Survey Bias: What it is and how to avoid it

Trustmary teamTrustmary team
Last edited: November 16th, 2022

Do you want to gather some information, but don’t know whether or not you’ll get honest responses and accurate results? This article will tackle the matter of survey bias and provide you with the best advice on how to avoid it in your research.

Disclaimer: Survey Bias Is Always Present

Before we go into why and how you can avoid response bias in your surveys, we should clear one important thing up: no survey is entirely free of bias. There are plenty of things that can impact the survey results negatively. 

Common Reasons Causing Bias

  • The survey takers are confused about your questions
  • Respondents can be tired
  • Some start the survey but are uninterested in completing the form
  • Technical issue prevents from submitting
  • Responses can be given without thinking about them much – or at all

You need to understand that not all forms of bias are preventable. However, it is your job as the survey creator to minimize it as much as you can to obtain more accurate data.

Now that you know this, let’s start at the beginning.

What is Survey Bias?

In the Cambridge dictionary, bias is defined in the following ways:

the action of supporting or opposing a particular person or thing in an unfair way, because of allowing personal opinions to influence your judgment

“the fact of allowing personal opinions to influence your judgment in an unfair way

When businesses create surveys, they’re bound to be influenced by each respondent’s personal experience. In general, the term survey bias refers to some kind of deviation of the results from the truth.

what is survey bias

Example of Survey Bias

As an example: If you only send an NPS survey to the customers you know are happy with your services, you’ll end up getting a very good NPS score. However, this isn’t an accurate representation of the performance of your company, when you purposefully left out all unhappy customers.

Bias occurs in even the best surveys, but it can be minimized to ensure the accuracy of most answers. While it’s impossible to eradicate bias altogether, there are some ways to minimize some forms of it. 

Every person’s opinion is different and subjective. This includes the opinion of the survey takers, as well as that of the researcher who creates the survey. 

In order to create unbiased surveys and influence the accuracy of survey takers, you need to tread very carefully. Next, you’ll learn how to handle and avoid the most common types of bias.

Different Types of Survey Bias and How to Avoid It

There are many different types of survey bias that you should be familiar with. Let’s take a look at the most common ones. The types can be divided into three main groups: Sampling bias, Questionnaire bias and interviewer bias.

types of bias in surveys

Sampling Bias

Sampling bias happens during the selection of survey respondents. The respondents are also known as the survey’s sample.

Sampling bias can be split into different types:

sampling bias in surveys

Exclusion Bias

Sometimes, you’ll decide to remove some participants because of their responses, or behavior, or they won’t come to the follow-up after filling out your initial survey. This can cause sampling bias because you won’t get complete information from some of the respondents.

How to Avoid Exclusion Bias

There’s just one way to avoid this and sometimes it is impossible. To the best of your abilities, try and keep the sample for the initial survey and the follow-up the same. Don’t remove people just because you dislike them or don’t have the time. 

However, you can’t really avoid this from happening if the respondents decided not to do the follow-up for their own reasons. What you can do is remove their initial survey responses from the data you collected altogether.

Representation Bias

Representation bias occurs when you select a sample that isn’t very inclusive. For example, if you walk down the street and ask the people passing by to answer your survey, you could exclude key members of the population or people that your survey is aimed for. It’s a very common type of sampling bias.

How to Avoid Representation Bias

There’s a simple solution to representation bias. Many researchers today collect responses by using digital sampling methods. You can select people digitally through email or social media to avoid the in-person sampling bias that often happens.

Pre-Screening Bias

In the rush to collect data, we often create pre-screening bias. Many surveyors gather volunteers that might or might not have the qualifications and characteristics that the survey requires. 

How to Avoid Pre-Screening Bias

One way to avoid pre-screening bias is to add screening questions to the sample selection process. This will allow you to screen out the people that aren’t the right target audience for your survey.

Survivorship Bias

We all want to get positive responses from customers regarding our work and their experience. However, you should aim to get accurate and honest and not focus solely on getting positive answers. Survivorship bias happens when you include only the customers that keep buying from you and fail to include the ones that you failed or disappointed.

How to Avoid Survivorship Bias

This is pretty straightforward. When you do customer satisfaction surveys, include different customers, including your current buyers as well as the ones that aren’t buying from you anymore. 

Most surveys request feedback only from current customers that are most likely to be satisfied and from potential customers that aren’t certain about how they feel about a brand. You need to include former customers also, people that you probably failed to satisfy.

This is the only way to get accurate data in terms of customer satisfaction.

Questionnaire Bias

Next on our list of bias categories is questionnaire bias. This includes every form of survey bias that is the result of how the survey questionnaire is built i.e. how you ask the questions.

Questionnaire bias can come in many forms. These are also known as response biases and, while in many cases they occur unintentionally, you can take some measures to avoid them.

To help you do exactly that, let’s consider the different types of survey response biases.

types of questionnaire bias

Acquiescence Bias

This type of response bias is also known as “saying yes”. It happens very often in surveys with too many questions or questions that are too lengthy. In such cases, respondents will start overly agreeing, and you can notice contradictory responses in their answers. 

How to Avoid Acquiescence Bias

You can’t really avoid this altogether. Some respondents will click on ‘yes’ or ‘no’ simply to finish the form or won’t bother to read the questions thoroughly. However, if you use a variety of question types to keep it interesting, as well as keep your survey short and to the point, you can minimize acquiescence bias. 

And while you might not be able to eliminate it altogether, you can get rid of some of the biased answers afterward, too.

When you’re reviewing the results, you will be able to detect contradictory answers in some surveys and learn where this type of response bias occurred. Those answers are biased and therefore can be eliminated from your data analysis

Self-Selection or Non Response Bias

Non-response bias occurs when respondents opt-out of completing the survey because of its content. Even with a perfect selection of the sample, some of the respondents can choose not to provide answers.

There are tons of reasons for this. Some people don’t like surveys. Others don’t use their email, so if you are doing an online survey, they might not receive your message. A third group might not like the brand and won’t be willing to spend time on it, or won’t understand the purpose of the survey. 

Essentially, this means that you won’t get the full sample that you planned and, if you have many unresponsive sample members, you can miss out on important data to analyze.

How to Avoid Non Response Bias

First of all, you must understand that, for every survey, there will be people that won’t answer it. Your goal here is not to force everyone to answer your questions but to keep the number of people that leave the survey small.

One way to avoid nonresponse bias is to remove the option for participants to opt out once they see the topic of the survey or the questions. However, this sounds and looks too forceful and can cause frustration in the respondents, which can lead to even more bias. 

Another way is to keep the topic or brand hidden throughout the survey to avoid people opting out. Also, you can offer some incentives to convince people to keep providing answers in your survey.

Social Desirability Bias

Social desirability bias is a very frequent form in surveys. Many of the participants will choose the answers that they think they are supposed to provide when completing your survey. In other words, they’ll choose answers that are socially desirable and might not equal their honest answer.

Some people simply want to be good subjects or fear that their honest responses won’t be socially acceptable. These people will try to discern what you hope will be their answer and provide it regardless of their real opinion. 

How to Avoid Social Desirability Bias

You can avoid social desirability bias by keeping the topic or brand hidden from the participants. If they don’t know what the answers will serve for or who they are for, they’ll respond more honestly. 

In addition to this, you should try to eliminate any loaded or leading questions that can prompt response bias. Finally, you should inform your respondents that their answers can be anonymous so that they won’t fear that they’ll provide socially undesirable answers.

Order Bias

The order in which you ask your questions and more importantly, the order of the answer options can bias the respondents toward making choices that are higher on the list. This is order bias. 

How to Avoid Order Bias

One excellent way to avoid this type of bias is to shuffle the responses and questions. Many survey platforms have this feature, and it will minimize order bias if you use it. 

Extreme Response Bias

Some of the people that will be filling out your survey will like extremes. There are such people so, if given the opportunity, they will pick the most extreme answer. 

How to Avoid Extreme Response Bias

No surveyor loves to get extreme responses. The solution is similar to that for acquiescence bias – ask different types of questions. Extreme responding is often the result of similar questions, the same type of questions asked all over again, or too many questions. You should minimize the number of questions and ask different types to avoid this. 

Neutral Response Bias

This type of response bias occurs when the surveyor creates non-specific questions. It also happens when the survey doesn’t evoke strong enough responses. So, the participants take a neutral position, especially when the survey comes in the form of a Likert answer scale.

Let’s face it – plenty of neutral answers don’t provide you with any actual data. A mixture of extreme and neutral responses is not good either.

How to Avoid Neutral Response Bias

There’s only one way to avoid this – ask the right questions. You need to be very specific and consider the sample when creating them. For example, if you ask respondents how they feel about different animals with a Likert scale, you can expect non-pet owners to be neutral. Pet owners, on the other hand, will most likely have extreme views on how much they like them.

Interviewer Bias

Sometimes the reason behind less than truthful responses lies in the person that asks the questions. You can get contradictory statements or dishonest answers because of the actions of the interviewer.

There are plenty of reasons why interviewer bias happens. The research participants can provide inaccurate answers because of how the question was asked, how the interviewer made them feel, etc. There’s even friendliness bias in case the respondent knows the interviewer, which can affect their responses.

This is why your choice of an interviewer is highly important. In this case, you cannot make the survey anonymous in a complete sense like you could with an online survey. However, it is the job of the interviewer to make respondents feel safe about answering honestly, comfortable during the interview, and provide more accurate answers as a result.

Here are some of the types of interviewer bias you might come across:

types of interviewer bias

Demand Characteristic Bias

If you are a participant in a survey in an interview setting, this can make you uncomfortable and affect your answers. Interviews are nerve-wracking for many people. If an interviewer isn’t able to create a comfortable situation for the participants, they can create this type of bias. 

How to Avoid Demand Characteristic Bias

To avoid this type of bias, you need to help participants feel comfortable in the interview. The interviewer’s job is to find a nice place to hold the interview, be nice and professional, and basically make them forget that they are in an interview. 

An unwelcoming setting for interviews can cause this type of bias. Also, if the interviewer fails to share the purpose of the study at the beginning, the interviewee can spend their time trying to figure it out instead of focusing on the questions.

Reporting Bias

Reporting bias occurs when the surveyors decide to report on the findings from the survey partially or not share the information because they are unsatisfied with the results. 

How to Avoid Reporting Bias

Not only is this unethical, but it destroys the whole purpose of your survey. When you’re implementing any type of survey, it’s important to be prepared to get some negative answers.

You should expect your respondents to answer honestly because that is the only way to get accurate, actionable data. If your respondents are biased to strongly agree with everything you ask, you aren’t really getting accurate information. 

7 General Tips to Avoid Bias from Your Survey Respondents

Now that you know about the different types of surveys, let us present you with 7 general tips that will help you minimize response bias. 

1. Be Selective with Your Survey Questions

This is one of the best ways to avoid different types of bias and encourage honest responses. Structure your questions openly and don’t add any unnecessary adjectives. Consider the respondent’s position in this. Would you feel comfortable selecting: I strongly disagree with the question you’ve asked?

There are a few types of questions that we recommend avoiding to minimize bias:

Loaded Questions

Such questions make assumptions that might or might not be accurate. For example, if you ask this question, you are asking a loaded question:

example of loaded question

Some of the respondents might not wake up in the morning, so their answers will be biased. 

Leading Questions

Leading survey questions are pushing the respondents toward the desired response. To avoid them, remove language that implies intent. Don’t write anything positive or negative about your products and services, but give them the chance to share their opinion.

If you ask the following, you are asking a leading question.

example of leading question in survey

What if they don’t think your customer service was amazing?

Prestige Questions

When people get statements and learn that an expert said something, this can influence their answer and convince them to agree with that expert. For example, if you say the following, many people will be prompted to agree:

example of prestige questions in surveys

Double-Barrelled Questions

These survey questions are the most confusing of them all and they almost always result in bias. They happen when you blend two questions into one, and the respondents are given a chance at just one answer.

Tempted to ask more questions in a shorter survey, many researchers do this to save space. Still, questions like the following can have two different answers and lead to bias.

example of double-barrelled question

Stereotype Questions

It’s not uncommon to stereotype respondents. This is often done unintentionally. Racial, gender and social stereotypes do exist. If you don’t try to eliminate these, you can get many inaccurate responses.

To avoid this, remove any language from your survey that can sound stereotypical, and never ask respondents to describe themselves by age, ethnicity, religion, or age.

2. Give Them the Option to Elaborate

Many of the questions asked can have more complicated answers than just yes or no. One of the best survey research methods is to add a field to your surveys or give interviewees space to elaborate on their answers. They will often read your question in the online surveys and think: Yes, but…

3. Provide Background Information

A bit of context can clear a lot of things. If you want to avoid people giving biased answers because they misunderstood or don’t understand the question, add some information to it. Don’t go overboard, though, since they won’t be willing to read through tons of content.

4. Choose the Sample Carefully

If your surveys aim to gather data about a specific buyer persona, you need to find the right people as well as tailor your questions to the audience. You need to consider where you’ll reach them and how, too.

For example, if you are trying to learn how people feel about your applications, you won’t send your survey to the ones that haven’t used it for years. Respondents tend to forget about their experience in a while, so you’ll get inaccurate answers from them.

Another good idea is to avoid relying solely on volunteers because this often causes voluntary and non-response bias.

5. Give Them an Out

In addition to different answer options, you should give people the option not to answer. People should not feel forced to fill out your survey. If they don’t have a choice but to answer your questions, they will most likely pick any answer without giving it much thought. 

6. Disregard the Obviously Biased Surveys

When we discussed the different types of bias before, we mentioned that you can notice it sometimes in the answers. It is easy to notice biases such as neutral or extreme. When you see that the respondent kept choosing one of the first few answer options in every case, or that they selected neutral for every answer, it is safer to remove their responses entirely from the data set. 

7. Ask Different Question Types

Unless you want your answers to be filled with selection bias or people to keep leaving your survey, you need to keep it interesting and organized. Mix your strategies in it. Do not ask similar questions that you can do without, and use a variety of question types to keep them engaged. 

Benefits of Unbiased Responses for Your Survey Research

Survey bias can cause endless problems for researchers. If you manage to minimize these, you will obtain data that accurately reflects the participants’ opinions, which is basically the reason why you’ve been conducting surveys in the first place. 

benefits of minimizing survey bias

Some other benefits of unbiased responses include:

  • Actionable data. Unbiased responses equal accurate data. Based on it, you can take actions that will improve your brand and user satisfaction. 
  • Smart investments. Imagine making a big investment and losing a fortune just because you trusted the biased answers that resulted from a poorly-crafted survey. If you have unbiased responses, you can get amazing insights and decide how to invest your money, resources, and time wisely.
  • High return on investments. Bad information leads to bad performance and product quality. If you obtain the wrong information, you can destroy your brand’s image instead of fixing it. If you ask the right questions, you can make the right investments and get a high return on them. 
  • Higher satisfaction. People love brands that care about what they think. Many will appreciate you taking the time to ask them questions, but they’ll appreciate it even more if their answers are really heard. If you manage to obtain honest answers and use them to improve your brand, you’ll be building loyal customers as a result. 
  • More satisfied investors and stakeholders. Investors and stakeholders will get accurate information if you avoid bias. If you don’t, they can easily be dissatisfied with your performance levels and may decide to reduce the research budget soon afterward. 
  • Cost-effective. If there’s bias in surveys, these can easily turn into inconclusive research. As you know, these cost a lot. It takes people time, money, resources, as well as tons of effort to create and implement surveys for the right people. You might not get a new shot at it if your first try doesn’t succeed, both because of financial limitations and lack of interest in your respondents. After all, you can’t expect them to take survey after survey because you cannot avoid bias.

Further Reading

Creating surveys can take time and effort, but it can offer you information about your brand and customers like no other tool. To learn more about the types of surveys you can create, you should read this guide on surveys.

To avoid bias, you need to know how to recognize it first. Take a look at these examples of survey bias to learn what exactly it is that you should work to avoid.

Lastly, this guide will teach you how to ask the right questions in your survey.

FAQ

How can I know if my survey is biased or not?

If your survey questions are biased, they’ll be phrased, formatted, or skew people toward a certain answer. They can also be unclear, hard to understand, ask more questions than one, and basically make it hard for customers to give an honest answer. 

Does bias affect the validity of my surveys?

It most definitely does. Bias exists in all research and it is difficult to eliminate. It’s important to try and minimize it because it can impact both the reliability and validity of your findings. 

Can I remove bias altogether?

No. Removing all types of bias from your research is impossible. There will always be some kind of bias that will not be possible to prevent. However, if you manage to minimize it, you can expect more accurate answers and insights. 

How can I reduce nonresponse bias?

If your surveys aren’t getting as many answers as you wanted them to, you need to edit them. You might want to consider providing an incentive. You should mix up the questions and ask different types of them. Most importantly, your survey needs to be short and simple so that people will be willing to dedicate time to it. 


Trustmary team
Trustmary team


Trustmary

Trustmary is the most effective way to convert more sales by improving digital trust.