How to Minimize Question Order Bias in Your Surveys

What is Question Order Bias? 

Order bias is a term to define a condition in which the order of your questions and answer options can affect how respondents give feedback.

For example, respondents are more likely to choose options higher on the list.

Moreover, sometimes survey takers give context to certain questions, affecting the following questions. In return, respondents will be influenced by it as they answer the next one.

Practical Example of Question Survey Bias

You ask how many times respondents exercise in a week. 

Then, in the next question, you ask them how to develop a healthy lifestyle. 

In the second question, their first answer is more likely to be exercising rather than eating healthy food or getting enough sleep.

As order bias can deviate the survey result, survey takers must carefully sort their questions and answer options. One effective way is by reshuffling the questions and answer options sequence. This means your questions will appear in random order for every respondent.

With the world getting more sophisticated, tackling question order bias is now only a simple step. Many survey platforms provide reshuffling features, so you don’t have to do the job manually.

Why is Question Order Important? 

It’s in human nature that specific actions may influence others. 

• kids mirror their parents, 
• encouraging utterances improve self-esteem, and 
• CTAs in videos convert audiences.

Those examples prove that people’s actions greatly depend on circumstances and surroundings. But, you don’t want that to happen in your survey. The goal of your survey is not to influence respondents with your question sequence to be able to generate actual data.

To sum up, the order of questions you have can significantly affect how respondents perceive and answer your questions. Your data can be inaccurate and misleading if you’re unaware of your question order.

How Can Question Order Bias Affect Your Survey Results? 

As mentioned previously, the order of your questions may define respondents’ feedback. 

Researchers have concluded that order bias can affect respondents in choosing or giving answers.

The order bias comes in two types—the first type of bias deals with the relation between the question order and the specificity level. 

For example, the initial question can influence respondents’ feedback on the following questions. Or, the answer options are tendentious for respondents who appeal more to the higher options or vice versa.

The other bias type relates to biased questions which aren’t randomized. Once you randomize them, you can see significantly different data. The second type often happens when surveying campaigns or concepts. 

6 Strategies to Minimize Question Order Bias 

We know it’s impossible to obliterate survey bias completely. However, as a surveyor, you should know how to tackle this problem to result in accurate data. 

It’s vital to remember that you will use the data for particular purposes, such as building product development or marketing strategy. In other words, you’ll be wasting time if the data you base your decisions on is inaccurate, to begin with.

To help you overcome this hurdle, we’ve listed the best practices to minimize question-order bias in your surveys. 

Let’s break down each of them and reveal more details.

1. Reduce the number of scale-type questions  

5-scale questions are one of the most-used types of close-ended questions in a survey. Naturally, it has advantages and disadvantages for survey takers and respondents.

This type is also known as the Likert scales, invented by Renis Likert, an American psychologist. Likert refers to respondents’ statements within some degree of agreement and disagreement.

This particular survey has long been famous among businesses. The Likert scale is suitable for B2C and B2B as it allows them to quickly find out customers’ behavior and satisfaction.

Businesses commonly use the 5-point scale to describe respondents’ attitudes that scratches between strong disagreement and strong agreement. Here is an example of 5-point scale options.

• Strongly Disagree
• Disagree
• Neutral
• Agree
• Strongly Agree

However, the Likert scale survey also has some downsides and limitations, one of which can lead to biased answers. Below are some disadvantages of having scale questions in a survey.

• The scales are vulnerable to accuracy as respondents can’t express their opinions in detail.
• Sometimes the proceeding questions influence respondents in giving answers to the next one, which turns into bias.
• People tend to avoid extreme degrees (strongly disagree or degree) and choose the less extreme ones to be more neutral.

The above three points can affect your survey results, leading to answer bias and misleading data. 

To minimize those disadvantages, try the following:

• Segment respondents by demographic 
• Give room for respondents’ opinions
• Encouraging respondents to elaborate their answers
• Showing gratitude to respondents for taking the survey

The scale questions and answers can’t be inseparable in businesses-related surveys in most cases. 

It helps them gather quality data in a relatively short time and analyze them quickly. But, you can also minimize the use of this type of survey to avoid survey bias.

2. Make your questions as engaging as possible  

Engaging survey questions are key to make respondents give sincere answers, but making one has never been easy. 

We’ve put together some of the best ways to structure engaging survey questions for your audience.

Evoke Emotions

One effective way to create engaging survey questions is by evoking emotions. You can use powerful action words, significant numbers to show data, and strong adjectives (positive and negative).

Combining those features will help you create engaging survey questions. That way, respondents will be intrigued to give accurate and sincere answers.

Make questions easy to understand 

Questions with good readability help respondents digest the context better, resulting in honest answers. The way you deliver and structure sentences in your question are essential.

Your respondents come from different educational backgrounds. , Keep in mind that making sure everyone can grasp and understand your point is vital. 

• Use short and simple sentences as long it’s possible
• Use more active sentences
• Use subheadings to separate topics and bullet points to make a list. That way, respondents won’t face hard times filling out your survey. In addition, consider using writing tools, such as Grammarly, to improve your readability.

3. Randomize your questions and answer options  

Randomizing survey questions and answer options is the best way to avoid bias. 

Many survey platforms have randomized features that enable you to resequence and reshift questions and answer options easily and quickly.

If you run a survey to research your target market, consider randomizing three vital things: 

1. the order of questions, 
2. page, and 
3. blocks. 

Here’s why.

Question Randomization

When you do concept testing, it’s better to switch the subject order. 

For example, you’re comparing two ads and asking respondents for their opinions related to them.

If you put ad A always higher on the list in every question, it will develop a bias in respondents’ answers as they may value it more. 

However, constantly switching the order between ads A and B can minimize bias and you’re more likely to see varied answers. This is because the first ad shown can influence respondents’ viewpoints and answers to the following questions. 

Page Randomization

Suppose you try to find out the effectiveness of some ad videos. Rather than putting all videos on a page, spreading them on different pages is better.

So, each video will have its own page. Not only does it avoid bias, but you can also add open-ended questions as a follow-up to generate more comprehensive data.

Block Randomization

Continuing the previous point, consider doing block randomization if you have many survey pages consisting of a video or picture.

Each page represents a block so that you will have several blocks from your pages. Block randomization will help you reorder the pages or give a single random block to every respondent.

4. Group your questions around a common topic  

You can minimize survey bias by providing grouped questions. 

It refers to those questions that allow respondents to choose more than one answer. For example, choose our best products (pick up three). 

Research suggests that question grouping has higher reliability and facilitates hypothesis guessing. Most importantly, question grouping can reflect more of the “true” picture of respondents’ feedback.

5. Adding branching in surveys  

Branching in a survey can help you collect relevant data from the most relevant respondents. In branching surveys, certain questions appear only if respondents are correlated to them. If not, they’ll be redirected to another question.

For example, you ask whether respondents have ever purchased your products. Those who answer ‘yes’ will go to the next questions like ‘what products did you buy.’ 

On the contrary, those who never purchase your products will go to another question like ‘choose the reason why you don’t want to buy our products,’ or which of our products you’re more likely want to buy,’

6. Pre-test the survey questions  

A survey pre-test means trying out your survey to a small group of respondents before running it for real. Doing a survey pre-test can give more benefits than you think. 

Through testing, you can check and spot some problems and fix them immediately.

This way, it becomes less problematic compared to if you need to get things fixed in an actual situation. You can save more time, develop a more robust survey, and, more importantly, keep the headache away.

There are some efforts that you can make to run a pre-test. You can run it to your colleagues for a smaller survey. Or, run a pilot test with a larger number of respondents from the sample population. 

Takeaway 

Among many survey biases, question order bias is one of the most important to consider. 

Some may think it won’t affect much of your survey results. Still, some studies have proved that question order can influence correspondents’ attitudes.

If such a bias happens, it will be a waste of effort since you will only generate inaccurate and misleading data. Therefore minimizing question order bias is essential. 

The above points can be your guide to avoiding question order bias and running a comprehensive offline or online survey.

Further Reading

• The Ultimate Guide To Surveys
• Survey Bias: What it is and how to avoid it
• What is Non-Response Bias and How to Overcome It?