The process of collecting feedback is a dialogue between an interviewer and a respondent. And like in every dialogue, there can be misunderstandings. Due to the indirect form of the conversation, they become even more troublesome than usual.

If you want to receive valid and useful feedback from your survey, you need to express your thoughts clearly and concisely, avoiding biased questions.

What is a biased question?

A biased question is an unquiry asked in a way to solicit a certain answer from a respondent or unclear to them. Sometimes, they are born from wrong choice of words and questions order, sometimes - caused by social and psychological factors. Both surveyor and respondent might be unaware of it, still falling to its trap subconsciously.

Biased questions prevents people from sharing honest impressions and damage the feedback received. Although eliminating them completely is next to impossible as you cannot predict the reaction of each person, but knowing what types of bias are out there will help you watch out for them and mitigate their influence.

Types of biased questions

Just like there are many ways to affect a respondent's choice, there are many different types of biases and questions born from them:

• Double-barreled question
• Absolute questions
• Leading questions
• Loaded questions
• Dichotomous questions
• Vague questions
• Social desirability bias
• Demand characteristics bias
• Assimilation effect
• Contrast effect
• Acquiescence bias
• Straightlining.

Double-barreled questions

A double-barreled question combines two questions in one and asks for one answer to both. It is easy to miss because adding more information often seems like a good idea.

This sentence asks about two different matters: the effectiveness of the materials and their structure. It’s not necessarily that both of them are on the same quality level. A respondent might find materials extremely helpful, but not very well organized. Thus, they will be at a loss as to what to answer. And when you will be reading their replies, you also won’t be able to pinpoint which of the two matters caused problems.

How to avoid double-barreled questions? Try to be concise and simple whenever possible. Stay away from long descriptions that use several words to define one point. Make sure that you address only one side of the topic in each field.

To check yourself, imagine how you would be analyzing results. If you receive a negative answer, will you be able to understand where the problem lies?

Absolute questions

Absolute questions include such words as “all, always, every, none, never”. These ultimate terms carry an extreme connotation making people want to deny it.

This question sounds too pushy. If a respondent liked only some of the tasks, their first impulse will be just to say No, without looking at the other options.

For someone, it might sound almost like you are accusing them of not participating in your events earlier.

How to avoid absolute questions? It’s best not to use absolutes in your form except when it’s unavoidable. In most cases, these words can be omitted or the sentence can be rephrased.

Leading questions

A leading question is a statement that a surveyor wants to be confirmed or denied asking it in a way that leads to a certain answer.

It’s obvious from this sentence what answer is expected. You risk receiving negative feedback or no answer at all because no one likes being manipulated.

The example above expresses an interviewer's preferences explicitly. But not all leading questions are that obvious.

At first glance, there are two choices — old and new programs. But are there really? This sentence clearly favours the new program by emphasizing that it was enhanced. Whether your respondents subconsciously follow the lead or realize it — the outcome will be biased anyway.

How to avoid leading questions? Carefully check if you give respondents enough freedom. Do not include your personal opinion keeping the sentences neutral.

Loaded questions

A loaded question contains a certain assumption that might be false for some respondents. It is similar to a leading question as it implies the direction of an answer too.

The interviewer is certain that all participants have enjoyed the event and all that’s left is to find out to what degree. What if someone did not enjoy it at all? Even if you include the corresponding option in the list of responses, the question will still leave a negative impression.

How to avoid loaded questions? The countermeasures are similar to those for leading questions. Take all possible outcomes into account and stay neutral allowing respondents to decide for themselves.

Dichotomous questions

Dichotomous questions always have only two possible answers, with yes/no being the most frequent options.

Using two definite options is tempting because it makes the analysis of the results an easy task. But at the same time, it backs some of the respondents into a corner. They won’t be able to give a reply if they have not decided on their future attendance yet. Consequently, they will either drop out or pick an answer randomly.

How to avoid dichotomous questions? Include the middle option that expresses the uncertainty or neutral position — for example, “Maybe”. Or give a way out by adding the “I don’t know” answer. It’s not that convenient for analysis but provides a suitable choice for those who do not have a clear opinion on a matter.

Vague questions

In contrast with the previous types of bias that limit respondents’ freedom, a vague question gives too much of it.

A vague question includes words that can be interpreted in different ways: "good, bad, interesting, many". People define them depending on their values, beliefs, experience, and goals. If you use these words in your questionnaire, there is a high chance that each of your respondents will be answering their own question and you won’t know which one.

Are you sure that you and event participants have the same understanding of success? For you, the criteria might be delivering new knowledge and teaching new skills. And for others, it might be an engaging and lively discussion or the level of organization.

How to avoid vague questions? Focus on the qualities and parameters that matter the most for you and ask as precisely as you can. Try not to leave the room for different interpretations.

Social desirability bias

This type of bias is different from previously mentioned biased questions. Social desirability is a response bias that makes the survey participants choose a socially-approved answer.

People want to look good in front of others. Even without realizing it, they usually deny traits that can be viewed negatively by those around them. This makes it difficult to receive honest answers to sensitive questions and to make accurate statistics.

It’s considered that learning and self-improvement are essential qualities of any successful person. If a respondent doesn't have any intention to learn anything new, there is still a high chance they won’t admit it and give you a positive answer that would be socially approved.

How to avoid social desirability bias? Making a questionnaire anonymous helps people relax and share honest thoughts.

If anonymity is not an option, at least assure participants that you will keep this information private (for example, in an attached letter).

You can also try to rephrase a sentence to make it sound less aggressive.

If you list possible directions and include “I haven’t decided yet” or “I don’t know” options, it will help two groups of people. You will give hints to those who want to grow but haven’t yet decided how to proceed. And you will add a way out for those who do not intend to raise their skills for now.

Demand characteristics bias

Demand characteristics bias make respondents give answers based on their assumption of what you need the most. They do not reflect on their actual thoughts and emotions but try to pick the most favorable answer.

Respondents might think that they help you by choosing “Yes”, so they will mark this reply solely because of this reason. At the same time, they could be satisfied with the current location and don’t see any need to change it.

How to avoid demand characteristics bias? Do not leave the ground for assumptions and stay as neutral as possible. Ask participants to share their impressions on the current location or write their suggestions on what can be improved on this matter.

If you get a lot of negative feedback, changing a location can really be a good idea, even if you don’t ask about it directly.

Assimilation effect

Survey results can be negatively affected not only by the wording of questions but by the survey structure too.

The assimilation effect takes place when a respondent uses their reply to the first question as a basis for a reply to the next question.

Let’s look at the order of the following fields:

Knowledge of the topic can be considered as a part of a trainer's performance, which means that the detailed question here precedes the general question. In this situation, when a participant is asked to rate the performance in general, the first thing that comes to their mind is how knowledgeable the trainer was because it has been mentioned just before. And they might give their rating based on it alone, ignoring all the other aspects of the performance.

It happens because our mind always looks for a shortcut. To find an answer, we go through the data stored in our memory. Of course, we stumble upon the most recent information first. If it suits our purposes, we tend to use it and don’t try to look further.

But the most recent information is not always what we need. A participant might be impressed with the trainer’s profound knowledge, but have complaints about the presentation style or the way the trainer interacted with the audience. If they rate the performance high thinking only of knowledge, some serious problems might go unnoticed.

How to avoid the assimilation effect? Placing a general question first is the easiest solution. It helps get rid of the assimilation effect because this way you don’t give respondents any hints.

If there are several detailed questions related to one topic, it’s safe to put them before a general question. A respondent will have a broader picture and will be less susceptible to the assimilation effect.

If, for some reason, you still need to put one detailed question first, and a general question second, try providing an explanation for the second case. For example, elaborate on what you mean by performance in brackets. Not a very elegant way out, but still better than invalid results.

Contrast effect

Contrast effect occurs when in their next answer, respondents exclude the aspect they were asked about before. That’s because they think that if you have already mentioned this one aspect you mean something else in the next field.

In their answer about the event’s venue, respondents might not take transportation into account. Thus, you will get confusing data when participants had various troubles with getting to the location, but at the same time, they gave the venue 5 stars out of 5.

How to avoid the contrast effect? The contrast effect can be avoided the same way as the assimilation effect — either put a general question first or precede a general question with several detailed questions.

Acquiescence bias

In some cases, the problem is not how you ask questions or position them. Sometimes, respondents don’t feel like answering so they just pick options randomly.

You can’t really control their motivation, but you can at least help them find it. One way is to explicitly convey the importance of their feedback and design a feedback form with respondents’ convenience in mind.

Read our recommendations on how to collect event feedback effectively.

And another way is to wisely use question order and question types to keep respondents focused.

Two effects caused by low motivation are acquiescence and dissent biases. Acquiescence or agreement bias means a respondent agreeing with everything in a survey. And dissent bias is a respondent always choosing “No”.

How to avoid acquiescence bias? Maintain the balance by using different question types. It will help participants not to get bored while answering them.

Do not ask for agreement or disagreement and offer different options instead.

There is still a chance that some respondents give random replies, but others will feel interested to share their real thoughts.

Straightlining

Straightlining means marking the same option in all questions. It stems from the same reason as the acquiescence bias — respondents are not motivated enough to think about their replies.

It happens with ratings but can affect multiple choice questions as well.

How to avoid straightlining? The advice is the same here — the balance of question types. It’s better to avoid several rating scales in a row or matrix questions, asking to rate several aspects.

It will be harder for participants to click on the same answer, if a multiple choice question is followed by an open-ended question, then a Likert scale. Or when answer options in multiple-choice are different from the previous field.

Conclusion

It will be harder for the survey bias to get in your way if you stay alert of it. Step back to look at your questionnaire from your respondents’ perspectives before sending an email with a survey link or handing out a form.

If you are collecting feedback for your event, staying impartial might be difficult because you have put so much effort into it and you are rooting for its success.

But that’s exactly why you have to keep your head cool and your questions neutral. It will allow you to transform surveys from a mere formality into a really powerful tool that will support your events and give them additional value.

And to enhance this tool even further, you can automate the whole process of collecting feedback — sending evaluation requests, structuring and storing reviews, and converting them into testimonials.

You can easily do it with Workshop Butler's evaluation tools — a set of features designed specifically for the needs of training businesses.