Bias is the mortal enemy of all surveys, and as a survey creator it’s important to guard against it to make sure you get reliable results. Over the years we’ve developed many tips to promote honest answers from your respondents, and best practices for designing surveys that use unbiased wording, structure, and styling. But if you’re not careful, there are a few ways you can still introduce bias without even knowing it.
One of the leading causes of misleading survey data is researcher bias that comes directly from the survey writer. This bias is sneaky. It’s caused by survey creators who innocently influence the results to reach an outcome they hope or expect to reach. It’s sneaky because survey creators are typically unaware it’s happening.
Can you avoid this kind of bias?
Don’t worry! We have you covered with a researcher bias prevention list. Below are four things to avoid when conducting your survey:
1. Asking the wrong questions
It’s impossible to get the right answers if you ask the wrong questions. Unfortunately, survey results are easily compromised by questions that fall short of capturing the entire scope of a survey’s issue. Say, for example, your survey was created to understand your employees’ favorite type of pizza. You ask, “Do you like pepperoni, meat lovers, or vegetarian pizza the best?” Though there are many other types of pizza, they did not come to your mind and were left out of the question. Now instead of measuring the most popular pizza, the study measures the preference between these three types.
Tip: Exploratory research is the best way to make sure your questions are exhaustive and on point. By first surveying a small group with open-ended questions on your subject, you’ll gain a better perspective of the scope of your survey topic, and be less likely to overlook options that may matter to respondents. You may also want to review similar surveys to learn what categories and topics were popular with respondents in the past.
2. Surveying the wrong people
Choosing your respondent group may seem like a no-brainer, but it often leads to something called selection bias. When conducting a survey, it’s imperative to target a population that fits your survey goals. If you incorrectly exclude or include participants, you may get skewed data results.
Usually this bias happens when you lack of a clearly defined target population. For example, say you want to limit your survey to people with low economic standings. This population could be defined in many ways: people with low income, people who lack disposable income, or people who have a low net worth after taking into account their property, income, and debt. Each of these three descriptions can successfully be used to address the broad population you hope to reach. But, each definition could provide different results for your study.
Tip: To avoid surveying the wrong people, make sure you clearly define the respondent requirements you need to meet your survey objectives before beginning your project. This step will give your survey results a proper scope. Also remember to be specific in your reports and findings when referring to your population. Using broad terms like poor, rich, large, or small can lead to misinterpretation.
3. Using an exclusive collection method
Some surveying methods can make it difficult, or even impossible, for certain people to take part in your study. For example, if you survey commuters you meet walking around on the street, you might not get a representative sample of people who drive or ride bicycles. By excluding potential respondents in a non-random way, you can instill bias into your survey if the people who aren’t part of the panel have views that differ than those who are.
Tip: The best way to limit this type of researcher bias is to give all potential respondents an even chance to participate in your survey. In our commuter example, you might be better off sending an online survey to everyone who lives in your town, or asking some local businesses to send your survey out to all their employees.
4. Misinterpreting your data results
This form of bias is introduced when raw data is transformed into misinterpreted findings. Usually it’s a case of inappropriate or inaccurate statistical techniques, which lead to the incorrect interpretation of the survey results. For example, bias can come into play when a survey creator gets excited about a finding that meets their hypothesis but overlooks the fact that the survey result is only based on a handful of respondents.
Tip: Most often, this form of bias is caused by gathering information and then later developing your data analysis strategy. To avoid this type of bias, create a data analysis plan before you write your survey, then write questions that you know will work well with the analysis you have in mind. For example, try creating multiple choice questions with rating scales that will work well when you want to quantify your results. It also helps to take note of the different analytical tools available to you from your survey software before you create your survey.
Avoiding these four mistakes may seem difficult at first, but by remaining true to your survey’s purpose and having a firm understanding of the topics of your research you’ll be well on your way to eliminating researcher bias from your survey.
Put simply: You need to do some planning before you start your survey. Take a second to think about each of the four points above and whether your survey plan addresses them. Once you’re sure that your research methodology is sound, you can rest easy that your final results won’t lead you astray.
Questions for Rick? Let us know in the Comments below!
Tags: survey methodology