The order questions appear in your survey can directly impact the responses you gather. One of the more well known examples (Iyengar & Kinder, 1987) of question order effects is in the domain of politics. When pollsters ask people, “What is the most important problem facing the nation?,” the answer they give becomes the object of focus for their answer to a subsequent question—“Do you approve or disapprove of the way [Barack Obama] is handling his job as president?” By and large people answer the approval question while judging the president primarily on his performance on the issue they consider to be most important.
This phenomenon is called priming. Respondents are primed to think about one issue while answering the subsequent question.
Priming has an impact even if the survey topic is not political, or even controversial. Let’s take a seemingly harmless survey. Let’s say I’m trying to gauge sentiment for a company softball team. I may be tempted to ask “What is your favorite sport?” as part of my survey. Seems like a fair start. But if my next question is, “How interested are you in playing on a company softball team?” and softball isn’t high on your list of favorite sports, then you may be tempted to rate your interest lower in joining the company team than if I had asked you first about the team, then about your favorite sport.
Another reason question order matters is that respondents may have a desire to appear consistent in their responses. For example, if you ask students to answer a very difficult math problem first, then ask how much they enjoy math, they may be tempted to rate their interest lower if they struggled to solve the math problem.
Even response options from an earlier question can impact subsequent answers. If I ask “Which of these four fruits is your favorite?” and then ask how much fruit you eat in a week, you may focus your attention on just those four items and report a lower number than if you were to think of a longer list of fruit.
How can you address question order effects in your own surveys? One option is to randomize your questions so that respondents are not all answering questions in the same order. My colleague, Tim, will be posting this week on one of our NEW advanced logic features, Question Randomization, that allows you to do just that—randomize questions to reduce question order effect.
Have additional questions on why question order matters? Please feel free to ask in the comments below.