Online Surveys = Ongoing Insights

Question Order Matters

Question Order Matters

The order questions appear in your survey can directly impact the responses you gather. One of the more well known examples (Iyengar & Kinder, 1987) of question order effects is in the domain of politics. When pollsters ask people, “What is the most important problem facing the nation?,” the answer they give becomes the object of focus for their answer to a subsequent question—“Do you approve or disapprove of the way [Barack Obama] is handling his job as president?” By and large people answer the approval question while judging the president primarily on his performance on the issue they consider to be most important.

This phenomenon is called priming. Respondents are primed to think about one issue while answering the subsequent question.

Priming has an impact even if the survey topic is not political, or even controversial. Let’s take a seemingly harmless survey. Let’s say I’m trying to gauge sentiment for a company softball team. I may be tempted to ask “What is your favorite sport?” as part of my survey. Seems like a fair start. But if my next question is, “How interested are you in playing on a company softball team?” and softball isn’t high on your list of favorite sports, then you may be tempted to rate your interest lower in joining the company team than if I had asked you first about the team, then about your favorite sport.

Another reason question order matters is that respondents may have a desire to appear consistent in their responses. For example, if you ask students to answer a very difficult math problem first, then ask how much they enjoy math, they may be tempted to rate their interest lower if they struggled to solve the math problem.

Even response options from an earlier question can impact subsequent answers. If I ask “Which of these four fruits is your favorite?” and then ask how much fruit you eat in a week, you may focus your attention on just those four items and report a lower number than if you were to think of a longer list of fruit.

How can you address question order effects in your own surveys? One option is to randomize your questions so that respondents are not all answering questions in the same order. My colleague, Tim, will be posting this week on one of our NEW advanced logic features, Question Randomization, that allows you to do just that—randomize questions to reduce question order effect. Have additional questions on why question order matters? Please feel free to ask me in the comments below.

Do a Better Job with SurveyMonkey Enterprise

Give yourself—and the rest of your team—the answers you need to make smarter decisions. Access our most powerful tools with SurveyMonkey Enterprise.

Learn More →

Tags: , , ,

  • Pingback: New Advanced Logic Feature: Question Randomization | The SurveyMonkey Blog()

  • Jeroen Egelmeer

    At the moment I’m busy with my graduation project about a survey from a large company. Now I think that one questions in the should be asked as first to avoid a bias. (it’s now on the fourth place) The only problem I have is to convince the company to put that specific question to the first place. So my question is, do you have some scientific information for me to convince the company to change te place of the question?

  • Felicity

    I would like to design a survey where the answers to some of the questions will be in other questions later on. Therefore, it is important that people cannot go back and change their answers. Is this possible?

  • dimondn

    I am new to surveys but wondering if someone might opt out of a survey if unable to go back and adjust answers to questions.
    Is that a factor?

  • Matt

    I am in the process of creating a survey using several already existing measurement scales. However, I have to counterbalance the order in which each scale is given without changing the order of the questions within each scale itself. Is there a way for me to do this?

    • Phil G

      Hi Matthew,

      It sounds like you want to minimize order bias by offering flipped or rotated response options, at random, to your respondents. Flipping involves presenting answer choices as either 1, 2, 3, 4 or 4, 3, 2, 1. Rotation requires showing, for example an order of 1, 2, 3, 4, 5 for the first respondent and 2, 3, 4, 5, 1 for the second respondent and 3, 4, 5, 1, 2, for the third respondent etc. Both methods guard against satisficing.

      Both of these options are available in SurveyMonkey.

  • Susan

    I would like to create a brief survey regarding students visits to the tutoring lab that I oversee. I have approximately six questions, but any feedback on setup would be appreciated. I am looking at such things as helpfulness of the tutor, convenience of lab hours, how can the tutoring be improved, and so forth.
    Thank you!

    • Hi Susan –

      Try our new Question Bank. When you go to create the new survey, take a look at the Question Bank options on the left.
      Or, you can use one of our expert survey templates. I’m not sure if we have one that fits the tutoring lab exactly, but the University Student Satisfaction might work or at least give you a starting point.

  • Sandy K

    How do I get construct my survey so that Survey Monkey gives me a total point value and the end of the survey. For example, we are asking questions with a 1 to 5 point value. There may be 15 questions and I want to know the points scored for each question and the total points for all questions. For example If someone strongly agrees that is 5 points. Also is it possible to give a weighting percentage factor to the questions and have Survey Monkey provide the weighted total?

  • Minoo

    Do you have any recommendations that how many randomised versions of a questionnaire should be produced in order to reduce the bias? (e.g. for 10 questions with 100 sample size).

    • Hanna J

      What size sample you need depends on what population you’re trying to estimate, how accurate you need the estimate to be, and how many variables you’re estimating.
      For example, for our population of roughly 200,000 pro subscribers we would need at least 750 responses in our sample to estimate ONE question.
      Want to add more questions? You’ll need more sample. (For a basic estimator tool to figure out your own sample needs, check out this page:

  • Brashani

    I am currently working on a research project where we need to counter-balance two portions of the online survey. Does SurveyMonkey allow you to do this? Thank you!

  • I have really learned new things from your blog post. Yet another thing to I have noticed is that in most cases, FSBO sellers will reject a person. Remember, they will prefer to not use your services. But if a person maintain a gentle, professional connection, offering assistance and staying in contact for about four to five weeks, you will usually have the capacity to win a conversation. From there, a listing follows. Cheers

    • Kayte K

      We’re glad you found this helpful and thank you for the feedback!

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in

Write Surveys Like a Pro

Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »