Online Surveys = Ongoing Insights

Order Matters: Don’t Let it Get In Your Way

 → 
 → 
Order Matters: Don’t Let it Get In Your Way
surveyscience

Pop quiz!

Read this list of words quickly: pen, airplane, shell, butter, duct, blueberry, wrapper, zebra

Now cover the list with your hand.

Now list all of the words you remember.

If you’re like most people, you’ve remembered the first one (pen) and the last one (zebra), but not so many in between. How did we know that? Well, the order of a list can be just as important as the actual content. The order itself changes how you think and respond.

The same thing happens in surveys when you give respondents a list of answer choices. The first and the last answer choice stick out, and the middle tends to get ignored. Take this question for example:

Best Young Adult Literary Series?

More respondents will pick Harry Potter and Percy Jackson than would have otherwise simply because they are the first and last choices. In survey language, that’s called a response choice order effect.

Why might people pick the first response choices?

  1. Satisficing. Respondents start with the first option, thinking about all the reasons the Harry Potter series might or might not be the best series of the last ten years. They then do the same thing for the rest of the options on the list. As respondents go down the list, they start getting mentally tired. They then begin to satisfice–they stop paying close attention and skim over the last options in the list, neglecting to consider them carefully. This can leave the top response choices in the list with a much stronger case than the bottom ones.
  2. Priming. We talked before about how this works with questions. Ask someone their income and that’ll sway how they respond to a question asking if they believe in the American Dream. Believe it or not, the same thing happens with response options too. For example, if Harry Potter is first on the list, respondents might first think about how the series made billions of dollars. They’ll then use this same standard–which is pretty unique to Harry Potter–to judge the other options in the list. None of the other options made billions of dollars, so Harry Potter is left as the easy choice.

Why might people pick the last response choices?

  1. Satisficing. Yes, satisficing rears its ugly head here too. Again, respondents do the hard critical thinking work for the first few options, but get mentally tired and begin to satisfice. When they stop thinking critically about the last few choices, this can actually make them seem more appealing, not less. It becomes more of a gut instinct selection than a rationally thought-out opinion.
  2. Recency. Sometimes it’s just the last thing you hear that sticks in your mind. It’s not because you’re being sloppy or because you’re not thinking clearly but just because sometimes the last thing that you’re hearing seems the most compelling. Think of it as years of conditioning that you always “save the best for last.”

Does response order REALLY matter that much?

YES! For example, studies of voting have shown that the order of candidates’ names on a ballot affects vote totals. For example, in the 2000 presidential election, George W. Bush got 9 percent more votes when he was listed first on the ballot than when he was listed last–that’s a huge difference! This is one of the reasons that ballots randomize the order of candidates’ names.

Okay, then how do I fix it?

  1. You can randomize the options, so that each person sees the list in a different order.
  2. Sometimes, if you have an order list or a rating scale, randomizing the options can feel awkward. In this case, you can flip the options, so some of your respondents see a list that begins with Harry Potter and ends with Percy Jackson, and others see a list that begins with Percy Jackson and ends with Harry Potter.

If you randomize or flip the options, any order effects will balance themselves out and your data quality will be protected! And guess what? Both of these solutions are just a click away with SurveyMonkey.

Be sure to let us know how it goes in the Comments section below!

Tags: , ,

  • Polymars

    This is a good article. When there are 4 choices, as in your film choice question, would the respondent be more inclined to a balanced response if the choices were split into 2 sets of 2 and then and then a 3rd eliminator question?

    • kaytek

      Hi there,

      A creative suggestion but we recommend against complicating it too much. The best way to go is always to randomize the order of the responses, rather than try to guess how the respondent will be biased by the order. That way, biases wash out across your whole sample.

      Thanks for the question!

  • Thanks for that, great blog and useful advice. I remember learning about the recency and latency effect at Uni (many years ago). Thanks again.

    • kaytek

      You’re very welcome!

      • Ali Ahmed

        Hi kaytek, i need some serious and continous help from you, as i am working on a small survey related project for my university ….

        if u can just leave an email from your email account, we can have our discussion on emails, so that this blog doest gets a personal help forum…

        My email sshjssh@gmail.com

        Awaiting ur response

        • kaytek

          Hi Ali,

          We’d be happy to help. If you could kindly send a message with the email address connected to your account to support@surveymonkey.com, our team will respond asap. Usually within 2 hours, often less.

          Thanks!

  • Practical, usable advice. And it upsells too -for me, this adds value to the paid version.)

    • kaytek

      Awesome, glad to hear it. 🙂

  • Folkestone Winter Shelter

    great article, very informative thanks

  • Caritas_Fleet

    What about questions that show a rating from high to low, like:
    1. Strongly Disagree
    2. Disagree
    3. No Opinion
    4. Agree
    5. Strongly Agree

    — OR —

    The reverse order. Which would be the best way? or would should this be randomized too???

  • kaytek

    Hi there, with answer choices like these, we recommend not randomizing these to keep them methodologically sound. Start with Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree.

    Check out this quick video tutorial on rating scale best practices here: https://www.surveymonkey.com/blog/2013/06/10/jumpstart-video-series/

    And from our Help Center: http://help.surveymonkey.com/articles/en_US/kb/What-is-the-Rating-Average-and-how-is-it-calculated

    Hope this helps!

  • lana_s

    What about the order of rating scales – negative to positive or reverse?

    • kaytek

      Great question, Lana. We’re checking with our Survey Research team and will get back to here asap.

      • kaytek

        With scales, you should try and rotate the options if possible, but sometimes given the constraints of our platform, you aren’t able to. We’ve alerted our Product team on this and thank you for the feedback.
        More info for you on how to ask this type of question—
        You should rotate the responses to this question so that sometimes positive comes first, sometimes negative comes first.

        Q: Compared to other days, has your day today been:
        – Very positive
        – Somewhat positive
        – Neither positive nor negative
        – Somewhat negative
        – Very negative
        But for a question like this, it’s better to keep the answer options the same as the question wording so you don’t confuse the respondent:

        Q: Has your day today been positive, negative, or neutral?
        Note, for questions that use a Likert scale like “extremely, very, somewhat, not too, not at all” we wouldn’t recommend rotating.

        Hope this helps!

      • Muhubullah Abbasi

        Hi
        kaytek
        I want to ask question regarding my sample size. my total population is 10,000 people in textile industry. how much people do i need to survey to fill questionnaire. is it 30%+3000 or 10%=1000. plz suggest.
        forgive for asking question on different topic then above. Am new on DISQUS. And i dont know how to use it.

Inspired? Create your own survey.

Inspired? Create your own survey.

Sign Up FREE or Sign in

Write Surveys Like a Pro

Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »