Ask more, know more, do more

Order Matters: Don’t Let it Get In Your Way

Order Matters: Don’t Let it Get In Your Way

Pop quiz!

Read this list of words quickly: pen, airplane, shell, butter, duct, blueberry, wrapper, zebra

Now cover the list with your hand.

Now list all of the words you remember.

If you’re like most people, you’ve remembered the first one (pen) and the last one (zebra), but not so many in between. How did we know that? Well, the order of a list can be just as important as the actual content. The order itself changes how you think and respond.

The same thing happens in surveys when you give respondents a list of answer choices. The first and the last answer choice stick out, and the middle tends to get ignored. Take this question for example:

Best Young Adult Literary Series?

More respondents will pick Harry Potter and Percy Jackson than would have otherwise simply because they are the first and last choices. In survey language, that’s called a response choice order effect.

Why might people pick the first response choices?

  1. Satisficing. Respondents start with the first option, thinking about all the reasons the Harry Potter series might or might not be the best series of the last ten years. They then do the same thing for the rest of the options on the list. As respondents go down the list, they start getting mentally tired. They then begin to satisfice–they stop paying close attention and skim over the last options in the list, neglecting to consider them carefully. This can leave the top response choices in the list with a much stronger case than the bottom ones.
  2. Priming. We talked before about how this works with questions. Ask someone their income and that’ll sway how they respond to a question asking if they believe in the American Dream. Believe it or not, the same thing happens with response options too. For example, if Harry Potter is first on the list, respondents might first think about how the series made billions of dollars. They’ll then use this same standard–which is pretty unique to Harry Potter–to judge the other options in the list. None of the other options made billions of dollars, so Harry Potter is left as the easy choice.

Why might people pick the last response choices?

  1. Satisficing. Yes, satisficing rears its ugly head here too. Again, respondents do the hard critical thinking work for the first few options, but get mentally tired and begin to satisfice. When they stop thinking critically about the last few choices, this can actually make them seem more appealing, not less. It becomes more of a gut instinct selection than a rationally thought-out opinion.
  2. Recency. Sometimes it’s just the last thing you hear that sticks in your mind. It’s not because you’re being sloppy or because you’re not thinking clearly but just because sometimes the last thing that you’re hearing seems the most compelling. Think of it as years of conditioning that you always “save the best for last.”

Does response order REALLY matter that much?

YES! For example, studies of voting have shown that the order of candidates’ names on a ballot affects vote totals. For example, in the 2000 presidential election, George W. Bush got 9 percent more votes when he was listed first on the ballot than when he was listed last–that’s a huge difference! This is one of the reasons that ballots randomize the order of candidates’ names.

Okay, then how do I fix it?

  1. You can randomize the options, so that each person sees the list in a different order.
  2. Sometimes, if you have an order list or a rating scale, randomizing the options can feel awkward. In this case, you can flip the options, so some of your respondents see a list that begins with Harry Potter and ends with Percy Jackson, and others see a list that begins with Percy Jackson and ends with Harry Potter.

If you randomize or flip the options, any order effects will balance themselves out and your data quality will be protected! And guess what? Both of these solutions are just a click away with SurveyMonkey.

Be sure to let us know how it goes in the Comments section below!

Tags: , ,

10 thoughts on “Order Matters: Don’t Let it Get In Your Way

  1. Polymars says:

    This is a good article. When there are 4 choices, as in your film choice question, would the respondent be more inclined to a balanced response if the choices were split into 2 sets of 2 and then and then a 3rd eliminator question?

    1. kaytek says:

      Hi there,

      A creative suggestion but we recommend against complicating it too much. The best way to go is always to randomize the order of the responses, rather than try to guess how the respondent will be biased by the order. That way, biases wash out across your whole sample.

      Thanks for the question!

  2. Sally Mayor says:

    Thanks for that, great blog and useful advice. I remember learning about the recency and latency effect at Uni (many years ago). Thanks again.

    1. kaytek says:

      You’re very welcome!

      1. Ali Ahmed says:

        Hi kaytek, i need some serious and continous help from you, as i am working on a small survey related project for my university ….

        if u can just leave an email from your email account, we can have our discussion on emails, so that this blog doest gets a personal help forum…

        My email

        Awaiting ur response

        1. kaytek says:

          Hi Ali,

          We’d be happy to help. If you could kindly send a message with the email address connected to your account to, our team will respond asap. Usually within 2 hours, often less.


  3. Ann Druce says:

    Practical, usable advice. And it upsells too -for me, this adds value to the paid version.)

    1. kaytek says:

      Awesome, glad to hear it. :)

  4. Folkestone Winter Shelter says:

    great article, very informative thanks

  5. kaytek says:

    Hi there, with answer choices like these, we recommend not randomizing these to keep them methodologically sound. Start with Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree.

    Check out this quick video tutorial on rating scale best practices here:

    And from our Help Center:

    Hope this helps!

Leave a Reply

Your email address will not be published. Required fields are marked *

Inspired? Create your own survey.

Make your own survey

Sign In

Sign up to get started

PRO Sign Up 
Sign Up FREE

Latest Features

Curious about what we’ve been working on?

View New Features

Get expert survey help

Get expert survey help


Best practices for planning expert surveys

Planning makes writing your survey easy.

Download eGuide

How many responses do you need?

Use our sample size calculator to find out!

Calculate now

Analyze survey data like a pro

Learn to slice and dice data using the Analyze tool.

Download eGuide