Ask more, know more, do more

Comparing Isn’t Always a Piece of Cake: Ranking vs. Rating Explained

Comparing Isn’t Always a Piece of Cake: Ranking vs. Rating Explained

We’re still a few days away from the weekend and we’ve already got cake on the brain, dear survey fans.

Comparisons are a basic impulse. Tell your friend you like angel food cake and she might counter, “Really?! More than birthday cake? Or carrot cake?” Naturally, this will turn into a heated discussion over how to rank the deliciousness of these kinds of cakes.

Your survey might look a little something like below (we whipped up this practice survey just for you so feel free to get in there and start ranking):

Create your own user feedback survey

The problem: A ranking question may seem like the natural fit, however, there is a drawback to using a ranking question. Although ranking different kinds of cake gives you a relative sense of whether one is liked more than the other—you don’t know how much more. Someone may like chocolate cake only slightly more than carrot cake, but thinks cheesecake is disgusting. A ranking question will not let you know the strength of preferences—only the order.

Let’s stick to the sweets to help illustrate another shortcoming of ranking questions, shall we?

M&Ms come in six colors, so your ranking question might look like this:

Create your own user feedback survey

Now, almost everyone has a favorite and a least favorite color M&M, but it’s somewhere in the middle where it starts to get a little blurry. Red is towards the top and brown towards the bottom, but as for whether yellow is better than green or vice versa? It’s all kind of a toss-up. Could depend on your mood, the color of your outfit, or maybe it just doesn’t matter at all.

The moral of the sweet story here? Ranking questions can get difficult in a hurry. Once you get past three choices in a ranking question, it gets downright sloppy and confusing—and psychological research has proven that it’s a very tough thing for people to rank more than three options.

The problem: When you give people a cognitively difficult task like ranking question choices of say, six things, often they will end up getting frustrated and will start to rank choices randomly. This will lead to higher dropout rates from your surveys and even worse, messy data for you to analyze from those who do manage to power through and finish. Remember, although ranking questions gives you a relative sense of whether one thing is liked more than the other—you don’t know how much more.

So, what can you do instead? Well, that depends on why you’re asking! Just one more reason you need to decide why you’re asking before you come up with how to ask it.

Use Case #1:

  • Sample Research Question: The Mars company wants to double the number of the favorite color of M&M and eliminate the least favorite color of M&M. To know which to double and which to keep, Mars needs to know which is the best and which is the worst.
  • Sample Survey Solution: If you just want to know what people’s favorite and least favorite thing is and you don’t really care about all that middle stuff, try splitting it into two closed-ended questions:

The color that is chosen the most times in the favorite question is the winner you’ve been looking for, and the color that is chosen the most times in the least favorite question is the loser you’ll need to eliminate.

Use Case #2:

  • Sample Research Question: The Mars company wants to add MSG to the less-liked colors of M&Ms to give them a flavor boost so that consumers find the colors all equally tasty. To know how much MSG to add, Mars needs to know the relative standing of the colors.
  • Sample Survey Solution: If you really do care about the relative standing of all six colors of M&Ms (or anything else non-candy related), split up the one ranking question into six rating questions:

Respondents rate each of the six M&M colors one at a time. Using these separate ratings, you can then compute an average score for each M&M. You then have an overall ranking of liking for each color based on the average score for each color.

So what’s the best approach to use overall? Every question type has its strengths and weaknesses. It’s just a matter of knowing your research question. If you know why you’re asking the question, the best trade-off for your particular survey should be pretty clear. So with that, it’s off to chow down on some cake and M&Ms!

Still need help thinking it through? Let us know in the Comments section below!

Image courtesy of BigStock photo

Do a Better Job with SurveyMonkey Enterprise

Give yourself—and the rest of your team—the answers you need to make smarter decisions. Access our most powerful tools with SurveyMonkey Enterprise.

Learn More →

Tags: , , , , , , ,

6 thoughts on “Comparing Isn’t Always a Piece of Cake: Ranking vs. Rating Explained

  1. Deepak says:

    Thanks for the tip, explained with great examples.

    1. Kayte K says:

      Hi Deepak, thank you for the support! Don’t hesitate to let us know if you ever need help with any of your survey projects.

  2. Oh my goodness! an wonderful post dude. Thank you But I am experiencing problem with ur rss . Do not know why Unable to subscribe to it. Is there any individual obtaining identical rss problem? Everyone who knows kindly respond. Thnkx

    christian louboutin shoes

    1. Kayte K says:

      We’re A-ok on our end here so far. What specifically seems to be happening for you?

  3. mlb says:

    My ranking questions averages appear to be “reversed” (for example, 5 potential rankings, scores are 6-average). The example on your help screen does not reverse the score. Am I interpreting my results correctly?

    1. Kayte K says:

      Hi there, not sure we’re understanding your question fully. Could you clarify a bit? Here’s more info on ranking questions too from our Help Center just in case:

Leave a Reply

Your email address will not be published. Required fields are marked *

Inspired? Create your own survey.

Make your own survey

Sign In

Sign up to get started

PRO Sign Up 
Sign Up FREE

Latest Features

Curious about what we’ve been working on?

View New Features

Get expert survey help

Get expert survey help


Best practices for planning expert surveys

Planning makes writing your survey easy.

Download eGuide

How many responses do you need?

Use our sample size calculator to find out!

Calculate now

Analyze survey data like a pro

Learn to slice and dice data using the Analyze tool.

Download eGuide