Why “Select All” Makes Your Life Hard

surveyscience

We spend a lot of time on this blog telling you all about how a good survey is like a good conversation. Well, here’s the post where we rein it in a little. Sometimes, survey writers fall into the trap of writing a survey a little too much like a conversation. This can be a problem because unlike a real conversation there’s no opportunity for dynamic follow-up in a survey so you’ve got to get it right the first time.

Let’s say David runs a real estate agency and is trying to make sure his agents have a competitive edge with customers. He decides to go straight to the customers to find out what they need and writes up the following question:

He sits back, feeling pretty proud of himself. Not only is this just how he would ask the question in a conversation, but, he’s also read our blog and knows that closed-ended questions are the way to go.

Okay David, hate to burst your bubble here but this question’s got some issues.

1. Survey takers zone out.

Why? Well, this question is pretty complicated. David may feel like he got away with asking five questions in one–but trust me, his survey takers will notice that. Answering a select all question can be both complicated and tiring, and when things get complicated and tiring, survey takers stop paying attention. Also, because a survey taker can check 0-5 boxes here, the number of boxes he or she actually checks may not reflect his or her actual opinions–but just the moment he or she got tired of checking boxes. Bottom line? Select all questions can spell trouble for your data quality.

2. Analysis is tough.

Why? When survey takers can choose more than one answer to a question, data analysis gets complicated fast. When you measure one concept at a time, like we’ve suggested in the past, it makes analysis simple. You just take averages of responses for each concept, compare the numbers, and you’re done! With a multiple answer question, however, David doesn’t have a measure of the strength of each of those adjectives. So, he can’t assign them a number and is stuck treating them just as percentages. He has essentially made five Yes/No questions, and that’s not a recipe for success.

3.  Next steps are unclear.

Why? If you don’t know the relative importance of all these qualities, how do you prioritize what to focus on? As a business owner, David is running a survey to get actionable insights–to get information that he can use to make real improvements in his business. Just because friendliness gets picked the most, doesn’t make it the most important to focus on! It might be that 70% of people agree that friendliness is slightly important, but 50% of people agree that honesty is extremely important. With a “select all” question, however, David doesn’t know the strength of these preferences. Consequently he doesn’t know how to prioritize training for his employees. Without more detailed information, he doesn’t have enough information to proceed.

How can David fix it?

David should turn his one select all question into five separate questions, each with a Likert scale measuring the strength of survey takers’ importance ratings of the five qualities. For example:

So on your next survey, learn from David’s mistakes. Leave those “select all” questions on the shelf and break each of them down into specific questions. Your survey takers will stay focused, your analysis will be easy, and your next steps will be clear. A recipe for success at every step of the survey process!

Have questions? You know where to go, folks.

Tags: , ,

Inspired? Create your own survey.

Inspired? Create your own survey.