However, this particular matrix question type is different from a regular matrix where the columns are response options and rows are items to be rated.
Take a look at an example of a regular matrix:
Now in a menu matrix question type, the response options are presented using dropdown menus. In that sense, a menu matrix groups not multiple choice questions, but rather several regular matrix questions together. Seems like an efficient way of presenting your survey questions, right? But is it really the right question type for you?
To find out, we conducted 4 survey experiments. We compared menu matrix, regular matrix, and multiple choice question types. In the first experiment, we asked the respondents to answer 3 questions for each of 11 types of beverages and alcoholic drinks. Respondents were randomly assigned into 1 of 3 conditions that determined the presentation of the questions: in a menu matrix (Image 1), regular matrix (Image 2), or multiple choice questions (Image 3). The questions were not required, so respondents could skip any questions they didn’t want to answer.
What we learned was striking! The percentage of skipped questions for the first question (first column in the menu matrix) is low and similar across the 3 question types (Image 4). However, over 10% of the respondents skipped the later 2 questions in the menu matrix.
In the regular matrix and multiple choice questions, the missing data rates are still pretty low. Now when examining the data that came in for mobile respondents and desktop/laptop respondents, we found an even more striking effect.
Among mobile respondents, about a quarter of the menu matrix respondents skipped the second and the third questions but the missing data remains low for the regular matrix and multiple choice questions.
Skipping a question often suggests that the survey taker experienced some difficulties in responding. That can lead to poor data quality. Given the increasing number of mobile respondents, the problem caused by using the menu matrix question type should NOT be ignored.
After seeing these results, we were wondering whether reducing the size of the menu matrix could reduce the missing data. So we ran a second experiment where we asked the same 11 items but presented them in 2 smaller menu matrices, with 5 and 6 items respectively.
The results? We learned that the shortened menu matrix doesn’t reduce the missing data. Still over 10% of survey takers skipped the second and third questions in the menu matrix. Those on mobile? Missing rates were close to 30%.
Next, we further simplified the menu matrix by removing the last column. This didn’t reduce the difficulty of the task since similar amount of respondents skipped the second question in the menu matrix (11.5%) and more among the mobile respondents (22.5%).
There are 2 possible explanations for the low missing rate for the first question of menu matrix but high missing rate for the later questions.
#1: It’s possible that respondents answered the menu matrix column-wise. After they completed the first question on the first column, survey fatigue (in other words–“I’m sick of taking this survey!”) may have caused some respondents to drop out from taking the survey. Also, mobile respondents need to horizontally scroll in order to see the second and third columns, and it requires even more effort.
#2: There’s something unique with the first question and respondents simply didn’t want to answer the second or third question.
To find out the reason, we conducted a fourth experiment where we reversed the order of the 2 columns used in the experiment 3. The result shows that the missing data is still high for the second question in the menu matrix overall (8.3%) and higher for mobile respondents (23.4%).
This confirms our hunch that respondents went through the menu matrix column-wise and the high missing rate for the second and third column was NOT due to the content of the questions. Regardless of which question is presented second, the missing rate is always higher than the first question.
In all 4 experiments, we also asked respondents to rate the difficulty of the survey and we found that menu matrix respondents consistently rated the survey to be more difficult than the other 2 question types.
What’s the takeaway for these experiments? Menu matrix question types are really hard for survey takers to answer. And even more so for people taking surveys on mobile. Unless you want to risk losing a quarter of your respondents and make you respondents unhappy, considering replacing it with a regular matrix or multiple choice questions.
Check out our blogs on how to design a good matrix and multiple choice questions. Hope this was helpful and see you next time!
Questions for Mingnan? Let him know below.