Poorly designed questions and scaling problems can derail your research efforts faster than you can say "the cat in the hat"!
To help you avoid a few of the more common and onerous problems, we will explore two separate but related questionnaire-design issues: matrix questions, especially the big, scary kind, and unbalanced scales, which provide data that is, at best, difficult to analyze and, at worst, useless.
Scary Matrix Questions
First, we examine what we will refer to as scary matrix questions (SMQs). A colleague of mine who likes that phrase said he imagined research professionals texting in acronyms: "OMG did U R the RI with all the SMQs? LOL." (Translation: Oh my god, did you read the research instrument with all the scary matrix questions? Laughing out loud.) Don't be the subject of such laughter.
The use of matrix questions that could scare a respondent into dropping out is counterproductive. Yet matrix questions are used frequently and written with less care than you would write an email.
Those considerable and overwhelming questions are used typically to collect a large number of data points in a relatively small amount of space.
One justification is to create a data set that will provide enormous differentiation using a large set of variables and values. The more likely outcome is tired respondents who straight-line their answers (i.e., select all 3s, 4s, or 5s) as a way to move through the task quickly.
What's the outcome? Rather than a highly differentiated set of responses, the research creates the equivalent of white bread and adds very little to the study's ability to gather insights.