Real-World Education for Modern Marketers

Join Over 609,000 Marketing Professionals

Start here!
N E X T
Text:  A A

Fatal Mistakes in Questionnaire Design: Leading and Misleading Questions

by   |    |  7,206 views

In this article, learn how to...

  • Identify biased, leading survey questions and correct them
  • Identify and correct questions with parallel-construction problems
  • Identify poorly worded questions and correct them

Leading and misleading questions always yield questionable data, based on which you are highly likely to report findings that can misguide stakeholders. Moreover, decisions they make based on such data could cause an organization's failure rather than lead to its success.

Leading Questions

Questions like those in the following examples lead respondents to give answers they might not otherwise provide if given a less-biased question and a set of balanced answer choices. Although these real-world examples in this article may seem a little extreme, they illustrate the point.

Examples of Leading Questions

Q10. When did you (or will you) stop investing in print marketing?


1. Last year
2. This year
3. Next year

Q20. Vendors that offer free upgrades always win higher satisfaction scores than those that don't. How much more satisfied are you when a vendor offers free upgrades?

1. Slightly more satisfied
2. Moderately more satisfied
3. More satisfied
4. Much more satisfied
5. Definitively more satisfied
 
Assuming a precondition (e.g., a behavior that is no longer practiced, such as discontinuing an investment behavior) and predisposing respondents with an introductory statement, even if true, are common types of leading questions. Avoid both.

Lack of Parallelism

Another way to mislead respondents is to present a confusing question. It's never good if respondents read a question and then say to themselves, What do they want? That can happen if you have not used parallel construction.

Lack of parallel construction between a question and its answer choices can confuse respondents and increase the prob¬ability of invalid data. Parallel construction is not hard to achieve, but researchers must be diligent when writing questions to ensure parallelism.

The following examples illustrate what to watch for and how to write parallel questions.

The list of responses within question Q30 does not match the list of answers the respondents are asked to give. Either the items listed in the question or the answer choices must change.

Examples of Parallel-Construction Problems

Q30. What percent of per-unit product costs are associated with run-time software elements, electronic components, and mechanical components? (Enter percentages in the boxes provided.)

Percent of Per-Unit Product Cost

Software components (per unit royalties) ____ ____%
Components such as semiconductors and boards ____ ____%
Components like physical housing and engines ____ ____%
Other (please specify): ___________________ ____ ____%

Q40. Approximately how many labels does a single barcode scanner in your organization scan per week? (Select one.)

Scans Per Day

1. Less than 25
2. 25 to 49
3. 50 to 99
4. 100 to 249
5. 250 to 499
6. 500 to 999
7. 1,000 or more
8. Don't know
 
Question Q40 could be very hard to answer and presents more than one problem.

First, if a company owns two or more scanners, the weekly number of scans could be radically different for each type of scanner in use.

Implicitly, you have asked the respondents to choose one scanner. But which one? Should they choose the newest scanner, the most-active scanner, or the fastest scanner, or use some other criterion for selection?

Asking about the average number of scans is another problem, because the average could be for a very heterogeneous scanning environment, rendering an average essentially meaningless. Additional clarity is required if the researcher wants to interpret the data with a degree of accuracy.

Finally, the question asks about barcodes scanned per week, but the answer section says "Scans Per Day"; does the researcher want the respondents to do the math?

More than likely, during questionnaire revisions, the time frame was changed but edits were not made carefully. Regardless, the question is confusing, and it will yield confusing data that researchers cannot interpret with confidence.

The revised versions, Q30R and Q40R, show a solution for each question. However, the revisions are merely one way to address the issues.

Examples With Parallel Construction

Q30R. What percent of per-unit product costs are associated with run-time software unit royalties, electronic components, and mechanical components? (Enter percentages in the boxes provided.)

Percent of Per-Unit Product Cost

Run-time-software unit royalties     ____ ____%
Electronic components (e.g., semiconductors, boards)   ____ ____%
Mechanical components (e.g., physical housing, engines)  ____ ____%
Other (please specify): ___________________    ____ ____%

Q40R. Approximately how many bar-code labels does your organization's newest scanner scan per day? (Select one.)

Scans Per Day

1. Less than 25
2. 25 to 49
3. 50 to 99
4. 100 to 249
5. 250 to 499
6. 500 to 999
7. 1,000 or more
8. Don't know

Mismatches

Another easy mistake to make is to mismatch the question text with the type of question you're asking.

One common example is the use of the term "primary" in a question that is structured as a multiple-response question, thus creating an oxymoron.

Example of a Mismatch

Q50. What is the primary reason your company will add new [type here] partners?
(Select all that apply.)
[Randomize]

1. Company reputation for excellent staff
2. Product complements existing areas of focus
3. Offers high margins
4. Offers marketing services with highest co-op funds
5. Provides alternative product choices
6. Superior service in the form of sales support
7. Best technology in the market
8. None of the above  
9. Other (specify) ___________________________

The text says "primary," but the question is structured as a multiple-response question with instructions to "Select all that apply." The word "primary" means "main"—and so, in the example, the one reason or primary reason. If you want more than one answer, don't use the word "primary."

Poor or Awkward Wording

Always work to root out poor and awkward wording (which leaves room for ambiguity) and the use of poor English (e.g., improper verb tense or poor choice of words and phrases, use of double negatives).

Poor writing will undermine your credibility with respondents and your client. Taking the time to be thoughtful, to write a clear and precise question, is its own reward, but it will also pay dividends.

Examples of Poorly Worded and Confusing Questions

Q60. According to you, what are the key factors that contribute to the success of a company in the present-day poor economic conditions?

A more typical and more efficient way to structure that open-ended question is shown in question Q60R. The respondent has clear instructions, and the answer portion of the question is structured to aid the respondent and to make life easier for the person coding the answers.

Q60R. What, in your opinion, are the key factors that contribute to the success of a [type of] company in the present recessionary economy? (Please briefly describe up to three.)

1. _____________________________________________________
2. _____________________________________________________
3. _____________________________________________________
 
Q70. Rank the following value-added services in terms or order of importance or level of desirability beginning with the most important 1 meaning "most important" through 5 meaning "least important."

Rank (1 to 5)
1. Access to demonstration equipment ____
2. Co-op marketing funds ____
3. Lead generation ____
4. Access to Web tools ____
5. Sales support ____

How would you answer that question? It includes a typo (the first "or" should be an "of") and two different ways in which to rank the services listed (by "order of importance or level of desirability"), and, as if those problems weren't enough, its instructions are awkwardly structured.

Let's take a look at the same question reworked.

Q70R. Rank the following value-added services in terms of their importance to your organization's success, where the most important is ranked 1, the next most important is ranked 2, and the least important is ranked 5.

Rank from 1 to 5:
____ Access to demonstration equipment
____ Co-op marketing funds
____ Lead generation
____ Access to Web tools
____ Sales support

If "level of desirability" were the criterion, it could be substituted for "order of importance," but you have to pick one criterion and go with it. Otherwise, ask about each in separate questions or at least in separate columns, but only if there is a compelling reason to do so.

* * *

As the examples in this article illustrate, it is easy to make errors when developing a questionnaire. However, the errors are not difficult to correct if you know what to look for and if you keep the study objectives in mind.

For more insights on questionnaire design, see Questionnaire Design for Business Research (Tate Publishing, 2010).


Join over 609,000 marketing professionals, and gain access to thousands of marketing resources! Don't worry ... it's FREE!

WANT TO READ MORE?
SIGN UP TODAY ... IT'S FREE!

We will never sell or rent your email address to anyone. We value your privacy. (We hate spam as much as you do.) See our privacy policy.

Sign in with one of your preferred accounts below:

Loading...
Carey V. Azzara is the principal and founder of AtHeath, LLC (www.atheath.com), a strategic research planning and consulting firm. He is the author of Questionnaire Design for Business Research (Tate Publishing, 2010). Reach him via carey.azzara@atheath.com.

Rate this  

Overall rating

  • Not rated yet.

Add a Comment

Comments

  • by Andrew Whalley Mon Dec 6, 2010 via web

    Your link to Digg doesn't work

MarketingProfs uses single
sign-on with Facebook, Twitter, Google and others to make subscribing and signing in easier for you. That's it, and nothing more! Rest assured that MarketingProfs: Your data is secure with MarketingProfs SocialSafe!