Real-World Education for Modern Marketers

Join Over 602,000 Marketing Professionals

Start here!
N E X T
Text:  A A

Questionnaire Auditing: The Quality-Control Step You Cannot Afford to Skip

by   |    |  4,113 views

In this article, learn...

  • Why the survey questionnaire is the foundation of solid data
  • What a questionnaire audit can do for you
  • What can go wrong with a questionnaire

The value of strong questionnaire design, complemented by the task of high-quality sample development, is not fully appreciated. Too often researchers relegate those two essential building blocks of market research to a status secondary to the analytics and presentation of results. Big mistake!

A strong questionnaire design and, specifically, conducting an audit to ensure quality, is the focus of this article.

Questionnaire Design

Research Axiom No. 1: You can never fully recover from a poorly written questionnaire.

No manipulation of data points, regardless of how cleverly done; no amount of analysis, regardless of how brilliantly executed; and no degree of interpretation, regardless of the intellectual insights, can save you from a poor research foundation. Your building will collapse as if shaken by an earthquake!


Questionnaire design is a task repeatedly given insufficient time and attention. Clients and research professionals alike often underestimate the effort it will take to develop a well-structured and concise research instrument.

What amazes me most? Project leaders relegate the task to a status depicted in this attitude: "Once the questionnaire is done, we can get on with the important stuff, like analysis and reporting."

The assumption that analysis work is the essence of the research, and the expectation that interpreting the results is where the mastery of research ultimately lies, is a mystery to me.

Have we not pounded the concept of garbage-in garbage-out into the heads of research students across the globe? Can new Internet tools substitute for critical thinking and the hard work of aligning the research instrument to the purpose of the study to answer the business questions that sponsors paid to learn?

If that sounds like a rant, I'm guilty as charged, but I will end it with an analogy: If you have not studied and then practiced writing poetry, would you expect to publish a book of poems simply because the VP of marketing asked you to? Designing a good-quality research instrument probably takes less talent than being a good poet, but it's close.

What does that have to do with questionnaire auditing? That's simple. If you work hard up front to create a great questionnaire, the audit will be quick and painless. If you don't... well, you already know the answer.

Questionnaire Auditing

Our research on research has shown that even experienced researchers are less diligent about quality than one would hope. Research and questionnaire design, in particular, require not only science but also art; and if the proper fundamentals are not applied, the product is less than artful.

The purpose of an audit is to help you avoid errors in structure and allow you to focus more on the art.

A Questionnaire Quality Control (QQC) audit using a standard set of parameters provides valuable feedback to designers, the project teams, and clients. A good QQC audit has three parts:

  1. A set of standardized quality checks: (a) Checks that maximize objective assessment and minimize subjective evaluation; (b) A QQC score based on the results of the standardized portion of the audit, taking into account repeat mistakes
  2. A clear mechanism for testing a questionnaire's overall length: (a) Testing the questionnaire during development; (b) Testing the questionnaire after it is finalized and programmed; (c) Checking the longest paths
  3. An expert review to examine unique aspects of the questionnaire and to identify issues or problems that might need attention, correction, or enhancement

Questionnaire Design Problems

The QQC audit can identify problems and errors commonly encountered during the task of questionnaire design. However, in our experience, after providing a review of a questionnaire and telling the author it needs work, the person frequently asks, "What's wrong with my questionnaire? It seemed perfectly fine to me."

Authors don't want to hear that what they've created is less than masterful, but there is more to questionnaire design than asking questions the way you do in everyday discourse. A well-structured audit will be helpful and provide actionable guidance to the author(s) of research instruments. It always helps to receive an impartial review.

Here is a partial list of audit items to help answer the "What's wrong with my questionnaire?" question:

  • Answers not mutually exclusive
  • Double-barrel questions
  • Unbalanced scales
  • Lack of parallel construction
  • Terminology that creates ambiguity
  • Definition problems
  • Scary matrix questions
  • Lack of instructions
  • Poor or confusing wording
  • Inventory question overload
  • High respondent burden (e.g., too much math)
  • The overly long research instrument

If you are having trouble designing a questionnaire, you are not alone. Through the application of a multipoint questionnaire audit, we have repeatedly shown that questionnaires do not meet the requirements of sound questionnaire design. Here are three examples of QQC audits we've conducted:

  1. A director of research from a well-known university had a QQC score of 65.
  2. The owner of a LinkedIn Group related to market research had a QQC score of 67 (and the questionnaire had fewer than 10 questions).
  3. The owner of a market-research company repeating a study done more than five times scored 58 on the QQC audit.

No, you are definitely not alone.

Final Message

Questionnaire design is not an end in itself; rather, it is a means to an end.

Its purpose is to illuminate the subject under investigation, to bring forth content that one can use to develop insights into the marketplace. It should help decision makers make better decisions than they otherwise would make if the information were not available.

To accomplish that goal, practitioners must construct questionnaires with great care and skill while avoiding errors throughout the process.

As Albert Einstein once said, "We can't solve problems by using the same kind of thinking we used when we created them."

If you do not care about the quality of the research you conduct, well, shame on you—but at least recognize that a questionnaire of good quality is a necessity for self-preservation. Enough said.


Join over 602,000 marketing professionals, and gain access to thousands of marketing resources! Don't worry ... it's FREE!

WANT TO READ MORE?
SIGN UP TODAY ... IT'S FREE!

We will never sell or rent your email address to anyone. We value your privacy. (We hate spam as much as you do.) See our privacy policy.

Sign in with one of your preferred accounts below:

Loading...
Carey V. Azzara is the principal and founder of AtHeath, LLC (www.atheath.com), a strategic research planning and consulting firm. He is the author of Questionnaire Design for Business Research (Tate Publishing, 2010). Reach him via carey.azzara@atheath.com.

Rate this  

Overall rating

  • This has a 1 star rating
  • This has a 1 star rating
  • This has a 1 star rating
  • This has a 1 star rating
  • This has a 1 star rating
2 rating(s)

Add a Comment

Comments

  • by Olga Thu Oct 28, 2010 via web

    So how exactly would one calculate QQC? What is the formula?

  • by Harley Axelrod Sat Nov 6, 2010 via web

    Being a Respondent more these days that a Questionnaire designer, it sometimes seems apparent that the Questionnaire in the final stages has been edited, redrafted and tested by the same person (or committee). A totally fresh set of eyes 'test driving' the final draft version might help improve the Quality of the input.

  • by Arie Mon Nov 8, 2010 via web

    Same as @Olga - what formula do you suggest?

  • by BD Mon Nov 15, 2010 via web

    This is a rather useless article. It does not go on to give us a QC checklist. What's the point? Do it right? Okay. But how?

  • by Erik Sat Nov 27, 2010 via web

    Well, now I know that the author feels that there are a lot of bad questionnaires out there and they should be created by a professional.

    "practitioners must construct questionnaires with great care and skill while avoiding errors throughout the process."

    And this article does nothing to help one do that.

MarketingProfs uses single
sign-on with Facebook, Twitter, Google and others to make subscribing and signing in easier for you. That's it, and nothing more! Rest assured that MarketingProfs: Your data is secure with MarketingProfs SocialSafe!