When you receive an invitation to take a survey, are you more likely to click on a link labeled Take a brief survey, or one that says Take a 5-minute survey? Suzanne Norman suspected it was the latter, and decided the hypothesis called for a test. "[A]fter all," she says at the Emma blog, "five minutes means five minutes, whereas 'brief' may just be marketing-speak for 'the longest survey of your life, sucker.' When you're asking for someone's time, there's little question that it's good to be as specific as possible."
It turned out Norman's hunch was correct. Although the two didn't deliver dramatically different results, the specificity of a "5-minute" promise consistently outperformed the implied open-endedness of "brief." The five-minute version garnered:
- An 8-percent increase in click-throughs.
- A 9-percent increase in the number of people who started the survey after arriving on the page.
- An 11-percent increase in the number of people who completed the 18-question survey.
"While setting an expectation [of] a specific time [limit] isn't the silver bullet to boosting your survey participation numbers," says Norman, "the people who appreciate the specificity will be [more] likely to visit, start and complete your survey."
The Po!nt: Get out that stopwatch. If you haven't done a similar "time" test with your own surveys, try it out: you might find a way to drive even greater participation.
Source: The Emma Blog. Read the full post here.
by Christina "CK" Kerley
Comments
by Glenn Fri May 15, 2009
One point I would add to this discussion.
Forget about the importance of "5 minutes" vs."brief" in the context of click-through analytics and survey conversions/completions- unless the purpose of said survey is purely a lead generation tool (which seems to be an accepted mindset among email marketing platforms these days). It's fairly obvious that more concrete definitions around the time parameters would yield the 9-11 point spikes as evidenced herein.
This advice if implemented incorrectly, will compromise the scientific process and may yield faulty or inconclusive results if you can't compensate for this in the data validation / cleaning phase. The actionable insights that you then produce will lack the predicative capabilities to ensure absolute success and you will likely end up sending your colleagues on a wild goose chase.
Instead, you should be looking at the variances in response behavior and the quality of thought invested in survey completions. Dozens of reliable metrics already exist to measure these response patterns and countless studies have been published on response bias as a result of participation incentives.
Quantifying the "briefness" of a survey is no different than offering a chance to win something- it is a form of motivational encouragement that inherently skews the sample population plain and simple.
A better approach, might be to advise marketers to work with research vendors and/or survey tools that can precisely measure the median time of participation from a healthy sample of survey beta testers and then use this metric as a concrete and trustworthy statistic for potential participants to measure their own participation ROI. Empower your community with the authenticity of the survey; don't hoodwink them into something by just parsing the right words.
If it actually takes 12m and 32s at the median in beta tests, then publish this number (and feel free to round up to the nearest minute). This may very well end up being the aforementioned "5 minutes"! I have met so many DIY social scientists in my years that think that because they can actually fill in the survey and whip through it in 5 minutes, then that's the number they should publish in the survey introduction. This is naive at best and fraudulent at worst. It will only serve to diminish your brand esteem and the professions of marketing and market research.
When it comes to surveys and sample populations, don't forget the old adage Garbage In = Garbage Out. If you have doubts about the quality of your sample population, you can always apply a gross technique like using a bell curve to remove participants outside of a set number of deviations from the median participation time, but there is also great value in understanding those voices outside of the norm and crudely removing them due to methodological weakness could diminish the survey insights as well.
My point is simple- worry less about volume metrics and focus more on quality outcomes. Long-term business intelligence will grow a company much more effectively than short-term, feel-good popularity spikes.
by Williams Ford Tue Jun 2, 2009
Great Job on getting some good metrics. More marketers should follow this example. ;%3E)
Williams Ford
WGF Enterprises, Inc
wgfenterprisesinc.com/1SourceBlog