Know-How Exchange : Research/Metrics

Good for you for asking your customers!


As a recovering former statistics student at Johns Hopkins, I can tell you that if your population is 10000 and you want a confidence level of 95% and a confidence interval of plus or minus five, then you need a sample size of 370. If you want greater precision, you need to up the sample size.

The ideal number of questions depends on the question structure (multichoice, TF, Likert scale, fill in), the nature of the topic, the interest of the interviewee, and the likely variability of response. It also depends on how you deliver the questionnaire. What you do is pretest the questionnaire Who the heck gave you "5" as the answer? Sounds like they need to watch "Hitchhiker's Guide to the Galaxy."

I would offer credits like its done at MarketingProfs. Best answers get awarded most credits . Or offer them a copy of the research report - that should peak their curiosity enough to join in the fun. (Or you could tell them that if they don't respond, you are going to have a contest. First prize will be dinner with the CEO. Second prize will be two dinners with the CEO. Third prize will be having dinner with the survey developers.)

People will share information about industry or vertical if you set up the research protocol right. Make sure you tell them you aren't going to use it to target them for individual sales efforts.

Good luck, and thank you for asking us.

Regards,

Win

Hi, Larry,

1. Sample size depends on the population size (10,000, you say), maximum acceptable error margin (5% is commonly), the confidence desired (95% is commonly used), and probability of a a particular answer (the percent of respondents answering a certain way). Sample size increases as the population increases, as the acceptable error margin decreases, as the confidence increases, and as the probability increases (to 50% where it peaks - e.g. 50% gives the largest sample size). Since most people don't know the probability before they survey, most people design for the worst case, 50%. If you use 5% margin of error and 95% confidence for your sample design, you will be able to say, "the results of the sample will reflect the population results to within 5% with a confidence of 95%. For a population of 10,000, with 5% margin of error, and a confidence of 95%, the maximum sample size would be 371. More than anything, the right number for confidence and margin of error depends on your budget. If you felt OK for 10% error and 90% confidence, your sample size would be 68. You can experiment with the numbers and see the effect on sample size at:

https://www.berrie.dds.nl/calcss.htm

2. As far as ideal number of questions, I know of no research to draw on here. As you say, simple is better. The more questions, the less likely someone will answer/complete the survey. However, I personally, I think that 10-15 questions are OK - depending on the complexity. One thing that has been observed: If a respondant knows how long the survey will take ahead of time, they are more likely to continue to the end - even if it takes more time than the estimate. Translation: Let the respondant know how many questions and about how much time it will take ahead of time.
3. Incentives increase the number of responses. The larger the value of the incentive, the higher your response rate likely will be. For longer surveys, researchers include questions to check consistency by asking questions a different way a couple times. For your survey, this probably isn't practical. Better to spend a few lines up front to explain the respondant's WIFFM - what's in it for me? If you state that the survey will help define the level of service and increase customer satisfaction, etc.
4. As far as asking questions about the respondant's business, as long as it's "demographic" information or generally publically available data - what industry they are in, vertical markets they serve, etc, I think the respondents will answer truthfully. If you ask proprietary information - revenues, purchase amounts, profits - people are unlikely to trust you with their proprietary information.

I hope this helps.

Wayde

The answers to your questions will depend almost entirely on what it is you want to learn and how you plan to use the results of your research.

The mechanics are easy ... sample sizes, etc. ... It's really about the objective of the research. For example, if you can accomplish the objective with a sample size that yields only 90% reliability, there's no reason to spend the extra money shooting for 95% accuracy. And if you need the answer to only one question, there's no reason to ask 5 or 10 questions.

Said another way, you need to first decide what it is you need to learn. State the objective of the research in very clear terms. Then force yourself to determine exactly what you'll do with the results when you have them. What are the action items that this research will trigger?

When you have those answers, then we can answer the mechanical questions. And if you can afford it, you'd be well advised to get some professional assistance from an experienced market researcher. They'll be well worth a consulting fee, if the research itself is really important to your business.

1. Excellent answers to question one, so I won't contribute more.

2. Ideal number of questions to ask is just enough to get the data and responses you need, but not so many as to scare off respndents or force them to start making up answers to finish the survey. This may take a little trial and error (like you have to time to experiment now, right?). Precisely why you need to run surveys more often...you gain experience and insight, and your customers get a better service/product.

3. Incentives can work for you, or against you. Usually, incentives are used to encourage or increase participation rates. Once again, experience with your customers would tell you what inceptive works best for them....but given the short time frame....I wouldn;t worry about an incentive leading to poor or misleading data. It may happen, but its more your fear about being ripped off than there are real scammers in the world. Keep your incentive appropriate to the demands of the survey, and you'll be fine.

4. It doesn't hurt to ask, but I understand your concern. Ease the particpant's fear by making the question optional to answer rather than mandatory. Mandatory answers are rather dictatorial...and 'personal' identifying information is a bit uncomfortable to divuldge when it's demanded of you. Soften the terms, and make it optional....you may need a larger sample size to offset the numbers who exercise their 'skip this question' option, but...you'll end up with enough data to gauge your results.

Hope your survey goes well, and do more of them.

Darcy Moen
www.customerloyaltynetwork.com

Larry,
I have spent much of my time during the past 14 years interviewing the presidents of small companies. One of the questions I often ask is how much time are you willing to spend giving feedback to suppliers.
In the form of an email based survey, I feel 5 minutes is pretty much tops. If you want/need more time, then I suggest a message from the CEO of the surveying company explaining why you want/need the information and how it will be used for the betterment of humanity, customer service etc....
Frank Hurtte
www.riverheightsconsulting.com