Know-How Exchange : Research/Metrics

I've faced this many times. It is especially common in B2B situations where the target audience is somewhat limited and/or hard to reach.

One possible solution is to come up with a small panel of experts who have a good track record in assessing marekting strategies and consumer behavior. Just poll them on key questions (and pay them well for their time and expertise). You don't have to represent this as quantitative, projectable research; it's not. But it can be quite valuable and extremely cost-effective.

If you haven't read Malcolm Gladwell's "Blink," that's a good place to get the theory behind this approach. Often an expert's gut reaction is worth more than even quantitative research. They somehow "know" what will work and what won't -- even if they don't have a tight rationale to explain it. If you have a small panel of these experts -- say 6 or 8 of them -- you may have the best technique of all -- and a new product offering for your company.

If you decide to pursue this, I may be able to help you assemble an expert panel. Many of them are here on this forum, and recently retired executives from advertising, marketing, and market research would be good candidates.

This could be the wave of the future for certain kinds of market research. Read "Blink."

Hi there B2B

This is a problem I had to battle with when a marketing manager for an instrumentation company. The products sold in the low thousands per annum, fresh sales stats were in the mid hundreds and customer feedback forms were in the high 10’s. The CEO wanted next year’s strategic indicators! I sorted it by borrowing a few pattern recognition techniques from electrodynamics and resurrecting the use of Bayesian statistics. That is the maths of conditional probability distribution and the idea that one probability is actually related to another if there is a historic connection. See https://en.wikipedia.org/wiki/Bayes'_theorem for a reasonably up-to-date interpretation of Bayes. (He’s been out of fashion for around 200 years!)

I think that the idea of using a feedback group as well as analysing your statistics has its merits, but any qualitative approach is always going to suffer from the problem of producing something that is measurable. Also, don’t forget the warning about asking groups of experts – gather any 6 in a room, ask them a question and you’ll get 6 answers. Moderate the group with a strong chairman and you’ll end up with the chair’s opinion.

Here’s a Bayesian view. I understand your worry about using small samples and the risks of reducing the confidence level in your results, but confidence in what? A confidence level of 95% is only two limits within which 95% of your results lie. Obviously, I don’t know what it is that you are measuring, but if you can infer a trend, albeit imprecisely, then you can take multiple measurements over time as well as discrete samples.

Does a pattern occur in your measurements, either on their own or plotted over time? (Forgive the ignorance but I’m not sure what the first differential of customer satisfaction with time actually represents!) Likewise, is the variance or standard deviation against some norm small? If the answer is yes, graphically or otherwise then you can start to use smaller samples with a greater statistical level of confidence than a pure binomial distribution of measurements would suggest.

If there is a pattern then it is reasonable to assume, for the purpose of this study only, that the results are depending on each other in some way – perhaps in the nature of the sample, perhaps through the nature of the questions, perhaps because they do actually conform to a pattern. That means that for a sample of a given size, you can, as we used to do in chemistry, “Discard the wrong readings” or more aptly, accept that our all too few readings are a good fit because they look to be so.

If the assumption is wrong and there is no pattern, then the model fails. The stats will we be invalid. If the assumption is right, the results will be far more accurate than a binomial distribution and standard probability would or could assume.

In other words, go with smaller samples if your measured samples have either a small variance or they conform to a tight fit to a curve or pattern. Bayes says that the errors in such measurements themselves fit into a bell curve, with the majority of the results being within the confidence level you sought in the first place!

The downside is that a few analyses will be badly wrong and you won’t know by how much. Whether that’s better or worse than asking for opinions I don’t know – personally, I’d like to have the benefit of the two approaches, then you could test the statistics of answers given in surveys around the areas where the experts concur. Now that’s interesting!

Steve