Know-How Exchange : Research/Metrics

Using demographic characteristics -- such as the ones you mention -- is the typical way to weight survey data to better represent a population of interest. If you can start with some universe estimates of these characteristics, e.g., what percentage for each age group, race, income group etc., weighting ACROSS modes using this should be sufficient. The idea here is that you used different modes to "fill in" under-represented populations, therefore the demographic weighting should smooth this out.

The bigger concern you should have is the bias inherent in response mode. There's evidence that people answer telephone questions differently from mail or web (aural vs. visual), not to mention the ever-present interviewer bias that creeps in. Some research companies claim to have invented ways to compensate for this type of bias, but it remains to be seen whether these are effective approaches. There are tests you can do to see if otherwise-similar respondents differ by response mode, and I'd encourage you to do so.

You may consider keeping these disparate surveys separated and use techniques like convergent analysis to compare the results across studies rather than combining them and wondering whether this hurt your ability to draw effective conclusions from these data.

There have been a number of researchers who have looked at this question, and how to weight in a number of situations. It can be difficult to generalize across industries, and the proliferation of web-based surveys and available respondents using this mode has rendered some of this research to be somewhat outdated. However, there is some information available that can give you some direction. See for example:

https://www.marketstreetresearch.com/online_research_article.htm

https://www.jmir.org/2004/1/e2

https://www.bus.wisc.edu/nielsencenter/research/studies_in_comparability.pd...

Interesting problem...I believe you should measure the variables you suspect the two groups to be different on (such as willingness to travel, willingness to pay, cost of acquisition, customer life time value once acquired,...). This is a lot of work, but you can simply do it once for a cohort of customers, and then you can use these weights for the several years in the future: these differences are unlikely to change much over time.

To give you an example from another application: a firm found out that customers acquired through WOM referrals yield higher life time value than those acquired through advertising, which in turn yielded more life time value than those acquired through price promotions. Based on this knowledge, it now has different weights for acquired customers, and rewards eg the sales force based on these weights.

Please do keep us posted on what you find out!

I’d keep the data separate and analyse each group accordingly. If you market to people through the same channels, then the separate data is the best indicator of the responses you might get. Combining them will result in numbers which have no relevance to the marketing you deploy.

One quick way to have a look at the weighting would be if you had an overlap between the telephone samples and the email samples.

By identifying the overlap in these two groups in the total data, you can get a decent feel for the actual weightings which could be applied. Regardless of being able to do this I’d still look at the two sets of data separately!

Steve Alker