Know-How Exchange : Research/Metrics

Dear Pint,

I believe most respondents do understand a ranking question, and see no problems in your industry in this regard. However, you can experiment with a few different ways for getting the information you need:

a) evaluate EACH of the service attributes (eg out of 10, or 100, or indicate the importance on a 1-7 scale)
b) DIVIDE 100 points over the following 7 service attributes
c) evaluate following service PROFILES that combine different levels of the attributes (conjoint analysis)

a) is easy for respondents to fill out as it requires little thought (which increases response rates but may yield lower quality of answers)
b) forces respondents to carefully consider how much each attribute really contributes to their purchase decision. Therefore, it takes more time, but may yield information that better represents their evalution process. Typical example is price, which often gets the highest score in a), but often ends up a lot less important than other attributes in b) and c)
c) is more representative of how people evaluate competing offers, but may be way too cumbersome with 7 attributes. Can you reduce them to 4-5 key ones with 2-3 levels each?

Practically, I would work out the above (especially a and b) and present them to a few respondents you know very well. They can give you feedback about which is better, after which you can conduct your survey

Dear Pint

I too think that a ranked question would work as it forces the recipient to evaluate their priorities. Even if they do this off the top of their head there is research which states that gut instincts are actually more accurate in a survey than careful reasoning.

Another reason for going with a ranked question is that setting separate questions each with a separate score would make the questionnaire 6 questions longer than it need be. Participants shortly followed by you will have to unravel potentially conflicting answers and come up with a valid statistical treatment of the results.

Juliet (She’s in the top 100 here) and I used SurveyGizmo whose directors are also members of this site to do a survey with several ranked questions which was sent out to school teachers. They confounded our estimates of a 10% response rate by contributing about a 37% response. We attributed this excellent result to the subject under consideration and the fact that the questionnaire was easy to complete – ranked questions included.

The ranked questions were hardest to evaluate, but to give the varying opinions the correct weighting we multiplied the number of responses with their ranking in order to skew the answers more dramatically from strongly dislike though to strongly like. This is simply akin to cranking up the gain in a system and does not change the interpretation of the result – it simply makes it easier to see!

Good luck

Steve Alker
Xspirt