Know-How Exchange : Research/Metrics

You might run a comparison column chart with the two series placed side by side.

You could also take the average score when they started working and compare against the average score after eight months.

However, you are dealing with small numbers so don't make big decisions based on the results.

You also need to think about what happened during the eight months that might have had a bearing on the outcome.

Sounds like a great learning opportunity. Of course, with just 10 respondents it's almost impossible to generalize. What you're really looking for is whether the difference (in scores) is striking (e.g., 20 first time, 90 second time).

I suggest you look at the individual scores for each person and each question. That way you'll know whether there's a difference based on the individuals (e.g., some people learn faster/better than others), and you will learn whether some "lessons" are more memorable than others.

Beyond that, I'd be careful about using aggregate overall results from such a small respondent panel.

Statistically, I would calculate the difference in scores for each employee. Next, calculate the average and the standard deviatoin of these 10 individual 'improvements' to address whether your employees' improvement has been significant (= significantly different from 0, eg. at the 10% significance level). If you do find a significant difference, think about what may have caused it (Chris' response) - if you do not, it may pay to increase the numbers.

Managerially, I would ask what is really measured in these questions and what is crucial for the employee's value to the company. Maybe certain questions are more important, or maybe it is more important to get (almost) all employees over a threshold (eg a score of 70) than to improve individual scores further. And I agree with mgoodman that your small sample size may not allow you to make general statements

I wonder if this is an opportunity to take the statistics of Mr Bayes for a walk. After all we have some information which is connected to the individuals in the test which is already given – their first test results.

Before doing that, it is important to look at what outcome you are trying to measure. Common sense says that you would expect the scores to increase from low ones to higher ones. If this is the outcome, your inferences about an individual and your success in educating them as a group are obvious and the treatment suggested by Koen is appropriate.

Due to the small sample size, you would be unwise to infer too much about the validity of the outcome as a whole as your standard deviations and variance would show – but what do you do if the results are perverse? What will you do with the data if individuals or worse still, the entire lot show lower scores? Have they failed in some way or have you failed? Or have you educated them to a level in your business whereby they no longer accept that there is a trite right or wrong answer as you originally saw it? I mean, what are you actually going to do with the scores if they don’t behave in the direction you expect?

Once you’ve told us that, we can tell you if it is appropriate to use Bayesian statistics to take account of their original scores!

Steve Alker
Xspirt