Question

Topic: Research/Metrics

Minimum Number Of E-mail Contacts For Useful A/b Testing

Posted by tech_marketer on 9/3/2010 at 1:46 PM ET 250 Points
Is there a statistical minimum number of people on your e-mail list to do legitimate A/B testing.

Sometimes we have a targeted e-mailer with 1,000 people or less, and wondering if A/B testing is not useful in these circumstances,

Thanks
To continue reading this question and the solution, sign up ... it's free!
SUBSCRIBE
Or, SIGN IN

RESPONSES

  • Posted on 9/3/2010 at 2:22 PM Member
    Hmmm, the minimum number of people on the list probably depends on your response rate. If CTR is most important to you, and it's above 3%, then some statisticians would say that 30 clicks is enough to make predictions of future performance. If the difference in performance between A and B is significant enough, you can learn something of value with even a lower response.

    Matt
  • Posted by wnelson on 9/3/2010 at 5:36 PM Accepted
    Ehab,

    The answer to your question is, "YES, there is a minimum number of people to survey for an A/B test." Absolutely! The sample size depends on how accurate you want to detect. For instance, if you have an "open" rate of 30% and you want to test if the "new" email is better by 33% - i.e. you'd get 40% to open, you'd need 200 people minimum. If you want to detect the difference between 30% open rate and 33% open rate, you'd need 2000 people. With 1000 in your email, you could tell the difference between 30% open and 34,5% open. What this means is that if the "old" format open rate was 30% and the new format open rate was greater than 34.5%, then with a 95% confidence, you could say that the new format was better than the old. You can play around with different numbers using the following tool:

    https://www.emarketingdynamics.com/plancalc.asp

    The underlying assumptions here are that your 1000 people can be considered as "random" samples. That means that in a LARGE population, you have an equal chance of selecting those 1000 as any other. Also, you are assuming that other factors are the same - like the time of day, day of the week, etc.

    All this statistical stuff being said, the real thing that matters is how important is this to your business? For instance, let's say you want to detect the difference between 30% open and 34.5% open. If you send 1000 emails, that's the difference between 30 and 35 people opening. What does the business derive from an "open?" If it's $10,000 per open, that's $30,000 or $35,000 - or a $5,000 difference. That means the experiment and the use of the data is worth $5,000. That might be worth it. But, if the value of an "open" is $1, then we're talking a difference of $5. Who cares. Make sure you understand the value of the information before you look at the sample size.

    I hope this helps.

    Wayde
BACK TO TOP

Post a Comment

Related Discussions

Community Info

Top 25 Experts
(Research/Metrics)