Question

Topic: Research/Metrics

Sufficient # Of Direct Mail For A/b Testing?

Posted by Anonymous on 3/3/2008 at 4:59 PM ET 25 Points
I'm planning on doing direct mail mailouts to businesses in my state. Is there a sufficient number of mail I should send out to do A/B testing correctly?

For instance, can I send out 500 of each (1000 total)?

I'm not a stats guru by any stretch, so any help would be greatly appreciated. Thank you.
To continue reading this question and the solution, sign up ... it's free!
SUBSCRIBE
Or, SIGN IN

RESPONSES

  • Posted by arthursc on 3/3/2008 at 5:18 PM Accepted
    Generally, in simple A/B or A/B/C tests, it's recommended that the control panel and each test panel or split be at least 5000, to get statistically valid data. However, sometimes we don't have that many records, so I've mailed as few as 2000 in a test panel. But unless that test panel shows a huge difference in response rates, I do not change my strategy or tactics based on the results of one test with that low qty. I would repeat the test enough times until I felt secure about the results.

    If you can't do even 2000 in test panel, then of course do what you can, but be careful about making changes unless the test response is overwhelming.

    More complex testing models create dozens or even hundreds of cells of one or a few hundred, but that is not applicable for your situation.

BACK TO TOP

Post a Comment

Related Discussions

Community Info

Top 25 Experts
(Research/Metrics)