Tool

Sample Size Calculator (Evan's Awesome A/B Tools)

Evan Miller · evanmiller.org Free Intermediate

Before you run anything, put your real numbers into this and see how many conversions per variant you would actually need. It takes your baseline conversion rate and the smallest effect you care about (the minimum detectable effect) and returns the sample size per group, which is the fastest honest way to learn whether a test can ever hit significance. If the number dwarfs your monthly traffic, that is your answer: skip the test and do something qualitative instead.

  • Enter your baseline conversion rate and minimum detectable effect, and it returns the sample size needed per variation.
  • Aiming to detect a small change requires a much larger sample, which makes the low-traffic trap concrete.
  • It is a free, no-signup, single-page tool from a widely trusted source in A/B testing statistics.
Open on evanmiller.org ↗

The question it answers

  1. How do I run a growth experiment properly when my user numbers are too small for statistical significance?

Read with it

  1. This is the honest counter-argument most growth advice skips: when you have low traffic, formal A/B tests usually cannot reach... Entrepreneur Essay The Case Against A/B Testing at Early-Stage Startups Andrew Cohen
  2. Torres takes the next step after you accept you cannot reach significance: figure out what you are actually trying to learn,... Product Talk Article What to Do When You Don't Have Enough Traffic to A/B Test Teresa Torres
eChai Partner Brands The partnership →