How do I run a growth experiment properly when my user numbers are too small for statistical significance?
The short answer
At low volume, classic A/B significance is out of reach, so lean on bigger, obvious bets and qualitative signal rather than pretending a 2 percent lift on 80 users is real. Run sequential changes you can attribute (ship one thing, watch the cohort, talk to users about why), reserve true A/B tests for when you have the traffic, and prefer effect sizes large enough to see with your eyes. Small-sample rigor means being honest that you are learning direction, not proving a number.
Go deeper, your way
3 hand-picked resources, 2 link-checked. Pick how you want to dig in.
📄 Article
✓ Link checkedFreeBeginner
Why we picked it
Torres takes the next step after you accept you cannot reach significance: figure out what you are actually trying to learn, then pick a method that fits. She walks through swapping A/B tests for customer interviews, paper prototypes, think-aloud sessions, and competitive research depending on the question, so low traffic stops being a dead end. The line to carry: do not let a lack of traffic keep you from learning what you need to learn.
Why we picked it
This is the honest counter-argument most growth advice skips: when you have low traffic, formal A/B tests usually cannot reach significance for weeks or months, and the setup cost is real. Cohen, a founder himself, says name the problem plainly and lean on focus groups, hallway usability tests, and judgment until you actually have the volume. Read it as a starting point for deciding whether a test is even worth running yet, not as a verdict against ever testing.
Young products rarely have the sample size to make an A/B result trustworthy, and running one anyway burns founder time better spent on product-market fit.
Early traffic is often skewed (one launch, one ad blast), so even a significant-looking result may not reflect real users.
Small qualitative checks like a quick usability session or talking to a few users can be enough signal at this stage.
Why we picked it
Before you run anything, put your real numbers into this and see how many conversions per variant you would actually need. It takes your baseline conversion rate and the smallest effect you care about (the minimum detectable effect) and returns the sample size per group, which is the fastest honest way to learn whether a test can ever hit significance. If the number dwarfs your monthly traffic, that is your answer: skip the test and do something qualitative instead.
