Growth & Marketing

How do I run a growth experiment properly when my user numbers are too small for statistical significance?

A starting point

At low volume, classic A/B significance is out of reach, so lean on bigger, obvious bets and qualitative signal rather than pretending a 2 percent lift on 80 users is real. Run sequential changes you can attribute (ship one thing, watch the cohort, talk to users about why), reserve true A/B tests for when you have the traffic, and prefer effect sizes large enough to see with your eyes. Small-sample rigor means being honest that you are learning direction, not proving a number.

Go deeper

Hand-picked from around the web, each with a note on why it earns your time.

3 resources 2 link-checked Read Use

Read

📄 Article

✓ Link checked Free Beginner

Why we picked it Torres takes the next step after you accept you cannot reach significance: figure out what you are actually trying to learn, then pick a method that fits. She walks through swapping A/B tests for customer interviews, paper prototypes, think-aloud sessions, and competitive research depending on the question, so low traffic stops being a dead end. The line to carry: do not let a lack of traffic keep you from learning what you need to learn.

What to Do When You Don't Have Enough Traffic to A/B Test

From Product Talk by Teresa Torres

Match the method to the question: value and desirability questions are better answered by talking to users than by an underpowered test.
Interviews, paper prototypes, and think-aloud usability sessions give directional signal without needing statistical power.
You can also buy temporary traffic (paid search) to a page if you genuinely need a quantitative read.

Open producttalk.org →

✍️ Essay

Free Beginner

Why we picked it This is the honest counter-argument most growth advice skips: when you have low traffic, formal A/B tests usually cannot reach significance for weeks or months, and the setup cost is real. Cohen, a founder himself, says name the problem plainly and lean on focus groups, hallway usability tests, and judgment until you actually have the volume. Read it as a starting point for deciding whether a test is even worth running yet, not as a verdict against ever testing.

The Case Against A/B Testing at Early-Stage Startups

From Entrepreneur by Andrew Cohen

Young products rarely have the sample size to make an A/B result trustworthy, and running one anyway burns founder time better spent on product-market fit.
Early traffic is often skewed (one launch, one ad blast), so even a significant-looking result may not reflect real users.
Small qualitative checks like a quick usability session or talking to a few users can be enough signal at this stage.

Open entrepreneur.com →

Use

🛠️ Tool

✓ Link checked Free Intermediate

Why we picked it Before you run anything, put your real numbers into this and see how many conversions per variant you would actually need. It takes your baseline conversion rate and the smallest effect you care about (the minimum detectable effect) and returns the sample size per group, which is the fastest honest way to learn whether a test can ever hit significance. If the number dwarfs your monthly traffic, that is your answer: skip the test and do something qualitative instead.

Sample Size Calculator (Evan's Awesome A/B Tools)

From evanmiller.org by Evan Miller

Enter your baseline conversion rate and minimum detectable effect, and it returns the sample size needed per variation.
Aiming to detect a small change requires a much larger sample, which makes the low-traffic trap concrete.
It is a free, no-signup, single-page tool from a widely trusted source in A/B testing statistics.

Open evanmiller.org →