Article

What to Do When You Don't Have Enough Traffic to A/B Test

Teresa Torres · Product Talk Free Beginner

Torres takes the next step after you accept you cannot reach significance: figure out what you are actually trying to learn, then pick a method that fits. She walks through swapping A/B tests for customer interviews, paper prototypes, think-aloud sessions, and competitive research depending on the question, so low traffic stops being a dead end. The line to carry: do not let a lack of traffic keep you from learning what you need to learn.

  • Match the method to the question: value and desirability questions are better answered by talking to users than by an underpowered test.
  • Interviews, paper prototypes, and think-aloud usability sessions give directional signal without needing statistical power.
  • You can also buy temporary traffic (paid search) to a page if you genuinely need a quantitative read.

The question it answers

  1. How do I run a growth experiment properly when my user numbers are too small for statistical significance?

Read with it

  1. This is the honest counter-argument most growth advice skips: when you have low traffic, formal A/B tests usually cannot reach... Entrepreneur Essay The Case Against A/B Testing at Early-Stage Startups Andrew Cohen
  2. Before you run anything, put your real numbers into this and see how many conversions per variant you would actually need. It... evanmiller.org Tool Sample Size Calculator (Evan's Awesome A/B Tools) Evan Miller
eChai Partner Brands The partnership →