Every CRO blog says the same thing eventually: "you should be A/B testing everything." Button colors. Headlines. Whether your CTA says "Get Started" or "Start Now." It's presented as a settled best practice, the way flossing is presented as a settled best practice — technically correct, mostly ignored, and rarely examined closely enough to explain why.
Here's the examination. For a lot of small businesses, right now, today, running that test is closer to guessing with extra steps than it is to real data.
The Sample Size Myth Actually Cuts Both Ways
The oversimplified version of the sample-size objection says "small businesses can't test, they don't have enough traffic." That's not quite right either — statistical power scales with the square root of sample size, so 10,000x fewer visitors costs only about 100x less statistical power, not 10,000x less. Technically, small-traffic tests aren't mathematically impossible.
They're just slow. A test that a high-traffic site resolves in three days might take a low-traffic site the better part of a year to reach the same confidence level — and by month four, the page, the offer, or the season has usually already changed underneath the test.
The Part Even Big Companies Get Wrong
Here's the number that should give anyone pause before treating testing as a default: companies with millions of users — genuinely enormous sample sizes, no traffic problem at all — report a median success rate around 10% for tests that come back statistically significant and positive. Nine times out of ten, even at that scale, the test either shows no meaningful difference or shows the "loser" variant actually performing better.
Statistical significance and practical significance are not the same thing. A clinical trial with 10,000 participants once found a "statistically significant" one-pound difference in weight loss between two groups — real, measurable, and reproducible, and also completely irrelevant to anyone deciding which approach to actually use. A test can be true and still not matter.
What Actually Works Instead, at Low Traffic
None of this means guessing is the alternative. It means the sequencing should flip: research first, test second — the opposite order most CRO advice implies.
Voice-of-customer research (pulling the actual words your customers use from reviews, support tickets, and calls) doesn't need a large sample to be directionally useful. Three customers independently using the same unusual phrase to describe your product is a real signal, available on day one, with zero visitors required to "reach significance." A/B testing that finding once it's implemented can confirm it later, at whatever pace your traffic allows — but the research is what should decide what gets tested in the first place, not a coin flip between "Get Started" and "Start Now."
If your traffic is genuinely large enough to reach significance in weeks, not months, testing earns its place in the process. If it isn't, spending the next six months waiting on a headline test is six months not spent finding out what your actual customers are already telling you, for free, in the reviews you haven't read yet.