Insights / Perspective / Conversion rate optimisation (CRO)
PerspectiveA/B testing without fooling yourself
A test result is only as good as its design. Stopping early, changing several things at once and measuring clicks instead of orders all produce false wins.
A/B testing has an air of scientific authority. Split the traffic, compare the results, let the data decide. It is a genuinely powerful discipline, and organisations that test well make steadily better decisions about their websites, emails and apps.
But a test result is only as good as its design. In our experience, a large share of the “wins” reported by testing programmes would not survive a careful second look. The changes are rolled out, the expected improvement never appears in the monthly figures, and confidence in testing quietly erodes. The causes are almost always the same few mistakes, and all of them are avoidable.
The temptation to stop early
The most common error is stopping a test as soon as one version looks ahead. Early in any test, results swing widely because the numbers are small. A variant can lead by a large margin after two days and be level after two weeks. If you check daily and stop the first time the result looks significant, you will declare many winners that are really just noise.
The remedy is to decide the sample size before the test starts. Free calculators, and most testing tools, will tell you how many visitors or conversions you need to detect a given improvement with reasonable confidence. The smaller the improvement you want to detect and the lower your baseline conversion rate, the more traffic you need. Run the test until you reach that number, and for at least one or two full weeks to capture weekday and weekend behaviour, whatever the interim results look like.
If the numbers do not add up
If the calculator says you need six months of traffic to detect the improvement you expect, the test is not viable as designed. Test a bolder change, test on a higher-traffic page, or accept that this decision will be made on judgement rather than a formal test.
Changing one thing at a time
Teams under pressure to deliver results often bundle changes: a new headline, a different button colour, a shorter form and a new image, all in one variant. If the variant wins, nobody knows why. If it loses, a good idea may have been sunk by a bad one alongside it.
Bundling has its place. A complete redesign tested against the current page can answer the question “is the new direction better overall?” But it cannot tell you which elements to keep. For learning that builds over time, change one meaningful thing per test, or use a properly designed multivariate test where traffic is high enough to support it.
Measuring orders, not clicks
It is tempting to judge a test on the metric nearest to the change. A new product page layout is measured on add-to-basket clicks; a new email subject line on opens. These measures react quickly and produce clear results. But they are means, not ends.
A more prominent button may increase clicks while attracting less committed visitors who abandon at checkout. A provocative subject line may lift opens while reducing purchases. Always measure the outcome the business cares about, usually completed orders or revenue per visitor, as the primary result. Treat click-through measures as useful diagnostics, not verdicts.
Where order values vary widely, look at revenue per visitor as well as conversion rate. A change that converts slightly more visitors but nudges them towards cheaper products can reduce revenue overall, and only a revenue measure will reveal it.
Other ways tests mislead
Several subtler problems are worth knowing about.
- Testing during unusual periods: results from a sale week or a major campaign may not hold at other times.
- Uneven splits: if a tracking or technical issue sends more of one type of visitor to one variant, the comparison is unfair. Check that the split is balanced by device and traffic source.
- Novelty effects: returning visitors may react to anything new at first, an effect that fades within weeks.
- Testing many variants at once without adjusting for it, which raises the chance of a false winner.
- Running overlapping tests on the same page or journey, so one test contaminates another.
Building a testing habit that holds up
The organisations that get the most from testing treat it as a discipline rather than a series of experiments. Before each test, they write down a hypothesis grounded in evidence, the primary measure, the sample size and the duration. They resist peeking. They record every test, winners and losers, in a shared log so that learning accumulates and the same ideas are not tested twice.
They also check whether winning tests deliver in practice. After a winner is rolled out, compare the relevant measure over the following months with the period before. If the improvement does not appear, investigate. It may be seasonal variation, or it may be a sign that the original test was flawed.
We used to celebrate a win every fortnight. When we started checking whether those wins showed up in revenue, we learned more about our testing process than about our website.
Digital director, consumer brand client
A short discipline for your next test
Before your next test, write five lines: the hypothesis, the single change, the primary measure, the sample size and the end date. Share them with the team and agree not to stop the test early. When it ends, record the result in your log, whatever it is. Over a year, that small discipline will make your testing programme far more trustworthy than any new tool.
Source: Intelliversal analysis.
