A/B Testing
A/B testing compares two versions of a page, email or ad to find which performs better. One variable changes at a time; everything else stays identical.
A/B testing is a controlled experiment that sends two versions of something to separate audiences and measures which performs better against a defined goal.
What A/B Testing Means in Marketing
You have a landing page converting at 3 percent. Someone suggests moving the call-to-action button above the fold. You could just make the change and see what happens. Or you could send half your traffic to the original and half to the new version, measure both for two weeks, and know with statistical confidence whether the change helped or hurt.
That’s A/B testing. The name comes from comparing version A (the control, which is what you have now) against version B (the challenger).
The rule is one variable at a time. If you change the headline, the button colour and the hero image all at once, and conversion goes up, you’ve learned nothing useful. You can’t attribute the change to any single element.
A/B testing is used everywhere: landing pages, email subject lines, ad creatives, onboarding flows, pricing pages. Anywhere there’s enough traffic and a clear metric, a test is possible.
How A/B Testing Works
- Identify the element to test and form a hypothesis: “Changing the CTA from ‘Submit’ to ‘Get my free guide’ will increase clicks.”
- Set a primary metric: button click rate, sign-up rate, revenue per visitor.
- Determine sample size using a significance calculator before starting.
- Split traffic randomly: half to A, half to B, simultaneously.
- Run the test until you reach statistical significance. Do not stop early.
- Implement the winner. Document the result for future reference.
Minimum detectable effect is the smallest improvement worth running the experiment for. Setting it too small means you need impractical traffic volumes. Setting it too large means you’ll miss real but modest gains.
A/B Testing Example
Booking.com built much of its growth on continuous A/B testing across their product, running hundreds of simultaneous experiments. Their approach treats every design change as a hypothesis to be confirmed, not a decision to be made. That culture is worth more than any single winning test.
Why A/B Testing Matters for Marketers
Intuition about what users want is usually wrong. A/B testing replaces opinion with evidence and makes it impossible for the most senior person in the room to override a result just because they prefer the old version.
More importantly, it compounds. A 5 percent improvement in conversion followed by another 5 percent and another is not a 15 percent gain overall. It’s roughly 16 percent, and over a year of testing, that compounds into significant commercial advantage over teams who rely on best guesses.
Frequently Asked Questions
How long should an A/B test run?
Until you reach statistical significance, which usually means a p-value below 0.05. Practically, most tests need at least one full business cycle (typically two weeks) to account for day-of-week variation. Stopping early because one variant is winning is one of the most common testing mistakes, because that lead often reverses.
What is the difference between A/B testing and multivariate testing?
A/B testing changes one element at a time and is easier to interpret. Multivariate testing changes multiple elements simultaneously to find the best combination. Multivariate tests require far more traffic to reach significance because you're splitting visitors across many more variants.
What is a good sample size for an A/B test?
It depends on your baseline conversion rate and the minimum effect you want to detect. A page converting at 2 percent needs tens of thousands of visitors to detect a half-point improvement reliably. Use a sample size calculator before launching, not after.