Analytics and Data Intermediate

Confidence Interval

A confidence interval is a range of values around a statistical estimate that indicates the likely true value, used in A/B testing to assess result reliability.

A confidence interval is a range of values around a sample-based estimate that gives you an indication of how precise that estimate is, used widely in A/B testing and statistical analysis to communicate the uncertainty in a result.

What Confidence Interval Means in Marketing

When you run an A/B test and find that variant B has a 3.2% conversion rate versus variant A’s 2.8%, that 0.4% difference is an estimate based on the visitors who came through your test. It is not a certain measurement of reality. If you ran the test again with different visitors, you’d get a slightly different number. The confidence interval expresses the range within which the true difference probably falls.

A 95% confidence interval around that 0.4% difference might run from 0.1% to 0.7%. This tells you that while your best estimate is 0.4%, the true effect could be as small as 0.1% or as large as 0.7%. You can be reasonably confident there is a positive effect, but its size is uncertain.

The confidence interval is one of the most important concepts for avoiding bad decisions in marketing experimentation. Tests stopped early, when the confidence interval is still very wide, often produce misleading results. A variant that looks like a 20% improvement after 200 visitors might be indistinguishable from the control after 2,000 visitors, because the early data was noise.

How Confidence Interval Works

The calculation. A confidence interval is calculated from the sample size, the observed value and the standard deviation of the data. Larger sample sizes produce narrower intervals because more data reduces uncertainty. Higher variance in your outcome data produces wider intervals.

The conventional threshold. A 95% confidence level is the standard in A/B testing. It means you’d be wrong about 5% of the time if you consistently used this method. Some testing tools default to 90% confidence, which sounds almost as good but accepts a higher error rate. Know what confidence level your tool uses.

Intervals in testing reports. Most A/B testing platforms display confidence intervals visually or numerically in their results dashboards. A wide interval that crosses zero means you don’t have a conclusive result yet. A narrow interval entirely above zero means you can conclude variant B outperforms variant A.

The relationship with statistical significance. A result is statistically significant at the 95% level when its confidence interval does not include zero (for a difference) or the null hypothesis value (for a single estimate). Significance and confidence interval are different ways of reporting the same underlying statistical information.

Confidence Interval Example

An email marketing team runs an A/B test on subject lines for a promotional email. After three days, variant B shows an open rate of 28% versus 24% for variant A. The testing tool reports: “85% confidence, confidence interval: -0.5% to +8.5%.” The interval includes zero, meaning the result is not yet conclusive. The team waits another four days. The interval narrows to “+1.2% to +6.8%”, no longer includes zero, and confidence reaches 96%. Now they declare a winner.

Why Confidence Interval Matters for Marketers

Marketers who read only point estimates (variant B is 4% better) and ignore confidence intervals make more bad decisions. The interval tells you how much to trust the estimate. Deploying a winning variant from an underpowered test is one of the most common ways to waste development effort on changes that don’t actually improve performance.

Frequently Asked Questions

What does a 95% confidence interval mean?

It means that if you ran the same experiment many times under the same conditions, 95% of the confidence intervals calculated would contain the true population value. It does not mean there is a 95% chance the true value is in your specific interval. The distinction matters: a 95% confidence interval is a statement about the reliability of the method, not about your particular result. In practice, 95% is the conventional threshold for declaring a statistically significant result.

How does confidence interval relate to statistical significance in A/B testing?

In A/B testing, a result is typically called statistically significant when the confidence interval for the difference between variants does not include zero (meaning you can be confident there is a real difference, not just noise). A wide confidence interval that includes zero means you cannot conclude that one version outperforms the other. Narrowing the confidence interval requires more data, which is why running tests until you have sufficient traffic is critical before drawing conclusions.

Why should marketers care about confidence intervals?

Because a result with a wide confidence interval might look like a big improvement but could easily be noise. If your A/B test shows a 15% conversion rate improvement but the confidence interval runs from -5% to +35%, the true effect could be negative. Declaring a winner based on early data with a wide confidence interval leads to deploying changes that don't actually improve performance, or worse, hurt it. Waiting for the interval to narrow before declaring a winner is one of the most important practices in ethical A/B testing.

Consulting & Strategy

Need help putting this concept into practice on your campaigns? Share your current targets and I will suggest the right approach.

Discuss your campaigns