A/B Test Significance Calculator

What does an A/B test significance calculator actually tell you?

An A/B test significance calculator tells you whether the conversion gap between two versions is real or just noise. You show two headlines, two ad texts or two pricing layouts to separate visitors over the same period, then enter each version's visitors and conversions. From those four numbers the tool derives the p-value, the confidence level and a confidence interval for the lift.

On its own, the calculation does not answer “which one is better?”. Instead it answers a narrower question: would this difference most likely hold if you collected the data again? For example, B may show a 17.5% lift, yet with a small sample the same gap can vanish the following week. That is why I recommend reading three outputs together:

• Badge: is the difference significant at your confidence level?
• Confidence interval: where could the true lift sit?
• Data check: did traffic split evenly, and is there enough data?

If you run the test through ads, clean tracking comes first. For instance, tagging each variant's traffic with the UTM builder keeps sources apart when you analyse the result later.

What do the p-value and confidence level really mean?

The p-value is the probability of seeing a difference at least as large as yours if A and B actually performed the same. So a p-value of 0.0079 means two equal versions would drift this far apart only about 8 times in 1,000. At 95% confidence the tool compares it with a 0.05 threshold; below that, the badge turns “Significant”.

The confidence card shows 1 minus the p-value. However, it does not mean “B is better with 99.2% probability”, which is a very common misreading. The American Statistical Association's statement on p-values makes two points worth remembering:

• A p-value does not measure the size of an effect or the importance of a result.
• Business decisions should not rest only on whether a p-value passes a threshold.

Therefore the calculator shows the confidence interval of the lift next to the p-value. If the interval runs from +4.6% to +30.4%, the lift is significant, but the true effect could be modest or large. In practice, compare the lower end, your most pessimistic case, with the business goal before you ship.

How do you size a test before using an A/B test significance calculator?

Half the work with an A/B test significance calculator happens before launch. The Sample Size tab needs four inputs: your baseline conversion rate, the minimum detectable effect (MDE), the confidence level and statistical power. Power is the chance of catching a difference that truly exists; 80% is the usual default in statistics.

The logic is simple: small differences need a lot of data. Worked example: to detect a 10% relative lift on a 5% baseline with 95% confidence and 80% power, you need 31,234 visitors per variant. Raise the target to 20% and the need falls to 8,158. In short, halving the MDE roughly quadruples the sample.

Add your daily visitors and the tool turns the sample into a duration, rounded up to full weeks so weekdays and weekends weigh equally in both variants. If the duration looks too long, you have three levers:

• Test a bolder change, which means a larger MDE.
• Send more traffic to the test; my guide on setting a Google Ads budget helps with the paid side.
• Move the goal higher up the funnel, for example add to cart instead of purchase.

Why does stopping a test early crown false winners?

The most expensive A/B testing habit is checking results daily and stopping the moment the dashboard says “significant”. Every extra look gives chance another shot at showing you a false winner. In the example scenario of Evan Miller's article How Not To Run an A/B Test, this habit pushes the false positive rate you believe is 5% up to 26.1%.

According to the same source, if you plan to peek 10 times, the reported significance has to reach 1.0% to keep a true 5% level. In practice the cleanest routine has three steps:

1. Fix the sample size and duration before launch with the Sample Size tab.
2. While the test runs, watch only for technical problems: pages that fail to load and SRM warnings.
3. Once you reach the planned number of visitors, read the result once and decide.

Also keep in mind that a winner's effect can shrink after the first weeks once the novelty wears off. That is why I keep watching the conversion rate for a few weeks after shipping a major change.

What should you do when the SRM check fires?

A sample ratio mismatch (SRM) occurs when you planned an even split but the visitor counts per variant drift apart more than chance allows. The calculator tests this with a chi-square test and shows a warning in the data box when p < 0.01. For instance, a 20,000 vs 19,000 split looks harmless, yet at that volume it is a real deviation.

According to Microsoft Research's guide to diagnosing SRM, roughly 6% of A/B tests at Microsoft show an SRM, and the team applies an even stricter threshold of p < 0.0005 before revealing results. Common causes include:

• Bugs in the code that assigns users to variants.
• Visitors lost while redirecting to the variant page.
• A variant that loads slowly or breaks, so fewer of its visits reach your logs.
• Changing the traffic split by hand mid test.

Do not trust the p-value while SRM shows. First find the cause, fix it and restart the test. That said, if you split traffic unevenly on purpose, say 90/10, this check does not apply and you can ignore the warning.

Why does a third variant need a Bonferroni correction?

When you press + Variant C, the calculator compares B and C with A separately. Each comparison gives chance another opening, so with two comparisons the 5% error rate climbs to almost 10% overall. The Bonferroni correction therefore divides the significance threshold by the number of comparisons; at 95% confidence the tool judges each comparison at α = 0.025.

Worked example: A has 5,000 visitors and 150 conversions, B has 5,100 and 180, C has 4,950 and 190. C shows a +27.9% lift with p = 0.0214, which stays below 0.025, so C is significant. B, on the other hand, reaches +17.6% but p = 0.1346, so it is not. The headline shows the variant with the highest rate, and the chart lists each variant's lift over A with its p-value.

The correction has a price: it lowers power, so each variant needs more visitors to catch the same effect. As a result the Sample Size tab applies the same correction when you raise the number of variants. With limited traffic, I would rather test two strong ideas than three average ones.

How should you read the Bayesian probability to beat A?

The P(B > A) card answers a different question than the classic test: given the data, how likely is B's true conversion rate higher than A's? The tool builds a Beta(conversions + 1, visitors - conversions + 1) distribution for each variant and approximates the difference with a normal distribution. Because the prior is flat, no variant gets a head start.

It is the easiest output to explain to stakeholders, because “B has a 99.6% chance of being better” needs no translation. Still, I have two caveats:

• A high probability can come with a tiny lift, so read it together with the confidence interval.
• The Bayesian view is not immune to peeking either; the rule of a sample size fixed in advance still applies.

The observed power card, meanwhile, is informational rather than a planning tool. It derives from the p-value and cannot prove after the fact that a test was large enough. For new tests, always rely on the power setting in the Sample Size tab.

How do you run clean tests in Google Ads and on landing pages?

On the ads side, Google Ads experiments split traffic and budget so you can test campaign changes safely. In its experiment help pages Google recommends a 50% split, running experiments for at least 4 to 6 weeks and waiting for 1 to 2 conversion cycles; it also discards the first 7 days of data as ramp-up. To double check a result, enter the clicks and conversions from the experiment report into this calculator.

Landing page tests also need to protect SEO. Google Search Central recommends these rules while a test runs:

• Never show Googlebot different content than visitors see, which counts as cloaking.
• If variants live on separate URLs, add a canonical tag that points to the original page.
• If you test through redirects, use a temporary 302 redirect instead of a permanent 301.
• When the test ends, ship the winner and remove the test code.

For clients I set up this routine as part of Google Ads management. I cover the conversion side step by step in my article on the CRO process for business websites. To see how a winning variant changes ad profitability, the ROAS calculator also helps.