When to Stop an A/B Test: Statistical Significance Guide

Short answer
When to stop an AB test?
You stop an AB test when it reaches the sample size and the full weekly cycles you calculated before launch. A significance badge turning green for the first time is not a reason to stop. Unplanned stopping is correct only for broken data or serious harm to a guardrail metric.
That answer sounds simple, but this is where most teams go wrong. They close the test at the first significant result, ship the "winner," and then see no lift a few weeks later. In this guide, we walk through stopping rules, sample size planning and the peeking trap, one step at a time.
At the end, you will find a decision table and a numbered checklist. With both, you can finish every test with the same discipline.
This article is about ending a test correctly, not about setting one up. If you still need to build your first test, start with the beginner guide we link below, and then come back. The stopping rule is the quietest and most expensive decision in any experiment.
Why does the first significant result mislead you about when to stop an AB test?
Every test produces a difference that wobbles as data arrives. So the question of when to stop an AB test matters more than most teams expect. In the first days, a handful of conversions can swing the gap in either direction. For example, variant B may lead after two days, and then the gap may close the following week.
The reason is that significance math assumes the sample is completed as planned. If you check daily and stop at the first green light, you capture a lucky swing. In other words, you pick a good-looking moment instead of a real winner.
NIST's engineering statistics handbook explains that the significance level (alpha) sets the false-positive risk for a single, planned test. That risk does not stay fixed when you check the same test again and again. You can read the details in the NIST section on hypothesis tests.
Think of it this way: five coin flips with four heads do not prove a rigged coin. Also, small samples produce extreme results often. Large samples pull those extremes back toward the average, so early big gaps deserve suspicion.
Which decisions should you write down before the test starts?
A good stopping rule exists before the test begins. If you write the rule after seeing results, you quietly choose an end point that suits you. So prepare a one-page test plan first.
The plan should include these items:
- Hypothesis: what you change, why, and which behavior you expect to move.
- Primary metric: one conversion measure that drives the decision.
- Guardrail metrics: page speed, refunds, cancellations or form errors that signal harm.
- Minimum detectable effect: the smallest lift that justifies the investment.
- Confidence level and power: for example 95% confidence and 80% power.
- Sample size and duration: visitors per variant and the number of full weeks.
- Stop date: a calendar date you do not move.
If you need help designing the page or producing the new variant, our web design services can turn the plan into a working page. First, though, put the plan on paper.
Writing a plan may feel like bureaucracy at first. However, once results arrive, everyone has a different expectation. Marketing wants conversions, leadership wants revenue, and design wants clicks. One agreed primary metric ends that argument before the test finishes. In addition, it gives you a document to return to if the result is disputed.
How do you plan the sample size?
Sample size depends on four inputs: your current conversion rate, the smallest effect you want to detect, the confidence level and the power. The lower your baseline rate and the smaller the effect, the more visitors you need.
You do not have to do the math by hand. Open our free ab test calculator, go to the sample size tab, and enter the baseline rate, the minimum effect, the confidence level and the power. Then the tool returns visitors per variant, total visitors and an estimated duration. It also rounds the duration up to full weeks.
Run this step before launch, because it answers a second question: can you even run this test on your site? If the required duration is many months, redesign the test around a larger effect.
For example, instead of testing one button color, try a bold variant that changes the offer and the layout together. Also, big changes tend to produce big differences. As a result, you reach a conclusion with fewer visitors. However, changing too many things at once hides what worked, so match the size of the change to your hypothesis.
What does a worked example look like?
Here is a sample scenario. This is an example calculation, not a promise about your site.
Say your current conversion rate is 4%. You want to notice if a variant lifts it to 4.4%, which is a 10% relative increase. Choose 95% confidence, 80% power and a two-sided test. The standard two-proportion formula gives roughly 39,000 visitors per variant.
If your site gets 2,000 visitors a day in total, each variant receives 1,000. At that pace, you need about 40 days. Rounded up to full weeks, the test runs for 6 weeks.
- A larger target effect shortens the test.
- A page with a higher baseline rate shows the same relative lift with fewer visitors.
- Adding variants increases the total you need.
These numbers only show the logic. Because your baseline, traffic and target will differ, run the calculation with your own data. So take your daily traffic from recent full weeks, and leave out campaign weeks that distort the average.
What do significance, the p-value and confidence level tell you?
Significance means the chance that the gap appeared by luck stays below a threshold you chose in advance. The p-value reports that chance. For example, with a threshold of 0.05, a result this extreme would appear about once in twenty experiments if no real difference existed.
Meanwhile, the confidence level is the complement of that threshold. A 95% confidence level means you picked alpha at 0.05. NIST notes that 0.1, 0.05 and 0.01 are common choices.
However, the p-value is not "the probability that B beats A." Drop that misreading. It only describes how surprising your data would be if there were no difference.
In addition, a significant result does not mean the difference matters to your business. We cover that gap in the practical significance section below.
A two-sided test is the safer default as well. It checks whether B is better and whether B is worse. Since a new variant can lower conversions, you want the test to be able to show that.
Why do power and minimum detectable effect matter so much?
Power is your chance of catching a difference that truly exists. According to NIST, power is the probability of correctly rejecting a false null hypothesis. A low-power test can wave a good variant through as "no difference."
The minimum detectable effect (MDE) is the smallest difference you aim to catch. Therefore a business decision sets this value. For example, ask what lift you need to cover the cost of building the new page.
Two mistakes are common:
- Choosing a tiny MDE and launching a test that runs for months.
- Choosing a huge MDE and missing small but valuable gains.
So MDE is an investment decision first and a statistical input second.
Power deserves a deliberate choice, too. A common pick is around 80%, and higher power asks for more visitors. For a critical change, such as pricing or checkout, raising the power is sensible. For a low-risk copy test, the standard value is usually enough.
What is peeking, and how does it change when to stop an AB test?
Peeking means looking at results before the test ends and acting on them. First, looking alone is harmless. The problem starts when what you see decides whether you stop. Each look then becomes another chance to find a lucky swing.
A practical rule: monitor, but do not decide. For instance, check data flow, errors and traffic allocation while the test runs. Announce a winner only when the planned sample is reached.
Some tools offer sequential testing. In that method, you plan the number of looks in advance and the threshold adjusts for them. If you use such a tool, follow its documented stopping rule exactly. A classic fixed-sample test gives you no such allowance.
Set a team rule as well. Let only two people open the results screen, and hold the interpretation until the planned date. In addition, send leadership a weekly status with traffic and error notes only, and leave out any winner guesses. This small discipline prevents most early mistakes.
In short, you may say "let me look today," but wait for the plan date before you say "let me close it today."
Why do tests run in full weeks?
First, visitor behavior follows a weekly rhythm. Weekday and weekend visitors arrive with different intent. Also, some businesses see pay-day spikes, sale days or newsletter days that reshape demand.
If you start a test on a Tuesday and end it on a Friday, you over-represent some days and under-represent others. Therefore, run the test for whole weeks. In many cases, covering at least two full weeks is healthier.
When setting the duration, ask yourself:
- Does the buying decision take days or weeks?
- Does a seasonal campaign, holiday or big announcement fall in the middle of the test?
- Does your traffic source mix stay steady throughout?
If your buying cycle is long, a short test pushes you into an early call. In that case, account for conversion delay and extend the duration.
For instance, if a form lead becomes a sale a week later, sales data is incomplete on the day the test ends. For that reason, it is often more practical to tie the primary metric to a step you can measure in time, such as form submissions. Just confirm that this step really relates to sales.
When to stop an AB test before the plan date, and which exceptions count?
Early stopping is an exceptional, pre-agreed decision. Three situations justify it.
First, there is broken data. If the conversion tag fails in one variant, the result means nothing. Stop the test, fix the tag, and restart.
Second, there is serious harm to a guardrail metric. For example, form errors jump in the new variant, or the page slows down badly. When harm like that appears, the loss outweighs the learning.
Third, there is a sample ratio mismatch, which means the test setup is broken. We cover it in the next section.
In none of the three cases do you close the test as "winner found." Instead, you record the problem and the decision in the test log. That way, the team does not repeat the mistake.
Write a harm threshold for each guardrail metric before launch. If you set the threshold mid-test, you bend the rule toward the result you want. Choose the current threshold value together with your team, based on your own business.
Why is sample ratio mismatch the first thing to check?
A sample ratio mismatch (SRM) happens when traffic does not split between variants in the planned ratio. For example, a 50/50 test that sends 5,000 visitors to A and 4,400 to B shows it. The gap looks small, yet chance may not explain it.
This usually signals a technical problem. A redirect might run slowly in one variant, bot traffic might pile up on one side, or a cookie rule might exclude some users. As a result, every metric you compute on a broken setup is unreliable.
So check the traffic split before you read any results. Our calculator flags it with a chi-square test. That check assumes an equal split, though, so if you set up an uneven allocation on purpose, interpret the warning accordingly.
If an SRM warning appears, do not share the result. So find the cause first.
To investigate, follow this order: redirect and load order first, bot and internal traffic filters next, and browser and device mix last. Often, one variant loads slowly and some visitors leave before the tag fires. In that case, restarting without a fix is pointless.
When to stop an AB test that shows no significant result?
A non-significant test is not a failed test. It only shows that your data cannot separate the variants. NIST is clear on this point: failing to reject the null hypothesis does not prove the null hypothesis true.
First, ask why. Did the sample fall short of the plan? Was the effect smaller than expected? Was the hypothesis weak? Each cause has a different fix.
Then you have three options:
- Keep the current version. If the variant did not win, you have no reason to take the risk.
- Try a bolder change. Small copy and color differences often cannot be detected on most sites.
- Do not extend the test with more data. An unplanned extension is just peeking in another form.
So when should you stop an AB test that shows nothing? You stop at the planned date and log it as inconclusive.
Write down what you learned. For instance, the note "headline change alone was not enough" points the next test toward the offer itself. In this way, even an inconclusive result gives direction.
How do multiple variants change the rule?
As the number of variants grows, so does the number of comparisons. Each comparison is another chance, so the odds of finding a lucky "significant" result go up.
Correction methods balance this. The Bonferroni correction is the simplest: divide the threshold by the number of comparisons. For example, at 95% confidence with two comparisons, each one uses an alpha of 0.025. Our calculator applies this automatically from three variants upward.
In addition, every extra variant increases the total visitors you need. If your traffic is limited, run two variants at a time.
Another trap is slicing the data after the test and hunting for a winner. If you scan mobile, new visitors and certain sources one by one, you will find something "significant" somewhere. Unless you planned those segments, treat any finding as a hypothesis for a new test.
Put simply, exploratory analysis creates hypotheses, and a test confirms them. Do not mix the two. If you spot an interesting pattern in a segment, test it separately and on a plan.
What is practical significance?
However, statistical significance only says a difference may be real. It does not say how large the difference is. NIST warns that a very large sample can flag a trivial difference as significant, while a small sample can miss an important one.
So ask two questions together:
- Is the difference clear enough that chance is an unlikely explanation? (statistical significance)
- Is the difference large enough to change your decision? (practical significance)
For example, if the confidence interval is wide, the lower bound of the gain may sit near zero. Shipping the variant is risky in that case. A result where even the lower bound covers your costs is the safer call.
In addition, if conversions rise but revenue or refund metrics get worse, do not declare a winner. Look at the whole table instead of one number.
This approach also avoids a common reporting mistake: presenting a small lift as a big win. So show the confidence interval in your report. Telling leadership "we expect a lift between these two values" is more honest than quoting a single percentage.
How do stopping rules work in Google Ads experiments?
Experiments in an ad account follow the same logic. Google Ads Help explains that a statistically significant result is marked with a blue asterisk and suggests that your data is likely not due to chance. See the Google Ads Help page on monitoring experiments for the details.
The same page recommends running an experiment for roughly two to three weeks so it can gather enough data. If results stay inconclusive, causes can include low traffic, a small split or a truly small difference. The default confidence interval level can also be adjusted, so check the current value in your own account summary.
For setup help, read our Google Ads drafts and experiments guide. The rule stays the same, though: set the duration in advance and do not cut it short.
Ad experiments add one more trap. Automated bidding goes through a learning period, so early days can swing. Count the learning period inside your duration, and do not decide from the first days. Also avoid changing the budget or the traffic split during the experiment.
When to stop an AB test: how do you read the decision table?
The table below gives a practical call for each situation you may meet during a test. Adjust it to your own plan date and guardrail metrics.
| Situation | Decision | Why |
|---|---|---|
| Sample and full weeks complete, result significant | Stop and ship | The plan finished, so the decision is reliable |
| Sample complete, result not significant | Stop and log "inconclusive" | The difference cannot be separated; extending is peeking |
| Significance appears before the plan date | Keep running | The swing may be temporary |
| Tag or measurement bug found | Cancel, fix, restart | The data is untrustworthy |
| SRM warning present | Cancel and find the cause | The setup is broken |
| Guardrail metric seriously worse | Stop at the agreed harm threshold | The loss outweighs learning |
| Full weekly cycle not finished | Keep running | The daily rhythm is missing |
What should your pre-stop checklist include?
Before you call a winner, go through this list in order. Every item matches a trap covered above.
- Did you reach the planned sample size?
- Did you complete the full weekly cycles?
- Does the traffic split match the planned ratio (SRM check)?
- Does the conversion tag work in both variants?
- Is the primary metric significant at your chosen confidence level?
- Does even the lower bound of the confidence interval add value for your business?
- Are the guardrail metrics the same or better?
- Did you change a campaign, price or page during the test?
- Did you write the result and the lesson into the test log?
If any of these nine is a "no," postponing the decision or calling the test inconclusive is the more honest choice. Attach the list to your test plan and tick it in the same order each time. Once it becomes a habit, it takes only a few minutes.
How do you read the result in the calculator?
When the test ends, enter the numbers yourself. Open the A/B test significance calculator, then type visitors and conversions for A and the same for B. Set the confidence level to the value in your plan, and the test type to two-sided.
The tool shows a significance badge, the relative lift and a confidence interval for that lift. It also reports the p-value, observed power and data warnings. You see the sample ratio check on the same screen. In other words, one screen tells you both the result and the health of the setup.
If conversions are very few, for example under 10, the tool shows a warning. Do not decide with data like that.
The tool handles binary conversion metrics only. For continuous measures such as revenue or order value, you need a different test.
If you want to sanity-check the conversion rate itself first, the conversion rate calculator does the job.
How do you log results and feed the next test?
Every test produces two things: a decision and a lesson. The decision ships, and the lesson disappears unless you write it down. So keep a short record for each test.
Then include these fields: hypothesis, date range, sample, result, confidence interval, decision and lesson. Three months later, you will not need to retest the same idea, because the record shows why it failed last time.
To build the idea list and the priority order, our marketing experiment backlog guide helps. When you set up your first test, follow getting started with AB testing. For a broader view of common mistakes, see 8 things marketers should know about AB testing.
Search results need a similar discipline, and how to run SEO A/B tests explains where the differences lie.
Which tools and data should feed your test ideas?
Heatmaps and session recordings help you form hypotheses. However, you need to know which tool answers which question. Our sibling article, which CRO tools you need, answers that in detail.
To feed hypotheses with real behavior, see Microsoft Clarity heatmaps and session recordings. To decide whether optimization pays off for your site at all, conversion rate optimization: when it pays off is a good start.
If traffic is low, think about simple fixes first. For example, our landing page optimization guide lists work that most sites should do before testing. You do not have to test everything. Some problems are obvious, and fixing them and watching the metric is faster and cheaper.
What if you cannot run these tests on your own?
Many businesses build the test but get stuck on duration, sample size and interpretation. That is normal, because this work combines statistics, measurement and design. You can handle many parts yourself with the guide above.
Still, if you need help with the page design, the measurement setup or reading the results, Talha Aslan and our team are here for you. For this part of the process, you can review how we work on our business website design page.
One note: no test guarantees a result in advance. A good team will not promise you a lift. Instead, it gives you tests that are set up correctly, read cleanly and turned into learning.
Core guide: What Is Digital Marketing? Its types, examples and how it is done, on one page with an interactive film that travels through the brain.



