AB Testing for Beginners: How to Set Up Your First Test and Pick a Winner

Short answer
What is AB testing and how do you set up your first test?
AB testing (also written A/B testing) is a controlled experiment that shows two versions of a page or element to randomly split visitors and measures which version drives more of the action you want. For your first test, write a hypothesis, choose one primary metric, plan your sample size and duration, and change only one thing.
This guide walks you through each decision in order. In practice, our focus is hypothesis, duration and sample size. Deciding when to stop a running test is a separate topic, so we keep that part short and point you to our guide on when to stop an AB test.
Why is guessing not enough? Because conversion rates move with seasons, ad traffic and campaigns. An AB test removes that noise, since both versions run at the same time under the same conditions. As a result, you learn what caused the change instead of hoping.
When is AB testing the right method?
However, not every question needs a split test. A test answers "is A or B better?" For "why do visitors leave?" you usually get faster answers from session recordings, surveys and interviews.
AB testing makes sense when:
- You have two clear options and do not know which is better.
- The page gets enough traffic to finish the test in a reasonable time.
- The behavior you measure, such as a purchase, a form or a click, is clearly defined.
- You have the authority to ship the winning version.
Also, building a variant usually takes design and development work. A clean layout, a fast template and consistent components make tests much easier to run. If your team cannot handle that work, web design services can help you prepare test versions that stay clean and measurable.
Where do you find test ideas?
Good tests come from observation, not from guesses. First, start with where the data points. Find the step where you lose the most people, watch session recordings, and list the objections your customers repeat.
Three sources work especially well:
- Analytics data: Which page, device and step show the biggest loss?
- Behavior recordings: Where do visitors stall or click on things that do nothing? Microsoft Clarity heatmaps and session recordings are a practical starting point.
- Customer voice: Which questions do support messages, sales calls and reviews repeat?
Next, put your ideas in a list and rank them by impact, ease and confidence. A medium idea at a high-loss step beats a brilliant idea on a page nobody visits. For general diagnosis, see conversion rate dropped: where to start.
Share the list with sales, support and design too. For example, each team sees a different part of the visitor's doubts. That way your tests rest on real customer questions, not only on marketing assumptions.
How do you write a good hypothesis?
A hypothesis is the compass of a test. For instance, "make the button red" is an idea, not a hypothesis. A hypothesis links an observation, a change and an expected effect, and it explains why.
A practical template reads like this: "We observed X. If we change Y, we expect Z to improve for this audience, because of this reason."
Example scenario: session recordings show visitors hunting for shipping details on a product page. Your hypothesis could be: "If we show the delivery time directly under the add to cart button, more visitors will move from product page to cart, because they no longer need to leave the page to find that information."
A hypothesis like this gives you three things:
- It tells you exactly what to change.
- It tells you which metric to watch.
- It teaches you something whatever the result, because you tested the reason.
What hypothesis mistakes should you avoid?
Five mistakes show up often in first tests. As a result, all of them make results hard to interpret.
| Mistake | Why it hurts | How to fix it |
|---|---|---|
| An idea with no reason ("change the headline") | You cannot learn why it worked | Write the "because" part |
| Several target metrics | You drift toward whichever metric won | Pick one primary metric |
| A vague audience | You do not know who it worked for | Name the device or source |
| A tiny change | The effect may be too small to see | Try a more visible change |
| Writing the hypothesis after the result | You risk reading the data to fit a story | Record it before launch |
Writing the hypothesis after the result is the most common trap. So date it in your test log before you start the test.
How do you choose the success metric?
Choose one primary metric per test. This metric decides the winner. For a sales page that is purchases, for a form page it is submissions, for a content page it might be signups. So pick a behavior close to your real goal.
Mid-funnel metrics such as click rate are easy to measure but can mislead. For example, a button with more clicks does not always bring more sales. Therefore, treat those as supporting data and judge the winner on the primary metric.
You also need guardrail metrics. They check that the winner does not hurt something else. For example, a shorter form may collect more submissions while the quality of those leads drops. Refund rate, page load time or lead quality can also reveal that risk.
Example scenario: for a lead form, your primary metric is submissions. Your guardrail is the share of submissions with a valid phone number. If the variant raises submissions but lowers that share, the gain may be a quality loss. To double-check your baseline, try a conversion rate calculator.
Should you test one change or several?
For your first AB testing rounds, change one thing. Only then do you know what caused the difference. If you change the headline, image and button together, you cannot say which one worked.
Sometimes a full redesign deserves a test. Then the test checks one hypothesis, not one element. For example, "a shorter, trust-focused page brings more signups" can include several changes. Note that it is a bundle and read the result as a bundle.
Here is how the two approaches compare:
- Single-element test: It shows the cause clearly, but it may produce a small effect and needs more traffic.
- Bundle (redesign) test: It can catch a big effect, but it does not separate the causes.
On low-traffic sites, a bundle test is often more realistic, because a large difference is easier to detect.
Which elements are worth testing?
Not all elements carry the same weight. In general, elements closest to the decision moment usually have the biggest effect. So test the offer first, the message second, and visual details last.
Good starting candidates include:
- Value proposition and headline: Does the visitor understand your offer in the first seconds?
- Form structure: Field count, order and error messages. Our guide to web form design covers the details.
- Call to action: Button text and position. For ideas, see CTA button examples.
- Social proof: The placement of reviews and trust signals.
- Section order: Which block, placed first, carries visitors further?
Tiny details such as color shades or corner radius rarely produce an effect you can measure. Leave them for last.
Why do you calculate sample size before the test?
Sample size is the number of visitors in the test. So you decide it before launch. Otherwise, you will be tempted to stop the moment the numbers look good.
Three inputs set the sample size you need:
- Baseline conversion rate: How does the page convert today?
- Smallest difference you care about: How big must the gain be to matter to you?
- Error risk you accept: How low do you want the chance of crowning a false winner?
In short, detecting a small difference takes far more visitors than detecting a large one. For that reason, every test should start with a "smallest meaningful difference." Otherwise, you end up with tests that run for weeks and say nothing.
Example scenario: if your page gets few visitors a month and a modest relative gain would need months of waiting, you want to know that before you build the test. Then you can choose a bigger change or a different method.
How do you use an AB test calculator in practice?
Let's turn theory into practice. First, open the ab test calculator. Enter the visitors and conversions for the control, then the visitors and conversions for the variant. The tool helps you judge whether the difference is likely to be chance.
Example scenario: the control converts 200 of 10,000 visitors, and the variant converts 230 of 10,000. When you enter those numbers, you can see whether the variant's lead is statistically meaningful. These figures are only an example, so enter your own data.
Use the tool in two stages:
- Before the test: estimate the sample you need for the difference you want to detect.
- After the test: enter raw counts and check whether the difference is significant.
Instead, look at the result when you reach your planned sample, not the same day you launch. You can also bookmark the A/B test significance calculator page for quick checks. A calculator does not make the decision. It gives you one piece of evidence for it.
How long should a test run?
Duration depends on two things more than the calendar: reaching your target sample and finishing complete weekly cycles. Because weekday and weekend behavior differ on most sites, plan whole weeks. Ending a test mid-week can therefore give you a skewed picture.
When you plan duration, keep these points in mind:
- Think about your buying cycle. Do people decide in a day or in two weeks?
- Run in full weekly blocks, and do not stop in the middle of one.
- Avoid overlap with holidays, sales and big campaigns. If overlap happens, note it.
- Write the duration before you start, and do not extend it just because the result looks good.
Running too long carries risk as well. Cookie and device changes can show the same person different versions. Google also notes that running a test longer than necessary could be seen as an attempt to deceive search engines. So start with a plan and finish with a plan.
Example scenario: on a site where buying takes several days, a two-day test will undercount the variant, because conversions arrive late. For longer decision cycles, plan a longer test.
How do you split traffic and choose who enters the test?
The healthiest split is random, and usually even. Once a visitor lands in one version, that person should therefore keep seeing it for the whole test. Otherwise, the experience becomes inconsistent and the result breaks.
Who you include matters too. Do you include all page visitors, only new visitors, or only mobile users? If your hypothesis is about mobile shipping information, testing mobile traffic only gives a cleaner result.
Keep three points in mind:
- Randomness: Assign visitors randomly, not by behavior.
- Consistency: The same person sees the same version.
- Audience limits: Exclude internal staff, bots and test orders from the start.
Do not change the split ratio mid-test. Instead, if one version causes a problem, stop and restart. Note also that one person may visit from a phone and a laptop, which can create cross-device inconsistency. You cannot fully remove it, but knowing about it keeps your reading careful.
What technical checks should you run before launch?
Most test errors come from setup, not from design. So a short quality check before launch saves you weeks of bad data.
Use this checklist:
- Do both versions display correctly on all major browsers and devices?
- Does the conversion event fire in both versions? Try it with a test order or a test form.
- Is there a flicker, where the original appears for a moment before the variant loads?
- Does the variant make speed or layout stability worse? The web.dev Core Web Vitals guide explains which loading, interaction and visual stability metrics to watch.
- Is the traffic split close to what you planned? If visitor counts differ a lot, the setup may have a bug.
Check your cache and content delivery settings as well, because caching can serve the wrong version to some visitors. Open both versions in a private window yourself before launch. If even one item fails, do not trust the results. A good test passes quality assurance first.
What should you avoid while a test is running?
Once a test starts, impatience is the biggest danger. Also, looking at results each day is not harmful in itself. However, stopping the moment a version "looks like it is winning" distorts the outcome, because early data holds more noise than real signal.
While the test runs, avoid these actions:
- Editing one of the versions.
- Changing the traffic split.
- Starting another test on the same page.
- Launching a big campaign that shifts your traffic mix.
- Changing the success metric after you see the data.
When to stop is not the subject of this article. For significance and stopping rules, read when to stop an AB test.
What conditions must hold before you call a winner?
One number is never enough to call a winner. Several conditions must hold together. The table below lists what to check.
| Condition | Question | Effect on the decision |
|---|---|---|
| Sample size | Did you reach the planned number? | Keep waiting if not |
| Duration | Did you finish full weekly cycles? | Do not decide if not |
| Significance | Is the gap too large to be chance? | A winner candidate appears if yes |
| Effect size | Is the difference meaningful for the business? | A tiny gap may not justify the rollout cost |
| Guardrail metric | Did something else get worse? | Reconsider the winner if so |
| Consistency | Does the direction hold across device and source? | Investigate why if it flips |
Then, if all six look good, ship the winner with confidence. If one is missing, record the result as a directional signal and run a new test if needed.
How do you read the outcome: win, loss or inconclusive?
An AB testing round ends in one of three ways, and each one has value.
A win: Ship the variant, clean test code from your site, and write down what you learned. Then try the same logic on similar pages with a new hypothesis.
A loss: This is not a failure. Instead, you learned your hypothesis was wrong and avoided shipping a bad change. Think about why you were wrong, because that becomes the raw material for the next hypothesis.
Inconclusive: The difference is not significant. In practice, this means you cannot measure a gap between the versions. Test a bigger change, or move to a page with more traffic.
Not every test produces a winner, and that is normal. What matters is that each test teaches you something. Our article on 8 things marketers should know about AB testing supports this mindset. After you ship a winner, also watch the metric for a few weeks to confirm the gain holds in real life.
How do novelty effects and traffic imbalance spoil results?
A new design can attract extra attention in the first days simply because it looks unfamiliar. That is called a novelty effect. As visitors get used to it, the effect can fade, so the exciting first-week result may not reflect the real difference.
Another risk is a sample ratio problem. If the number of visitors in each version drifts far from the split you set, something may be wrong technically. For example, a slow-loading variant may lose some visitors before tracking fires, which unbalances the counts.
To reduce both risks:
- Run the test in full weekly blocks and long enough to pass the first-week excitement.
- Look at new and returning visitors separately.
- Compare the visitor count of each version roughly every day.
- If counts look wrong, audit the setup before you interpret anything.
Why should you read mobile and desktop results separately?
The same change can help on mobile and hurt on desktop. For example, screen size, connection speed and usage context all differ. Even when the total looks good, an opposite direction in one device group deserves a closer look.
Check the total first, then the device split. However, hunting for splits after the test leads you to random differences. So write down which splits you will inspect before you launch.
Also, mobile behavior is a topic of its own. You can continue with our guide to mobile conversion optimization for ecommerce. When you study click behavior, a CTR calculator helps as well.
Does AB testing affect SEO?
Done correctly, it should not. Google's guide on website testing and Google Search lists four main rules.
- Do not cloak: Showing the crawler one version and visitors another breaks Google's spam policies.
- Use rel canonical: Point alternate test URLs to the original. Google prefers this over a noindex tag.
- Use temporary redirects: If you send visitors to a test URL, choose a temporary redirect, not a permanent one.
- Do not run too long: When the test ends, ship the winner and remove test elements as soon as you can.
Testing titles and snippets that affect search results is a different subject. For that, read how to run SEO AB tests. The AB testing in this guide is about conversion tests that measure on-page experience.
How do you run AB testing on a low-traffic site?
Not every site has enough traffic for AB testing. With very few visitors, small differences are almost impossible to detect. So what can you do?
Three approaches work:
- Test bigger changes: A large gap shows up even in a small sample. Test the offer or page structure instead of button color.
- Add qualitative methods: Even five participants in a usability session reveal serious problems. See how to run a usability test with 5 users.
- Measure a broader metric: Instead of purchases, measure a more frequent step such as add to cart. However, confirm that it links to your end goal.
Before-and-after comparisons look tempting, but they cannot separate seasonality and campaign effects. So treat them as a hint, not as proof. Traffic growth also helps, so plan your testing program together with your acquisition work.
How do you keep a test log?
If you do not record what you learned, you will repeat the same test a year later. A simple table is enough, as long as you keep it consistent.
For each test, write these fields:
- Hypothesis: Observation, change, expected effect and reason.
- Primary and guardrail metrics: Which number decides the winner?
- Sample and duration plan: The targets you set before launch.
- Result: Win, loss or inconclusive, with raw counts.
- Lesson: What does it say about the next idea?
Over the years, then, this log becomes the most valuable knowledge base your team owns. It also gives you concrete evidence when you ask when conversion rate optimization pays off.
Which test method fits which situation?
One method does not fit every case. The table below helps you choose at the start.
| Situation | Suitable method | What to watch |
|---|---|---|
| Enough traffic, two clear options | Classic AB test | Calculate sample size first |
| Low traffic, big change | Bundle test plus qualitative data | Read the result as a direction |
| Offer or messaging question | Value proposition test | See how to test your value proposition |
| Ad image or copy | Creative test | Read the Meta ads creative testing guide |
| True ad contribution | Incrementality test | See incrementality testing explained |
Every method in this table starts with the same discipline: hypothesis, metric, sample and duration. In short, the method changes, but the logic does not.
When should you get web design support for AB testing?
You can set up your AB testing program yourself. In some cases, though, expert help saves time. If variants need design and code, if your pages are slow, or if your measurement setup is unreliable, you need to fix the ground first.
Typical signs include:
- Your tests keep hitting technical problems and you cannot trust the results.
- Your page template makes changes hard.
- Mobile and desktop experiences differ a lot.
- Your tracking setup is incomplete or inconsistent.
At this point, Talha Aslan and team help turn test ideas into measurable page versions and strengthen the foundation. However, we do not promise results. Our approach is to measure, learn and then apply. If you want to work through it together, take a look at our business website design page.
If you also wonder where to look for conversion leaks next, our guide to finding conversion leaks with GA4 funnel exploration is a natural next step.
What does a short action plan for your first AB test look like?
Let's gather all steps into a single plan. For your first test, follow this order:
- Pick a problem step from analytics and recordings.
- Write a hypothesis with observation, change, expected effect and reason.
- Choose a primary metric and a guardrail metric.
- Plan sample size and full weekly duration with a calculator.
- Build the variant with one change and run the technical checklist.
- Launch with a random split and avoid changes mid-test.
- When you reach the target sample, judge the result against the six conditions.
- Ship the winner, clean test elements, and update your test log.
At first glance, this plan looks long. From the second test on, however, it turns into an hour-long habit. In other words, the goal is not speed but accuracy. This way, every test teaches you a little more about why your visitors convert.
Core guide: What Is Digital Marketing? Its types, examples and how it is done, on one page with an interactive film that travels through the brain.



