The peeking problem: why stopping an A/B test early breaks your stats
A methodology guide for CRO teams. Published 7 July 2026.
Peeking is the habit of watching a running A/B test and stopping it the moment the result reads significant. It feels responsible. You are paying attention, you spotted a winner, you shipped it fast. In reality it is the single most common way teams ship false winners, because the classic significance test was never designed to be checked over and over.
The peeking problem is the inflated false positive rate that results from repeatedly checking a running A/B test and stopping at the first significant reading. A test designed for a 5% false positive rate can reach 20% to 40% actual error if you check daily over several weeks. The fix is to either commit to a pre planned sample size and read the result once, or use a sequential testing method that accounts for repeated looks. ABTestly Pro offers sequential (always valid) testing for exactly this purpose.
What peeking actually is
A standard fixed-horizon test works like this: you decide the sample size in advance, run until you reach it, then look once and read the result. The 5% significance threshold (p<0.05) is the promise that if there is truly no difference between A and B, you will be fooled into calling a winner only 5% of the time.
Peeking breaks the promise. Instead of looking once at the planned finish line, you look every day and stop as soon as p drops below 0.05. That single change quietly rewrites the maths, and not in your favour.
Why repeated looks inflate false positives
Think of each look as a separate coin toss for a false alarm. One look at a true null gives you a 5% chance of a spurious significant result. But the p-value of a live test wanders up and down as data trickles in, driven by ordinary random noise. Every time you look, you give that wandering line another chance to dip below the threshold, even when nothing real is happening.
This is the multiple-comparisons problem wearing a disguise. You are not comparing many variants, you are testing the same variant many times, and each test carries its own 5% risk. Stack enough looks together and the chance that at least one of them trips the wire climbs steeply.
A concrete intuition: if you check a genuinely null test once a day for a few weeks and stop at the first significant reading, your real false-positive rate is not 5%. Depending on how often you look, it can climb to 15%, 25%, or higher. You have quietly converted a strict test into a loose one, and you did it by being diligent.
Worked example: how peeking inflates your error rate
Imagine you plan a test that needs 20,000 visitors per variant to detect your minimum detectable effect at 95% confidence and 80% power. The test will take four weeks to reach full sample. There is no real difference between control and variant; the true effect is zero.
If you look once at the end, your chance of a false positive is 5%, exactly what you designed for. But suppose instead you check the dashboard every day and plan to stop and ship the moment p drops below 0.05. Each day the p value wanders randomly, and each look gives it a fresh chance to cross the threshold by noise alone.
After four weeks of daily checks (28 looks total), the probability that the wandering p value dipped below 0.05 at least once is far higher than 5%. Simulation studies and the mathematical theory of repeated significance tests (Armitage, McPherson, and Rowe, 1969) show that the cumulative false positive rate climbs roughly as follows:
| Check frequency | Looks over 4 weeks | Nominal false positive rate | Actual false positive rate |
|---|---|---|---|
| Once at the end (no peeking) | 1 | 5% | 5% |
| Weekly | 4 | 5% | ~11% |
| Every other day | 14 | 5% | ~20% |
| Daily | 28 | 5% | ~25% |
| Twice daily | 56 | 5% | ~30% |
| Hourly (continuous monitoring) | ~670 | 5% | ~40% |
To make this concrete: you check after day 7 (5,000 visitors per variant) and see p = 0.03. That looks convincing. You stop the test and ship the variant. But at only 25% of the planned sample, the estimate is noisy and the threshold crossing is likely a random fluctuation. If you had waited until day 28, the p value would most likely have drifted back above 0.05, because there was never a real effect to sustain it.
The core lesson is that a single p value of 0.03 means something very different when it is the one planned look versus one of 28 opportunistic looks. The number on the screen is the same, but the statistical guarantee behind it is not. You can verify the sample size you need for your own test in the significance card of the A/B test calculator.
Why the false winners feel so convincing
Early stopping does not just inflate the count of false positives, it also biases the size of the effect you report. A test that crosses significance early tends to do so on a lucky upward swing, so the measured lift is exaggerated. You ship the variant, the swing regresses toward the true (often zero) effect, and the win you celebrated never shows up in revenue. The test looked productive. The decision was noise.
What to do instead
There are two honest ways out, and they are not mutually exclusive.
Fix the horizon in advance. Decide the sample size before you launch, based on your baseline conversion rate, the minimum detectable effect you care about, 95% significance, and 80% power. Then run to that number and read the result once. Our guide on how long to run an A/B test turns that sample size into a calendar date, and the A/B test calculator does the arithmetic for you.
Use a method built for continuous monitoring. Sequential testing and always-valid inference are designed so that you can look as often as you like without inflating error, by adjusting the threshold to account for the repeated looks. These methods trade a little raw power for the freedom to peek safely.
ABTestly offers both routes. Its fixed-horizon frequentist method asks for a sample size you plan in advance and reads Still collecting until you reach it, the honest state for a test that has not gathered enough evidence yet. On Pro, its sequential (always-valid) method is built for continuous monitoring, so you can peek safely while the test runs. You choose the method per experiment, and the results view shows the uncertainty measure for the method you chose (a confidence interval for the frequentist method, a credible interval for Bayesian) without declaring a winner before that method's decision threshold is met.
If you want the details of how we count conversions and compute significance, the results methodology docs are public, and the SRM docs explain the chi-square check that flags a broken split before you trust any result.
The short version
Peeking feels like diligence and behaves like cheating. Looking at your dashboard is fine. Stopping the moment it turns green is where the damage happens. Decide your sample size before you launch, run to it, and read the result once, or adopt a sequential method that is honest about repeated looks. Either way, let the plan end the test, not your enthusiasm.
Further reading
- A/B test calculator to compute sample size and check significance for your own test.
- Minimum detectable effect, explained covers how to choose the MDE that sets your sample size.
- How long to run an A/B test turns the sample size into a calendar finish date you can commit to.
- Statistics engines documentation details ABTestly's frequentist, sequential, and Bayesian methods, including how sequential testing eliminates the peeking problem.
FAQ
Is it OK to check my A/B test results before it finishes?
Looking at the dashboard is fine. The problem is acting on it: stopping the test the moment it crosses p<0.05. Each extra look is another chance to cross the line by chance, so early stopping is what inflates your false-positive rate, not viewing the numbers.
How much does peeking inflate the false-positive rate?
It depends on how often you look, but the effect is large. A test with a nominal 5% false-positive rate can reach 15% to 25% or higher if you check it daily over several weeks and stop at the first significant reading. The more looks you take, the worse it gets.
Does ABTestly let me stop a test early when it looks significant?
ABTestly offers both a fixed-horizon frequentist method and a sequential, always-valid method on Pro. The frequentist method will not declare a winner before the test reaches significance at the sample size you planned for, and until then it shows Still collecting. The sequential method is designed for continuous monitoring, so you can peek safely while the test runs. You choose the method per experiment.
How ABTestly computes results is public: see the results methodology docs and the SRM docs.