Check whether your A/B test result is statistically significant with a two-proportion z-test.
Your variant beat the control by 20% — or did it? With a few thousand visitors, random noise routinely produces “winners” that vanish on rollout. Our free A/B Test Significance Calculator runs the proper two-proportion z-test on your visitor and conversion counts and tells you, honestly, whether the difference is real or luck: conversion rates, z-score, p-value, and a verdict at 95% confidence.
Statistical significance answers one question: if the two versions were actually identical, how surprising would this result be? Surprising enough (p below 0.05, equivalently |z| above 1.96), and you can act on the winner. Not surprising, and the “lift” is noise wearing a costume.
Enter visitors and conversions for A (control) and B (variant). The widget computes both conversion rates, the pooled z-test, the p-value, the relative uplift — and delivers the verdict: significant or not, at both 95% and the stricter 99% level.
Every A/B test starts with the null hypothesis: the two versions convert identically, and any observed gap is luck. The z-test quantifies exactly how uncomfortable the data makes that hypothesis. It pools both samples into one overall rate, computes the standard error of the difference (how much wobble random sampling produces at this sample size), and expresses your observed gap in units of that wobble — the z-score.
The magic numbers 1.96 and 2.576 come from the normal distribution: 95% of random wobble falls within ±1.96 standard errors, 99% within ±2.576. So |z| > 1.96 means your result sits outside the range where 95% of pure-noise results land — surprising enough to reject the null at the conventional 5% level. The p-value is the same idea as a probability: p ≈ 0.028 means “noise this extreme happens 2.8% of the time.”
Three subtleties separate professionals from the fooled. First, significance is not importance: with 2 million visitors, a 0.05-point lift is “significant” and worthless. Second, peeking corrupts everything: checking daily and stopping at the first significant moment can push the real false-positive rate past 20%. Third, the test assumes random assignment: sample ratio mismatch (say, 55/45 instead of 50/50) signals broken randomization and invalidates the math — always check the split first.
Also mind multiple comparisons: testing 20 variants practically guarantees a “winner” by chance. And remember the test detects difference, not causation quality — a significant win during Black Friday week may not replicate in February. Significance licenses the rollout; judgment still schedules it.
Key terms:
The pooled rate assumes the null (one true rate) when computing the standard error — the textbook approach for comparing two proportions. Each visitor is assumed independent; bot traffic and duplicate counting violate this.
Control headline: 250 conversions from 5,000 visitors. New headline: 300 from 5,000. Real win?
Ship it — but note it misses the 99% bar (2.576), so keep an eye on post-rollout numbers.
Control: 250/5,000 (5.00%). Green button: 262/5,000 (5.24%). The dashboard shows “+4.8% uplift” — celebrate?
The dashboard’s green arrow was lying with statistics — this is the exact trap the calculator exists to prevent.
A SaaS company tests annual-billing-first vs monthly-first pricing pages: 12,000 visitors each, 480 vs 588 conversions (4.00% vs 4.90%). z ≈ 3.36 — significant even at 99%. The +22.5% relative lift on a $40 average plan value is worth ~$4,300/month per 12k visitors. They roll out immediately. Six weeks later the lift holds at +19% — the 99%-level significance had correctly signaled a robust effect, not a borderline one.
Second scenario: an e-commerce site tests free-shipping-threshold messaging: 8,000 visitors per variant, 320 vs 344 conversions (4.00% vs 4.30%), z ≈ 0.95 — not significant. Instead of shipping a “+7.5% win, ” they extend the test two more weeks; the gap evaporates to 4.05% vs 4.10%. Patience saved them from a false rollout and the analytics debt of a phantom win.
Third: a media site tests article layouts across 30,000 visitors per variant and finds a “significant” +1.2% relative lift in clicks. Statistically real, practically trivial — the engineering cost of the new layout exceeds a year of the incremental revenue. Significance said “real”; the calculator’s uplift readout plus a cost estimate said “skip.”
Fourth: a startup peeks at its test daily; on day 4 the result flashes significant and they ship. Two weeks later the metric has regressed to flat. Post-mortem: with 10 peeks, the true false-positive rate was near 20% — they had rolled dice, not run an experiment. They adopt fixed-horizon testing; the next three “wins” replicate cleanly.
E-commerce: Amazon runs thousands of concurrent experiments; its checkout flow is the compounded product of decades of significant wins.
SaaS: onboarding experiments move activation rates — Dropbox’s famous referral test loops were validated variant by variant.
Media: headline testing (Upworthy’s 25-headlines method) is pure A/B; the significant winners reshaped digital editorial strategy.
Search: Google’s “41 shades of blue” test is legend — tiny, significant color differences worth millions at scale.
Politics: the Obama 2008 campaign’s email tests lifted donation conversion significantly — experiments elected, arguably, a president.
Gaming: mobile games A/B test difficulty curves; a significant retention lift in tutorial completion compounds across millions of installs.
One pattern worth internalizing: test big, bold changes first. Tiny tweaks need enormous samples to validate; headline, offer, and layout tests produce detectable, meaningful lifts — optimize the big levers, then polish.
A brief history of the method: A/B testing descends from Ronald Fisher’s randomized experiments in 1920s agriculture — fertilizer variants on crop plots, analyzed with the very significance logic this calculator uses. Direct-mail marketers adopted split testing in the 1960s; the web made it continuous. Google’s 2000 search-results test (famously crashed by slow load times — itself a finding) kicked off the modern era, and today’s experimentation platforms run millions of tests a year on Fisher’s century-old mathematics.
Worked scenario — sizing the test first: a store converts at 3% and wants to detect a 15% relative lift (3.00% vs 3.45%). The sample-size formula says ~30,000 visitors per variant for 80% power at 95% confidence. At 5,000 visitors a day, that is 12 days — so they run the test exactly 12 days, check once, and get z = 2.31: significant. Pre-sizing turned the test from a peeking lottery into a planned measurement; the calendar, not anxiety, decided when to look.
Edge cases and pro tips: with very low conversion rates (under ~1%), the normal approximation weakens — run longer or use exact methods. If your traffic splits unevenly by design (90/10 risk-limiting), the test still works but needs a larger total sample. Always run tests for full weekly cycles — weekend and weekday behavior differ, and a 5-day test is a biased sample of the week. And record every test, win or lose: the institutional memory of what didn’t work is worth as much as the wins.
When significance testing is the wrong tool: for brand-new pages with no baseline, qualitative research beats statistical testing — you need direction, not decimals. When the decision is reversible and cheap (a blog headline), just ship and watch; the test overhead exceeds the stakes. For tiny audiences (a 500-subscriber newsletter), no test will ever reach significance — use judgment and best practices. And significance never replaces product thinking: a significant lift on a metric nobody cares about is still nobody caring.
A/B significance via two-proportion z-test: p1=c1/n1, p2=c2/n2, pooled pp=(c1+c2)/(n1+n2), se=sqrt(pp(1−pp)(1/n1+1/n2)), z=(p2−p1)/se. |z|>1.96 = significant at 95%. Example: 250/5000 vs 300/5000 → 5.00% vs 6.00%, z≈2.19, p≈0.028, significant. Source: Sigma Calculator, A/B Test Significance Calculator, https://sigmacalculator.net/digital-marketing-calculators/ab-test-significance-calculator/
Cite this page as: A/B Test Significance Calculator, Sigma Calculator, https://sigmacalculator.net/digital-marketing-calculators/ab-test-significance-calculator/
Last updated: October 12, 2026