How do you calculate statistical significance for an A/B test?

Use a two-proportion z-test: compute each variant's conversion rate, pool them to get the standard error, form a z-score from the difference, and convert it to a p-value. Below 0.05 is significant at 95% confidence.

What does a p-value of 0.05 mean?

It means there is about a 5% chance of seeing a difference this large if the two variants were actually identical. Below 0.05 is the conventional threshold for calling a result statistically significant.

How big does an A/B test sample need to be?

It depends on your baseline rate and the lift you want to detect, but small samples (a few hundred per variant) rarely reach significance for modest lifts. Underpowered tests are the most common reason a real effect looks like noise.

Omega Point Systems

Calculator / 05

A/B test significance, solved.

Two-proportion z-test for conversion rate comparison. p < 0.05 = 95% confidence.

Start from a preset

Inputs

Visitors — variant A

Conversions — variant A

Visitors — variant B

Conversions — variant B

∎Statistically significant lift at 95% confidence. Ship B.

Outputs

Rate A

10.00%

variant A conversion

Rate B

12.00%

variant B conversion

Lift

20.00%

relative change vs A

p-value

0.0432

significant

Formula · z-statistic

z = (p̂ᵦ − p̂ₐ) ÷ SE

SE = √(p(1−p)(1/nₐ + 1/nᵦ))

Formula · p-value

p = 2 · (1 − Φ(|z|))

two-tailed normal CDF

Take it with you

Open this calculator as an editable Google Sheet — your inputs and live formulas, ready to fork.

This free calculator runs a two-proportion z-test on an A/B test — conversion rates, lift, z-score, and p-value — to tell you whether a result is statistically significant at 95% confidence.

How to calculate A/B test significance

Compute each variant's conversion rate (conversions ÷ visitors). Pool the two rates, derive the standard error, and form a z-score: (rate B − rate A) ÷ standard error. The p-value is two times one minus the normal CDF of the absolute z. A p-value below 0.05 means the difference is significant at 95% confidence.

How to read the result

Significant and positive: ship B. Significant and negative: keep A. Not significant: the test has not separated signal from noise yet — keep running or accept the null. Significance is not the same as a meaningful lift; an underpowered test can miss a real effect, and a huge sample can flag a trivial one.

Worked example

2,000 visitors each, 200 vs. 240 conversions: rates of 10% and 12%, a 20% relative lift, z ≈ 2.0, p ≈ 0.045. Just under 0.05 — significant at 95%, but barely, so the prudent read is to confirm rather than celebrate.

Frequently asked questions

How do you calculate statistical significance for an A/B test?: Use a two-proportion z-test: compute each variant's conversion rate, pool them to get the standard error, form a z-score from the difference, and convert it to a p-value. Below 0.05 is significant at 95% confidence.
What does a p-value of 0.05 mean?: It means there is about a 5% chance of seeing a difference this large if the two variants were actually identical. Below 0.05 is the conventional threshold for calling a result statistically significant.
How big does an A/B test sample need to be?: It depends on your baseline rate and the lift you want to detect, but small samples (a few hundred per variant) rarely reach significance for modest lifts. Underpowered tests are the most common reason a real effect looks like noise.