Creatives & ad policy

A/B test calculator

Enter visitors and conversions for each variant — get the uplift, the p-value from a two-proportion z-test and a clear significance verdict.

Test data

Variant A (control)

Visitors

Conversions

Variant B (test)

Visitors

Conversions

Confidence level

Hypothesis

Result

—

CR — A

—

CR — B

—

Relative uplift

—

p-value

—

Confidence

—

1 − p

95% CI on diff

—

absolute, pp

Two-proportion z-test with a pooled standard error. It assumes visitors are independent and randomly split, one conversion event per visitor, and that you fixed the sample size in advance — peeking at the result and stopping when it turns green inflates false positives.

Bayesian view

P(B beats A)

—

uniform prior

P(uplift > 0)

—

Expected loss picking B

—

CR points, if B is worse

The Bayesian panel uses Beta(1,1) posteriors on each conversion rate and a normal approximation of their difference, so it does not need a fixed sample size and can be read as the test runs. "P(B beats A)" of 95%+ is a common ship bar; "expected loss" is how much conversion rate you would give up on average if you pick B and B is actually the worse variant.

Minimum sample size

Baseline CR, %

Min. detectable effect, % rel.

Power

—

Per variant, for a two-sided test at the confidence level chosen above. A relative MDE of 10% on a 4% baseline means detecting a move to 4.4%.

What a significance test actually answers

When variant B converts better than A in a test, the question is whether that gap is a real effect or just noise. This calculator runs a two-proportion z-test: it pools the two conversion rates, computes the standard error, and turns the gap into a z-score and a p-value. The p-value is the probability of seeing a difference at least this large if the two variants truly convert identically. If it is at or below your alpha — 5% at 95% confidence — the result is unlikely to be chance and the tool calls it significant.

Fix the sample size before you start

This is a fixed-horizon test. Decide how much traffic you will collect before running it — the minimum-sample-size box works that out from your baseline rate, the smallest effect worth detecting, the confidence level and the power. Checking the result repeatedly and stopping the moment it turns green massively inflates false positives; a test that looks significant at 200 conversions often is not at 2,000. The tool also assumes visitors are split randomly and independently and that each one has a single binary outcome — for revenue per visitor or a multi-step funnel you need a different test.

Frequently asked questions

What test does this run?

A two-proportion z-test with a pooled standard error — the standard check for comparing two conversion rates. It gives a z-score, a p-value and a verdict at the confidence level you pick.

What does the p-value mean here?

It is the probability of seeing a difference at least this large if the two variants actually convert identically. A p-value at or below your alpha (5% at 95% confidence) means the result is unlikely to be pure chance.

Two-sided or one-sided?

Use two-sided by default — it tests whether B is different from A in either direction and is the honest choice when you would ship B only on a win but want to know about a loss too. One-sided only tests "B beats A" and reaches significance sooner, so use it only if a worse B has no consequences.

Why is my result not significant even though B looks better?

Either the sample is still too small for the size of the difference, or the difference is genuinely within noise. The minimum-sample-size box tells you how much traffic a difference of a given size actually needs.

Can I stop the test as soon as it turns green?

No. This is a fixed-horizon test: decide the sample size before you start and only read the result once you reach it. Checking repeatedly and stopping on the first significant reading massively inflates the false-positive rate.

How is the minimum sample size calculated?

From the baseline conversion rate, the relative minimum detectable effect, the chosen confidence level and statistical power (default 80%), using the standard normal-approximation formula for comparing two proportions. It is per variant.

Does it work for revenue or multi-step funnels?

It is built for a binary outcome — converted or not. For revenue per visitor or average order value you need a t-test or a bootstrap, not this. For a funnel, test one clearly defined step at a time.

Is my data sent anywhere?

No. Every number is computed in your browser; nothing is transmitted or stored.

All tools →