How to read this Bayesian A/B calculator
Use this when you already have visitors and conversions for a control (A) and a variant (B), and you want a practical answer to: how likely is B’s conversion rate higher than A’s, and what do you stand to lose if you pick the wrong one?
It is not a sample-size planner, and it is not a p-value calculator.
What to enter
- Visitors and conversions for A and B. Conversions cannot exceed visitors.
- Credible interval (CI) — how wide the posterior conversion-rate range in the table is (90%, 95% or 99%). This does not change chance-to-beat.
- Winner threshold — the bar for naming a winner (90%, 95% or 99%). Until one variation’s chance-to-beat clears that bar, you will see “No winner yet”. That is a decision rule, not a law of nature.
Share results copies a link with those inputs (not a frozen “winner”), so anyone opening it gets the same recalculation.
How to read the output
Chance-to-beat is the probability that one variation’s true conversion rate is higher than the other’s, given these counts and a uniform prior. The two bars add up to (about) 100%. A 92% chance B is better is strong evidence, not a guarantee — and at a 95% winner threshold it is still “no winner yet”.
CR in the table is the observed rate: conversions ÷ visitors. That is the number you would quote from analytics. It is not the Bayesian posterior mean, which is slightly pulled toward 50% on small samples.
Uplift is the observed relative change of B versus A: (CR B − CR A) ÷ CR A. On 100 conversions from 1,000 visitors vs 120 from 1,000, that is +20%. Treat it as a description of what you saw, not a promise of what you will get if you roll out.
CI is the posterior range for each variation’s conversion rate at the level you chose. Wide intervals mean the chance-to-beat can still move a lot with more data. We only warn when an interval is wider than five percentage points.
Expected loss (in percentage points) is the average conversion-rate cost if you pick that arm and it is actually worse. It is the useful “what if we are wrong?” number. A high chance-to-beat with tiny expected loss on B is a much more comfortable rollout than the same chance-to-beat with a large loss if B is wrong.
A worked example
Default numbers: A 1,000 visitors / 100 conversions (10% CR); B 1,000 / 120 (12% CR).
You should see roughly a 92% chance B is higher, +20% observed uplift, and no winner at a 95% threshold. Expected loss if you choose B is small; expected loss if you stick with A is about two percentage points of conversion rate. That is a “B is probably better, but we have not crossed our own bar yet” read — which is more honest than calling it a win.
Raise B to 150 conversions and the same threshold will typically name B as the winner, with observed CR 50% higher than A.
When to be careful
- Unequal traffic. The SRM warning only compares visitor counts to an assumed 50/50 split. Ignore it if you planned 80/20 (or any other split). Take it seriously if you did plan equal traffic.
- No conversions yet. Chance-to-beat is mostly prior noise until both arms have completions.
- 100% conversion rate on an arm, or very unequal sample sizes, make the posterior awkward. The calculator still runs; treat the charts as indicative.
- Peeking. This does not correct for checking results every day and stopping when you like the number. Bayesian chance-to-beat is easier to live with than p-values in that setting, but it is not a licence to ignore a sequential testing plan.
- Two variations only. Multivariate or bandits need a different tool.
When this is the right tool
Use it to debrief a live or finished A/B test: paste the counts, set a winner threshold your team actually uses, and read chance-to-beat together with expected loss. Use Share results when you want a colleague to see the same inputs.
If you need “how many visitors until 80% power at 20% MDE”, that is a frequentist sample-size question — not this calculator.
If you want a hand interpreting a messy test (unequal split, multiple metrics, or a rollout decision), get in touch.
