What the test does
It asks how likely a gap this size would be if the two versions really performed the same.
The two rates are pooled into a single standard error, and the gap is measured in units of that error to give the z score.
Worked example
Group A gets 120 conversions from 1,000 visitors, group B gets 150 from 1,000. The rates are 12% and 15%, a relative difference of 25%.
The z score is 1.96305 and the p value is 0.04964, which is significant at the 5% level but not at 1%.
What significant does not mean
Significant does not mean important. With a large enough sample, a tiny and useless difference will come out significant.
That is why the confidence interval is shown next to the p value: it says how big the gap might really be.
Choosing the level
5% is the usual threshold, and 1% is stricter. The lower the threshold, the stronger the evidence needed.
Deciding the threshold after seeing the result is the mistake to avoid, because it turns a test into a search for a number you like.
Inputs for the statistical significance calculator
Visitors and conversions
The number of people in each group and how many did what you wanted. Conversions cannot exceed visitors.
Groups A and B
The two versions being compared. The order does not change the p value, only the sign of the gap.
CALCULATOR QUESTIONS
Statistical significance calculator questions
Is 120 out of 1,000 against 150 out of 1,000 significant?
Just: the p value is 0.04964, below the 5% threshold but above 1%.
What does a p value of 0.05 mean?
If the two versions truly performed the same, a gap this large would happen about 5% of the time.
Why show a confidence interval as well?
It shows the size of the gap in percentage points, which a p value alone cannot tell you.
Can I test more than two groups?
Not on this page. Three or more groups need a chi-square test.