Avoid declaring an A/B test winner after a handful of conversions

Variant B gets two early purchases while A gets none, and the team wants to stop the test. Small samples can create dramatic-looking percentages that do not represent a stable difference.

Why this happens

Repeatedly checking results and stopping when they look favourable can bias decisions. Unequal allocation, duplicate exposure and changing the page during the test further undermine interpretation.

Step-by-step solution

1. Choose the primary metric, allocation unit and evaluation plan before launch. Define what effect would be useful rather than only looking for a positive percentage.

2. Check assignment and exposure counts. Investigate a large allocation imbalance before interpreting conversion differences.

3. Use a statistical method appropriate to the planned design and report uncertainty. If the sample is insufficient, continue according to the plan or report the result as inconclusive.

Worked example

Four conversions from 40 visitors and three from 40 visitors differ by one event. That observation alone is not a dependable reason to redesign an entire acquisition strategy.

Verify the fix

Check that users remain in their assigned variant and that both variants record the same conversion definition. Review uncertainty alongside the estimated difference.

Prevent a repeat

Keep a test log that includes stopped and inconclusive experiments. Avoid repeatedly testing many metrics and publishing only the one that appears favourable.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.