Detect sample ratio mismatch in A/B tests by comparing observed vs expected group sizes with a chi-square test.
Last updated
Expected traffic split
Result
The observed traffic split is consistent with the expected ratio. Traffic allocation appears healthy.
Interpretation
No Sample Ratio Mismatch detected (p = 0.1552). The observed ratio 50:49 is consistent with the expected 1:1. Traffic allocation appears healthy.
What is SRM?
A Sample Ratio Mismatch occurs when the actual traffic split in an A/B test deviates from the intended allocation. Common causes include buggy assignment logic, bot traffic, redirects dropping users, or data pipeline issues. A p-value below 0.01 indicates the mismatch is unlikely due to chance.
A/B Testing
A sample ratio mismatch (SRM) calculator checks whether the observed traffic split in an A/B test matches the expected allocation ratio. SRM is a critical data quality check — if the actual ratio of users in each variant differs significantly from the planned split, the experiment results may be biased and unreliable.
The calculator uses a chi-square goodness-of-fit test to compare observed sample sizes against expected proportions.
For a two-variant test with expected ratio r and total sample N: expected counts are E_A = N × r/(1+r) and E_B = N × 1/(1+r). The chi-square statistic is χ² = Σ(O − E)²/E with 1 degree of freedom. A p-value below 0.001 (the standard threshold) indicates a significant mismatch.
χ² = Σ (Oᵢ − Eᵢ)² / Eᵢ
Chi-square goodness-of-fit statistic for sample counts.
A worked example helps translate the sample ratio mismatch calculator maths into a realistic scenario so the user can compare the headline result with a concrete set of inputs.
That matters because a result is easier to trust when the page shows how the same logic behaves in a practical case instead of leaving the formula abstract.
Use the sample ratio mismatch calculator output as a planning aid, then compare it with the assumptions, units, and caveats shown elsewhere on the page before acting on the number alone.
That extra interpretation step matters because a calculator can simplify the arithmetic but still cannot replace real-world context such as local rules, contract terms, or individual circumstances.
Frequently asked questions
The standard threshold is p < 0.001. This is more conservative than the typical 0.05 used in hypothesis testing because SRM is a data quality check — you want to be very sure the mismatch exists before invalidating experiment results.
Generally no. SRM indicates a systematic bias in how users were assigned to variants, which means the treatment and control groups are not comparable. You should investigate and fix the root cause before drawing conclusions from the experiment.
The safest manual check is to follow the same formula or rule one step at a time and compare that working with the calculator output. That catches sign errors, bracket mistakes, and input-order mixups without requiring any extra method beyond the underlying maths itself.
Also in Statistics
Related
These related calculators come from the same leaf category, nearby sibling categories, or the same top-level topic.
Run a chi-square calculator for goodness-of-fit or independence tests with p-value, expected counts, selected-alpha decision, contribution rows.
Calculate p-values from z-scores, t statistics, chi-square statistics, and F statistics with one-tailed or two-tailed options, alpha-level decisions.
Calculate descriptive statistics from one dataset.
Convert a chi-square statistic and degrees of freedom into the right-tail p-value.
