Choose a meaningful effect and a feasible experiment before launch. Few outcomes can leave both versions plausible; no winner is a useful result, not permission to invent certainty.
A small difference in a small test is not automatically a reliable winner. Low-traffic websites often need to choose between running a long experiment, testing a larger change or using qualitative evidence for a bounded design decision. The correct answer can be that the experiment did not distinguish the versions.
This guide uses fictional counts to explain decisions. It does not provide a significance result for a real website, prescribe one statistical method for every situation or promise that a particular number of visitors is always enough.
State the business question and one primary outcome
Write a testable question rather than “make the page better.” A fictional repair service asks whether clearer scheduling information helps eligible visitors complete an accepted appointment request. A button-click increase alone would not answer that question if the requests are invalid or never accepted.
Choose one primary outcome and define its denominator. Distinguish assigned visitors, sessions, form starts and accepted requests. Define important guardrails, such as error rates or unsuitable enquiries, without replacing the primary outcome whenever another chart looks more favorable.
The GOV.UK introduction to A/B studies describes a control and variation, random allocation, sample planning and analysis. Its examples provide a method framework, not a traffic threshold guaranteeing that a small commercial website will obtain a decisive answer.
Check whether the proposed test is feasible
Estimate the relevant eligible traffic and outcome frequency using a comparable period. A website with many article views may have few visits to the particular enquiry page. Use the population exposed to the experiment, not the whole site’s impressive-looking total.
Decide what change would matter commercially and whether the site can collect enough appropriate observations within a stable period. Tiny effects generally require more information to distinguish. A test lasting through a major seasonal shift or complete offer change can become hard to interpret.
NIST’s sample-size discussion for proportions makes planning assumptions explicit. Use a method suited to the experiment’s design and outcome. A calculator’s result remains dependent on the inputs and assumptions; it is not a universal minimum for every page.
Read the fictional result without exaggeration
| Version | Assigned eligible visitors | Accepted requests | Observed rate |
|---|---|---|---|
| A: current page | 100 | 4 | 4% |
| B: clearer schedule | 100 | 6 | 6% |
B has two more observed requests, two percentage points higher and a fifty-percent relative increase in this invented sample. Those descriptions are arithmetic, not a demonstration that B’s underlying performance is reliably better or that the same uplift will continue.
With few outcomes, plausible values can remain broad. NIST’s confidence-interval guidance explains why interval methods and small-sample behavior matter for proportions. Do not apply an unsuitable approximation merely because a spreadsheet makes it easy.
A complete analysis must match the actual assignment unit, independence assumptions, outcome definition and planned method. Separate descriptive counts from inferential conclusions. This guide deliberately does not invent a p-value or declare a statistical winner from the table.
Keep assignment and measurement trustworthy
Use a supported random assignment mechanism and a stable version identity. Alternating versions by day can confound the result with weekday effects. Sending one traffic source to A and another to B can compare audiences rather than page designs.
Check that people see a coherent version across the relevant journey and that recording works for both variants. A form event implemented only on B can manufacture an apparent improvement. Inspect accepted outcomes and duplicate handling before interpreting the experiment.
Keep consent and collection limitations visible. The measured sample may not represent everyone who saw the page. Do not circumvent visitors’ choices to make the experiment look more complete. Document known missingness and decide whether it limits the question beyond usefulness.
Set stopping rules before viewing the outcome
Define the intended sample, observation period, analysis approach and operational stop conditions. A serious error or misleading page can require stopping early for reader protection. That is different from stopping because a favorable percentage appeared on Tuesday.
Repeatedly checking until a threshold is crossed can invalidate a fixed-horizon interpretation. If using a sequential approach, use a method designed for it and understand its assumptions. Do not attach a sophisticated label to ordinary daily peeking after the fact.
Record concurrent changes: traffic campaigns, pricing, outages or revised eligibility. A random split can help comparison, but severe implementation problems or a changing outcome definition can still compromise the result. Do not hide those events in the final report.
Choose alternatives when a decisive test is unrealistic
Use moderated task sessions or direct usability observation to understand whether readers can find important information. These methods can reveal confusion and broken journeys without claiming a measured conversion uplift. Keep findings attached to observed tasks and participants rather than extrapolating a precise site-wide percentage.
Fix clear defects directly when the correct behavior is already established. A broken mobile button or unreadable label does not need an A/B test that deliberately leaves some visitors with the defect. Verify the repair and describe it as a repair.
For an uncertain design preference, make a reversible decision using the best available evidence, document why and monitor it. That can be reasonable even when a statistical winner is unavailable. The important distinction is between an informed product choice and a proven causal improvement.
Report no winner in a useful way
Include the question, versions, assignment method, dates, eligible counts, primary outcomes, analysis limitations and operational incidents. State whether the experiment distinguished the alternatives under the chosen method. Avoid “the versions are identical” when the result only failed to establish a difference.
Specify the next decision: retain the control, use a separately justified clearer design, gather more appropriate information or test a materially different hypothesis. Keep guardrails and practical costs visible. No winner can prevent overconfidence and wasted redesign cycles.
The acceptance criterion is a decision another person can understand without trusting a headline uplift. If a test lacks enough information, say so. A small site benefits more from honest uncertainty and clear improvements than from a sequence of fragile winners that cannot be reproduced.
Continue with the next useful check
Sources and verification
Editorial review 2026-10-18. Source check dates are recorded separately. Notes describe what each source supports and its limits.
View sources
- GOV.UK: A/B testing comparative studiesChecked Control, variation, randomization and study planning.
Limits: Not a specific commercial-site power calculation. - NIST: Sample sizes requiredChecked Sample planning depends on hypotheses and assumptions.
Limits: Method must match actual experiment design. - NIST: Confidence intervals for proportionsChecked Interval methods and small-sample limitations.
Limits: Does not establish a winner for fictional counts.