Separate known repairs from uncertain choices
Repair a known broken action, fix invalid tracking and reserve an experiment for a meaningful unresolved choice with feasible measurement. Define the primary outcome and guardrails before allocation. The comparison-layout example needs a real statistical plan; a few early percentages cannot establish a reliable business decision.
Worked example — illustrative data.
| Change | Treatment to consider | Reason |
|---|---|---|
| Buy button fails | Repair and verify | Known blocking defect |
| Two plausible comparison layouts | Consider a controlled experiment | Uncertain buyer decision |
| Very few qualified demos | Use task evidence and cautious rollout first | Outcome sample may not resolve the question |
| Tracking sends duplicate orders | Repair measurement before testing | Invalid outcome data |
Describe the uncertain mechanism: for example, showing usable dimensions beside a lamp image may help buyers select a compatible product. Define one primary outcome and relevant guardrails, including errors, returns or qualification where they matter. Plan sample, allocation, duration and stopping with a competent analyst using the actual traffic and effect that matters. Avoid peeking at unstable early percentages and declaring a winner. Preserve the original plan and exceptions.
Write the decision rule before opening an experiment
State the uncertain choice, eligible audience, primary outcome and meaningful downside before allocating visitors. For the comparison layouts, keep qualified outcomes distinct from raw submissions. Use the site’s own stable baseline and assess whether the available sample can answer the question. Repair a broken button or duplicate purchase event before treating the release as an experiment.
More submissions can produce fewer qualified requests
A variant that raises form submissions can reduce qualified requests. A narrow submit-rate winner may therefore be the wrong business decision. Include downstream quality and allow for its reporting delay.
Test the comparison layout only if uncertainty and feasible measurement justify it. Do not invent a universal minimum traffic threshold or claim that every visual change needs an experiment.
Should an experiment stop as soon as one version leads?
Follow the chosen statistical method and a documented stopping rule rather than reacting to a temporary lead. Define guardrails and analysis conditions before launch. If the design cannot resolve the question, use task evidence or a cautious reviewed rollout instead.
Continue with Guides for stores, software teams and local businesses, Conversion experiment design and analysis and Choose a conversion rate baseline you can use.