AI PM training

A/B testing with AI: design the experiment before reading the result

Use AI to draft hypotheses, audit instrumentation and explain results while keeping experiment decisions statistically defensible.

Workflow: Hypothesis, Assignment and metrics, Quality checks, Decision rule
AI Product Manager • Original instructional diagram

Use AI as an experiment assistant

AI can generate hypotheses, draft an analysis plan, inspect queries and help explain results. It cannot turn weak traffic into sufficient evidence or justify repeatedly checking a conventional fixed-horizon test until something looks significant. Keep the statistical design and product decision explicit.

Step 1: write a causal hypothesis

For our fictional appointment product: making available times easier to compare may increase completed first bookings. Define who is eligible, what changes and the expected mechanism. Avoid changing copy, pricing and flow simultaneously unless you deliberately want to test that combined intervention.

Step 2: define metrics and assignment

Choose one primary metric and define its numerator, denominator and measurement window. For example, eligible users completing a first booking within seven days divided by all eligible assigned users. Predefine assignment at user or account level and keep it stable. Watch for cross-device exposure and spillover between users.

Add guardrails such as cancellation rate, errors or support burden. Give each a measurement window and acceptable boundary. Distinguish guardrail metrics from secondary diagnostics.

Step 3: plan sample size and stopping

Use a suitable calculator or reviewed statistical code with your baseline, smallest practically meaningful effect, significance level and power. State whether the method is fixed-horizon or a valid sequential design. Fixed-horizon inference needs the planned stopping discipline; sequential analysis needs its own valid method. Do not mix them after seeing results.

Draft an experiment plan using only the supplied baseline and constraints. Define the causal hypothesis, eligibility, assignment unit, primary metric formula, guardrails, minimum worthwhile effect, required sample-size inputs and stopping rule. Mark missing values. Flag contamination, multiple comparisons and delayed outcomes. Do not invent traffic, calculate a sample size from missing inputs, or declare a winner.

Step 4: check data before effects

Verify exposure events, assignment persistence, deduplication and outcome windows. Investigate sample ratio mismatch: an unexpected allocation imbalance can indicate a broken experiment. Review bot traffic, exclusions and missing events using rules established before looking at outcomes. A large apparent lift does not rescue invalid measurement.

Step 5: explain results with uncertainty

Report sample sizes, baseline, absolute difference, relative lift, interval estimate, method and guardrail results. A hypothetical move from 10% to 11% is a one-percentage-point increase and 10% relative lift; this arithmetic says nothing about significance without sample sizes and a valid design.

Do not interpret “not statistically significant” as proof that variants are equivalent. Label underpowered results inconclusive. Investigate segments only with appropriate caution about multiple comparisons and exploratory analysis. Ask an analyst to review calculations and the final recommendation.

Low traffic: choose a better learning method

If the required sample cannot be reached in a useful period, consider moderated research, a concierge pilot or a larger meaningful intervention. These can answer different questions; they do not provide the same causal estimate as a randomized experiment.

Practice assignment

In 60 minutes, fill the linked experiment plan before generating variants. Include launch checks, stop conditions and a decision table for ship, revise or inconclusive. Read the site's product metrics guide for additional measurement and stopping-rule guidance.

Return to AI for Product Managers training