Experiment design
Builds on Topic 10.
What you'll produce
An A/B test design to prove whether your PRD feature actually works. You'll write a testable hypothesis, define the control and variant, the metric and guardrails, and how you'll judge the result honestly. This proves the Topic 10 skills: framing a hypothesis, designing a clean A/B test, and respecting significance and limits.
Instructions
- Write a hypothesis in the form "If we [change], then [metric] will [improve], because [reason]." Use the feature metric from your metrics framework (Deliverable 4).
- Define control vs. variant — what each group sees. Change one thing so the result is attributable.
- Name the primary metric you'll measure and the guardrail/counter-metric you'll watch (so a "win" doesn't hide harm).
- Describe the setup: who's eligible, how they're split (random), and roughly how long / how much traffic before you'd trust the result (significance — "small samples lie").
- State your decision rule: what result means ship, what means don't, and that you won't "peek" and stop early.
- Note one limitation: something the A/B test won't tell you (e.g., why), and how you'd follow up (e.g., user interviews).
Worked example
(Feature: personalized "why you started" reminder)
- Hypothesis: If we send lapsing users a reminder showing their personal reason, then week-2 retention will increase, because reconnecting them to their motivation reduces motivation-driven drop-off.
- Control: No reminder (current experience). Variant: The personalized reminder. Only that one change differs.
- Primary metric: Week-2 retention. Guardrail: Notification opt-out rate (must not spike).
- Setup: New users randomly split 50/50 on signup; run until enough users for a trustworthy result, not just a few days.
- Decision rule: Ship if retention shows a meaningful, sustained lift with no guardrail harm; don't ship if flat/negative or if opt-outs spike. No peeking/early stop.
- Limitation: The test shows whether retention moved, not why; follow up with interviews of users who returned.
Rubric
Levels: Needs work (1) / Solid (2) / Excellent (3). Passing = every criterion at Solid or above.
- Testable hypothesis — 1: vague/no metric · 2: proper if/then/because with a metric · 3: sharp, with a clear mechanism.
- Clean control/variant (one change) — 1: many changes/confounded · 2: clear A vs. B · 3: cleanly isolates one change.
- Primary metric + guardrail — 1: missing · 2: has both · 3: well-chosen metric and a genuine guardrail.
- Sound setup (random split, enough data) — 1: flawed/none · 2: random split, mentions sample size · 3: clearly addresses randomization and significance.
- Honest decision rule (no peeking) — 1: missing or peeking · 2: a decision rule · 3: clear ship/no-ship rule and explicitly avoids early stopping.
- States a limitation — 1: missing · 2: notes one · 3: notes a real limit and a sensible follow-up.