Topic 10 — Experimentation and A/B testing

Goal: Learn how PMs prove what actually works instead of guessing.

Lesson 10.1 — Why experiment

The meeting had been going for an hour. Priya wanted Lumi's sign-up form longer and friendlier, with a warm welcome line. The CEO kept saying shorter is always better. Dan chimed in with a customer who'd complained the form felt "cold." Everyone had a take, and everyone said it with total confidence.

Maya noticed something. Nobody actually knew. They were all guessing, just at different volumes.

That's the moment that pushes modern PMs toward experiments — running a real, controlled test and letting actual user behavior pick the winner. Experienced teams are wrong about what users want more often than they'd like to admit, so instead of trusting the room, you measure.

The quiet benefit is what it does to arguments. "Let's test it" replaces "let's argue about it," and the data decides instead of the most senior or loudest person.

A good experiment doesn't help you win the meeting. It makes the meeting unnecessary.

This is the Measure step of Build–Measure–Learn (Topic 3), done with rigor, and it leans on the metrics you met in Topic 9.

Lesson 10.2 — Frame ideas as hypotheses

So Maya wants to test the sign-up form. Where does she start? Not with "let's see what happens." That's a fishing trip, not an experiment.

She starts by writing the idea as a hypothesis — a clear prediction she can be proven wrong about. The shape that works:

If we [make this change], then [this metric] will [improve], because [reason].

Hers comes out as: "If we shorten the sign-up form from 6 fields to 3, then completed sign-ups will increase, because fewer steps means less drop-off."

Look at what that one sentence pins down. The change (6 fields to 3). The metric (completed sign-ups). The direction she expects (up). And the reason (less friction, less drop-off).

The reason matters more than it looks. It's the part that forces you to admit you might be wrong.

And here's the test of whether you're actually ready to run a test: can you name the metric you expect to move? If Maya couldn't say "completed sign-ups," she wouldn't really know what success looks like. No metric, no hypothesis, no experiment yet.

Lesson 10.3 — How an A/B test works

Maya has her hypothesis. Now she needs a fair fight between the old form and the new one.

The standard way to set that up is an A/B test. You build two versions and show them to two randomly split groups of users at the same time.

• Version A, the control — the current experience. Here, the 6-field form.
• Version B, the variant — the same thing with your one change. Here, the 3-field form.

Two details do all the heavy lifting. The split is random, so the two groups are basically the same kind of people. And both versions run at the same time, so a holiday or a marketing email can't quietly skew one group. When the groups match and the timing matches, a real difference in sign-ups is most likely caused by your change and not by luck. You compare the metric you care about and keep whichever version wins.

Sam, the engineer, has watched teams botch this before, and he'd give Maya two warnings.

Change one thing at a time. If Version B has a shorter form and a new button color and friendlier copy, a win tells you nothing about which change earned it.

Wait for enough data. With a handful of users, a "win" is often just noise wearing a costume.

Lesson 10.4 — Significance and common pitfalls

That second warning has a real name. Statistical significance means a result is big enough and consistent enough that it's unlikely to be a fluke.

You don't need the math. You need the instinct, and the instinct is this: small samples lie. A test screaming "B is winning!" after 20 users can flip the other way after 2,000. Early numbers are loud and frequently wrong.

So Maya's test is running, B is up 40% after two days, and her thumb is hovering over the "ship it" button. This is exactly where good PMs trip. Watch for four traps.

Peeking and stopping early. Calling a winner the moment it looks good, before the test has gathered enough data, manufactures false victories. Let the test run its planned course.

Testing too many things at once. Bundle five changes together and you can't credit the win to any of them.

Ignoring practical significance. A result can be statistically real and still pointless. A reliable 0.1% lift might not be worth a week of engineering. Ask whether the size of the win actually matters.

Cherry-picking from many tests. Run twenty tests and a few will look like winners by pure chance, the way someone eventually flips five heads in a row. Treat a surprising "win" with suspicion until it holds up.

Under all four sits one discipline: honesty. It is so tempting to declare every test a win, especially when you championed the idea. Experimentation only pays off if you read the results truthfully.

Lesson 10.5 — What experiments can't do

Maya ships the short form. Sign-ups climb. Win, right?

Mostly. But the test has a blind spot worth naming, and so do A/B tests in general.

It tells you what happened, not always why. The numbers showed sign-ups rose; they can't explain which dropped fields were scaring people off. For the why, you go back to talking to users (Topic 5). What changed and why it changed are two different questions, and tests only answer the first.

A few more limits keep a PM honest:

• A/B tests fit small, measurable tweaks best — wording, flows, button placement. A giant strategic bet, like building the CEO's vague "AI assistant," can't be cleanly split-tested and may take years to pay off.
• They need enough traffic. A product with a trickle of users can't run a meaningful A/B test, and qualitative methods carry more weight there.
• They never replace judgment. Data informs the call; it rarely makes the call. A PM who follows every result without thinking is as lost as one who ignores data completely.

No single test is the real prize. What you're building is a culture of learning: write a clear hypothesis, test it cheaply, read the result honestly, and feed what you learned back into the loop. Humble, curious, evidence-driven. That's the habit that separates strong PMs from merely opinionated ones.

Worked example — Settling a debate with a test

Back to that hour-long meeting. Priya wants the longer, friendlier form; the CEO insists shorter wins. Same volume problem as before, nobody actually knows.

Maya cuts through it with a hypothesis: "If we cut the form from 6 fields to 3, completed sign-ups will rise, because less friction reduces drop-off."

She sets up an A/B test. Half of new visitors get the 6-field form (control), half get the 3-field form (variant), at the same time, assigned at random.

After two days, the 3-field version looks 40% better. Dan wants to call it and move on. Maya doesn't peek. She lets it run until it reaches enough users for the result to mean something.

The number settles down. The real lift is 12% — smaller than the early excitement, and trustworthy. She ships the short form, then runs a few user interviews to understand why the dropped fields mattered, because the test showed what changed but not why.

The debate ended on evidence instead of who talked loudest. That's the whole job, compressed into one form.