Topic 6 — Defining the question and the metrics

Goal: Learn that good analysis starts with a sharp question, not the data.

Lesson 6.1 — The request that sends you down the wrong road

Marcus, Perch's Head of Marketing, drops by Nadia's desk on a Monday: "Hey, can you send me the sales numbers?" Nadia's first instinct, fresh from her retail-ops days, is to dive straight in. She opens her SQL editor, pulls total revenue for the year, builds a tidy chart, and sends it over before lunch. Marcus replies: "Oh — I meant sign-ups from last week's ad campaign. Did it work?"

An hour gone. The data was perfect. The answer was useless.

This is the single biggest mistake new analysts make: diving into the data before understanding the question. The query ran fine, the chart looked sharp, and none of it mattered, because Nadia answered a question nobody asked.

Priya, her mentor, has a rule she repeats until it sticks. Before you touch a table, answer three things:

• What decision is this for? Marcus is deciding where to spend next month's ad budget.
• What does success look like? More sign-ups than the campaign cost to run.
• Which exact numbers answer that? Sign-ups in the campaign week, ideally split by the channel that drove them.

Slow down at the question and the analysis is half done. Rush past it and even flawless SQL produces the wrong answer.

The skill here is putting the question before the data. It feels backwards when you're eager to prove you can pull numbers, but the analysts people trust are the ones who ask first.

Lesson 6.2 — Turning a vague wish into an answerable question

"How's the business doing?" is not a question you can run.

Dana, Perch's VP of Operations, asks Nadia exactly this in a hallway. It's a real concern, but it's a wish, not a query. There's no number that answers it, because it doesn't say what to measure, over what stretch of time, or for whom.

An answerable question names three things:

1. A metric — the specific number you'll report (revenue, sign-ups, repeat-purchase rate).
2. A time frame — over what period, compared to what (last six months, week over week, this quarter vs. last).
3. A breakdown — usually a way to split it (by product category, by customer type, by channel).

Watch Nadia translate Dana's wish into something she can actually pull:

Did revenue grow month over month over the last six months, and which product category — sofas, desks, or shelving — drove the change?

That version has a metric (revenue), a time frame (month over month, last six months), and a breakdown (product category). Now it's a job, not a mood. Nadia learned to do this conversion out loud, in front of the stakeholder, so Dana can nod and say "yes, that's what I meant" — or correct her before the work starts instead of after.

The vague-to-sharp move is the most reusable trick in this whole topic. Almost every request you'll ever get arrives vague. Sharpening it is your job, not the stakeholder's.

Lesson 6.3 — A metric is only as good as its definition

Nadia builds a "revenue" report. Tom, the data engineer, glances at it and asks a question that stops her cold: "Is that before or after refunds? And are you counting tax?"

She hadn't decided. And that's the whole problem.

A metric is a specific number that measures something — but the same word means different things to different people, so the analyst's job is to write down the exact definition before pulling anything. Three words that quietly cause chaos:

• "Active users" — active how recently? In the last day, week, or month? And doing what — logged in, or actually made a purchase?
• "Revenue" — before or after refunds? Including tax and shipping, or just the product price?
• "Conversion rate" — converted out of whom (everyone who visited? everyone who added to cart?), over what period?

This isn't pedantry. It prevents the classic disaster every data team eventually lives through: two people stand up in the same meeting and present two different "revenue" numbers from the same database, and neither is wrong — they just defined the word differently. One counted refunds, the other didn't.

Agree the definition out loud before you pull the data. The cheapest argument to win is the one you have before the numbers are on the screen.

When Nadia now hears a metric name, she writes its definition in one plain sentence at the top of her query — "Revenue = product price after refunds, excluding tax and shipping" — and shows it to whoever asked. Tom stopped finding holes in her reports the week she started doing that.

Lesson 6.4 — Rates beat counts when sizes differ (and the metrics that fool you)

Marcus is comparing two ad campaigns. Campaign A brought in 1,200 orders; Campaign B brought in 800. "A won," he says.

Nadia checks how many people each campaign reached. A was shown to 120,000 people; B to 40,000. So A converted 1% of its audience, and B converted 2%. Per person reached, B was twice as effective. The raw counts pointed the wrong way.

That's the case for preferring rates and ratios over absolute counts when you're comparing groups of different sizes. A rate (conversion rate 2%) divides by the size of the group, so it puts unequal things on a fair footing. A raw count (800 orders) doesn't — a bigger group will almost always have a bigger count, which tells you about its size, not its quality.

Counts still matter. If Dana needs the total impact on the warehouse, "1,200 orders" is the number she wants, not a percentage. The rule of thumb: rates for comparing, counts for totals and impact.

There's a sharper trap hiding nearby. A vanity metric is a number that looks impressive but doesn't change any decision. Total page views, total all-time sign-ups, social-media followers — they reliably go up, they feel like progress, and they tell you almost nothing about what to do next. An actionable metric is one where you can say "we changed X and Y moved," so it actually guides a choice.

Nadia caught herself once putting "total visitors this year: 4.1 million" at the top of a deck. Priya asked the only question that matters: "What does Marcus do differently if that number is 4.1 million versus 3.9 million?" Nothing. She swapped it for repeat-purchase rate by channel — a number Marcus could act on — and the deck got useful.

Lesson 6.5 — The overall number is hiding something

"Sales are flat," Nadia reports. Total revenue this quarter looks almost identical to last quarter. Case closed.

Priya isn't satisfied. "Flat for everyone?"

So Nadia splits the number by customer type. The flat line was two strong trends cancelling each other out: revenue from new customers was surging, while revenue from returning customers was collapsing — exactly the group Perch is betting its whole strategy on. "Flat" was the most dangerous possible summary, because it hid a fire.

This is segmentation: breaking a metric down by groups — time, region, customer type, product, acquisition channel — and it's where the real insight lives. The overall number is an average, and averages blur.

The extreme version of this trap has a name: Simpson's paradox, where a trend in the whole dataset reverses once you split it into groups. The famous case is a university that looked like it admitted men at a higher rate than women overall — yet within almost every individual department, women were admitted at an equal or higher rate. The aggregate said one thing; every segment said the opposite. (It happened because women applied more often to the most competitive departments.)

You don't need the math. You need the habit — one question Nadia now asks before she trusts any single number:

Is this true for everyone, or is an average hiding something?

Look at the overall figure and a few key segments, every time. That one reflex is most of what separates a real analysis from a confident, well-formatted, wrong one.

Worked example — Marcus asks "how are sign-ups doing?"

Marcus stops by: "How are sign-ups doing?" The old Nadia would already be querying. The new one runs the whole topic in about ninety seconds.

Question before data. She asks what it's for. Marcus is deciding whether to keep funding the Instagram campaign that's been running for six weeks. Success means sign-ups that turn into actual furniture buyers, cheaply enough to justify the spend.

Make it answerable. Together they sharpen "how are sign-ups doing?" into: Over the last six weeks, what was the weekly sign-up count by acquisition channel, and what share of each channel's sign-ups made a first purchase within 30 days? Now it has a metric, a time frame, and a breakdown.

Define the metric. A "sign-up" is a confirmed email account, not a half-finished form. "First purchase within 30 days" excludes refunded orders. She writes both definitions at the top of her query so Marcus can't be surprised later. Tom confirms the warehouse has a clean channel column, so the breakdown is possible.

Compare with rates. Instagram drove the most sign-ups in raw count, but only 3% of them bought within 30 days, versus 11% for organic search. By the count, Instagram won; by the rate that actually predicts revenue, it was the weakest channel.

Segment before trusting it. Before sending it, she splits by week and spots that Instagram's purchase rate was climbing sharply in the last two weeks after a creative change. The six-week average (3%) hid a real improvement. Her recommendation to Marcus: don't cut Instagram yet — the recent trend is the story, not the average.

One vague question. A sharp answer, a defined metric, the right ratio, and a segment that flipped the conclusion. That's the job.