Interview Drills — Root-Cause Analysis & Metrics

"A metric moved — why, and what do you do?" questions test calm, structured diagnosis. Each drill has a Framework, a Model answer, and a Rubric. Don't jump to a fix; segment, hypothesize, and check before acting.

The universal RCA structure: Clarify the metric and the change (size, when, how measured) → check for data/tracking issues first → segment (by user type, platform, geography, new vs. existing, time) → form hypotheses (internal changes? external events? seasonality?) → say how you'd verify each → then propose action.

D1

difficulty: medium
concept: significance-and-pitfalls Daily active users dropped 20% this week. What do you do?
Framework: Clarify (which users, exactly when, real or tracking glitch?) → rule out a logging/data bug → segment (platform, geography, new vs. existing) → hypotheses (a release? an outage? a competitor? seasonality/holiday?) → verify each → then act.
Model answer: "First I'd confirm it's real, not a tracking break. Then segment: is it one platform (iOS update?), one region (outage?), or new vs. existing users (acquisition vs. retention)? I'd line it up against our releases and external events. Say it's only Android after our last release — that points to a bug to roll back. I diagnose before I 'fix.'"
Rubric: Strong answers check data integrity first, segment systematically, generate internal and external hypotheses, and verify before acting. Weak answers immediately propose a fix with no diagnosis.

D2

difficulty: hard
concept: vanity-metrics Daily active users are up, but revenue is flat. What's going on?
Framework: Clarify definitions → consider that the new users may be low-intent/non-paying → segment activity vs. paying behavior → check the conversion step and pricing → hypotheses (a vanity-metric trap? wrong audience? a paywall change?).
Model answer: "Rising DAU with flat revenue suggests the new activity isn't converting. I'd segment: are the extra actives free users, a different geography, or driven by a feature that doesn't lead to purchase? I'd check the free→paid conversion step. It may be a vanity-metric situation — engagement that doesn't reflect value to paying users."
Rubric: Strong answers recognize the possible vanity-metric/disconnect, segment paying vs. non-paying, and inspect the conversion path. Weak answers celebrate the DAU rise or can't explain the disconnect.

D3

difficulty: medium
concept: north-star How would you measure the success of a new feature?
Framework: Tie to the feature's goal → pick a primary metric that reflects real value → add a guardrail/counter-metric → define the timeframe and the success bar up front → mention how you'd isolate the feature's effect (e.g., A/B).
Model answer: "Start from what the feature is for. Pick a primary metric tied to that outcome (e.g., the action it should drive or a retention lift), set a guardrail so we don't harm something else, define success up front, and isolate the impact with an A/B test rather than crediting it for unrelated changes."
Rubric: Strong answers connect the metric to the feature's purpose, include a guardrail, predefine success, and isolate causality. Weak answers pick a vanity metric or have no success definition.

D4

difficulty: hard
concept: aarrr-funnel Sign-ups doubled after a campaign, but week-2 retention fell. What's happening, and what do you do?
Framework: Recognize acquisition vs. retention split → hypothesize the campaign drew lower-intent/wrong users → segment campaign cohort vs. baseline → check activation (did they reach value?) → decide: fix targeting and/or activation, not just chase more sign-ups.
Model answer: "More sign-ups but worse retention usually means the campaign attracted the wrong users who never hit the 'aha' moment. I'd compare the campaign cohort's activation and retention to baseline. If they're worse, the fix is better targeting and a stronger activation/onboarding — not pouring more people into a leaky funnel."
Rubric: Strong answers separate acquisition from retention/activation, suspect audience quality, segment the cohort, and avoid 'just acquire more.' Weak answers treat doubled sign-ups as success or propose more acquisition.

D5

difficulty: medium
concept: north-star Pick a North Star metric for a food-delivery app and justify it.
Framework: State the real value delivered → choose one metric capturing it (not a vanity metric) → explain why it aligns users and business → name a counter-metric → note what it could miss.
Model answer: "North Star: completed orders per active user per month — it captures real value (people getting meals) and aligns business (revenue) with users (utility). Counter-metric: late/cancelled orders, so we don't grow volume while service quality drops. It could miss restaurant-side health, which I'd track separately."
Rubric: Strong answers pick a value-aligned North Star, justify the alignment, add a counter-metric, and note a blind spot. Weak answers pick a vanity metric (downloads) or can't justify the choice.

D6

difficulty: hard
concept: significance-and-pitfalls Conversion jumped right after a release, and the team wants to declare victory. What's your response?
Framework: Be appropriately skeptical → check it's not noise/short window (significance) → confirm the release is the cause vs. a coincident event → look for a confound or tracking change → recommend confirming via a controlled comparison before crediting it.
Model answer: "I'd be glad but cautious. Is the window long enough to be significant, or could it be a short-term blip? Did anything else change at the same time (a promo, seasonality)? I'd verify the release is actually the cause — ideally via an A/B or a clean before/after with guardrails — before we declare victory."
Rubric: Strong answers resist premature conclusions, raise significance and confounders, and seek causal confirmation. Weak answers accept the jump at face value or peek/declare victory early.