Interview Drills — Case Study (Take-Home / Live Analysis)

Open-ended interview questions for the deciding round at most companies: you're handed a dataset and a vague ask, and you must clarify the question, define the metric, plan the cleaning and segmentation, and reason to an honest conclusion. Each drill has a Framework (the structure a strong answer follows), a Model answer (a concise example), and a Rubric (what an interviewer listens for). Practice thinking aloud: state your assumptions, show your skepticism about the data, and always end with a "so what?" tied to a decision. The app can role-play these as mock interviews (see mock-interview.md).

The universal case-study structure: Clarify the question & the decision → define the metric precisely → inspect and plan to clean the data → segment to find the real story → reason to a conclusion with its caveats → recommend an action. Use it on almost any "here's some data, tell us what's going on" prompt.

D1

difficulty: easy
concept: defining-the-question Your stakeholder Slacks you: "Can you pull the numbers on how our app is doing?" Walk me through what you do before touching the data.
Framework: Resist exporting → ask what decision this feeds and what "doing well" means to them → pin down one primary metric, a time frame, and a likely breakdown → restate the sharpened question back for confirmation → only then plan the pull.
Model answer: "I wouldn't pull anything yet — 'how's the app doing' could mean ten different reports. I'd ask: what decision is this for, and what would 'good' look like to you? Say they're worried about growth. I'd sharpen it to: 'Did weekly active users grow week-over-week over the last 8 weeks, and is the change coming from new or returning users?' That names a metric, a time frame, and a breakdown. I'd send that one-liner back — 'is this the question?' — and pull only once they confirm."
Rubric: Strong answers slow down, surface the decision behind the ask, and convert a vague request into one answerable question naming metric + time frame + breakdown, then confirm before working. Weak answers immediately propose a dashboard or start writing SQL, leaving the real question undefined — the classic way to deliver a technically correct but useless report.

D2

difficulty: medium
concept: metric-definition The take-home says: "Build a query for our conversion rate and tell us if it's healthy." Before you write SQL, how do you define "conversion rate," and why does it matter?
Framework: Name the ambiguity (converted out of whom, into what, over what window) → write down one explicit definition → state edge cases (refunds, repeat buyers, bots, the denominator population) → note that "healthy" needs a baseline or benchmark → flag that you'll show your definition in the deliverable so numbers are reproducible.
Model answer: "'Conversion rate' is dangerously vague — two analysts will hand back different numbers from the same table. I'd define it explicitly: of users who started checkout in a given week, the share who completed a paid order within 24 hours, excluding refunded and test orders. I'd write that definition at the top of the deliverable. 'Healthy' is meaningless without a comparison, so I'd judge it against last quarter's trend and any stated target, not in isolation. If the prompt is silent on the window or the denominator, I'd state my assumption rather than guess silently."
Rubric: Strong answers treat the metric definition as the actual work — they pin the numerator, denominator, time window, and exclusions explicitly, and insist "healthy" requires a baseline. They surface assumptions instead of burying them. Weak answers jump to SQL with an unstated definition, conflate rate with raw count, or call a number "good/bad" with nothing to compare it to.

D3

difficulty: medium
concept: cleaning-data You open the take-home CSV of 50,000 orders. Before any analysis, walk me through how you'd make sure this data is trustworthy.
Framework: Look at the raw rows first (min, max, a sample) → check the usual suspects: missing values, duplicates, inconsistent formatting, outliers, wrong types → decide and document a treatment for each (exclude / fill / flag), never silent deletion → sanity-check totals against a known reference → record every change so the work is reproducible.
Model answer: "First I'd eyeball the raw data — sort by order value, scan the min and max, look at 20 random rows; many errors are obvious on sight. Then I'd check for duplicate order IDs (they silently inflate revenue), missing or blank order dates, country fields like 'USA' / 'U.S.A.' / 'us' that break grouping, amounts stored as text that won't sum, and outliers like a 10,000-unit order — which I'd investigate, not assume is wrong. For each issue I'd pick a treatment and write it down: 'removed 412 duplicate IDs, excluded 1,100 rows with no order date.' Finally I'd sanity-check: does total revenue roughly match what finance reports? If a result later looks surprising, I'd suspect the data before believing the surprise."
Rubric: Strong answers start by looking at raw data, name concrete problem types, choose a documented treatment per issue, and sanity-check totals against an external anchor — showing care and skepticism. They never silently drop rows. Weak answers assume the data is clean, jump straight to charts ("garbage in, garbage out"), auto-delete anything odd, or treat cleaning as a box to tick rather than the bulk of trustworthy analysis.

D4

difficulty: medium
concept: segmentation Your headline result is "average revenue per user is flat month-over-month." The hiring manager nods, but you suspect that's not the whole story. What do you do?
Framework: Distrust the single overall number → ask "is this true for everyone, or is an average hiding something?" → pick segments that map to the business (new vs. returning, region, plan tier, acquisition channel, device) → compare segment trends, not just the total → watch for a reversal hidden in the aggregate (Simpson's paradox) → report the overall and the segments that drive it.
Model answer: "'Flat overall' is exactly when I get suspicious — a single average often hides opposite trends. I'd split revenue per user by new vs. returning customers and by plan tier. A flat total can easily mean new-customer revenue is collapsing while a few returning power users are surging — same average, completely different story and a completely different action. I'd specifically check whether a trend reverses when split, because a real movement can cancel out in the aggregate. My deliverable would lead with the overall number but immediately show the two or three segments that actually explain it."
Rubric: Strong answers treat an overall average as a question, not an answer; they segment along business-meaningful dimensions, look for hidden or reversing trends (Simpson's paradox), and report what's driving the total. Weak answers accept the flat headline at face value, segment randomly with no hypothesis, or never check whether the average is masking diverging groups.

D5

difficulty: hard
concept: correlation-causation In the take-home you notice users who use Feature X have 3× higher retention. The prompt asks: "Should we push everyone to use Feature X?" How do you answer?
Framework: Separate correlation from causation explicitly → name the likely confounder / selection effect (who chooses X may already be your best users) → resist the causal claim the stakeholder wants → state what evidence would establish cause (a controlled A/B test) → give an honest, useful interim answer with appropriate hedging rather than a false certainty.
Model answer: "I'd flag this as a correlation, not proof. The most likely explanation isn't that Feature X causes retention — it's that already-engaged users are the ones who discover and adopt X, so the same loyalty drives both adoption and retention. That's a classic confounder. 'Ice cream and drownings' move together because of summer, not because one causes the other. So I wouldn't recommend a company-wide push on this evidence alone. What would settle it is an experiment: randomly prompt a held-out group to try X and compare retention against a control. Until then my honest answer is 'X is a strong signal of an engaged user and worth investigating, but we can't yet say it creates retention — let's test it before betting on it.'"
Rubric: Strong answers refuse to convert correlation into causation, name the specific confounder or selection effect, and propose a controlled experiment as the real test — while still giving a usable, honestly hedged interim recommendation. Weak answers announce "Feature X causes retention, roll it out," ignore self-selection, or hedge so vaguely the stakeholder gets no guidance at all. Best answers balance scientific honesty with decision-usefulness.

D6

difficulty: hard
concept: storytelling Time's up on the take-home. You have 5 minutes and one slide to present your finding to a non-technical exec. The data was noisy and your conclusion is only an early signal. What do you say?
Framework: Lead with the answer in one sentence ("so what?"), not the process → back it with two or three numbers/charts, no more → translate impact into their terms (money, users) → make an explicit recommendation tied to a decision → be honest about uncertainty without burying the finding → keep methodology and caveats as details-on-demand.
Model answer: "I'd open with the headline: 'New-customer revenue dropped about 20% last month, and it traces to mobile checkout failing after the redesign.' Then two pieces of evidence: the new-vs-returning split, and the mobile-vs-desktop completion gap. I'd quantify it in their terms — 'roughly $40K of lost monthly revenue.' Recommendation: 'prioritize fixing mobile checkout; I'd validate with a quick funnel check first.' And I'd be upfront about uncertainty: 'this is based on four weeks of fairly noisy data, so treat it as a strong early signal, not a proven number.' Methodology and the cleaning notes go in an appendix for anyone who asks. Overclaiming on noisy data would cost me credibility; calibrated honesty builds it."
Rubric: Strong answers are answer-first, use plain language, quantify impact in the audience's terms, end on a clear recommendation, and state uncertainty calibrated to the data without hiding the finding — methodology relegated to details-on-demand. Weak answers bury the conclusion under process, use jargon ("churn velocity"), present every chart they made, omit a recommendation, or either over-claim certainty on noisy data or hedge so heavily the exec can't act.