Interview Drills — Judgment, Risk & Responsible Use

Open-ended interview questions for AI-adjacent roles (AI Operations, AI quality, trust & safety, data/annotation, AI-product support). Each has a Framework (the structure a strong answer follows), a Model answer (a concise example), and a Rubric (what an interviewer listens for). These test the limits-and-ethics judgment from Topics 6–7: how you tell whether AI output can be trusted, how you handle hallucination, bias, and privacy, when you keep a human in the loop, and when you disclose AI use. The app can role-play these as mock interviews (see mock-interview.md).

The universal responsible-use structure: Name the risk (hallucination / bias / privacy / over-reliance) → say how you'd detect it (verify against a source, check who's affected) → decide who must approve before acting (human-in-the-loop for consequential calls) → state what you'd disclose and to whom. Lead with "I'd verify before I trust," not "the AI said so." Use it on almost any "how would you handle…" question.

D1

difficulty: easy
concept: evaluating-ai-output An AI tool gives you a confident, well-written answer with specific facts and a citation. How would you decide whether it's trustworthy enough to use?
Framework: Separate fluency from truth (confident ≠ correct) → identify the checkable claims (facts, names, numbers, the citation) → verify each against an independent, authoritative source → judge the stakes (who's hurt if it's wrong?) → decide: use, fix, or escalate.
Model answer: "I'd remember that AI produces plausible text, not verified truth, so a confident tone tells me nothing. I'd pull out the checkable parts — the date, the statistic, and especially the citation — and verify each against a primary source. I'd actually open the cited link, because hallucinated citations are common. If anything fails to check out, or it's a high-stakes use like a customer-facing claim, I don't use it as-is — I correct it or flag it for review."
Rubric: Strong answers explicitly distinguish fluency from factuality, name which elements they'd verify, insist on an independent source (and check the citation actually exists), and scale rigor to the stakes. Weak answers trust it because it "sounds right," check nothing, or treat the AI as the source of truth.

D2

difficulty: medium
concept: ai-limitations You're reviewing AI-drafted content before it's published. Walk me through how you'd catch a hallucination.
Framework: Define hallucination (a confident, plausible-sounding falsehood) → scan for the high-risk slots where it hides (specific facts, figures, quotes, citations, names, dates) → verify each against an authoritative source rather than against the AI's own confidence → document the failure specifically → note the pattern if it recurs.
Model answer: "Hallucinations hide in the specific, checkable claims — a precise statistic, a named study, a quote, a date. I go through the draft and flag every one of those, then verify each against a trusted source; the polished writing around them is exactly what makes a false fact slip through. When I find one I document it precisely — 'the 2023 figure in paragraph two is unsupported; no such study exists' — not just 'this is wrong,' because specific feedback is what improves the system. If the model keeps inventing statistics, that pattern is worth reporting."
Rubric: Strong answers target the specific checkable claims, verify against external sources (not the AI), give precise documented feedback, and spot recurring failure patterns. Weak answers reread for tone, "trust their gut," or say "I'd ask the AI if it's sure."

D3

difficulty: medium
concept: bias-and-fairness You notice an AI hiring-screen tool consistently rates candidates from one demographic group lower. What do you do?
Framework: Name the risk (AI learns from human data and can absorb/amplify societal bias) → don't dismiss it as a one-off — check whether it's systematic across cases → trace the likely source (biased or unrepresentative training/feedback data) → escalate, because this affects real people's livelihoods and may be a legal/fairness issue → push for a human-in-the-loop safeguard until it's fixed.
Model answer: "Because AI inherits bias from its data, a consistent skew against one group is a red flag, not noise. I'd first confirm it's systematic by looking across many cases, then document the pattern with specific examples. Since this affects people's livelihoods and touches fairness and legal lines, I'd escalate rather than quietly work around it. Until it's investigated I'd argue no candidate should be auto-rejected on the model's score alone — a human must review, so a biased system isn't making consequential decisions unchecked."
Rubric: Strong answers recognize bias originates in data, verify it's systematic before acting, escalate because people are affected, and insist on human review for consequential decisions. Weak answers explain it away, assume the AI is neutral, or "fix it themselves" without escalating a fairness/legal issue.

D4

difficulty: medium
concept: privacy-and-data-security A colleague suggests pasting a spreadsheet of customer records into a public AI chatbot to clean it up faster. How do you respond?
Framework: Name the risk (inputs may be stored or used to train; this is confidential personal data) → state the principle (never feed private/confidential data into a tool unless you know it's safe) → don't just block — offer a responsible alternative → tie it to consent and obligations to the people in the data.
Model answer: "I'd push back, kindly. Whatever we paste into a public tool may be stored or reused, and these are real customers' personal records — putting them in isn't ours to do without knowing the tool is safe and approved. So I wouldn't use a public chatbot here. Instead I'd suggest an approved enterprise tool with a data agreement, or anonymizing/removing the identifying fields first so no personal data leaves our control. The goal — cleaning the data faster — is fine; the method has to respect privacy and the customers' consent."
Rubric: Strong answers recognize inputs can be retained, refuse to expose confidential personal data, and offer a safe alternative (approved tooling, anonymization) rather than only saying no. Weak answers see no problem, focus only on speed, or block without understanding why or offering a path forward.

D5

difficulty: hard
concept: human-in-the-loop Leadership wants to fully automate a decision your team currently makes with AI assistance — to cut costs. When is removing the human in the loop acceptable, and when is it not?
Framework: Restate the principle (AI augments judgment; don't let it make consequential choices unchecked) → propose a decision rule based on stakes and reversibility → keep humans where errors are costly, irreversible, or affect people; automate low-stakes, easily-reversible, high-volume cases → propose a middle path (AI handles routine, humans handle edge cases and audits) → name the metric you'd watch.
Model answer: "I'd frame it by stakes, not cost. Where a wrong call is costly, hard to reverse, or directly affects a person — denying a claim, flagging someone — a human stays in the loop, because AI is confidently fallible and shouldn't make those unchecked. Where the decision is low-stakes, high-volume, and easily reversible, automating is reasonable. So I'd propose a tiered model: AI auto-handles the clear, low-risk cases and routes anything uncertain or high-impact to a human, with humans also auditing a sample of the automated ones. I'd track the error rate and escalation rate so we'd see immediately if automation started causing harm."
Rubric: Strong answers tie human oversight to stakes/reversibility/impact-on-people, distinguish what's safe to automate from what isn't, propose a tiered design with sampling/audit, and name a monitoring metric. Weak answers either automate everything for cost or refuse all automation — neither shows judgment about where the human matters.

D6

difficulty: hard
concept: ai-disclosure-and-transparency You used AI to help produce a report that will go to clients. Should you disclose that AI was involved, and how would you decide? Walk me through a case where you would and one where you wouldn't.
Framework: State the transparency principle (disclose AI-generated content where appropriate; don't pass off unverified AI work as authoritative) → identify what drives the call: the stakes, audience expectations, whether AI shaped substance vs. just polish, and any policy/legal duty → give a "disclose" case and a "no formal disclosure needed" case → tie disclosure to verification (you still own correctness either way).
Model answer: "My default leans toward transparency, scaled to how much AI shaped the substance and what the audience expects. If AI generated the analysis or claims the client will rely on, I'd disclose it and make clear a human verified the facts — passing unverified AI work off as authoritative is exactly what to avoid. If AI only helped tighten wording on content I authored and stand behind, a formal disclosure usually isn't necessary, though I'd follow any client or company policy. Either way disclosure doesn't replace verification — I'm accountable for the report being correct whether or not AI touched it."
Rubric: Strong answers make disclosure depend on stakes/audience/whether AI shaped substance and on policy, give a concrete disclose vs. not-disclose contrast, and stress that disclosure never substitutes for verifying and owning the output. Weak answers give a blanket "always/never," ignore who relies on it, or treat disclosure as a way to offload responsibility for accuracy.