Topic 5 — Evaluating AI output and quality

Goal: Learn to judge AI output rigorously — the core of AI Operations.

Lesson 5.1 — The confident wrong answer

In her second week as an AI quality analyst at Trellix Health, Bina Velasquez is reviewing a draft from Scribe — the company's AI assistant that writes clinical visit notes for small clinics. The note is beautiful. Clean paragraphs, the right medical vocabulary, a tidy summary a busy doctor would happily sign. It reads like it was written by someone who knew exactly what they were doing.

Then she checks the chart. The patient is on metformin. The draft says lisinopril.

The note didn't sound unsure. There was no stumble, no "I think," no flag. The AI delivered the wrong drug with the same calm fluency it used for everything else. And that is the whole reason her job exists.

AI is fallible but confident. It hallucinates facts, makes subtle errors, and produces a different answer each time you ask — yet it phrases every answer like it's certain. The dangerous output is the one that looks perfect and is quietly wrong, not the one that looks broken.

Evaluation is the work of catching that — judging whether AI output is correct, useful, and safe before it reaches a human who'll trust it. At Trellix, a hallucinated drug name can land in a real patient's chart, so evaluation sits at the center of the product, not off to the side.

Think of it as QA for software — except you're not testing code, you're testing the AI's reasoning and content. The bug hides inside a sentence that reads just fine.

Bina spent eight years teaching high-school biology. Grading two hundred essays a week trained her to spot the student who writes with total confidence and gets the mitochondria wrong. That instinct — a nose for the fluent answer that doesn't hold up — turns out to be the exact instinct this work rewards.

Lesson 5.2 — More than "does it sound good"

When Bina started, her first instinct was to read a draft and ask, is this good? That question is too vague to act on. A note can be well-written and wrong, or accurate and rude, or correct and missing the one thing that mattered.

So she learned to break "good" into dimensions — separate things to check, each on its own:

• Accuracy / factuality — is it actually true? Hallucinated facts, names, dosages, numbers. This is the most important one, because it's where confident AI fails most and does the most harm.
• Relevance — does it answer what was actually asked, or wander?
• Completeness — does it cover what's needed, or skip the patient's chief complaint?
• Coherence / clarity — is it organized and readable?
• Tone / appropriateness — is the voice right for a clinical note versus a patient message?
• Safety / harm — does it avoid harmful, biased, or unsafe content? In a patient-facing reply, this one is non-negotiable.
• Instruction-following — did it obey the format, the length, the constraints it was given?

The trick is checking them separately. A draft can score high on clarity and fail on accuracy, and if you only ask "is it good?" the clarity quietly papers over the error — which is exactly how the lisinopril note almost slipped through.

Dr. Soledad Marchetti, a physician at one of Trellix's pilot clinics, taught Bina why factuality sits at the top. "I can fix clumsy wording in five seconds," she said. "A wrong dosage I might not catch, because the note reads like I wrote it." That's also where domain knowledge earns its keep — to judge whether a clinical claim is right, someone on the chain has to actually know medicine.

Lesson 5.3 — A rubric, then a score or a comparison

Here's a problem Bina hit fast. She'd rate a draft a 4. A teammate would rate the same draft a 2. Neither was careless. They just carried different pictures of "good" in their heads.

This is the quiet failure of gut-feel evaluation: without a written standard, two trained people reviewing the same output disagree roughly 30–40% of the time. Not because one is wrong, but because "good" lives in two different heads.

The fix is a rubric — the criteria for "good," written down in advance, the way QA writes acceptance criteria before a feature ships. A Scribe rubric doesn't say "the note should be accurate." It says: no factual claim unsupported by the chart; all medications and dosages match the record; chief complaint in the first sentence; under 250 words. The strongest rubrics also carry anchor examples — a real "this is a 5" note and a real "this is a 2" note pinned beside the scale, so the numbers mean the same thing to everyone. Anchors are what drag that 30–40% disagreement toward consistency. Bina built Trellix's first Scribe rubric in a shared doc, then had three reviewers grade the same ten drafts against it; where they split, she sharpened the wording until they stopped.

Once the rubric exists, there are two everyday ways to turn judgment into data, and Bina uses both. The first is scoring: rate one response against the rubric, say 1 to 5. It tells you how good something is on an absolute scale — and the anchor examples are what keep a "4" meaning the same thing on a tired Friday and a fresh Monday.

The second is pairwise comparison: put two responses side by side and answer one question — which is better? People are far steadier at "B beats A" than at "A is a 3.5." Marc Devlin, the AI product manager, leans on this when the team is choosing between two prompt versions. They don't ask reviewers to score each note; they ask which one they'd rather a doctor sign.

That comparison data is worth more than it looks. A pile of "B is better than A" judgments is exactly the preference signal used to improve the model itself — the human feedback behind RLHF, which Topic 6 covers. When Bina marks one draft better than another, she isn't only grading today's output. She's producing the raw material that teaches the next version of Scribe what "better" means.

Scoring tells you how good one answer is. Comparison tells you which of two to keep — and that second kind of judgment is what models learn from.

Lesson 5.4 — Be specific, then look for the pattern

Early on, Bina logged a verdict that read, in full: "This one's wrong." Pri Reddy, the engineer who builds Scribe's pipeline, came back with a fair question — which part, and how? She can wire up models and RAG, but she can't read a clinical note and see what's off. The whole point of Bina's review is to tell her.

So the rule became: specific, documented feedback. Not "this is wrong," but "the date in paragraph 2 is wrong — visit was the 14th, note says the 4th; the dosage claim in paragraph 3 is unsupported by the chart." Specific feedback is both more useful to Pri and more usable as a training signal, because it points at the exact failure instead of a vague mood.

Then comes the move that separates a good evaluator from a great one: spotting patterns. One wrong note is a note to fix. The same kind of wrong note, over and over, is a system to fix. After her first month, Bina's log let her tell Marc two things she could never have seen one draft at a time: Scribe invents specific statistics when the chart is thin and it struggles with this whole drug class.

That reframes the work. Patching one output helps one patient. Naming the pattern tells the team where to fix Scribe itself — a prompt change, a guardrail, a gap in the RAG data — so a thousand future notes come out right. Her teacher's habit of reading a stack of papers and noticing the whole class missed question 4 is the same habit, pointed at AI.

Lesson 5.5 — When the judge is itself an AI

By 2026, Bina can't hand-review everything. Trellix generates thousands of drafts a day; she reviews dozens. So the team adopted a tool that's reshaping this work: LLM-as-a-judge — using a strong model to score outputs automatically, against the same rubric a human would use.

The economics are blunt. A careful human review costs real expert time and effort — only a few dozen items fit in a day, and at a clinic that time is expensive. An LLM judge runs for fractions of a cent and thousands a minute. The savings are only half of it: a well-built judge reaches about 80% agreement with human preferences, which happens to be about how often two trained humans agree with each other. The judge doesn't get tired, distracted, or drift between Friday and Monday.

So is Bina automated away? No — and understanding why is the real lesson.

A judge is only as trustworthy as its calibration. The team builds a "golden set" — 30–50 drafts that domain experts like Dr. Marchetti have annotated as ground truth — and checks the judge against it. A good judge hits 85–90% agreement with that human reference. When it drifts below, Bina rewrites the judge's rubric prompt until it climbs back. The judge is fast; the human-annotated golden set is what makes it correct.

And on the cases that matter most — a subtle clinical error, a culturally off patient message — human review still catches what automated scoring misses. So the picture is the skilled human leveraged, not human replaced by judge: Bina now governs an army of automated judges, spends her own attention on the high-stakes 5%, and keeps the whole system honest. Theo Brandt-Okonkwo, who onboards new clinics, puts it plainly to nervous staff: "The AI grades the easy thousand. Bina decides what 'right' even means, and watches the hard ones."

Worked example — Bina ships a quality cycle

A pilot clinic complains: Scribe's patient-message replies "feel off." Vague. Bina turns it into an evaluation.

Dimensions first. She pulls 60 recent replies and scores each on accuracy, tone, safety, and instruction-following — separately, not as one blurry "good?"

Rubric. She writes down what each dimension means for a patient message: no medical claim beyond what the doctor approved (accuracy); warm, plain, no jargon (tone); never gives dosing advice (safety); under 120 words (instruction-following). Two anchor examples — one clearly great reply, one clearly bad — pinned to the scale.

Compare, don't just score. For a candidate prompt fix, she runs pairwise: old reply versus new, which would you rather a patient receive? The new prompt wins 47 of 60. That preference data goes to Pri.

Specific feedback. Her log doesn't say "tone is off." It says: "14 of 60 replies open with clinical jargon ('per your laboratory findings') instead of plain language."

The pattern. That's not 14 unrelated misses — it's one failure: Scribe defaults to clinical register in patient-facing text. Marc now has a fixable target, not a mood.

Scale it. Once the new prompt ships, Bina can't re-read every reply forever. She builds an LLM judge for tone and length, calibrates it against a 40-reply golden set Dr. Marchetti annotated, gets it to 88% agreement — and reserves her own eyes for anything the judge flags as a safety risk.

Vague complaint in. A fixed system, a measured improvement, and a running quality check out. That cycle — dimensions, rubric, comparison, specific feedback, pattern, scaled judge — is the core skill of AI Operations.