Topic 8 — AI Product Management

Goal: Understand how AI products are built and what makes building with AI different.

Lesson 8.1 — The day the same input gave two answers

Bina is testing Scribe, Trellix Health's AI assistant that drafts clinical visit notes for small clinics. She pastes the same doctor's dictation in twice, expecting the same draft both times. She gets two. Close, but not identical: one phrases a medication instruction clearly, the other buries it in a longer sentence. Nothing broke. The thing just answered differently.

Eight years teaching biology, and this is the moment that reframes everything for her. A login button does the same thing every time you press it. Scribe does not.

That gap has a name. Traditional software is deterministic — it does exactly what it was programmed to do, every single time. AI is probabilistic — it produces plausible output that varies run to run and can be confidently wrong (you saw why in Topics 3 and 7). Two correct-looking drafts from one input are not a bug Pri needs to fix; they're the material the product is made of.

With normal software you ask "does it work?" With AI you ask "how often is it good enough, and what happens when it isn't?"

For Marc, Trellix's AI Product Manager, this one fact reshapes the job. He can't promise a specific output. He can only shape the range of outputs and decide what the product does when one lands at the bad end.

Lesson 8.2 — Designing for uncertainty

Once you accept the output will sometimes be wrong, your design questions change.

A deterministic feature is built around the happy path: define what it should do, build it, test it once, ship it. An AI feature has no single "should do." So Marc starts from the opposite end — he assumes Scribe will occasionally draft a note that's subtly off, and designs the experience around being wrong first.

That's designing for uncertainty: planning for the unsure, the empty, and the plain incorrect, rather than only the moment it nails it.

• When Scribe isn't confident a lab value was dictated correctly, it flags that line instead of presenting it as settled fact.
• When retrieval finds nothing relevant, the draft says so rather than inventing a plausible detail.
• Every draft lands in an editable review screen, never straight into the patient record.

It also means you can't test once and walk away. Because behavior drifts and the inputs never stop varying — every doctor dictates differently — Marc and Bina evaluate continuously (Topic 5) instead of signing off one pass. Quality here is a moving range you keep watching, not a checkbox you tick once.

Lesson 8.3 — AI suggests, the human decides

Dr. Soledad Marchetti, a family physician at one of Trellix's pilot clinics, told Bina the line that the team now repeats to every new clinic: "I'll use it, but I read every word before it goes near a patient."

That instinct is the most important design principle in the topic. Keep the human in control. Let people review, edit, or override the AI — especially for actions that matter. AI suggests, human decides is the sweet spot for high-stakes work, and a clinical note that could reach a real patient is about as high-stakes as it gets.

A few principles cluster around it, and Marc weighs all of them when shaping a feature:

• Set honest expectations. Scribe tells users plainly that it's AI-drafted and can make mistakes (the AI-disclosure idea from Topic 7). Over-promising "magic" shatters trust the first time the AI slips.
• Design for failure with graceful fallbacks. Decide up front what happens when the AI is wrong, unsure, or returns nothing useful — an easy edit, a way to reject the draft, a path to a human.
• Make feedback effortless. A thumbs-up/down, a one-click "this was wrong," the edits users already make — capture all of it. It improves the product and feeds the loop from Topic 6.
• Use AI only where it earns its place. Scribe applies AI to drafting and summarizing language, where it shines. Bolting AI onto something that worked fine without it is a gimmick, and users smell it.

The goal is a product where being wrong is cheap, visible, and easy to fix — not an AI that's never wrong.

Lesson 8.4 — "If you can't evaluate it, you can't ship it"

Pri, the ML engineer, can wire Scribe to a new model in an afternoon. What she can't do is judge whether a clinical note reads correctly — she's an engineer, not a clinician. That judgment is Bina's, and turning it into something measurable is the heart of the AI PM's job.

Marc has a rule he says out loud in planning: if you can't evaluate it, you can't ship it. Before a feature goes near a clinic, he and Bina build an evaluation framework — the machinery from Topic 5, owned now as a product responsibility:

• A dataset of real, representative dictations — the messy and ambiguous ones, rather than only clean demo cases.
• A rubric defining "good enough" in concrete terms: factually accurate, no invented details, safe phrasing, clinically clear.
• An LLM-as-judge prompt — a second AI scoring each draft against that rubric at scale — calibrated against Bina's spot-checks so judge and expert agree.
• Continuous monitoring in production, because a model that scored well last month can drift this month.

Quality stops being a yes/no and becomes numbers you track, reaching past "does it work." Marc watches task-completion rate (did Scribe fully draft a usable note end-to-end without a human rescue?), alongside quality scores, safety flags, and the human-override and edit rate — how often a doctor rewrites the draft. A rising edit rate is an early warning that quality is slipping, long before anyone files a complaint.

An AI PM also needs to understand the building blocks the AI team works with, even without coding them:

• RAG (retrieval-augmented generation): feeding the model relevant private, up-to-date documents — a clinic's own templates, a patient's prior notes — so it answers from real data instead of guessing, which sharply cuts hallucination. It's the backbone of most enterprise AI features.
• Guardrails: the limits and allowlists that keep the AI's actions safe — what it may say, do, or touch.
• Agents: AI that plans and takes multi-step actions using tools, rather than answering in one shot.

Marc doesn't write the RAG pipeline. He decides which documents Scribe retrieves from, what the guardrails forbid, and whether a step is a single suggestion or a full agent — all product calls.

Lesson 8.5 — The career-changer's opening

So what is an AI Product Manager? Everything a regular PM does — discovery, prioritization, working with the team, measuring success — plus a layer that only exists because the product is probabilistic.

Marc spends his week translating: telling Trellix's founders what Scribe can realistically promise a clinic, and telling Pri what "clinically clear" requires. He defines "good enough." He owns the data and feedback loops — the pipeline where doctors' edits and thumbs-downs become the next round of improvement (Topic 6). He carries the ethics and risk (Topic 7): bias, safety, privacy, a confident hallucination reaching a real patient. And he stays current, because what's possible shifts month to month.

Theo, the consultant who onboards clinics, lives the business half of that bridge — turning "what the AI can do" into "what this clinic needs." No single person here is the AI genius. The product comes from the translation between them.

For Bina, the ex-teacher, this is where her own future comes into focus. An AI PM needs strong product and communication skills plus solid AI literacy — and not the ability to build models. The AI literacy is exactly this course; the product skills are the PM track. Her classroom instincts — clear writing, patient judgment, a nose for a confident-but-wrong answer — are the rare, hard-to-hire half. The model-building she leaves to Pri. That's why this is one of the most accessible, and fastest-growing, on-ramps in tech.

Worked example — Shipping the "patient-message reply" feature

Trellix wants Scribe to draft replies to patient messages ("Is it normal to feel dizzy on this medication?"). Higher stakes than notes — this text could reach a patient. Watch the AI-PM thinking run end to end.

Marc starts with where AI adds value: drafting clear, empathetic replies is a real time-sink, and language is AI's strength. Worth building.

He immediately designs for uncertainty and keeps the human in control: no reply auto-sends. Scribe drafts, the draft lands in a review box, the clinician edits and presses send. AI suggests, human decides.

Pri proposes RAG so replies pull from the clinic's own guidance and the patient's chart, plus guardrails that hard-block anything resembling a new diagnosis or dosage change — those escalate to a human, never get drafted.

Before launch, Marc invokes his rule: if you can't evaluate it, you can't ship it. Bina builds a dataset of real patient questions, writes a rubric (accurate, safe, warm, no medical advice beyond scope), and stands up an LLM-as-judge calibrated to her spot-checks. The feature ships only above a quality bar, with task-completion rate and edit rate on a dashboard from day one.

Dr. Marchetti pilots it. Week one, the edit rate is high — doctors keep softening the tone. That's the feedback loop working: the edits flow back, the prompt is tuned, the rate drops, quality climbs. One feature, every principle in this topic inside it.