Topic 7 — AI limitations, risks, and ethics

Goal: Understand AI's limits and how to use it responsibly.

Lesson 7.1 — The confident wrong answer

Three weeks into her job at Trellix Health, Bina is reviewing a Scribe draft of a visit note. The note cites a medication dose, neat and specific: "Lisinopril 40mg daily." It reads like something a careful clinician wrote. The problem is that the patient was never on 40mg. The model produced a plausible-sounding number, and a plausible-sounding number is exactly what it's built to produce.

Bina was a biology teacher for eight years. She has graded ten thousand exams, and she knows the worst kind of wrong answer is the one written with total confidence. That instinct is now her most valuable professional skill.

This failure has a name. Hallucination is when AI confidently generates false information — invented facts, fake citations, wrong numbers — because it is producing plausible text rather than retrieving verified truth. It isn't lying and it isn't broken. Generating fluent, likely-sounding language is the whole mechanism (Topic 3), and sometimes the most likely-sounding sentence is simply untrue.

This is the headline limitation, the one every other lesson in this topic orbits around. At Trellix the stakes make it vivid: a hallucinated dose in a clinical note can reach a real patient. So the rule Bina lives by is short.

Treat every factual claim from AI as unverified until a human or a trusted source confirms it.

That's how you use the tool correctly, not distrust of it. Scribe drafts; a person verifies. The draft saves Dr. Marchetti twenty minutes per note, and the verification step is what makes those twenty minutes safe.

Lesson 7.2 — Smart at patterns, shaky at logic

Pri, the ML engineer, once showed Bina a transcript that delighted them both. They'd asked a model a simple word problem — a clinic has 3 exam rooms, each booked 6 times a day, how many appointments — and the model wrote two paragraphs of confident reasoning and landed on the wrong total. Three rooms, six each, and it said 21.

Brilliant at language, occasionally hopeless at arithmetic a fourth-grader nails. That gap surprises people, so it's worth naming the family of limits it belongs to.

AI is extraordinary at patterns and unreliable at strict reasoning. It can fail at logic, math, or plain common sense, and it fails confidently, with the same fluent tone it uses when it's right. There's no internal alarm that goes off when it's wrong.

A handful of related limits travel with this one:

• Knowledge cutoff. A base model's knowledge stops at its training date. Ask it about something newer and it may answer from a stale world, or fill the gap with invention, unless it's connected to live tools or current documents.
• Inconsistency. AI is probabilistic. Ask the identical question twice and you can get two different answers. Useful to know when you're tempted to trust one lucky-looking response.
• Context limits. It can only hold so much at once (the context window), and it has no memory of past conversations unless you deliberately give it some.
• Sensitivity to phrasing. A small change in wording can swing the output hard. That's the lever behind prompt engineering (Topic 4), and also a fragility: it means the output you got partly depended on how you happened to ask.

None of this is a reason to avoid AI. It's the map of where to keep your hands on the wheel.

Lesson 7.3 — Bias travels in the data

Theo, the consultant who onboards new clinics, brings Bina a worry from the field. One clinic wants Scribe to help triage which patient messages get a same-day reply. Reasonable idea. Bina's teacher-radar pings anyway.

Here's why. Bias in AI is the tendency to absorb — and sometimes amplify — the unfairness present in its training data. AI learns from human-created text, and human-created text carries human assumptions about gender, race, age, and more. A model can inherit those patterns without anyone intending it to, and then apply them at scale.

That turns abstract harm concrete fast. A triage helper that, say, systematically rates the concerns of older patients as less urgent is a clinic treating people unfairly, automatically, all day — not a quirky bug.

So in anything that touches people directly — hiring, lending, healthcare — recognizing and reducing bias is not an optional nicety. It's part of the job description for an AI-adjacent professional. Bina's role in the triage idea is to test it deliberately: feed it varied patient profiles, hold the medical content constant, and watch whether the urgency scores drift by anything that shouldn't matter. That's an evaluation (Topic 5) aimed straight at fairness.

You don't need to be a statistician to do this well. You need to expect bias, look for it on purpose, and refuse to ship the thing that fails the look.

Lesson 7.4 — What you put in matters as much as what comes out

Bina's first instinct, back when Scribe flagged that wrong dose, was to paste the patient's full note into a free public chatbot and ask it to double-check. She caught herself with her hand on the keyboard.

That pause is the whole lesson on privacy and data security. Information you enter into an AI tool may be stored, logged, or used to improve the model. So the rule is simple and firm: never feed private personal data, secrets, or confidential business information into a tool unless you know it's handled safely.

At Trellix this is concrete. Patient data flows only through the compliant, contracted systems the company has vetted and signed agreements for. A random consumer chatbot is not one of them, no matter how convenient it looks at 4pm. Pasting a patient's note into it would have been a real privacy breach, even with good intentions.

The general skill underneath the Trellix rule: before any sensitive data goes into a tool, know where it goes and who can see it. If you can't answer that, the data doesn't go in.

This sits beside the other risks worth naming plainly:

• Misinformation at scale. Convincing false content is now cheap to generate in bulk, which is a genuine societal concern and another reason careful use matters.
• Over-reliance. Skipping verification, outsourcing your judgment to the tool. The quiet failure mode, and the most common one.
• Security misuse. Any powerful tool can be abused. Responsible builders ask how their AI feature could be turned against people, and guard against it.

The throughline across all of them: AI should augment human judgment, not replace it.

Lesson 7.5 — Responsibility is now written into the law

For a while, "use AI responsibly" was advice. As of 2026 it is, in growing part, regulation — and that changes the conversation Marc the product manager has every week.

Responsible AI rests on a few principles you can hold in your head: keep a human in the loop for consequential decisions, verify facts and check for harm before you act or publish, be transparent about AI-generated content where it matters, protect privacy, and actively reduce bias and harm. Trellix builds all five into Scribe on purpose — human review sits at the center of the product, not bolted on at the edge.

What's new is that governance has teeth. Three reference points an AI-adjacent professional should recognize by name:

• The EU AI Act. Its high-risk and transparency obligations become enforceable from August 2, 2026, with penalties up to €35 million or 7% of global annual turnover for the most serious breaches. Article 50 means people must be told when they're dealing with AI-generated content or talking to a bot. A clinical tool like Scribe lands squarely in the high-risk, must-disclose territory.
• The NIST AI Risk Management Framework. A voluntary US framework that organizes the work into four functions: Govern, Map, Measure, Manage. It's the practical playbook many teams use to structure their risk process.
• ISO/IEC 42001. The first international standard for an AI management system — the certifiable shape an organization can build its whole AI-governance practice around.

Disclosure and human oversight are no longer just good manners. They're increasingly written into both products and law.

Bina doesn't need to be a lawyer. She needs to know these exist, know that Scribe discloses its AI involvement to clinics, and know that the verification work she does every day is the ground floor of compliance, not a separate chore.

Worked example — Scribe drafts a patient message

A pilot clinic asks Trellix to let Scribe draft replies to patient messages, going beyond visit notes. Marc green-lights a trial. Watch every limit in this topic show up in one small feature.

A patient messages: "Can I take ibuprofen with my new blood pressure med?" Scribe drafts a warm, fluent reply that says yes and cites a guideline. Confident, well-written, and — Dr. Marchetti spots — wrong for this patient, whose kidney history makes that combination risky. A textbook hallucination with a real person on the other end.

Bina runs an evaluation before the feature goes anywhere. She checks consistency by sending the same question five times (it varies — inconsistency), and she tests for bias by varying the patient's name and age while keeping the medical facts fixed, watching whether the tone or caution level drifts. It does, slightly, for older patients' names. That's a finding, not a footnote.

On privacy, the team confirms the message content never leaves Trellix's contracted, compliant pipeline — no patient text touches an outside tool. On governance and disclosure, every drafted reply carries a clear note that it was AI-assisted and reviewed by clinic staff, satisfying the kind of transparency the EU AI Act's Article 50 expects.

The decision: Scribe may draft, a clinician must approve before send, and nothing reaches a patient unread. Human in the loop, by design. The feature ships not because the AI got smart enough to trust, but because the humans around it stayed responsible. That blend of capability and conscientiousness is exactly what made Bina the right person to own the call.