Here’s something I see a lot.
A team has built a clinical AI product. RAG pipeline, vector database, the works. They run their evals. Retrieval looks great. Recall is high. Their top-k returns the right source documents most of the time.
So they ship.
Three months later, a clinician flags an answer that’s wrong. They trace it back. The retrieval was correct. The right source was there. The model just didn’t say what the document said.
The team is confused. Their tests passed. Their metrics looked good. What happened?
What happened is they measured the wrong thing.
Two questions, not one
When a clinical AI system answers a question, there are actually two things that have to go right.
The first is retrieval. Did the system find the right source material for the question being asked?
The second is interpretation. Did the model accurately convey what that source said?
Most teams test the first one. Almost nobody tests the second one. And the gap between them is where reliability quietly dies.
You can have perfect retrieval and a confidently wrong answer. The model pulled the right Cochrane review. It also paraphrased it in a way that flipped a key conclusion. Or it stitched together two correct passages into a synthesis that neither source actually supports. Or it dropped a critical qualifier and turned a “consider in selected patients” into a recommendation.
If you’re only watching retrieval, you don’t see any of this. Your metrics keep looking good while your product quietly loses clinician trust.
Why this is invisible in normal testing
Standard RAG evaluation focuses on retrieval metrics. Precision at k. Recall. Mean reciprocal rank. These are useful, and they tell you something real. But what they tell you is whether the right document is in the candidate set. They tell you nothing about what the model does with it.
There’s a reason this is the default. Retrieval is measurable in a clean, automated way. You have ground truth (the right document). You have a candidate set (what the retriever returned). You can score it.
Interpretation is harder to measure. The ground truth is “what does this source actually say about the question.” Comparing the model’s answer against that requires either a domain expert reading the output and the source side by side, or another model doing the comparison and inheriting its own biases.
So most teams don’t measure it. They measure retrieval, they ship, and they hope.
In casual products, this works fine. The cost of a bad answer is low. The user shrugs and moves on.
In clinical AI, every wrong answer is expensive. Even if the clinician catches it, they remember. Even if they don’t catch it, somebody downstream might. Once enough wrong answers accumulate, trust collapses and the system gets quietly turned off.
You don’t get a warning when this is happening. You just notice, six months in, that adoption has stalled.
What interpretation failures actually look like
Let me get concrete about the failure modes, because “the model didn’t say what the document said” is too vague to be useful.
There are a few common patterns I see in production clinical AI.
The paraphrase flip. The model rewords a passage and accidentally inverts its meaning. The source says “X is generally not recommended in patients with Y.” The model summarizes it as “X may be used in patients with Y.” Same words. Opposite clinical guidance.
The dropped qualifier. The model strips a clinically important hedge. The source says “small studies suggest X may help in selected patients.” The model returns “X helps.” The qualifier was the whole point.
The composite hallucination. The model takes correct content from two different sources and stitches them into a synthesis that neither source supports. Each piece is real. The combination is invented. This one is especially insidious because each citation checks out individually.
The confident gap fill. The model is missing a piece of the answer and fills it in with something plausible. Sometimes the fill is correct. Sometimes it’s wrong. The model treats both the same way: it presents them with the same confidence. No flag, no uncertainty, no “this part I’m less sure about.”
The wrong-population answer. The retrieved document discusses adults. The question was about pediatrics. The model uses the adult guidance to answer the pediatric question without flagging the mismatch.
Every one of these can happen with perfect retrieval. The right document is in the system. The model still produced a wrong answer. And if you’re only watching retrieval metrics, you’ll never see any of them.
What testing for interpretation actually requires
If interpretation is the failure mode that’s quietly killing your reliability, you need a way to measure it. Here’s what that looks like in practice.
You need a test set of questions where you’ve manually written or validated the correct answer based on the source material. Not just “is the right source returned.” The actual right answer, in the form your system would output it.
Then you run your full pipeline (retrieval plus generation) and compare the generated answer against the validated answer. Not “is it semantically similar.” Specifically: does it convey the same clinical meaning, with the same qualifiers, applied to the same population, with no invented content?
That comparison can be done by a domain expert (slow, expensive, but high quality) or by another model with a carefully designed evaluation prompt (faster, cheaper, but you have to validate the evaluator). Most production teams end up doing both. Expert review on a smaller golden set, model-based evaluation at scale, with periodic expert audits to confirm the evaluator is calibrated.
The numbers you watch are different from retrieval metrics. Faithfulness. Completeness. Hedge preservation. Population fit. Composite hallucination rate. These are the metrics that tell you whether your interpretation layer is reliable.
If you’ve never measured these, you don’t actually know how often your system is wrong. You know how often your retrieval is wrong. Those are not the same number.
What to do about it architecturally
Measuring interpretation is one thing. Designing for it is another.
A clinical-grade system has a few patterns that reduce interpretation failure rate by design.
Bounded synthesis. The model is constrained to summarize what was retrieved, not speculate beyond it. If the retrieved set doesn’t contain the answer, the system says so. This kills the confident gap fill failure mode at the architecture level.
Source-tied output. Every claim in the generated answer is traceable to a specific source. Not just “here are the documents.” This claim came from that paragraph. This makes paraphrase flips and composite hallucinations visible to anyone reviewing the output.
Qualifier preservation. The synthesis layer is designed to retain clinical hedges. If the source says “may,” the output doesn’t say “does.” If the source says “in selected patients,” the output doesn’t drop “selected.” This is a prompt and architecture pattern, not a model behavior you can hope for.
Population matching. Before synthesis, the system checks whether the retrieved document’s population matches the question’s population. Adult guidance does not answer pediatric questions without flagging the mismatch.
Confidence on the right level. The model’s confidence is calibrated against the evidence it actually has, not against its general fluency. If half the answer is from the retrieved sources and half is gap fill, the system should flag the second half. Not pretend the whole thing is equally well supported.
None of these are features you bolt on. They’re architecture choices applied at the synthesis layer. Applied late, they’re a rebuild. Applied early, they’re just how the system works.
Where to start
If you have a clinical AI product in production and you’ve never run an interpretation evaluation, start there.
Pick 30 questions your system has actually been asked. Manually write the correct answer for each one based on the source material your system would retrieve. Now run those questions through your live system and compare the outputs against the validated answers.
Don’t grade on semantic similarity. Grade on whether the clinical meaning matches. Same recommendation. Same qualifiers. Same population. No invented content.
Most teams who run this exercise for the first time are surprised by their results. Not in the good direction.
What you do with those results is the next step. Sometimes it’s a prompt change. Sometimes it’s a retrieval change. Sometimes it’s a synthesis architecture change. Usually it’s some combination of all three.
But you can’t fix what you haven’t measured. And if you’re only measuring retrieval, you’re not measuring what your clinicians actually experience.
That’s the gap. That’s where reliability lives. And that’s the first thing we look at on a Reliability Assessment.
If you’d rather find this gap yourself before a partner or a clinician does, that’s a conversation worth having early.