Most clinical AI teams test the wrong thing. And they test it the wrong way.
They run the model against some questions. They look at the answers. The answers look good. They ship. Then something breaks in production and they cannot reproduce it, cannot measure how often it happens, and cannot prove they fixed it.
I want to walk through what real reliability testing looks like. Not retrieval metrics, most teams already have those. The harder layer underneath. Does the system actually say the right thing, every time, under the messy conditions a hospital will throw at it.
This one is technical. If you are building clinical AI, the technical part is the part that matters most, so stay with me.
A quick note before I start. I spent years doing software testing the old way, the boring way, writing test cases and scenarios and tracing every requirement. People think testing is about confirming the thing works. It is not. Good testing is about trying to make the thing fail. That mindset is the whole difference here, and almost nobody applies it to AI.
First, get the definition of “wrong” right
The first problem is that most teams have a fuzzy idea of what a wrong answer even is. “It hallucinated” is not measurable. You cannot build a test suite around a word that vague.
Here is a more useful way to break it down. In a retrieval based clinical system, an answer can be wrong in a few separate ways, and they are not the same problem.
It can be wrong because retrieval failed and the right source never showed up. That is a retrieval problem. Most teams already measure this one.
It can be wrong because the right source showed up but the model misread it. It flipped a recommendation. It dropped a qualifier. It used adult guidance to answer a pediatric question. It stitched two sources into a conclusion neither one actually supports. That is an interpretation problem. Almost nobody measures it.
It can be wrong because the model filled a gap the sources did not cover, and served the invented part with the same confidence as the real part. That is a synthesis boundary problem.
It can be wrong because the system should have refused to answer and it didn’t. That is a calibration problem.
Four different failure types. Four different fixes. If your only metric is “accuracy,” you cannot tell them apart. And if you cannot tell them apart, you cannot fix any of them efficiently, because you do not even know which one you are looking at. The first step in testing for reliability is committing to a list of failure types specific enough that you can actually do something about each one.
Build a test set that looks like reality, not your demo
The second problem is the test set itself.
Most teams test against questions that look like their demo. Clean inputs. Well formed questions. The happy path. That tells you the system works when everything goes right. Which is exactly the condition that never happens in production.
A real reliability test set has to include the inputs that actually break systems. Vague questions, the kind a busy clinician really types between patients. Questions where two legitimate sources disagree. Questions about a population the literature barely covers. Inputs with missing context. Questions sitting right on the edge of what the system should even be willing to answer.
And for each one, you need a validated correct answer. Not “the right document.” The actual right answer, in the form your system would give it, written or checked by someone who has the domain knowledge to know. This is the expensive part. This is the part teams skip. And skipping it is exactly why they have no ground truth to measure against later.
A few hundred good cases beat tens of thousands of clean ones. The value lives in the cases that poke at the edges, not the ones that confirm the middle. I learned this writing test scenarios years ago, long before AI. You sit a few people in a room, everybody writes scenarios for the same feature, you rotate and build on each other’s, and in an hour you have fifty or a hundred real scenarios nobody would have thought of alone. Same trick works here. The edge cases are sitting in the heads of your clinical users. Go get them.
Measure interpretation, not similarity
Once you have a test set, the instinct is to score answers by how semantically similar they are to the validated answer. Don’t. Semantic similarity is the same broken signal that caused the problem in the first place. Two answers can be very similar and clinically opposite, because “X is recommended” and “X is not recommended” are one word apart and a whole world apart.
What you actually want to measure is a set of specific properties.
Faithfulness. Does every claim in the answer trace back to something in the retrieved sources, or did the model add stuff that was not there.
Qualifier preservation. Did the answer keep the clinical hedges. “May help in selected patients” cannot become “helps.”
Population fit. Does the answer apply to the population the question asked about, or did it quietly import guidance from a different group.
Conflict handling. When the sources disagreed, did the system surface the disagreement or just pick one and move on.
Refusal calibration. On the questions where the right move was to decline or escalate, did it.
You can score these with domain experts on a smaller set, and with a carefully validated model based evaluator at scale. The key word is validated. If you use a model to grade your model, you have to keep checking the grader against expert judgment. Otherwise you are just measuring one model’s bias with another model’s bias and calling it a number.
Test for reproducibility, not just correctness
Here is the part that separates a real reliability process from a one time eval.
In normal software, a passing test means the behavior is locked. Run it again, same result. That is the whole contract. LLM systems break that contract. The same input can give you a different output across runs, across model versions, across a tiny prompt change. An answer that was right yesterday can drift wrong after a model update you did not even control.
So reliability is not a one time measurement. It is a regression suite you run over and over. Every model version. Every prompt change. Every retrieval change. You are watching two things at once. Is the system correct today. And did it stay correct since the last change.
Reproducibility is also what makes a failure fixable. When a clinician reports a bad answer, you need to rebuild exactly what the system retrieved, how it ranked, what it synthesized, and why. If you cannot reproduce it, you cannot fix it. And you definitely cannot prove to a hospital that you fixed it. This is impact analysis, plain and simple. When something changes, what else might it have broken, and how do you know. Old discipline, new technology.
Watch behavior over time
Last piece. Drift.
A clinical AI system is not static. The model changes. Your corpus grows. Usage shifts as more clinicians adopt it and start asking questions you never planned for. A system that was reliable at launch can rot quietly over months without a single line of code changing, because the inputs changed and the model changed underneath you.
So reliability is a thing you monitor, not a box you check at launch. You need production telemetry watching the same things your test suite watches. Refusal rates. Conflict surfacing. Faithfulness on a sample of live traffic. The mix of question types compared to what you tested against. When the live mix drifts away from your test mix, that is your early warning that your test set no longer looks like reality and needs to grow.
What this gets you
A team that does all this has something most clinical AI teams simply cannot produce. They can say, with evidence, how often the system fails, in which specific ways, under which conditions, and what they did to reduce each one. They can reproduce any failure. They can prove a fix held. They can show a hospital security team a reliability process, not just a claim.
That is the difference between “our AI is accurate,” which every vendor says and no buyer believes anymore, and “here is exactly how our system behaves under pressure, including the places it still has limits.” The second one builds trust. The first one is noise.
The work is not glamorous. It does not make the demo prettier. Nobody claps for a regression suite. But it is the work that decides whether your product survives contact with a real hospital, and most teams are not doing it because nobody told them they had to. This is the layer underneath the broader shift I wrote about recently — the model was never the hard part, and the reliability process is most of what “the part around the model” actually means.
If you are building clinical AI and you do not have a reliability process like this yet, that gap is usually the first thing I map when I look at a system. Happy to walk through what it would look like for yours.