Health-Tech
How AI Is Changing Clinical Decision Support
By Miracle Olajuyigbe 10 min read
3:40 in the afternoon
A nurse practitioner is finishing notes on the fourth patient of a busy afternoon when a banner appears across the chart: elevated sepsis risk, score 82, recommend lactate and blood cultures.
The patient is a 68-year-old woman admitted for a hip fracture. Her heart rate has been running around 104, up from 88 that morning. Her temperature is 37.8. Her white count came back mildly raised. Those three inputs are most of what drove the score.
She is also in pain, has had nothing for it in three hours, and is anxious about surgery in the morning. Pain and anxiety raise heart rate. Post-operative and post-injury patients run mild fevers for reasons that have nothing to do with infection. She has no cough, her wound looks clean, her urine is clear, and she is fully oriented and irritated about the noise on the ward.
The clinician has, realistically, twenty seconds to decide. Order the workup, which costs a couple of hundred dollars, sticks a needle in an anxious patient, and might delay surgery if something equivocal comes back. Or dismiss the alert, which is what she did with the last eleven sepsis alerts this week, all of which were noise.
She dismisses it.
Here is the part that should worry anyone building these systems. Nothing in that paragraph tells you whether she was right. Both outcomes are plausible: the patient is fine, or the patient is at hour two of a urinary source that will declare itself by midnight. The alert has not resolved the ambiguity. It has added a data point of unknown weight and moved the burden of judgment back onto a person who has less time to exercise it than the system’s designers assumed.
That is the actual state of AI in clinical decision support. Not a robot doctor. A stream of probabilistic suggestions arriving into a workflow with no slack in it.
What clinical decision support does today
Strip out the marketing and current systems do a handful of specific things.
Risk stratification. Models that assign a probability of an event within some window: sepsis, deterioration requiring ICU transfer, readmission within 30 days, no-show at an appointment, falls. These are the most widely deployed and the most variable in quality.
Pattern detection in images and signals. Diabetic retinopathy screening from fundus photographs, nodule detection on chest imaging, large vessel occlusion flagging on CT angiography, arrhythmia detection from ECG and wearable data. This is where the technology is strongest, because the task is narrow, the input is standardised, and the ground truth is comparatively clean.
Documentation and information retrieval. Ambient scribes generating draft notes from a recorded consultation, summarising a chart, surfacing the relevant prior result. Not a diagnostic aid at all, but plausibly the highest-value near-term application, because it gives time back and time is the constraint everything else runs into.
Rules-based safety checks. Drug interaction warnings, allergy flags, dose limits. Mostly not AI, mostly decades old, and important to this discussion because it is where we already learned what happens when clinicians receive more alerts than they can attend to.
Triage and routing. Sorting inbound messages, prioritising worklists, deciding which imaging study gets read next. Quietly among the most useful applications, and the least discussed, because reordering a queue does not make a compelling demo.
Where it genuinely helps
The strongest case for these tools is not that they are smarter than clinicians. It is that they do not get tired, do not get distracted at 4 a.m., and do not skim.
Sensitivity at volume is a real advantage. A radiologist reading her hundredth study of the day has measurably different performance than on the tenth. A model reads the hundredth exactly as it read the first. For small findings that are easy to miss and consequential when missed, that consistency is worth a lot, and second-reader arrangements in mammography and lung nodule detection have shown it.
Screening programmes benefit differently: they extend reach where specialists do not exist. Autonomous diabetic retinopathy screening from retinal photographs is the clearest case. In a district where the nearest ophthalmologist is a day’s travel away and the screening rate is near zero, the comparison is not model versus specialist. It is model versus nothing, and I would take the model. That framing gets lost in debates held in countries with a surplus of specialists.
Time-critical pathways benefit for reasons that have little to do with diagnostic cleverness. Software that spots a suspected large vessel occlusion and pages the stroke team while the radiologist is still opening the study is mostly compressing a communication delay. Unglamorous, and it saves brain tissue.
On the documentation side, the honest win is simpler still: a clinician looking at a patient rather than a keyboard is a better clinician.
Where it fails
The base rate problem, which produces the alert flood. This is arithmetic, not a bug, and it is poorly understood by people who have not deployed these systems.
Suppose a condition occurs in 2% of admitted patients and your model has 80% sensitivity and 90% specificity, which would be respectable performance. In a thousand admissions, twenty patients have the condition; you catch sixteen. Of the 980 who do not, you flag 98. So of 114 alerts, 98 are wrong. Most of what a clinician sees is a false positive, and that is with good numbers.
Real numbers have been worse. When researchers externally validated a widely deployed proprietary sepsis prediction model across a large health system, publishing in JAMA Internal Medicine in 2021, it achieved an area under the curve of 0.63, missed roughly two thirds of sepsis cases, and generated alerts on about 18% of all hospitalised patients. The tool was live in hundreds of hospitals. Clinicians had been told it worked.
Alert fatigue, which is the predictable consequence. Studies of medication alerting have consistently found override rates between roughly 50% and over 90%. Clinicians are not being reckless. They are responding rationally to a signal that has proven unreliable, and the adaptation happens fast and generalises. Once a clinician has learned to dismiss your alerts, they dismiss the true ones at the same speed as the false ones.
This is the specific danger, and it deserves to be stated without hedging: a poorly calibrated alerting system can make care worse than no alerting system at all, because it trains people to ignore a channel that then also carries the important message. That is not a hypothetical risk to be managed. It is the default outcome of shipping a noisy model into a busy ward.
Bias in training data. A model learns the population it was trained on, including that population’s inequities.
Pulse oximetry is the cleanest illustration, and it is not even AI. Analysis published in the New England Journal of Medicine in 2020 found that occult hypoxaemia, arterial oxygen saturation below 88% despite a pulse oximeter reading of 92% to 96%, was nearly three times more common in Black patients than in white patients. Now consider that oxygen saturation feeds deterioration scores, sepsis models, and triage algorithms across essentially every hospital system. The bias is not in the model. It is in the sensor, and the model inherits it, launders it through a number, and hands it to a clinician as an objective risk score.
Dermatology models trained overwhelmingly on lighter skin perform worse on darker skin. Models trained on utilisation data learn that patients who historically consumed less care were less sick, when they may simply have had less access. Deploy that in a different population and it will be confidently wrong in a direction that compounds an existing disadvantage.
Distribution shift. A model validated at an academic centre in Boston is not validated at a district hospital in Ibadan, or in Boise. Different case mix, different documentation habits, different lab platforms, different thresholds for admission. Performance degrades, sometimes sharply, and often silently. A model that has quietly stopped working looks exactly like a model that is working, because the output format has not changed.
Automation bias, running the other way. The mirror image of alert fatigue is over-trust. Where a model has a good reputation, clinicians defer to it, including in cases where their own assessment was correct. Trainees are particularly susceptible, which has an unpleasant long-term implication: if the model’s judgment substitutes for the reasoning that builds clinical skill, we degrade the very expertise the oversight model depends on.
Why “the model said so” is not enough
Explainability gets discussed as a technical property. In practice it is a workflow requirement, and the reason is simple: a clinician cannot integrate a number they cannot interrogate.
When a colleague tells me they are worried about a patient, I ask what is worrying them. The answer, the wound looks worse than yesterday, she is not herself, tells me what to weigh it against and how it interacts with everything else I know. A score of 82 offers nothing to weigh. I cannot tell whether it is 82 because of the heart rate, which I have already explained with pain, or because of something I have not noticed.
That is why the alert in the opening scenario was ignored, and why a version showing the three contributing variables would have fared differently. Not because she would have trusted it more, but because she could have checked its reasoning against hers in the twenty seconds available.
There is a liability dimension too. When a clinician accepts a recommendation they cannot explain and the outcome is bad, they own it. Vendors structure their agreements so the tool is decision support and the clinician is the decision maker, which is defensible only if the clinician was equipped to evaluate the recommendation. Where the interface makes real evaluation impossible, that arrangement transfers risk to the person least able to manage it.
Human oversight, argued properly
“Keep a human in the loop” has become a slogan that ends discussions instead of starting them. It is worth asking what the human is meant to be doing, and whether the conditions for doing it exist.
Real oversight requires four things, and every one of them costs money.
Time. Reviewing a recommendation means reconstructing enough of the clinical picture to have an independent opinion. That is minutes, not seconds. If a clinician receives forty alerts a shift and has ten seconds each, oversight has not been implemented. It has been asserted.
Training in what the tool actually is. Not a demo. Clinicians need to know what the model was trained on, its performance in a population like theirs, its known failure modes, and what it does not look at. Most clinicians using these systems today cannot answer any of those questions about the tools in their chart. That is a procurement and implementation failure, not a personal one.
Interface design that supports scrutiny. Contributing factors visible without a click. Confidence communicated plainly. Dismissal that takes one action and acceptance that takes a deliberate one, rather than the reverse. Any design where accepting is easier than examining will produce acceptance, and the resulting agreement rate will be reported as validation.
Authority to override without penalty. If overriding requires a written justification and disagreement is audited against outcomes, the rational strategy is to accept everything. You will get compliance and call it oversight.
Without those four, what you get is the rubber stamp: a clinician clicking approve because the queue is long, the alert is probably wrong, and disagreeing is expensive. The signature exists. The judgment does not. And because the audit trail records a human decision, the system produces documentation of oversight that never happened, which is worse than having no human in the loop at all, because at least then everyone would know.
What good implementation looks like
- Validate locally before go-live. Run the model silently against your own historical data and measure sensitivity, specificity, and positive predictive value in your population. Vendor-reported performance is a starting hypothesis.
- Compute the expected alert volume per clinician per shift and get agreement from the people who will receive them that the number is workable. If it is not, tune the threshold before launch, not after complaints.
- Set the threshold against the cost of the recommended action. A flag prompting a low-cost, low-harm step tolerates false positives. A flag prompting an invasive intervention does not.
- Show the contributing factors in the alert itself. Top three drivers, direction of each, and what the model does not consider.
- Track override rate as a primary safety metric from day one. A rising override rate is your system losing credibility, and it is measurable weeks before anyone files an incident report.
- Monitor for drift on a schedule and define in advance what performance level triggers a pause. Deployment is not the end of validation.
- Audit performance by subgroup, including by race, sex, age, language, and insurance status. If you are not measuring differential performance, you are not going to notice it.
- Give clinicians a one-click way to flag a wrong alert, route it to someone with authority to change the threshold, and close the loop visibly. Silence teaches people that reporting is pointless.
- Name an accountable clinical owner, not a committee. Someone whose job includes the question of whether this tool is still helping.
The uncomfortable part
The failures that matter in clinical AI are mostly not model failures. They are implementation failures dressed up as model successes.
A vendor can ship an accurate model and still make care worse, if the alert volume is untenable, the interface discourages scrutiny, and nobody is watching the override rate. Every one of those is a decision made by people, on a roadmap, usually with a deadline.
So the question worth putting to anyone building or buying these systems is not whether the model is good. It is this: in the workflow you have actually built, what is the shortest path to dismissing this recommendation, and what is the shortest path to examining it? If dismissal is faster, and it usually is, then you have not built decision support. You have built a system that generates recommendations and a record of someone having glanced at them.
That gap is where patients get hurt, and it will not be closed by a better model.