AI hallucinations in healthcare: risks and safeguards

An AI hallucination is output that reads as clinically plausible but has no basis in the source material: an examination documented that never happened, a dose nobody mentioned, a guideline citation that does not exist. In clinical documentation the measured rates are low but never zero — one evaluation of 12,999 clinician-annotated sentences reported 1.47%, while a blinded specialty review flagged hallucinations in 31% of ambient-scribe notes. Both studies were run by scribe vendors on their own products, so read every published rate as provisional. The number that matters is not the rate anyway. It is whether the error reaches the signed record.

This guide is written for clinicians who already use — or are about to use — an AI scribe, a large language model or any generative tool near patient care. It covers what a hallucination actually is, why these systems produce them by design rather than by accident, what the published rates are and why they disagree so wildly, the specific slots in a note where fabrications hide, and the review habits that catch them before you sign.

What is an AI hallucination in healthcare?

A hallucination is AI-generated content that appears plausible but has no basis in reality — that is the definition used in a 2025 commentary on AI scribe risks in npj Digital Medicine. In a clinical setting that means the model asserts something that was never said, never found and never done, in the register of a competent colleague. The fluency is the problem: a fabricated sentence in a clinical note does not look like a bug. It looks like a note.

"Hallucination" is often used loosely to mean any AI error, which muddies the review task. The same commentary separates four distinct failure modes in ambient documentation, and they are worth keeping apart because each is caught differently:

• Fabrication (true hallucination) — entirely fictitious content, such as documenting an examination that never occurred or a diagnosis that does not exist. Caught by asking "did this happen?"
• Omission — something clinically important that was discussed but is absent from the note: a symptom, a concern, an assessment finding. Caught by asking "what is missing?", which is a separate sweep and an easier one to skip.
• Misinterpretation — a context-dependent statement misconstrued into the wrong treatment, medication or care plan. Caught by reading for meaning rather than for form.
• Speaker attribution error — a patient's statement recorded as the clinician's, or the reverse. Caught by asking "who said this?"

There is also a layer below the language model. Ambient tools transcribe before they summarise, and the transcription step hallucinates too. In Careless Whisper, presented at the ACM FAccT conference in 2024, Koenecke and colleagues found that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences that existed nowhere in the audio, and that 38% of those hallucinations carried explicit harms. Critically, they occurred more often in speech with longer non-vocal durations — a common feature of aphasia, and of any patient who pauses, struggles or speaks slowly. The failure is not evenly distributed across your list.

Why do language models hallucinate at all?

Because they are built and scored to produce an answer rather than to decline one. Kalai and colleagues set out the argument in Nature in 2026, under a title that states the conclusion: evaluating large language models for accuracy incentivises hallucinations. Next-word pretraining, they show, creates statistical pressure towards hallucination even with idealised, error-free training data — and the dominant headline metric, accuracy, then systematically rewards guessing over admitting uncertainty. Uncertainty is penalised; confident invention is not.

One detail of that analysis matters more than the rest for clinical work. The errors the authors describe as unavoidable are those involving facts that lack repeated support in the training data — one-off details — while recurring regularities such as grammar are learned reliably. Generalising from pretraining to summarising a specific consultation is an inference rather than their finding, but it points the right way: the structure of a note is a regularity a model learns well, and the particulars of your patient are the opposite of one. The fluent scaffolding is the part the model is good at.

Roustan and Bastardot make the clinical case directly in a clinicians' guide to large language models in the Interactive Journal of Medical Research, attributing hallucinations to both training-dataset factors and the auto-regressive nature of these models — generating each token from the ones before it. Their warning goes further than inaccurate output: they flag the reinforcement of flawed diagnostic reasoning pathways, which is a subtler harm than a wrong number in a note. Two consequences follow for practice, and both are structural rather than fixable by better prompting. A model has no internal signal separating "this was in the consultation" from "this is what usually appears here". And the pressure to complete a plausible document is strongest exactly where the source material is thinnest: a brief encounter, a quiet patient, a template section with nothing to put in it. A generative tool asked to fill a structured note will fill it.

How often do AI tools hallucinate in clinical work?

Nobody knows, and the honest answer has to start there. Published rates run from roughly 1% to over 90%, the spread is mostly definitional rather than real, and the two studies closest to everyday ambient documentation were both produced by companies selling ambient documentation. Read the unit of analysis and the author affiliations before you read the number.

• 1.47% of sentences — vendor study. Asgari and colleagues evaluated 18 experimental configurations of an LLM clinical-note pipeline across 12,999 clinician-annotated sentences and reported a 1.47% hallucination rate alongside a 3.45% omission rate, concluding that refined prompts and workflows brought major errors below previously reported human note-taking rates (npj Digital Medicine, 2025). Every author is affiliated with an AI scribe company, the pipeline evaluated is its own, and its own clinicians did the annotating — so the figure describes a product tuned by the people measuring it, under conditions they chose. The paper discloses this. Treat 1.47% as a best case achieved in-house, not a rate you should expect.
• 31% of notes, against 20% for physicians — also a vendor study. Palm and colleagues took 97 real visits across general medicine, paediatrics, obstetrics and gynaecology, orthopaedics and adult cardiology, generated an ambient note and a physician-drafted "gold" note for each, and had two blinded specialty reviewers score all 194 notes. Hallucinations were detected in 20% of gold notes and 31% of ambient notes, p = 0.01; ambient notes scored higher on thoroughness and lower on succinctness, and reviewers preferred them 47% to 39% (Frontiers in Artificial Intelligence, 2025). The blinding and the human comparator are real methodological strengths. The affiliation is the same problem: the product assessed is the sponsor's, across five specialties and fewer than a hundred visits.
• Roughly 1–3% overall error rates for modern ambient scribes, against a typical 7–11% for older automated speech-recognition dictation, per the npj Digital Medicine commentary — which also notes that in randomised trials human scribes were more than four times as likely as self-documentation to produce notes physicians rated accurate.
• 28.6–91.4% for generated references. Chelli and colleagues asked GPT-3.5, GPT-4 and Bard to reproduce the reference lists of 11 published systematic reviews, analysing 471 references. Counting a paper as hallucinated when any two of title, first author or year were wrong, rates were 39.6% for GPT-3.5, 28.6% for GPT-4 and 91.4% for Bard (Journal of Medical Internet Research, 2024).

The missing row in that table is an independent one. The npj Digital Medicine commentary's first recommendation is precisely that: independent evaluation against standardised metrics for accuracy, completeness and time saved, required before widespread implementation rather than after it. Until that exists, the sensible posture is not to pick a favourite number but to treat all of them as provisional — informative about what is achievable under good conditions, uninformative about what your tool will do in your clinic, with your patients and your accents. The one rate that should govern your own review habits is the one you measure yourself.

Within those limits, two readings are defensible at once: ambient documentation is now accurate enough to be clinically useful, and not accurate enough to be left unreviewed. The 20% figure for physician-written notes is the most interesting number in the set, and the one to handle most carefully — it is a single vendor-affiliated study of 97 visits across five specialties, so it is a hint rather than a base rate. If it replicates independently, it means unsupported content in the record is not a problem AI introduced, and safeguards aimed only at the AI will miss most of it. What AI plainly does change is the volume, the speed and the plausibility of the output. For how these rates are measured and what accuracy claims actually mean, see our deeper look at how accurate AI medical scribes are.

What does a hallucination look like in a clinical note?

Rarely like nonsense. It looks like the sentence a thorough clinician would have written, which is why skimming for oddity does not work. The npj Digital Medicine commentary cites documented harms from the earlier speech-recognition era that show the shape of the risk: a transcript recording "no vascular flow" instead of "normal vascular flow", prompting an unnecessary procedure, and a confused tumour location that led to surgery on the wrong site. Single words, clinically inverted, in otherwise unremarkable prose.

The following patterns are illustrative — composites of the failure modes described in the literature, not transcripts of real incidents — but they map onto where experienced reviewers find problems:

• The completed examination. The consultation covered a cough; the note records "chest clear to auscultation bilaterally, no added sounds" because that is what follows a respiratory history in most notes. Nothing was auscultated.
• The confident negative. "Denies chest pain, no red-flag features" appears in a note where red flags were never asked about. A fabricated negative is more dangerous than a fabricated positive: it closes a question instead of opening one.
• The inverted modifier. "No" and "normal", "left" and "right", "increase" and "reduce" — words that invert clinical meaning while leaving the sentence grammatical.
• The plausible dose. A medication discussed without a dose acquires a standard one. The number is not wrong in general; it is simply not what was agreed.
• The attributed concern. The patient's worry about a relative's cancer appears as the clinician's assessment of malignancy risk.
• The invented citation. Asked to justify a plan, a general-purpose model supplies a reference with a real journal, real-sounding authors and a DOI that resolves to nothing — the failure Chelli and colleagues quantified.

Why a 1% hallucination rate is not a 1% risk

Because the clinician reviewing the draft is not a neutral detector. Automation bias — the tendency to over-rely on automated output — has been studied in healthcare for decades. Goddard, Roudsari and Wyatt screened 13,821 papers and reviewed 74 in a systematic review for the Journal of the American Medical Informatics Association, finding the effect mediated by trust and confidence in the system, and by workload, task complexity and time constraint — the three conditions that describe a full clinic. Mitigators they identified are notably mundane: training, explicit user accountability, and design choices such as attaching confidence levels to output and presenting information rather than a recommendation.

The sharper finding for note review comes from Lyell and Coiera, who reviewed 40 studies and reported that automation bias appeared in single tasks with high verification complexity, and more often in diagnosis than in monitoring (JAMIA, 2017). Translated into documentation: the statements you are least likely to challenge are the ones that would cost you the most effort to check. A fabricated examination finding is cheap to query if you remember the encounter and expensive if it is the ninth note of the afternoon — so that is precisely the sentence that survives review. The World Health Organization made the same point in its January 2024 guidance on large multi-modal models in health, warning both of documented risks of false, inaccurate, biased or incomplete statements and of these tools encouraging automation bias among clinicians and patients.

There is a workload trap on top of it. The commentary cites a study in which AI scribes saved only 34 seconds per note, with wide individual variability — and warns that organisations may raise patient-volume expectations on the promise of efficiency, leaving clinicians with more encounters and the added cognitive burden of auditing generated text. Review time is not free, and a safeguard that assumes unlimited review time is not a safeguard. Our piece on the hidden cost of clinical documentation covers the underlying arithmetic.

How do you catch a hallucination before you sign the note?

By reviewing against the encounter rather than against plausibility, and by checking the high-risk slots first instead of reading top to bottom. The question is never "does this read correctly?" but "do I specifically recall this happening?" Anything you cannot place should be deleted or confirmed, not kept because it sounds right.

• Check the slots where fabrication does the most damage, in this order: medications and doses, allergies, laterality, negations, numeric values, examination findings, and the follow-up or safety-netting instructions the patient will act on.
• Run omissions as a second, separate pass. Fabrications announce themselves as text; omissions are invisible by nature. The sentence-level evidence found omissions at more than double the hallucination rate.
• Verify every negative you did not personally establish. "Denies", "no", "unremarkable" and "within normal limits" are the highest-yield words to interrogate.
• Use source traceability where the tool offers it. Notes that link each generated statement back to the transcript segment it came from turn an expensive verification into a cheap one — which, given the verification-complexity finding, is the single highest-leverage product feature.
• Never let a generated citation, guideline reference or statistic stand unchecked. Open it. A DOI that does not resolve is the cheapest hallucination to catch and the most embarrassing to publish.
• Review while you still remember. The value of your memory of the consultation decays through the session; batching every note to the end of the day is the condition under which automation bias is strongest.
• Keep your own error log for a fortnight. Vendors report aggregate rates; the pattern that matters is which failures your tool makes in your specialty, with your patients and your accents.

Shifaa AI is built on this premise rather than around it. It is an AI clinical assistant, currently in private beta, designed as decision support: it drafts a structured note from the consultation for the clinician to review, edit and sign (AI medical scribe), and it surfaces differentials and considerations for the clinician to weigh (differential diagnosis support). It does not diagnose and it does not sign anything. The broader framing is in our guide to AI in healthcare, and if you want to tighten documentation structure before any software is involved, the free SOAP note template is a paper-first place to start.

What should you ask a vendor about hallucinations?

Ask for numbers with their definitions attached, and for the governance that sits behind them. A vendor quoting a hallucination rate without stating the unit of analysis, the annotator and the specialty mix is quoting a marketing figure. The npj Digital Medicine commentary calls for independent validation using standardised metrics, mandated transparency on how systems work and regular reporting of error rates, clear regulatory frameworks defining accountability when errors occur, institutional protocols and training in auditing AI-generated content, and dedicated funding for independent long-term research.

• What is your hallucination rate, measured per sentence or per note, and who annotated it — your clinicians or independent ones?
• Has any evaluation of this product been published independently of the company?
• How does the note let me trace a statement back to the transcript that produced it?
• Does the tool ever tell me it is unsure — flagging a low-confidence statement rather than presenting everything in the same flat, confident register?
• What are your error rates across accents, languages and speech patterns, and have you tested patients with limited fluency in the consultation language? The speech-recognition disparities documented in the literature are systematic, not incidental.
• Is the audio or transcript retained, for how long, and is any of it used to train models?
• Where does liability sit when a generated error reaches the record — and what does your contract say about it?
• Is the tool classified as a medical device in the jurisdictions you sell into? Many ambient scribes are positioned as administrative rather than clinical tools, which keeps them outside device oversight despite their effect on the record.

Governance is moving in the same direction. WHO's 2024 guidance recommends that governments assign a regulatory agency to assess and approve large multi-modal models for health use, and introduce mandatory post-release auditing and impact assessments when such a model is deployed at scale, with results published and broken down by user groups including age, race and disability. For where generative tools sit inside the wider decision-support landscape, see our explainer on clinical decision support systems and the honest answer to whether AI can diagnose patients.

Frequently asked questions

What is an AI hallucination in simple terms?

Content the AI generated that looks plausible but has no basis in the source material — a finding, dose, event or citation that was never there. It is a fabrication rather than a mistake in reasoning, which is why it cannot be spotted by judging whether the text sounds sensible.

How common are hallucinations in AI-generated clinical notes?

Nobody reliably knows yet, and it depends on what is counted. A sentence-level evaluation of nearly 13,000 clinician-annotated sentences reported 1.47%; a blinded note-level review found hallucinations in 31% of ambient notes, and in 20% of the physician-written notes used as its comparator. Both studies were authored by AI scribe companies assessing their own products, and no equivalent independent evaluation has been published, so these are provisional figures rather than established rates.

Can AI hallucinations be eliminated?

Not with current architectures. Prompt design, retrieval from verified sources, narrower tasks and source-linked output all reduce the rate substantially, but systems trained and scored to produce an answer rather than abstain will sometimes produce one without support. Detection is advancing — work published in Nature uses uncertainty estimated at the level of meaning to flag when an answer is likely to be fabricated — but it flags risk rather than removing it. Clinician review remains the control, not a formality.

Who is responsible if a hallucination reaches the patient record?

In practice, the clinician who signed it — they attest to the record. Professional bodies have called for civil liability frameworks to be updated to clarify accountability for AI-related harms, and the position differs by jurisdiction, so check your own regulator's guidance and your indemnity terms rather than assuming the vendor carries the risk.

References

• Topaz M, Peltonen LM, Zhang Z. Beyond human ears: navigating the uncharted risks of AI scribes in clinical practice. npj Digital Medicine, 2025;8:569. nature.com/articles/s41746-025-01895-6
• Asgari E, Montaña-Brown N, Dubois M, Khalil S, Balloch J, Yeung JA, Pimenta D. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine, 2025;8:274. nature.com/articles/s41746-025-01670-7
• Palm E, Manikantan A, Mahal H, Belwadi SS, Pepin ME. Assessing the quality of AI-generated clinical notes: validated evaluation of a large language model ambient scribe. Frontiers in Artificial Intelligence, 2025;8:1691499. frontiersin.org
• Koenecke A, Choi ASG, Mei KX, Schellmann H, Sloane M. Careless Whisper: Speech-to-Text Hallucination Harms. ACM Conference on Fairness, Accountability, and Transparency, 2024. arxiv.org/abs/2402.08021
• Chelli M, Descamps J, Lavoué V, Trojani C, Azar M, Deckert M, Raynier JL, Clowez G, Boileau P, Ruetsch-Chelli C. Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews. Journal of Medical Internet Research, 2024;26:e53164. pmc.ncbi.nlm.nih.gov/articles/PMC11153973
• Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 2012;19:121-127. pmc.ncbi.nlm.nih.gov/articles/PMC3240751
• Lyell D, Coiera E. Automation bias and verification complexity: a systematic review. Journal of the American Medical Informatics Association, 2017;24:423-431. pmc.ncbi.nlm.nih.gov/articles/PMC7651899
• World Health Organization. WHO releases AI ethics and governance guidance for large multi-modal models, 18 January 2024. who.int
• Kalai AT, Nachum O, Vempala SS, Zhang E. Evaluating large language models for accuracy incentivizes hallucinations. Nature, 2026;653:1047-1051. nature.com/articles/s41586-026-10549-w
• Roustan D, Bastardot F. The clinicians' guide to large language models: a general perspective with a focus on hallucinations. Interactive Journal of Medical Research, 2025;14:e59823. pmc.ncbi.nlm.nih.gov/articles/PMC11815294
• Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature, 2024;630:625-630. nature.com/articles/s41586-024-07421-0