Same Illness, Different Voice: Restating the Patient's Account Narrows Narrative Anchoring in Clinical Language Models
cs.CL, cs.AI
Submitted: 2026-07-29
Updated: 2026-10-13
Code: https://github.com/prabhjotschugh/narrativeshield-sdoh-medqa
License: http://creativecommons.org/licenses/by/4.0/
The gist: Clinical language models are evaluated on vignettes written in a clinician's voice, but patients describe the same illness in their own.
Terminology
Abstract
Clinical language models are evaluated on vignettes written in a clinician's voice, but patients describe the same illness in their own. We pair 1,000 MedQA cases with three retellings that carry the same audited original facts, in a clinical, a socioeconomic and a cultural voice, and evaluate 12 open models (3B to 24B, including two medical fine-tunes). Every model is less accurate when the patient speaks in a marked voice: 26% more errors, a pooled penalty of-9.6 accuracy points (95% CI [-11.0, -8.2]), with no detectable reduction at larger scale. The penalty is not explained by any retelling property we measured (length, omitted age, MedQA split, added clinical content) and persists when the marked narrative contains the original vignette verbatim; because generation settings differ between voices, we read it as sensitivity to these marked retellings, not proof that register alone is the cause. Chain-of-thought and explicit debiasing instructions together remove only about a fifth of the gap. Having the same model first rewrite the patient's account as a clinical vignette, then answer from the rewrite, halves the gap while raising marked-voice accuracy by 3.9 points (means over the 11 models that can run the pipeline). A matched pipeline using a structured schema instead of prose closes the same gap but is 7.1 points less accurate in all 11 models, and adding reasoning agents or tools costs more still. Rewriting is not free: it recovers more facts than extraction but also invents plausible details, which we quantify. In our experiments, the representation a model reasons over matters more than how much it reasons.
Sources
- The Llama 3 Herd of Models
- Mistral 7B
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Qwen2.5 Technical Report
- Gemma 3 Technical Report
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering