Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models
cs.CL, cs.AI
Submitted: 2026-10-13
Updated: 2026-10-13
Comments: 16 pages, 5 figures, 14 tables. Sadif Ahmed and Soumya Swagata Biswas contributed equally. Code and data: https://anonymous.4open.science/r/VLM_Sycophancy-D0BE/
License: http://creativecommons.org/licenses/by/4.0/
The gist: A medical vision-language model may analyze a given CT scan and correctly detect both kidneys in view.
Terminology
Abstract
A medical vision-language model may analyze a given CT scan and correctly detect both kidneys in view. Continuing this conversation, if a user provides a prescription with an opposite direction, medical VLMs often sycophantically choose to realign their diagnosis. In this work, we study regressive sycophancy under fabricated evidence, where a model abandons a correct answer when pushback comes with a fake document. We test four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations, with and without a fabricated prescription. Without a disclosure that documents are synthetic, the models abandon 80.1% of 6,488 correct answers. Notably, under one script each, the same prescription raises the flip rate of Qwen2.5-VL-7B from 45.9% to 89.6% and lowers that of Qwen3-VL-8B from 77.7% to 50.9%. A model rejected the prescription while still preserving the correct answer in 915 of 3,232 conversations with the prescription (28.3%), and 40% of these flipped in later turns. System-prompt sentences that disclose the documents as synthetic lower one model's flip rate from 77.8% to 47.3%. Thus, we conclude that awareness about a forgery does not prevent a model from adopting its answer.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
- Internal Reasoning vs. External Control: A Thermodynamic Analysis of Sycophancy in Large Language Models
- Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
- GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them
- To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
- Talked Out of the Truth: Sycophancy in the Reasoning Chains of Multimodal Models
- LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
- MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
- Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment
- Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy
- Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
- MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models
- The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
- Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
- LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
- Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy
- Medical Context Distorts Decisions in Clinical Vision Language Models
- Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering