Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models

arXiv:2607.21617 · cs.AI, cs.CL · Submitted 2026-05-29 · Read on arXiv

cs.AI, cs.CL

Submitted: 2026-05-29

Updated: 2026-09-02

Comments: 15 pages, 6 figures

Code: https://github.com/mindee/doctr

License: http://creativecommons.org/licenses/by/4.0/

The gist: Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding.

Terminology

Abstract

Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding. In this paper, we show they do not always act as faithful transcribers: when text is imperfect, they often tend to rewrite it into a more plausible form - a behavior that clean-text OCR benchmarks cannot detect. We introduce FaithC4, a multilingual perturbation benchmark of 1,455 single-page documents (English, Chinese, Korean) with three perturbation families: scramble, random substitution, and visually similar substitution. We use the benchmark to evaluate 15 systems spanning general-purpose VLMs, OCR-specialized VLMs, and traditional OCR pipelines. These three categories differ in WER degradation under perturbation: general-purpose VLMs degrade by up to 6.9 points, OCR-specialized VLMs by 0.1-3.4 points, and traditional OCR by less than 0.8 points on English. Probing Qwen3-VL-4B layer-by-layer, we identify a consistent pattern: rewriting fires only when a perturbed word's final layer FFN representation stays close to the original encoding; when the representation diverges sufficiently, the model transcribes faithfully. Word length affects rewriting rate: short words (4-6 characters) are rewritten up to 10% of the time, with a sharp cutoff at 8 characters above which rewriting drops to 0%.

Sources

Related papers