BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models

The gist The authors introduce BEAR-Bench, a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents.

BEAR-Bench Introduction and Scope

BEAR-Bench is designed to address limitations in existing multimodal Large Language Models (MLLMs) by evaluating their ability to reason about text-dense, professional documents in both English and Russian languages The benchmark comprises 1000 human-annotated questions targeting text-dense enterprise and scientific documents The science section tests the ability to interpret academic figures, transcribe mathematical formulas, and analyze data from plots The enterprise tasks target data crossreferencing in financial reports and logical reasoning over business charts The questions follow two design principles: they require multiple logical steps rather than direct extraction from a single piece of evidence, and they are fully answerable from the document alone, without external expert knowledge The inclusion of Russian-language tasks further broadens evaluation beyond predominantly English- and Chinese-language resources

Domain Taxonomy and Data Collection

The benchmark spans two primary domains, the Business Domain and the Science Domain, each subdivided into thematically coherent sub-categories The Business Domain covers three document types: financial reports (SEC Forms 10-K and 10-Q) requiring tabular reasoning and year-over-year calculations; investor presentations (Form 8-K exhibits) combining charts, KPI tiles, and infographic maps; and flowcharts and organisational diagrams depicting corporate ownership structures and process pipelines The Science Domain covers three categories: mathematical and physical formulae from physics and mathematics preprints targeting symbol-level recognition; scientific figures and plots (line plots, scatter diagrams, heatmaps) requiring axis and legend interpretation; and academic layouts with multi-column pages testing readingorder resolution and cross-referential reasoning Source collection for the Business Domain involved retrieving English documents from the U.S. Securities and Exchange Commission (SEC) EDGAR system, supplementary English documents from official government portals, and Russian-language documents from state corporate-disclosure platforms e-disclosure.ru and moex.com For the Science Domain, English papers were downloaded from arXiv via its official Python API across four STEM categories, while Russian articles were collected from CyberLeninka using an asynchronous Playwright-based crawler Raw PDFs underwent a two-stage filtering pipeline: Stage 1 involved Visual Content Classification using Gemini 2.5 Pro to obtain silver labels for content types like contains diagrams and contains equations, which trained a lightweight classifier Stage 2 retained only pages at or above the 67th percentile of OCR character count within their respective language group

Annotation Pipeline and Quality Control

Human annotation was conducted by 13 domain experts, with every annotator processing its own subset of images and authoring exactly one question–answer pair per image Annotators were required to re-verify visual-content labels produced by the automatic classifier and compose a multi-step, multi-hop question with a detailed ground-truth answer Questions were required to elicit compositional reasoning, such as aggregating values across table rows or interpreting plotted trends in the context of equations Model responses were scored by GPT-4o used as an LLM-as-a-judge, which assesses semantic equivalence and returns a binary verdict v ∈ [true, false] with a brief explanation in structured XML tags Quality control involved querying eight state-of-the-art proprietary VLMs on every item, flagging items where three or more models returned identical responses and the LLM judge assigned false to all answers After quality-control filtering, BEAR-Bench comprises 1,000 document images paired with 1,000 human-authored QA instances across four domain–language groups

Experimental Evaluation and Results

The evaluation involved 16 MLLMs, including proprietary models like Gemini 3.1 Pro and open-weight models such as Qwen3.5-397B All models were evaluated zero-shot under a fixed protocol: each instance received only the image and the raw question, with no few-shot examples or prompt engineering, and a uniform decoding temperature of 0.6 The main results show that Qwen3.5-397B-A17B and Gemini 3.1 Pro achieve the highest overall accuracy (75.4% and 75.1%, respectively), indicating clear headroom even for the strongest systems A marked drop from English to Russian is observed across all evaluated models, confirming that the linguistic gap persists even for frontier proprietary models Explicit chain-of-thought (CoT) prompting improved accuracy for some open-weight models, such as Qwen3-VL-8B-Instruct, which improved by 13.7 percentage points The effect of image resolution was critical, as accuracy decreased sharply from 49.3% at the original resolution to 12.6% at a downsampling factor of c=3

Error Analysis and Hallucination Detection

An exploratory error analysis clustered incorrect responses into five final categories: spatial misgrounding (C1), counting/aggregation (C2), OCR/visual-attribute (C3), chart-value extraction (C4), and semantic/reasoning errors (C5) Visualperceptual errors, specifically spatial misgrounding and OCR/visual-attribute errors, dominate the failures on BEARBench To compare existing hallucination detection methods, the study assessed token-level uncertainty scores, representation-based detectors, supervised hidden-state probes (SUQ), and MLLM-as-a-judge methods across two deployment regimes The SUQ Probe achieved the highest Balanced Accuracy on all eight proprietary models (BalAcc 0.67–0.74), while the VLM-as-judge baseline achieved a high BalAcc of 0.

1. Bold header: Integration of BEAR-Bench for Robust Reasoning Evaluation

This allows researchers to evaluate how often models fail on BEAR-Bench but also how reliably those failures can be identified by comparing token-level uncertainty scores, representation-based detectors, supervised hidden-state probes, and MLLM-as-a-judge methods.

2. Bold header: Enhanced Hallucination Detection Capabilities

The system can now compare existing detection methods under different access regimes (direct access to internal signals for openweight models and proxy-based detection for proprietary ones), providing a practical comparison for deployment settings with and without access to model internals.

3. Bold header: Targeted Error Mode Mitigation

By analyzing the error taxonomy, systems can be improved specifically against spatial misgrounding (C1), counting/aggregation (C2), OCR/visual-attribute (C3), chart-value extraction (C4), and semantic/reasoning (C5) errors, focusing on improving perceptual accuracy and logical consistency.

While Multimodal Large Language Models (MLLMs) have made significant strides in visual comprehension, their ability to reason about text-dense, professional documents remains incompletely evaluated. Existing benchmarks emphasize information extraction, require external domain knowledge, or cover professional documents only as one of many settings. They are also largely English- or Chinese-centric, leaving other languages and Russian, in particular, substantially underrepresented. To address these limitations, we introduce BEAR-Bench (Bilingual Enterprise and Academic Reasoning), a self-contained, complex English-and-Russian benchmark comprising 1000 human-annotated questions based on text-rich business and scientific documents. We evaluate 16 proprietary and open-weight MLLMs, including Gemini 3.1 Pro and Qwen3.5-397B, on BEAR-Bench and observe clear headroom even for the strongest systems. Finally, we use the resulting model outputs to compare existing hallucination detection methods, evaluating not only how often models fail on BEAR-Bench but also how reliably those failures can be identified.