Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs

arXiv:2502.16435 · cs.CV, cs.CL · Submitted 2025-02-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs".

Jane: Here's the gist of the paper. The paper introduces VISFACTOR, a novel benchmark, and evaluates it across 39 frontier models to establish clear performance boundaries for genuine visual reasoning.

Tom: Here is the detailed synthesis of the research: Core Thesis: The Foundational Gap in MLLM Visual Cognition The central argument of this work is that current MLLMs bypass the necessary bottom-up cognitive hierarchy that humans utilize to develop perception—a hierarchy spanning from basic primitives and Gestalt principles up to high-level semantics.

Jane: In contrast, MLLMs are trained directly on complex downstream tasks, effectively skipping the foundational visual capabilities required for genuine human-like visual cognition.

Tom: This deficiency is framed as a critical risk for safety-critical applications (e.g., robotics, medical imaging) where brittle spatial reasoning can lead to catastrophic hallucinations.

Title and authors: Jane: So we know the paper calls this work "Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs." What does that title tell us about the core concern?

Tom: It tells us that there’s a gap between what humans can actually do visually and what these models are capable of when it comes to reasoning. It’s not just about accuracy on a task, but about the underlying cognitive process.

Lu: The paper sets up this investigation by noting that human cognition is structured in a bottom-up hierarchy, starting from basic primitives and Gestalt principles up to high-level semantics. In contrast, MLLMs are trained directly on complex downstream tasks, which often bypass those foundational visual capabilities>

Jane: It sounds like they’re saying that if you skip the basics, you can’t really build the advanced understanding that humans have>

Tom: That’s right. The paper introduces VISFACTOR to measure this gap by testing models across four domains: visualization and spatial reasoning, perceptual and closure, memory, and reasoning>

Meng: Four distinct areas of visual intelligence. I wonder how rigorously they set up these tests so we know it’s not just a guessing game for the model.

Lalam: The paper says they engineered VISFACTOR to reduce the overall chance-level accuracy down to only two point nine percent, which is meant to make sure that any score you see is really reflecting true reasoning ability> <ref:2502.16435#pg1>

Jane: That low chance level is important because it prevents models from just succeeding by guessing randomly on the test items, which is a major problem with some existing benchmarks>

Tom: Right. And they made sure the test battery has high internal consistency, achieving a Cronbach’s alpha of zero point nine four, meaning all those subtests are actually measuring that shared visual language capability> <ref:2502.16435#pg1>

The paper's summary: Jane: So we know they built this benchmark to check the foundational gap. Can you give us a clearer picture of what VISFACTOR actually tests in practice?

Lu: VISFACTOR is based on the Factor-Referenced Cognitive Test, or FRCT, which uses twenty vision-centric subtests <ref:2502.16435#pg1>. These include things like Hidden Figures Tests, Gestalt Completion Tests for objects and words, and memory tests for pictures and maps>

Tom: It covers visualization and spatial reasoning—seeing how things rotate or relate to each other—and perceptual closure, which is just recognizing what’s missing in a pattern>

Meng: And then they have memory tests, like the Shape Memory Test or Map Memory Test, which are about holding visual information in your head after seeing it>

Lalam: Plus, there’s a reasoning domain where they look at diagramming relationships between different objects like desks and pencils to see if the model can map those connections>

Jane: So it’s covering everything from simple object recognition to complex spatial manipulation and even memory recall of visual scenes>

Tom: The key thing is that this isn't just one big test; they use factor analysis to break down the performance across these different cognitive domains, giving a fine-grained profile of the model’s strengths and weaknesses>

Lu: And they also designed algorithms to automatically construct and validate unlimited test cases with controllable difficulty, which means they can tweak things like rotation angles or grid sizes>

Meng: That part about automated test case generation is interesting because it means we don't have to manually create every single hard visual puzzle. It lets us generate a lot of data quickly.

The paper's improvements: Jane: So the paper isn’t just stopping at building this benchmark; they suggest ways to actually use it better or fix the models with it. What are the suggested improvements for this work?

Lu: One big suggestion is to use this benchmark to train models specifically on VISFACTOR so they can achieve genuine mastery of human-like visual cognition, instead of just surface-level performance on downstream tasks>

Tom: That means moving toward training approaches that start with these foundational visual primitives before tackling the more complex semantic stuff>

Meng: I’d also be interested in how they handle the cost. If we want to train models on this, generating data through their automated test case construction sounds like it would have some computational overhead>

Lalam: They suggest using parametric generation for test cases so you can create unlimited test cases with controllable difficulty by adjusting parameters like rotation angle or grid size>

Jane: That sounds like they are trying to solve the problem of benchmark saturation by letting us generate more varied, controlled challenges>

Lu: Another idea is developing an automatic evaluation protocol that reduces the chance-level baseline to two point eight nine percent by using methods like decomposed multiple choice and symmetry variants> <ref:2502.16435#pg1>

Tom: So they’re not just saying "this model failed"; they are proposing a way to measure failure more reliably, making sure the score reflects real reasoning ability>

Meng: And I think that’s smart because if the baseline is too high, you just get inflated scores from lucky guesses in the evaluation pipeline.

Conclusion: Jane: So we’re coming to the end of this discussion on "Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs." To wrap up, what are the main implications for the AI research community?

Tom: The core implication is that current performance gains on general visual benchmarks might not actually mean these models have achieved genuine visual mastery. We’re seeing evidence of a structural mismatch between how we train them and what humans use to perceive things>

Lu: They conclude that the best model they evaluated only achieved a score of fifty-four point zero percent, which is significantly lower than human performance, implying that existing methods are not capturing true cognitive ability>

Meng: From my side, I worry about deploying models based on these scores if they aren't truly robust across all those domains, especially when we look at real-world traffic where things can get messy>

Lalam: They end by saying the research provides a definitive map of MLLM limitations and advocates for shifting development toward bottom-up hierarchical training approaches, where models first master basic visual primitives before tackling complex semantic reasoning>

Jane: So it’s a call to action: we need to change how we train these systems fundamentally, focusing on building that foundational visual understanding first>

Tom: It’s a map of what we’re missing. We need to start training models on the visual basics before expecting them to handle the big picture reasoning correctly.

Jen-tse Huang, Dasen Dai, Jen-yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, *Wenxiang Jiao*, *Pinjia He*, *Zhaopeng Tu*, *Haodong Duan*

CUHK University · PKU University · CUHK-KSZ Institute of Science and Technology Center of Science and Technology · RUC Institute of Science and Technology Center

cs.CV, cs.CL

Submitted: 2025-02-23

Updated: 2026-10-21

Comments: Accepted to NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The paper introduces VISFACTOR, a novel benchmark, and evaluates it across 39 frontier models to establish clear performance boundaries for genuine visual reasoning.

Key concepts

VISFACTOR: A novel benchmark digitizing 20 vision tests from the FRCT. It is designed to measure foundational visual intelligence across four domains: Spatial Reasoning, Perceptual Closure, Memory, and Reasoning. It is engineered to be highly sensitive to true visual ability.
Foundational Visual Gaps: The central argument that current multimodal models bypass the necessary cognitive hierarchy humans use for perception. MLLMs skip basic visual primitives and Gestalt principles, leading to a deficit in genuine spatial reasoning required for safety-critical tasks.
Chance-Level Accuracy: A metric used in VISFACTOR to ensure performance scores reflect actual reasoning rather than luck. The benchmark is set with a very low chance level (2.9%) across all subtests, making high scores highly indicative of true cognitive mastery.
Spatial Orientation Bias: A specific architectural weakness found in models where they consistently misclassify angles as 45-degree approximations instead of perceiving continuous angular relationships. This shows models lack fine-grained spatial perception.

Terminology

Benchmark — 4× in this paper · explained in 2 episodes
MLLMs — 3× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes

Summary

The paper introduces VISFACTOR, a novel benchmark, and evaluates it across 39 frontier models to establish clear performance boundaries for genuine visual reasoning.

Here is the detailed synthesis of the research:

Core Thesis: The Foundational Gap in MLLM Visual Cognition

The central argument of this work is that current MLLMs bypass the necessary bottom-up cognitive hierarchy that humans utilize to develop perception—a hierarchy spanning from basic primitives and Gestalt principles up to high-level semantics. In contrast, MLLMs are trained directly on complex downstream tasks, effectively skipping the foundational visual capabilities required for genuine human-like visual cognition. This deficiency is framed as a critical risk for safety-critical applications (e.g., robotics, medical imaging) where brittle spatial reasoning can lead to catastrophic hallucinations.

The Benchmark: VISFACTOR

The authors introduce VISFACTOR, the first benchmark to systematically digitize 20 vision-centric subtests derived from the well-established Factor-Referenced Cognitive Test (FRCT). This benchmark is specifically designed to measure foundational visual intelligence, ensuring that any observed success reflects genuine reasoning rather than mere chance.

Structure and Rigor of VISFACTOR:

Cognitive Domains: VISFACTOR spans four critical cognitive domains: (1) Visualization and Spatial Reasoning (S), (2) Perceptual and Closure (CS), (3) Memory (MV/MA), and (4) Reasoning (I/RL).
Chance-Level Accuracy: The benchmark is engineered to reduce the overall chance-level accuracy to a very low 2.9%, ensuring that performance metrics are highly sensitive indicators of true visual reasoning ability.
Internal Consistency: The test battery exhibits exceptional reliability, achieving a Cronbach’s alpha coefficient of 0.94 (or 0.9392 in one excerpt), confirming that the subtests collectively measure a shared, underlying general visual-language capability rather than isolated artifacts.
Validity: VISFACTOR demonstrates strong convergent validity, showing high positive correlations (r about 0.81) with comprehensive multi-modal benchmarks like MMMU-Pro and structurally similar tasks such as VisualPuzzles (r = 0.93). Crucially, it exhibits excellent discriminant validity, yielding near-zero or negative correlations with orthogonal domains like OCRBench-v2 (r = 0.02) and MathVista (r = -0.49), proving its unique contribution to the evaluation landscape.

Evaluation Methodology

The study evaluates 39 frontier MLLMs, encompassing both proprietary models (e.g., GPT, Gemini) and open-source models (e.g., LLaMA, Qwen). The evaluation is conducted in a zero-shot setting, omitting few-shot prompting to prevent model misdirection, as the input context already contains a substantial number of images. To manage computational costs associated with evaluating all 39 models comprehensively, the authors constrain their evaluation to a single run per model.

Key Findings and Anomalies

The results reveal a striking gap between MLLM performance and human capability:

Overall Performance: The best model evaluated achieved a score of only 54.0%, significantly lower than human performance (which is implied to be near 78.8% for comparable tasks).
Task-Specific Failures: Models consistently fail on tasks that require genuine, foundational visual understanding, specifically mental rotation, spatial relation inference, and figure–ground discrimination, regardless of model size or prompting strategy.
The Middle Score Anomaly: The authors expose a critical phenomenon where models achieve intermediate performance on certain tasks (like the Identical Pictures Test/P3), suggesting a lack of genuine reasoning capabilities—they are neither random nor near-perfect. This is supported by analysis showing that high-performing examples typically involve easily verbalizable content, while failures are associated with visually complex and linguistically demanding patterns.
Prompting Inefficacy: Chain-of-Thought (CoT) prompting yields only marginal performance improvements for the GPT-4 family and can sometimes decrease accuracy on perceptual and closure tasks. This highlights a structural mismatch: text-mediated reasoning in current MLLMs does not effectively compensate for the lack of intuitive, bottom-up spatial reasoning present in humans.
Architectural Vulnerabilities: The analysis reveals specific architectural weaknesses:
Spatial Orientation Bias: Models consistently misclassify various directions as 45-degree angles, indicating only coarse categorical representations rather than continuous angular perception.
CoT Degradation: Longer CoT sequences can degrade performance on perceptual and closure tasks (P3, CS2), suggesting that the sequential traversal forced by text reasoning interferes with holistic spatial judgment. Conversely, CoT consistently improves performance on reasoning tasks like I3 and RL2, which are more related to linguistic understanding.

Conclusion and Implications

The collective evidence from VISFACTOR strongly suggests that performance gains observed on existing general benchmarks are likely castles in the air rather than evidence of genuine mastery over human-like visual cognition. The benchmark successfully isolates distinct cognitive capabilities through fine-grained factor analysis, confirming that VISFACTOR is not just a homogeneous difficulty variant but a set of diagnostic probes capable of isolating specific architectural vulnerabilities.

The authors conclude with a clear call to action: the research provides a definitive map of MLLM limitations and advocates for shifting development toward bottom-up hierarchical training approaches, where models first master basic visual primitives before tackling complex semantic reasoning.

Improvements for AI systems

Establish foundational visual reasoning capabilities by training models specifically on VISFACTOR, which digitizes 20 vision-centric subtests from FRCT to reveal foundational visual gaps. This allows AI systems to achieve genuine mastery of human-like visual cognition rather than superficial performance on downstream tasks alone.
Implement parametric generation for test cases, enabling the creation of unlimited test cases with controllable difficulty by modulating parameters like rotation angle and grid size. This provides robust performance tracking without benchmark saturation and ensures that model improvements are genuinely reflective of cognitive mastery.
Develop an automatic evaluation protocol that reduces chance-level baseline to 2.89% by using Decomposed multiple choice, Grouped-consistency items, and Symmetry variants. This prevents inflated scores from lucky guesses, ensuring measured accuracy reflects actual reasoning ability rather than random guessing.
Introduce reasoning effort modulation via Chain-of-Thought (CoT) prompting, noting that CoT improves performance on tasks like I3 and RL2 while potentially hurting others. This allows AI systems to leverage extended reasoning chains when tackling complex logical synthesis, while avoiding the Verbal Thinking Hurts Performance effect observed in certain visual tasks.
Integrate a mechanism to detect the Middle Score Anomaly by analyzing performance on simple yet human-easy tasks like P3 (Identical Pictures Test). This exposes a missing substrate for genuine visual reasoning, guiding research toward models that possess true cognitive processing rather than just concept-level recognition.
Fine-tune model architectures using data generated by the parametric generator to enhance specific weaknesses, such as length, angle, and proportion assessment in tasks like VZ1 and VZ3. This addresses limitations where models struggle with accurately gauging line lengths and proportions or misclassify angles as 45-degree approximations.
Utilize factor analysis to isolate distinct cognitive capabilities by tracking performance across the ten latent factors (CF, CS, I, MA, MV, P, RL, S, SS, VZ). This provides a fine-grained diagnostic profile of model strengths and weaknesses that goes beyond holistic scores.