Human-centered evaluation of data visualization understanding in vision-language models

arXiv:2505.17202 · cs.HC, cs.CL, cs.CV · Submitted 2025-05-22 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Human-centered evaluation of data visualization understanding in vision-language models".

Jane: Here's the gist of the paper. The gist The core finding is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations, and that this gap is especially pronounced for assessments that were developed to measure these skills in humans.

Tom: Data visualizations are powerful tools for communicating patterns in quantitative data They do so by leveraging a combination of visual features (e.g., color, shape, size, position) and text-based annotations (e.g., axis labels, legends) to efficiently convey patterns in quantitative data.

Jane: Interpreting any data visualization relies upon the ability to correctly combine visual, linguistic, and quantitative information to answer some question at hand.

Tom: The acquisition of data visualization literacy — a robust ability to parse data visualizations and derive insights from them — has been a longstanding priority in STEM education.

Paper summary: Tom: So, we’re looking at this paper today, "Human-centered evaluation of data visualization understanding in vision-language models." Basically, the main point is that these vision language models are promising for modeling how humans understand data visualizations, but we need to know if they actually mimic human behavior on those tasks.

Jane: That sounds like a big question because right now, we evaluate these AI systems mostly by just looking at their accuracy scores. This paper argues that accuracy alone isn't enough; we have to compare the models directly against how people actually solve the problems.

Lu: It sets up a human-centered evaluation protocol using eight different vision language models across six specific assessments designed for humans, which they call CHART-six <ref:2505.17202#pg1>. The core finding they highlight is that these vision language models performed worse than humans on average, and this gap even stayed there even when we used criteria that were pretty lenient.

Meng: Wait, so it’s not just about whether the AI got the right answer on a test, but whether its *pattern* of errors looks like what a human makes? That seems like a necessary step if we want to use these models as scientific cognitive models.

Lalam: Yes, Lalam agrees. The paper found that while relative performance across different items was somewhat similar between the models and humans, all the error patterns they produced were reliably different from those of human participants.

Tom: That’s a key distinction there—even if they get some things right in terms of accuracy, their internal reasoning process is fundamentally different from a human's. What did we learn about these vision language models from this study?

Jane: The paper shows that current state-of-the-art vision language models consistently underperform humans on reasoning tasks involving data visualizations, and this performance gap persisted even under lenient criteria to assess model performance.

Lu: They used a suite of six tests, CHART-six which included five established tests of human visualization literacy and one test specifically designed for evaluating these models <ref:2505.17202#pg1>. These tests cover a wide array of graph types, from bar plots to treemaps and pie charts.

Meng: So they tested them on something like GGR and HOLF-Multi which have quite a few items, spanning thirteen to two hundred sixteen different plot types across those tests. That’s a lot of complexity for one set of evaluations.

Lalam: Right, they used those various graph types to test the models against human performance on tasks like retrieving values or finding trends and correlations across different plots.

Tom: And when we looked at the results, humans consistently outperformed GPT-4V on most types of graphs, regardless of whether the item came from a multiple choice or numerical response test <ref:2505.17202#pg2>.

Paper summary: Jane: That’s interesting because it suggests that even though these models are advanced, they aren't yet matching human skill in interpreting visual data patterns. The paper also noted that humans outperformed GPT-4V across all three task types: value identification, arithmetic computation, and statistical inference <ref:2505.17202#pg2>.

Lu: What’s really compelling is the analysis of error patterns. They constructed two versions of an error pattern vector for each model—one from valid responses and one including all responses—and found that these vectors were reliably distinct from those produced by humans.

Meng: That’s where the real insight for deployment comes in, because if we can reliably distinguish the error patterns, it means we can actually tell when an AI is making a mistake in a way that aligns with human reasoning versus just being randomly wrong.

Lalam: And they found that even when focusing only on items for which models did produce valid responses, they still achieved reliably lower performance than the adult human participants represented in this work.

Tom: So, to wrap up what we’ve heard about this paper, the authors are suggesting that while vision language models show promise as a class of models that can reason over a broad class of visualization and question types, there are still significant opportunities to improve their alignment with human behavior.

Jane: They point to the gap between these models and human participants on several of the tests in CHART-six including GGR, CALVI and HOLF, as evidence for the need to use a diversity of independently designed measures when identifying opportunities for further model development <ref:2505.17202#pg1>.

Lu: The researchers envision this CHART-six protocol being used to track the progress of artificial systems towards achieving human-like behavior on tasks involving data visualizations <ref:2505.17202#pg1>. This suggests a path forward by creating controlled experimentation against real-world tasks involving complex, naturalistic inputs.

Meng: From an engineering standpoint, it sounds like they are providing a concrete benchmark for what success looks like when training these models for scientific reasoning rather than just pattern matching. It tells us where the current limitations of the architectures are showing up most clearly.

Lalam: And they noted that while proprietary systems like GPT-4V were most aligned with human behavior among the models in their suite, open-source models such as LLaVA1 point 6-Yi-34B and Pix2Struct-0 point 3B did not necessarily lag far behind in terms of error patterns <ref:2505.17202#pg2>.

Tom: It’s clear that this CHART-six evaluation protocol is a tool for cognitive AI benchmarking, revealing gaps between humans and AI systems on common tasks involving complex inputs like data visualizations <ref:2505.17202#pg1>.

Jane: So, the implication for us is that if we want to build tools that actually help scientists communicate and analyze data better, we need to focus on developing those models toward more human-like reasoning rather than just maximizing raw accuracy scores.

Conclusion: Tom: So we’re wrapping up our look at this paper, "Human-centered evaluation of data visualization understanding in vision-language models." The main point is that these vision language models are promising for modeling how humans understand data visualizations, but we need to know if they actually mimic human behavior on those tasks.

Jane: That’s the central tension here. We’ve seen a lot of accuracy scores for these AI systems, but this study pushes back on that by asking if the answers match what people actually produce when they try to visualize data.

Lu: Yeah, the authors set up this human-centered evaluation protocol using eight different vision language models across six specific tests called CHART-six. It’s a bunch of assessments designed to cover a wide variety of graph types and understanding skills that humans need.

Meng: And what they found is pretty clear—these vision language models performed worse than humans on average across those tasks, and that performance gap stayed there even when they used lenient criteria to judge the models.

Lalam: What’s really interesting is how they looked at the errors. They didn't just look at whether an answer was right; they compared the pattern of mistakes made by the AI to what humans make, and those patterns were reliably different.

Tom: So if we take away the idea that models are just trying to get a high accuracy number, this suggests that even if an AI gets some things right, its internal reasoning process is fundamentally different from a human's way of thinking about charts and graphs.

Jane: Exactly. The implication for someone who only listens to this is that we can’t just build these models based on how well they pass a standard test; we have to look at the actual *process* they use to reason about the data.

Lu: The authors are suggesting that there’s a real opportunity here to develop artificial systems that actually serve as useful models of how humans reason about data visualizations. It points toward a need for better alignment with human behavior in these complex reasoning tasks.

Meng: That means if we want to use these AI systems in real-world tools or education, we can’t just rely on the current benchmarks; we need to test them against this kind of human comparison protocol.

Lalam: And they even noted that while some models were closer to human behavior than others, the error patterns were still consistently distinct. It shows that it’s not a one-size-fits-all issue across all vision language models.

Tom: So the overall message is that there are significant opportunities to improve these systems by focusing on how they reason rather than just their final score on a test. This leads us into what this means for building better tools for STEM learning environments, which we’ll discuss next.

Arnav Verma, Kushin Mukherjee, Christopher Potts, Elisa Kreiss, Judith E. Fan

Stanford University · University of California, Los Angeles

cs.HC, cs.CL, cs.CV

Submitted: 2025-05-22

Updated: 2026-10-20

Journal ref: Scientific Reports (2026)

DOI: 10.1038/s41598-026-68119-z

Code: https://github.com/cogtoolslab/davinci_public2023

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The core finding is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations, and that this gap is especially

Key concepts

Data Visualization Understanding: This is the ability to correctly combine visual elements (like colors and shapes), linguistic information (like labels), and numerical data to derive insights from a chart or graph. It requires parsing complex inputs into meaningful conclusions, which is crucial for interpreting quantitative patterns.
Vision-Language Models (VLMs): These are advanced AI models that can process both images (vision) and text simultaneously. The paper tests if these models can effectively model the complex reasoning required to understand how visual data representations relate to written descriptions, mimicking human understanding.
CHART-6 Suite: This is a comprehensive set of six distinct, human-centered assessments designed to test different aspects of data visualization literacy. It includes tests for identifying patterns in various graph types, handling misleading visualizations, and performing arithmetic on plots.

Terminology

Vision-language models — 3× in this paper · explained in 2 episodes

Summary

The gist The core finding is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations, and that this gap is especially pronounced for assessments that were developed to measure these skills in humans.

Introduction and Motivation

Data visualizations are powerful tools for communicating patterns in quantitative data They do so by leveraging a combination of visual features (e.g., color, shape, size, position) and text-based annotations (e.g., axis labels, legends) to efficiently convey patterns in quantitative data. Interpreting any data visualization relies upon the ability to correctly combine visual, linguistic, and quantitative information to answer some question at hand. The acquisition of data visualization literacy — a robust ability to parse data visualizations and derive insights from them — has been a longstanding priority in STEM education. There is a need for computational models that can contend with the diversity of real-world visualizations and are adaptable to common visualization understanding tasks.

Methodology: CHART-6 Suite

The researchers developed CHART-6 (Comparative Human-AI Graphical Reasoning Tests), a human-centered suite of data visualization understanding assessments. This test suite spans a wide array of different approaches to designing such assessments, ensuring broad coverage of the skills that are considered to be important when assessing human data visualization literacy. The suite includes six tests:

GGR (GGR is a 13-item test containing three bar plots, three line plots, an icon array, and a pie chart).
VLAT (The Visualization Literacy Assessment Test (VLAT) is a 53-item test containing 12 graph types).
CALVI (The Critical Thinking Assessment for Literacy in Visualizations (CALVI) is a 60-item test focusing on the ability to compensate for misleadingly constructed data visualizations).
HOLF (HOLF is a 384-item test containing 64 bar plots procedurally generated from eight real-world datasets).
HOLF-Multi (HOLF-Multi is a 216-item extension of HOLF containing 72 bar, line, and scatter plots).
ChartQA-Human (ChartQA-Human ChartQA [56] is a data visualization understanding benchmark containing plots obtained from various web sources such as Statista and Pew Research).

Evaluation Protocol for Models

The evaluation protocol involved presenting each test item 10 times to every model, yielding a total of 8,510 responses per model. To assess the performance of vision-language models against human behavior, the researchers employed a three-step process:

Process responses
Validate responses
Analyze responses.

For processing model output, general instructions describing the nature of the task were prepended to each question and all prompts were formatted to match the model-specific prompts used during training. Model output was then processed independently by a different large-language model to extract the response in the correct format. For numerical-response items, any strings prefixed or suffixed to the floating-point value were removed.

Key Findings on Performance Gaps

The core finding is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations.

Even when focusing on items for which models did produce valid responses, we found that they still achieved reliably lower performance than did the adult human participants represented in this work.

The researchers analyzed accuracy using two metrics:

Accuracy when considering only valid responses (upper bound).
Accuracy when considering all responses, including invalid ones, where invalid responses were marked as incorrect (lower bound).

For multiple-choice tests (GGR, VLAT, and CALVI), the difference between model and human performance was generally smaller when considering only valid responses. However, for numerical-response tests (HOLF, HOLF-multi, and ChartQA), the gap widened substantially when all responses were considered.

Variation Across Tasks and Graphs

The analysis showed that Humans consistently outperformed GPT-4V on most types of graphs regardless of whether an item came from a multiple-choice or numerical-response test. However, the kinds of graphs on which Humans did well also tended to be the ones on which GPT-4V also performed well. Furthermore, Humans outperformed GPT-4V across all three task types (value identification, arithmetic computation, statistical inference).

Error Pattern Comparison

When scrutinizing the patterns of performance from models and humans more comprehensively, current vision-language models generate error patterns that are reliably distinguished from those produced by humans. The researchers constructed two versions of an error-pattern vector for each model—one derived from valid responses and one including all responses—and computed how correlated these vectors were between humans and all models. This analysis suggests that when scrutinizing the patterns of performance more comprehensively, current vision-language models generate error patterns that are reliably distinguished from those produced by humans.

Conclusion

The findings suggest that while vision-language models show promise as a class of models that can reason over a broad class of visualization and question types, there remain opportunities to improve their alignment with human behavior. The gap between these models and human participants on several of the tests in the suite, including GGR, CALVI and HOLF, suggests the value of using a diversity of independently designed measures for identifying opportunities for further model development. The researchers envision CHART-6 being used to track the progress of artificial systems towards achieving human-like behavior on tasks involving data visualizations. The findings suggest that while proprietary systems like GPT-4V was most aligned with human behavior among the models in our suite, open-source models such as LLaVA1.6-Yi-34B and Pix2Struct-0.3B did not necessarily lag far behind. The results contribute to a growing body of cognitive-AI benchmarking efforts that employ large-scale controlled experimentation to reveal gaps between humans and AI systems on a common set of real-world tasks involving complex, naturalistic inputs. The researchers believe it to be likely that AI systems displaying more human-like understanding of visual, linguistic, and mathematical concepts could be used to design more effective STEM learning environments and tools to support scientific communication.

--- Page 1 ---

CHART-6: Human-Centered Evaluation of Data Visualization Understanding

Understanding in Vision-Language Models

Arnav Verma, Kushin Mukherjee, Christopher Potts, Elisa Kreiss, and Judith E. Fan Abstract— Data visualizations are powerful tools for communicating patterns in quantitative data. Yet understanding any data visualization is no small feat — succeeding requires jointly making sense of visual, numerical, and linguistic inputs arranged in a conventionalized format one has previously learned to parse. Recently developed vision-language models are, in principle, promising candidates for developing computational models of these cognitive operations. However, it is currently unclear to what degree these models emulate human behavior on tasks that involve reasoning about data visualizations. This gap reflects limitations in prior work that has evaluated data visualization understanding in artificial systems using measures that differ from those typically used to assess these abilities in humans. Here we evaluated eight vision-language models on six data visualization literacy assessments designed for humans and compared model responses to those of human participants. We found that these models performed worse than human participants on average, and this performance gap persisted even when using relatively lenient criteria to assess model performance. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced patterns of errors that were reliably distinct from those produced by human participants. Taken together, these findings suggest significant opportunities for further development of artificial systems that might serve as useful models of how humans reason about data visualizations. Index Terms—chart understanding, graph comprehension, artificial intelligence, visualization literacy, cognitive-AI benchmarking 1 INTRODUCTION Humans can engage with a wide range of visual input modalities, ranging from natural scenes and drawings to diagrams and data visualizations [22, 26, 79]. Data visualizations — also commonly known as graphs, charts, and/or plots — are indispensable tools for supporting exploratory analysis and statistical reasoning [8, 20, 78]. They do so by leveraging a combination of visual features (e.g., color, shape, size, position) and text-based annotations (e.g., axis labels, legends) to efficiently convey patterns in quantitative data [5,61,77,81]. As such, interpreting any data visualization relies upon the ability to correctly combine visual, linguistic, and quantitative information to answer some question at hand (Figure 1). Moreover, the acquisition of data visualization literacy — a robust ability to parse data visualizations and derive insights from them [8, 10, 21, 24, 27, 28, 73] — has been a longstanding priority in STEM education [16].

Improvements for AI systems

Automated response validation for vision-language models can be improved by employing a multi-stage processing pipeline, as described in Fig 3, which involves extracting the answer from a model's raw output using GPT-4 and then applying specific rules based on the question type (e.g., requiring an exact match for multiple-choice or a single floating-point value for numerical responses).
Model fine-tuning should be focused on improving visual property detection, as current models often fail to detect visual properties that are generally salient to humans, such as intersections between lines, overlap between shapes, and the number of simple visual elements in a scene (Page 9). This improvement would enable the system to better perform tasks like reading beyond the data or identifying anomalies.
Evaluation protocols should incorporate comparison across different assessment strategies, as the paper suggests that when we considered only valid responses generated by models, the model-human performance gap narrowed considerably, indicating that a single metric is insufficient for robust evaluation.

Abstract

Data visualization understanding, which requires integrating visual, numerical, and linguistic inputs, poses fundamental challenges to multimodal reasoning. Recently developed vision-language models (VLMs) are promising candidates for modeling the computations that underlie human data visualization understanding. However, whether they are useful as scientific models depends not only on whether they produce correct answers, but also whether they approximate the patterns of responses that people produce on the same tasks. This distinction has been ignored in recent evaluations of VLMs, which typically only report overall model performance (accuracy) without direct comparison to human behavioral patterns. Here we introduce a human-centered evaluation protocol that compares VLM and human responses on the same set of visualization understanding items. We apply this protocol to eight VLMs across six assessments: five established tests of human visualization literacy, plus one designed specifically for evaluating VLMs. Models performed worse than humans on average, and this gap persisted even under lenient criteria. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced error patterns that were reliably distinct from those of humans. We envision our evaluation protocol helping to identify artificial systems that might serve as promising scientific cognitive models of how humans reason about data visualizations.