Human-centered evaluation of data visualization understanding in vision-language models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Human-centered evaluation of data visualization understanding in vision-language models".

Jane: Here's the gist of the paper. The gist The core finding is that current state-of-the-art vision-language models consistently underperform humans on reasoning tasks involving data visualizations, and that this gap is especially pronounced for assessments that were developed to measure these skills in humans.

Tom: Data visualizations are powerful tools for communicating patterns in quantitative data They do so by leveraging a combination of visual features (e.g., color, shape, size, position) and text-based annotations (e.g., axis labels, legends) to efficiently convey patterns in quantitative data.

Jane: Interpreting any data visualization relies upon the ability to correctly combine visual, linguistic, and quantitative information to answer some question at hand.

Tom: The acquisition of data visualization literacy — a robust ability to parse data visualizations and derive insights from them — has been a longstanding priority in STEM education.

Paper summary: Tom: So, we’re looking at this paper today, "Human-centered evaluation of data visualization understanding in vision-language models." Basically, the main point is that these vision language models are promising for modeling how humans understand data visualizations, but we need to know if they actually mimic human behavior on those tasks.

Jane: That sounds like a big question because right now, we evaluate these AI systems mostly by just looking at their accuracy scores. This paper argues that accuracy alone isn't enough; we have to compare the models directly against how people actually solve the problems.

Lu: It sets up a human-centered evaluation protocol using eight different vision language models across six specific assessments designed for humans, which they call CHART-six <ref:2505.17202#pg1>. The core finding they highlight is that these vision language models performed worse than humans on average, and this gap even stayed there even when we used criteria that were pretty lenient.

Meng: Wait, so it’s not just about whether the AI got the right answer on a test, but whether its *pattern* of errors looks like what a human makes? That seems like a necessary step if we want to use these models as scientific cognitive models.

Lalam: Yes, Lalam agrees. The paper found that while relative performance across different items was somewhat similar between the models and humans, all the error patterns they produced were reliably different from those of human participants.

Tom: That’s a key distinction there—even if they get some things right in terms of accuracy, their internal reasoning process is fundamentally different from a human's. What did we learn about these vision language models from this study?

Jane: The paper shows that current state-of-the-art vision language models consistently underperform humans on reasoning tasks involving data visualizations, and this performance gap persisted even under lenient criteria to assess model performance.

Lu: They used a suite of six tests, CHART-six which included five established tests of human visualization literacy and one test specifically designed for evaluating these models <ref:2505.17202#pg1>. These tests cover a wide array of graph types, from bar plots to treemaps and pie charts.

Meng: So they tested them on something like GGR and HOLF-Multi which have quite a few items, spanning thirteen to two hundred sixteen different plot types across those tests. That’s a lot of complexity for one set of evaluations.

Lalam: Right, they used those various graph types to test the models against human performance on tasks like retrieving values or finding trends and correlations across different plots.

Tom: And when we looked at the results, humans consistently outperformed GPT-4V on most types of graphs, regardless of whether the item came from a multiple choice or numerical response test <ref:2505.17202#pg2>.

Paper summary: Jane: That’s interesting because it suggests that even though these models are advanced, they aren't yet matching human skill in interpreting visual data patterns. The paper also noted that humans outperformed GPT-4V across all three task types: value identification, arithmetic computation, and statistical inference <ref:2505.17202#pg2>.

Lu: What’s really compelling is the analysis of error patterns. They constructed two versions of an error pattern vector for each model—one from valid responses and one including all responses—and found that these vectors were reliably distinct from those produced by humans.

Meng: That’s where the real insight for deployment comes in, because if we can reliably distinguish the error patterns, it means we can actually tell when an AI is making a mistake in a way that aligns with human reasoning versus just being randomly wrong.

Lalam: And they found that even when focusing only on items for which models did produce valid responses, they still achieved reliably lower performance than the adult human participants represented in this work.

Tom: So, to wrap up what we’ve heard about this paper, the authors are suggesting that while vision language models show promise as a class of models that can reason over a broad class of visualization and question types, there are still significant opportunities to improve their alignment with human behavior.

Jane: They point to the gap between these models and human participants on several of the tests in CHART-six including GGR, CALVI and HOLF, as evidence for the need to use a diversity of independently designed measures when identifying opportunities for further model development <ref:2505.17202#pg1>.

Lu: The researchers envision this CHART-six protocol being used to track the progress of artificial systems towards achieving human-like behavior on tasks involving data visualizations <ref:2505.17202#pg1>. This suggests a path forward by creating controlled experimentation against real-world tasks involving complex, naturalistic inputs.

Meng: From an engineering standpoint, it sounds like they are providing a concrete benchmark for what success looks like when training these models for scientific reasoning rather than just pattern matching. It tells us where the current limitations of the architectures are showing up most clearly.

Lalam: And they noted that while proprietary systems like GPT-4V were most aligned with human behavior among the models in their suite, open-source models such as LLaVA1 point 6-Yi-34B and Pix2Struct-0 point 3B did not necessarily lag far behind in terms of error patterns <ref:2505.17202#pg2>.

Tom: It’s clear that this CHART-six evaluation protocol is a tool for cognitive AI benchmarking, revealing gaps between humans and AI systems on common tasks involving complex inputs like data visualizations <ref:2505.17202#pg1>.

Jane: So, the implication for us is that if we want to build tools that actually help scientists communicate and analyze data better, we need to focus on developing those models toward more human-like reasoning rather than just maximizing raw accuracy scores.

Conclusion: Tom: So we’re wrapping up our look at this paper, "Human-centered evaluation of data visualization understanding in vision-language models." The main point is that these vision language models are promising for modeling how humans understand data visualizations, but we need to know if they actually mimic human behavior on those tasks.

Jane: That’s the central tension here. We’ve seen a lot of accuracy scores for these AI systems, but this study pushes back on that by asking if the answers match what people actually produce when they try to visualize data.

Lu: Yeah, the authors set up this human-centered evaluation protocol using eight different vision language models across six specific tests called CHART-six. It’s a bunch of assessments designed to cover a wide variety of graph types and understanding skills that humans need.

Meng: And what they found is pretty clear—these vision language models performed worse than humans on average across those tasks, and that performance gap stayed there even when they used lenient criteria to judge the models.

Lalam: What’s really interesting is how they looked at the errors. They didn't just look at whether an answer was right; they compared the pattern of mistakes made by the AI to what humans make, and those patterns were reliably different.

Tom: So if we take away the idea that models are just trying to get a high accuracy number, this suggests that even if an AI gets some things right, its internal reasoning process is fundamentally different from a human's way of thinking about charts and graphs.

Jane: Exactly. The implication for someone who only listens to this is that we can’t just build these models based on how well they pass a standard test; we have to look at the actual *process* they use to reason about the data.

Lu: The authors are suggesting that there’s a real opportunity here to develop artificial systems that actually serve as useful models of how humans reason about data visualizations. It points toward a need for better alignment with human behavior in these complex reasoning tasks.

Meng: That means if we want to use these AI systems in real-world tools or education, we can’t just rely on the current benchmarks; we need to test them against this kind of human comparison protocol.

Lalam: And they even noted that while some models were closer to human behavior than others, the error patterns were still consistently distinct. It shows that it’s not a one-size-fits-all issue across all vision language models.

Tom: So the overall message is that there are significant opportunities to improve these systems by focusing on how they reason rather than just their final score on a test. This leads us into what this means for building better tools for STEM learning environments, which we’ll discuss next.