Estimating problem difficulty without ground truth using Large Language Model comparisons

arXiv:2512.14220 · cs.LG, cs.AI · Submitted 2025-12-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Estimating problem difficulty without ground truth using Large Language Model comparisons".

Jane: Here's the gist of the paper. The gist: LLM compare is a new method for estimating problem difficulty that does not rely on ground truth data, using an LLM to perform pairwise difficulty comparisons and then computing Bradley-Terry scores based on the outcomes.

Tom: Problem Difficulty Estimation Context Recent advances in the finetuning of large language models (LLMs) have significantly improved their performance on established benchmarks, emphasizing the need for increasingly difficult, synthetic data The ultimate goal is to create synthetic out-of-distribution data, i.e.

Jane: problems that are currently unsolvable by both humans and LLMs Current approaches, such as human calibration or performance-based scoring, fail to generalize to out-of-distribution problems because they are not scalable, time-consuming, and ground truth dependent Traditional problem difficulty was measured in a human-centric way, such as Classical Test Theory (CTT) and Item Response Theory (IRT) However, these methods fundamentally depend on human responses and do not generalize to newly generated problems.

Tom: The LLM Compare Methodology LLM compare consists of two parts First, an LLM plays a fixed number of matches per problem in a dataset we want to score by letting it decide which out of two problems is more difficult with a basic prompt.

Paper summary: Tom: So, looking at "Estimating problem difficulty without ground truth using Large Language Model comparisons," the authors are proposing this LLM compare method as a way to estimate problem difficulty by having an LLM do pairwise comparisons and then using those outcomes to calculate Bradley-Terry scores.

Jane: It really highlights that we can get a measure of difficulty that is dynamic and continuous, and it doesn't require any ground truth data for the initial estimation.

Tom: And the key thing they showed is that this method isn't just good at ranking problems; it can actually identify distinct tiers in a benchmark because they found clear separation between easy problems and hard problems with a Cohen’s d of two point five six on their synthetic set <ref:2512.14220#pg3>.

Jane: For someone who only listens to the show, this means we are looking at tools where AI can help structure the difficulty of new content in a way that is interpretable, unlike some other ranking methods that might just bin things into groups.

Tom: The authors are positioning this as a fine-grained and reliable alternative to existing measures like CTT or IRT because it’s robust to hallucinations and shows high correlation with human labels.

Jane: It opens up possibilities for using AI in curriculum design or adaptive testing systems where the difficulty of problems needs to be constantly evaluated without relying on slow, time-consuming human annotations.

Tom: This paper is a significant step toward creating synthetic data that has a structure, not just random noise.

Lu: It’s important because it moves us past the ground truth dependency that has limited how much we can generate challenging problems for large language models to learn from.

Meng: From my side, it shows that AI can be used as an evaluator in a way that is scalable and doesn't require us to rely on constantly running human annotation pipelines.

Lalam: So the paper essentially gives us a tool where the difficulty itself is derived from how LLMs compare problems against each other, and it’s a method that seems quite solid for moving forward with synthetic data generation.

Conclusion: Tom: So, we're wrapping up this discussion on "Estimating problem difficulty without ground truth using Large Language Model comparisons." Basically, these researchers developed a new way to figure out how hard a problem is by just having one AI compare two problems against each other.

Jane: It’s about ditching the need for human labels or perfect answers when we want to measure difficulty. They use the AI to do pairwise comparisons and then calculate scores using something called Bradley-Terry models.

Lu: The core idea is that you don't need ground truth data upfront; you just need a set of problems, and the LLM iteratively estimates how strong one problem is compared to another based on those matches.

Meng: It’s interesting that this method doesn't rely on any pre-existing human calibration, which is a big deal for scaling up difficulty testing without needing massive annotation teams.

Tom: What I find compelling is that it’s dynamic and continuous, meaning we can test new problems as they come in without having to re-calibrate the whole system every time.

Jane: And they showed that these scores actually align well with human judgments, showing a Pearson correlation of over zero point eight for a large set of problems.

Lu: That alignment is strong because it means the AI’s internal estimation process isn't completely detached from what humans consider hard or easy.

Meng: But the paper also points out that this method can actually reveal tiers in a benchmark, meaning it can tell you there are distinct groups of problems that are significantly harder than others.

Tom: That’s significant because traditional measures often just give you a single score, not the nuance of having clear difficulty levels within your test set.

Jane: So, what this means for us is we might be able to build synthetic data sets that have a more structured difficulty curve rather than just random problems.

Lu: Exactly. This moves us toward generating out-of-distribution problems that are truly challenging because the AI has learned a consistent relative measure of difficulty across the entire set.

Tom: It’s a step toward making those synthetic problems more useful for training better models, and it’s certainly less reliant on manual labeling than we've seen before.

Jane: Next up, we're going to look at how these scores actually separate the hard problems from the easy ones in practice.

Data Analytics Lab, Vrije University Brussel · imec-SMIT, Vrije University Brussel · School of Engineering and Applied Sciences, Harvard University

cs.LG, cs.AI

Submitted: 2025-12-16

Updated: 2025-12-16

Comments: 19 pages, 10 figures

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: The gist: LLM compare is a new method for estimating problem difficulty that does not rely on ground truth data, using an LLM to perform pairwise difficulty comparisons and then computing

Key concepts

Problem Difficulty Estimation Context
The goal is to create synthetic, out-of-distribution problems that current methods cannot solve. Traditional measures like CTT fail because they rely on human responses and cannot generalize to new problems, making LLM Compare a scalable alternative.
LLM Compare Methodology
This method involves an LLM comparing pairs of problems. The LLM decides which problem is harder in a match, and these pairwise wins are used to estimate the underlying difficulty strengths ($eta$) using the Bradley-Terry model and iterative Luce Spectral Ranking (ISLR).
Bradley-Terry (BT) Model
The BT model is a statistical framework used to calculate scores based on pairwise comparisons. It assumes the probability of problem 'i' beating problem 'j' is determined by their respective difficulty strengths ($eta$), allowing for the estimation of relative difficulty without needing absolute ground truth scores.

Terminology

Summary

The gist: LLM compare is a new method for estimating problem difficulty that does not rely on ground truth data, using an LLM to perform pairwise difficulty comparisons and then computing Bradley-Terry scores based on the outcomes.

Problem Difficulty Estimation Context

Recent advances in the finetuning of large language models (LLMs) have significantly improved their performance on established benchmarks, emphasizing the need for increasingly difficult, synthetic data The ultimate goal is to create synthetic out-of-distribution data, i.e. problems that are currently unsolvable by both humans and LLMs Current approaches, such as human calibration or performance-based scoring, fail to generalize to out-of-distribution problems because they are not scalable, time-consuming, and ground truth dependent Traditional problem difficulty was measured in a human-centric way, such as Classical Test Theory (CTT) and Item Response Theory (IRT) However, these methods fundamentally depend on human responses and do not generalize to newly generated problems.

The LLM Compare Methodology

LLM compare consists of two parts First, an LLM plays a fixed number of matches per problem in a dataset we want to score by letting it decide which out of two problems is more difficult with a basic prompt. Then, based on the number of wins per question, the difficulty score is estimated by its strength under the Bradley–Terry (BT) model. The BT model assumes that the probability that problem i beats j is modeled as pij = e βi / (e βi + e βj). The total number of matches and the pairwise outcomes are known, allowing for the estimation of strengths (β1, β2,..., βn) using iterative Luce Spectral Ranking (ISLR).

Conceptual Framework and Positioning

The paper proposes a conceptual framework that positions existing difficulty measures on three orthogonal planes—construction, scale and dependence The first plane encodes by whom an how a measure is constructed, with dimensions human vs. LLM and performance vs. judgement. The second plane shows what form the output of the measure takes, with dimensions dynamic vs. static (flexibility to introduce a new problem) and discrete vs. continuous. The third plane relates to dependence on external anchoring, encoding if the measure depends on a specific model and if it needs context information to provide non-trivial difficulty scores. LLM compare naturally occupies two empty quadrants in this space of difficulty measures It is the first measure that is dynamic and continuous, as well as model-agnostic and independent of ground truth information.

Empirical Validation and Robustness

LLM compare demonstrates strong alignment with human annotations, showing a Pearson r ≥ 0.80 for n = 1876 It is robust to hallucinations, with less than 6% degradation in Pearson correlation for 10% noise injection. Across all datasets, problems deemed more difficult by LLM compare are also considered more difficult by human labels, human performance, LLM labels and LLM performance. Furthermore, the Kendall τ lies naturally lower for all difficulty measures because it punishes discordant pairs and ties explicitly.

Identification of Difficulty Tiers

BT scores are able to identify the presence of difficulty tiers in a benchmark because they are linear in the log-odds of success. The paper applied LLM compare to a synthetic dataset consisting of 50 Omni-Math Tier 1 (=easiest problems according to human labels) and 50 Tier 4 (=hardest problems) The resulting distributions show a clear separation between the two difficulty tiers, with Cohen’s d between Tier 1 and Tier 4 for OpenAI o3 being 2.56.

Conclusion and Implications

LLM compare is a fine-grained, reliable and broadly applicable alternative for existing measures of difficulty, resolving their key limitations It provides a key step towards (superhuman) synthetic data generation with LLMs. Our findings highlight the potential of LLMs as difficulty assessors and have implications for curriculum design, model evaluation, adaptive testing and AI-assisted research ideation. The relative differences in difficulty between problems are also interpretable, in contrast to ranking- or binning-based difficulty measures. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. The relative differences in difficulty between problems are also interpretable, in contrast to ranking- or binning-based difficulty measures. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. The relative differences in difficulty between problems are also interpretable (Section 2.2), in contrast to ranking- or binning-based difficulty measures. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems. It is an important step towards solving the need for time-consuming human annotations as it shows high correlation to human labels and is robust to hallucinations. The BT scores reflect the presence of tiers in benchmark difficulty, e.g. when a dataset contains a group of equivalent problems.

Improvements for AI systems

  1. Bold header: New problem difficulty estimation pipeline

This system can estimate problem difficulty for out-of-distribution problems by using LLM compare, which performs pairwise difficulty comparisons, and then Bradley-Terry scores are computed based on the outcomes. This allows for scoring synthetic data without relying on ground truth data, addressing the limitations of current approaches.

  1. Bold header: Model-agnostic difficulty assessment

The proposed method is model-agnostic and independent of ground truth information, which makes it suitable for replacing existing measures because it naturally occupies all desirable quadrants in the space of difficulty measures. This ensures that the evaluation method is not limited by a specific model, as demonstrated by showing strong alignment across different LLMs like OpenAI o3 and Gemini 2.5 Pro.

  1. Bold header: Robustness against LLM hallucinations

The system exhibits high reliability because LLM compare is the most robust to hallucinations among all LLM-based difficulty measures, with its scores degrading the least under noise injection. This robustness is achieved because LLM compare is a continuous measure, so each hallucination creates a discordant pair, unlike discrete rank-based measures.

  1. Bold header: Automated curriculum and test design

The ability of the method to identify difficulty tiers in a benchmark allows for adaptive testing; specifically, it can show a clear separation between the two difficulty tiers using Cohen’s d and Wasserstein distance on synthetic datasets, guiding the creation of increasingly challenging synthetic data.

  1. Bold header: Fine-grained relative difficulty quantification

The Bradley-Terry scores are linear in the log-odds of success, meaning the differences in difficulty between problems are interpretable, which is superior to ranking or binning measures because it allows for clear interpretation of relative difficulty between specific problems within a set.

Abstract

Recent advances in the finetuning of large language models (LLMs) have significantly improved their performance on established benchmarks, emphasizing the need for increasingly difficult, synthetic data. A key step in this data generation pipeline is a method for estimating problem difficulty. Current approaches, such as human calibration or performance-based scoring, fail to generalize to out-of-distribution problems, i.e. problems currently unsolvable by humans and LLMs, because they are not scalable, time-consuming, and ground truth dependent. Therefore, we propose a new method for estimating problem difficulty, LLM compare, that addresses these limitations. An LLM performs pairwise difficulty comparisons, and then Bradley-Terry scores are computed based on the outcomes. To validate our method, we first propose a conceptual framework that positions existing approaches on three orthogonal planes--construction, scale and dependence--identifying which quadrants a measure needs to occupy to score out-of-distribution problems. LLM compare naturally occupies all desirable quadrants as the first measure that is continuous and dynamic, model-agnostic and independent of ground truth information. As a second validation, we show that LLM compare demonstrates strong alignment with human annotations: Pearson r at least 0.80 for n=1876. Thirdly, we show that LLM compare is robust to hallucinations, with less than 6% degradation in Pearson correlation for 10% noise injection. Our work represents a significant step towards replacing time-consuming human annotations and synthetic data generation, and will be an important driver for curriculum design, model evaluation, and AI-assisted research ideation.

Sources

Related papers