Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

arXiv:2602.02589 · cs.AI, cs.LG · Submitted 2026-02-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment".

Jane: Here's the gist of the paper. The gist The introduction states that PeerRank introduces a fully autonomous end-to-end evaluation framework in which models generate evaluation tasks, answer them with category-scoped live web grounding, judge peer responses and aggregate dense peer assessments into relative performance estimates, without human supervision or gold references.

Tom: How it works PeerRank treats evaluation as a multi-agent process where each model participates symmetrically as task designer, respondent, and evaluator In Phase 1, "Endogenous Question Generation," each model independently generates a set of questions from a fixed category set, ensuring that "No filtering or deduplication is applied".

Jane: The full evaluation corpus Q is then formed by combining the generated questions from all participating models.

Tom: In Phase 2, "Web-Grounded Answer Generation," all models answer all questions using live web access enabled to evaluate retrieval and synthesis under open-world conditions.

Paper summary: Tom: So to wrap up, this paper introduces Agents-as-Peers as a way to evaluate LLMs without human gold references by making the models themselves participate in the task design, answering, and judging process.

Jane: It’s about moving away from those fixed benchmarks that quickly age and not accounting for open-world web access or synthesis that we need today > <ref:2602.02589#pg3>.

Lu: The implication is that we can get stable, discriminative rankings using this multi-agent approach because they are explicitly measuring identity bias, name bias, and position bias > <ref:2602.02589#pg1>.

Meng: For deployment, this means we need to account for how the evaluation protocol itself might influence the final score because the biases are measurable and controllable > <ref:2602.02589#pg3>.

Lalam: The paper suggests peer-based evaluation can serve as a scalable complement to human-anchored benchmarks, provided that we measure those biases alongside performance metrics > <ref:2602.02589#pg1>.

Conclusion: Tom: So we've seen how these AI models can actually create their own evaluation tasks and then judge each other instead of relying on some fixed human list or a single central judge for LLM performance > <ref:2602.02589#pg1>.

Jane: It sounds like the main idea is that we can get a more honest, relative measure of how good different AI models are by letting them peer-assess each other instead of just checking them against some old benchmark > <ref:2602.02589#pg3>.

Tom: Right. They call it Agents-as-Peers, and they’re making sure to measure things like self bias and identity bias while the models are judging > <ref:2602.02589#pg3>.

Lu: The authors found that on actual benchmarks like TruthfulQA, these peer scores line up really well with actual accuracy, showing a Pearson correlation of about zero point nine hundred four > <ref:2602.02589#pg1>.

Meng: But they also found some interesting stuff about the biases themselves. Like how most models tend to rate themselves higher than their peers, which is kind of expected but still worth tracking when you're deploying something > <ref:2602.02589#pg3>.

Lalam: It really suggests that if we want a scalable way to evaluate these large language models without relying on human labels, we need to treat the evaluation process itself as a system that has its own biases and factors influencing the scores > <ref:2602.02589#pg1>.

Jane: So what does this mean for us, especially when we're thinking about putting models into production environments where things are constantly changing?

Yanki Margalit, Erni Avram, Ran Taig, Oded Margalit, Nurit Cohen-Inger

Caura.ai · Ben-Gurion University of the Negev

cs.AI, cs.LG

Submitted: 2026-02-01

Updated: 2026-10-20

Comments: Version 2.0 of PeerRank

Code: https://github.com/caura-ai/caura-PeerRank

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The introduction states that PeerRank introduces a fully autonomous end-to-end evaluation framework in which models generate evaluation tasks, answer them with category-scoped live web

Key concepts

Agents-as-Peers
A distributed protocol where every LLM acts as a task designer, respondent, and evaluator. Models generate questions, answer them using live web access, and then score the responses of their peers without any single privileged judge involved.
Peer Score (Pj)
The primary metric for ranking a model's performance. This score is calculated by aggregating scores from all other agents that evaluate a specific model's answer, explicitly excluding the model's own self-ratings to ensure an unbiased peer assessment.
Bias Quantification
A method to measure systematic errors in evaluation. The protocol tests and quantifies three types of bias: self-bias (models overrating themselves), name bias (scoring based on model identity), and position bias (scoring based on the order answers are presented).
Web Grounding
The process where LLMs use live web access to generate answers. This allows models to evaluate retrieval and synthesis capabilities in an open-world setting, using a shared external retrieval layer for category-scoped context.

Terminology

Summary

The gist The introduction states that PeerRank introduces a fully autonomous end-to-end evaluation framework in which models generate evaluation tasks, answer them with category-scoped live web grounding, judge peer responses and aggregate dense peer assessments into relative performance estimates, without human supervision or gold references.

How it works

PeerRank treats evaluation as a multi-agent process where each model participates symmetrically as task designer, respondent, and evaluator In Phase 1, Endogenous Question Generation, each model independently generates a set of questions from a fixed category set, ensuring that No filtering or deduplication is applied. The full evaluation corpus Q is then formed by combining the generated questions from all participating models.

In Phase 2, Web-Grounded Answer Generation, all models answer all questions using live web access enabled to evaluate retrieval and synthesis under open-world conditions. Web grounding is implemented using a single external retrieval layer that is shared across all evaluated models, and Category-scoped grounding fetches context only for questions in the current events category.

Peer Evaluation and Aggregation

In Phase 3, Peer Evaluation, each model evaluates all answers to each question, producing scores si,j,q from 1 to 10. Judging is performed without web access so judges score only the submitted answer. To control systematic judge effects, evaluations are conducted under three regimes: (i) shuffle-only (randomized answer order), (ii) blind-only (fixed order, hidden identities), and (iii) shuffle+blind. The resulting evaluation tensor S has dimensions K × K × N.

Metrics and Bias Quantification

The observed score of model mj is Oj = Ei,q[si,j,q], which includes self-evaluations. The primary ranking metric is the peer score Pj = Ei̸=j,q[si,j,q], which excludes self-ratings. Robustness is checked via an Elo rating approach based on pairwise model comparisons. PeerRank explicitly measures evaluation bias by comparing scores across regimes. Self bias is defined as ∆self j = Eq[sj,j,q] − Pj, Name (identity) bias is measured as ∆name j = P shuffle j − Pj, and position bias as ∆pos j = P blind j − Pj.

Empirical Validation and Findings

In a large-scale study over 12 commercially available models and 420 autonomously generated questions, PeerRank produces stable, discriminative rankings. The study validates the methodology on TruthfulQA and GSM8K where peer scores correlate with objective accuracy. Peer evaluation outperforms self-evaluation in tracking correctness, showing a Pearson r = 0.905 for cross-evaluation versus r = 0.538 for self-evaluation on TruthfulQA. Furthermore, peer scores strongly correlate with GSM8K ground-truth performance (Pearson r = 0.873). The study also reveals that judge generosity is negatively correlated with peer performance across models (Pearson r = −0.755).

Behavioral Insights

The analysis of bias shows that Self bias is typically positive in our study, indicating that most models exhibit self-overrating when evaluating their own answers. Name bias is non-trivial, with more recognizable model identities receiving higher scores when visible. Position bias is measurable, as answers shown first receive a +0.39 score lift on average, motivating shuffling as a control. The study also observes substantial heterogeneity in reasoning allocation across models, finding that some models employ hidden or budgeted deliberation during generation.

Conclusion

PeerRank suggests that peer-based evaluation can serve as a scalable complement—and in some settings an alternative—to human-anchored benchmarks, provided that bias is explicitly measured and controlled. The framework offers a continuously updatable evaluation paradigm aligned with deployment conditions, where models retrieve information, synthesize across sources, and operate without fixed reference answers. By treating bias as a measurable quantity alongside performance metrics, PeerRank shifts evaluation from matching static answer keys toward understanding relative performance under a shared, bias-aware protocol.

Improvements for AI systems

  1. Implement fully endogenous peer evaluation to create a continuously updatable evaluation paradigm aligned with deployment conditions rather than relying on static benchmarks or human-authored prompts. This allows models to retrieve information, synthesize across sources, and operate without fixed reference answers.

  2. Integrate live web grounding during the answering phase but disable it during judging to ensure judges score only the submitted answer, not additional evidence retrieved at scoring time, keeping judgments blind and comparable across models.

  3. Develop bias-aware protocols by systematically controlling presentation effects through shuffle-only, blind-only, and shuffle+blind regimes to explicitly measure self bias (self minus peer), name bias (shuffle-only minus shuffle+blind), and position bias (blind-only minus shuffle+blind).

  4. Enhance evaluation outputs by reporting multi-dimensional model characterization across normalized dimensions like quality, speed, consistency, humility (inverse self bias), and strictness to avoid relying on single leaderboards. This reveals trade-offs like Models with high peer scores may exhibit higher latency or lower humility.

  5. Incorporate reasoning effort as a first-class evaluation dimension by using proxies like the character-to-token ratio to infer hidden or budgeted deliberation during generation, allowing for comparisons between models based on their internal processing policies.

Sources

Related papers