Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
cs.CL
Submitted: 2026-10-13
Updated: 2026-10-13
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Cost-Optimal Active AI Model Evaluation
- Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
- Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings
- Efficient Benchmarking Is Just Feature Selection and Multiple Regression
- Training Verifiers to Solve Math Word Problems
- MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
- QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions
- Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
- Large Language Model Selection with Limited Annotations
- BayesAME: Bayesian Active Model Evaluation
- Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
- CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
- Agent psychometrics: Task-level performance prediction in agentic coding benchmarks
- Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
- Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
- Fluid Language Model Benchmarking
- Efficient Sequential Evaluation of Large Language Models
- How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
- ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
- Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering