Large Language Model Selection with Limited Annotations

arXiv:2605.24981 · cs.CL, cs.LG · Submitted 2026-05-24 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-05-24

Updated: 2026-09-25

Comments: This submission was uploaded as a separate arXiv entry in error. It is a revised version of arXiv:2510.09418, which will be updated instead

Code: https://github.com/InternLM/InternLM-techreport

License: http://creativecommons.org/licenses/by/4.0/

The gist: Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets.

Terminology

Abstract

Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.

Sources

Related papers