Binary Agreement for Efficient Unsupervised Confidence Calibration of Reasoning LLMs

The gist The method introduces an unsupervised confidence calibration technique for reasoning LLMs that derives a self-consistency-based proxy target from offline sampling on unlabeled data to distill it into a lightweight deployment-time predictor, which substantially outperforms baselines in various settings.

How it works

1. The core idea treats repeated sampling not as a deployment-time inference procedure but as an offline source of unlabeled supervision, where the approach uses repeated sampling on unlabeled examples to estimate a self-consistency signal.

2. Formally, the method defines a self-consistency score for an input–answer pair as s(x, a) = 1/k sum j=1 1/h if a(j) = a i.

3. Instead of computing this score at test time, the approach uses it offline on unlabeled data to construct supervision for a single-pass predictor by selecting a sampled response y˜i = (˜ri, a˜i) together with its score si = s(xi, a˜i)

4. This process yields weakly labeled examples of the form (xi, y˜i, si) which are then used to train a predictor fϕ(x, r, a) to estimate the empirical support for the response’s answer under repeated sampling.

5. At test time, given a new input x and response y = (r, a), the confidence score is reported as c(x, y) = fϕ(x, r, a) using this learned predictor.

Evaluation and Results

1. The method was evaluated across 5 math and question-answering tasks using 9 reasoning models from the Qwen, DeepSeek, and Nemotron families.

2. Empirical results showed that Self-consistency is generally well-calibrated with respect to correctness in reasoning LLMs.

3. The proposed method was found to be the strongest approach among the methods that satisfy our deployment constraints, both on average and in the worst case.

4. The results demonstrated that Our method significantly outperforms baselines and offers competitive performance to the more expensive full test-time self-consistency measure.

5. The confidence estimates were shown to improve downstream outcomes in applications such as selective prediction (without producing any tokens) and enhancing the calibration of stronger models.

Robustness and Decision-Making Usefulness

1. The method proved robust under distribution shift, showing that it remains robust across different temperatures and with a relatively small number of samples (e.g., 5-10)

2. Under distribution shift in language, the method offers reliable performance even when calibration data does not match test data.

3. For selective prediction, the method yields the strongest overall selective prediction performance among the methods available in our setting.

4. In linguistic calibration, conditioning a stronger decision-maker LLM on their confidence estimates consistently leads to the best-calibrated downstream decisions.

Conclusion and Limitations

1. The findings suggest that calibrated confidence for reasoning LLMs need not require either labeled calibration data (and possibly compute-intensive RL training) or expensive test-time sampling.

2. The method's effectiveness is shown to be independent of model ownership, as our black-box results suggest that the same strategy can remain effective even when the underlying reasoning model is only accessible through a proprietary interface.

3. A limitation noted is that "the method can only be as good as the self-consistency signal it distills: if agreement among samples is weakly related to correctness, or models are consistently wrong with high internal agreement, the learned predictor may inherit those failures".

4. Overall, the work suggests that calibration for reasoning LLMs can be learned from unlabeled data and deployed with only one generation.

1. Unsupervised confidence calibration for reasoning LLMs from a single generation allows deployment in settings where labeled calibration data is unavailable and repeated sampling is too costly by using offline sampling on unlabeled data to derive a self-consistency-based proxy target. This enables the creation of a lightweight deployment-time confidence predictor that maps a single generated response to an answer-level confidence estimate.

2. The method can be applied in black-box settings where model internals are unavailable, as it only requires the ability to sample outputs and access to a reasonably strong embedding model rather than the reasoning model's own hidden states. This expands applicability beyond open-weight models and proprietary APIs by demonstrating that the method can remain effective even when the underlying model is exposed only through an API or other proprietary interface.

3. The resulting confidence estimates improve downstream decision-making in selective prediction, enabling systems to decide whether to answer directly or defer based on calibrated confidence, potentially bringing not only accuracy gains but also inference cost savings roughly equal to the abstention rate.

4. The system can achieve better-calibrated downstream decisions when communicating with a stronger model; for instance, GPT-4o-mini conditioned on the assistant's confidence estimates reduces expected calibration error significantly compared to using token probabilities or verbalized confidence alone.

Reasoning language models often lack the calibrated confidence needed for reliable deployment, particularly when correctness labels are unavailable and sampling budgets are limited. We introduce Binary Agreement for Self-consistency Calibration (BASC), which uses self-consistency as an unlabeled proxy for correctness without constructing costly self-consistency targets for each calibration query. Our key observation is that agreement between an anchor response and a single independent probe is a Bernoulli observation whose conditional mean is the anchor's latent self-consistency. Pooling these observations across unlabeled queries, BASC supports both recalibrating existing confidence scores and learning more discriminative scores before recalibration, using as few as two responses per calibration query and only one at deployment. The use of binary isotonic recalibration provides a distribution-free calibration guarantee with respect to latent self-consistency. Across 5 tasks and 9 reasoning models, BASC substantially improves calibration with respect to correctness.