From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation

arXiv:2604.07667 · cs.AI, cs.MA, cs.SI · Submitted 2026-04-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "From Debate to Decision".

Jane: Here's the gist of the paper. The gist: Conformal Social Choice introduces a post-hoc decision layer that converts multi-agent debate outputs into calibrated act-versus-escalate decisions, ensuring safety by flagging wrong consensus cases with high probability, which is not evidence of correctness<ref:2604.07667#pg2> How it works The framework is a four-stage pipeline that transforms multi-agent debate outputs into prediction sets with marginal coverage guarantees<ref:2604.07667#pg3>.

Tom: Stage 1 involves a Multi-Agent Debate with Verbalized Probability Elicitation, where heterogeneous LLM agents produce verbalized probability distributions for the input question x<ref:2604.07667#pg4>.

Jane: These verbalized scores are noisy proxies for true beliefs and are postprocessed into numerical probabilities<ref:2604.07667#pg4>.

Tom: Stage 2 aggregates agent beliefs into a full probability distribution using a linear opinion pool, defined by the formula P(t) social(y x) = X N i=1 wi w i · π(t) i (y x)<ref:2604.07667#pg5>.

Paper summary: Tom: To recap, in "From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation", the paper introduces a post-hoc decision layer that turns debate outputs into calibrated act versus escalate decisions.

Jane: Basically, they take verbalized probability distributions from different agents and aggregate them using a linear opinion pool, then use split conformal prediction to produce prediction sets with a marginal coverage guarantee.

Lu: What makes this setup important is that it requires no retraining and no access to model internals for the decision layer itself.

Meng: That means we don't have to overhaul our entire LLM architecture just to get a safety net on top of the debate process.

Tom: The method aggregates those verbalized probability distributions into a collective belief via that linear opinion pool, and then applies split conformal prediction to produce sets satisfying a marginal coverage guarantee of greater than one minus alpha.

Jane: It's about producing sets where the correct answer has a probability greater than one minus alpha, without needing assumptions on how well the individual models are calibrated.

Lu: The paper uses a hierarchical action policy to map singleton sets to autonomous action and larger sets to human escalation based on those prediction sets.

Meng: It’s about making that decision explicit: if you have a single opinion, act autonomously; if there's more disagreement, escalate to a human.

Tom: And they test this across eight MMLU-Pro domains and two other benchmarks, showing that the fixed-round coverage stays close to their target. Eighty-one point nine percent of wrong consensus cases were flagged when alpha is zero point zero five.

Jane: That finding really suggests the mechanism itself provides a safety net for when agents get stuck in a bad agreement during that debate process.

Lu: And they also show that at any pre-specified round, the correct answer is included with probability greater than one minus alpha, which is exactly what you need for setting deployment thresholds.

Meng: For deployment, knowing that you have this calibrated refusal layer is crucial because it tells you exactly when to stop debating and escalate to a human expert.

Tom: So, we’ve seen how they use verbalized confidence and social choice to build this structured way of managing uncertainty during the debate process itself.

Conclusion: Jane: So, wrapping up this discussion on "From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation", the main thing is how they shift focus from just trusting one final answer to managing the risk of the entire debate process.

Tom: This paper shows that by aggregating verbalized probabilities through a linear opinion pool and calibrating with split conformal prediction, you can get calibrated act versus escalate decisions without retraining or model access.

Lu: The authors show you can use this framework to manage risk without needing perfect knowledge of the underlying model calibration, which is really important for deploying complex systems.

Meng: It means we can have these multi-agent discussions where we know exactly when to trust the AI and when to pull in a human expert without having to rewrite the models themselves.

Jane: It’s about moving beyond just looking at the final answer and giving us a structured way of managing the uncertainty inherent in agent disagreement during that debate process itself.

Tom: And they get a fixed-round marginal coverage guarantee, meaning at any specific round, say round five, there's a chance greater than alpha that the correct answer is included in your set.

Lu: That guarantee holds without assuming individual model calibration, which is what makes it robust for real-world black-box settings.

Jane: So, the big finding isn't that the debate itself becomes more accurate overall, but that this layer makes its failures actionable; it flags eighty-one point nine percent of wrong consensus cases when alpha is zero point zero five.

Tom: That suggests the mechanism itself provides a safety net for when agents get stuck in a bad agreement during that debate process.

Meng: So, the framework successfully converts debate outputs into set-valued act versus escalate decisions under marginal coverage control.

Jane: And that brings us to how this framework changes the way we think about controlling AI behavior in complex, multi-agent settings.

Mengdie Flora Wang, Haochen Xie, Guanghui Wang, Aijing Gao, Guang Yang, Ziyuan Li, Qucy Wei Qiu, Fangwei Han

AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China

cs.AI, cs.MA, cs.SI

Submitted: 2026-04-09

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: Conformal Social Choice introduces a post-hoc decision layer that converts multi-agent debate outputs into calibrated act-versus-escalate decisions, ensuring safety by flagging wrong

Key concepts

Multi-Agent Debate: This involves multiple different LLM agents discussing a question. Each agent provides its own answer and a verbalized probability distribution over possible answers. The core idea is that debating these different views can lead to better reasoning, but agreement doesn't guarantee the correct answer.
Linear Opinion Pool: This stage aggregates the individual probability distributions from all agents into one overall social probability distribution. It treats the agents' beliefs as inputs, combining them using a simple weighted average where each agent's belief contributes proportionally to the final pool.
Split Conformal Prediction: This is a statistical method used to create prediction sets that guarantee coverage. Instead of making assumptions about how well individual models are calibrated, it uses a held-out calibration set to determine a threshold. This ensures that at any given round, the correct answer is included in the resulting set with a guaranteed minimum probability.
Act-versus-Escalate Policy: This is the final decision layer that takes the statistical prediction sets and converts them into an action plan. If the predicted set contains only one answer (a singleton), it suggests autonomous action. If the set is larger, indicating high ambiguity or disagreement, it triggers human escalation for review.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Calibration — 1× in this paper · explained in 9 episodes
Conformal Prediction — 1× in this paper · explained in 11 episodes
Split Conformal Prediction — 1× in this paper · explained in 4 episodes

Summary

The gist: Conformal Social Choice introduces a post-hoc decision layer that converts multi-agent debate outputs into calibrated act-versus-escalate decisions, ensuring safety by flagging wrong consensus cases with high probability, which is not evidence of correctness<ref:2604.07667#pg2>

How it works

The framework is a four-stage pipeline that transforms multi-agent debate outputs into prediction sets with marginal coverage guarantees<ref:2604.07667#pg3>. Stage 1 involves a Multi-Agent Debate with Verbalized Probability Elicitation, where heterogeneous LLM agents produce verbalized probability distributions for the input question x<ref:2604.07667#pg4>. These verbalized scores are noisy proxies for true beliefs and are postprocessed into numerical probabilities<ref:2604.07667#pg4>. Stage 2 aggregates agent beliefs into a full probability distribution using a linear opinion pool, defined by the formula P(t) social(y x) = X N i=1 wi w i · π(t) i (y x)<ref:2604.07667#pg5>. The weight strategy employed is uniform weights wi = 1/N throughout the process<ref:2604.07667#pg6>.

Conformal Calibration and Decision Making

Stage 3 applies split conformal prediction to the aggregated social probabilities to produce prediction sets C(x) satisfying a marginal coverage guarantee of Pr[y ∈ C(x)] ≥ 1 − α<ref:2604.07667#pg7>. The non-conformity score is defined as snc(x, y) = 1 − Psocial(y x)<ref:2604.07667#pg9>. The conformal threshold qˆ is determined by the quantile of these scores from a held-out calibration set Dcal, ensuring the guarantee holds without assumptions on individual model calibration<ref:2604.07667#pg10>. Stage 4 constructs the prediction set C(x) = y ∈ Y: snc(x, y) ≤ qˆ.

Action Policy and Empirical Findings

A hierarchical action policy maps singleton sets to autonomous action and larger sets to human escalation. Key empirical findings show that at α=0.05, the conformal layer intercepts 81.9% of wrong-consensus cases before they reach automated action. Because the layer refuses to commit on these cases, the remaining conformal singletons are up to 22.1 percentage points more accurate than consensus stopping—a selection effect of calibrated refusal, not a reasoning improvement.

Safety and Trade-offs

The system provides a calibrated risk budget over the population of queries, not a per-instance certificate. Table 3 demonstrates that at α=0.05, conformal prediction correctly flags 81.9% of wrong-consensus cases across all domains. The net effect of conformal prediction on safety is overwhelmingly positive, with the ratio of errors prevented to errors introduced being approximately 240:1 at α=0.05. The framework escalates most cases to human review on high-ambiguity domains such as Law, which is viewed as a success case of calibrated refusal rather than a limitation.

Limitations and Future Directions

The coverage guarantee of Theorem 2 is marginal, not per-instance correctness. Conditional coverage requires additional techniques such as conformal risk control or group-balanced calibration. The paper suggests that α ∈ [0.05, 0.10] represents a practical sweet spot for conformal social choice in multi-agent debate settings. Future work includes exploring online and adaptive calibration methods to handle distribution shift in non-stationary environments. The framework is currently restricted to closed-set classification, with extensions to open-ended generation being a promising future direction.

Conclusion

Conformal Social Choice provides a principled mechanism for deciding when to act autonomously and when to defer to human review without retraining or model access. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. By aggregating verbalized probabilities from heterogeneous agents via a linear opinion pool and calibrating with split conformal prediction, the framework offers a principled mechanism for deciding when to act autonomously and when to defer to human review without retraining or model access. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control. The framework successfully converts debate outputs into set-valued act-versus-escalate decisions under marginal coverage control.

Improvements for AI systems

Improve decision-making from consensus-based stopping to a calibrated act-versus-escalate decision layer that converts debate outputs into calibrated action choices rather than committing to potentially wrong answers. This system can safely act on singleton predictions while escalating larger sets of uncertain predictions to human experts, as the paper states, the conformal layer makes its failures actionable.
Implement a post-hoc decision pipeline that aggregates verbalized probability distributions from heterogeneous agents via a linear opinion pool and calibrates them with split conformal prediction to yield prediction sets with a marginal coverage guarantee of Pr[y in C(x)] ≥ 1−α, without assumptions on individual model calibration. This allows the system to maintain control over its error rate at a user-adjustable miscoverage rate alpha.
Use the hierarchical action policy derived from the prediction set size to map singleton sets to autonomous action and larger sets to human escalation, effectively transforming calibrated uncertainty into an operational decision rule. This provides a selection effect of calibrated refusal, not a reasoning improvement by intercepting 81.9% of wrong-consensus errors at alpha=0.05.
Enhance the system's ability to detect wrong-consensus convergence by identifying cases where agents initially disagree but later converge to unanimous wrong consensus, which the paper shows is most susceptible in domains like Law and Psychology (34.8% and 33.3% respectively).
Deploy an entropy-based weighting strategy for social probability aggregation, as this strategy upweights agents that are more confident and is shown to produce only small differences in coverage and accuracy compared to uniform weighting after debate convergence, suggesting it is a robust choice for combining agent distributions.

Abstract

Multi-agent debate improves LLM reasoning, yet agreement is not evidence of correctness. When agents converge on a wrong answer through social reinforcement, consensus stopping commits that error to automated action with no recourse. We introduce Conformal Social Choice, a post-hoc layer that converts debate outputs into calibrated act-versus-escalate decisions. Verbalized probability distributions from heterogeneous agents are aggregated via a linear opinion pool and calibrated with split conformal prediction, yielding fixed-round marginal coverage: at any pre-specified round, the correct answer is included with probability at least,1 - α, without assumptions on individual model calibration. A hierarchical action policy maps singletons to autonomous action and larger sets to human escalation. Across eight MMLU-Pro domains and two additional benchmarks, fixed-round coverage stays close to target. The key finding is not that debate becomes more accurate, but that the conformal layer makes its failures actionable: at the final round, 81.9% of wrong-consensus cases are flagged by non-singleton sets at α = 0.05. Separately, among cases that reach a singleton before the final round, accuracy reaches 90.0--96.8% on MMLU-Pro (up to 22.1pp above consensus stopping): a selection effect, not a reasoning improvement. Simple probability- and entropy-threshold rules attain comparable interception; conformal's distinct value is a finite-sample fixed-round coverage guarantee together with explicit candidate sets for escalation. Adaptive first-singleton stopping is evaluated empirically and not covered by this guarantee.