Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

arXiv:2606.00820 · cs.CL · Submitted 2026-05-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Not All Flips Are Conformity".

Jane: Here's the gist of the paper. The gist: The conventional answer flip rate conflates three distinct mechanisms—spontaneous instability, stance-induced conformity, and reasoning-induced persuasion—which this paper decomposes using a controlled experimental design.

Tom: Decomposition Framework The study employs a three-source decomposition framework to isolate answer changes into (1) spontaneous change that would occur regardless of peer input, (2) stance-induced conformity triggered by peer positions alone, and (3) additional change induced by exposure to peer reasoning<ref:2606.00820#pg2> The decomposition is operationalized through a controlled experimental design in which each agent-question pair is observed under three counterfactual conditions (self-reflection, stance-only, and full reasoning exposure), allowing us to attribute each observed change to its proximate cause<ref:2606.00820#pg4> The decomposition is exhaustive over peerinduced changes (conditional on self-reflection stability) and mutually exclusive by construction<ref:2606.00820#pg4>.

Jane: Key Findings on Mechanisms The analysis reveals that spontaneous instability is a large baseline source of answer change, with "spontaneous degradation" being nearly twice as frequent as "harmful conformity" in the primary GPT-4o setting<ref:2606.00820#pg5> Strict conformity is substantially lower than raw flip-rate estimates and remains "predominantly harmful," with 63.6% correct-to-wrong in the primary setting<ref:2606.00820#pg5>.

Tom: Furthermore, reasoning often increases error adoption rather than correcting it, as only 30% of reasoning-induced changes improve accuracy<ref:2606.00820#pg6> The information-gradient experiment shows that "reasoning-like presentation leads to 20–39% error adoption among resistant agents, even when the reasoning contains no valid logical content"<ref:2606.00820#pg6>.

Title and authors: Tom: Now we’re going through the actual summary of "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," which lays out what they actually did in their experiment. They use a three-source decomposition framework to isolate answer changes into spontaneous instability, stance-induced conformity, and reasoning-induced persuasion <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Jane: And they show how they do this by using controlled counterfactual conditions to attribute each observed change to one of those three sources <ref:2606.00820#pg4> <ref:2606.00820#pg4>.

Lu: The big picture here is that the authors are showing that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Meng: This means for us in production, we can stop treating every answer change the same way; we can start asking what mechanism caused it so we know what to fix at scale <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Lalam: I'm sure of my answer here: this work shows that the persuasive power of reasoning isn't just about the logic itself, but heavily dependent on how it’s structured <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Tom: And they found that spontaneous degradation is actually the larger source of change, being nearly twice as frequent as harmful conformity in the primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Jane: So if spontaneous instability is so common, it means most answer shifts are just the model being inherently unstable or shaky without any external input at all, which is a bit different from what we usually think about social influence <ref:2606.00820#pg2> <ref:2606.00820#pg2>.

Lalam: Then there's the third source, reasoning-induced persuasion, which they investigate by showing that reasoning-like presentation leads to twenty to thirty-nine percent error adoption among resistant agents, even when the reasoning contains no valid logical content <ref:2606.00820#pg6> <ref:2606.00820#pg6>.

The paper's summary: Tom: So we’ve covered the core findings of "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," which breaks down answer flips into spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those three causes using counterfactual conditions. It really shifts how we think about what's happening when AI agents debate each other.

Lu: The big picture is that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in the primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Lu: The mitigation part is interesting because they found a way to reduce that harmful conformity by thirteen point six percentage points when we target it in a diagnostic setting where we have ground truth labels available <ref:2606.00820#pg7> <ref:2606.00820#pg7>.

Meng: But then they also showed that without those correctness labels or self-reflection controls, reducing peer adoption doesn't actually improve accuracy because they can’t tell if the influence we’re trying to reduce is harmful or helpful <ref:2606.00820#pg2> <ref:2606.00820#pg2>.

Lalam: And they also looked at what this means for deployment policies, showing that while a targeted policy reduces peer adoption by two point five times more effectively than a random one, neither policy produces a significant accuracy change without those runtime correctness signals or verification mechanisms we talked about before <ref:2606.00820#pg9> <ref:2606.00820#pg9>.

Jane: So the paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before <ref:2606.00820#pg2> <ref:2606.00820#pg1>.

The paper's improvements: Tom: So we’ve broken down "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" showing how answer flips are actually caused by three things: spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those causes using counterfactual conditions. It really shifts how we think about what's happening when AI agents debate each other.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in their primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Tom: So while they give us tools to predict and mitigate the bad stuff when we have labels, it really underlines that reliable deployment will always require some kind of runtime correctness signal or a way to verify what the agents are doing in real-time <ref:2606.00820#pg9> <ref:2606.00820#pg9>.

Jane: The paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before <ref:2606.00820#pg2> <ref:2606.00820#pg1>.

Conclusion: Tom: So we’ve broken down "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," showing how answer flips aren't just random noise, but are caused by three things: spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those causes using counterfactual conditions. It really shifts how we think about what's actually happening when AI agents debate each other.

Lu: The big picture is that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers.

Meng: That means for us in production, we can stop treating every answer change the same way; we can start asking what mechanism caused it so we know what to fix at scale.

Lalam: I'm sure of my answer here: this work shows that the persuasive power of reasoning isn't just about the logic itself, but heavily dependent on how it’s structured, which is a very important insight for building better agents.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in their primary GPT-4o setting.

Jane: So if spontaneous instability is so common, it means most answer shifts are just the model being inherently unstable or shaky without any external input at all. That’s a bit different from what we usually think about social influence.

Lu: And then they look at strict conformity, which they say is substantially lower than the raw flip-rate estimates and remains predominantly harmful, with fifty-seven to seventy-seven percent correct-to-wrong across model replications in that primary setting.

Meng: Harmful conformity is worrying because it means the agents are actually getting worse when they try to agree with their peers, even if they think they’re being helpful. That’s a real risk for any system we put into production.

Lalam: Then there's the third source, reasoning-induced persuasion, which they investigate by showing that reasoning-like presentation leads to twenty to thirty-nine percent error adoption among resistant agents, even when the reasoning contains no valid logical content.

Jane: Wow, so even if the argument itself is nonsense, just presenting it in a reasoned way can actually get other agents to adopt those wrong answers. That's a very strange effect for persuasive power.

Tom: And they’ve done some prediction work too, showing that harmful conformity can be predicted from Round zero features with an AUC of zero point seven nine, which is useful for setting up risk-targeted interventions before the debate even really gets going.

Lu: The mitigation part is interesting because they found a way to reduce that harmful conformity by thirteen point six percentage points when they target it in a diagnostic setting where we have ground truth labels available.

Meng: But then they also showed that without those correctness labels or self-reflection controls, reducing peer adoption doesn't actually improve accuracy because they can’t tell if the influence we’re trying to reduce is harmful or helpful. That means we need verification mechanisms, not just risk scoring.

Tom: So while they give us tools to predict and mitigate the bad stuff when we have labels, it really underlines that reliable deployment will always require some kind of runtime correctness signal or a way to verify what the agents are doing in real-time.

Jane: The paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before.

Lu: The future work they suggest is crucial: extending this decomposition framework to other model families and different debate architectures so we can see if these principles hold up elsewhere.

Meng: And I also think it’s worth seeing how consistency checks during debate can help bridge that gap between diagnosing the risk and actually deploying a solution in a real-time system.

1Xiqi Hao, 2Zengqing Wu, 1Yu-Xuan Qiu, 2Chuan Xiao, 1Ruiqi Xu, 2Shuyuan Zheng, 3Jianbin Qin

Beijing Institute of Technology Department of Information Science and Technology University of Osaka Shenzhen University

cs.CL

Submitted: 2026-05-30

Updated: 2026-10-18

Code: https://github.com/47hao030808/Not-All-Flips-Are-Conformity

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: The conventional answer flip rate conflates three distinct mechanisms—spontaneous instability, stance-induced conformity, and reasoning-induced persuasion—which this paper decomposes

Key concepts

Three-Source Decomposition Framework
This framework isolates answer changes into three distinct causes: spontaneous instability (change regardless of peers), stance-induced conformity (change due to peer positions), and reasoning-induced persuasion (change due to exposure to peer reasoning). It uses controlled experiments with counterfactual conditions to attribute observed changes precisely.
Harmful Conformity
This refers to agents changing their answers simply because other agents have chosen a different option, even if the original answer was correct. The study shows this is predominantly harmful, leading to significant accuracy drops when it occurs.
Reasoning Form vs. Content
The paper tests whether agents are persuaded by the actual logical content of another agent's reasoning or merely the structural form of that reasoning. Findings indicate that the presentation style ('form effect') is substantial and can lead to error adoption even when the reasoning itself is invalid.

Terminology

Summary

The gist: The conventional answer flip rate conflates three distinct mechanisms—spontaneous instability, stance-induced conformity, and reasoning-induced persuasion—which this paper decomposes using a controlled experimental design.

Decomposition Framework

The study employs a three-source decomposition framework to isolate answer changes into (1) spontaneous change that would occur regardless of peer input, (2) stance-induced conformity triggered by peer positions alone, and (3) additional change induced by exposure to peer reasoning<ref:2606.00820#pg2> The decomposition is operationalized through a controlled experimental design in which each agent-question pair is observed under three counterfactual conditions (self-reflection, stance-only, and full reasoning exposure), allowing us to attribute each observed change to its proximate cause<ref:2606.00820#pg4> The decomposition is exhaustive over peerinduced changes (conditional on self-reflection stability) and mutually exclusive by construction<ref:2606.00820#pg4>.

Key Findings on Mechanisms

The analysis reveals that spontaneous instability is a large baseline source of answer change, with spontaneous degradation being nearly twice as frequent as harmful conformity in the primary GPT-4o setting<ref:2606.00820#pg5> Strict conformity is substantially lower than raw flip-rate estimates and remains predominantly harmful, with 63.6% correct-to-wrong in the primary setting<ref:2606.00820#pg5>. Furthermore, reasoning often increases error adoption rather than correcting it, as only 30% of reasoning-induced changes improve accuracy<ref:2606.00820#pg6> The information-gradient experiment shows that reasoning-like presentation leads to 20–39% error adoption among resistant agents, even when the reasoning contains no valid logical content<ref:2606.00820#pg6>.

Prediction and Intervention

Harmful conformity can be predicted from Round 0 features (AUC = 0.79), and a risk-targeted intervention reduces it by 13.6 percentage points<ref:2606.00820#pg2> In the diagnostic setting, this targeted intervention reduces harmful conformity by 13.61 pp in the high-risk group (95% CI [7.5, 19.7]), and improves accuracy by 14.63 pp<ref:2606.00820#pg7>. However, without correctness labels or self-reflection controls, reducing peer adoption does not improve accuracy because harmful and beneficial influence cannot be distinguished<ref:2606.00820#pg2>.

Reasoning Form vs. Content

The study investigates whether agents respond to the logical content of reasoning or merely its structural form<ref:2606.00820#pg6> The information-gradient experiment reveals a form effect (stance → invalid reasoning, +29.66 pp) and a content effect (invalid → wrong reasoning, +24.83 pp), where the form effect alone is substantial<ref:2606.00820#pg6>. This suggests that the persuasive power of reasoning in debate does not depend solely on logical content<ref:2606.00820#pg6>.

Deployment Implications

The deployment setting tests whether peer-adoption risk prediction yields practical accuracy gains<ref:2606.00820#pg2> While the targeted policy reduces peer adoption 2.5× more effectively than the random policy, neither policy produces a significant accuracy change because opposing effects of harmful and beneficial adoption approximately cancel out<ref:2606.00820#pg9>. This implies that reliable deployment requires runtime correctness signals or verification mechanisms in addition to Round 0 risk scoring<ref:2606.00820#pg2>

The work shows that stance convergence in MAD cannot be interpreted from answer flip rates alone<ref:2606.00820#pg8>. The paper is organized around three questions: measure (RQ1: decomposing stance convergence), explain (RQ2: reasoning form vs. content), and mitigate (RQ3: prediction and intervention)<ref:2606.00820#pg2> Future work should extend the decomposition to additional model families and debate architectures, and explore whether consistency checks during debate can bridge the diagnostic-to-deployment gap<ref:2606.00820#pg9>. The study has several limitations, including using homogeneous GPT-4o agents and only studying singleround debate<ref:2606.00820#pg10>. Ethical considerations confirm that all experiments were conducted using LLM agents with no human subjects involved<ref:2606.00820#pg10>. The paper is partially supported by NSFC 62472289, 62532007, 62502034, and Guangdong Province Key Laboratory of Popular High Performance Computers 2017B030314073<ref:2606.00820#pg10>. The references include Solomon E Asch. 1956, Chi-Min Chan et al. 2024, Justin Chen et al. 2024, Young-Min Cho et al. 2025, Hyeong Kyu Choi et al. 2026, Min Choi et al. 2025b, Yu Cui and Hongyang Du. 1955, Morton Deutsch and Harold B Gerard. 1986, Yilun Du et al. 2024, Andrew Estornell and Yang Liu. 2024, Jack Hessel et al. 2023, Jie Huang et al. 2024a, Mehran Kazemi et al. 2015b, Mehran Kazemi et al. 1955b, Miles Turpin et al. 2023, Qineng Wang et al. 2024a, Yubo Wang et al. 2024b, Jerry Wei et al. 2023, and Hangfan Zhang et al. 1955].

--- Page 1 ---

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

1Xiqi Hao, 2Zengqing Wu†,

1Yu-Xuan Qiu, 2Chuan Xiao,

1Ruiqi Xu, 2Shuyuan Zheng, 3Jianbin Qin†

1Beijing Institute of Technology, Zhuhai, 2University of Osaka, 3

Shenzhen University

zengqing.wu@ist.osaka-u.ac.jp, qinjianbin@szu.edu.cn

Abstract

Multi-agent debate (MAD) is a promising strategy for improving LLM reasoning, but when agents converge on a shared answer, it is unclear whether that convergence reflects genuine deliberation or social compliance<ref:2606.00820#pg2> We show that the conventional answer flip rate conflates three distinct mechanisms: spontaneous instability, stance-induced conformity, and reasoninginduced persuasion<ref:2606.00820#pg2> Our three-source decomposition framework isolates each through controlled counterfactual conditions<ref:2606.00820#pg4>. In the primary MMLU-Pro setting, 37% of agent-question observations change under self-reflection alone, while robustness tests show substantial modeldependent instability across GPQA-Diamond and three model families<ref:2606.00820#pg2>. strict conformity is 29% in the primary setting and remains predominantly harmful across model replications (57–77% correct-to-wrong)<ref:2606.00820#pg2>. A controlled information-gradient experiment reveals that even vacuous reasoning is associated with 20–39% error adoption among resistant agents, with reasoning-like presentation carrying substantial persuasive weight<ref:2606.00820#pg2>. Harmful conformity can be predicted from Round 0 features (AUC = 0.79), and risk-targeted intervention reduces it by 13.6 percentage points (p. However, without correctness labels or self-reflection controls, reducing peer adoption does not improve accuracy, because harmful and beneficial influence cannot be distinguished<ref:2606.00820#pg2>.

The paper is organized around three questions: measure (RQ1: decomposing stance convergence), explain (RQ2: reasoning form vs. content), and mitigate (RQ3: prediction and intervention)<ref:2606.00820#pg2>. Our contributions are: • A three-source decomposition framework tested primarily on GPT-4o, with robustness replications across two additional model families and four benchmarks (MMLU-Pro, GPQA-Diamond, BIG-Bench Extra Hard (BBEH) (Kazemi et al.

Improvements for AI systems

  1. Decompose answer changes into three sources: spontaneous change, stance-induced conformity, and reasoning-induced persuasion to isolate genuine social influence from model instability. This allows systems to distinguish between spontaneous instability (where no external peer information is introduced) and stance-induced conformity (where the agent is stable under self-reflection but changes to a peer’s answer under stance exposure).

  2. Implement a risk-targeted intervention strategy based on Round 0 features, achieving a reduction in harmful conformity by 13.6 percentage points while improving accuracy by 14.63 percentage points in diagnostic settings where ground truth is available. This provides a mechanism to reduce negative peer adoption without sacrificing beneficial corrections when correctness labels are present.

  3. Develop a reasoning-like presentation detector that can identify persuasive weight, as reasoning-like presentation carrying substantial persuasive weight leads to 20–39% error adoption among resistant agents. This allows the system to specifically flag and treat structural reasoning forms as high-risk signals, even when the logical content is vacuous.

  4. Create a pre-interaction risk scoring model using features like peer disagreement structure and largest wrong-answer coalition to predict harmful conformity with an AUC of 0.79. This enables systems to prioritize intervention for agents exhibiting high risk before they engage in debate, focusing resources where they will have the most impact on accuracy.

  5. Design a mitigation prompt that requires agents to independently re-evaluate the question and your initial reasoning before changing an answer, specifically instructing them not to change merely because other agents selected a different option. This counteracts context-induced compliance by requiring agents to identify concrete reasons why their initial answer is wrong or incomplete.

Abstract

Multi-agent debate (MAD) among large language models (LLMs) is often judged by answer changes and convergence, yet these outcomes alone cannot establish that peers improved the answer. We study this gap with matched self-reflection, stance-only, and full-reasoning conditions. With five GPT-4o agents on MMLU-Pro disagreement questions, 37% of agent-question pairs change even without peer information. Among pairs stable under this control, 29.3% adopt a peer answer after seeing peer stances. These adoptions include corrections, new errors, and changes between wrong answers. The outcomes vary with correct-answer availability. When a correct peer answer is available, correction exceeds harmful adoption after accounting for peer support in the primary setting. Full reasoning improves individual accuracy but provides no additional majority-vote gain over self-reflection while using 57% more tokens. Responses to constructed peer materials vary across GPT-4o, DeepSeek-V4-Flash, and Qwen3-Plus. An answer-agnostic intervention suppresses both harmful and beneficial peer adoption without a detectable accuracy gain. Together, the results show that convergence is an ambiguous signal of value and that debate must be judged by what peer information contributes to the final group answer.

Sources

Related papers