Circuit Claims Depend on What Is Extracted and How It Is Compared

arXiv:2607.18921 · cs.LG, cs.AI · Submitted 2026-07-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Circuit Claims Depend on What Is Extracted and How It Is Compared".

Jane: The gist The claim it supports depends on which circuit is reported and how two circuits are compared, because preserving behavior does not single out one circuit,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper "Circuit Claims Depend on What Is Extracted and How It Is Compared". Basically, it argues that when you try to identify which parts of a large model are actually responsible for a certain behavior by removing them, there isn't just one single circuit you find.

Jane: That sounds like the core idea: preserving behavior doesn't point to one specific set of components, and whether we report the compact circuit or something bigger makes a huge difference in what we actually learn about the model.

Tom: Exactly. The authors made this concrete using a synthetic Lean tactic-prediction benchmark, which is basically a controlled setting with fixed proof rules and randomized surface form to make sure they were testing the extraction itself, not just some random task.

Lu: What's really interesting here is how they set up the comparison. They vary four factors: model sparsity, whether attention heads are merged into single query and key nodes or kept separate, the allowed post-ablation loss for a pruning graph, and which supervised checkpoint starts the reinforcement learning process.

Meng: That sounds complicated to run at scale. I'm curious what they actually found about stability across all those settings. Does the core circuit description stay consistent no matter how you prune it?

Lalam: The paper explores this stability across different checkpoints, including dense and weight-sparse ones, and they look at atomic versus compositional proofs to see how the results hold up.

Jane: And what they found is that some things are pretty stable, like the ranking of conditions that initialize reinforcement learning. But other things change depending on whether you're looking at exact edges or attention head sets.

Tom: That's a key point because it means if you just report the exact edges, your claim might be unstable across different pruning strategies. The paper notes that for dense and seventy-five percent weight-sparse checkpoints, the exact edge lists don't consistently overlap as much as the attention head sets do <ref:2607.18921#pg2,dense and 75% weight-sparse>.

Lu: And they quantified this overlap using Jaccard@ten metrics <ref:2607.18921#pg2>. For instance, the attention-head set Jaccard@ten success was zero point six six six at the primary dense versus seventy-five percent weight-sparse comparison point, while the exact-edge Jaccard@ten success was only zero point five five four at that same point <ref:2607.18921#pg2,attention-head set Jaccard@10 success>.

Paper summary: Meng: So it means reporting the attention head set gives you a better overlap measurement than just looking at which specific connections are kept in the exact edge list under those conditions. What does that mean for deployment?

Tom: It means if you're building an automated system to find these circuits, focusing on the attention heads might be a more stable starting point than trying to pin down every single connection immediately. The paper also looked at how the query and key representations—single merged node versus separate nodes—affected graph sizes, and it said that change was only slight.

Jane: That's curious because representation learning is such a big part of transformer design. So, when you change the way you model heads, does it really matter much for the overall structure of what's preserved?

Lu: The authors say representing each attention head with separate query and key nodes rather than a single merged one changes graph sizes only slightly. But they did find that across query and key choices and loss budgets, all three extracted graphs—the core circuit, the support graph, and the fixed-loss pruning graph—keep the same ranking of conditions that initialize reinforcement learning.

Tom: So even if your definition of the circuit changes a little bit, the overall structure regarding when RL starts up stays consistent across those tested parameters. That's a solid piece of evidence for some stability.

Meng: Okay, so we have this stable ordering for RL initialization conditions, but what about the actual behavior? Does just having that stable ordering mean we can trust the circuit extraction to explain *why* something works?

Lalam: The paper connects circuit structure to behavior by looking at compositional proofs versus atomic proofs. They found that the largest gains in compositional task accuracy come with the largest bridge fraction—that's the share of nodes in the compositional circuit that are outside of those corresponding atomic circuits.

Jane: So, even when we look at complex tasks, if a component is outside its corresponding atomic circuit, it contributes most to that extra gain. It seems like behavior and structure are correlated in this way.

Paper summary: Tom: That association is interesting because the authors stress it's an observed link between behavior and core-circuit structure rather than a direct cause-and-effect claim. They see that when you ablate those bridge components in a sparse reinforcement learning model, the compositional accuracy drops while the atomic accuracy stays pretty stable.

Lu: That suggests that these extra components are important for the composition effect, but removing them doesn't totally destroy the behavior on smaller tasks, just less so.

Meng: From a production standpoint, if we only keep those bridge components in our pruning budget, we might lose the compositional benefit without losing the basic function of the atomic task. That gives us a trade-off to consider for efficiency versus performance gain.

Jane: So what does this all mean for how we think about these model circuits? It really hinges on which circuit you choose to report and how you compare it against others.

Tom: Right, so the whole point of "Circuit Claims Depend on What Is Extracted and How It Is Compared" is that the claim you make about a circuit's role isn't fixed by the behavior itself.

Lu: The paper gives us a reporting practice: you have to specify which graph you report, what extraction rule and pruning threshold you used, how query or key nodes were represented, and what comparison level supports your claim.

Jane: So for anyone trying to use this research for analysis or building tools, the crucial thing is being explicit about those choices because behavior alone doesn't define the interpretation of the circuit.

Tom: That's the practical advice: you have to be transparent about how you extract and compare things, or your mechanistic conclusion is shaky. It moves circuit analysis from a single answer to a set of well-defined interpretations based on your setup.

Meng: It's good that they laid out those specific requirements for reporting because without them, anyone could just pick the prettiest graph and make an unsupported claim about its function.

Lalam: I agree with Tom. The empirical claim supported by a circuit extraction depends on what is extracted and how it is compared.

Jane: It really shows that in complex AI systems, there isn't one universal mechanism hiding underneath everything; the interpretation depends entirely on the lens you use to look at it.

Conclusion: Tom: So, we’re talking about this paper titled "Circuit Claims Depend on What Is Extracted and How It Is Compared." Basically, it argues that you can't just look at a model and claim you found the mechanism for its behavior because which circuit you report and how you compare it matters a lot.

Jane: That makes sense. So if we find a set of components that helps the AI perform a task, but we pick one specific way to describe that set, does it change what we think the AI actually learned?

Lu: Exactly. The authors show this using a synthetic benchmark where they deliberately randomized things like the proof rules and surface form. This lets them test if the claim holds up under different extraction choices.

Meng: I’m still thinking about how this translates to actual deployment. If I'm trying to prune a model for efficiency, should I be focusing on the compact circuit or that bigger graph they mention?

Tom: The paper gives us a reporting practice for this. You have to state which circuit you report, how you extracted it, what your pruning threshold was, and how you compared it against others. Behavior alone doesn't tell you the interpretation of the model.

Jane: It sounds like the core message is that circuit analysis isn't a single answer. It’s a set of specific answers based on your setup. And this paper lays out exactly what those setups need to be transparent about.

Lalam: The finding they push is that for any claim you make, you have to be explicit about the extraction rule and the comparison level you use. If you don't say that, your analysis is shaky.

Tom: So it moves circuit extraction from just finding a neat little component list to being very deliberate about how we measure those components against each other. It’s a lot of fine-tuning for anyone building these tools.

Jane: Right, so the next thing we need to figure out is what kind of specific comparisons the authors used—because that's where all the real detail is hiding in this study.

Fudan University · Shanghai Innovation Institute

cs.LG, cs.AI

Submitted: 2026-07-21

Updated: 2026-10-15

Comments: 35 pages. Accepted at NeurIPS 2026. Extended version with appendices

Code: https://github.com/Stepuuu/circuit-extraction-stability

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The claim it supports depends on which circuit is reported and how two circuits are compared, because preserving behavior does not single out one circuit, and this unstated choice decides it.

Key concepts

Circuit Extraction
This process identifies a small set of model components whose presence is necessary to maintain a specific target behavior when other parts of the model are removed (ablation). The paper shows that this extraction is ambiguous because preserving behavior doesn't point to just one specific circuit.
Comparison Choices
The study tested how different ways of comparing extracted circuits—such as using exact edge lists versus routing-head summaries, or varying the allowed pruning loss budget—affect the results. These choices significantly influence which structural claims can be made about the model's behavior.
Reported Circuit Objects
The researchers analyzed three different types of extracted structures: a compact prediction-preserving subgraph, an extended support graph containing surrounding read/write/routing info, and a fixed-loss pruning graph. These objects differ in their scope and selection criteria.
Behavioral Correlation
The largest gains in compositional task accuracy are associated with the largest fraction of core circuit nodes that lie outside the corresponding atomic circuits. This suggests a structural relationship between the most effective components and how they contribute to complex tasks.

Terminology

Summary

The gist The claim it supports depends on which circuit is reported and how two circuits are compared, because preserving behavior does not single out one circuit, and this unstated choice decides it.

Circuit Extraction Ambiguity

Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior. The paper argues that this reading is underdetermined because preserving behavior does not single out one circuit, because the claim it supports depends on which circuit is reported and how two circuits are compared. This issue was made concrete in a synthetic Lean tactic-prediction benchmark where fixed proof rules with randomized surface form let differences between extracted circuits be attributed to these choices rather than to the task.

Stability Across Comparison Choices

The study investigated stability across four factors: model sparsity, whether each attention head is represented with a single merged node for query and key or with separate query and key nodes, the allowed post-ablation loss for the pruning graph, and which supervised checkpoint initializes reinforcement learning (RL). Some descriptions stay stable under this variation while others change, such as the overlap of exact, component-to-component edges being low and sensitive to these choices. Specifically, the set of selected attention heads and the circuit-size ranking of conditions that differ in which supervised checkpoint initializes reinforcement learning (RL) are stable across those same conditions.

Comparison of Circuit Objects

The paper reports three extracted circuit objects: the core circuit, a compact prediction-preserving subgraph; the extended support graph, a wider graph that also retains surrounding read, write, and routing components; and the fixed-loss pruning graph. These objects differ in which components they keep and how they are selected. The core circuit is a compact extracted subgraph selected because, after the other candidate components are ablated, the remaining components still preserve the target tactic prediction.

Key Findings on Comparison Metrics

The research found that routing-head sets are more reproducible than exact edge lists under the same extraction method for dense and 75% weight-sparse RL checkpoints. The head-level summary groups top routing edges by attention head identity before computing Jaccard@10, raising Jaccard@10 to 0.666 on the success split and 0.554 on the near-miss split. Furthermore, across query/key choices and loss budgets, all three graphs keep the same ranking of the RL initialization conditions by graph size.

Behavioral Correlation

The largest compositional-task gains are accompanied by the largest core-circuit bridge fraction—the share of compositional-task circuit nodes that lie outside the union of the corresponding atomic-task circuits. The RL conditions with the largest compositional-task accuracy gains also have the largest bridge fractions, but this is an observed association between behavior and core-circuit structure, not a causal claim.

Recommended Reporting Practice

The results suggest a reporting practice for circuit-extraction studies: state the reported graph, the extraction rule and pruning threshold, the query/key node representation, the comparison level supporting the claim, and any boundary or negative-control checks that bound the interpretation. The empirical claim supported by a circuit extraction therefore depends on what is extracted and how it is represented, pruned, and compared.

Scope and Limitations

The synthetic Lean benchmark provides fixed proof-state rules and atomic and compositional structures known by construction for auditing circuit extraction in a controlled setting. The practical benefit is methodological: studies that name which graph is reported, how it is extracted, and which comparison supports a mechanistic conclusion are better positioned to produce reproducible and comparable circuit analyses. The reporting practice recommended in the conclusion is intended to mitigate this risk, and extending the controlled comparison to deployed models would require validation across larger models, longer contexts, and more heterogeneous tasks.

Reproducibility Package

The source code and reproduction scripts are released at https://github.com/Stepuuu/circuit-extraction-stability. The package is designed to support reproduction of the main experimental pipeline and contains the synthetic data-generation and tokenizer scripts, supervised and GRPO training entry points, circuit-extraction scripts for the core circuit and factorized query/key graph objects, fixed-loss pruning sweep code, a minimal runtime module, a requirements file, and representative commands. The reported experiments do not redistribute third-party datasets or pretrained models as experimental assets.

Training Workload

The dominant supervised recipe in Table 3 corresponds to a 5B-token run of the shared 8-layer, 415.7M-parameter transformer which corresponds to approximately 1.2 × 10 19 training FLOPs under the standard dense-transformer estimate. The paper and supplemental package specify the model size, token budgets, batch sizes, gradient-accumulation settings, sequence length, evaluation cadence, and RL sampling settings that determine the training and analysis workload scale.

Boundary Tests

Boundary tests in Appendix H measure behavior under shift. Negative controls show that matched atomic controls preserve more target predictions after ablation than mismatched controls, while mismatched controls remain well above random. The controlled comparisons isolate sensitivity to extraction choices, with broader behavioral transfer left for future validation.

Final Conclusion on Description

The exact edge-list level of description is the least reproducible one, while coarser summaries stay stable. The empirical claim supported by a circuit extraction therefore depends on what is extracted and how the extracted graph is represented, pruned, and compared. The reporting practice recommended in the conclusion is compatible with either reading. The claim it supports depends on which circuit is reported and how two circuits are compared >

--- Page 5 ---

The gist Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior.

Circuit Extraction Ambiguity

Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior. The paper argues that this reading is underdetermined because preserving behavior does not single out one circuit, because the claim it supports depends on which circuit is reported and how two circuits are compared. This issue was made concrete in a synthetic Lean tactic-prediction benchmark where fixed proof rules with randomized surface form let differences between extracted circuits be attributed to these choices rather than to the task.

Comparison of Circuit Objects

The paper reports three extracted circuit objects: the core circuit, a compact prediction-preserving subgraph; the extended support graph, a wider graph that also retains surrounding read, write, and routing components; and the fixed-loss pruning graph.

Improvements for AI systems

  1. Improved circuit analysis robustness by stating that "A circuit-level claim is therefore well defined only once one states which circuit is reported, the pruning threshold used to extract it, and the level at which circuits are compared; otherwise these unstated choices, not behavior alone, decide it." This forces researchers to explicitly state their methodological assumptions for any mechanistic claim.

  2. Improved interpretation of model behavior by emphasizing that the empirical claim supported by a circuit extraction therefore depends on what is extracted and how the extracted graph is represented, pruned, and compared. This warns downstream users that an exact edge list may not be the most reproducible measure of a mechanism.

  3. Improved component identification by leveraging the finding that Routing-head sets are more reproducible than exact edge lists under the same extraction method for dense and 75% weight-sparse RL checkpoints, suggesting that focusing on attention head sets offers a more stable mechanistic view than relying on fine-grained edges.

  4. Improved circuit structure understanding by noting that the largest compositional-task gains are accompanied by the largest fraction of compositional-task circuit nodes that lie outside the union of the corresponding atomic-task circuits. This allows for targeted investigation into what novel components (bridge fraction) are responsible for significant behavioral improvements under reinforcement learning.

  5. Improved parameter sensitivity analysis by noting that the RL initialization condition plots aggregate means over five sparsity levels and three seeds per family-condition cell, enabling researchers to assess how the choice of supervised checkpoint initialization affects the resulting circuit structure across different model training distributions.

Sources

Related papers