Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs

arXiv:2605.27715 · cs.CL · Submitted 2026-05-26 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond Input Understanding".

Jane: Here's the gist of the paper. The gist The gap between strong mathematical reasoning performance in English and lower reliability in many low- and medium-resource languages is often explained by a failure to understand non-English problem statements, but this view is incomplete because controlling the model’s reasoning language can substantially reduce accuracy, suggesting that language also affects reasoning execution itself.

Tom: Input-Reasoning Language Settings The study evaluates four input-reasoning language settings to disentangle input-side understanding from trace-side reasoning execution, including the en → x setting where the problem is in English but the reasoning trace is generated in a target language x.

Jane: The analysis shows that "performance is strongly influenced by the language of the reasoning trace, not only by the input language".

Tom: This effect is most severe when "medium- and high-difficulty problems require low-resource reasoning traces".

Paper summary: Tom: So, we’re talking about the paper "Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs." The main idea here is that we usually think the gap between strong English math skills and weaker performance in other languages is because the model just doesn't understand what's written in that language.

Jane: That’s what most people assume, right? They think the problem statement itself is a barrier. But this paper argues that view is incomplete because controlling which language the model uses for its actual reasoning can really hurt accuracy, suggesting language affects how the reasoning happens internally.

Lu: Exactly. They show that even when you give the problem in English, forcing the model to generate its steps in a different target language can substantially reduce performance. That points to something happening during execution rather than just understanding the initial input.

Meng: So, this isn't just about translating the prompt; it’s about how the AI processes those intermediate steps when it switches languages during its work. That makes sense for deployment, because if a model gets confused mid-process, that’s a failure mode we need to catch.

Lalam: I see what they are doing with this "Beyond Input Understanding" title—it shifts the focus from the input side to the trace side of reasoning execution itself. They introduce something called DATG to look at that process structure directly.

Tom: Right, and the core idea behind DATG is mapping those long reasoning traces into something language independent, like mathematical anchors and dependencies. This lets them check if those target-language traces are actually hitting all the necessary mathematical nodes and respecting the required steps.

Jane: So they’re not just looking at whether the final answer is right, but whether the path taken to get there follows a correct mathematical structure, regardless of which language it’s written in. It seems like a really precise way to diagnose execution failure.

Conclusion: Tom: The paper "Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs" suggests that the performance gap isn't just about input understanding; it’s also about failures in how the model executes its reasoning traces when switching languages. They introduce DATG to diagnose this by aligning target-language traces against language independent mathematical anchors and dependencies.

Jane: And their findings across twelve languages on models like Qwen3 show that non-English reasoning often struggles with both anchor coverage and dependency fidelity, especially in low and medium resource settings. This means the failure isn't uniform; it’s specific to the language resources available for tracing.

Lu: What I find most interesting is how they define those metrics—Committed Anchor Recall, Path Monotonic Fidelity, and Harmful Action Rate. These are concrete ways to measure structural integrity rather than just looking at the final score on a benchmark.

Meng: From an engineering standpoint, knowing that low-resource reasoning traces often omit key intermediate facts or break the dependency order gives us a clear target for debugging when we deploy these systems in those environments. It tells us exactly where the execution breaks down.

Lalam: The authors then propose two test-time controls, Loop-Retry and Formula-Retry, which they say consistently improve performance in those low-resource languages by targeting these specific failures exposed by DATG.

Tom: So, the implication is that we need to look beyond just the final answer when evaluating multilingual reasoning. We need to care about the structure and ordering of those generated reasoning traces instead of just focusing on whether they match a perfect English answer.

Jane: It suggests that future evaluation for multilingual systems should focus more on how stable and structurally sound the reasoning process is across different languages, not just the final output accuracy. That’s a big shift in how we think about these models.

Jiaqiao Zhang, Zhoujun Li, Raoyuan Zhao, Jian Lan, Thomas Seidl, Michael A. Hedderich, Hinrich Schütze, Yihong Liu

Southwest University · LMU Munich & MCML

cs.CL

Submitted: 2026-05-26

Updated: 2026-10-16

Comments: EMNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The gap between strong mathematical reasoning performance in English and lower reliability in many low- and medium-resource languages is often explained by a failure to understand

Key concepts

Input-Reasoning Language Settings: This setting tests whether the model's reasoning quality depends on the language of the initial problem versus the language used during the step-by-step reasoning process. The study found that even when problems are in English, using a target language for reasoning can substantially reduce accuracy.
Directed Acyclic Trace Graph (DATG): DATG is a framework that translates a model's reasoning trace into a graph of language-independent mathematical anchors and dependencies. This allows researchers to check if the generated steps cover all necessary math nodes, respect the correct order between steps, and avoid errors.
Committed Anchor Recall (CAR): CAR is a metric used by DATG to measure how well the reasoning trace covers all required mathematical concepts or 'anchors' needed for a correct solution. Low scores indicate that key intermediate facts are missing from the generated reasoning path.
Loop-Retry and Formula-Retry: These are test-time controls designed to fix execution failures identified by DATG. Loop-Retry resamples when early repetitive errors occur, while Formula-Retry provides a symbolic scaffold of known mathematical relations to guide the model toward a correct result.

Terminology

Trace — 4× in this paper · explained in 2 episodes
Low-Resource Languages — 2× in this paper · explained in 4 episodes
Accuracy — 1× in this paper · explained in 1 episode
Failure Modes — 1× in this paper · explained in 3 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Reasoning Traces — 1× in this paper · explained in 4 episodes

Summary

The gist The gap between strong mathematical reasoning performance in English and lower reliability in many low- and medium-resource languages is often explained by a failure to understand non-English problem statements, but this view is incomplete because controlling the model’s reasoning language can substantially reduce accuracy, suggesting that language also affects reasoning execution itself.

Input-Reasoning Language Settings

The study evaluates four input-reasoning language settings to disentangle input-side understanding from trace-side reasoning execution, including the en → x setting where the problem is in English but the reasoning trace is generated in a target language x. The analysis shows that performance is strongly influenced by the language of the reasoning trace, not only by the input language. This effect is most severe when medium- and high-difficulty problems require low-resource reasoning traces.

Directed Acyclic Trace Graph (DATG)

To diagnose trace-side failures, the paper introduces DATG, a Directed Acyclic Trace Graph framework that maps reasoning traces to language-independent mathematical anchors and dependencies. This allows researchers to align target-language traces with reference DAGs and measure whether they cover required mathematical nodes, respect dependency edges, and avoid harmful mathematical actions. The framework abstracts reasoning traces into languageindependent mathematical anchors and dependency edges.

Diagnostic Metrics and Findings

DATG is used to compute three key metrics: Committed Anchor Recall (CAR), Path Monotonic Fidelity (PMF), and Harmful Action Rate (HAR). The analysis across 12 languages on the Qwen3 series shows that non-English reasoning often degrades both anchor coverage and dependency fidelity, with the strongest failures appearing in medium- and lowresource languages. Specifically, "low-resource target-language reasoning traces often lose part of the derivational structure needed for correct mathematical reasoning: they may omit key intermediate facts, break dependency order, or introduce harmful mathematical actions".

Test-Time Controls

Motivated by these diagnostic findings, the authors propose two simple test-time controls targeting DATG-exposed failure modes: Loop-Retry and FormulaRetry. Loop-Retry resamples when early repetitive degeneration is detected, while FormulaRetry provides a compact symbolic scaffold of known quantities, relations, and formula templates without revealing the final result. These controls consistently improve target-language reasoning performance in low-resource languages.

Conclusion

The paper concludes that multilingual mathematical reasoning gaps arise not only from input-side understanding failures, but also from trace-side reasoning execution failures. The findings suggest that simple test-time controls partially recover accuracy and that future multilingual reasoning evaluation should look beyond final answers to the structure, ordering, and stability of generated reasoning traces.

--- Page 1 ---

--- Page 2 ---