LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

arXiv:2506.04019 · cs.SE, cs.CL, cs.LG, cs.PL · Submitted 2025-06-04 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LLMs are Brittle to Simple Code Transformations".

Jane: Here's the gist of the paper. The gist The CETBench dataset is constructed via transformations over programs to benchmark LLMs for code-equivalence checking, revealing that very simple code transformations can result in a significant drop in performance of SOTA LLMs for the task of code-equivalence checking How it works The CETBench dataset is constructed by taking a pair of programs from a repository and applying a random series of pre-defined code transformations to generate (non-)equivalent pairs.

Tom: This process involves starting with a repository of programs written to solve various programming tasks, constructing pairs that may solve the same or different tasks, and then subjecting each pair to a sequence of transformations such as variable renaming or swap of if-else blocks.

Jane: The perturbation pipeline in Algorithm 1 formalizes this procedure for generating perturbed code pairs via controlled semantic modifications.

Tom: The algorithm randomly decides whether the full sequence should preserve the original semantic relationship between the program pair or introduce a semantic break.

Title and authors: Jane: Well, this paper introduces CETBench as a novel benchmark constructed by applying random series of pre-defined code transformations over a repository of programs to check for code equivalence. The goal is to see if LLMs can accurately determine if two programs are functionally equivalent or not after these changes.

Tom: So it’s taking existing pairs, mixing in pairs that solve the same task versus pairs that solve different tasks, and then running those pairs through a sequence of transformations like variable renaming or swapping blocks.

Lu: They note that some of these transformations are semantic-preserving, or sp, while others are not preserving meaning at all, or np. This process generates the set of perturbed pairs we call CETBench.

Meng: The construction involves starting with a repository from competitive programming problems and creating pairs that might solve the same task or different tasks initially before applying these controlled semantic modifications to create the final dataset of perturbed pairs.

Tom: So it’s not just a static test; it’s a dynamic process where they are actively creating new, potentially broken, code examples for the LLMs to judge.

Jane: The resulting dataset is structured with a prefix that describes the initial relationship between the pair and a flag that indicates the net semantic effect of all those transformations.

Lu: They perform a detailed analysis of relative performance across different pair categories both before and after introducing these perturbations, which is what really sets this work apart from previous benchmarks like GPTCloneBench or EqBench.

Meng: It’s important to note that CETBench applies composed sequences of up to seven diverse transformations, often across distinct but algorithmically similar programs.

Tom: That depth of transformation sequence seems key; it’s not just one small change, it’s a whole chain of modifications that can confuse the model.

Jane: It really shows the limitations of older benchmarks because they were limited in scale and diversity, like EqBench which only had one hundred forty-seven equivalent and one hundred twenty-five non-equivalent C/Java pairs <ref:2506.04019#pg3,147 equivalent and 125 non-equivalent C/Java pairs>.

Lu: This work is pushing the evaluation toward models like GPT-4o, which are frontier models we're currently seeing used for testing these kinds of complex reasoning tasks <ref:2506.04019#pg1>.

Meng: It’s good to see them extend beyond just single-step transformations, which is where many simpler benchmarks fall short.

Tom: So the summary is that CETBench is a new way to stress test LLMs on code equivalence by systematically breaking those equivalences with controlled semantic modifications.

The paper's summary: Jane: The paper suggests a simple fine-tuning approach as a remedy for the performance drop they observed on these transformed pairs. They found that fine-tuning using a combination of both the perturbed and unperturbed datasets yields better results than just using one dataset alone.

Tom: So, they found that if you train the model on both normal code and this messy, perturbed code together, it helps boost its performance on both types of test sets.

Lu: They also suggest that for models like Mistral or Qwen that are weaker, explicitly asking them for a chain of thought helps them significantly in both the original pairs and the perturbed pairs when you prompt them without that extra guidance.

Meng: It sounds like the solution isn't just about better data, but about using specific prompting techniques to guide weaker models through this complex reasoning task.

Jane: They also suggest analyzing how sensitive different transformation types are; they found that logic-altering perturbations like boolean inversion or if-condition flipping have a slightly higher delta in performance drop compared to simple renaming.

Tom: So, the paper is giving us a way to tune our prompting strategy based on what specific type of code manipulation is causing the most trouble for a given model.

Lu: They also point toward model specialization, suggesting that different models might perform better under certain perturbation scenarios; for example, Deepseek-Coder showed high performance in some of those challenging scenarios.

Meng: If we can identify which models specialize well in handling specific types of code changes, it gives us a clearer path for deploying them where they’ll actually be useful in production.

Tom: So the authors are suggesting we move from a one-size-fits-all approach to more targeted fine-tuning and prompting based on the specific failure mode.

Jane: It seems like they are acknowledging that we don't need a single magic fix, but rather a combination of better data mixing and smarter ways to prompt the models.

The paper's improvements: Tom: Overall, they conclude that LLMs are definitely brittle to simple code transformations, especially when we look at how they handle code equivalence checking. The key finding is that their performance drops significantly on these perturbed datasets for almost all SOTA LLMs.

Jane: They emphasize that while models *can* work with perturbations, they absolutely require explicit supervision and training to effectively deal with this task well.

Lu: They wrap up by saying future work should include experimenting with additional datasets and figuring out how we can get models to provide explanations for their equivalence judgments.

Meng: I think getting those explanations is the next big thing because it moves us from just knowing *if* the answer is right to understanding *why* it was right or wrong.

Tom: So, they are looking toward better interpretability as a way to push beyond statistical pattern matching for this task of code equivalence checking.

Jane: It’s a solid piece of work because it clearly defines the challenge and points us toward needing more than just bigger models; we need better training strategies and better reasoning capabilities.

Lu: The CETBench benchmark gives us a concrete way to measure this brittleness, which is valuable for directing future research in representation learning for code tasks.

Meng: For practical deployment, this means we have a clearer idea of the kinds of adversarial inputs that could cause our deployed models to fail during equivalence checks in real traffic.

Tom: So it’s a good warning sign: simple changes are surprisingly hard for LLMs to handle semantically, and we need more targeted methods to make them reliable for this kind of judging.

Conclusion: Tom: So we've been looking at CETBench, which is this new benchmark designed to see how brittle Large Language Models are when checking if two pieces of code are equivalent after some simple changes have been made.

Jane: Right, so the core idea is they take pairs of programs—maybe two that solve the same problem—and they apply a random series of transformations, like renaming variables or swapping if-else blocks, to see if the AI can still tell them apart.

Lu: They constructed this dataset by extending existing work, taking stuff from CodeContests and creating these perturbed pairs labeled with prefixes that describe the initial relationship and flags that show what happened after the transformations.

Meng: What I find interesting is how they generate these pairs; they decide randomly whether to keep the original meaning or introduce a break in meaning at each step of up to Kmax perturbations. It’s a controlled way to break things.

Lalam: Lalam here, from my side, I see that the analysis shows a significant drop in performance across SOTA LLMs for code-equivalence checking when they look at these perturbed datasets.

Tom: That confirms what we suspected, Lu—the simple transformations cause a big hit to their accuracy on this task. But what about the fix? What’s the solution they propose?

Jane: The remedy they suggest is a fine-tuning approach, but it's not just fine-tuning on the perturbed data alone; they found that combining pairs from both the original and the perturbed datasets gives better results than training only on one set.

Lu: They show that LLMs do have some capability with these perturbations, but they absolutely need that explicit supervision to handle this task effectively.

Meng: From a production standpoint, if we want to deploy models for code checking, this tells us we can’t just rely on out-of-the-box performance; we gotta supervise the training process with this kind of mixed data.

Tom: And there's more than just fine-tuning. They also found that for models like Mistral and Qwen, explicitly asking them to follow a chain of thought gives them a significant boost in both the original and the perturbed pairs compared to just prompting without that extra guidance.

Jane: That makes sense; when the logic gets twisted by those transformations, they need that structured reasoning help to keep track of what they're supposed to be checking.

Lu: They also pinpoint specific types of transformations where models struggle more—like boolean inversion or if-condition flipping—showing where their understanding breaks down most severely.

Meng: So, we can start thinking about tuning our prompting based on the type of code change we expect to see in real-world scenarios, rather than just hoping the model figures it out.

Tom: It sounds like CETBench isn't just a test; it’s a roadmap showing us exactly what kind of data and training strategy we need to make LLMs actually reliable for these kinds of critical code checks.

Jane: It really highlights that even simple changes in code can have a big impact on how much an AI understands the underlying meaning.

Lu: So, while they point out limitations, like only testing Python and using a binary equivalence scheme, the main message is about how to supervise these models effectively.

Meng: I'm still cautious because this evaluation is restricted to one language; we need more breadth before we trust these results for general use.

Lalam: Lalam confirms that the paper demonstrates that with targeted fine-tuning and better prompting, the performance gap on those transformed pairs shrinks considerably for most models tested.

Tom: So, to wrap up, CETBench is a crucial benchmark showing us that code equivalence checking requires more than just raw pattern matching from LLMs; it needs careful supervision and tailored training data.

Jane: That’s what this paper suggests about how we should approach making these models more robust for real-world tasks.

Lu: The authors conclude that future work should focus on getting better explanations from the models themselves so we can understand *why* they think two programs are equivalent or not.

Meng: I agree, moving toward explainability is a necessary step before we can really deploy these tools in high-stakes environments.

Tom: So, CETBench gives us a clear idea of the brittle spots in current LLM code reasoning, and now we have some direction for how to improve them through better training and prompting.

Jane: Next time we talk about benchmarks, I want to look at something that shows how models handle visual reasoning under stress.

Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla

Indian Institute of Technology Delhi · IBM Research AI

cs.SE, cs.CL, cs.LG, cs.PL

Submitted: 2025-06-04

Updated: 2026-10-20

Comments: Published in ACL Findings 2026

Journal ref: Findings of the Association for Computational Linguistics: ACL 2026

DOI: 10.18653/v1/2026.findings-acl.2070

Code: https://github.com/google-deepmind/code_

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The CETBench dataset is constructed via transformations over programs to benchmark LLMs for code-equivalence checking, revealing that very simple code transformations can result in a

Key concepts

CETBench: A dataset created by applying random code transformations to program pairs to test how well LLMs can still tell if the programs are equivalent. It measures the robustness of an LLM's ability to check for code equivalence when the input code is slightly altered.
Code-Equivalence Checking: The task of determining whether two different pieces of source code perform the exact same function, even if they look syntactically different. This is a crucial test for understanding if an LLM truly grasps the underlying logic rather than just recognizing patterns.
Perturbation Pipeline: A formal process used to create new, slightly modified versions of original code pairs. It randomly applies a sequence of small changes, like renaming variables or swapping conditional blocks, to generate pairs that might be equivalent or not.
Semantic Understanding: The deep comprehension an AI has about what the code actually *does*, beyond just matching surface-level text. The paper suggests LLMs often lack this true semantic understanding, instead relying on statistical correlations found in their training data to guess equivalence.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes
Ablation — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 4 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Semantic Understanding — 1× in this paper · explained in 3 episodes

Summary

The gist The CETBench dataset is constructed via transformations over programs to benchmark LLMs for code-equivalence checking, revealing that very simple code transformations can result in a significant drop in performance of SOTA LLMs for the task of code-equivalence checking

How it works

The CETBench dataset is constructed by taking a pair of programs from a repository and applying a random series of pre-defined code transformations to generate (non-)equivalent pairs. This process involves starting with a repository of programs written to solve various programming tasks, constructing pairs that may solve the same or different tasks, and then subjecting each pair to a sequence of transformations such as variable renaming or swap of if-else blocks.

The perturbation pipeline in Algorithm 1 formalizes this procedure for generating perturbed code pairs via controlled semantic modifications. The algorithm randomly decides whether the full sequence should preserve the original semantic relationship between the program pair or introduce a semantic break. It then applies a sequence of up to Kmax perturbations, randomly selecting one of the two programs (s1 or s2) at each step and applying a randomly selected perturbation type, with the variant (semantic-preserving or semantic-non-preserving) determined by the overall goal for the sequence.

Dataset Construction and Categorization

The benchmark is constructed by extending the CodeContests dataset, which consists of approximately 10,000 competitive programming problems curated from platforms such as Codeforces. For each problem, solutions are grouped into clusters denoted as Pi. The entire collection of these clusters across all N problems is denoted P = P1,..., PN.

Each resulting pair from Algorithm 1 is assigned a structured label ⟨prefix, k, flag⟩. The prefix encodes the initial relationship of the pair before any perturbations were applied, with four possible values: id identical correct submissions, fe functionally equivalent but syntactically different solutions, ne one correct, one incorrect submission, and di correct submissions of different problems. The flag indicates the net semantic effect of the perturbations.

Analysis and Findings

The analysis on this dataset reveals that for a variety of SOTA LLMs, including both open source and closed source, performance drops significantly on the perturbed dataset. A specific finding is the drop in performance when the starting pair represents a single program. The analysis points to the fact that LLMs may still be far from what could be termed as a semantic understanding of underlying code, likely relying on statistical patterns for the task of code equivalence.

Remedies and Future Directions

To remedy this performance drop, a simple fine-tuning-based approach is presented to boost LLM performance on the transformed pairs of programs. The best results are obtained when the fine-tuning is done using a combination of pairs from the perturbed as well as unperturbed datasets. This demonstrates that LLMs do have the capability to work with perturbations but require explicit supervision to effectively deal with this task. Future work should include experimenting with additional datasets and thinking about whether models can be made to explain why they think two programs are equivalent (or not), resulting in better explainability.

Limitations

Several limitations remain, including the evaluation being primarily restricted to a single programming language (Python). The dataset uses a binary Yes/No labelling scheme for functional equivalence, which oversimplifies nuanced cases where equivalence may be partial, conditional, or input-dependent. Furthermore, real-world coding variations—such as advanced refactoring patterns, concurrency-related modifications, or differences in library usage—are not fully captured by the transformation set. The effectiveness of fine-tuning on unseen perturbation types or real-world scenarios remains unclear.

References

Ajmain I Alam, Palash R Roy, Farouq Al-Omari, Chanchal K Roy, Banani Roy, and Kevin A Schneider. 2023. Gptclonebench: A comprehensive benchmark of semantic clones and cross-language clones using gpt-3 model and semanticclonebench. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 1–13. IEEE

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. ArXiv

Sahar Badihi, Yi Li, and Julia Rubin. 2021. Eqbench: A dataset of equivalent and non-equivalent program pairs. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 610–614. IEEE

Ira D Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant’Anna, and Lorraine Bier. 1998. Clone detection using abstract syntax trees. In Proceedings of International Conference on Software Maintenance (Cat. No. 98CB36272), pages 368–377. IEEE

Saikat Chakraborty, Shuvendu Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy. 2023. Ranking llm-generated loop invariants for program verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9164–9175

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov., Heidy Khlaaf., Girish Sastry., Pamela Mishkin., Brooke Chan., Scott Gray,, and 34 others. 2021. Evaluating large language models trained on code. ArXiv

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990–998. PMLR

Yijiang Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126–10141

Aram Ebtekar. 2019. How to interpret contest ratings. Codeforces Blog. Accessed: 2025-05-18

Mark Gabel, Lingxiao Jiang, and Zhendong Su. 2008. Scalable detection of semantic clones. In Proceedings of the 30th international conference on Software engineering, pages 321–330

Robert Goldblatt and Marcel Jackson. 2012. Wellstructured program equivalence is highly undecidable. ACM Transactions on Computational Logic (TOCL), 13(3):1–8

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu,, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.

Improvements for AI systems

Automated code equivalence checking robustness: The LLM system can now be evaluated on its ability to distinguish between semantically equivalent and non-equivalent pairs even when simple transformations are applied, as demonstrated by CETBench revealing a significant drop in performance of SOTA LLMs for the task of code-equivalence checking.
Targeted performance boosting via fine-tuning: The AI system can be improved by fine-tuning on combined original and perturbed training sets, which show improvement in performance on both the original and perturbed test sets and are better than vanilla GPT-4o models.
Reasoning enhancement with Chain of Thought prompting: For weaker models like Mistral and Qwen, explicitly asking for a chain of thought helps them significantly, showing a significant boost in their performance in both original pairs as well as the perturbed pairs as compared to prompting without chain of thought.
Difficulty level generalization: The system's capability improves across problem complexity because the analysis shows that the drop [in accuracy] remains constant across difficulty levels which suggests that whether the problem is basic or complex, introducing perturbations impacts model’s reasoning in a similar way.
Targeted perturbation sensitivity analysis: The AI can be tuned to understand specific transformation types; logic-altering perturbations like boolean inversion (section 3.3.5), if-condition flipping(section 3.3.3), and if-else swapping (section 3.3.1) have a slightly higher delta, indicating that the model struggles to understand the transformed logic.
Model specialization for equivalence tasks: Different models can be optimized based on their strengths; for instance, models like Deepseek-Coder show high performance in certain perturbation scenarios, suggesting potential for specialized LLM deployment.

Abstract

We study how well LLMs can determine whether two programs are functionally equivalent. This is an important problem because benchmarking code equivalence helps assess LLM capability in tasks such as code rewriting and translation. To this end, we introduce CETBench - Code Equivalence with Transformations Benchmark - built from a repository of programs that may solve the same or different tasks. Each dataset instance is created by sampling a program pair and applying a random sequence of predefined code transformations, yielding either equivalent or non-equivalent pairs. Our analysis shows that even simple transformations cause a significant performance drop in state-of-the-art LLMs on code-equivalence checking. These challenges are further amplified in the cross-lingual setting when comparing programs written in different languages. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, supporting cross-lingual equivalence checking, the generation of program pairs with varying difficulty levels, and the application of diverse transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code.