Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools

arXiv:2610.21309 · cs.CR, cs.SE · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools".

Elias: The gist The LLM4SAST framework is presented as a novel universal testing framework that leverages large language models and dynamic feedback to detect faults in static application security testing tools…

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: So we're looking at this paper now, "Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools." It seems like they are tackling a problem where SAST tools—those static analysis tools—are supposed to catch security flaws early, but the tools themselves can have faults that make them unreliable or hard to use.

Elias: Yeah, that's the core issue. If the tool is broken, you either miss real vulnerabilities or get flooded with false alarms, which kills your workflow. The authors are proposing a new universal testing framework called LLM4SAST to fix this reliability problem across different programming languages and tools.

Nadia: It’s interesting because they aren't just looking at one tool or one language; they want something that works universally, which is what the title suggests. They use large language models to help generate test cases for these SAST tools.

Priya: Before we get into the how, I want to know what this means for us in terms of real data. What kind of testing are they actually doing? Are we talking about just running a few tests, or something more robust?

Nadia: They are using large language models as the program transformation engine to produce test inputs that have rich semantics across different languages. They use three specific metamorphic relations—semantically preserving, program repair, and vulnerability injection—to guide how these variants are generated.

Elias: That's the meat of it. The paper describes MR1, which is about preserving the meaning while transforming code, MR2 where they try to make sure the modified program is still safe or at least doesn't introduce new problems, and MR3 where they actively inject known vulnerabilities into the code.

Nadia: And they are using an LLM to produce these multi-language variants based on three inputs: the SAST tool itself, an input program for that tool, and documentation from the tool describing those vulnerabilities. That’s a pretty comprehensive setup for testing.

Priya: That sounds ambitious. Since LLMs can sometimes make things up or hallucinate, how does the framework handle those mistakes when generating these test variants? Is that a major concern for them?

Title and authors: Elias: Yes, it is. They address this by using dynamic feedback-driven variant selection to filter out invalid variants. This means they combine runtime status and dynamic analysis results of the program variants during testing to make sure only valid ones move forward, which helps mitigate those hallucinations.

Nadia: And then there’s this other part where they try to find unique faults efficiently. They use a program-differencing-based clustering method, using QWen3-Embedding-0 point 6B embeddings of normalized edit scripts to group similar faults together and identify the unique ones faster.

Priya: So, if I understand right, they are trying to reduce the manual effort required for a developer to review all these generated variants? That sounds like something that would actually make a difference in terms of time spent.

Elias: Exactly. They claim this clustering method reduces the effort needed to review variants by about ninety-eight percent when it comes to identifying unique faults, which is quite a big number for reducing manual labor.

Nadia: The results they show are pretty compelling too. They evaluated LLM4SAST on six mainstream open-source SAST tools for both Java and Rust and found that it detected sixty-six unique faults.

Priya: Sixty-six unique faults across six tools is a significant number, but what about the quality of those findings? Did they confirm any of them as actual, real bugs that developers can actually fix?

Nadia: They did. They reported that thirty-two of those sixty-six faults were either confirmed by developers or fixed by them, which shows it’s not just generating noise.

Elias: And the efficiency is what really stood out when compared to prior work. They found that LLM4SAST finds three and a half times more faults while only needing about forty percent of the variants that older approaches required for fault detection.

Priya: That comparison against existing research, like Statfier, shows a significant improvement in efficiency and coverage. It suggests their approach is much better at finding bugs without having to generate an overwhelming number of tests first.

Nadia: The dynamic-feedback-driven variant selection heuristic also helped them reduce false positives by nearly forty-eight percent while still keeping the fault-finding capability strong, which is a big win for usability.

Title and authors: Elias: So, to summarize what they’ve done with "Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools," they've created a universal framework that uses LLMs to generate diverse, language-aware test inputs guided by specific metamorphic relations, and then filters those results dynamically using runtime data to find sixty-six unique faults across six tools.

Priya: It sounds like it’s moving SAST testing from something manual and slow toward something automated and much more thorough in terms of coverage. But I wonder where the limits are? What does this framework actually struggle with?

Nadia: They did mention that one internal threat to the validity is potential implementation bugs within LLM4SAST itself, which they've mitigated by releasing it publicly for verification. Externally, they flagged the conservative nature of differencing-based clustering as a limitation that might miss some faults in future evaluations.

Elias: That makes sense. They’re acknowledging that while the system is powerful, you still have to be careful about what the underlying model is actually seeing and how those clusters are being interpreted. But overall, it seems like they've put together a solid piece of work for testing security tools that spans different languages and features.

Nadia: So, to wrap up on this paper, "Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools," the implication is that developers building SAST tools across multiple languages can seriously consider incorporating this approach into their own testing suites to improve the reliability and usability of those tools.

Priya: I think the real impact here is shifting how we trust these security checks. If a tool can be tested this thoroughly by an LLM framework, it gives us more confidence that when a developer runs that tool later, the results are more reliable and less likely to be misleading.

Elias: Yeah, it’s about creating a better feedback loop between the code generation and the testing process. It’s about making sure those tests actually stress the tool in meaningful ways across different contexts.

Nadia: That covers what we have on this paper today. We'll keep an eye out for more papers that tackle these kinds of universal testing problems next time we air.

The paper's summary: Nadia: So, this paper is about taking static analysis tools that check code for bugs and trying to test them really well using large language models and dynamic testing to see if those tools are actually trustworthy across different languages.

Elias: Yeah, the core idea they're pushing is building a universal testing framework, LLM4SAST, that uses these models not just for generating code, but as a way to transform the inputs for the security checker itself.

Priya: So what exactly does this mean for us in terms of how we check our own tools? Are they claiming this can find bugs where other methods fail?

Nadia: They did show that it found sixty-six unique faults across six different SAST tools, and thirty-two of those were confirmed or fixed by developers.

Elias: The efficiency numbers are what really grab my attention; they claim the framework needs only about forty percent of the test variants that previous research required to find those same bugs.

Priya: Forty percent is a big reduction in testing time, but I want to know what that means for real-world data; how reliable is this method when it's generating these complex code transformations?

Nadia: They used three types of metamorphic relations to guide the transformation process—preserving meaning, program repair, and vulnerability injection—which makes the generated test inputs much more diverse.

Elias: That diversity is key because it means they aren't just testing one single type of input; they're exploring different ways a bug might manifest in that tool.

Priya: And you mentioned dynamic feedback driven selection to handle the LLM's tendency to hallucinate; how does that actually filter out those invalid variants in practice?

Nadia: It combines runtime status and dynamic analysis results from the test execution, so if a variant doesn't behave as expected during testing, it gets filtered out immediately.

Elias: That’s smart because it adds an external check to the LLM’s internal generation process; it grounds the language model in actual program behavior.

Priya: So, for someone who just listens to the show, what's the big picture here? Is this going to change how we approach building security tools?

Nadia: It suggests that developers of SAST tools can seriously incorporate this into their test suites to make those tools more reliable and usable across multiple programming languages.

Elias: The implication is that we can get a much better sense of the tool's actual behavior, not just what the static text suggests.

Priya: But they did flag a limitation; they admit that relying on differencing-based clustering for finding unique faults might still miss some undetected bugs in future evaluations.

Nadia: Yeah, so it’s a powerful starting point for testing these tools, but we still need to keep an eye out for those blind spots.

Elias: That makes sense; the system is strong because of the LLM's ability to explore space, but its clustering method is conservative about what it flags as unique.

Priya: So we have a framework that finds more bugs with fewer tests, but we still need to trust the way those unique faults are being categorized.

Nadia: Exactly; it’s a step forward in making security tool validation much more systematic and efficient.

Elias: And the next thing for this work is exploring using multi-agent systems instead of just prompt-driven models for generating those varied program programs.

The paper's improvements: Tom: So, we're looking at how they suggest making this LLM4SAST framework even better than what they just described by pointing out some future steps in their research.

Nadia: They mentioned using a multi-agent system instead of just relying on a single prompt-driven model to generate those program variants.

Elias: That makes sense; if you have different agents, you can get more varied and complex code transformations than one LLM alone could handle for all those metamorphic relations.

Priya: So, what does that actually mean for the quality of the test cases they generate? Does it help them avoid some of those semantic issues we talked about earlier?

Nadia: It should give them more control over generating inputs that respect the constraints defined by those three metamorphic relations—preserving meaning, program repair, and injection.

Elias: Precisely; more agents mean they can better manage the complexity of mapping those desired transformations onto a real code base.

Priya: That sounds like it could tighten up the loop where we get test cases from the LLM to make sure we’re not just getting high-level ideas that don't translate well to actual execution.

Nadia: Yeah, and they also talked about adopting a learning based approach, specifically fine-tuning an LLM to identify false positives.

Elias: Fine tuning sounds like a way to train the model specifically on what constitutes a false positive versus a real fault in those SAST tools.

Priya: That could significantly improve the precision of the framework because it learns from past mistakes rather than just following pre-set rules for filtering invalid variants.

Nadia: It’s about making the AI smarter about what it considers noise and what it considers a genuine security flaw in that context.

Elias: And they also proposed mining large code repositories to automatically discover and refine those metamorphic relations based on historical bug patterns.

Priya: So, instead of researchers manually defining all those transformation rules, the system could learn them by looking at massive amounts of existing code and past bugs?

Nadia: Yeah, that would automate the discovery part, helping to make the framework more universal across different programming languages naturally.

Elias: It shifts the burden from manual rule definition to pattern recognition in large datasets; that’s a big change for a cryptographic or security tool.

Priya: It makes sense if you want something truly universal, because you can't manually define every single transformation logic for every new language or tool.

Nadia: So the next step is moving toward more autonomous systems that learn the rules from data rather than having humans hardcode them into the framework.

Conclusion: Nadia: So, to wrap up this talk on "Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools," we're seeing a framework that uses large language models to generate diverse test inputs and then filters them dynamically using runtime results.

Elias: The paper shows how this approach finds sixty-six unique faults across six different tools, which is significant because it requires far fewer test variants than previous methods.

Priya: I still have some questions about the validity of those numbers; how much of that fault detection is from confirmed bugs versus what the AI just found by chance?

Nadia: The authors were pretty clear; thirty-two of those sixty-six faults were either confirmed by developers or fixed by them.

Elias: And they did point out that the dynamic filtering, while helpful against hallucinations, still relies on a certain level of conservative clustering that might miss some issues in future tests.

Priya: So it's a powerful tool for improving the efficiency of testing security tools right now, but we can't assume it catches every single edge case without more rigorous validation.

Nadia: Exactly; it’s an applied method that makes the testing process much more systematic across different programming languages and features.

Elias: The implication is that developers building SAST tools can seriously look at incorporating this into their own testing suites to boost the reliability of those tools.

Priya: For someone listening who isn't deep in security, it just means we have a new way to test the testing tools, making the whole process more efficient and less prone to getting stuck on false alarms.

Nadia: It’s about creating a better feedback loop between generating code variants and actually testing those variants against the tool's output.

Elias: We’re moving toward systems where we don't just generate tests, but where the system learns what kind of tests are needed based on historical data from codebases.

Priya: That moves us closer to a more robust way of verifying security tools that isn't limited by how much manual effort a human can put into generating those test cases.

Nadia: So we’ve seen how this LLM4SAST framework performs in practice, and it shows a clear path forward for universal testing.

Elias: It’s a solid piece of work demonstrating the potential of combining semantic program transformations with dynamic analysis for security tool validation.

HUAIEN ZHANG, YU PEI, HAO CHEN, SHIN HWEI TAN

The University of Hong Kong · The Hong Kong Polytechnic University · Concordia University

cs.CR, cs.SE

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/pmd/pmd

Project page: https://llm4sast.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The LLM4SAST framework is presented as a novel universal testing framework that leverages large language models and dynamic feedback to detect faults in static application security testing

Key concepts

Metamorphic Relations: These are defined rules that guide how a program is transformed to create new versions while keeping the original functionality or changing it in a specific way. LLM4SAST uses three types—isSemPreserving, isProgRepair, and isVulInjection—to systematically generate semantically diverse test cases for the SAST tool.
LLM as Transformation Engine: The large language model (LLM) acts as the core engine for generating program variants. Because LLMs are trained on vast amounts of code from many languages, they can understand and transform code across different programming languages based on prompts derived from analysis reports, allowing the framework to be universal.
Dynamic Feedback-Driven Selection: This heuristic is used to efficiently select the best program variants. It combines runtime status and dynamic analysis results to quickly eliminate invalid or erroneous variants generated by the LLM, thereby reducing false positives and focusing testing efforts on potentially faulty inputs.

Terminology

Clustering — 1× in this paper · explained in 2 episodes

Summary

The gist The LLM4SAST framework is presented as a novel universal testing framework that leverages large language models and dynamic feedback to detect faults in static application security testing tools across multiple programming languages.

Motivation and Problem Statement

Static application security testing (SAST) tools are crucial for the Shift-Left security paradigm by identifying vulnerabilities early in the development phase, but their own implementation faults undermine reliability and usability In contemporary software development lifecycle, the significance of static application security testing (SAST) tools is paramount Ensuring the correctness and reliability of SAST tools is a critical research topic because their implementation faults can result in both missed detections of security vulnerabilities and a high false positive rate that severely compromises their usability. Existing quality assurance paradigms for SAST tools include differential testing, metamorphic testing, random testing, and program generation. These conventional approaches suffer from limitations such as the inability to support multi-language SAST tools (C1), limited variant diversity by a predefined transformation set (C2), and a lack of an effective approach for filtering invalid variants (C3).

LLM4SAST Framework and Methodology

LLM4SAST is designed as a novel metamorphic testing framework that is universal, capable of testing SAST tools that support multiple programming languages and their evolving features. The core innovation involves harnessing large language models as a program transformation engine, capitalizing on their extensive pre-training on diverse code corpora spanning multiple programming languages. The framework takes three key inputs: (1) the SAST tool S to be tested by LLM4SAST, (2) an input programs for S, and (3) the documentation of S that contains vulnerability descriptions. It first invokes S to generate analysis reports, then constructs input prompts based on predefined templates to generate semantics-diverse program variants.

Metamorphic Relations and Variant Generation

LLM4SAST employs three types of metamorphic relations to guide program transformations from multiple views, significantly increasing the semantic diversity of the program variants. These relations are defined as follows: MR1 is isSemPreserving(psi) =⇒ RS (P) = RS (P′); MR2 is isProgRepair(psi) =⇒ RS (P′) ⊆ RS (P); and MR3 is isVulInjection(psi) =⇒ RS (P) ⊆ RS (P′). To generate variants, LLM4SAST utilizes context-guided prompt templates, which are constructed using comprehensive contextual information derived from analysis reports and documentation to mitigate hallucinations.

Validation Heuristics and Fault Identification

To mitigate issues caused by inherent drawbacks of LLMs, particularly hallucination [49], a dynamic-feedback-driven variant selection technique is used to efficiently eliminate invalid variants by combining runtime status and dynamic analysis results. Additionally, a program-differencing-based clustering method is proposed to efficiently identify unique faults by abstracting edit scripts using GumTree and employing QWen3-Embedding-0.6B for vector representations. The process involves comparing the maximum cosine similarity MaxSim between feature vectors and existing clusters to assign them or create new ones, followed by manual analysis of representative faults to categorize them into “Fixed”, “Confirmed”, and “Pending”.

Evaluation Results

LLM4SAST was evaluated on six mainstream open-source SAST tools for Java and Rust, detecting 66 unique faults, with 32 confirmed or fixed by developers. The results demonstrate that LLM4SAST identifies 3.4 times more faults while requiring only 39.8% of the variants compared to existing approaches. The dynamic-feedback-driven variant selection heuristic reduced false positives by 47.8% while preserving fault finding capability, and program-differencing-based clustering reduced the effort required to review variants by approximately 98%. LLM4SAST outperforms existing research approaches, such as Statfier, detecting 5 times more faults and requiring significantly fewer variants for fault detection.

Future Work

Future improvements could involve using a multi-agent system instead of directly employing prompt-driven LLMs to generate more diverse variant programs. Furthermore, adopting a learning-based approach, such as fine-tuning an LLM to identify false positives, can improve the precision of LLM4SAST. Another direction is enabling LLM4SAST to automatically discover and refine metamorphic relations by mining large code repositories and analyzing historical bug patterns. The primary internal threat to validity stems from potential implementation bugs in LLM4SAST, mitigated by public release for verification. The main external threat involves LLM hallucinations and the conservative nature of differencing-based clustering, which may harbor undetected faults in future evaluations.

Conclusion

LLM4SAST is a LLM-based universal testing framework that automatically generates input programs via semantic-preserving, program repair, and vulnerability injection transformations for discovering faults in SAST tools. Our experiments show that LLM4SAST outperforms the existing researches by finding more bugs yet iterating through fewer variants. Overall, LLM4SAST has discovered 66 bugs, of which 32 have been confirmed or fixed. Our results confirms that developers of SAST tools supporting various programming languages can incorporate our approach into their test suite construction to further improve the tools’ usability and reliability >.

Improvements for AI systems

Bold header: Universal LLM-driven SAST Tool Testing Framework

The improved system can detect 66 unique faults across six SAST tools by leveraging a framework that leverages large language models as a program transformation engine to produce semantically-rich, language-aware test inputs.

Bold header: Multi-Language and Evolving Language Support

The system can overcome the limitation of existing research where existing research efforts are predominantly designed for single programming languages by utilizing LLMs' training on diverse code corpora spanning multiple programming languages to provide a universal testing solution applicable across multiple programming languages.

Bold header: Enhanced Fault Detection via Diverse Metamorphic Relations

The system can identify deeper semantic faults by employing three types of metamorphic relations (semantics-preserving, program-repair, and vulnerability-injection) which the paper states allows for detection beyond equivalence preservation to uncover faults that may otherwise remain hidden.

Bold header: Robust Variant Filtering via Dynamic Feedback

The system can mitigate LLM hallucination by implementing the heuristic described as dynamic-feedback-driven variant selection, which involves comparing runtime status and dynamic analysis results of program variants during test execution to filter out invalid variants that fail to satisfy the constraints specified in the metamorphic relations.

Bold header: Efficient Fault Identification via Clustering

The system can reduce manual analysis effort by employing a program-differencing-based clustering method using QWen3-Embedding-0.6B embeddings of normalized edit scripts to identify unique faults, which reduces the effort required to analyze the evaluation results.

Bold header: Automated Fault Repair and Injection

The system can perform complex program modifications by utilizing LLMs for program repair and vulnerability injection transformation, allowing it to generate variants that are not only diverse but also capable of generating fixes, as illustrated by the prompt template for the program-repair metamorphic relation.

Abstract

Static application security testing (SAST) tools help detect potential software vulnerabilities, but they themselves contain faults undermining reliability and usability. Automated testing can address these issues, yet current techniques suffer from limited program transformation diversity and lack mechanisms to verify generated variants, leading to reduced fault detection and high manual effort. We present LLM4SAST, the first universal framework detecting faults in SAST tools using large language models and dynamic feedback. Our key insight is leveraging LLMs to understand and transform code in diverse programming languages. We design novel metamorphic relations with transformations to guide variant generation, using an LLM to produce multi-language variants that satisfy these relations. To reduce erroneous variants and identify unique faults, we propose two heuristics: dynamic-feedback-driven selection and program-differencing-based clustering. For each metamorphic relation, a report-centric test oracle detects variants that fail tests and identifies underlying faults by differentially analyzing SAST reports of the seed program and its variants. LLM4SAST identified 66 unique faults across six SAST tools, with 32 confirmed or fixed after being reported to developers, demonstrating effectiveness. The proposed heuristics eliminate 47.8% of false positives, and LLM4SAST detects 3.4 times more faults while requiring only 39.8% of the variants compared to existing approaches.