Unpacking Security Scanners for GitHub Actions Workflows

arXiv:2601.14455 · cs.SE, cs.CR · Submitted 2026-01-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Hi, I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Unpacking Security Scanners for GitHub Actions Workflows".

Elias: Here's the gist of the paper. The gist The first systematic comparison of 9 GitHub Actions Workflows security scanners reveals that no single scanner covers all identified security weaknesses, highlighting fundamental differences in their analysis strategies and leading to actionable recommendations for developers.

Nadia: Classification of Security Weaknesses The authors establish a classification of 10 common security weaknesses that can be found in GitHub Actions Workflows to provide a common ground for comparing the scanners.

Elias: This classification process involved manually analyzing documentation and source code to cluster rules into these ten common weaknesses.

Nadia: The identified weaknesses include Artifact Integrity Weakness (AIW), Control Flow Weakness (CFW), Excessive Permission Weakness (EPW), GitHub Runner Compatibility Weakness (GRCW), Hardening Gap Weakness (HGW), Injection Weakness (IW), Known Vulnerable Component Weakness (KVCW), Privileged Trigger Weakness (PTW), Secrets Exposure Weakness (SEW), and Unpinned Dependency Weakness (UDW).

Paper summary: Nadia: So we're looking at this paper called "Unpacking Security Scanners for GitHub Actions Workflows," and basically, the big idea is that you can't just pick one scanner and think you've got everything covered when it comes to securing those automation workflows. Elias Right, so the authors did a systematic comparison of nine different security scanners to see how they stack up in terms of what weaknesses they find, how good their detection is, and how fast they run.

Priya: So what's the main claim here? Nadia The paper claims that no single scanner covers all the common security weaknesses found in GitHub Actions Workflows because each one uses a totally different way of looking at things. Elias It says that this diversity in analysis strategies leads to major gaps in both what kinds of weaknesses are reported and how many there are.

Nadia: So why should we care about this difference? Priya Because it helps developers understand where their workflows might be vulnerable if they only rely on one tool for security. Elias It also gives us a clear picture of the landscape when we think about hardening our systems against those supply chain attacks that target GitHub Actions because they exploit things like excessive permissions or missing artifact integrity checks.

Nadia: It sounds like the paper is setting up a common language for talking about these security weaknesses, which is important since different tools might be flagging things differently. Priya Exactly. They establish a classification of ten common security weaknesses to use as a baseline for comparing all nine scanners because they need that common ground to make their comparison systematic and fair.

Elias: And they listed those ten specific weaknesses, which include things like Artifact Integrity Weakness, Control Flow Weakness, Excessive Permission Weakness, GitHub Runner Compatibility Weakness, Hardening Gap Weakness, Injection Weakness, Known Vulnerable Component Weakness, Privileged Trigger Weakness, Secrets Exposure Weakness and Unpinned Dependency.

Nadia: That's a lot of specific terms to keep track of. So the comparison itself isn't just about which scanner finds more things. Elias No, it’s about scope—which specific weaknesses they target—detection capabilities, and performance, which is how long it takes to scan a workflow.

Priya: I was looking at the performance part of this paper because that's where I usually get skeptical about scanner claims. Nadia It actually measured the time each scanner took to analyze a single workflow file using wall-clock time as the primary metric. Elias The results showed that most scanners are actually fast enough for seamless integration into a continuous integration pipeline, with median times ranging from two point seven one seconds for seven out of nine scanners <ref:2601.14455#pg2>.

Nadia: That's pretty good news for developers who are worried about slowing down their CI/CD process because adding multiple scanners adds only a modest runtime overhead relative to the security benefits they bring. Priya It also pointed out that Semgrep stands apart as the slowest scanner in terms of setup, with median elapsed times ranging from twenty-six point one three seconds to thirty-nine point one nine seconds.

Elias: So overall, this comparison of GitHub Actions Workflows Security Scanners is really showing that we need to think about combining complementary scanners instead of just trusting one tool blindly because they implement fundamentally different analysis strategies.

Nadia: That leads us into the conclusion section where they discuss what this means for how developers should actually set up their security posture. Priya They suggest a tiered approach: Fundamental Scanning using tools like actionlint and poutine, Essential Scanning by adding frizbee, and then Advanced Scanning which would include semgrep.

Elias: It’s about giving developers an option to choose between a fundamental level of security or more comprehensive scanning depending on what their specific security requirements actually are. Nadia That’s the practical advice here: don't just pick one scanner; look at what you need and combine them.

Priya: And they also mentioned that future work should focus on improving these scanners by analyzing the deeper network of transitive GitHub Action reuse because that seems like a gap they identified in their study. Elias It’s about looking beyond the immediate workflow file to see how actions are pulled in, which is where a lot of the complexity lies.

Nadia: So, to wrap up on "Unpacking Security Scanners for GitHub Actions Workflows," the paper provides a set of guidelines for developers to integrate these scanners based on their needs and an open science repository so others can reproduce the comparison. Elias The main implication is that because there's this diversity in analysis strategies, developers really need to be thoughtful about selecting which scanners they use to protect their workflows against those supply chain attacks.

Conclusion: Nadia: So we're wrapping up this talk on "Unpacking Security Scanners for GitHub Actions Workflows."

Elias: It looks like the paper is doing a systematic comparison of nine different security scanners against ten common weaknesses in those workflows.

Priya: What that really means is they're trying to figure out which tools are actually good and which ones are just guessing.

Nadia: Exactly, because they found that no single scanner covers all the identified weaknesses, showing these scanners use fundamentally different ways of looking at things.

Elias: And the authors provide some real evidence showing those different analysis strategies lead to big gaps in what kind of security issues get reported and how many there are.

Priya: It’s about moving past just trusting one tool and understanding the actual limitations when you try to secure your GitHub Actions pipelines.

Nadia: So, the authors offer some recommendations for developers on how they should set up their scanning based on whether they need a basic level of security or something more advanced.

Elias: They suggest three levels: fundamental scanning, essential scanning, and then advanced scanning including tools like Semgrep.

Priya: And the practical advice is to combine them so you get both a baseline and some deeper checks on your workflows.

Nadia: It’s about choosing scanners based on what your specific security requirements actually are instead of just picking the most popular one.

Elias: The paper also points out that these scanners are all fast enough for CI pipelines, though some, like Semgrep, take a noticeable amount of time to run.

Priya: So we know how they perform technically but it also flags that you can't ignore the runtime costs when you stack up multiple security tools.

Nadia: It’s about being smart with your security setup and understanding the trade-offs between depth and speed for your automation workflows.

Elias: That leads us next to looking at those specific nine scanners themselves, so we can see what each one actually does under the hood.

Université de Montréal

cs.SE, cs.CR

Submitted: 2026-01-20

Updated: 2026-10-19

Comments: 19 pages, 7 figures, 6 tables

Code: https://github.com/sparkrew/github-actions-security

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: The gist The first systematic comparison of 9 GitHub Actions Workflows security scanners reveals that no single scanner covers all identified security weaknesses, highlighting fundamental differences

Key concepts

Security Weaknesses Classification: The authors defined ten specific types of vulnerabilities found in GitHub Actions Workflows, such as Artifact Integrity Weakness or Secrets Exposure Weakness. This common framework was used to compare how well each scanner detected these distinct problems.
Scanner Comparison Metrics: Scanners were evaluated based on scope (what weaknesses they target), detection capabilities (how many weaknesses they find), accuracy (how well findings match manual checks), and performance (scanning speed). These metrics allowed for a structured, objective comparison of the tools.
Analysis Strategies: The core finding is that different scanners employ fundamentally different methods to find vulnerabilities. For example, two scanners might search for the same type of weakness but use entirely different logic. This diversity explains why they report different numbers of issues.
Recommended Scanning Levels: The paper suggests three tiers of scanning: Fundamental (using actionlint and poutine), Essential (adding frizbee), and Advanced (including semgrep). This tiered approach helps developers choose a security level that matches their needs, balancing security with pipeline performance.

Summary

The gist The first systematic comparison of 9 GitHub Actions Workflows security scanners reveals that no single scanner covers all identified security weaknesses, highlighting fundamental differences in their analysis strategies and leading to actionable recommendations for developers.

Classification of Security Weaknesses

The authors establish a classification of 10 common security weaknesses that can be found in GitHub Actions Workflows to provide a common ground for comparing the scanners. This classification process involved manually analyzing documentation and source code to cluster rules into these ten common weaknesses. The identified weaknesses include Artifact Integrity Weakness (AIW), Control Flow Weakness (CFW), Excessive Permission Weakness (EPW), GitHub Runner Compatibility Weakness (GRCW), Hardening Gap Weakness (HGW), Injection Weakness (IW), Known Vulnerable Component Weakness (KVCW), Privileged Trigger Weakness (PTW), Secrets Exposure Weakness (SEW), and Unpinned Dependency Weakness (UDW).

Comparison of Security Scanners

The study compared 9 state-of-the-art open-source workflow security scanners across scope (which security weaknesses they target), detection capabilities (how many weaknesses they detect), and performance (how long they take to scan a workflow). The analysis showed significant differences in the design of the scanners, leading to discrepancies in the number of reported weaknesses.

Detection Consistency and Coverage

The research questions guide a systematic differential analysis around three aspects: scope, detection consistency, and performance. For instance, poutine and zizmor essentially search for the same common weaknesses but implement different strategies. poutine reports 107 Injection Weaknesses (IW) in 61 workflows, searching for scripts that are run without proper checks and unsafe debug settings.

Runtime Performance Analysis

The performance evaluation measured the time each scanner takes to analyze a single workflow file using wall-clock time as the primary metric. The results showed that most scanners are fast enough for a seamless integration in a CI pipeline, with median times ranging from 2.71 seconds for 7 out of 9 scanners. Semgrep stands apart as the slowest scanner in the setup with median elapsed times ranging from 26.13s to 39.19s.

Recommendations for Developers

Based on the findings, the paper recommends three levels of scanner configurations: Fundamental Scanning (actionlint and poutine), Essential Scanning (adding frizbee), and Advanced Scanning (including semgrep). This approach allows developers to achieve a fundamental level of security or more comprehensive levels of security scanning. The study also suggests that future work should focus on improving scanners with the analysis of the deeper network of transitive GitHub Action reuse.

The paper provides a curated set of 9 state-of-the-art, open-source GitHub Actions Security Workflow scanners and guidelines for developers to integrate them depending on their security requirements. The study also provides an open science repository for reproducibility.

The paper concludes that the landscape of GitHub Actions Workflows security scanners is very diverse, with both general purpose and focused scanners implementing fundamentally different analysis strategies. This diversity highlights the importance of selecting scanners based on how they align with specific security requirements. The study also notes that GitHub itself is currently strengthening built-in security checks for GitHub Actions Workflows. The final classification of 10 common weaknesses forms the basis for differential analysis of the 9 security scanners. All evaluated scanners are fast enough to be integrated into continuous integration pipelines, with adding multiple scanners adding only a modest runtime overhead relative to the added security benefits. The study mitigates threats related to dataset size and absence of ground truth through comparative study and manual inspection. All data and code for this work is available at https://github.c om/sparkrew/github-actions-security. All these findings emphasize the importance of combining complementary scanners. The paper contributes to the GitHub Actions Workflows security body of knowledge with a systematic analysis of workflow security scanners. This work provides software development teams with concrete evidence about the strength of each scanner. The paper is available as arXiv:2601.14455v2 [cs.SE] 17 Mar 2026 <ref:2601.14455 pg1, arXiv:2601.

Improvements for AI systems

The improved system can perform a systematic differential analysis of security scanners to determine if no single scanner covers all identified security weaknesses, providing developers with actionable recommendations for developers to harden their GitHub Actions Workflows based on empirical evidence.
The system can implement three levels of scanner configurations—Fundamental, Essential, and Advanced—allowing users to choose between scanning with poutine and actionlint for fundamental security or including semgrep in addition to actionlint, poutine, and frizbee for advanced detection of complex weaknesses like code injections.
The system can provide runtime performance guarantees by predicting the execution time of various scanners on different workflow sizes, specifically noting that semgrep stands apart as the slowest scanner in our setup, which is crucial for ensuring fast feedback is essential in CI pipelines.

Sources

Granite: Granular Runtime Enforcement for GitHub Actions Permissions