GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models

arXiv:2602.23826 · cs.CL, cs.LG · Submitted 2026-02-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models".

Jane: Here's the gist of the paper. The gist The GLUScope tool is an open-source tool for analyzing neurons in Transformer-based language models, focusing on gated activation functions to provide more fine-grained interpretability research.

Tom: What it addresses The paper presents GLUScope, an open-source tool for analyzing neurons in Transformer-based language models, intended for interpretability researchers who focus on more recent models than previous tools do; specifically we consider gated activation functions such as SwiGLU This introduces a new challenge: understanding positive activations is not enough.

Jane: Instead, both the gate and the in activation of a neuron can be positive or negative, leading to four different possible sign combinations that in some cases have quite different functionalities Accordingly, for any neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs.

Tom: How it works The tool is designed to account for the complexity introduced by gated activation functions like SwiGLU or GEGLU For each neuron, GLUScope records not only the strongest activations overall, but the strongest activations within each of the sign combinations.

Title and authors: Tom: So, we’re starting with the title and who wrote this paper. It’s GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models, by Sebastian Gerstner and Hinrich Schütze from LMU Munich. Jane That sounds pretty technical, but what it really means is they built a specific tool for interpretability researchers who are looking at newer models. Lu They’re tackling a gap because most existing tools often assume vanilla activation functions like ReLU or GELU, but modern language models use more complex things like SwiGLU. Tom So the challenge they highlight is that if you only look for positive neurons, you miss how much complexity there is when both the gate and input activations can swing in different directions. Jane So GLUScope aims to give researchers a way to see those four distinct functional behaviors by showing text examples for each sign combination and tracking their frequency.

Meng: From a deployment standpoint, I’m interested in the tool because it moves beyond just general neuron analysis into this specific gated structure. Tom Right, so the paper describes how this complexity shows up in the activations of SwiGLU neurons when you look at both parts separately. Lu They record not just the strongest activations overall, but specifically the strongest ones within each of those four sign combinations, which is a key technical detail.

Jane: And they make that data accessible through a website called GLUScope, so researchers can actually visualize this information for a few selected neurons. Tom So what we’re looking at here is how this tool helps bridge the gap between theoretical complexity in the model architecture and actual observable behavior in the activations.

Lalam: I see that they are releasing a dataset summarizing neuron activations and a website for visualization, which is useful because it provides concrete artifacts for others to build on.

The paper's summary: Tom: So, let’s move into what the paper actually proposes. The paper summarizes GLUScope as an open-source tool specifically designed to analyze neurons in Transformer models that use gated activation functions like SwiGLU or GEGLU. Jane It goes beyond just finding strong activations by focusing on these four possible sign combinations for the gate and input activation, which dictates different functionalities. Lu They record detailed information for each neuron, including the average value and the top-k or bottom-k values within each of those specific sign combinations, along with where those activations happened in the dataset.

Tom: It sounds like they are creating a structured way to quantify how often these four sign patterns appear across different pieces of text. Jane They release a dataset with this summary information and the GLUScope website for visualizing it on selected neurons. Meng What I like about that is that it’s not just abstract theory; they provide concrete data, which helps researchers move past just guessing what the neuron is doing.

Jane: And they provide text examples corresponding to each entry in that summary table, so you can actually see the text when a neuron activates under a specific sign combination. Tom That connects the mathematical finding directly back to how the model processes language, which is exactly what we need for interpretability research.

Lu: They also include code that lets you reproduce their artifacts or create new ones for other models or even more neurons, which makes it reusable.

The paper's improvements: Tom: Now let’s talk about the improvements they suggest over what was there before. The main improvement is explicitly accounting for the complexity introduced by gated activation functions like SwiGLU or GEGLU. Jane They move beyond just looking at positive activations to considering that both the gate and input can be positive or negative, leading to four different sign combinations that have distinct functionalities. Lu So instead of a single view, they track the strongest activations within each of those four sign patterns.

Tom: That means for any given neuron, you get a much richer picture than just seeing which ones fire strongly overall. Jane They provide summary statistics on the GLUScope website directly from that dataset, including frequency counts for each combination. Meng I mean, having those frequency counts is important because it tells you how robust or rare a certain activation pattern is across the entire test set.

Lu: They also show detailed information about each intermediate activation within those sign combinations, like the Swish(xgate), xin, and the final result Swish(xgate) · xin. Jane And they provide text examples for each of those specific entries in the summary table, which gives real context to the statistical data.

Tom: So these improvements make it much more precise for understanding neuron behavior in these advanced architectures compared to older tools.

Jane: And they show that this approach can lead to new insights, like finding a strong negative correlation between cos(win, wout) and the frequency of xgate being positive for any given neuron. Lu That kind of model-wide analysis is powerful because it links the internal neuron behavior to the overall model structure.

Conclusion: Tom: So to wrap up on GLUScope, this tool is a new way for interpretability researchers to look at neurons in Transformer models that use gated activation functions by tracking four distinct sign combinations of the gate and input activations. Jane It gives us a data-driven way to see how these complex activation patterns appear across different texts, supported by visualizations and frequency statistics. Lu This is a valuable contribution because it addresses the complexity introduced by SwiGLU and GEGLU, which traditional tools often miss entirely.

Meng: From my side, I’m interested in how this could be used to debug issues when deploying models at scale; having a way to see these specific activation patterns might help us understand where failures are happening in production. Jane That makes sense because understanding the behavior under different conditions is crucial for reliability. Tom Exactly, it moves us from saying "this neuron is active" to "this neuron is active when the gate and input are both positive."

Lalam: I’m pretty sure that GLUScope will be used a lot because it provides actionable data, not just theoretical findings.

Lu: Yes, and they’ve made the analysis reproducible by releasing the code, which is important for building on this kind of work.

Meng: It does sound like a solid piece of work focused on making sense of these specific model components.

Jane: And that’s where we wrap up our discussion on GLUScope, and we’ll be looking at new papers next time.

Sebastian Gerstner, Hinrich Schütze

LMU Munich and Munich Center for Machine Learning

cs.CL, cs.LG

Submitted: 2026-02-27

Updated: 2026-10-21

Comments: 6 pages for main body, 10 pages in total. 4 figures

Code: https://github.com/sjgerstner/gluscope

Project page: https://sjgerstner.github.io/gluscope

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 80/100

The gist: The gist The GLUScope tool is an open-source tool for analyzing neurons in Transformer-based language models, focusing on gated activation functions to provide more fine-grained interpretability

Key concepts

Gated Activation Functions (GLU variants): These are modern activation functions used in large language models, such as SwiGLU or GEGLU. They introduce complexity because both the 'gate' and the 'input' part of the function can be positive or negative, creating four different sign combinations that dictate how a neuron behaves.
Four Sign Combinations: Because gated activations have both a gate and an input component, there are four possible combinations of positive or negative signs for these parts. GLUScope tracks the activations for each of these four specific sign patterns to understand the unique functionality associated with each combination.
GLUScope Tool: This is an open-source tool and a website that visualizes neuron data. It records summary statistics, including frequency and average activations, for each of the four sign combinations across a model's dataset. It also provides text examples corresponding to these patterns for detailed inspection.

Terminology

Transformer — 2× in this paper · explained in 2 episodes
Activation Function — 1× in this paper · explained in 1 episode
Interpretability — 1× in this paper · explained in 15 episodes

Summary

The gist The GLUScope tool is an open-source tool for analyzing neurons in Transformer-based language models, focusing on gated activation functions to provide more fine-grained interpretability research.

What it addresses

The paper presents GLUScope, an open-source tool for analyzing neurons in Transformer-based language models, intended for interpretability researchers who focus on more recent models than previous tools do; specifically we consider gated activation functions such as SwiGLU This introduces a new challenge: understanding positive activations is not enough. Instead, both the gate and the in activation of a neuron can be positive or negative, leading to four different possible sign combinations that in some cases have quite different functionalities Accordingly, for any neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs.

How it works

The tool is designed to account for the complexity introduced by gated activation functions like SwiGLU or GEGLU For each neuron, GLUScope records not only the strongest activations overall, but the strongest activations within each of the sign combinations. Concretely, we release a dataset containing summary information on the activations of each neuron (of one model), and a website, GLUScope, for visualizing this neuron data for a few selected neurons.

Key components and artifacts

The released artifacts include several key components designed to support analysis

A dataset containing summary information on the activations of each neuron (of one model).
A website, GLUScope, for visualizing this neuron data for a few selected neurons.
The dataset we ran the model on.
Code that can be used to reproduce our artifacts or create new ones of the same kind (activation datasets for other models and/or visualizations for more neurons).

Data and Visualization

The activation dataset contains detailed information about each of the intermediate activations of a neuron. Specifically, for each neuron the xgate and xin activations can take four possible sign combinations. Within each sign combination, we record detailed information about each of the intermediate activations of the neuron (xgate, Swish(xgate), xin, and the final activation Swish(xgate) · xin): the average value, and the top-k (or bottom-k, depending on the sign) values together with the dataset indices where they appeared. The GLUScope website presents summary statistics directly read off this dataset as a table. These statistics include Frequency, which shows how often each combination appeared when running the model on the dataset, and Detailed information about each of the intermediate activations of the neuron (the terminology here largely follows TransformerLens). The text examples section presents text examples (highest absolute activations) for each of the entries in the summary table.

Usage and Insights

The tool is showcased through two usage examples that lead to new insights into neuron behavior. One example involves using the activation dataset for a model-wide analysis, where a strong negative correlation between cos(win, wout) and frequency of xgate > 0 of any given neuron is found. The second example demonstrates inspecting a neuron with GLUScope, which reveals that while the prediction based on weights might suggest one pattern, the actual activations often show different patterns depending on the signs of xgate and xin. Specifically, for the gate-in- activations we do find a pattern where again is usually a plausible next token. This finding could not have been discovered with a more traditional tool designed for vanilla activation functions, like Nanda’s (2022) Neuroscope.

Limitations

The limitations of GLUScope include that it cannot directly be used to analyze MoE models or non-Transformer models like Mamba. Moreover, the focus is on individual neurons, instead of other hypothesized causal variables like SAE features. The tool also does not address the problem of finding interpretable directions that do not correspond to individual neurons, as suggested by Elhage et al. (2022).

The paper concludes by releasing GLUScope, a tool for inspecting neurons in transformer language models. In the near future, the authors plan to extend the demo with more models and include weight-based analyses in each neuron page. The limitations of focusing on gated activation functions are also noted.

--- Page 1 ---

GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models Sebastian Gerstner & Hinrich Schütze LMU Munich and Munich Center for Machine Learning (MCML) Munich, Germany sgerstner at cis dot lmu dot de Abstract We present GLUScope, an open-source tool for analyzing neurons in Transformer-based language models, intended for interpretability researchers. We focus on more recent models than previous tools do; specifically we consider gated activation functions such as SwiGLU. This introduces a new challenge: understanding positive activations is not enough. Instead, both the gate and the in activation of a neuron can be positive or negative, leading to four different possible sign combinations that in some cases have quite different functionalities. Accordingly, for any neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs. We describe examples of how our tool can lead to novel insights. A demo is available at https://sjgerstner.github.io/gluscope 1 Introduction Transformer-based (Vaswani et al., 2017) large language models (LLMs) have shown impressive performance, but there is still much that is unclear about how they achieve it. This had led to large-scale interest and effort in understanding their inner workings, an endeavor initially called NLP interpretability and later mechanistic interpretability (Elhage et al., 2021; Saphra and Wiegreffe, 2024). ACL researchers have also been involved in these efforts, as shown for example by the existence of the interpretability and model analysis track at ACL conferences, or the popularity of the BlackboxNLP workshops (Saphra and Wiegreffe, 2024). As a part of this endeavor, several works have analyzed individual neurons,1 especially via text examples that strongly activate them (Dalvi et al., 2019; Geva et al., 2021; Nanda, 2022; Voita et al., 2024; Gurnee et al., 2024). We want to support these interpretability researchers, especially those working on neuron interpretation, by publishing a new tool for neuron analysis. There are several such tools already (see below, section 3), but there is a crucial gap to fill: they (often implicitly) assume vanilla activation functions such as ReLU, GELU1 We use neuron to refer to a hidden dimension inside the MLP layer. (Ramachandran et al., 2017), or Swish (Hendrycks and Gimpel, 2016); but modern language models (e.g., Groeneveld et al., 2024; Touvron et al., 2023) often use gated activation functions (a.k.a. GLU variants) such as SwiGLU or GEGLU (Shazeer, 2020). These activation functions introduce a new challenge: understanding positive activations is not enough. Instead, both the gate and the in activation of a neuron can be both positive and negative, leading to four different possible sign combinations. Each of these sign combinations will have different patterns of appearance. For this reason, gated neurons can have much more complex behavior than vanilla ones. Our tool, GLUScope, is the first to take this complexity into account: For each neuron, we record not only the strongest activations overall, but the strongest activations within each of the sign combinations. Concretely, we release the following artifacts (section 4): a dataset containing summary information on the activations of each neuron (of one model), and a website, GLUScope, for visualizing this neuron data for a few selected neurons. Additionally, we release the dataset we ran the model on, as well as code that can be used to reproduce our artifacts or create new ones of the same kind (activation datasets for other models and/or visualizations for more neurons). We showcase the utility of our tool by two usage examples (section 5) that lead to new insights into neuron behavior, including ones that could not have been achieved with other tools. 2 Preliminaries 2.1 Gated activation functions (GLU variants)We work on gated activation functions like SwiGLU or GEGLU (Shazeer, 2020). Gated activation functions are used widely; e.g., OLMo (Groeneveld et al., 2024) and Llama (Touvron et al., 2023) use SwiGLU, and Gemma (Gemma, 2024) uses GEGLU.

Improvements for AI systems

Improved neuron analysis for gated activation functions by recording four different possible sign combinations for each neuron, allowing researchers to understand how both gate and in activation can be positive or negative. This enables a more complex understanding of neuron functionality beyond simply looking at positive activations.
GLUScope website visualization of neurons will show summary statistics for each sign combination, including frequency and maximum, minimum, and average activations for intermediate states like hook post, hook pre linear, etc., based on the recorded dataset. This allows users to directly compare patterns across different activation sign combinations.
The system can perform model-wide analysis by calculating correlations between properties of neurons, such as finding a "strong negative correlation between cos(win, wout) and frequency of xgate > 0," which helps uncover relationships between neuron behavior and model structure.
GLUScope enables detailed inspection of specific neurons by showing text examples corresponding to each entry in the summary table for a given sign combination, providing context on when a neuron activates under specific conditions like when 'again' is already highly ranked.

Abstract

We present **GLUScope**, an open-source resource for analyzing neurons in Transformer-based language models (LMs), intended for interpretability researchers. We focus on the now-standard *GLU-based activation functions* such as SwiGLU, used in virtually all recent open-weights LMs. This introduces a new challenge: understanding positive activations is not enough. Instead, both the *gate* and the *in* activation of a neuron can be positive or negative, leading to four different possible sign combinations (*activation quadrants*) that in some cases have quite different functionalities (e.g. promoting vs. suppressing a specific set of tokens). Accordingly, for any neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs. Our resource can lead to novel insights: in particular, thanks to analyzing all four activation quadrants, it can surface interpretable behavior that other tools would miss. We also release the code used to create the resource, thus enabling researchers to build on it by adding more models, text corpora, or neuron pages.