Non-Gaussianity Reveals Interpretable Directions in Language Models

arXiv:2606.11722 · cs.LG, cs.AI, cs.CL · Submitted 2026-06-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Non-Gaussianity Reveals Interpretable Directions in Language Models".

Jane: The gist The introduction states that Independent Component Analysis (ICA) can serve as a compact lens for language-model interpretability by identifying non-Gaussian directions in activation geometry without requiring the training…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at "Non-Gaussianity Reveals Interpretable Directions in Language Models," it really boils down to this idea: we can find meaningful directions without training another dictionary by looking for non-Gaussian distributions in the residual stream activations >

Jane: The authors are essentially saying that because interpretable directions are selective on tokens and less Gaussian than random ones, we can use ICA as a compact lens to find them >

Lu: It’s about using non-Gaussianity as a direct unsupervised signal for discovering interpretable structure in activations rather than relying solely on indirect learning objectives like sparse reconstruction >

Meng: For those of us dealing with production models, the implication is that we might be able to audit model behavior or find specific biases by running this ICA Lens workflow without having to re-train a massive dictionary for every new layer >

Tom: The results show that these recovered directions are useful—they capture lexical, syntactic, and semantic patterns, and they are competitive with SAEs in terms of reconstruction and concept prediction >

Jane: But the paper also flags a limitation: the authors state that while their random component audit shows most sampled components admit usable working labels under their protocol, they still need to be careful about how those labels are interpreted in practice >

Lu: They also note that the analysis of polysemous words requires interpretation relative to the information available at the target position, which means context matters for every label they assign >

Meng: So it's not a magic solution where you just run this and get perfect answers; you still have to do some careful evaluation >

Tom: The main point is that non-Gaussianity is a direct way to find structure, but we have to use the ICA Lens workflow carefully, and the interpretation of what those components mean must be grounded in context >

Conclusion: Tom: So we’re wrapping up this look at "Non-Gaussianity Reveals Interpretable Directions in Language Models." This paper is essentially saying you don't need to train another huge dictionary like a Sparse Autoencoder to find useful directions in how language models work.

Jane: Right. The core idea here is that there’s a statistical signature—non-Gaussianity—in the activation patterns of language models that points directly toward those interpretable directions we want to find.

Lu: Specifically, they found that the projection distributions of natural text residual stream activations along these sparse autoencoder decoder directions are much more non-Gaussian than if you just picked random directions.

Meng: So what does this mean for us in production? Does it give us a shortcut to understanding why a model makes a specific decision?

Lalam: I’m confident that this non-Gaussianity is indeed a direct unsupervised signal for discovering the structure we're looking for, and they tested it across GPT-two Small, Gemma two 2B, and Qwen three point five 9B Base models <ref:2606.11722#pg2>.

Tom: The results show these recovered directions actually capture coherent lexical, syntactic, and semantic patterns when both humans and other AIs evaluate them. That’s a big deal for interpretability because it means the model isn't just showing us noise; it's showing us recognizable stuff.

Jane: And what they found is that this non-Gaussianity is positively linked to both human-label confidence and automated interpretability, which suggests we can trust these directions more.

Lu: The resulting coordinates are competitive with Sparse Autoencoders in terms of sparse reconstruction and concept prediction, but they do it much faster—fitting in minutes per layer and using way less memory than training those large dictionaries.

Meng: Faster fits are good for iteration speed, but what about the real world? Does this method work reliably when we try to deploy it on live traffic instead of these clean benchmark datasets?

Lalam: The authors tested their random component audit, and they found that most sampled components admit usable working labels under their protocol. That’s a promising sign for practical application.

Tom: It shows that non-Gaussianity is a direct way to find useful structure in activations without needing another training phase, which cuts down on the computational overhead of building those massive dictionaries.

Jane: So, we’ve seen how this works and what it can recover. The big question now is how much do these directions actually change how we think about model behavior or control?

University of Maryland

cs.LG, cs.AI, cs.CL

Submitted: 2026-06-10

Updated: 2026-10-19

Comments: This version has been substantially revised from v1, with a comprehensive re-evaluation of the experiments and extensive additional experiments and analyses

Code: https://github.com/HoagyC/sparse_coding

Project page: https://liusida.github.io/ica-lens-paper/future-projects.html

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 87/100

The gist: The gist The introduction states that Independent Component Analysis (ICA) can serve as a compact lens for language-model interpretability by identifying non-Gaussian directions in activation

Key concepts

Non-Gaussianity: This is a statistical measure used to determine if the data distribution in an activation direction is 'normal' (Gaussian) or not. In this context, high non-Gaussianity suggests that the signal along that direction is structured and specific, unlike random noise which tends to be Gaussian. It acts as a key indicator for finding meaningful patterns.
Sparse Autoencoder (SAE): SAEs are a type of neural network used to learn compact, efficient representations of data by forcing the internal activations to be sparse—meaning only a few neurons are active at any time. The paper uses directions derived from SAE decoders as a baseline for what an 'interpretable' direction should look like.
Independent Component Analysis (ICA): ICA is a statistical method that attempts to separate mixed signals into statistically independent components. It is used here as a tool to find the underlying, distinct directions in the high-dimensional activation space of language models without needing to train complex dictionaries like SAEs.

Terminology

Interpretability — 2× in this paper · explained in 15 episodes
Non-Gaussianity — 1× in this paper · explained in 3 episodes
Sparse Autoencoders (SAEs) — 1× in this paper · explained in 4 episodes

Summary

The gist The introduction states that Independent Component Analysis (ICA) can serve as a compact lens for language-model interpretability by identifying non-Gaussian directions in activation geometry without requiring the training of large overcomplete dictionaries like Sparse Autoencoders (SAEs).

How it works

ICA is motivated by the intuition that interpretable directions should be selective on tokens and look less Gaussian than typical random directions, which is measured through non-Gaussianity (Page 4).
ICALens introduces a practical workflow that combines an optimized GPU-parallel FastICA pipeline with LLM-specific stability recipes to enable stable, efficient, and auditable ICA analysis of LLM representations (Page 3).
The fitting pipeline includes three recipes: Row normalization reduces the influence of activation-norm outliers before whitening (Page 8), a p95-LIM rule accepts a fit when most components have stabilized but a small tail remains difficult to converge (Page 9), and adaptive refitting lowers the target component count for difficult layers (Page 10).

What ICA recovers

**. ICA does not return an arbitrary rotation of the residual stream (Page 4). It recovers directions that are substantially more non-Gaussian than random directions and public SAE decoder directions across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base (Page 11). This confirms that ICA components successfully recover the statistical objective of the method on LLM activations (Page 10). The effective receptive field (ERF) measures how much textual evidence is needed for a direction to activate (Page 12). ERF reveals a layer-wise shift from token-local components in shallow layers to broader context-dependent components in deeper layers (Page 13). High excess kurtosis is consistently found to be negatively correlated with the effective receptive field, suggesting that high kurtosis components are typically more local (Page 14). The annotation protocol uses ERF to estimate the expected scope of evidence for each component side (Page 15). This allows for a more grounded interpretation of components (Page 20). The random component audit shows that most sampled components admit usable working labels under this protocol (Page 39). The resulting labels capture lexical form, local syntax, phrase templates, discourse context, semantic ambiguity, long-range repetition (Page 18). The embedding-layer ICA analysis reveals that familiar analogy pairs are supported by overlapping component directions (Page 23). This suggests that embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24). The token-wise analysis shows that ICA directions often vary across short spans, including the financial context around bank, deposit, check, withdraw, and cash (Page 28). The analysis of polysemous words shows that the mixture must be interpreted relative to the information available at the target position (Page 19). The embedding-layer ICA reveals that different analogy families decompose in different ways, showing that familiar pairs are represented as a mixture of semantic, gender-associated, morphological, and surface-form directions (Page 23). The token-wise analysis shows that ICA directions track the financial interpretation of bank across related words such as deposit, check, withdraw, and cash (Page 28). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 23). The token-wise analysis shows that ICA directions vary across short spans, including the financial context around bank, deposit, check, withdraw and cash (Page 28). The embedding-layer ICA reveals that different analogy families decompose in different ways (Page 23). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24). The token-wise analysis shows that ICA directions track the financial interpretation of bank across related words such as deposit, check, withdraw and cash (Page 28). The embedding-layer ICA reveals that different analogy families decompose in different ways (Page 23). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24). The token-wise analysis shows that ICA directions track the financial interpretation of bank across related words such as deposit check withdraw and cash (Page 28). The embedding-layer ICA reveals that different analogy families decompose in different ways (Page 23). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24). The token-wise analysis shows that ICA directions track the financial interpretation of bank across related words such as deposit check withdraw and cash (Page 28). The embedding-layer ICA reveals that different analogy families decompose in different ways (Page 23). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24). The token-wise analysis shows that ICA directions track the financial interpretation of bank across related words such as deposit check withdraw and cash (Page 28). The embedding-layer ICA reveals that different analogy families decompose in different ways (Page 23). The embedding-layer ICA turns a coarse analogy view into a component-level explanation (Page 24).

Improvements for AI systems

Adaptive ICA fitting pipeline: Implement row normalization reduces the influence of activation-norm outliers before whitening and adaptive refitting lowers the target component count for difficult layers, enabling a more stable and efficient layer-wise analysis across modern LLMs.
Contextual decomposition diagnostic: Utilize the effective receptive field (ERF) to characterize ICA components, revealing a layer-wise shift from token-local components in shallow layers to broader context-dependent components in deeper layers, allowing for better layer description.
Interactive inspection protocol: Build an interactive explorer that keeps the relevant evidence for each component visible during analysis, supporting active hypothesis testing with targeted prompts and providing a structured annotation process where experts assign labels based on evidence-controlled, ERF-guided protocols.
Feature utility evaluation: Apply ICA to SAEBench metrics like sparse probing for predictive feature utility and Targeted Probe Perturbation (TPP) for selective feature intervention, demonstrating that ICA remains highly competitive with public SAEs and consistently outperforms PCA and ITDA.
Cross-model interpretability comparison: Use the ICA-SAE overlap analysis to audit public SAE labels, finding that some are incomplete or overly narrow by comparing them against ICA's behavior on real examples.

Abstract

Finding human-interpretable directions in language-model activations is important for understanding and controlling model behavior. These directions are usually discovered through indirect learning objectives such as sparse reconstruction, but it remains unclear what existing statistical structure distinguishes interpretable directions and whether that structure can be used directly for feature discovery. We find that non-Gaussianity provides such a signature: projection distributions of natural-text residual-stream activations along sparse autoencoder (SAE) decoder directions are substantially more non-Gaussian than those along random directions, with most contexts forming a concentrated background and recurring patterns often occupying one tail. This raises a natural question: if interpretable directions tend to have non-Gaussian projection distributions, can we discover them by searching for non-Gaussian directions directly in activation space, without training another sparse dictionary? To test this hypothesis, we introduce ICA Lens, a practical workflow for efficient discovery, reliable annotation, and faithful evaluation of non-Gaussian directions in LLM residual-stream activations. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find that: (1) the recovered non-Gaussian directions capture coherent lexical, syntactic, and semantic patterns under both human and LLM-based evaluation; (2) non-Gaussianity is positively associated with both human-label confidence and automated interpretability; and (3) the resulting coordinates are competitive with SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7-43 times less memory. These results show that non-Gaussianity is a direct unsupervised signal for discovering interpretable and useful structure in activations.