SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

The gist The SYNAPSE framework proposes a systematic, training-free method for understanding and stress-testing the internal behavior of Transformer models across different domains by extracting per-layer [CLS] representations and applying controlled, inferencetime interventions.;

How it works

SYNAPSE is a framework that extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference to enable controlled, repeatable experiments on internal representations without altering the original modelSYNAPSE extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference to enable controlled, repeatable experiments on internal representations without altering the original model

The framework operates as a cyclic workflow consisting of six steps:

1. Ranking: Compute global and class-specific rankings on a held-out data splitRanking: Compute global and class-specific rankings on a held-out data split

2. Selection: Identify the top-k indices based on the desired p and scopeSelection: Identify the top-k indices based on the desired p and scope

3. Intervention: Register PyTorch forward hooks to intercept and modify activations during inferenceIntervention: Register PyTorch forward hooks to intercept and modify activations during inference

4. Inference: Run the model on the test set and record performance metricsInference: Run the model on the test set and record performance metrics

5. Cleanup: Remove all active hooks to restore the model to its original stateCleanup: Remove all active hooks to restore the model to its original state

6. Verification: Recompute the baseline to ensure no permanent changes were made to the weightsVerification: Recompute the baseline to ensure no permanent changes were made to the weights

Key Components and Strategies

The framework is designed for systematic analysis by introducing three main blocks:

1. Explainability Block: This foundational component performs a non-destructive extraction of internal activations, specifically extracting per-layer [CLS] activationsExplainability Block: This foundational component performs a non-destructive extraction of internal activations, specifically extracting per-layer [CLS] activations

2. Analysis Block: This block converts probe weights into global and per-class rankings, mapping top-k neuron indices to specific layers and hidden dimensions to enable reproducible, scope-controlled interventionsAnalysis Block: This block converts probe weights into global and per-class rankings, mapping top-k neuron indices to specific layers and hidden dimensions to enable reproducible, scope-controlled interventions

3. Adversarial Block: This block implements a variety of inference-time perturbations via PyTorch forward hooks, including global undirected silencing, label-directed silencing, and per-class silencingAdversarial Block: This block implements a variety of inference-time perturbations via PyTorch forward hooks, including global undirected silencing, label-directed silencing, and per-class silencing

Intervention Mechanisms

The Adversarial Block features three complementary intervention mechanisms to probe functional redundancy and sensitivity:

(i) Global undirected silencing targets the top-k neurons from the global ranking to measure overall sensitivityGlobal Undirected Silencing: Targets the top-k neurons from the global ranking to measure overall sensitivity

(ii) Global directed silencing selects neurons that have the strongest influence on a specific target label according to the linear probe, but applies a zeroing interventionGlobal Directed Silencing: Selects neurons that have the strongest influence on a specific target label according to the linear probe, but applies a zeroing intervention

(iii) Per-class silencing focuses exclusively on the top-k neurons belonging to a specific class’s ranking to probe class-conditional brittleness and potential leakage between labelsPer-Class Silencing: Focuses exclusively on the top-k neurons belonging to a specific class’s ranking to probe class-conditional brittleness and potential leakage between labels

Other stress tests implemented include logit bias, Gaussian noise injection, and weight manipulationThis work introduces three complementary intervention mechanisms: (i) global undirected silencing, which removes the top-k neurons according to a global ranking to measure overall sensitivity; (ii) global directed silencing, which selects globally important neurons with maximal influence on a target label to perform label-aware analysis; and (iii) per-class silencing, which suppresses neurons associated with a specific class to directly probe class-conditional brittleness

Experimental Validation and Findings

Experiments were conducted across two domains: malware detection using system-call sequences and emotion classification using the GoEmotions corpusTo validate the framework’s versatility across different domains, two distinct datasets are employed. First, a tokenized version of the MalwSpecSys dataset is used, containing system call sequences labeled as Normal, Bashlite, Bdvl, RansomwarePoC, or TheTickFirst, a tokenized version of the MalwSpecSys dataset is used, containing system call sequences labeled as Normal, Bashlite, Bdvl, RansomwarePoC, or TheTick

Second, the GoEmotions corpus is evaluated using a monologg/bert-base-cased-goemotions-original modelSecond, the GoEmotions corpus is evaluated using a monologg/bert-base-cased-goemotions-original model

The results consistently indicate that task-relevant information is distributed across broad and overlapping sets of neurons rather than concentrated in a small number of isolated unitsAcross tasks and model architectures, the results consistently indicate that task-relevant information is distributed across broad and overlapping sets of neurons rather than concentrated in a small number of isolated units

A key finding is that neuron-level interventions tend to produce gradual performance degradation, requiring increasingly large fractions of ablated neurons to induce substantial failuresAcross tasks and model architectures, the results consistently indicate that task-relevant information is distributed across broad and overlapping sets of neurons rather than concentrated in a small number of isolated units. Consequently, neuron-level interventions tend to produce gradual performance degradation, requiring increasingly large fractions of ablated neurons to induce substantial failures

The framework reveals architecture- and class-dependent sensitivities, where targeted interventions on specific neuron subsets can disproportionately affect certain decision pathwaysAt the same time, the analyses reveal architecture- and class-dependent sensitivities, where targeted interventions on specific neuron subsets can disproportionately affect certain decision pathways

The work demonstrates that small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profilesIn contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles

Overall, the findings show that SYNAPSE supports interpretable inspection of internal representations and acts as a practical tool for stress-testing robustness in both cybersecurity and NLP applicationsOverall, the findings show that SYNAPSE supports interpretable inspection of internal representations and acts as a practical tool for stress-testing robustness in both cybersecurity and NLP applications</ref:5

Limitations

The study notes several limitations, including the focus exclusively on [CLS] representations for tractability, omitting token-level structure and attention pathwaysIn many cases, large fractions of neurons must be silenced to match the effect of simple input-space perturbations, underscoring the intrinsic robustness of the evaluated models to localized neuron ablations and the distributed nature of their internal representations.

1. Automated Robustness Budget Estimation: The system can quantify how much targeted ablation is needed before performance breaks by tracking performance degradation to correlate with silencing key neurons across different architectures and domains. This provides a concrete failure budget for internal representation manipulation, moving beyond simple input-space perturbation analysis.

2. Architecture-Agnostic Robustness Profiling: The framework enables cross-domain experimental validation, allowing direct comparison of neuron-level behavior across heterogeneous modalities like malware detection and emotion classification without retraining or task-specific pipelines. This reveals a consistent, domain-independent organization of internal representations.

3. Identifying Heterogeneous Specialization: The system can expose heterogeneous specialization patterns by analyzing how different models allocate capacity, showing that each model develops idiosyncratic islands of specialization rather than uniform distribution across labels. This guides the development of more interpretable models by highlighting areas where certain decision pathways are disproportionately sensitive to targeted perturbations.

4. Complementary Vulnerability Mapping: The framework can map complementary vulnerability profiles by comparing the effects of neuron silencing with structured manipulations in weight or logit space, such as Logit Bias (TH-4) and Class-Directed Output Reweighting (TH-2). This reveals attack surfaces that are not captured by neuron-centric analysis alone.

5. Training-Free Stress Testing: By employing forward hooks, the system performs controlled, repeatable experiments on internal representations without altering the original model, ensuring that all analyses are conducted in a non-destructive intervention setting. This allows for systematic evaluation of robustness across tasks and architectures efficiently.

In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework (requiring no retraining of the base model) for understanding and stress-testing the internal behavior of Transformer models across domains. SYNAPSE extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled, repeatable experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. SYNAPSE reveals a consistent organization of internal representations across the two evaluated domains, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable fine-grained, label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust and interpretable Transformer models.