Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
cs.LG, cs.AI, cs.CY
Submitted: 2026-05-31
Updated: 2026-10-01
Comments: Second version, 40 pages, updated methodology and results; COLM AIW 2026 workshop
License: http://creativecommons.org/licenses/by/4.0/
The gist: Emergent misalignment (EM) occurs when narrow finetuning induces dangerous behavior outside the finetuning task.
Terminology
Abstract
Emergent misalignment (EM) occurs when narrow finetuning induces dangerous behavior outside the finetuning task. Detecting this shift through repeated behavioral evaluation is costly, motivating our checkpoint-level monitoring from internal representations. We define a fixed coordinate system from seven alignment-relevant activation directions and use it to track representational drift during LoRA finetuning of four open-source 7-9B language models. Finetuning drift in this space exhibits a dominant axis that explains 78.6% of variance and remains stable across datasets, extraction choices, and parameter-update capacities. Across 468 checkpoints from three EM-relevant held-out datasets, the resulting monitors attain 1.8% FNR, 2.0% FPR, and 0.989 AUROC, outperforming semantic, random, PCA, and SAE feature baselines. On a fourth dataset, a matched benign-dangerous control shows that substantial representational drift can also occur under benign finetuning, while changes across the 7D profile still distinguish dangerous from benign runs. Stress tests across two 14B models, full finetuning, longer training horizons, and misaligned starting states show that the signal can persist across shifts in training configuration, while reliable deployment may require recalibration.
Sources
- A General Language Assistant as a Laboratory for Alignment
- Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models
- Training Verifiers to Solve Math Word Problems
- Assessing BERT's Syntactic Abilities
- The Llama 3 Herd of Models
- Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Editing Models with Task Arithmetic
- Mistral 7B
- In-Training Defenses against Emergent Misalignment in Language Models
- LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
- Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models
- Natural Emergent Misalignment from Reward Hacking in Production RL
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- Large Language Models Often Know When They Are Being Evaluated
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Qwen2.5 Technical Report
- Convergent Linear Representations of Emergent Misalignment
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks