Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders

arXiv:2601.02978 · cs.CL, cs.AI · Submitted 2026-01-06 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-01-06

Updated: 2026-10-13

Comments: The paper has been substantially restructured, and the new version has been submitted as arXiv:2607.26853

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Steering — 4× in this paper · explained in 2 episodes
• Activation Addition — 1× in this paper · explained in 1 episode
• Activation Steering — 1× in this paper · explained in 5 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
• Mechanistic interpretability — 1× in this paper · explained in 4 episodes

Sources

• Persona Vectors: Monitoring and Controlling Character Traits in Language Models
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
• Qwen3 Technical Report
• Instruction Tuning for Large Language Models: A Survey
• SocialEval: Evaluating Social Intelligence of Large Language Models
• Fine-Tuning Language Models from Human Preferences

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering