The Autonomy Tax: Defense Training Breaks LLM Agents
cs.CR, cs.AI, cs.LG
Submitted: 2026-03-19
Updated: 2026-08-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks.
Terminology
Abstract
Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines. Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- SecAlign: Defending Against Prompt Injection with Preference Optimization
- Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
- Generative AI in Transportation Planning: A Survey
- The Llama 3 Herd of Models
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Deliberative Alignment: Reasoning Enables Safer Language Models
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Mistral 7B
- "Someone Hid It": Query-Agnostic Black-Box Attacks on LLM-Based Retrieval
- A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations
- Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
- CMOOD: Concept-based Multi-label OOD Detection
- Defenses Against Prompt Attacks Learn Surface Heuristics
- StruQ: Defending Against Prompt Injection with Structured Queries
- Mitigating Hallucinations in Large Language Models via Causal Reasoning
- JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model
- Ignore Previous Prompt: Attack Techniques For Language Models
- Computational complexity of three-dimensional Ising spin glass: Lessons from D-Wave annealer
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills