TRACER: Trace-Regularized Training Against Indirect Prompt Injection
cs.LG
Submitted: 2026-10-13
Updated: 2026-10-13
Comments: 19 pages, 13 figures, 12 tables. Accepted to Findings of EMNLP 2026
Code: https://github.com/Mishrakshitij/TRACER
License: http://creativecommons.org/licenses/by/4.0/
The gist: Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior.
Terminology
Abstract
Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.
Sources
- Training Verifiers to Solve Math Word Problems
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Defeating Prompt Injections by Design
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Hidden-in-Plain-Text: A Benchmark for Social-Web Indirect Prompt Injection in RAG
- A Critical Evaluation of Defenses against Prompt Injection Attacks
- CORE: Collaborative Reasoning via Cross Teaching
- Qwen2.5 Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- PromptArmor: Simple yet Effective Prompt Injection Defenses
- Outcome-based Exploration for LLM Reasoning
- ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction
- Defense Against Indirect Prompt Injection via Tool Result Parsing
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks