Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation
cs.CR, cs.CL
Submitted: 2026-10-07
Updated: 2026-10-07
Comments: 20 pages, 5 figures, 9 tables
Code: https://github.com/luka-krixvon/routing-metadata-leakage
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off.
Terminology
Abstract
LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off. We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts. We run pre-registered studies on 1.7 million real requests (WildChat-1M, LMSYS-Chat-1M) with two cost/quality routers and a domain router, survey eleven systems' logging, and test post-processing defenses. At matched length, the shift's direction depends on category and router. For RouteLLM at the 50% operating point, harassment and self-harm requests reach the strong model 19 points less often than comparable ones on prompts unseen in exploration, medical requests (exploratory: LLM labels failed their gate) 31 points less often on distinct prompts (both post hoc), and sexual requests 10 points more often (secondary); the other router's four are negative. Twenty RouteLLM decisions separate frequent medical askers with AUC 0.71, exploratory and below the pre-registered primary endpoint's 0.75 (domain router: 0.92, an upper estimate). Per-category length-matched parity with accurate labels removes the gap on real traffic, costing at most 0.2 accuracy points on RouterBench (post hoc), where routers' gaps on sensitive subjects (13-42 points, pre-registered) exceed those of an oracle routing by realized accuracy gain (1-11, post hoc). Per-conversation stickiness, per-user budget bands, and pooled parity fail, the last as categories' shifts differ in size or sign. A post hoc exact per-user rate hides only even-prefix strong counts and forfeits most self-assessed routing value; it preserves odd-position decisions, from which a post hoc log attack reaches AUC 0.73 after 20 RouteLLM requests (exploratory).
Sources
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- AutoMix: Automatically Mixing Language Models
- RouteLLM: Learning to Route LLMs with Preference Data
- RouterBench: A Benchmark for Multi-LLM Routing System
- Register Bias in Complexity-Based Large Language Model Routing
- Privacy-Preserving LLMs Routing
- Whisper Leak: a side-channel attack on Large Language Models
- A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services
- From Privacy to Workflow Integrity: Communication-Graph Metadata in Autonomous Agent Interoperability
- Selection-Invariant Communication Compilers for Privacy-Aware Multi-Agent LLM Workflows
- AVEC: Bootstrapping Privacy for Local LLMs
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks
- Dynamic LLM Routers are Often Misguided
- Remote Timing Attacks on Efficient Language Model Inference
- RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
- Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use
- Equality of Opportunity in Supervised Learning
- Fair and Optimal Classification via Post-Processing
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills