Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot
cs.CR, cs.AI
Submitted: 2026-06-14
Updated: 2026-08-29
Comments: Accepted by EMNLP 2026 (Main)
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Large language models deployed as commercial APIs are vulnerable to model extraction attacks, while existing defenses either act too late or degrade utility for legitimate users.
Terminology
Abstract
Large language models deployed as commercial APIs are vulnerable to model extraction attacks, while existing defenses either act too late or degrade utility for legitimate users. We propose Knowledge Trap, a defense that redirects extraction attacks toward low-transferability knowledge through a Honeypot Knowledge Graph (HKG) and breadcrumb-guided exploration. Instead of blocking queries or perturbing outputs, Knowledge Trap consumes the attacker's limited query budget on knowledge with negligible downstream utility while preserving benign-user performance. Experiments in medical and financial domains show that Knowledge Trap reduces surrogate Agreement by 6.2% on average without degrading legitimate-user accuracy, outperforming existing defenses that impose measurable user impact. These results suggest that defending knowledge-space traversal is a practical direction for mitigating LLM extraction attacks.
Sources
- Stealing Part of a Production Language Model
- MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models
- Increasing the Cost of Model Extraction with Calibrated Proof of Work
- GPT-4 Technical Report
- The Llama 3 Herd of Models
- Model Leeching: An Extraction Attack Targeting LLMs
- GPT-4o System Card
- Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks
- Thieves on Sesame Street! Model Extraction of BERT-based APIs
- HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model
- GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
- NSmark: Null Space Based Black-box Watermarking Defense Framework for Language Models
- LLMPot: Dynamically Configured LLM-based Honeypot for Industrial Protocol and Physical Process Emulation
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills