Lossless Anti-Distillation Sampling
cs.LG, cs.CR
Submitted: 2026-05-12
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model at drastically lower cost.
Terminology
Abstract
Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model at drastically lower cost. Existing defenses either modify the generation to degrade distillation performance, sacrificing response quality, or rely on behavioral detection mechanisms that can be readily bypassed through multi-account querying. In this work, we propose Lossless Anti-Distillation Sampling (LADS), which leaves the generation itself unchanged while substantially reducing the effectiveness of distillation. Concretely, LADS controls the latent randomness underlying inference through a coupling mechanism that preserves within-account generation independence while inducing cross-account dependence. By construction, each benign user, who typically holds only a single account, receives the same experience under LADS as they would without any defense, thereby enjoying a lossless experience. However, for a multi-account task-specific distiller, semantically similar queries submitted across different accounts are assigned coupled randomness, inducing dependence in the harvested data and thereby degrading the generalization performance of the distilled model. Using uniform convergence theory, we show that LADS provably degrades the distiller's generalization gap relative to standard i.i.d. sampling. Experiments on image generation, mathematical reasoning, and code generation confirm that LADS substantially degrades the performance of distilled students while preserving exact statistical fidelity for individual users.
Sources
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- The Llama 3 Herd of Models
- On the Partition Function and Random Maximum A-Posteriori Perturbations
- Measuring Mathematical Problem Solving With the MATH Dataset
- Elucidating the Design Space of Diffusion-Based Generative Models
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct
- Undistillable: Making A Nasty Teacher That CANNOT teach students
- Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks
- Qwen2.5 Technical Report
- Antidistillation Sampling
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Score-Based Generative Modeling through Stochastic Differential Equations
- Magicoder: Empowering Code Generation with OSS-Instruct
- Antidistillation Fingerprinting
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks