Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregation

The gist: BAGEL, a modular, lightweight, and incrementally updatable framework for malicious prompt detection, achieves an F1 score of 0.92 by employing a bootstrap aggregation and mixture of experts inspired ensemble of fine-tuned models that outperforms larger baselines while maintaining computational efficiency

Motivation and Problem

Large Language Models (LLMs) are susceptible to malicious prompts that induce unsafe or policy-violating behavior through harmful requests, jailbreak techniques, and prompt injection attacks. Existing defenses face fundamental limitations where black-box moderation APIs offer limited transparency and adapt poorly to evolving threats, while whitebox approaches using large LLM judges impose prohibitive computational costs and require expensive retraining for new attacks Current systems force designers to choose between performance, efficiency, and adaptability.

BAGEL Framework

BAGEL is a modular, lightweight, and incrementally updatable framework for malicious prompt detection that employs a bootstrap aggregation and mixture of experts inspired ensemble of fine-tuned models. At inference, it uses a random forest router to identify the most suitable ensemble member, then applies stochastic selection to sample additional members for prediction aggregation. When new attacks emerge, BAGEL updates incrementally by fine-tuning a small prompt-safety classifier (86M parameters) and adding the resulting model to the ensemble.

Key Advantages of BAGEL

BAGEL offers three key advantages over prior approaches.

1. Computational Efficiency: Each member is a finetune of an 86M-parameter base promptsafety classifier, which keeps inference lightweight (for example, setting the selection size to 5 yields an effective footprint of 430M parameters).

2. Incremental Updates: When new attack datasets become available, LLM providers can finetune the same base model and add the resulting classifier to the ensemble, avoiding expensive end-to-end retraining of the full system while preserving performance on previously observed attacks.

3. Interpretability: The random forest router relies on transparent structural features allowing clear insight into which structural patterns of prompt attacks are most indicative of malicious intent.

Methodology

The framework modifies the classic bootstrap aggregating technique by training the ensemble models on entirely different datasets rather than subsets of the same dataset. It modifies the classic mixture of experts technique by routing the incoming prompt to a subset of more than one ensemble member by predicting an ideal member combined with a stochastic selection.

The process involves several steps:

1. Base Model: BAGEL utilizes a small base prompt-safety classifier as the backbone for each promptcop, such as Prompt Guard 2 (86M parameters).

2. Creation of Specialized PromptCops: Each dataset Di is partitioned into three disjoint subsets: Finetuning Set (70%), Calibration Set (10%), and Test Set (20%).

3. Dynamic Routing via Random Forest: A Random Forest classifier, R, is trained on structural features derived from prompt samples to serve as a router, predicting the index of the ideal promptcop i∗.

4. Selection and Aggregation: A subset Sx of size n is constructed as Sx = Mi∗ ∪ Mrand1, …, Mrandn-1, where Mrand are models drawn uniformly at random from P excluding Mi∗. The final maliciousness score yˆ(x) is computed via simple averaging: yˆ(x) = 1/n ∑Mj∈Sx pj(x).

5. Threshold Calibration: The optimal threshold τ∗ is updated by evaluating the F1 score across the cumulative calibration data to maximize it using a heuristic two-stage search strategy.

Results and Evaluation

BAGEL was evaluated on a diverse collection of nine largescale, real-world datasets covering multiple categories of malicious prompts. The results show that BAGEL outperforms popular black-box moderation APIs and competitive white-box baselines, achieving 0.095 Attack Success Rate (ASR) and 0.066 False Positive Rate (FPR), resulting in an F1 score of 0.922 when utilizing the previously mentioned 430M parameters subset which is 45% smaller than the entire ensemble. Performance remains robust after nine incremental updates, since the final F1 scores never drop below 0.92 even after introducing new datasets nine times. The performance curves show that increasing the selection size n further from n = 5 to n = 9 yields only marginal gains, confirming that it is unnecessary to select and request inference from every promptcop in the ensemble. BAGEL achieves comparable or superior results while being significantly more lightweight and transparent compared to methods like Perspective API and ToxicDetector.

Conclusion

BAGEL establishes a blueprint for sustainable LLM security systems that prioritize modularity, efficiency, and adaptability. It demonstrates that small, specialized models can match or exceed the performance of billion-parameter alternatives while significantly reducing computational overhead and enabling efficient incremental updates through simple fine-tuning and ensemble addition. This work shows that modular architectures with intelligent routing can deliver robust protection while remaining practical for real-world deployment.

--- Page 1 ---

The gist: BAGEL, a modular, lightweight, and incrementally updatable framework for malicious prompt detection, achieves an F1 score of 0.92 by employing a bootstrap aggregation and mixture of experts inspired ensemble of fine-tuned models that outperforms larger baselines while maintaining computational efficiency

Motivation and Problem

Large Language Models (LLMs) are susceptible to malicious prompts that induce unsafe or policy-violating behavior through harmful requests, jailbreak techniques, and prompt injection attacks. Existing defenses face fundamental limitations where black-box moderation APIs offer limited transparency and adapt poorly to evolving threats, while whitebox approaches using large LLM judges impose prohibitive computational costs and require expensive retraining for new attacks. Current systems force designers to choose between performance, efficiency, and adaptability.

Results and Evaluation

BAGEL was evaluated on a diverse collection of nine largescale, real-world datasets covering multiple categories of malicious prompts. The results show that BAGEL outperforms popular black-box moderation APIs and competitive white-box baselines, achieving 0.

1. Adaptive Ensemble Detection: Implement BAGEL to detect malicious prompts using an ensemble of smaller fine-tuned models, achieving high performance (F1 score of 0.92) while using significantly fewer parameters (430M effective) than billion-parameter baselines.

2. Incremental Defense Evolution: Allow the system to update incrementally by fine-tuning a small classifier on new attack data and adding it to the ensemble, avoiding expensive end-to-end retraining of the full system while preserving performance on previously observed attacks.

3. Interpretable Routing: Utilize a random forest router trained on structural features derived from prompt samples to identify the most suitable promptcop, providing clear insight into which structural patterns of prompt attacks are most indicative of malicious intent.

4. Computational Efficiency and Sustainability: Reduce computational overhead by employing a subset selection strategy where setting the selection size to 5 yields an effective footprint of 430M parameters, making it feasible for low-latency deployments.

5. Robustness Against Evolving Threats: Maintain robust performance across incremental updates, as the framework demonstrates that the final F1 scores never drop below 0.92 even after introducing new datasets nine times.