From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

arXiv:2610.20907 · cs.CR · Submitted 2026-10-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "From Detection to Decision".

Elias: The gist The results show that training for calibrated outputs alone does not ensure reliable security decisions under distribution shift.

Nadia: First, who's behind it and why it matters.

Title and authors: Nadia: Now we're looking at the specific title and who wrote this paper: "From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection." It really zeroes in on moving beyond just having a detector and focusing on making actual security decisions like allow, block, or defer.

Elias: The authors are Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, and Indranil Sanyal from eBay Inc. They set up this comparison between Jev, the Prompt Guard two classifiers at twenty-two million and eighty-six million parameters, and an LLM judge across forty-seven thousand six hundred eleven held-out examples covering various attack types.

The paper's summary: Nadia: The paper summarizes their findings by showing that while some models have great discrimination, like Jev with an AUROC of zero point eight seven one, their native probabilities aren't always better calibrated than the temperature-scaled classifier scores <ref:2610.20907#pg1>.

Elias: They give us specific numbers for this calibration check: Jev gets ECE zero point five four one, Brier zero point four four zero, and NLL one point four three two when compared to calibrated Prompt Guard two at eighty-six million parameters which got ECE zero point five seven one, Brier zero point four three nine, and NLL one point one zero four for instance <ref:2610.20907#pg2>.

Priya: That difference in calibration metrics is significant because it suggests that a simple training goal isn't enough to ensure the model's output is trustworthy when we deploy it in a changing environment.

Nadia: Right, Priya. And they also looked at the decision policy behavior, seeing how uncertainty at the gate translates into those allow block defer decisions when you have that frozen imperfect reviewer.

Elias: They found that to hit a very strict false negative rate of less than one percent, you needed over ninety-one point eight percent of inputs to be escalated to that reviewer for Jev native, and similar high rates for the Prompt Guard variants <ref:2610.20907#pg2>.

Priya: So even with those high escalation rates, the false positive rate at these crossing points was still quite high, hovering around forty to forty-five percent. That shows that just hitting a low false negative rate doesn't automatically mean you have a good overall system policy.

The paper's improvements: Nadia: One of the main improvements suggested by the authors is moving toward a risk-aware decision pipeline where we realize that a probability of zero point nine only matters if high scores reliably indicate attacks and stay meaningful even when attacks or benign prompts change.

Elias: They also argue that for routing decisions, you can use native models like Jev because their native probabilities aren't consistently better calibrated than temperature-scaled classifier scores, as we saw with the numbers comparing Jev to Prompt Guard two at eighty-six million parameters <ref:2610.20907#pg1>.

Priya: And they suggest that if we want to improve the reliability of those LLM-based security decisions, we should use temperature scaling on the classifier scores because it substantially improves metrics like ECE and NLL without changing how good the model is at discrimination.

Nadia: So, one way to fix things is to apply that specific calibration adjustment—temperature scaling—to enhance the utility of those classifier scores for deployment.

Elias: And they also propose establishing operational policies based on empirical escalation rates, which means finding the smallest escalation rate on the deferral curve where you hit that target false negative rate of less than one percent <ref:2610.20907#pg2>.

Priya: But they also stress a limitation here: calibration alone doesn't determine a useful decision policy, so we still need to evaluate reviewer behavior and the security and utility objectives of the deployed system together.

Conclusion: Nadia: To wrap up, the authors conclude that discrimination, calibration, reviewer behavior, and policy utility all need to be evaluated as parts of one single decision pipeline when dealing with prompt injection detection under distribution shift.

Elias: They reiterate that pooled metrics can obscure composition effects because the confirmatory cohort is heavily weighted by attacks from BIPIA <ref:2610.20907#pg3>, and the quality of the reviewer itself limits how much benefit deferral can actually achieve.

Priya: So for us, it means we need to be careful not to just look at one metric in isolation; we have to look at how those metrics work together under real operational constraints.

Nadia: They also point out that the comparison itself differs because Prompt Guard two is temperature-scaled on target calibration data while Jev uses its native probability, which is another factor we have to consider when comparing these systems <ref:2610.20907#pg1>.

Elias: And they flag a practical limitation: the cost model used in their evaluation is just illustrative, so we should use application-specific costs and review budgets instead of that general model.

Priya: Ultimately, this work on "From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection" shows us that while detection models are getting better at spotting injections, the real challenge remains making a dependable decision when those attacks evolve in ways we haven't seen before.

Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, Indranil Sanyal

eBay Inc.

cs.CR

Submitted: 2026-10-14

Updated: 2026-10-14

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The results show that training for calibrated outputs alone does not ensure reliable security decisions under distribution shift.

Key concepts

Probabilistic Decision Models: These are models that output a probability score (like 0.8) instead of just a yes/no answer. They aim to help systems make nuanced choices like allow, block, or defer based on the confidence level of the input being malicious.
Distribution Shift: This refers to situations where the data used to train a model is different from the data it encounters in real-world deployment. In this paper, it means testing if models trained on one set of attacks still work well when faced with new, unseen types of prompt injections.
Calibration: Calibration measures how well a model's predicted probabilities match reality. A perfectly calibrated model that says an input has a 90% chance of being malicious should be correct 90% of the time. The paper tests if native probabilities are reliable compared to scores adjusted after training.
Allow-Block-Defer Policies: These are operational rules for handling detected threats. Instead of just blocking everything, these policies decide what to do with an input based on its risk score: ALLOW it, BLOCK it immediately, or DEFER it for further review by a human or another system.

Terminology

Calibration — 5× in this paper · explained in 9 episodes
Distribution Shift — 3× in this paper · explained in 10 episodes
AUROC — 1× in this paper · explained in 3 episodes
Deferral — 1× in this paper · explained in 2 episodes
Indirect Prompt Injection — 1× in this paper · explained in 2 episodes
LLM judge — 1× in this paper · explained in 2 episodes

Summary

The gist The results show that training for calibrated outputs alone does not ensure reliable security decisions under distribution shift.

Detection Models Comparison

The study compares three detection models: Jev, Prompt Guard 2 (22M and 86M), and an LLM judge, across 47,611 held-out confirmatory examples covering various attack types The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714) However, Jev’s native probabilities are not consistently better calibrated than temperature-scaled classifier scores across calibration metrics

Calibration Reliability

The evaluation examines whether the native probabilities of an off-the-shelf decision model are more reliable on the target workload than raw classifier scores or classifier scores adjusted with conventional post-hoc calibration, particularly under distribution shift Jev obtains ECE 0.541, Brier 0.440, and NLL 1.432, compared with calibrated Prompt Guard 2 86M scores of ECE 0.571, Brier 0.439, and NLL 1.104 Temperature scaling substantially improves the Prompt Guard calibration metrics, particularly NLL, without changing discrimination

Decision Policy Behavior

The research examines how uncertainty at the gate translates into ALLOW–BLOCK–DEFER decisions when paired with a frozen imperfect reviewer The empirical confirmatory escalation rate required to achieve system FNR ≤ 1% is found to be more than 91.8% for PG2-86M, 92.45% for PG2-22M, and 92.55% for Jev-native These crossing points are descriptive test statistics rather than deployment thresholds selected using confirmatory labels The system FPR remains between 40.6% and 45.8% at these crossings, showing why FNR alone is not sufficient to characterize the resulting policy

Key Findings on Performance

The LLM judge provides the strongest discrimination and overall probability estimates among the evaluated methods, but at substantially higher latency Jev outperforms both Prompt Guard variants on discrimination metrics, but its native 0.5 decision boundary misses 64.9% of attacks Under the pooled shift mixture, none of the evaluated methods support autonomous blocking at a low false-positive rate

Limitations and Conclusions

The study concludes that discrimination, calibration, reviewer behavior, and policy utility need to be evaluated jointly as parts of the same decision pipeline Pooled metrics can obscure composition effects because the confirmatory cohort is dominated by attacks from BIPIA The reviewer quality constrains the benefit of deferral, as a weak reviewer limits what deferral can achieve Calibration needs to be evaluated on the target workload rather than inferred from the model’s training objective The comparison also differs in how the models are adapted to the target workload, as PG2 is temperature-scaled on target calibration data while Jev uses its native probability The cost model used in evaluation is illustrative, and deployment policies should instead be selected using application-specific costs and review budgets The evaluation focuses on detection and routing rather than end-to-end agent compromise or downstream harm

--- Page 1 ---

FROM DETECTION TO DECISION: CALIBRATION AND DEFERRAL FOR PROMPT-INJECTION DETECTION Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, Indranil Sanyal eBay Inc. ABSTRACT Prompt-injection defenses are commonly evaluated as binary detectors, but deployed systems must decide whether to allow, block, or escalate an input Recent decision models claim to provide probabilities suitable for making these decisions automatically We examine whether these probabilities lead to better security decisions We compare Jev, a native probabilistic decision model, with Llama Prompt Guard 2 classifiers (22M and 86M parameters) and a frontier LLM judge Our evaluation uses 47,611 held-out confirmatory examples covering human-written direct attacks, indirect injections, hard-benign prompts, obfuscation, and distribution shift Prompt and calibration choices are determined on separate development and calibration splits and fixed before confirmatory evaluation The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714) Jev’s native probabilities are not consistently better calibrated than temperature-scaled classifier scores: Jev obtains ECE 0.541, Brier 0.440, and NLL 1.432, compared with 0.571, 0.439, and 1.104 for calibrated Prompt Guard 2 86M On the frozen confirmatory cohort, the empirical deferral curves satisfy FNR ≤ 1% only when more than 91.8% of inputs are escalated, with substantial false-positive rates This crossing is a descriptive confirmatory statistic, not a deployment threshold chosen using test labels These results indicate that training for calibrated outputs alone does not ensure reliable security decisions under distribution shift Discrimination, calibration, reviewer behavior, and policy utility need to be evaluated jointly Keywords Prompt Injection Detection · Probability Calibration · Selective Prediction · Uncertainty Estimation · Distribution Shift · Large Language Models

--- Page 2 ---

CALIBRATION AND DEFERRAL Figure 1: Evaluation design and decision pipeline. (A) Prompts, models, calibration parameters, and the gate policy are selected using development and calibration data only and then frozen (B) Each confirmatory input is scored by a frozen security gate, and a fixed policy maps the resulting attack probability to ALLOW, DEFER, or BLOCK Only deferred inputs are sent to the frozen LLM reviewer, which produces the final decision The confirmatory cohort is evaluated without further tuning

--- Page 3 ---

CALIBRATION AND DEFERRAL Table 1: Frozen confirmatory cohort. Dataset Benign Attack Total TensorTrust 0 526 526 BIPIA 53 36,075 36,128 NotInject 89 0 89 PIDS-Bench v3 5,488 5,380 10,868 Total Total

--- Page 4 ---

CALIBRATION AND DEFERRAL Table 2: Overall detection performance on the confirmatory cohort. Model AUROC AUPRC TPR@1% FPR FNR@0.5 Jev-native 0.871 0.973 0.125 0.649 0.048 PG2-86M 0.757 0.920 0.012 Jev misses attacks, Prompt Guard boundaries transfer poorly, and the default Prompt Guard boundaries transfer particularly poorly

--- Page 5 ---

CALIBRATION AND DEFERRAL Table 3: Overall calibration and false-negative confidence. Lower values are better. FN confidence is the mean p(attack) among false negatives Jev-native 0.541 0.440 1.

Improvements for AI systems

The system can implement a risk-aware decision pipeline by using a probability of 0.9 is useful only if high scores reliably indicate attacks and remain meaningful as attacks, domains, or benign prompts change. This allows for more nuanced security actions than binary detection, enabling the system to decide whether to allow a low-risk input, block a high-risk one, or defer an uncertain case to a more capable reviewer.
The system can utilize native decision models like Jev for routing decisions because "Jev’s native probabilities are not consistently better calibrated than temperature-scaled classifier scores: Jev obtains ECE 0.541, Brier 0.440, and NLL 1.432, compared with 0.571, 0.439, and 1.104 for calibrated Prompt Guard 2 86M."
The system can improve the reliability of LLM-based security decisions by employing Temperature scaling substantially improves PG2’s ECE, Brier score, and NLL without altering discrimination metrics, suggesting a specific calibration adjustment to enhance the utility of classifier scores for deployment.
The system can establish operational policies based on empirical escalation rates by defining the empirical confirmatory escalation requirement, defined as the smallest escalation rate on the confirmatory deferral curve at which the observed system satisfies FNR ≤ 1%. This provides a quantitative measure for when to escalate inputs to a frozen reviewer to meet a target False Negative Rate.
The system can mitigate false positives by recognizing that probability calibration alone does not determine a useful decision policy, and instead evaluate reviewer behavior and the security and utility objectives remain important properties of the deployed system. This guides the selection of deployment intervals based on both FNR and FPR trade-offs observed in the empirical curves.

Abstract

Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs. Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M), and an LLM-based judge on 47,611 held-out examples spanning direct and indirect prompt injections, hard-benign inputs, obfuscation, and distribution shift. Using separate development and calibration splits, we assess detection performance, probability calibration, and allow-block-defer policies under a frozen evaluation protocol. The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). However, Jev's native probabilities do not consistently outperform post-hoc calibrated classifier scores, and temperature scaling improves calibration without improving discrimination. At an empirical false-negative-rate target of 1%, all evaluated gate-reviewer systems require more than 91.8% escalation and incur substantial false-positive rates. These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift. Effective LLM security systems must jointly evaluate detection, calibration, reviewer behavior, and operational decision policies.