Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

The gist: This paper proposes an uncertainty-aware ROC framework for interval-valued predictions, introducing AUCL and AUCU metrics to quantify ranking performance beyond point scores.

Problem Setup and Interval Comparisons

The framework addresses the limitation of standard ROC analysis, which is designed for point scores and fails to capture the impact of predictive uncertainty on ranking performance <ref:2602.04775#pg2> In this setting, interval overlap represents explicit ranking ambiguity rather than a mere statistical coincidence of identical scores <ref:2602.04775#pg4>. For two intervals I1 = (L1, U1) and I0 = (L0, U0) from positive and negative classes, we define three comparison outcomes: (i) confidently correct if L1 > U0, (ii) confidently incorrect if U1. These rules induce a partial ordering that distinguishes definitively ordered pairs from ambiguous ones <ref:2602.04775#pg4>.

Two ROC-Style Curves and Interval-Based AUC Quantities

The framework defines two ROC-style curves using rate functions with different levels of strictness: TPRL vs FPRU (strict criteria requiring entire intervals to exceed the threshold) and TPRU vs FPRL (relaxed criteria allowing intervals to contain the threshold) <ref:2602.04775#pg6>. The associated areas under these curves are denoted by AUCL = Z TPRL(t) d[FPRU (t)] and AUCU = Z TPRU(t) d[FPRL(t)] <ref:2602.04775#pg7>. These measures have simple probabilistic interpretations: AUCL = P(I1 > I0), AUCU = 1 − P(I0 > I1) <ref:2602.04775#pg4>. Together with the overlap probability, these yield a three-region decomposition: P(I1 > I0) + P(I1 < I0) + P(overlap) = 1, providing a complete characterization of ranking uncertainty <ref:2602.04775#pg4>.

Connection to Classical AUC When prediction intervals collapse to points

The framework recovers standard ROC analysis as a special case when prediction intervals collapse to points, i.e., L = U almost surely, where the overlap probability vanishes and the partial ordering becomes total <ref:2602.04775#pg4>. In this scenario, TPRL(t) = TPRU (t), and FPRL(t) = FPRU (t), so that AUCL = AUCU reduces to the classical AUC <ref:2602.04775#pg4>.

Selective Prediction and Uncertainty-Aware AUC

The framework naturally supports selective prediction by identifying pairs where the model should abstain due to ranking ambiguity, utilizing the uncertainty-aware AUC (uAUC) as the conditional probability of a correct ranking given that a decisive ordering exists <ref:2602.04775#pg4>. The uAUC captures the intrinsic quality of the model’s high-confidence rankings, and its reliability is shown to be non-decreasing as uncertainty requirements become more stringent <ref:2602.04775#pg9>.

Estimating the Bounds of the Optimal AUC

The framework provides principled bounds on the optimal AUC achievable by any scoring function under mild coverage assumptions <ref:2602.04775#pg4>. Specifically, Theorem 3 establishes that given interval-valued predictions I(x) with class-conditional miscoverage rates α1 and α0, the optimal AUC satisfies AUCL − ppair ≤ AUC∗ ≤ AUCU + ppair <ref:2602.04775#pg10>. This result provides a principled way to estimate the range of the best achievable discrimination under observed interval uncertainty <ref:2602.04775#pg10>.

Empirical Validation

Experiments on real-world benchmark datasets, using bootstrapbased intervals as one instantiation, validate the framework’s correctness and demonstrate its practical utility for uncertainty-aware evaluation and decision-making <ref:2602.04775#pg4>. For instance, experiments verify the theoretical equivalences by showing that AUCL vs. P(I1 > I0) and AUCU vs. 1 − P(I1. Furthermore, the framework supports selective prediction, showing that at a 90% confidence level, the classifier abstains on 33.7% of pairs while achieving a uAUC of 0.916 on the remaining instances <ref:2602.04775#pg9>.

Discussion and Conclusion

The paper concludes that iAUC offers a rigorous, practical toolkit for evaluating and deploying uncertainty-aware classification models by providing AUCL and AUCU metrics that partition pairwise rankings into correct, incorrect, and ambiguous categories <ref:2602.04775#pg4>. The framework establishes formal links between interval-based metrics and theoretical discrimination limits, allowing practitioners to quantify the degree to which model-based ranking performance is definitive versus ambiguous <ref:2602.04775#pg4>. This approach enables selective prediction strategies that yield significant benefits in high-stakes risk prediction tasks <ref:2602.04775#pg4>. The framework also establishes conservative bounds on the optimal AUC, providing a principled range for the best achievable discrimination under observed interval uncertainty <ref:2602.04775#pg10>.

References

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021 <ref:2602.04775#pg4>.

Johannes Bracher, Evan L Ray, Tilmann Gneiting, and Nicholas G Reich. Evaluating epidemic forecasts in an interval format. Journal of the American Statistical Association, 2021 <ref:2602.04775#pg4>.

St´ephan Cl´emen¸con, G´abor Lugosi, and Nicolas Vayatis. Ranking and empirical minimization of u-statistics. The Annals of Statistics, 36(2):844–874, 2008 <ref:2602.04775#pg4>.

Corinna Cortes and Mehryar Mohri. Auc optimization vs. error rate minimization. In Advances in Neural Information Processing Systems, 2004 <ref:2602.04775#pg4>.

Thomas J DiCiccio and Bradley Efron. Bootstrap confidence intervals. Statistical Science, 11(3):189–228, 1996 <ref:2602.04775#pg4>.

Bradley Efron and Robert J Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, 1994 <ref:2602.04775#pg4>.

Ran El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11:1605–1641, 2010 <ref:2602.04775#pg4>.

Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861–874, 2006 <ref:2602.04775#pg4>.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016 <ref:2602.04775#pg4>.

Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian Data Analysis. CRC Press, 2013 <ref:2602.04775#pg4>.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007 <ref:2602.04775#pg4>.

David J. Hand and Robert J. Till. A simple generalisation of the area under the roc curve for multiple class classification problems. Machine Learning, 45(2):171–186, 2001 <ref:2602.04775#pg4>.

James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29–36, 1982 <ref:2602.04775#pg4>.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, 2018 <ref:2602.04775#pg4>.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.

1. Uncertainty aware classification systems can be improved by using AUCL and AUCU metrics to evaluate ranking performance, which provides an informative three-region decomposition of the ROC plane, partitioning pairwise rankings into correct, incorrect, and uncertain orderings.

2. Selective prediction capabilities can be enhanced by using the uncertainty aware AUC (uAUC) to define a threshold for abstention on ambiguous pairs, allowing models to abstain from ranking cases with overlapping intervals while maintaining a highly reliable discriminative core.

3. Model selection can be refined by using the bounds provided by interval-based metrics to characterize the physical limit of achievable discrimination, specifically by relating AUCL and AUCU to bounds on the optimal AUC under class-conditional coverage.

In high-stakes risk prediction, interval-valued predictions provide a natural way to represent predictive uncertainty. However, standard evaluation tools such as the receiver operating characteristic (ROC) curve and the area under the curve (AUC) are designed for point-valued predictions and do not capture how uncertainty affects discrimination performance. To address this gap, we develop an uncertainty-aware ROC framework and corresponding interval-based AUC (iAUC) metrics for evaluating the performance of interval-valued risk predictions. The framework constructs two ROC-style curves with associated areas AUC L and AUC U, and shows that these quantities induce a three-region decomposition of positive-negative rankings into confidently correct, ambiguous, and confidently incorrect orderings. Under valid class-conditional coverage, AUC L and AUC U, together with the pairwise miscoverage rate, provide theoretical bounds on the optimal AUC, linking interval coverage to achievable discrimination. The framework is compatible with a range of methods for quantifying epistemic uncertainty in risk prediction, including Bayesian, bootstrap, and ensemble-based procedures, and provides a practical way to compare predictive models and interval-generation methods. Experiments on clinical benchmark data and MIMIC-IV mortality prediction show that models with similar point AUC can have different uncertainty-aware ranking profiles, and that the proposed decomposition can support risk-aware model selection and threshold-based abstention.