Generative Conformal Prediction with Optimized Coverage Allocation

The gist The proposed Generative Conformal Prediction with Vectorized Non-Conformity Scores (GCP-VCR) enhances prediction set efficiency by utilizing vectorized non-conformity scores and ranked samples to create adaptive uncertainty sets that are more flexible than traditional methods.

Generative Framework

The framework leverages a generative model to sample multiple predictions from the fitted data distribution, which encodes nuanced information of the prediction error distribution such as multimodal patterns and tail behaviors <ref:2410.13735#pg3>. This approach moves beyond deterministic models that output limited summary statistics like mean or quantiles <ref:2410.13735#pg3>.

Vectorized Non-Conformity Score

The method constructs a vectorized non-conformity score by aggregating prediction errors across all generated samples, and further ranks these samples by their empirical density <ref:2410.13735#pg5>. The non-conformity score vector is obtained where each element Ei,r is the distance between the true label Yi and the r-th sorted sample Yˆi,j(r) <ref:2410.13735#pg7>. This ranking mechanism allows for localized uncertainty allocation by distinguishing high-confidence and low-confidence samples <ref:2410.13735#pg7>.

Efficiency-Enhanced Prediction Set Construction

The prediction set is constructed as a union of regions centered at each generated sample, where the radius of each region is adapted to the model’s uncertainty level of each sample <ref:2410.13735#pg9>. Specifically, for the Euclidean distance measure, it becomes a union of balls centered at each generated sample Yˆn+1,j(r), with rank-dependent radius determined by Qr(βr) <ref:2410.13735#pg10>. This allows for individualized optimization of the ball radius for each sample <ref:2410.13735#pg10>.

Quantile Optimization and Approximation

The optimal quantile vector β is found by minimizing the total volume of all balls while ensuring coverage validity, formulated as a constrained optimization problem <ref:2410.13735#pg7>. To handle the computational infeasibility of this problem for larger K, an iterative algorithm that approximates the optimal quantile vector in linear time by simplifying the search process is introduced <ref:2410.13735#pg10>. This approximation method has a computational complexity of O(Kn) (or O(K2n) when iterating over all random sample positions) <ref:2410.13735#pg10>.

Empirical Validation

Extensive numerical studies demonstrate that the GCP-VCR approach outperforms state-of-the-art methods on both synthetic and real-world datasets <ref:2410.13735#pg3>. In the MNIST experiment, GCP-VCR consistently attains a lower miscoverage rate than PCP, indicating a higher efficiency of the prediction set <ref:2410.13735#pg10>. Furthermore, on real datasets like Blog Feedback and Bike Sharing, GCP-VCR achieves the highest efficiency while maintaining the coverage rate <ref:2410.13735#pg10>.

Conclusion

GCP-VCR introduces a novel conformal prediction method for generative models that leverages vectorized non-conformity scores with ranked samples to improve prediction set efficiency <ref:2410.13735#pg6>. By ranking conditional samples, the method captures nuanced error distribution and explores structural relationships among samples <ref:2410.13735#pg10>. Optimizing each entry of the non-conformity score vector separately enables a more flexible and data-adaptive prediction set <ref:2410.13735#pg6>. Through extensive experiments on both synthetic and real-world datasets, GCP-VCR consistently demonstrated superior efficiency while maintaining coverage validity <ref:2410.13735#pg10>.

--- Page 1 ---

Generative Conformal Prediction with Vectorized Non-Conformity Scores Minxing Zheng Shixiang Zhu University of Southern California Carnegie Mellon University Abstract Conformal prediction (CP) provides modelagnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution’s complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation – yielding larger prediction sets in high-confidence regions and smaller or excluded sets in lowconfidence regions – enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms stateof-the-art techniques on synthetic and realworld datasets. 1 Introduction Conformal prediction (CP) (Vovk et al., 2005) has gained prominence as a robust framework for uncertainty quantification (UQ), offering prediction sets with guaranteed coverage under minimal distributional assumptions. As a model-agnostic approach, CP can enhance any trained machine learning models to output not just point estimates but also provide uncertainty sets that contain the unobserved ground truth with user-specified high probability. As a result, CP has found broad applications across fields such as healthcare (Seoni et al., 2023), finance (Fujimoto et al., 2022; Blasco et al., 2024), and autonomous systems (Michelmore et al., 2020; Su et al. 11 Feb 5 Generative Conformal Prediction with Vectorized Non-Conformity Scores Minxing Zheng Shixiang Zhu University of Southern California Carnegie Mellon University Abstract Conformal prediction (CP) provides modelagnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution’s complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation – yielding larger prediction sets in high-confidence regions and smaller or excluded sets in lowconfidence regions – enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms stateof-the-art techniques on synthetic and realworld datasets. 1 Introduction Conformal prediction (CP) (Vovk et al., 2005) has gained prominence as a robust framework for uncertainty quantification (UQ), offering prediction sets with guaranteed coverage under minimal distributional assumptions. As a model-agnostic approach, CP can enhance any trained machine learning models to output not just point estimates but also provide uncertainty sets that contain the unobserved ground truth with user-specified high probability. As a result, CP has found broad applications across fields such as healthcare (Seoni et al., 2023), finance (Fujimoto et al., 2022; Blasco et al. 2024), and autonomous systems (Michelmore et al., 11 Feb 5 Generative Conformal Prediction with Vectorized Non-Conformity Scores Minxing Zheng Shixiang Zhu University of Southern California Carnegie Mellon University Abstract Conformal prediction (CP) provides modelagnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution’s complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation – yielding larger prediction sets in high-confidence regions and smaller or excluded sets in lowconfidence regions – enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms stateof-the-art techniques on synthetic and realworld datasets. 1 Introduction Conformal prediction (CP) (Vovk et al., 2005) has gained prominence as a robust framework for uncertainty quantification (UQ), offering prediction sets with guaranteed coverage under minimal distributional assumptions. As a model-agnostic approach, CP can enhance any trained machine learning models to output not just point estimates but also provide uncertainty sets that contain the unobserved ground truth with user-specified high probability. As a result, CP has found broad applications across fields such as healthcare (Seoni et al., 2023), finance (Fujimoto et al., 2022; Blasco et al.

1. Bold header: Flexible Uncertainty Sets for Regression

The improved system can construct discontinuous prediction sets as a union of regions centered at random samples, where the radius of each region is adapted to the model’s uncertainty level of each sample, leading to more flexible and data-adaptive prediction set compared to methods that use a uniform radius.

2. Bold header: Enhanced Efficiency via Adaptive Quantile Selection

The system can optimize the quantile vector by minimizing the total volume of all balls, which serves as an upper bound on the true size of the prediction set, enabling more efficient and tighter uncertainty sets through the optimization problem in equation (6).

3. Bold header: Scalable Uncertainty Estimation via Heuristics

The system can achieve computational practicality for large datasets by employing a heuristic algorithm to approximate the optimal quantile vector, making it computationally feasible with a complexity of O(Kn) (or O(K2n) when iterating over all random sample positions).

4. Bold header: Superior Performance on Complex Distributions

The system will outperform state-of-the-art techniques on synthetic and real-world datasets, specifically demonstrating that GCPVCR provides a more flexible and locally-adaptive prediction set, i.e., the coverage region of GCP-VCR is larger in the high-density region and smaller in the low-density region.

5. Bold header: Robust Uncertainty Quantification for High Dimensions

The system can be applied to high-dimensional data like images (MNIST) by constructing prediction sets in latent embedding space, where the sampled images of GCP-VCR align closer with the target X, indicating a tighter and more efficient prediction set of GCP-VCR than PCP.