Post-Selection Distributional Model Evaluation

arXiv:2603.23055 · stat.ML, cs.IT, cs.LG, math.IT · Submitted 2026-03-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Post-Selection Distributional Model Evaluation".

Jane: The gist The authors introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called Post-Selection Distributional Model Evaluation. Essentially, the thesis is that formal model evaluation methods usually just check if a model hits a specific target key performance indicator level. But in many real applications, you don't know what that target level is beforehand. Instead, users want to compare different candidate models by looking at all the trade-offs between how well they perform and how reliable they are at test time.

Jane: That sounds really practical for things like deploying an AI model where you need to balance speed and accuracy in the real world. But the paper points out that this task gets complicated because you often have to use the same data both to pick a subset of good models and to estimate their performance distributions, which can introduce a kind of bias after selection.

Lu: That potential post-selection bias is a really important hurdle when we're talking about model evaluation, especially when we're dealing with data-dependent pre-selection procedures. The paper introduces something called Post-Selection Distributional Model Evaluation or PS-DME, which is a general framework for statistically valid distributional model assessment even after you have arbitrarily selected models based on the data.

Meng: So it's about handling that selection bias while still getting a statistically sound idea of the performance distributions for those selected models. I need to know what this actually means for deployment at scale, because if my pre-selection method is flawed, my distribution estimates might be useless when I try to compare candidates.

Lalam: PS-DME builds on e-values and controls something called the post-selection false coverage rate or FCR for those distributional KPI estimates. The paper establishes explicit conditions under which this framework is provably more sample efficient than a baseline method that relies on splitting the data differently.

Tom: So, to summarize, PS-DME is this framework designed to give you statistically valid distributional model assessment even when you've already pre-selected your models using some arbitrary, data-dependent procedure. It focuses on controlling that false coverage rate for your KPI estimates.

Jane: And the paper claims that by building on e-values, this PS-DME method actually controls the post-selection false coverage rate for those distributional KPI estimates. That sounds like a big step in making sure our comparisons aren't just based on a lucky subset of models.

Paper summary: Lu: They go on to establish explicit conditions under which PS-DME is provably more sample efficient than a baseline method that uses sample splitting. This means they've shown that you can get tighter results using this method without needing as much data overall, compared to just splitting the dataset for selection and evaluation separately.

Meng: Sample efficiency is key when you're dealing with large datasets; it means less computational cost and faster iteration cycles. If PS-DME can be more efficient than sample splitting, that’s something I can actually work with when trying to ship models to production across many configurations.

Lalam: They validate this methodology on three specific areas: synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation. These experiments demonstrate that PS-DME lets you reliably compare candidate configurations across a range of reliability levels, supporting the exploration of performance and reliability trade-offs in a statistically reliable way.

Tom: So we've seen it works on synthetic data, LLM decoding, and even telecom networks. That shows the framework isn't just theoretical; it has some real traction across different domains where you care about those performance-reliability trade-offs.

Jane: What this means for us is that we can move beyond just checking if a model hits a fixed target KPI and start analyzing the full picture of what happens at test time, which is much more useful when you're trying to understand the actual trade-offs available.

Lu: The paper also discusses performance measures like false coverage proportion, which is defined as the fraction of selected configurations for which a specific miscoverage event occurs. They also look at confidence band width, quantified by epsilon k in PS-DME, and they show this can be smaller than that of sample splitting under certain conditions.

Meng: Confidence band width is what I care about most; it directly relates to the uncertainty around my KPI estimate. If PS-DME gives me narrower bands when the full data reuse is efficient, that means my uncertainty is smaller, which helps me make more confident decisions about which configuration to deploy.

Lalam: They show that the choice of an e-calibrator parameter tau can have a noticeable effect on how tight those inferential bands are in practice; intermediate values of tau provide the most informative bands when you're actually running things.

Paper summary: Tom: So, it sounds like the core mechanic is using these e-values to construct confidence intervals that account for the post-selection process, and tuning a specific parameter helps you get the best balance between certainty and accuracy in your estimates.

Jane: It’s a sophisticated way of saying we can get reliable estimates of things like latency quantiles or outage probabilities when we're dealing with these complex model selection scenarios, which is really useful for understanding the real operational limits.

Lu: The paper suggests that PS-DME enables an uncertainty-aware assessment of the full KPI distribution, supporting a more reliable exploration of performance and reliability trade-offs across different operating points like latency quantiles or outage levels.

Meng: It’s about using this framework to support that exploration, which means we can reliably test different configurations without getting bogged down in the statistical noise introduced by how we initially selected them.

Lalam: They also mention that they use Berk–Jones confidence bounds as an alternative implementation, and this input-dependent method can be tighter in the tails of the CDF when you need to focus on those extreme performance scenarios.

Tom: So, it seems like the main contribution is proving that PS-DME provides valid post-selection inference while often yielding tighter bands and stronger guarantees than sample splitting baselines, depending on whether your data reuse efficiency is high enough.

Jane: It sounds like this paper gives us a concrete tool to move past simply checking if a model meets one target KPI, and instead lets us map out the full performance envelope of different AI configurations.

Lu: The implication is that for complex systems where you have many choices for hyperparameters or decoding strategies, PS-DME offers a statistically sound way to navigate those trade-offs with confidence in your estimates.

Meng: For production, this means I can use this framework to compare my candidate configurations and select one that balances the best performance with the lowest uncertainty about its reliability metrics.

Lalam: The authors are looking forward to extending this framework to more general and multivariate performance measures in their future work, which suggests there's a lot more ground to cover in applying this idea.

Conclusion: Tom: So we’ve gone through how they set up this whole evaluation pipeline—the pre-selection step and then trying to figure out what kind of performance distribution you get once you've picked your models.

Jane: Right, so before we get to the results, let’s just nail down what this paper is actually about. The title is Post-Selection Distributional Model Evaluation by Angelopoulos and his team at the University of California, Berkeley.

Lu: They are tackling the problem where you can't know your target performance level upfront, so you want to compare models based on their full performance and reliability trade-offs at test time.

Meng: I hear "post-selection." That sounds like it’s about what happens *after* you’ve already picked some models using some data-dependent method.

Lalam: Exactly. And the paper introduces PS-DME, which is this framework that lets you do that distributional model assessment even after your selection process has already happened.

Tom: It seems they are focused on controlling something called the post-selection false coverage rate, or FCR for those KPI estimates. That sounds like a specific statistical hurdle they're trying to clear.

Jane: So, what does this mean for us on the ground? If we’re using AI models and we need to know not just if it works, but how reliably it works under different conditions—like latency quantiles or outage levels—this framework helps us do that comparison statistically soundly.

Lu: The core idea is they show explicit conditions where PS-DME is provably more sample efficient than a baseline method where you split your data for selection and evaluation separately.

Meng: More sample efficient means you can get the same level of certainty with less data, which is what I need when I’m shipping things to production. That sounds like a real win for deployment costs.

Lalam: And they show this holds up across synthetic data, text-to-SQL models running large language models, and even on telecom network performance evaluation sets like O-RAN.

Tom: So the conclusion is that PS-DME lets you reliably explore those performance versus reliability trade-offs by using these distribution estimates with statistical guarantees, rather than just making a guess based on a chosen subset.

Jane: It changes the way we think about testing AI configurations, moving from checking if they hit a number to understanding the whole range of outcomes they might produce at test time.

Lu: The implication is that for complex systems with many choices—like different decoding strategies or hyperparameters—you can navigate those trade-offs with more confidence in your statistical measurements.

Tom: It’s about getting uncertainty awareness into the evaluation process itself, which is something I need when we're trying to make sure our deployed models are actually robust across the whole operating range.

Jane: If you want to see how they actually built these confidence bands and how that parameter tau affects the tightness of those bounds, we can look at that next.

Institute for Intelligent Networked Systems Northeastern University London

stat.ML, cs.IT, cs.LG, math.IT

Submitted: 2026-03-24

Updated: 2026-10-19

Code: https://github.com/amirfar76/PS-DME

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist The authors introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent

Key concepts

Post-Selection Distributional Model Evaluation (PS-DME): A general framework designed to statistically validate the performance distributions of a model after you have already chosen a subset of models based on specific data. It allows for arbitrary pre-selection while controlling the error rate in estimating the true performance characteristics of those selected models.
False Coverage Proportion (FCP): A measure used to quantify how often the pre-selected models are incorrectly evaluated or 'miscovered' by the statistical estimation process. The goal of PS-DME is to keep this proportion below a user-defined threshold (delta), ensuring that the confidence bands accurately reflect reality.
Sample-Splitting Distributional Model Evaluation (SS-DME): A baseline evaluation strategy where the data used for selecting models is kept entirely separate from the data used to estimate their performance distributions. This method provides a guaranteed control over post-selection error, serving as a benchmark against more complex methods like PS-DME.
e-values: Statistical values derived from e-calibrators that are used within PS-DME to construct confidence bands for the model's Cumulative Distribution Function (CDF). These values help define the width of the confidence intervals, allowing for a statistically sound assessment of performance uncertainty.

Terminology

Post-Selection — 4× in this paper · explained in 2 episodes
Selection Bias — 1× in this paper · explained in 5 episodes
Text-to-SQL — 1× in this paper · explained in 3 episodes

Summary

The gist The authors introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection, which controls post-selection false coverage rate (FCR) for the distributional KPI estimates

Problem Setup

The problem involves an arbitrary two-step model evaluation pipeline consisting of data-based model pre-selection and reliable distributional model evaluation The goal is to guarantee the condition FCR ≤ δ for a user-defined target probability δ ∈ (0, 1) This requires controlling the false coverage proportion (FCP), which is defined as the fraction of pre-selected models in set K for which the miscoverage event (5) holds true

Post-Selection Distributional Model Evaluation (PS-DME)

PS-DME supports three steps:

Arbitrary model pre-selection, where the user can pre-select a subset K ⊆ K of promising configurations using any arbitrary and possibly data-dependent procedure
Distributional model evaluation, where for each selected model k ∈ K, PS-DME constructs confidence bands for the CDF of the KPI of interest
Model selection based on performance-reliability trade-off with statistical guarantees

Distributional Model Evaluation Strategies

The paper compares three strategies:

Sample-splitting distributional model evaluation (SS-DME), where the pre-selection subset Dsel k is used exclusively for model selection, and the model evaluation subset Deval k is reserved for distributional inference This method provides a guarantee FCR ≤ δ
Naive in-sample distributional model evaluation (N-PS-DME), which reuses the full dataset D for both model selection and distributional inference, but cannot generally guarantee the FCR requirement (8) due to selection bias
In-sample distributional model evaluation (PS-DME), which leverages the entire dataset D for both selection and inference using e-values to construct valid post-hoc CDF bounds that satisfy the FCR condition (8)

Performance Measures and Guarantees

Key performance measures include:

False coverage proportion (FCP), defined as the fraction of selected configurations k ∈ K for which the miscoverage event (5) occurs
Confidence band width, quantified by εk in PS-DME, which is smaller than that of SS-DME under certain conditions
Post-selection guaranteed KPI, which allows the user to select a final configuration that strikes the desired balance between performance and reliability while accounting for statistical uncertainty

Experimental Validation

The proposed PS-DME was validated on three benchmarks:

Synthetic data, where it showed that PS-DME provides miscoverage control while N-PS-DME does not provide error control guarantees due to selection bias
LLM decoding with text-to-SQL, where PS-DME consistently guarantees lower KPI values, corresponding to higher performance for the selected decoding strategy
Telecom network performance evaluation on an O-RAN dataset, where PS-DME consistently yields stronger guarantees than the SS-DME variants

Conclusion

PS-DME enables uncertainty-aware assessment of the full KPI distribution, supporting reliable exploration of performance–reliability trade-offs across operating points such as latency quantiles and outage levels Compared to sample splitting, PS-DME can yield narrower confidence bands when the efficiency gains from full data reuse outweigh the cost of post-selection correction Experiments demonstrate that PS-DME achieves valid post-selection inference while often providing tighter bands and stronger guarantees than sample-splitting baselines Future work includes extending the framework to more general and multivariate performance measures

How it works

PS-DME leverages e-values to construct confidence intervals with post-selection validity The confidence bands are defined by Lk(x) = max[0, Fbk(x) − εk] and Uk(x) = min[1, Fbk(x) + εk], with band width εk derived from the e-calibrator This method is shown to satisfy the requirement (8), FCR ≤ δ, for any selection rule S(·) and any e-calibrator f(·) The comparison between PS-DME and SS-DME reveals that PS-DME can be more efficient than SS-DME when the condition (15) holds, highlighting the trade-off between using the full sample size nk for both pre-selection and inference The use of Berk–Jones (BJ) confidence bounds offers an alternative implementation that is input-dependent, which can be tighter in the tails of the CDF The final results on synthetic data and real-world benchmarks confirm that PS-DME achieves valid post-selection inference while often providing tighter bands and stronger guarantees than sample-splitting baselines

Key Findings

PS-DME constructs confidence bands for the entire CDF of the KPI of interest, enabling reliable estimation of quantities such as latency quantiles or outage probabilities
The comparison between SS-DME and PS-DME shows that PS-DME can yield narrower band widths when the efficiency gains from full data reuse outweigh the cost of post-selection correction
In the O-RAN experiment, PS-DME consistently yields stronger guarantees than the SS-DME variants
The choice of e-calibrator parameter τ can have a noticeable effect on inferential tightness, with intermediate values of τ providing the most informative bands in practice

References

Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control

Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control

Luiz André Barroso, Urs Hölzle, and Parthasarathy Ranganathan. The datacenter as a computer: Designing warehouse-scale machines

Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values

Yoav Benjamini and Daniel Yekutieli. False discovery rate–adjusted multiple confidence intervals for selected parameters

Robert H Berk and Douglas H Jones. Goodness-of-fit test statistics that dominate the Kolmogorov statistics

Leonardo Bonati, Salvatore D’Oro, Michele Polese, Stefano Basagni, and Tommaso Melodia. Intelligence and learning in O-RAN for data-driven nextg cellular networks

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners

Abhinav Chakraborty, Junu Lee, and Eugene Katsevich. Comparing three learn-then-test paradigms in a multivariate normal means problem

Ben Chugg, Etienne Gauthier, Michael I Jordan, Aaditya Ramdas, and Ian Waudby-Smith. Post-hoc large-sample statistical inference

Jeffrey Dean and Luiz André Barroso. The tail at scale

Lutz Dümbgen and Jon A Wellner. A new approach to tests and confidence bands for distribution functions

Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz.

Improvements for AI systems

Post-selection Distributional Model Evaluation (PS-DME) enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance–reliability trade-offs. This allows AI systems to select optimal configurations, such as LLM decoding strategies or network slicing policies, by accounting for statistical uncertainty in their test-time Key Performance Indicator (KPI) distributions.
PS-DME provides confidence bands for the CDF of the KPI of interest that are guaranteed false coverage rate control, addressing the problem where the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. This ensures that when an AI selects models based on preliminary data, the subsequent evaluation of those selected models remains statistically valid.
The framework allows for the selection of final configurations by examining the trade-off between achievable performance and reliability levels, enabling users to select a final configuration that strikes the desired balance between performance and reliability, while accounting for statistical uncertainty. This is directly applicable to selecting LLM decoding parameters or telecom network configurations based on desired failure probabilities, as demonstrated in the experiments.

Abstract

Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and we establish explicit conditions under which it is provably more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.