Controlling the False Discovery Rate via Model-Agnostic Importance Measures

arXiv:2610.21360 · stat.ME, stat.ML · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Controlling the False Discovery Rate via Model-Agnostic Importance Measures".

Jane: The gist Selecting covariates while controlling the false discovery rate (FDR) is critical for interpretable scientific discovery, especially with complex machine learning models.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So, to wrap up on this paper "Controlling the False Discovery Rate via Model-Agnostic Importance Measures," what's the simple translation for someone listening right now? What's the actual implication for how we do things in AI research or data science?

Tom: The main implication is that you can control false discoveries without needing to know the exact mathematical form of your response function <ref:2610.21360#pg1>. You don't have to assume it's linear or Gaussian, which is a huge relief when you're dealing with complex AI models <ref:2610.21360#pg4>.

Lu: They developed Model-Agnostic Selection with agent-X importance, MAS, which combines a resplit-per-covariate strategy and a data aggregation technique to achieve FDR control via cross-fitting <ref:2610.21360#pg3>.

Meng: It sounds like it's designed for situations where you're testing many things at once, where you want to find the most important variables without getting too many false alarms in your results <ref:2610.21360#pg4>.

Jane: Right. The paper shows that even though they are bypassing distributional assumptions, they still get theoretical guarantees for asymptotic FDR control under mild conditions <ref:2610.21360#pg4>. This is important because it gives you confidence in your variable selection process as the problem gets bigger <ref:2610.21360#pg4>.

Tom: They showed that their method performs competitively with other existing FDR control methods in terms of computational speed and power during simulations <ref:2610.21360#pg4>. It's not just a theoretical exercise; it works better than some of the methods they compared it against, like BH-LOCO <ref:2610.21360#pg4>.

Lu: The agent-X importance measure seems to be a more flexible tool for multiple testing than the leave-one-covariate-out measures they built on <ref:2610.21360#pg3>.

Jane: So, if you're working in a high-dimensional setting where you can't model everything perfectly, this paper points toward a selection method that is more robust and less restrictive <ref:2610.21360#pg4>.

Tom: That’s the gist of it. MAS is about giving us a way to select variables with control over false discoveries when we can't assume too much about the underlying data distribution <ref:2610.21360#pg4>.

Conclusion: Tom: So we've been looking at this paper, "Controlling the False Discovery Rate via Model-Agnostic Importance Measures." Basically, they’re trying to figure out how to pick important variables in machine learning models without making too many mistakes when you test a lot of things at once.

Jane: That’s right. The authors are proposing something called MAS, which stands for Model-Agnostic Selection with agent-X importance. It sounds complicated, but the core idea is that they want to select variables in a way that doesn't rely on any specific assumptions about the data distribution itself.

Lu: They use this agent-X importance measure, which is a dual form of what they call Leave-One-Covariate-Out. The goal there is to find a way to measure how much the best model prediction changes when you mess with one variable.

Meng: So, if I’m shipping models in production, I need something that doesn't break when the data looks different than what I trained on. This approach sounds like it tries to be agnostic about that training data distribution.

Tom: Exactly. The method they use involves a resplit-per-covariate strategy and some kind of data aggregation technique to make sure the estimates for the importance measures don't depend too much on each other. That’s how they handle those dependencies between different variables.

Jane: And the theoretical part is pretty solid, according to their theorem. They set up conditions where, as you get more data, they can guarantee that the false discovery rate stays under a certain level while controlling for the number of tests you're running.

Lu: The asymptotic linearity of this importance measure is key there. It shows that the estimator they are using actually behaves nicely in the limit, which helps prove that their selection rule works for controlling FDR.

Meng: But what’s missing? I see them mentioning assumptions one through five. That means if my real-world data doesn't fit those specific structural requirements, this guarantee might not hold up as well.

Jane: That’s a fair point, Meng. It shows the method is robust under mild conditions, but it still has its boundaries where the underlying assumptions matter for the strongest guarantees.

Tom: So what does this mean for us? It means we have a way to select features or variables in complex models without needing to nail down every single detail of how that data was generated.

Lu: It’s about moving away from methods that are too tied to specific distributional assumptions, like the original Model-X knockoffs. This gives us more flexibility when the model structure is really messy.

Jane: And for someone just listening, it means this paper offers a new toolkit for selecting variables in big AI projects where you can't perfectly model the whole world your data lives in.

Tom: It’s about making variable selection less brittle when you're working with high-dimensional data and complex machine learning models. We’ll be looking at how they tested this against other methods next.

Mengtao Wen, Haojie Ren, Changliang Zou

School of Statistics and Data Science, Nankai University · School of Mathematical Sciences, Shanghai Jiao Tong University

stat.ME, stat.ML

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/Jeremy690/False-Discovery-Rate-via-Data-Splitting

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 85/100

The gist: The gist Selecting covariates while controlling the false discovery rate (FDR) is critical for interpretable scientific discovery, especially with complex machine learning models.

Key concepts

Agent-X Importance: This measure evaluates the difference in prediction capability between the original covariate distribution and one where a specific agent variable is substituted. It quantifies how much the presence of a variable affects model performance, allowing for selection based on this importance.
Resplit-per-Covariate Strategy: This strategy involves splitting data samples independently for each covariate in every fold during cross-fitting. This decoupling ensures that the estimates of null importance measures are approximately independent, which is crucial for achieving valid FDR control.
Data Aggregation: This technique enhances statistical power by aggregating evidence across multiple variables. It creates a model-agnostic ranking statistic that improves the signal-to-noise ratio, enabling p-value-free threshold rules for variable discovery.

Terminology

Model-Agnostic — 3× in this paper · explained in 2 episodes
Aggregation — 1× in this paper
Theoretical Guarantees — 1× in this paper · explained in 2 episodes

Summary

The gist Selecting covariates while controlling the false discovery rate (FDR) is critical for interpretable scientific discovery, especially with complex machine learning models. This study introduces Model-Agnostic Selection with agent-X importance (MAS), a novel method that utilizes a resplitper-covariate strategy and a new variable importance measure to bypass distributional assumptions on covariates.

How it works

The proposed MAS method integrates three major ingredients to achieve FDR control via cross-fitting strategy. These ingredients are Agent-X importance, a resplit-per-covariate strategy, and data aggregation.

The Agent-X importance measure is defined as a dual form of the Leave-One-Covariate-Out (LOCO) approach, named agent-X importance. This measure is defined as ψj = V (f∗, PZj) − V (f∗, PZ), where X˜j is constructed as some constant or an independent copy of Xj. The goal of replacing Xj by X˜j in Zj is to deteriorate the prediction capability of the best model f∗.

The resplit-per-covariate strategy employs a cross-fitting strategy where samples in each fold will be resplit independently for each covariate so that all estimated null importance measures are approximately independent. This strategy is employed to decouple the potentially strong dependence between estimators of importance measures and thus provide valid FDR control.

Data aggregation is used to improve power by providing a model-agnostic construction of ranking statistics by further aggregating the evidence of important variables. The resulting statistics improve the signal-to-noise ratio and satisfy the marginal symmetry properties, allowing for a p-value-free threshold rule to make discoveries.

Theoretical Results

The theoretical control of FDR is established under mild conditions through Theorem 1. This theorem states that if Assumptions 1–5 hold and p = O(n r) for some constant r > 0, then for any α ∈ (0, 1), the MAS procedure in Algorithm 1 has FDPA ≤ α + op(1) as n, p → ∞.

Key theoretical properties include:

Asymptotic Linearity:

The asymptotic linearity of the proposed estimator ψˆj in (2.4), which serves as a foundation for establishing the selection rule for FDR control, is shown in Proposition 1. This proposition shows that ψˆj − ψj = 2/n X Zi∈D1 νj (Zi) − 2/n X Zi∈D0 ν0(Zi) + op(n-1/2).

Non-negativeness of agent-X importance measure:

Proposition 2 proves the non-negativeness of ψj with mild conditions and provides concrete examples for predictiveness functions like MSPE. It shows that if the agent variable X˜j is constructed such that P(Y,(X−j,X˜j)) ∈ P0,j, then ψj ≥ 0 and the equality holds when Xj ⊥⊥ Y X−j.

Analysis of resplit-per-covariate strategy

The resplit-per-covariate strategy is designed to relieve the strong dependence between null importance estimators, which can produce inflated FDR. The lemma shows that for ψj = ψl = 0, Cov(√nψˆj, √nψˆl) ≈ 0 under mild conditions. This implies that those null importance estimators are almost the same, and it is difficult to distinguish them.

Data aggregation

The data aggregation technique is refined by suggesting the construction ψˆE j = βˆj × ψˆj, where βˆj is a positive auxiliary statistic converging to some quantity βj > 0. This decomposition allows for a higher signal-to-noise ratio (SNR) compared to using ψˆj directly. The discovery set is then determined as A = the set of variables where ψˆE j ≥ L, where L is defined by a threshold rule.

Numerical Studies

Extensive simulations and real-data analyses demonstrate that MAS provides competitive performances with existing works in FDR control, computational efficiency and power. Simulations show that setting X˜j = z0.95 strikes a favorable balance between power and FDR control under these settings.

A comparison of MAS with competing methods shows that both DS and BHq yield FDR control and higher power in comparison to our MAS and Knockoff. The empirical power and much faster computation are observed for MAS compared to other methods like BH-LOCO.

Overall, the agent-X importance measures are more suitable for the multiple testing problem than the LOCO measures, and the choice of X˜j = z0.95 yields valid FDR control under general settings while providing reasonably high detection power. The paper concludes that MAS is more recommended in high-dimensional sparse problems where leveraging structural assumptions for regression is often easier than modeling the joint distribution.

Improvements for AI systems

Agent-X importance measure (MAS) enables model-agnostic variable selection by evaluating the difference of oracle predictiveness on both the original distribution and the distribution with agent variable in population level, which bypasses distributional assumptions on covariates and avoids fitting (p + 1) functions required by Leave-One-Covariate-Out (LOCO).
Resplit-per-covariate strategy mitigates the potentially strong dependence between estimators of importance measures, ensuring that the resulting selection procedure has a valid FDR control by making null importance estimators asymptotically independent, as shown by the result: Cov √nψˆj, √nψˆl ≈ 0.
Data aggregation enhances discovery power by constructing a statistic such that The resulting statistics improve the signal-to-noise ratio and satisfy the marginal symmetry properties, allowing for p-value-free thresholding, as demonstrated by the construction: βˆEj = βˆj × ψˆj.
Structure-powered enhancements allow tailoring variable selection to known data structures by constructing enhancement parameters such that β1 = V (f∗, PZGl(j)) for group structure or βj = V (f∗, PZQj) for ordered structure.
Covariate-adaptive importance can be integrated into learning algorithms like tree-based methods to create a statistic where ψˆCAj,k = X T t=1 [V (ˆfk,t, PZj,D j k,1) − V (ˆfk,t, PZ,D j k 0)] I(uk,j,t ≠ 0), potentially leading to higher power compared to standard MAS.