Stability Selection via Variable Decorrelation

arXiv:2505.20864 · stat.ME, stat.CO, stat.ML · Submitted 2025-05-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Stability Selection via Variable Decorrelation".

Jane: Here's the gist of the paper. The gist The proposed method improves selection stability for Lasso by decorrelating variables before applying Lasso, and this improvement holds across different settings and variable selection techniques.

Tom: 1.

Jane: The general method is outlined in Algorithm 1, which involves three key steps: (i) reordering the variables, (ii) applying Gram-Schmidt orthogonalisation, and (iii) performing variable selection on Q using the stability selection with the Lasso.

Tom: 2.

Title and authors: Tom: We're starting with "Stability Selection via Variable Decorrelation" and the authors, Mahdi Nouraie, Connor Smith, and Samuel Muller. They’re showing how you can fix Lasso’s instability when variables are correlated by cleaning up the input matrix first.

Jane: Right. So if I were driving right now, I’d think about whether my chosen set of predictors stays the same every time I retrain my model with slightly different training data. This paper suggests that by using this decorrelation step—this Gram-Schmidt process they use to make a new matrix Q—that variability goes down.

Lu: Exactly. The main theoretical push is showing that under certain rules about how you order your variables, putting the most important ones first, this new matrix Q actually satisfies the irrepresentable condition for Lasso. That condition is what makes sure the selection process stays consistent even when things get messy in high dimensions <ref:2505.20864#pg1>.

Meng: Consistency is good for theory, but for me on the engineering side, consistency means I can trust my model to pick the same set of features whether I’m testing it on training data or new data from production. That stability they're describing—it’s like reducing noise in the feature selection process.

Tom: And the results back up that idea with concrete numbers. They show that when you apply stability selection using Lasso directly to Q, it actually beats applying it directly to the original matrix X when those two highly correlated variables are both important. It converges toward a specific stability frequency for those top two variables, around zero point five in some cases <ref:2505.20864#pg3>.

Jane: That’s a tangible result because it shows that this isn't just theoretical math; it translates into better performance metrics, like higher F1-scores across different decision thresholds when you compare it to other methods like Elastic Net or SCAD. It means the selection quality itself is actually getting sharper.

Lu: And they proved its robustness too. They showed that even if you completely flip the direction of the correlation—making it negative instead of positive—the improvements in both stability and accuracy stay there, which is a huge win for real-world biological or social data where we don't always know if features are positively or negatively related.

Meng: If this works reliably across different data structures, it simplifies my deployment pipeline immensely because I don't have to build a whole separate pipeline just to fix feature selection instability after every new dataset comes in. It’s about making the initial training step more reliable, which cuts down on debugging time later.

Tom: So the implication is that this pre-processing step isn't just academic; it’s a practical tool for building more robust feature selection pipelines when dealing with correlated inputs, especially when you have a lot of variables.

Jane: It definitely suggests that we can get more confident in our chosen predictors, even if the underlying data structure is complex and features are intertwined. But now we have to think about the setup—what assumptions they made about variable ordering—because that’s where we might need to be careful.

Lu: That's the limitation they flag, and it's important because if you mess up that initial variable reordering, the whole stability guarantee can break down. It depends entirely on how you structure your input before you run the Gram-Schmidt orthogonalization <ref:2505.20864#pg3>.

Meng: So, we’re looking at a method that trades some initial computational cost for a much more stable and reliable feature selection outcome in Lasso models. That sounds like something we should prototype for production testing to see if it scales well.

Tom: Exactly. It’s about taking the raw input matrix X and transforming it into this clean surrogate Q, which is mathematically guaranteed to give us better stability under these conditions <ref:2505.20864#pg3>.

Jane: And the big question now for anyone listening is whether that initial reordering step is easy enough to automate for real-world data streams, or if we still need a manual human touch there.

The paper's summary: Tom: Now we look at the specific improvements highlighted in "Stability Selection via Variable Decorrelation" and what those actually mean for model training stability. The big point is that this method consistently outperforms every other selection technique when you're measuring selection stability across different algorithms like Elastic Net or Sparse Group Lasso.

Jane: Right. So it’s not just that it’s a bit better; they show that when you use this approach, the system can reliably identify relevant variables even when the training data shifts slightly, which is what stability selection is all about. It proves that this decorrelation step gives you a more consistent set of predictors across different training runs.

Lu: The authors demonstrate this by showing that in simulation scenarios, they see a significant boost in stability, and crucially, they maintained accuracy even when the sign of the covariances within those correlated groups was completely reversed. That’s a really strong test for robustness <ref:2505.20864#pg3>.

Meng: I’m interested in that robustness because it means we don't have to be so rigid about how features are related—whether they are positively or negatively correlated—to get a reliable feature selection result. That flexibility is what makes it more deployable, even if the initial setup requires some work.

Tom: They also found a way to tune the regularization parameter lambda that is data-driven instead of manual guessing. They identify a "lambda stable" value as the smallest regularisation value where your stability number goes over zero point seven five, which balances stability and prediction accuracy nicely <ref:2505.20864#pg3>.

Jane: So it’s not just about picking features; it’s about finding the right penalty setting for that feature selection to actually work well for you. It moves beyond just getting a stable set to getting a stable set that also performs well on the actual predictive task, which is what we need in production.

Lu: The low-dimensional data findings are also key here, because they showed this method still improves selection stability even when the dataset is small. That means you don't have to wait for massive datasets to see these benefits; it works even when you’re dealing with fewer variables than samples <ref:2505.20864#pg1>.

Meng: If we can use this for low-dimensional problems, that opens up possibilities for smaller, targeted models where we might not have the computational budget for a massive re-orthogonalization step before every training run. That's a nice cost consideration.

Tom: So the implication is that if you’re dealing with high dimensionality and correlation issues, this method gives you a more reliable selection mechanism than just running standard Lasso on messy data. It’s about making sure your model gets the right features consistently, not randomly.

Jane: It suggests a pathway for building feature selection pipelines that are inherently more stable, regardless of the specific nature of the correlations present in the input data. But we still need to be mindful of those ordering assumptions they mentioned earlier because that’s where we have to check our own data structure before we start applying this trick.

Lu: That's a limitation they flag, and it's important because if you mess up that initial variable reordering, the whole stability guarantee can break down <ref:2505.20864#pg3>.

The paper's improvements: Tom: So to wrap up, we’ve looked at "Stability Selection via Variable Decorrelation" and what it does: it introduces a way to use Gram-Schmidt orthogonalization to decorrelate variables before applying Lasso, which really boosts stability in high-dimensional settings.

Jane: Right. Basically, they found that by creating this surrogate matrix Q, we get much more reliable variable selection results than just running Lasso on the raw data when features are highly correlated. It means better consistency across different training runs and even when the data correlation signs change around.

Lu: The paper establishes that this approach works well in low-dimensional settings too, showing that the benefits aren't just for massive datasets; they hold up even when you have fewer variables than samples <ref:2505.20864#pg1>.

Meng: From my perspective on deployment, if we can handle the initial computation of this Q matrix efficiently, it’s a solid tool for reducing the risk of deploying a model that suddenly fails when the data distribution shifts slightly. It adds a layer of mathematical rigor before we even start tuning the regularization parameters.

Tom: Exactly, and they also found a way to tune the regularization parameter lambda using their proposed data-driven strategy, finding that lambda stable value where stability hits zero point seven five or higher balances stability and prediction accuracy nicely <ref:2505.20864#pg3> <ref:2505.20864#pg3>.

Lu: The authors also pointed out their limitations, though; the entire guarantee relies heavily on how you structure your initial variable ordering before you run that Gram-Schmidt process, so messing up that setup can undermine the stability <ref:2505.20864#pg3>.

Tom: Yeah, so the paper "Stability Selection via Variable Decorrelation" gives us a concrete method to improve stability by transforming correlated features into an orthonormal surrogate matrix Q that Lasso can handle more consistently.

Lu: And while the authors did set some boundaries on when this works best, the evidence they present for its performance across different correlation structures is quite compelling <ref:2505.20864#pg3>.

Conclusion: Tom: So we’ve just gone through "Stability Selection via Variable Decorrelation" and what they did is use Gram-Schmidt orthogonalization to make a new matrix Q, and running Lasso on that actually makes feature selection much more stable when variables are correlated.

Lu: The paper establishes that this approach works well in low-dimensional settings too. They prove these benefits hold up even when you have fewer variables than samples, which is interesting because we often focus only on massive datasets <ref:2505.20864#pg1>.

Meng: From my point of view, if we can handle the initial computation of that Q matrix efficiently, it’s a solid tool for reducing the risk of deploying a model that suddenly fails when the data distribution shifts slightly. It adds a layer of mathematical rigor before we even start tuning our regularization parameters.

Tom: Exactly, and they also found a way to tune the regularization parameter lambda using a data-driven strategy, finding that lambda stable value where stability hits zero point seven five or higher balances stability and prediction accuracy nicely <ref:2505.20864#pg3> <ref:2505.20864#pg3>.

Lalam: I’m certain that this technique will be adopted because it addresses a fundamental weakness in using standard Lasso with correlated data.

Mahdi Nouraie, Connor Smith, §amuel Muller

School of Mathematical and Physical Sciences, Macquarie University · School of Mathematics and Statistics, The University of Sydney

stat.ME, stat.CO, stat.ML

Submitted: 2025-05-27

Updated: 2025-05-27

Journal ref: Statistics and Computing (2026), 36, 160

DOI: 10.1007/s11222-026-10916-7

Code: https://github.com/MahdiNouraie/Variable-Decorrelation

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist The proposed method improves selection stability for Lasso by decorrelating variables before applying Lasso, and this improvement holds across different settings and variable selection

Key concepts

Variable Decorrelation: This process involves reordering variables by predictive power and then projecting them onto a new space using Gram-Schmidt orthogonalization. The goal is to remove linear dependencies or correlations between variables in the design matrix. This creates an orthonormal matrix Q that serves as a stable surrogate for the original data X.
Surrogate Matrix Q: The resulting orthonormal matrix Q is used instead of the original design matrix X when performing variable selection with Lasso. Because its columns retain unique components of X's span, it acts as a reliable proxy for selecting which variables are truly important, especially in correlated settings.
Irrepresentable Condition: This is a mathematical condition required for the Lasso method to perform well. The paper shows that by using the decorrelated matrix Q, this condition is satisfied under specific assumptions about variable ordering and correlation structure. This ensures that Lasso can correctly identify relevant variables even when they are correlated with irrelevant ones.

Terminology

Consistency — 1× in this paper · explained in 3 episodes
Loss Function — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed method improves selection stability for Lasso by decorrelating variables before applying Lasso, and this improvement holds across different settings and variable selection techniques.

Methodology

The general method is outlined in Algorithm 1, which involves three key steps: (i) reordering the variables, (ii) applying Gram-Schmidt orthogonalisation, and (iii) performing variable selection on Q using the stability selection with the Lasso.
The columns of the design matrix first be sorted according to their predictive power for the response variable using Air-HOLP [Joudah et al., 2025].
Next, the variables are decorrelated within the ordered design matrix by projecting them onto a new space of orthonormal vectors by Algorithm 2.
The resulting orthonormal matrix Q is used as a surrogate for X in variable selection results, as it retains the component of Xj contributing uniquely to the span of X.

Justifying the Surrogate

The columns of Q are good proxies for columns of X because the jth column of R contains the normalising factor for the corresponding orthogonal vector in Q on the diagonal, with zeros below it. This enables us to use Q as a suitable surrogate for variable selection compared to X when the variables are correlated.

Stability and Consistency

The irrepresentable condition for the Lasso is satisfied after variable decorrelation under two assumptions, which is demonstrated by Corollary 1.
Corollary 1 states that if the columns of the design matrix X are not linearly dependent on each other, and if the variable ordering method positions all relevant variables in earlier columns relative to the irrelevant variables that are correlated with them, the irrepresentable condition for Lasso is satisfied by employing Q.
Corollary 2 demonstrates that when two highly correlated and equally relevant variables are present, applying stability selection with Lasso to the orthonormal matrix Q results in higher selection stability compared to applying stability selection with Lasso directly to X.

Numerical Results

In simulation scenarios, the application of the pre-processing steps significantly enhances stability, as shown in Figure 1a.
Figure 3a and 3b demonstrate that although the signs of the covariances are reversed, the improvements in both stability and accuracy are maintained.
The method outperforms all other candidates in terms of selection stability when compared to Elastic Net, SCAD, MCP, and Sparse Group Lasso.
Figure 7a and 7b show that for the Riboflavin data, lambda stable exists and its corresponding stability values have converged to 0.75 after almost 50 sub-samples.

Low-Dimensional Data

The method improves selection stability even in low-dimensional settings because Corollary 2 demonstrates that our method improves selection stability even in low-dimensional settings. Figure 4a illustrates that, as expected, our method leads to an improvement in the selection stability.

Comparison with Alternative Techniques

Lasso, when combined with our method, outperforms all other candidates in terms of selection stability. The SGL yields a slight positive effect on the stability of the results when compared to the previous selection techniques.

Future Work

It is recommended that further exploration be done in non-linear settings, as correlated variables also pose challenges in non-linear settings. It is hypothesized that the proposed approach could also be effective for non-regularisation-based methods, such as filter or wrapper methods.

Competing Interests

The authors declare that they have no conflict of interest.

Data Availability

The Riboflavin dataset is accessible via the hdi package in R [Dezeure et al., 2015]. The rat microarray data can be obtained from the National Center for Biotechnology Information (NCBI) website at www.ncbi.nlm.nih.gov, under accession number GSE5680. The source code used for the paper is accessible through the following GitHub repository: https://github.com/MahdiNouraie/Variable-Decorrelation. Furthermore, the DVS R package, which facilitates the use of the methodology introduced in this paper, is available through https://github.com/MahdiNouraie/DVS.

Acknowledgments

Mahdi Nouraie was supported by the Macquarie University Research Excellence Scholarship (20213605). Samuel Muller was supported by the Australian Research Council Discovery Project Grant (DP230101908).

References

˚A. Bj¨orck. Numerics of Gram-Schmidt Orthogonalization. Linear Algebra and its Applications, 197-198:297–316, 1994.

Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199–231, 2001.

Peter B¨uhlmann, Markus Kalisch, and Lukas Meier. High-dimensional statistics with a view toward applications in biology. Annual Review of Statistics and Its Application, 1(1):255–278, 2014.

Si-Bao Chen, Chris Ding, Bin Luo, and Ying Xie. Uncorrelated Lasso. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1):166–172, 2013.

Annie P Chiang, John S Beck, Hsan-Jan Yen, Marwan K Tayeh, Todd E Scheetz, Ruth E Swiderski, Darryl Y Nishimura, Terry A Braun, Kwang-Youn A Kim, Jian Huang et al. Homozygosity mapping with SNP arrays identifies TRIM32 as a Bardet–Biedl syndrome gene (BBS11). Proceedings of the National Academy of Sciences, 103(16):6287–6292, 2006.

Peng Cui and Susan Athey. Stable learning establishes some common ground between causal inference and machine learning. Nature Machine Intelligence, 4:110–115, 2022.

Ruben Dezeure, Peter B¨uhlmann, Lukas Meier, and Nicolai Meinshausen. High-dimensional inference: Confidence intervals, p-values and R-software hdi. Statistical Science, 30(4):533–558, 2015.

Gregory Faletto and Jacob Bien. Cluster stability selection. arXiv preprint arXiv:2201.00494, 2022.

Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456):1348–1360, 2001.

Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1, 2010.

Improvements for AI systems

textbf Pronounced Stability in Lasso Selection for Correlated Variables: The improved system can consistently identify relevant variables across different training sets, as shown by Corollary 2, which proves that the selection frequencies of the first two variables using X will converge to 0.5 when using the orthonormal matrix Q.
textbf Enhanced Predictive Accuracy via Consistent Selection: The system achieves a notable enhancement in the F1-score across most decision threshold values, as illustrated in Figure 1c and 1d, demonstrating that while stability improves, selection accuracy is also enhanced when correlation is removed from the data.
textbf Robustness to Covariance Sign Changes: The method maintains its performance across changes in variable association direction; the improvements in both stability and accuracy are maintained even when the sign of the covariances within correlated groups is reversed, as shown in Figure 3a and 3b.
textbf Optimized Regularization Selection: The system utilizes a data-driven tuning strategy by selecting the optimal regularisation parameter, specifically identifying lambda stable as the smallest regularisation value for which the corresponding stability exceeds 0.75, to balance selection stability and prediction accuracy.
textbf Superior Performance Across Algorithms: The proposed method outperforms alternatives across various selection techniques; for instance, it shows that Lasso, when combined with the decorrelation method, outperforms all other candidates in terms of selection stability (Figure 5).

Abstract

The Lasso is a prominent algorithm for variable selection. However, its instability in the presence of correlated variables in the high-dimensional setting is well-documented. Although previous research has attempted to address this issue by modifying the Lasso loss function, this paper introduces an approach that simplifies the data processed by Lasso. We propose that decorrelating variables before applying the Lasso improves the stability of variable selection regardless of the direction of correlation among predictors. Furthermore, we highlight that the irrepresentable condition, which ensures consistency for the Lasso, is satisfied after variable decorrelation under two assumptions. In addition, by noting that the instability of the Lasso is not limited to high-dimensional settings, we demonstrate the effectiveness of the proposed approach for low-dimensional data. Finally, we present empirical results that indicate the efficacy of the proposed method across different variable selection techniques, highlighting its potential for broader application. The DVS R package is developed to facilitate the implementation of the methodology proposed in this paper.