A Stable Lasso

The gist The proposed method improves selection stability of Lasso by integrating a weighting scheme into the Lasso penalty function, where weights are defined as an increasing function of a correlation-adjusted ranking that reflects predictive power <ref:2511.02306#pg5>

Introduction and Problem Context

The Least Absolute Shrinkage and Selection Operator (Lasso) is widely used for variable selection due to its simplicity and empirical performance <ref:2511.02306#pg7> However, Lasso's selection stability deteriorates when correlated predictor variables are present <ref:2511.02306#pg8> Specifically, an irrelevant variable highly correlated with relevant ones can cause Lasso to fail in distinguishing between them regardless of data size or regularization degree <ref:2511.02306#pg9> The paper reviews existing approaches and highlights their limitations <ref:2511.02306#pg10> Existing methods like the Independently Interpretable Lasso (IILasso) require the full covariance matrix, making them computationally expensive in high-dimensional settings

Proposed Methodology

The central idea is to assign weights to variables based on a correlation-adjusted ranking that reflects their predictive relevance These ranks are then transformed into variable-specific penalty factors and incorporated into the Lasso objective function The resulting model is referred to as the ‘Stable Lasso’

The methodology involves several key steps:

1. First, rank the predictors using a correlation-adjusted ranking method based on Air-HOLP.

2. Then, define the weight function as wj(rj) ≔ 1 − 1/rj for j = 1, 2,... p, where rj denotes the rank assigned to the jth predictor using Air-HOLP.

3. Equation (3) extends the Lasso objective function to include these variable-specific weights: arg min Q - Xβ2 + λw ⊙ β1

This weighting scheme provides prior information, enabling the model to prioritize promising variables within groups of correlated predictors The slope of the weight function decreases with increasing rank, making weights less sensitive to rank changes among lower-ranked variables

Stability Assessment and Tuning

The Stable Lasso is embedded within the Stability Selection framework to assess its stability across random sub-samples and to tune the hyper-parameter lambda The stability measure, Φˆ (M(λ)), is computed as defined in Nogueira et al. (2018) A key finding is that applying a fixed, non-uniform weighting scheme across all sub-samples yields greater stability than the standard equally weighted Lasso

Hyper-parameter tuning is achieved by jointly considering selection stability and prediction accuracy, adopting the approach proposed by Nouraie and Muller (2024) They define λstable as the smallest λ achieving a stability value of at least 0.75, or lambda stable-1sd if that is not attainable

Empirical Evaluations

Empirical evaluations on both simulated and real-world datasets demonstrate the efficacy of the proposed method In synthetic data experiments with n = 100 and p = 1, 000 variables with strong correlations across five groups, Stable Lasso exhibited significantly greater stability than Lasso and other methods like SGL or IILasso

In a real-world mouse PCR dataset with n = 60 and p = 22, 575 genes, applying the standard Lasso within Stability Selection resulted in a maximum stability value of 0.054, whereas Stable Lasso achieved a maximum selection stability of 0.85 Furthermore, when tested under less extreme correlation structures (correlations up to 0.8), Stable Lasso achieved the highest average F1-score across all decision thresholds The results also show that the performance advantages of Stable Lasso persist even under moderate correlation structures

Conclusion

The Stable Lasso is presented as a more stable variant of the Lasso and a generalized form of the Adaptive Lasso It demonstrates enhanced stability and accuracy compared to existing methods, including SGL and IILasso, even when compared to variable decorrelation approaches The method is shown to be effective in both high-dimensional and low-dimensional settings Moreover, the results show that Stable Lasso can deliver stable and accurate variable selection in complex and challenging settings

Competing Interests

The authors declare that they have no conflict of interest

Author Contributions Statement

Mahdi Nouraie was responsible for drafting the manuscript, the development of the research methodology and for writing the computer code used throughout Samuel Muller and Houying Zhu provided critical feedback on the content of the manuscript, refining the clarity and scope of the manuscript and computer code

Data and Code Availability

The mice PCR dataset is accessible JRSS(B) Datasets Vol.77(5) at https://fdslive.oup.com/www.oup.com/academic/content/rss-datasets/jrsssb/77-5-2015-B The source code used for the paper is accessible through the following GitHub repository https://github.com/MahdiNouraie/Stable-Lasso

Acknowledgments

Mahdi Nouraie was supported by the Macquarie University Research Excellence Scholarship (20213605) and a top-up scholarship from the Statistical Society of Australia Samuel Muller was supported by the Australian Research Council Discovery Project Grant (DP230101908)

References

˚A. Bj¨orck. Numerics of Gram-Schmidt Orthogonalization. Linear Algebra and its Applications, 197-198:297–316, 1994

Ma lgorzata Bogdan, Ewout Van Den Berg, Chiara Sabatti, Weijie Su, and Emmanuel J Candes. SLOPE—adaptive variable selection via convex optimization. The Annals of Applied Statistics, 9(3):1103–1140, 2015

Howard D. Bondell and Brian J Reich. Simultaneous Regression Shrinkage, Variable Selection, and Supervised Clustering of Predictors with OSCAR. Biometrics, 64(1):115–123, 2008

Patrick Breheny and Jian Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics, 5(1):232–253, 2011

Peter B¨uhlmann and Sara van de Geer. Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Berlin Heidelberg, 2011

Sourav Chatterjee, Trevor Hastie, and Robert Tibshirani. Univariate-guided sparse regression. arXiv preprint arXiv:2501.18360, 2025

Si-Bao Chen, Chris Ding, Bin Luo, and Ying Xie. Uncorrelated Lasso. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1):166–172, 2013

Emeline Courtois, Pascale Tubert-Bitter, and Isma¨ıl Ahmed.

1. Selected Lasso Improvement (Stable Lasso): The proposed method improves selection stability by integrating a weighting scheme into the Lasso penalty function where weights are defined as an increasing function of a correlation-adjusted ranking that reflects the predictive power of predictors. This allows the model to prioritize promising variables within correlated groups, mitigating the random selection among correlated predictors by the Lasso.

2. Enhanced Variable Selection Performance: The Stable Lasso can consistently outperform alternatives across various scenarios, as shown in Figure 1b where it ranks second for F1-score against Lasso and achieves superior stability compared to the standard Lasso, despite both being applied under Stability Selection framework. This suggests a more robust and reliable selection mechanism in high-dimensional settings with correlated predictors.

3. Low-Dimensional Setting Robustness: The method demonstrates effectiveness even when the number of variables is reduced to 80, showing that Adaptive Lasso (OLS) achieves the highest stability, with Stable Lasso ranking second. This confirms that the approach maintains its benefits in low-dimensional contexts where OLS estimation is feasible.

4. Enhanced Model Tuning: The hyper-parameter tuning strategy employs a joint consideration of selection stability and prediction accuracy to define a Pareto optimal solution on the stability–accuracy Pareto front, selecting the smallest lambda achieving a required stability level or utilizing lambda stable-1sd. This ensures that the resulting model is not only stable but also achieves high predictive performance.

5. General Applicability: The approach is shown to be a general-purpose solution by its effectiveness in stabilizing other regularization-based selection methods, as demonstrated by Additional numerical results demonstrate the effectiveness of the proposed approach in stabilizing other regularization-based selection methods. This suggests its potential application across different machine learning tasks beyond standard linear regression.

The Lasso has been widely used as a method for variable selection, valued for its simplicity and empirical performance. However, Lasso's selection stability deteriorates in the presence of correlated predictors. Several approaches have been developed to mitigate this limitation. In this paper, we provide a brief review of existing approaches, highlighting their limitations. We then propose a technique to improve the selection stability of Lasso by integrating a weighting scheme into the Lasso penalty function, where the weights are defined as an increasing function of a correlation-adjusted ranking that reflects the predictive power of predictors. Empirical evaluations on both simulated and real-world datasets demonstrate the efficacy of the proposed method. Additional numerical results demonstrate the effectiveness of the proposed approach in stabilizing other regularization-based selection methods, indicating its potential as a general-purpose solution.