Why Linear Models Can Still Work in a Complex and Changing World

The gist: Linear models can remain useful in complex and changing worlds when instability primarily lies in the nonlinear component, allowing linear models to omit unstable curvature and thus make fewer prediction errors in new environments.

Why Linear Models Can Still Work

Linear models can predict well even when underlying mechanisms are nonlinear because the complexity of the world exceeds what can be reliably learned from finite, partially observed data Flexible models exploit curvature, thresholds, and interactions that fit one environment but fail elsewhere When instability lies mainly in the nonlinear component, omitting it can give linear models lower prediction error in a new environment Stable nonlinear structure favors flexible models, and linear models remain vulnerable to linear spurious relationships Model complexity should therefore be chosen by cross-environment stability as well as training-data accuracy.

The Role of Partial Observation and Environmental Change

The practical question is whether the rule selected from the available data remains useful after deployment conditions change, rather than whether a nonlinear model class contains a better rule in principle. Data contain measurement error, selection in who or what is observed, unrecorded factors, and features that occur only in a particular hospital or period. When the relevant state is sufficiently observed and a nonlinear relationship remains stable across environments, a flexible model can learn it and outperform a linear model. The link between observed variables and unobserved factors must vary across environments.

Mathematical Formulation of Prediction

The true state Z passes through the environment-specific recording process he and measurement noise η to produce the observed value X. The outcome Y is determined from the true state Z through a rule f, together with an unpredictable outcome-side deviation ε. A prediction model uses the observed X, not the unobserved Z. The object to be learned is the function me that predicts the outcome from the observed X, which is called the conditional mean or regression function.

Conditions Favoring Linear Restriction

Several research traditions support different parts of this argument. One tradition concerns improper linear models where simple methods using equal or unit weights can outperform methods using estimated weights when data are limited. Another tradition compares clinical versus statistical prediction, where a fixed procedure can match or exceed expert judgment through consistent reuse of the same procedure, not linearity itself. Furthermore, regularization reduces the risk of learning accidental patterns from finite data by restricting the shapes a model can learn.

The Concept of Best Linear Prediction

Even when the true relationship is nonlinear, one can define the linear prediction rule with the smallest mean squared error among all linear prediction rules, which is called an orthogonal projection. A linear model can therefore be used as the best linear approximation without assuming that the true relationship is linear. However, if the values taken by X, and their frequencies, change across environments, the coefficients of the best linear approximation can also change.

Conclusion on Model Choice

A linear model should be treated neither as a universally correct baseline nor as a claim that reality is linear. The most important question is not whether reality is linear, but how much of the complex structure found in the observed data will remain in the environments where the predictions are actually used. Complexity should be selected by degree, rather than treated as a binary choice between linear and nonlinear models.

What Empirical Studies Should Examine

Researchers should compare models whose complexity increases gradually, examining not only the accuracy score but also which curve, threshold, or interaction produced the improvement. Validation design should match the conditions that are expected to change at deployment, holding out each environment as a whole for evaluation rather than randomly splitting individual records. Reports should include error in each environment and the worst-performing environment. The aim is not to discard every nonlinear relationship, but to use nonlinear relationships that have been shown to remain stable in other environments.

Discussion on Practical Implications

Useful performance by a linear model in a complex domain does not imply that the mechanisms of the world are linear. The advantage does not arise from poorer training fit itself; it arises when linear restriction prevents a model from carrying nontransferable complexity into the target environment. A strictly linear restriction causes an unnecessary loss of accuracy if the nonlinear rules are stable across environments. The most important question is how much of the complex structure found in the observed data will remain in the environments where the predictions are actually used.

Conclusion

Linear models sometimes achieve useful predictive performance in complex domains because they can exclude detailed rules that recorded data cannot establish as stable in other environments. The affirmative reason for using a linear model is that limited observations may not yet justify carrying complex rules into environments where their validity has not been established. If nonlinear rules are stable across environments, a flexible model should be used. If the relationships that change across environments are linear, a linear model offers no protection. The most important question is how much of the complex structure found in the observed data will remain in the environments where the predictions are actually used.

References

[1] Robyn M. Dawes, David Faust, and Paul E. Meehl. Clinical versus actuarial judgment. Science, 243(4899): 1668–1674, 1989

[2] William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz, and Chad Nelson. Clinical versus mechanical prediction: a meta-analysis. Psychological Assessment, 12(1):19–30, 2000

[3] Evangelia Christodoulou, Jie Ma, Gary S. Collins, Ewout W. Steyerberg, Jan Y. Verbakel, and Ben Van Calster. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology, 110:12–22, 2019

[4] Xiaohong Chen, Han Hong, and Denis Nekipelov. Nonlinear models of measurement errors. Journal of Economic Literature, 49(4):901–937, 2011

[5] Andrés Abeliuk, Zhijing Huang, Emilio Ferrara, and Kristina Lerman. Predictability limit of partially observed systems. Scientific Reports, 10:20427, 2020

[6] Matthew J. Salganik, Ian Lundberg, Alexander T. Kindel, Caitlin E. Ahearn, Khaled Al-Ghoneim, Abdullah Almaatouq, et al. Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15):8398–8403, 2020

[7] Thomas A. Severini. A note on the application of hermite polynomials to hidden truncation models. Journal of Statistical Theory and Practice, 20:17, 2026

[8] Hillel J. Einhorn and Robin M. Hogarth. Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13(2):171–192, 1975

[9] Robyn M. Dawes.

1. The AI system can implement environment-aware model selection by prioritizing cross-environment stability as well as training-data accuracy. This means instead of choosing a model based solely on training error, it evaluates how well a learned rule transfers to conditions outside the training distribution, thus avoiding the pitfall where a flexible model can use shortcuts tied to backgrounds, institutions, measurement processes, or interactions among variables that are useful only in the training environment.

2. The system will employ a conditional approach to model pruning: The advantage does not come from poor source fit itself. It comes from restricting the fitted relationship in a way that excludes unstable curvature, thresholds, and interactions. This allows the AI to discard complex features when they are found to be unstable in deployment environments, even if they performed well during training.

3. The system can perform best linear prediction as a baseline for new environments by selecting the linear predictor that has the smallest mean squared error among all linear prediction rules (Proposition 1). This ensures that in cases where the relationship between X and Y itself changes, the system defaults to the most robust, albeit simple, approximation rather than overfitting unstable nonlinearities.

4. The AI can dynamically adjust its complexity based on environmental uncertainty: When little is known about environmental change, a linear model may be a cautious choice. This provides a mechanism for intermediate levels of complexity (like generalized additive models or shallow decision trees) that retain only the curvature that the data support and exclude complexity for which the evidence is weak.

5. The system will incorporate rigorous validation protocols: Validation must be conducted by holding out entire environments (each such environment should be held out as a whole for evaluation) rather than relying on randomly splitting individual records, ensuring that performance is measured in the environment where the model will be used.

Linear models can predict well in complex domains even when underlying mechanisms are nonlinear. We argue that this can occur because the complexity of the world exceeds the complexity that can be learned reliably from finite, partially observed data. Flexible models can exploit curvature, thresholds, and interactions that fit one hospital, period, population, policy setting, or measurement process but fail elsewhere. When this instability lies mainly in the nonlinear component, omitting it can give linear models lower prediction error in a new environment. We connect this claim to work on simple linear rules, bias and variance, regularization, best linear prediction, measurement error, distribution shift, invariance, Rashomon sets, and underspecification. Proposition 1 gives a simple squared-error case in which a nonlinear rule learned in the training environment raises target error because it transfers too weakly; finite-sample estimation error can strengthen this result. Partial observation alone is not enough: the link between observed variables and unobserved factors must vary across environments. Stable nonlinear structure favors flexible models, and linear models remain vulnerable to linear spurious relationships. Model complexity should therefore be chosen by cross-environment stability as well as training-data accuracy.