Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

arXiv:2610.09709 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Pretraining Shapes Spectral Structure".

Jane: The gist: The OOD accuracy gap in foundation models can be determined from spectral structure encoded in pretrained weights,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper titled "Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models". It sounds like it's trying to figure out if we can predict if a foundation model will generalize well outside the data it was trained on before we even see any new data.

Jane: That’s the big question, right? Like, how do you check for bad generalization without having a test set or any target data at all? It seems to be proposing that the answer is actually hidden in the way the weights are structured after pretraining.

Lu: Exactly. The authors argue that this structure isn't random; it’s shaped by two main things: the architecture, which dictates how information gets stored in those weight matrices, and the pretraining strategy, which determines what kind of patterns the model is rewarded for learning.

Meng: I’m curious about that strategy part. What are we talking about when we say pretraining strategy? Is it just what dataset they use, or something more specific like how they structure the training objective?

Tom: It seems to be more than just the data; it's the whole setup. The paper suggests that this combination of architecture and strategy creates a unique geometric fingerprint in those weights. That fingerprint is what ultimately governs how robust a model is when we move to new data distributions.

Jane: So, if we look at different combinations—say, a Transformer architecture trained with one method versus another—we should see different geometric fingerprints that tell us something about its OOD performance.

Lu: Right. And they show that pooling these incompatible fingerprints cancels out the predictive signal entirely, but when you condition on them separately, you get the signal back. It’s like needing to look at all the fingerprint types individually instead of just averaging them out and losing the important detail.

Meng: So it’s about separating those distinct groups to find the robust ones? That makes sense if we're trying to deploy models safely.

Tom: Right, so before we even run a single forward pass on new data, we can use these spectral statistics from the pretrained weights to triage models by their inherent fragility.

The paper's summary: Jane: So what’s the actual diagnostic they are proposing? How do they turn these abstract concepts of "spectral structure" into something concrete that a model engineer can actually use? I’m picturing some kind of score or metric.

Tom: They use cell-specific statistics computed from the pretrained weights alone as a proxy for how tightly the source representations concentrate. This single statistic is supposed to tell us about the concentration, and it’s what we want to measure.

Lu: And they find that this concentration statistic serves as a good proxy for OOD accuracy because it's related to how well those representations behave under distribution shift. They also point out that the direction of this proxy flips depending on which architecture family you are looking at.

Jane: Flipping directions sounds tricky. How does that help us? Like, if one family has a positive concentration score and another has a negative one, what does that mean for robustness?

Tom: It means we have to be careful about which direction we interpret; the operationalization is done by defining a cell as the finest grouping they tested where the direction stays stable within one architecture and strategy combination.

Meng: So they are trying to find these specific groups of models where the diagnostic makes sense, instead of just looking at every model in isolation.

Jane: And they found that within these cells, this concentration statistic actually orders about ninety-two percent of model pairs by their OOD robustness for in-sample testing <ref:2610.09709#pg3>. That’s a pretty high correlation for a single statistic derived from the weights alone.

Lu: It really shows that when you isolate the right combination of architecture and strategy, this cell-specific statistic is a very reliable way to rank models by how they handle distribution shift.

The paper's improvements: Tom: Okay, moving on to what the authors suggest we actually *do* with this finding. They propose acting on this spectral concentration to narrow the OOD gap significantly.

Jane: How much of an improvement are we talking about? Is it a small tweak or something substantial?

Tom: The paper says that acting directly on spectral concentration can narrow the OOD gap by twenty-four percent at eighty-seven point five percent ID retention <ref:2610.09709#pg3,the OOD gap by 24% at>. That’s a measurable improvement in robustness, and it comes from using this diagnostic before committing any target data.

Lu: And they also suggest that metric-guided layer selection beats random targeting in nine out of ten models when applying these spectral concentration insights. It implies we should be smarter about where we apply the fix, not just blindly changing parameters everywhere.

Meng: That makes sense for production costs and stability. If we can target specific layers instead of guessing which layers are fragile, that reduces the risk of breaking things at scale.

Jane: So it’s not just about finding a good model; it’s about fixing the model's internal structure based on these spectral diagnostics to make it more resilient before deployment.

Tom: Yeah, they also have this failure mode taxonomy organized into six archetypal modes, each defined by its spectral signature and a recommended repair strategy. It helps guide the actual intervention.

Lu: And they validate this cell-specific approach using leave-one-cell-out analysis, showing that different cells prefer different metrics in every single round of testing, which confirms that the architecture times strategy combination is indeed the minimal unit where those metric relationships are consistent.

Conclusion: Jane: So to wrap up this work on "Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models," what’s the big picture for us? What does this mean for how we evaluate models going forward?

Tom: It means we can check OOD robustness using only released weights, before any target data is even available. We’re checking the model's internal spectral structure to see if it has a concentration that suggests fragility.

Lu: The core idea is that the architecture and pretraining strategy jointly define this geometry, and by looking at how those geometries compare across different cells, we can predict OOD degradation.

Meng: For deployment teams, this gives us a way to triage models as soon as they are released into the wild. It’s an auditing step before we spend compute on training a specific target domain model.

Lalam: From my perspective, this gives me high confidence that if I look at the spectral structure correctly—by conditioning on architecture and strategy—I can predict where I’m most likely to fail when things change outside my training set.

Jane: It seems like we’ve moved from needing external data to being able to check internal properties of the weights alone to find model fragility. This is a big step in how we think about deploying these massive models responsibly.

Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo, Jiook Cha

Seoul National University · Brookhaven National Laboratory

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Comments: 10 pages, 3 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist: The OOD accuracy gap in foundation models can be determined from spectral structure encoded in pretrained weights, specifically by identifying architecture- and strategy-conditional

Key concepts

Spectral Geometry: This refers to the distinct geometric fingerprint left in the pretrained weights by a specific combination of model architecture and pretraining strategy. Different combinations result in different spectral geometries, which dictate how the model's internal representations behave when faced with new, unseen data (OOD).
Cell-Specific Statistics: The authors define 'cells' as the finest groupings of models based on their spectral signatures. A specific statistic computed from weights within a cell reliably ranks OOD robustness for 92% of model pairs in-sample, proving that the architecture and strategy combination is the minimal unit for consistent metric relationships.
OOD Robustness Gap: This is the difference in accuracy between a model's performance on its training data versus its performance on new, out-of-distribution data. The paper uses spectral concentration statistics as a proxy to bound this gap, meaning you can estimate how much worse an OOD model will perform based on its weights alone.
Log-srfeat: This is a specific theoretical metric that establishes the hierarchy of OOD degradation within certain conditions. Theorem 1 proves that under defined structural conditions (like bounded core drift), this metric orders the drop in accuracy, providing a mathematical foundation for understanding why some models are inherently more robust than others.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist: The OOD accuracy gap in foundation models can be determined from spectral structure encoded in pretrained weights, specifically by identifying architecture- and strategy-conditional spectral statistics that govern representation geometry

How it works

The paper demonstrates that the OOD robustness of a foundation model can be checked using only its released weights without any target data or forward pass The answer is encoded in the spectral structure of pretrained weights, shaped by architecture and pretraining strategy This spectral geometry governs OOD robustness, where architecture determines how information is stored and pretraining strategy determines what is rewarded

Key components of the diagnostic include:

Architecture determines how information is stored in weight matrices
Pretraining strategy determines what is rewarded
Together they set a spectral geometry that governs OOD robustness

Spectral Geometry and Representation

Each (architecture × strategy) combination leaves a distinct geometric fingerprint in pretrained weights, because architecture controls how information is stored and pretraining strategy controls what is rewarded This geometry determines how representations behave under distribution shift The spectral signatures differ by (architecture, pretraining strategy, data diversity) The bulk mass is related to MP-energy, while effective rank is related to PR / SR

Cell-Specific Statistics and Robustness

The OOD accuracy gap is bounded by how tightly the source representations concentrate A statistic computed from the pretrained weights alone serves as a proxy for that concentration The direction of that proxy reverses between architecture families This operationalization is done by defining a cell as the finest grouping tested, where the direction is stable within one (architecture × strategy) combination Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly because cells of opposite direction cancel Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample

Theoretical Foundation and Hierarchy

Theorem 1 establishes that log-srfeat orders OOD degradation within the regime characterized by T1–T6 This theorem proves that log-sr(F′) > log-sr(F) under certain conditions, leading to an increase in OOD accuracy drop of (1 − δ)q − δ > 0 The proof relies on structured weak-tail spreading (T1), bounded core drift (T2), balanced spectral projection (T3), joint normalized-margin concentration (T4), and shared Gaussian local target shift (T5)

Intervention and Practical Application

Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain Interventions acting directly on log-srfeat should outperform those propagating through the longer weight→feature chain Metric-guided weight surgery beats random targeting in 9 of 10 models

Failure Mode Taxonomy and Validation

The empirical cell assignments are organized into six archetypal failure modes, each defined by its spectral signature, predictive metric, and recommended repair strategy The taxonomy serves to motivate the repair choice, as the optimal repair targets the failure mode rather than the architecture per se

The paper validates its findings through a leave-one-cell-out (LOCO) analysis, showing that distinct cells prefer distinct metrics in all 15 LOCO rounds, confirming that the architecture × strategy combination is the minimal unit where within-cell metric–OOD relationships are consistent The final diagnostic protocol involves identifying the architecture family, pretraining strategy, and domain to retrieve the theory-predicted metric and note its calibrated direction

The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance, and choosing models for new modalities before any evaluation set exists The framework ranks models within a cell rather than predicting absolute accuracy on a target domain This approach allows for triaging releases as they appear, auditing deployment for fragility in advance

Improvements for AI systems

Improved OOD Robustness Diagnosis: Implement a diagnostic on released weights alone that determines model fragility before any target data is available, as described by The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time (Page 1). This allows for triaging releases and choosing models for new modalities before any evaluation set exists (Page 9).
Architecture-Conditional Metric Selection: Dynamically select the appropriate weight-space proxy statistic based on the specific (architecture × pretraining strategy) cell, as detailed in Algorithm 1 Theory-Predicted Metric Selection (Page 38). This ensures that the selection does not leak the target and leverages the established hierarchy where Cell-specific statistics outperform unconditioned alternatives.
Targeted Weight Surgery for Robustness: Instead of random parameter changes, apply interventions by first identifying fragile layers using a per-layer diagnostic and then applying repairs only to those specific layers, as demonstrated by Targeted repair gives the largest improvement in every modality (Page 7). This approach reduces OOD drop by up to −0.348 for GPT-2-large (Table 14).
Cross-Family Robustness Transfer: Leverage the cell structure to predict robustness in unseen model families by noting that the same statistic transfers to families the matrix was never fitted on (Page 9). For instance, a positive correlation in an AR-CLM cell predicts robustness across different downstream tasks or modalities within that family.
Architectural Innovation Detection: Detect when architectural innovations cause within-cell scaling reversals by monitoring the knee detection procedure on sorted metric plots, which can reveal cases where the ESM-2 calibration fails for AMPLIFY (Page 37). This allows for the detection of emergent fragility in novel architectures without requiring external theoretical knowledge.

Abstract

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.