Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On

arXiv:2610.12694 · cs.LG, cs.CE · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On".

Jane: The gist The proposed optimized version of a criterion for structure-disjoint splits, allowing for computationally feasible application on public libraries,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper, "Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On". The main idea here is about how structure disjoint splits change when you move from a small set of molecules to a much larger one.

Jane: Right, so it tackles this issue where you fix a rule once at one scale, like setting a distance threshold, and then try to apply it to something much bigger. It claims that this fixed rule isn't really invariant as the library grows.

Lu: The core problem they are looking at is evaluating out-of-distribution performance using these structure disjoint splits where the construction rule is set once based on the initial library size, like a pairwise distance threshold, and then assigning whole connected components to folds.

Meng: But this field has settled on a specific method for predicting structures from mass spectra, which is single-linkage clustering under the myopic Maximum Common Edge Subgraph MCES distance at a threshold of ten <ref:2610.12694#pg1,single-linkage clustering under the myopic Maximum Common Edge Subgraph MCES distance>. This convention was set on twenty-nine thousand molecules but public libraries now hold two hundred twenty-seven thousand six hundred ninety molecules <ref:2610.12694#pg1,This convention was set on 29,000 molecules but public libraries now>.

Lalam: MCES is NP-hard per pair over a quadratically growing set, which means this criterion has never been evaluated at that scale before. The paper proposes an optimized version of this criterion to make it computationally feasible on the public libraries: Admissible bounds and a union-find skip reduce two point six × one thousand ten molecular pairs to only one point one five × ten seven integer programs, which is a saving of over two thousand five hundred percent, while still producing the lossless partition.

Tom: That reduction sounds huge, two thousand five hundred percent is significant when you're dealing with NP-hard problems. What does this optimization actually mean for the work they are doing?

Jane: It means they made it computationally possible to even look at these massive public libraries using this specific splitting rule. They show that at this scale, the graph percolates, meaning one single component holds ninety-five point five percent of the library.

Lu: And what really catches my attention is what happens when you look at how the holdout fraction changes with library size N. A single component holds ninety-five point five percent for a large set, capping any test set at four point five percent.

Paper summary: Meng: But it's not just that the holdout gets smaller, it gets systematically different. The paper says a held-out compound outweighs a training compound eighty-eight point one percent of the time because a fixed absolute budget is actually a tighter relative tolerance on larger molecules.

Tom: So, when you look at large library benchmarks using this splitting rule naively, they end up mixing up structural novelty with extrapolation to larger, more complex chemistry. That's the core implication here.

Jane: Exactly. The existing benchmark procedure doesn't give you an invariant measure of generalization improvement as today’s chemical libraries keep growing in size; a fixed threshold is inherently relative to the scale it was set on.

Lu: They also look at threshold dependence, showing that lowering T from ten to five buys you a factor of sixteen in achievable holdout and removes most of the mass bias, but that costs novelty <ref:2610.12694#pg2>. At T equals five the giant component is twenty-six point five percent, and the ratios become achievable <ref:2610.12694#pg2>.

Meng: And they say the mass bias is largely gone from that split when you use T equals five <ref:2610.12694#pg2>. That's a big deal for production models where you care about specific molecular weights.

Tom: So, we have to consider that an absolute threshold is actually a relative one when you're dealing with single-linkage clustering. A molecule joins the giant component as soon as one connection falls below the threshold, and ten edge edits is a large fraction of a small molecular graph but a small fraction of a large one.

Jane: And because the distance metric isn't normalized, larger molecules stay comparatively unconnected under the same absolute threshold and survive in those small components left for holdout. This concentrates the split in the high-mass regime as library size grows.

Lu: The paper points out a discrepancy between metrics too, where on MassSpecGym’s complete all-pairs matrix, two thousand eight hundred fifty-three pairs attain ECFP4 Tanimoto greater than or equal to zero point nine nine nine while being provably distant at MCES greater than or equal to ten.

Meng: That implies a blind spot in counting; a fingerprint just counts a hashed set of substructural environments, so molecules built from repeating units collapse onto the same bit vector.

Lalam: The paper notes that both trends—the decreasing achievable holdout and the rising mass holdout AUC—reappear across different library sizes, showing that at N equals one hundred sixty thousand the giant sits at median three hundred ninety-three Da against five hundred thirty-eight point four Da outside it.

Paper summary: Tom: That fifteen point four dash shift in molecular mass is significant. It shows that the generalization improvement isn't a steady thing; it shifts with the scale of your data.

Jane: So, what does this mean for someone who only listens to the show? It means if you use these splits without accounting for library size, you might think you're improving generalization when you're actually just getting better at predicting heavier molecules in a larger set.

Lu: They flag one limitation plainly: T defines near, and therefore defines the leakage the split exists to prevent. A lower threshold buys usability by weakening the guarantee; pairs at MCES distance five to ten, which at these molecular sizes can be a single ring substitution, become free to straddle folds.

Meng: That means you aren't removing that tail at Tanimoto zero point one. That failure mode is what the split should prevent, and it leaks through two distinct channels.

Tom: We also have this discussion about threshold dependence, where lowering T from ten to five buys a factor of sixteen in achievable holdout but costs novelty because pairs that were previously separated now cross folds.

Jane: The paper suggests that while you can get usable ratios by lowering T, you lose the guarantee that the split is truly meaningful across different scales.

Lu: The authors conclude by stressing this: an absolute threshold is a relative one under single-linkage clustering. This means the existing benchmark procedure doesn't provide an invariant measure of this generalization improvement as chemical libraries grow in size.

Meng: So, we have to be careful about what we call "improving" these benchmarks when they are built on different underlying scales of data.

Lalam: The paper, titled "Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On", explores how the fixed construction rule for structure disjoint splits drifts with the libraries they are built on.

Tom: It’s about showing that using a fixed pairwise distance threshold to define folds doesn't give you a stable measure of performance when you scale up your dataset significantly.

Jane: It points out that as libraries get bigger, the holdout shrinks and becomes skewed towards higher mass molecules, which isn't just noise; it’s a systematic effect of how the splitting rule interacts with the library size.

Conclusion: Tom: So, we're wrapping up on "Critical Mass: Structure-Disjoint Splits Drift With The Libraries They Are Built On." Basically, it shows that when you set a fixed rule for splitting molecules into groups—like using a certain distance threshold—that rule doesn't stay the same when your library gets huge.

Jane: It means that the way we split up molecules isn't stable across different sizes of data. The authors are looking at this specific method, which relies on single-linkage clustering under an MCES distance of ten.

Lu: They found that this fixed threshold causes the results to shift systematically as the library size increases. Specifically, for every molecule in a huge library, the holdout set becomes systematically heavier.

Meng: From a production standpoint, that's concerning because if your model is trained on one set and you test it on a larger one, you might accidentally be testing its ability to handle much larger molecules than it was ever designed for.

Tom: So the authors are arguing that this isn't just a statistical fluke; it’s baked into the structure of how we define those splits. They show that an absolute threshold is really just a relative one depending on the size of the library you start with.

Jane: That’s what they mean, Tom. When you lower that threshold, say from ten to five, you get a bigger test set and less mass bias, but you lose some of that guarantee about what your split actually represents.

Lu: And they prove this by looking at the math on how pairs are processed. They have an optimized way to handle this NP-hard calculation so they can actually run it on these massive public libraries, which is pretty impressive engineering work.

Meng: Impressive, sure, but it still doesn't solve the core problem of using a fixed rule that changes its meaning with scale. If you deploy this splitting method blindly, your generalization estimate won't be accurate for bigger datasets.

Tom: Exactly. The implication here is that if we want to build robust models that work across different scales of chemical data, we can't just use one fixed benchmark setting anymore. We have to account for the library size itself when we evaluate how well our splits generalize.

Jane: Right, and it opens up a new way to think about what a good split even means in this context. It forces us to be much more careful about the assumptions we make when we scale up our data collection.

Tomo Oga, Devesh Shah, Cailum M. K. Stienstra, Gabriel Asher, Antonio H. de O. Fonseca, Niall O’Connor, Michael Widrich

Matterworks, Inc.

cs.LG, cs.CE

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted at 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: AI Data Readiness for Scientific Discovery

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist The proposed optimized version of a criterion for structure-disjoint splits, allowing for computationally feasible application on public libraries, reveals that MCES-10 is not invariant to

Key concepts

Structure-Disjoint Splits: This is a method used to divide a large collection of molecules into separate groups (folds) based on structural similarity. The goal is to create test sets that are truly 'out-of-distribution' from the training data, ensuring model performance estimates are reliable.
Maximum Common Edge Subgraph (MCES): This is a distance metric used to determine how similar two molecules are by finding the largest set of common structural features (edges). The paper uses this metric at a threshold of 10 to decide which molecules belong together in the same fold.
Library Percolation: This describes what happens when you apply the splitting rule to very large libraries. The paper found that as library size grows, the resulting clusters become dominated by one giant component, meaning most molecules end up in one large group, leaving very little room for smaller test sets.

Summary

The gist The proposed optimized version of a criterion for structure-disjoint splits, allowing for computationally feasible application on public libraries, reveals that MCES-10 is not invariant to library size as percolation collapses the achievable holdout and enriches the survivors in high molecular mass.

Problem Formulation

The core problem involves evaluating out-of-distribution performance with structure disjoint splits where a construction rule fixed once at the scale its library happened to have, such as a threshold pairwise distance, then assign whole connected components to folds The field has converged on single-linkage clustering under the myopic Maximum Common Edge Subgraph (MCES) distance at a threshold of 10 This convention was set on 29,000 molecules but public libraries now hold 227,690 molecules MCES is NP-hard per pair over a quadratically growing set We propose an optimized version of this criterion, allowing for computationally feasible application on the public libraries: Admissible bounds and a union-find skip reduce 2.6 × 1010 molecular pairs to only 1.15 × 107 integer programs, a 2,259× saving, while provably producing the lossless partition

Optimized Algorithm

The proposed optimization involves two algorithmic modifications to enable the computation of myopic MCES at larger scale: Expanded bounds and union-find skipping Expanded bounds expand the admissible bounds of Kretschmer et al. [8] by (a) the difference in total bond weight W(A) − W(B), since any excess weight in one molecule has no partner in the other, and (b) the l1 distance between bond-weight histograms of unordered pairs of endpoint elements, since a surplus of one bond class cannot match outside it In union-find skipping, we leverage Kruskal’s cycle rule [20]: a pair whose endpoints are already connected by proven near pairs cannot change the partition, whatever its verdict We bucket candidates by ascending bound and solve cheapest first over a union-find, skipping any pair whose endpoints already share a component This pruning is lossless and returns the partition that solving all N squared pairs with the same solver would (Appendix B.6)

Results on Scalability

The public union induces 2.6 × 1010 pairs, 43× more than MassSpecGym, and materializes a 207GB dense distance matrix for clustering alone The vectorized bounds certify 87% of pairs as far, followed by a 97.8% reduction by union-find skip (Figure 1) filter2 applied per pair inside the solver to 1.15 × 10 7 integer programs (0.04% of all pairs), resulting in a 2,259× reduction (Appendix B)

Impact on Holdout and Mass Bias

The achievable holdout collapses, and its mass bias grows with it The holdout fraction for generalization estimation under an MCES-10 graph split shrinks with increasing library size (Figure 2a), falling to 4.5% for the union and 0.84% for Enveda-180 Moreover, the probability that a holdout compound is heavier than a training compound increases monotonically, from ≈ 0.70 ROC AUC at N = 10 4 to ≈ 0.88 at N = 10 5 (Figure 2b) The MCES holdout therefore becomes not only smaller, but progressively enriched in high-mass structures as libraries scale This bias is a property of MCES, not the library

Conclusion on Benchmark Invariance

An absolute threshold is a relative one Under single-linkage clustering, a molecule joins the giant component as soon as one connection falls below threshold Ten edge edits is a large fraction of a small molecular graph but a small fraction of a large one Low-mass molecules cross that threshold more readily and denser sampling at larger N gives them progressively more opportunities to do so (Appendix E) Because the distance is unnormalized, larger molecules stay comparatively unconnected under the same absolute threshold and survive in the small components left for holdout, concentrating the split in the high-mass regime as library size grows The existing benchmark procedure therefore does not provide an invariant measure of this generalization improvement as today’s chemical libraries grow in size

Threshold Dependence

Lowering T from 10 to 5 buys a factor of 16 in achievable holdout and removes most of the mass bias, but costs novelty At T = 5 the giant is 26.5%, the ratios become achievable, and folds can actually be shown The mass bias is largely gone from that split (Figure 6a)

Discrepancy between Metrics

On MassSpecGym’s complete all-pairs matrix, 2,853 pairs attain ECFP4 Tanimoto ≥ 0.999 while being provably distant at MCES ≥ 10 These pairs span 953 structures across 129 disjoint families, a general property of the representation rather than one pathological series The blind spot is one of counting A fingerprint records a hashed set of substructural environments, so molecules built from repeating units collapse onto the same bit vector

Corroboration

Both trends reappear (Figure 8a,b) The achievable holdout falls monotonically from 40.6% to 14.6%, and the mass holdout AUC rises monotonically from 0.647 to 0.726 At N = 160,000 the giant sits at median 393.1 Da against 538.4 Da outside it, a 145 Da shift The ladder stops at N = 160,000 due to compute constraints: the library percolates less and therefore skips less, dispatching 10.55% of its candidates at that size against the union’s 2.23%, which puts a complete run several days of wall clock beyond the others

Final Assessment

T defines near, and therefore defines the leakage the split exists to prevent A lower threshold buys usability by weakening the guarantee: pairs at MCES distance 5 to 10, which at these molecular sizes can be a single ring substitution, become free to straddle folds What neither threshold removes is the tail at Tanimoto 1.0 This is the exact failure mode that a split should prevent, and it leaks through two distinct channels

Acknowledgements

We thank all members of Matterworks, Inc. for their advice and help on completing this work All authors of this work are employees or shareholders of Matterworks, Inc.

References

[1] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts.

Improvements for AI systems

Bold Header: Scalable Partitioning for Molecular Machine Learning Benchmarks

The optimized MCES criterion allows for computationally feasible application on the public libraries by reducing 2.6 × 1010 molecular pairs to only 1.15 × 107 integer programs, a 2,259× saving. This enables ML systems to evaluate generalization across much larger chemical spaces than previously possible, addressing the issue that a fixed absolute budget is a tighter relative tolerance on a larger molecule.

Bold Header: Bias Quantification for Library Size Dependence

The system can quantify how benchmark performance degrades as libraries grow by showing that MCES-10 is not invariant to dataset size: as libraries grow, percolation collapses the achievable holdout and enriches the survivors in high molecular mass. This prevents researchers from naively conflat[ing] structural novelty with extrapolation to larger, more complex chemistry when using standard benchmarks.

Bold Header: Structure-Disjoint Evaluation on Mass and Fragmentation Regimes

The improved splitting rule allows for evaluation of generalization across different chemical regimes by showing that the benchmark conflates structural out-ofdistribution generalization with extrapolation across mass and fragmentation regime. This forces model development to explicitly account for these differences, as the benchmark conflates structural out-ofdistribution generalization with extrapolation across molecule size and fragmentation regime.

Bold Header: Lossless Partitioning Guarantee

The union-find skipping mechanism ensures that the evaluation is rigorous by guaranteeing lossless partition and proving that the pruning is lossless, and keeps the same leakage guarantee. This means the resulting test set is exactly what a dense computation over all pairs would produce, providing an exact measure of generalization.

Bold Header: Threshold Sensitivity for Usability

The system provides a clear roadmap for selecting evaluation parameters by demonstrating that Lowering T from 10 to 5 buys a factor of 16 in achievable holdout and removes most of the mass bias. This allows researchers to choose between high-novelty holds (T=10) and more usable, less biased splits (T=5) based on their specific research goals.

Bold Header: Distinguishing Structural Novelty from Mass Extrapolation

The analysis reveals a critical failure mode where fingerprint-identical pair[s] split across folds leak through, where a fingerprint records a hashed set of substructural environments, so molecules built from repeating units collapse onto the same bit vector. This informs the development of models that are robust against these specific representation-based blind spots.