MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?

arXiv:2610.12522 · physics.chem-ph, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?".

Jane: The gist Large language models (LLMs) can leverage physical properties for molecular structure prediction, but their reliance on this data varies substantially across models and property types,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, to build on that idea of varied reliance, what’s the core methodology of MolP2SBench in simpler terms? It sets up a task where the AI gets a molecular scaffold and some properties, and it has to pick the missing structure from candidates.

Tom: It formalizes property-to-structure prediction as a multiple-choice masked completion task. The whole point is to move beyond just getting the right final answer and instead test if the model actually uses those supplied physical properties during its decision process.

Lu: They introduce controlled intervention schemes to probe this, specifically removing properties entirely, shuffling them across tasks, or replacing them with counterfactual properties from another candidate in the same family. These interventions are designed to isolate whether a model's decision is robust or if it’s just brittle and reliant on one specific property pairing.

Meng: So they are testing three dimensions: recovery under the original property, static reliance by shuffling, and dynamic responsiveness by checking if predictions follow targeted substitutions when things start out correct. That sounds like a lot of work to set up.

Jane: It is detailed work because they want to disentangle those two concepts—static reliance versus dynamic responsiveness—which is what traditional accuracy metrics just can't do for us.

Tom: And the results show that these metrics give us different pictures. For example, Luna, Sol, and Astra all show a higher Reliance Margin for atomwise 1H NMR than for 13C peaks, but they have a lower Counterfactual Flip Rate in those cases <ref:2610.12522#pg2>.

Lu: That reversal between reliance and responsiveness is actually what motivates them to report those two metrics separately instead of just using one combined score. It shows that being relied upon doesn't automatically mean being responsive to change.

Jane: And when you look at combining properties, they found that simply adding the remaining properties to the strongest bundle doesn't always help across different models and property sets. It can actually hurt performance for some of the models, like Luna and Opus.

Meng: That's a practical warning for anyone trying to build multi-modal systems; sometimes mixing in more data just introduces more confusion than it solves when you don't have a strong reason why those pieces belong together.

Tom: It’s about finding that sweet spot where the physical evidence is used correctly without being overly constrained by heuristics that aren't actually supported by the data.

Lu: So, essentially, MolP2SBench is a systematic way to diagnose *why* an LLM makes a prediction, rather than just observing what the final prediction is.

Jane: So if we are looking at this paper for someone who doesn't read it, we can boil it down to this: they built a test to see if AI uses the physics they are given to make sense of a molecule.

The paper's summary: Tom: Let's talk about what the authors themselves suggest as improvements for using this benchmark. They focus on creating better ways to analyze and use these results, rather than just generating more data points.

Jane: They propose establishing a channel-wise recovery leaderboard using Property-Grounded Recovery, which they say balances source recovery with recovery under counterfactual substitutions, ensuring equal weighting across different property families.

Lu: That’s important because it lets us see model strengths without collapsing heterogeneous properties into one single score. They also report scores independently for each property channel, which reveals model-specific strengths clearly.

Meng: I like that idea of seeing the channel-wise results because it tells you exactly which physical properties the AI is actually good at using, rather than just giving you a general score.

Tom: They also suggest reporting Reliance Margin and Counterfactual Flip Rate separately because they measure fundamentally different things—static reliance versus dynamic responsiveness—which we saw earlier isn't always correlated.

Jane: And one of the most interesting things they point out is the case studies, where Astra was shown to have a superior ability to use local chemical structures for NMR interpretation, even when global dipoles contradicted local polarity rules.

Lu: That shows that models can struggle with conflicting information; they can recognize a local motif but fail to ground its consequence in the specific geometry of the candidate molecule they are choosing.

Tom: The limitation they note is that this framework is currently limited to small-molecule completion using computed properties and curated candidates, so it doesn't extend easily to experimental spectra.

Meng: That’s a fair point; if we want this benchmark to be useful for real drug discovery, we need it to handle the messy data from actual experiments, not just idealized computed values.

Jane: So the authors are essentially saying they’ve given us a better diagnostic tool to understand *how* an AI is using its knowledge, even if that knowledge isn't always perfectly aligned with physical reality.

The paper's improvements: Tom: So to wrap up the MolP2SBench paper, the main implication is that we need to stop just looking at whether an LLM can reconstruct a molecule and start testing if its choices are actually grounded in the physics it's given.

Jane: Right. It shows that property-to-structure reasoning is complex, and there isn't one single way for models to use physical evidence effectively across all property types.

Lu: The paper establishes this controlled framework as a reusable testbed for diagnosing evidence use and guiding future training strategies toward more reliable molecular inference.

Meng: From an engineering standpoint, this means we need better ways to diagnose where our models are failing—is it a failure to see the property, or is it a failure to apply the property correctly?

Lalam: I’m quite confident in this direction. Based on what I’ve seen across the literature, my assessment is that this framework for MolP2SBench provides a necessary foundation for training models that actually use molecular properties effectively.

Tom: Exactly. It moves us away from just hoping the model gets the structure right, toward actively engineering it to be physically sound.

Jane: And I think what we need to keep asking is how we can move beyond these curated small-molecule datasets to make this reasoning applicable in broader chemical discovery scenarios.

Lu: Future work needs to explore expanding molecular families and property channels by using LLM-assisted dataset construction while still requiring expert review and validation for the data integrity.

Meng: I'd want to see some benchmarks that test how these models handle real-world experimental spectra, because that’s where the practical problems live.

Lalam: My certainty is that this paper sets a very clear direction for making property-grounded molecular inference a measurable and achievable goal for large language models.

Conclusion: Tom: So we’ve gone through MolP2SBench, which is basically this controlled test to see if LLMs can actually reason from molecular properties to structure, not just guess the answer.

Jane: Yeah, and what they found is that it’s really about disentangling whether the AI is just mimicking correct structures or if it’s actually using the physical properties they were given.

Lu: The core idea there is using Property-Grounded Recovery to give equal weight to both recovering the original property and recovering a structure based on those counterfactual substitutions.

Meng: I’m still thinking about how much effort goes into building that benchmark, though. It sounds like a lot of work just to set up these controlled interventions.

Lalam: The paper shows that reliance and responsiveness can rank different ways across the various property channels they tested, which is what makes this whole framework so useful for diagnosis.

Tom: And it highlights how much reliance varies—Luna, Sol, and Astra all show different patterns when it comes to atomwise NMR versus those 13C peaks <ref:2610.12522#pg2>.

Jane: That’s a key point because it shows that you can’t just look at one number and think you know the model's behavior.

Lu: They also found that combining properties doesn't always improve things across all models; sometimes adding more data actually hurts the recovery for certain AI systems.

Meng: That tells us that we need to be careful when we try to feed these models massive bundles of information, because it doesn't automatically mean better performance.

Tom: Ultimately, MolP2SBench gives us a reusable testbed for figuring out where the evidence is actually being used and how we should adjust our training strategies for property-grounded inference.

Jane: It’s about moving past just getting high accuracy scores and actually understanding the reasoning behind those choices.

Lu: The authors leave the validity of the supporting chemical arguments open, which means we still have to be careful when interpreting what these models are actually "thinking."

Meng: So they established a controlled way to evaluate this, but they also pointed out that it only covers small-molecule completion with computed properties.

Tom: Right, so the next step is figuring out how to expand this beyond small molecules and into more complex chemical spaces.

Jane: Definitely. It’s a solid framework for diagnosing evidence use in molecular AI right now, but we still have a lot to explore before we get there.

Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, …

University of Tennessee, Knoxville · University of Illinois at Urbana-Champaign · University of California, Berkeley · University of Georgia · Florida State University

physics.chem-ph, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 44 pages

Code: https://github.com/DUAL-Xiao/MolP2SBench

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist Large language models (LLMs) can leverage physical properties for molecular structure prediction, but their reliance on this data varies substantially across models and property types,

Key concepts

Property-to-Structure (P2S) Reasoning: This is the core task being tested. It involves giving an LLM a partial molecule and a set of experimental or calculated physical properties (like NMR shifts or UV-Vis spectra), and then asking the model to select the correct, complete molecular structure from many possibilities. It measures if the model connects physical data directly to structural outcomes.
Property-Grounded Recovery (PGR): This metric evaluates a model's ability to recover a structure when it is grounded in the original supplied property. It assesses how well the model uses the correct source property during prediction. PGR is used alongside other metrics to see if models are reliably using the intended physical evidence.
Reliance Margin (RM): RM measures a model's static reliance on a specific piece of information. It quantifies how much better a model performs when given the correctly paired property compared to when that property is replaced by random or shuffled data. A higher RM suggests the model strongly depends on that particular physical observation.
Counterfactual Flip Rate (CFR): CFR measures a model's dynamic responsiveness—how quickly its prediction changes when a supplied property is swapped for an alternative, plausible property from the same chemical family. It tests if the model is sensitive to counterfactual changes, indicating whether it truly understands the physical relationship rather than just memorizing patterns.

Summary

The gist Large language models (LLMs) can leverage physical properties for molecular structure prediction, but their reliance on this data varies substantially across models and property types, revealing performance patterns that may differ from expectations based on conventional chemical intuition

MolP2SBench Framework

MolP2SBench introduces a benchmark for evaluating molecular property-to-structure (P2S) reasoning in LLMs by presenting a masked molecular scaffold with given properties, requiring the model to select the missing structure from plausible candidates The evaluation tests how models respond when properties are shuffled or replaced, thereby assessing their reliance on correct property-structure pairing and their responsiveness to counterfactual property changes This framework formulates property-to-structure prediction as a multiple-choice masked completion task Each task presents a masked molecular scaffold with given properties, requiring the model to select the missing structure from plausible candidates

Evaluation Metrics

The paper establishes three controlled intervention schemes to evaluate how well models use supplied physical properties These include removing them, shuffling them across tasks, and replacing them with counterfactual (CF) properties from an alternative candidate within the same family to assess responsiveness Three metrics are used for evaluation: Property-Grounded Recovery (PGR), Reliance Margin (RM), and Counterfactual Flip Rate (CFR) PGR jointly evaluates recovery under the original source property and candidate-targeted counterfactual substitutions, assigning equal weight to source and counterfactual recovery within each task family RM quantifies static reliance by measuring the performance gain from providing the correctly paired property relative to a shuffled-property control CFR measures dynamic responsiveness by quantifying whether predictions follow targeted substitutions of the supplied property among cases that are initially solved correctly

Model Performance and Property Dependence

The results show that while LLMs can leverage physical properties for molecular structure prediction, their reliance on this data varies substantially across models and property types Astra has the highest point estimate in ten of the eleven channels, while Opus leads on atomwise 13C NMR (69.3% versus Astra’s 68.1%) Model performance does not consistently reflect the relative difficulty of these properties for human chemists For example, a model may benefit more from 1H NMR than from 13C NMR, yet be less likely to change its prediction when the 1H data is replaced Astra achieves higher accuracy with UV–Vis data than with familiar NMR peaks

Reliance and Responsiveness Differences

Reliance and responsiveness show different observed property rankings Luna, Sol, and Astra all show a higher RM for atomwise 1H than for 13C peaks, but a lower CFR For Sol, the RM values are 25.2 versus 12.6 pp, whereas the CFR values are 53.6% versus 67.3% This observed reversal motivates reporting reliance and responsiveness separately

Combined Property Analysis

Combining properties does not always improve recovery; for instance, within the GPT family, Sol and Astra improve over the strongest constituent in all three bundles, but a successful within-bundle combination does not guarantee a successful cross-bundle combination For example, adding the remaining properties to the strongest bundle (vibrational for all four) improves Sol from 89.7% to 93.4% but reduces Luna from 83.8% to 78.7%

Case Studies and Limitations

Case studies reveal physical inconsistencies in model explanations, such as Astra's advantage for UV–Vis being higher than NMR peaks In Task 97CCF0B132ED0CC7, Astra selects the ester over the ether by noting that the masked carbonyl carbon is excluded from observation These analyses highlight a gap between recognizing local chemical motifs and grounding their consequences in candidate-specific geometry The limitation is that MolP2SBench covers small-molecule completion with computed properties and curated candidates, leaving experimental spectra outside its scope

Conclusion

MolP2SBench establishes a controlled framework for evaluating property-to-structure reasoning in LLMs beyond answer accuracy It provides a reusable testbed for diagnosing evidence use and evaluating future training strategies toward more reliable property-grounded molecular inference The results establish property-dependent selection, while leaving the validity of the supporting chemical argument open

The gist MolP2SBench establishes a controlled framework for evaluating property-to-structure reasoning in LLMs beyond answer accuracy. This benchmark system systematically evaluates how well models use supplied physical properties by presenting masked molecular scaffolds with given properties and requiring them to select the missing structure from plausible candidates It is crucial because it disentangles whether an LLM's decision genuinely tracks the physical evidence provided, rather than just achieving correct reconstruction

How it works

MolP2SBench formulates property-to-structure prediction as a multiple-choice masked completion task Each task presents a masked molecular scaffold with given properties, requiring the model to select the missing structure from plausible candidates The design isolates and evaluates two key dimensions of model capability beyond pure reconstruction: reliance on correctly paired properties and responsiveness to counterfactual properties

Metrics

Three metrics are used for evaluation: Property-Grounded Recovery (PGR), Reliance Margin (RM), and Counterfactual Flip Rate (CFR) PGR jointly evaluates recovery under the original source property and candidate-targeted counterfactual substitutions, assigning equal weight to source and counterfactual recovery within each task family RM quantifies static reliance by measuring the performance gain from providing the correctly paired property relative to a shuffled-property control CFR measures dynamic responsiveness by quantifying whether predictions follow targeted substitutions of the supplied property among cases that are initially solved correctly

Data and Property Channels

The benchmark contains 136 task families spanning 11 molecular-property channels, yielding 1,496 source-property instances and 2,766 eligible CF substitutions These channels cover seven property types: electronic properties (dipole magnitude, HOMO–LUMO gap, UV–Vis), vibrational properties (IR and Raman spectra), and NMR properties (1H and 13C observations) The NMR inputs provide calculated shielding values for nuclei in the visible scaffold

Analysis of Results

The results show that reliance and responsiveness rank property channels differently, with IR yielding the highest PGR for Luna, Sol, Opus, and Astra Model rankings depend on both the supplied properties and the diagnostic being evaluated The channel-wise PGR stability under family subsampling shows mean rank correlations ranging from 0.940 to 0.999 across all eleven single-property channels

Limitations

MolP2SBench covers small-molecule completion with computed properties and curated candidates, leaving experimental spectra outside its scope The current benchmark’s coverage is constrained by the effort required for careful dataset construction and manual validation Future work could expand molecular families and property channels through LLM-assisted dataset construction while retaining expert review and validation

The gist MolP2SBench establishes a controlled framework for evaluating property-to-structure reasoning in LLMs beyond answer accuracy.

Improvements for AI systems

Improved property-grounded reasoning for molecular structure prediction by implementing Property-Grounded Recovery (PGR) as a channel-wise leaderboard, which gives equal weight to source and counterfactual recovery. This allows AI systems to assess whether their predictions genuinely rely on the supplied properties across different property types.
Disentangling reliance and responsiveness through the introduction of Reliance Margin (RM) and Counterfactual Flip Rate (CFR), which allow models to be characterized by static reliance versus dynamic responsiveness, as noted in Reliance Margin (RM) quantifies the performance change when the correct property–task pairing is disrupted by shuffling.
Developing robust model diagnostics by enabling systems to measure specific behavioral metrics, such as RM and CFR, which reveal variations in property utilization that traditional accuracy metrics fail to reflect. This capability allows for a diagnosis of whether higher accuracy with a given property does not necessarily imply greater responsiveness to that property.
Enhancing performance across diverse molecular inputs by allowing models to benefit from complex inputs, as shown by the finding that extending the strongest property bundle to all properties improves recovery for Sol and Astra but reduces it for Luna and Opus. This guides training strategies toward better handling of combined physical evidence.
Improving model interpretability by providing case studies that highlight limitations in chemical heuristics, such as showing Models make more errors when property values conflict with local heuristics, thereby enabling systems to recognize when explanations can conflict with candidate-level physical values.

Abstract

Large language models (LLMs) are increasingly used in molecular discovery and structure elucidation, where decisions depend on physical observations. Existing evaluations primarily focus on end-to-end molecular reconstruction, where correct predictions do not necessarily indicate reliance on the supplied properties. We introduce MolP2SBench, a benchmark for molecular property-to-structure (P2S) reasoning in LLMs. Each task presents a masked molecular scaffold with given properties and asks the model to select the missing structure from plausible candidates. We shuffle or replace properties to test reliance on correct property-structure pairing and responsiveness to counterfactual property changes. A channel-wise leaderboard uses Property-Grounded Recovery (PGR), giving equal weight to source and counterfactual recovery. Evaluating seven models, we find that LLMs can use physical properties for structure prediction, but performance varies substantially across property types. Higher accuracy with a property does not necessarily imply greater responsiveness to it: a model may benefit more from 1 H NMR than from 13 C NMR, yet be less likely to change its prediction when the 1 H data is replaced. LLM performance also need not reflect the relative difficulty of these properties for human chemists. Although UV-Vis spectra provide less direct structural information than NMR spectra, Astra achieves higher accuracy with UV-Vis data than with NMR peaks. MolP2SBench supports studying how LLMs extract and use physical-property information, revealing patterns that may differ from conventional chemical intuition and providing a foundation for more reliably property-grounded models for molecular and quantum-chemical inference. Code and dataset: https://github.com/DUAL-Xiao/MolP2SBench.