MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?".

Jane: The gist Large language models (LLMs) can leverage physical properties for molecular structure prediction, but their reliance on this data varies substantially across models and property types,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, to build on that idea of varied reliance, what’s the core methodology of MolP2SBench in simpler terms? It sets up a task where the AI gets a molecular scaffold and some properties, and it has to pick the missing structure from candidates.

Tom: It formalizes property-to-structure prediction as a multiple-choice masked completion task. The whole point is to move beyond just getting the right final answer and instead test if the model actually uses those supplied physical properties during its decision process.

Lu: They introduce controlled intervention schemes to probe this, specifically removing properties entirely, shuffling them across tasks, or replacing them with counterfactual properties from another candidate in the same family. These interventions are designed to isolate whether a model's decision is robust or if it’s just brittle and reliant on one specific property pairing.

Meng: So they are testing three dimensions: recovery under the original property, static reliance by shuffling, and dynamic responsiveness by checking if predictions follow targeted substitutions when things start out correct. That sounds like a lot of work to set up.

Jane: It is detailed work because they want to disentangle those two concepts—static reliance versus dynamic responsiveness—which is what traditional accuracy metrics just can't do for us.

Tom: And the results show that these metrics give us different pictures. For example, Luna, Sol, and Astra all show a higher Reliance Margin for atomwise 1H NMR than for 13C peaks, but they have a lower Counterfactual Flip Rate in those cases <ref:2610.12522#pg2>.

Lu: That reversal between reliance and responsiveness is actually what motivates them to report those two metrics separately instead of just using one combined score. It shows that being relied upon doesn't automatically mean being responsive to change.

Jane: And when you look at combining properties, they found that simply adding the remaining properties to the strongest bundle doesn't always help across different models and property sets. It can actually hurt performance for some of the models, like Luna and Opus.

Meng: That's a practical warning for anyone trying to build multi-modal systems; sometimes mixing in more data just introduces more confusion than it solves when you don't have a strong reason why those pieces belong together.

Tom: It’s about finding that sweet spot where the physical evidence is used correctly without being overly constrained by heuristics that aren't actually supported by the data.

Lu: So, essentially, MolP2SBench is a systematic way to diagnose *why* an LLM makes a prediction, rather than just observing what the final prediction is.

Jane: So if we are looking at this paper for someone who doesn't read it, we can boil it down to this: they built a test to see if AI uses the physics they are given to make sense of a molecule.

The paper's summary: Tom: Let's talk about what the authors themselves suggest as improvements for using this benchmark. They focus on creating better ways to analyze and use these results, rather than just generating more data points.

Jane: They propose establishing a channel-wise recovery leaderboard using Property-Grounded Recovery, which they say balances source recovery with recovery under counterfactual substitutions, ensuring equal weighting across different property families.

Lu: That’s important because it lets us see model strengths without collapsing heterogeneous properties into one single score. They also report scores independently for each property channel, which reveals model-specific strengths clearly.

Meng: I like that idea of seeing the channel-wise results because it tells you exactly which physical properties the AI is actually good at using, rather than just giving you a general score.

Tom: They also suggest reporting Reliance Margin and Counterfactual Flip Rate separately because they measure fundamentally different things—static reliance versus dynamic responsiveness—which we saw earlier isn't always correlated.

Jane: And one of the most interesting things they point out is the case studies, where Astra was shown to have a superior ability to use local chemical structures for NMR interpretation, even when global dipoles contradicted local polarity rules.

Lu: That shows that models can struggle with conflicting information; they can recognize a local motif but fail to ground its consequence in the specific geometry of the candidate molecule they are choosing.

Tom: The limitation they note is that this framework is currently limited to small-molecule completion using computed properties and curated candidates, so it doesn't extend easily to experimental spectra.

Meng: That’s a fair point; if we want this benchmark to be useful for real drug discovery, we need it to handle the messy data from actual experiments, not just idealized computed values.

Jane: So the authors are essentially saying they’ve given us a better diagnostic tool to understand *how* an AI is using its knowledge, even if that knowledge isn't always perfectly aligned with physical reality.

The paper's improvements: Tom: So to wrap up the MolP2SBench paper, the main implication is that we need to stop just looking at whether an LLM can reconstruct a molecule and start testing if its choices are actually grounded in the physics it's given.

Jane: Right. It shows that property-to-structure reasoning is complex, and there isn't one single way for models to use physical evidence effectively across all property types.

Lu: The paper establishes this controlled framework as a reusable testbed for diagnosing evidence use and guiding future training strategies toward more reliable molecular inference.

Meng: From an engineering standpoint, this means we need better ways to diagnose where our models are failing—is it a failure to see the property, or is it a failure to apply the property correctly?

Lalam: I’m quite confident in this direction. Based on what I’ve seen across the literature, my assessment is that this framework for MolP2SBench provides a necessary foundation for training models that actually use molecular properties effectively.

Tom: Exactly. It moves us away from just hoping the model gets the structure right, toward actively engineering it to be physically sound.

Jane: And I think what we need to keep asking is how we can move beyond these curated small-molecule datasets to make this reasoning applicable in broader chemical discovery scenarios.

Lu: Future work needs to explore expanding molecular families and property channels by using LLM-assisted dataset construction while still requiring expert review and validation for the data integrity.

Meng: I'd want to see some benchmarks that test how these models handle real-world experimental spectra, because that’s where the practical problems live.

Lalam: My certainty is that this paper sets a very clear direction for making property-grounded molecular inference a measurable and achievable goal for large language models.

Conclusion: Tom: So we’ve gone through MolP2SBench, which is basically this controlled test to see if LLMs can actually reason from molecular properties to structure, not just guess the answer.

Jane: Yeah, and what they found is that it’s really about disentangling whether the AI is just mimicking correct structures or if it’s actually using the physical properties they were given.

Lu: The core idea there is using Property-Grounded Recovery to give equal weight to both recovering the original property and recovering a structure based on those counterfactual substitutions.

Meng: I’m still thinking about how much effort goes into building that benchmark, though. It sounds like a lot of work just to set up these controlled interventions.

Lalam: The paper shows that reliance and responsiveness can rank different ways across the various property channels they tested, which is what makes this whole framework so useful for diagnosis.

Tom: And it highlights how much reliance varies—Luna, Sol, and Astra all show different patterns when it comes to atomwise NMR versus those 13C peaks <ref:2610.12522#pg2>.

Jane: That’s a key point because it shows that you can’t just look at one number and think you know the model's behavior.

Lu: They also found that combining properties doesn't always improve things across all models; sometimes adding more data actually hurts the recovery for certain AI systems.

Meng: That tells us that we need to be careful when we try to feed these models massive bundles of information, because it doesn't automatically mean better performance.

Tom: Ultimately, MolP2SBench gives us a reusable testbed for figuring out where the evidence is actually being used and how we should adjust our training strategies for property-grounded inference.

Jane: It’s about moving past just getting high accuracy scores and actually understanding the reasoning behind those choices.

Lu: The authors leave the validity of the supporting chemical arguments open, which means we still have to be careful when interpreting what these models are actually "thinking."

Meng: So they established a controlled way to evaluate this, but they also pointed out that it only covers small-molecule completion with computed properties.

Tom: Right, so the next step is figuring out how to expand this beyond small molecules and into more complex chemical spaces.

Jane: Definitely. It’s a solid framework for diagnosing evidence use in molecular AI right now, but we still have a lot to explore before we get there.