INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models

The gist The benchmark reveals persistent gaps between parameter-level correctness and scientifically valid configurations, particularly in physically coupled settings such as DFT+U, magnetism, and correlated materials.

Benchmark Design

1. The generation task contains 48 materials drawn from 16 representative material families and covers a broad range of scientific scenarios commonly encountered in first-principles calculations, including metals, semiconductors, insulators, oxides, correlated materials, layered materials, magnetic materials, and battery materials The generation task contains 48 materials drawn from 16 material families and covers a broad range of scientific scenarios commonly encountered in first-principles calculations.

2. Each material is paired with four representative VASP workflow types: static self-consistent field (SCF) calculation, geometry relaxation, band-structure non-self-consistent field (NSCF) calculation, and density-of-states (DOS) NSCF calculation.

3. Applying all four workflow types to all 48 materials yields 192 generation cases.

4. The repair task is derived directly from the generation task.

5. Each generation case produces three repair variants: a correct-to-preserve variant, a workflow-mistake variant, and a physics-mistake variant.

6. The repair task results in 576 repair cases in total.

7. Challenge types are used throughout the evaluation to analyze model performance across different scientific regimes and to localize failure modes.

Evaluation Framework

1. Configuration quality is evaluated through three complementary dimensions: Semantic Correctness, Policy Correctness, and Task-Critical Correctness (TCC).

2. Semantic correctness measures whether the intended physical meaning of a calculation is correctly encoded.

3. Policy correctness evaluates whether numerical and methodological choices are appropriately configured.

4. TCC evaluates whether all task-critical requirements of a calculation are simultaneously satisfied.

5. Semantic parameters determine the physical meaning of a calculation and include settings associated with physical effects and workflow intent, such as ISPIN, MAGMOM, LDAU, and ICHARG.

6. Policy parameters define numerical and methodological choices, including ENCUT, EDIFF, SIGMA, and NELM.

7. Task-critical parameter groups are defined separately and consist of semantic and policy parameters whose incorrect configuration would invalidate the intended calculation.

8. A semantic parameter is counted as correct only when the normalized prediction exactly matches the reference value, defined by Ssem = 1/Psem X p∈Psem 1 [ˆp = p∗].

9. For policy parameters, correctness is evaluated using parameter-specific equivalence or tolerance rules where Spol = 1/Ppol X p∈Ppol 1 [dp(ˆp, p∗) ≤ τp].

10. The overall parameter-level score is defined as Soverall = Ssem + Spol.

11. TCC is a strict case-level metric where a single error in any task-critical parameter group causes the entire case to fail.

Results and Failure Modes

1. GPT-5.4 achieves the highest overall score (84.7%), followed by GPT-5.4 XH and GPT-5.4 NR.

2. TCC remains substantially lower across nearly all model families, demonstrating that parameter-level correctness does not necessarily translate into task-critical correctness.

3. Semantic and policy correctness are only partially coupled, with the Policy−Semantic gap varying substantially across models.

4. Models with positive gaps achieve stronger policy performance than semantic performance, suggesting they reproduce common parameter patterns more successfully than they capture the underlying physical intent of the calculation.

5. Models with negative gaps exhibit stronger semantic understanding than policy fidelity, indicating that they often identify the correct physical workflow while failing to fully recover the corresponding numerical strategy.

6. Repair performance remains far from saturation across all evaluated models.

7. Preservation rates are generally higher than error-fix rates, indicating that maintaining valid configurations is generally easier than correcting injected errors.

8. TCC remains substantially lower than both component-level repair metrics, highlighting the difference between local parameter correction and recovery of a scientifically valid configuration.

9. The Fix−Preserve gap measures the balance between active error correction and configuration preservation, with most models exhibiting negative values indicating that preservation generally exceeds error correction.

Failure Localization

1. Difficulties are concentrated in specific material–challenge combinations rather than being uniformly distributed across the benchmark.

2. The most challenging regions occur in transition-metal oxides, correlated oxides, and battery materials when DFT+U-related constraints are present.

3. Magnetic materials become substantially more difficult under magnetic configurations, while layered materials exhibit reduced performance in vdW-related settings.

4. Failures arise primarily from interactions between material-dependent physics and challenge-specific constraints rather than from isolated parameter prediction errors.

Conclusion

INCARBench treats first-principles input configuration as a scientific capability rather than a formatting task.

The benchmark suggests that scientific configuration quality cannot be adequately characterized by parameter-level accuracy alone.

The observed gaps among semantic correctness, policy correctness, and task-critical correctness indicate that physical understanding, numerical-policy selection, and workflow consistency represent related but distinct capabilities.

Conventional parameter-matching metrics may overestimate the ability of LLMs to support real scientific workflows.

The benchmark reveals persistent gaps between parameter-level correctness and scientifically valid configurations, particularly in physically coupled settings such as DFT+U, magnetism, and correlated materials.

It also shows that error correction and configuration preservation are distinct capabilities in scientific repair.

These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for more reliable AI-driven scientific workflows.

All benchmark data have been uploaded to GitHub at https://github.com/liuzhiyangnku/incarbench.

The gist The benchmark reveals persistent gaps between parameter-level correctness and scientifically valid configurations, particularly in physically coupled settings such as DFT+U, magnetism, and correlated materials.

How it works

The benchmark assesses whether models can translate scientific intent into computational configurations and modify existing configurations while preserving valid settings. Configuration quality is evaluated through three complementary dimensions: Semantic Correctness, Policy Correctness, and Task-Critical Correctness (TCC). These dimensions assess physical intent encoding, numerical-policy selection, and satisfaction of workflow-defining and physics-defining requirements respectively.

Parameters are grouped into three categories: semantic parameters, policy parameters, and optional parameters. Semantic parameters determine the physical meaning of a calculation and include settings associated with physical effects and workflow intent, such as ISPIN, MAGMOM, LDAU, and ICHARG. Policy parameters define numerical and methodological choices, including ENCUT, EDIFF, SIGMA, and NELM. Optional parameters are retained for completeness and reproducibility but are not considered task-defining requirements.

1. Accurate scientific configuration generation system: The improved AI system can generate executable VASP INCAR files that satisfy task-critical requirements by explicitly encoding physical intent into semantic, policy, and task-critical requirements.

2. Robust configuration repair agent: The system can perform configuration repair by distinguishing between error-fix rate and preservation rate to avoid unnecessary modifications to already-correct configurations, specifically targeting the Fix–Preserve gap to adopt either aggressive or conservative editing strategies as needed.

3. Failure mode localization module: The improved system can diagnose why LLMs fail by analyzing performance across material families and challenge types, identifying specific scientific regimes like transition-metal oxides, correlated oxides, and battery materials when DFT+U-related constraints are present where failures are concentrated.

Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+ U, magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science.