MA-Bench: Measuring Operational Planning in Language Models
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-13
Updated: 2026-10-13
Comments: 32 pages. Dataset: https://huggingface.co/datasets/ScaleAI/MA-Bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on
Abstract
How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss κ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.
Sources
- PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
- ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Determinants of LLM-assisted Decision-Making
- COA-GPT: Generative Pre-trained Transformers for Accelerated Course of Action Development in Military Operations
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
- Reinforcement Learning with Rubric Anchors
- Remote Labor Index: Measuring AI Automation of Remote Work
- OpenAI o1 System Card
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- UserBench: An Interactive Gym Environment for User-Centric Agents
- The AI Productivity Index (APEX)
- ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks