More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses
cs.AI, cs.LG, cs.SE
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: 26 pages, 4 figures, including references and appendices. Under review at ICLR 2027. Code: https://github.com/StatXzy7/harness-eval
Code: https://github.com/StatXzy7/harness-eval
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Evaluating Large Language Models Trained on Code
- From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
- HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems
- It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
- Inefficiencies of Meta Agents for Agent Design
- Automated Discovery Has No Universally Superior Harness
- Automated Design of Agentic Systems
- When Should LLMs Search? Counterfactual Supervision for Search Routing
- LLM-as-a-Verifier: A General-Purpose Verification Framework
- More Agents Is All You Need
- Let's Verify Step by Step
- Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
- HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
- GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents
- TTHE: Test-Time Harness Evolution
- RouteLLM: Learning to Route LLMs with Preference Data
- An Empirical Study of the Non-determinism of ChatGPT in Code Generation
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems