CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

arXiv:2608.08392 · cs.AI, cs.CL · Submitted 2026-08-09 · Read on arXiv

Zejun Xu, Taiyi Chen, Jin Li, Yongtong Gu, Qi Cheng, Aixuan Lv, Shuai Zhu, Pengfei Zhu, Kaichen Yang, Boyu Sun, Yixian Yang, Mulong Xie, Xin Liu, Dagang Li, Xiaoteng Ma, Hongru Wang

cs.AI, cs.CL

Submitted: 2026-08-09

Updated: 2026-08-11

Comments: Accepted to COLM 2026. Project page: https://warriorxu0302.github.io/CAP-Bench/

Code: https://github.com/browser-use/browser-use

Project page: https://warriorxu0302.github.io/CAP-Bench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 46× in this paper · explained in 2 episodes
• Verification — 30× in this paper · explained in 2 episodes
• Core — 8× in this paper · explained in 2 episodes
• Evaluation Framework — 7× in this paper · explained in 3 episodes
• Papers — 7× in this paper · explained in 3 episodes
• Editor — 6× in this paper · explained in 2 episodes
• Robustness — 5× in this paper · explained in 5 episodes
• Coherence — 4× in this paper · explained in 2 episodes
• Generator — 4× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Directed Acyclic Graph (DAG) — 2× in this paper · explained in 2 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Hallucination — 2× in this paper · explained in 4 episodes
• Provenance — 2× in this paper · explained in 5 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Computational overhead — 1× in this paper · explained in 2 episodes
• Evaluation Protocol — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 3 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• WebCanvas: Benchmarking Web Agents in Online Environments
• BEARCUBS: A benchmark for computer-using web agents
• Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary
• Acting Less is Reasoning More! Teaching Model to Act Efficiently
• The Rise and Potential of Large Language Model Based Agents: A Survey

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems