Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

arXiv:2610.09240 · cs.CR, cs.AI, cs.CL · Submitted 2026-10-07 · Read on arXiv

cs.CR, cs.AI, cs.CL

Submitted: 2026-10-07

Updated: 2026-10-07

Comments: 20 pages, 8 figures, 6 tables. Code: https://github.com/MoonTea0416/WebMirage

Code: https://github.com/MoonTea0416/WebMirage

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Grounding — 1× in this paper · explained in 3 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen2 Technical Report
• GPT-4 Technical Report
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
• WIPI: A New Web Threat for LLM-Driven Web Agents
• Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments
• WebGPT: Browser-assisted question-answering with human feedback
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• Attacking Vision-Language Computer Agents via Pop-ups
• AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery
• SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents
• WebInject: Prompt Injection Attack to Web Agents
• An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
• GPT-4o System Card
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• CogAgent: A Visual Language Model for GUI Agents
• VILA: On Pre-training for Visual Language Models
• OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Estimating the Social Cost of Corporate Data Breaches
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills