EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents
cs.CL, cs.CV
Submitted: 2026-01-23
Updated: 2026-08-31
Comments: EMNLP Findings 2026
Code: https://github.com/InternLM/EMemBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games.
Terminology
Abstract
We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games. Rather than using a fixed set of questions, EMemBench generates questions from environment-grounded trajectories, covering both text-only and visual game environments. Each template computes verifiable ground truth from underlying game signals, with controlled answerability and balanced coverage over memory skills: single/multi-hop recall, induction, temporal, spatial, logical, and adversarial. We evaluate memory agents with strong LMs/VLMs as backbones, using in-context prompting as baselines. Across 15 text games and multiple visual seeds, results are far from saturated: induction and spatial reasoning are persistent bottlenecks, especially in visual settings. Persistent memory yields clear gains for open backbones on text games, but improvements are less consistent for VLM agents, suggesting that visually grounded episodic memory remains an open challenge. A human study further contextualizes the difficulty and interpretability of EMemBench.
Sources
- MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems
- Qwen3-VL Technical Report
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
- Long Context vs. RAG for LLMs: An Evaluation and Revisits
- MemGPT: Towards LLMs as Operating Systems
- Qwen2.5 Technical Report
- Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
- StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Mem-{\alpha}: Learning Memory Construction via Reinforcement Learning
- A-MEM: Agentic Memory for LLM Agents
- Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
- Qwen3 Technical Report
- MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering