LMEB: Long-horizon Memory Embedding Benchmark

arXiv:2603.12572 · cs.CL · Submitted 2026-03-13 · Read on arXiv

cs.CL

Submitted: 2026-03-13

Updated: 2026-10-12

Comments: 32 pages, 7 figures, 22 tables

Code: https://github.com/embeddings-benchmark/mteb

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 3× in this paper · explained in 2 episodes
• Long-horizon Memory Embedding Benchmark — 2× in this paper
• Abstraction — 1× in this paper · explained in 1 episode
• Embeddings — 1× in this paper · explained in 2 episodes

Sources

• jina-embeddings-v5-text: Task-Targeted Embedding Distillation
• Toward Conversational Agents with Context and Time Sensitive Long-term Memory
• Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• SWE-Exp: Experience-Driven Software Issue Resolution
• PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering
• Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
• Memp: Exploring Agent Procedural Memory
• Human-inspired Episodic Memory for Infinite Context LLMs
• jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval
• KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model
• A Benchmark for Procedural Memory Retrieval in Language Agents
• REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
• Toward Multi-Session Personalized Conversation: A Large-Scale Dataset and Hierarchical Tree Framework for Implicit Reasoning
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
• ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
• VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
• Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering