Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

arXiv:2610.22323 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation".

Dev: The gist: Mem4Act introduces a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control, addressing challenges in coverage, content,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So, we’re looking at this paper called Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation. It’s about how robotic agents need memory to actually do complex tasks that take a long time.

Dev: Yeah, it tackles the problem of agents needing to remember things from past observations and interactions to guide their future actions. Existing methods often just keep raw history or compressed visual tokens, but those representations either expose the policy to too much history or they might throw away important stuff like time, meaning, or perception.

Taro: So what’s the core claim here? What does Mem4Act actually propose as a solution?

Rosa: Mem4Act introduces a framework for learning effective latent memory that is fixed-size and optimized specifically for long-horizon robotic control. The main idea is to solve three coupled challenges: how to efficiently cover long histories, what information the memory should keep, and making sure the policy actually uses that memory.

Dev: That sounds like it’s tackling coverage, content, and utilization all at once. How does it go about covering those long histories?

Rosa: They organize history using multi-granularity temporal aggregation. This means they partition the history frames into different temporal windows at various levels and progressively aggregate local events into longer-term context. It involves decreasing the number of temporal windows across levels and repartitioning memory slots into fewer groups that align with increasingly broader time windows.

Taro: So, it’s not just dumping everything in a big bucket of memory, but structuring the time history differently?

Rosa: Exactly. They do that to organize the history structure itself. And they also focus on what information to retain through two types of supervision: semantic question answering and key-frame DINO feature reconstruction.

Dev: Semantic QA means they derive questions from subgoal annotations to preserve task-relevant facts, and DINO reconstruction uses key-frame features for perceptual supervision. That should give the memory some complementary semantic and perceptual content.

Taro: So the memory isn't just a blob of pixels or text; it’s trying to capture both what needs to happen semantically and what is visually important at specific moments?

Paper summary: Rosa: Right. And finally, they use a memory-first two-stage training strategy to make sure the policy actually relies on this learned memory. Stage one trains the informative memory representations with auxiliary supervision, and then stage two jointly adapts the policy and memory for control.

Dev: The second stage involves a learning rate multiplier of zero point one times for only the memory encoder when joint training happens, which is meant to keep its pretrained representation stable while letting it adapt to control. That’s a specific detail about how they train it.

Taro: So, if I were driving and the road suddenly changes unexpectedly, what would this system be doing based on what you just said? How does it handle that unexpected event?

Rosa: The resulting memory is integrated into action generation through memory-conditioned feature modulation. At each action-expert layer, the action hidden states attend to the memory Mt through cross-attention and those retrieved features modulate the feed-forward branch using a feature-wise scale and shift mechanism.

Dev: That means when it decides what to do next, it’s not just looking at what’s right now; it's combining direct current observations with information retained from the full third-person history. That should give the robot context beyond just its immediate sensory input.

Taro: So, when I ask the naive question—how long can this work outside of a perfect lab setting?—what does this paper suggest about its robustness in those messy, real-world conditions?

Rosa: They evaluated Mem4Act across RoboMME with sixteen tasks, RMBench with nine tasks, and four real-world manipulation tasks <ref:2610.22323#pg2>. They achieved success rates of forty-nine point eighty-eight percent on RoboMME, sixty-one point zero percent on RMBench, and eighty-one point five percent on the real-world tasks.

Dev: That’s a high success rate for memory-dependent control in the real world, compared to what other systems might achieve. The ablation studies show that temporal organization is more important than just increasing the memory capacity itself.

Paper summary: Taro: So, even if you give it a huge amount of memory slots, if the structure of how that history is organized isn't right, it won't work well in practice?

Rosa: That’s what they found. They also showed that combining semantic QA and DINO supervision gives the best joint-task performance at thirty-seven point sixty percent, which is better than relying only on action supervision.

Dev: And the two-stage training strategy, specifically using that zero point one times learning rate multiplier in stage two, resulted in the best overall performance at forty-nine point eighty-eight percent and showed a stronger reliance on memory that was specific to the rollout.

Taro: So what does this mean for someone who only listens to the show? It means these systems are designed not just to remember things, but to store and use that memory in a structured way so they can handle long, unpredictable sequences of actions.

Rosa: Basically, Mem4Act successfully learns this effective latent memory by addressing coverage, content, and utilization through structured temporal organization, complementary supervision methods—QA and DINO—and a staged training approach. It keeps the interface at five hundred twelve latent memory tokens while achieving state-of-the-art performance on these tasks.

Dev: It highlights that explicitly learning and utilizing structured memory is really important for reliable long-horizon robotic control, as the results show across different benchmarks.

Taro: For future work, I wonder if this structure holds up when the task complexity moves beyond what’s tested in RoboMME or RMBench. How does it generalize to completely new manipulation scenarios?

Rosa: That’s a valid question for future work, exploring how this multi-granularity aggregation handles truly novel sequences. It shows that structuring history is key, but scaling that structure to totally unseen environments is the next challenge.

Dev: So, the main point of Mem4Act remains learning a fixed-size memory representation that captures long-range evidence while keeping the policy context fixed at history size across different granularities. That’s what we’ve heard about this paper on Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation.

Conclusion: Rosa: So we've been looking at Mem4Act, which is about learning effective latent memory for long-horizon robotic manipulation across these different benchmarks.

Dev: Yeah, it's a framework that tries to figure out how an agent can keep track of things over a very long task without getting overwhelmed by all the history.

Taro: It seems like the paper tackles the core idea of having a fixed-size memory that actually helps guide future actions.

Rosa: Exactly, it’s about structuring that memory so it captures important things like what's happening at different times and what those things mean semantically or visually.

Dev: The authors use multi-granularity temporal aggregation to organize the history frames into different time windows. That sounds like a way to manage the sheer volume of past observations.

Taro: And they pair that up with semantic question answering and DINO feature reconstruction for supervision, which is interesting because it makes sure the memory isn't just random data.

Rosa: Right, it’s trying to ensure that whatever memory is stored actually holds information relevant to the task, not just noise.

Dev: Then they train it with a memory-first two-stage strategy, where they first train the memory and then joint train the policy and memory together. That's how they make sure the agent actually uses that learned structure for control.

Taro: So when you look at the results, it shows strong performance across RoboMME, RMBench, and real-world tasks, with success rates up to eighty-one point five percent in those real-world scenarios.

Rosa: That's a solid result for memory-dependent control outside of just controlled lab settings. It means agents can handle longer sequences reliably in messy environments.

Dev: It changes things for us because it suggests that the way we structure history and supervise that memory is more important than just having a bigger memory capacity.

Taro: If you only listen to this, it means the future of these systems isn't just about brute-forcing memory; it’s about learning a smart, structured way to remember things.

Rosa: And what we haven't covered yet is how this fixed-size memory handles completely novel manipulation scenarios that were not in the training data.

Yun Zhang, Zhiyu Huang, Tianhui Cai, Chen Tang, Wei Wang, Jiaqi Ma

University of California, Los Angeles · North Carolina State University

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Project page: https://handsomeyun.github.io/Mem4Act/ABSTRACT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: Mem4Act introduces a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control, addressing challenges in coverage,

Key concepts

Multi-granularity Temporal Aggregation
This method organizes past observations by grouping frames into windows of different temporal scales. It starts with fine temporal windows and progressively aggregates local events into longer contexts, allowing the memory to capture both immediate actions and broader historical context efficiently.
Memory Supervision (QA & DINO)
To ensure the memory retains useful information, Mem4Act uses two types of supervision. Question Answering uses task subgoals to preserve facts, while key-frame DINO feature reconstruction captures rich visual details from important past moments, ensuring complementary semantic and perceptual data are kept.
Memory-First Two-Stage Training
The policy learns to use the memory through a two-stage training process. The first stage trains the memory encoder using auxiliary supervision while keeping the main model frozen. The second stage jointly trains both components, carefully controlling the memory's learning rate to ensure it relies on its learned representations for control.

Terminology

Summary

The gist: Mem4Act introduces a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control, addressing challenges in coverage, content, and utilization through multi-granularity temporal aggregation, semantic and perceptual supervision, and memory-first two-stage training.

How it works

Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory (Page 1). To organize history, it uses multi-granularity temporal aggregation (Page 5), which partitions history frames into contiguous temporal windows at different levels and progressively aggregates local events into longer-term context (Page 5). This structure involves decreasing the number of temporal windows across levels, repartitioning the same memory slots into fewer groups aligned with increasingly broader temporal windows (Page 5).

How it works

To preserve complementary semantic and perceptual information, Mem4Act employs two specific supervision methods (Page 5). Semantic supervision is achieved through Question Answering, where memory-dependent questions are derived from subgoal annotations to preserve task-relevant facts (Page 5). Perceptual supervision is handled by key-frame DINO feature reconstruction, which reconstructs DINO features for up to K front- or head-view keyframes available at timestep t (Page 5).

How it works

The policy learns to utilize this memory through a memory-first two-stage training strategy (Page 5). Stage 1 focuses on learning informative memory representations with auxiliary supervision, while Stage 2 jointly adapts the policy and memory for control (Page 5). In Stage 1, the VLA backbone is frozen, and gradients propagate through the memory inputs to train the memory encoder (Page 5). Stage 2 then jointly trains the policy and memory components, with only the memory encoder using a learning rate multiplier of 0.1× to limit disruption to its pretrained representation (Page 5).

How it works

The resulting memory is integrated into the action generation process via memory-conditioned feature modulation (Page 5). At each action-expert layer, the action hidden states attend to the memory Mt through cross-attention, and the retrieved features modulate the feed-forward branch through a feature-wise scale and shift mechanism (Page 5). This design allows action generation to combine direct current observations with information retained from the full third-person history (Page 5).

Evaluation and Results

Mem4Act was evaluated on 29 long-horizon robotic manipulation tasks across RoboMME, RMBench, and four real-world manipulation tasks (Page 5). The framework achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5% on RoboMME, RMBench, and real-world tasks, respectively (Page 5). For the real-world tasks, Mem4Act achieves an average success rate of 81.50%, compared to 42.50% for GPT-6 Astra-xhigh (Page 8).

Ablation Studies

Ablations demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training (Page 5). Table 4 shows that temporal organization is more important than simply increasing memory capacity (Page 9). Furthermore, combining semantic QA and DINO supervision achieves the best joint-task performance (37.60%) and overall success (49.88%) compared to action supervision alone (Page 9). The two-stage training strategy, specifically the 0.1× learning rate multiplier in Stage 2, yields the best performance (49.88%) and shows stronger reliance on rollout-specific memory (Page 10).

Conclusion

Mem4Act successfully learns effective latent memory for long-horizon robotic manipulation by addressing coverage, content, and utilization through structured temporal organization, complementary supervision, and a staged training approach (Page 5). It achieves superior success rates across diverse tasks while maintaining a fixed-size interface of 512 latent memory tokens (Page 5). The results highlight the importance of explicitly learning and utilizing structured memory for reliable long-horizon robotic control (Page 5).

References

Johan Bjorck, Fernando Castaneda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, ˜ Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025 (Page 11).

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024 (Page 11).

Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao et al. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025 (Page 11).

Tianxing Chen, Yuran Wang, Mingleyang Li et al. RMbench: Memory-dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026a (Page 17).

Yipeng Chen, Wentao Tan, Lei Zhu et al. Non-markovian long-horizon robot manipulation via keyframe chaining. arXiv preprint arXiv:2603.01465, 2026b (Page 17).

Egor Cherepanov, Nikita Kachaev, Alexey Kovalev et al. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. In International Conference on Learning Representations, volume 2026, pp. 9759–9815, 2026 (Page 17).

Cheng Chi, Zhenjia Xu, Siyuan Feng et al. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025 (Page 7).

Yinpei Dai, Hongze Fu, Jayjun Lee et al. RoboMME: Benchmarking and understanding memory for robotic generalist policies. arXiv preprint arXiv:2603.04639, 2026 (Page 11).

Haoquan Fang, Markus Grotz, Wilbert Pumacay et al. SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation. arXiv preprint arXiv:2501.18564, 2025 (Page 17).

Gemini Robotics Team, Abbas Abdolmaleki et al. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer. arXiv preprint arXiv:2510.03342, 2025 (Page 17).

Songhao Han, Boxiang Qiu et al.

Improvements for AI systems

  1. Bold latent memory representation: Implement Mem4Act's fixed-size latent memory which is explicitly optimized for long-horizon robotic control, ensuring it captures both high-level task state and fine-grained perceptual evidence.

  2. Multi-granularity temporal aggregation: Utilize the paper's strategy of organizing history via multi-granularity temporal aggregation to progressively integrate local events into longer-term context, thereby addressing the challenge of how to efficiently cover long histories.

  3. Complementary memory supervision: Incorporate semantic question answering directly supervises memory to preserve task-relevant facts and keyframe DINO feature reconstruction retains perceptual information, ensuring the policy is supervised beyond just action chunks by targeting both task facts and visual structure across time.

  4. Memory-first two-stage training: Adopt the memory-first two-stage training pipeline, where Stage 1 establishes representations through auxiliary supervision, followed by Stage 2 where the policy learns to adapt and use it for control, encouraging memory utilization rather than relying on the current observation alone.

  5. Memory-conditioned action generation: Augment the base action expert with memory-conditioned feature modulation via cross-attention, allowing the policy to condition its actions using retrieved historical features through a mechanism where the resulting parameters modulate the input to the feed-forward branch.

Sources

Related papers