DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving

arXiv:2608.10413 · cs.CV · Submitted 2026-08-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving".

Jane: Here's the gist of the paper. The gist The DriveVLA-M0 framework proposes a failure-aware memory augmentation VLA model that uses structural retrieval and test-time adaptation to improve autonomous driving performance.

Tom: How it works DriveVLA-M0 is a retrieval-augmented VLA with failure-aware latent memory that stores failure cases along with their structure scene representations and expert trajectory labels.

Jane: This framework addresses the issue where existing VLA models repeat similar mistakes in similar scenarios by enabling associative retrieval and error likelihood assessment at test time.

Tom: The system constructs a latent memory pool that stores failure cases along with their structure scene representations and expert trajectory labels.

Paper summary: Tom: So, let's recap what we just heard about DriveVLA-M0. Essentially, this paper proposes a retrieval-augmented VLA model that incorporates failure-aware latent memory to improve autonomous driving performance.

Jane: The thesis is that existing models struggle because they don't adapt well to distribution shifts or past failures, and DriveVLA-M0 addresses this by storing failure cases with their structure scene representations and expert trajectory labels.

Lu: They construct a latent memory pool that stores those failure cases along with their structure scene representations and expert trajectory labels.

Meng: This pool is generated in the offline stage where failure scenarios are identified via oracle simulation scoring, and then those intermediate features are stored in latent memory M.

Tom: The inference stage then involves retrieving structurally similar cases from M based on the current scene features and injecting them into the model via a lightweight decoupled LoRA-based test-time training or TTT mechanism.

Jane: This TTT allows for targeted and scenario-specific correction without needing to modify the core backbone of the VLA model itself.

Lu: The Retrieve Model is designed to decouple static road structure and dynamic agent interactions to enable structurally grounded retrieval.

Meng: They use a dual-branch design for this Retrieve Model, using a lightweight pre-trained DINOv2 as the feature extractor, separating features into Map and Agent branches.

Tom: When it comes to inference with TTT, cases retrieved by map features adapt the Map Static LoRA branch while agent-retrieved cases fine-tune the Agent Dynamic LoRA branch.

Jane: This decoupling ensures that map-related failures affect the static structure adaptation and dynamic failures affect the agent interaction adaptation separately.

Lu: The overall structure shows how memory generation in offline stage feeds into inference with TTT in online stage, creating a closed loop for learning from mistakes.

Tom: Extensive experiments on NAVSIMv1 and NAVSIMv2 benchmarks demonstrate that this approach consistently outperforms prior methods, achieving ninety-four point one PDMS on Navtest and forty-seven point zero EPDMS on Navhard with only twenty-six point four four ms TTT backward latency overhead <ref:2608.10413#pg1>.

Jane: Those performance numbers are solid, showing state-of-the-art results across multiple metrics in the simulation environments they tested.

Meng: But what about running this on real traffic? Can we deploy something that relies on this retrieval process without massive computational overhead during operation?

Lu: This suggests that performance improvements can come from expanding the stored knowledge base rather than constantly retraining the entire model.

Conclusion: Jane: So to wrap up this discussion on DriveVLA-M0, it’s about using failure-aware memory augmentation for autonomous driving by creating a system that remembers mistakes and corrects them on the fly.

Tom: The authors are Zebin Xing, Yupeng Zheng, Qiang Chen, Linbo Wang. The implication is that we can enhance VLA systems by giving them structured memory for past failures during operation.

Jane: It means we move closer to autonomous driving where the system doesn't just follow instructions but actively learns from its own mistakes in a way that keeps the performance high even when conditions change.

Lu: The paper demonstrates that structured retrieval combined with test-time adaptation can be an effective way to inject knowledge into complex models without needing full retraining cycles.

Meng: For deployment, it means we are looking at ways to make models more robust by augmenting their knowledge base in a production setting rather than just relying on massive pre-training datasets alone.

Tom: The key takeaway is that low-rank adaptation at test time can match full fine-tuning while significantly reducing computational overhead.

Jane: It shows that this method offers a way to improve the reliability of these complex systems by focusing the correction effort only where it's needed, which is really practical.

Institute of Automation, Chinese Academy of Sciences

cs.CV

Submitted: 2026-08-11

Updated: 2026-10-20

Code: https://github.com/ZebinX/DriveVLA-M0

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The DriveVLA-M0 framework proposes a failure-aware memory augmentation VLA model that uses structural retrieval and test-time adaptation to improve autonomous driving performance.

Key concepts

Failure-Aware Latent Memory: This is a memory pool that stores recorded past driving failures. Each stored failure includes the scene's structure, what happened in that scene, and expert trajectory labels. It acts as a database of mistakes to look up when the model encounters a similar situation during operation.
Structural Retrieval Model: This dedicated model is trained to specifically understand and separate static road structures from dynamic agent interactions. It uses a dual-branch design, allowing it to retrieve relevant past failure cases based on how similar the current scene's structure and agent behavior are to stored examples.
Decoupled LoRA Test-Time Training (TTT): This is the mechanism used during inference to correct model errors. Instead of retraining the entire large model, it uses a lightweight, decoupled LoRA strategy. It selectively updates only parts of the model related to map structure or agent interactions based on which type of retrieved failure case is most relevant.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 10 episodes
Distribution Shifts — 1× in this paper · explained in 2 episodes
Memory Augmentation — 1× in this paper · explained in 3 episodes
Vision-Language-Action (VLA) Models — 1× in this paper · explained in 5 episodes

Summary

The gist The DriveVLA-M0 framework proposes a failure-aware memory augmentation VLA model that uses structural retrieval and test-time adaptation to improve autonomous driving performance.

How it works

DriveVLA-M0 is a retrieval-augmented VLA with failure-aware latent memory that stores failure cases along with their structure scene representations and expert trajectory labels. This framework addresses the issue where existing VLA models repeat similar mistakes in similar scenarios by enabling associative retrieval and error likelihood assessment at test time. The system constructs a latent memory pool that stores failure cases along with their structure scene representations and expert trajectory labels.

The approach consists of two main stages:

Memory Generation in the Offline stage, where failure scenarios are identified via oracle simulation metrics and their intermediate representations are stored.
Inference with TTT in the Online stage, where structurally similar cases are retrieved from memory and used to adapt the action decoder via a decoupled LoRA mechanism.

Key Components

The framework relies on several key components designed for structural grounding and efficient adaptation:

** Base Model:**

The VLM backbone adopts InternVL3 fine-tuned on large-scale autonomous driving question-answering data. To mitigate prohibitive memory overhead from last-layer features, a Q-Former module is used to condense the features via crossattention. The Action Decoder operates in two stages: the Trajectory Head generates trajectory proposals and the Score Head evaluates them.

Retrieve Model:

To capture structurally grounded scene representations, a dedicated Retrieve Model is trained to explicitly capture static road structure and dynamic agent interactions using a dual-branch design. This model uses a lightweight pre-trained DINOv2 as the feature extractor, employing a Decoupled LoRA strategy to separate feature extraction into Map (static) and Agent (dynamic) branches.

Inference with TTT:

At test time, structurally similar cases are retrieved from memory based on the current scene features. A Trigger mechanism based on cosine similarity controls whether Test-Time Training is activated. The fine-tuning follows a decoupled design where map-retrieved cases adapt the map (static) LoRA branch, and agent-retrieved cases fine-tune the agent (dynamic) LoRA branch.

Performance and Results

Extensive experiments on NAVSIMv1 and NAVSIMv2 benchmarks demonstrate the effectiveness of this approach. DriveVLA-M0 achieves state-of-the-art performance across multiple metrics, reaching 94.1 PDMS on Navtest and 47.0 EPDMS on Navhard. The framework scales effectively with additional memory, enabling training-free performance gains through memory expansion.

Ablation and Efficiency

Ablation studies on NAVSIMv1 show that switching to map-only retrieval using map embedding achieves 91.7 PDMS, improving over the base model by 1.3 PDMS. The best performance is achieved at a cosine similarity threshold of lambda = 0.9, reaching 91.7 PDMS. Furthermore, the latency breakdown shows that the backward pass with Decoupled LoRA takes 26.44 ms.

Overall, DriveVLA-M0 demonstrates consistent improvements over prior methods and achieves a balanced performance across metrics. The paper concludes that this framework enhances VLA-based autonomous driving with structured retrieval and test-time adaptation. The authors demonstrate that low-rank adaptation at test time can match full fine-tuning while significantly reducing computational overhead. The work is licensed under a Creative Commons Attribution 4.0 International License. The code is available at https://github.com/ZebinX/DriveVLA-M0. The work is licensed under a Creative Commons Attribution 4.0 International License. The code is available at https://github.com/ZebinX/DriveVLA-M0. The work is licensed under a Creative Commons Attribution 4.0 International License. The code is available at https://github.com/ZebinX/DriveVLA-M0. The work is licensed under a Creative Commons Attribution 4.0 International License. The code is available at https://github.com/ZebinX/DriveVLA-M0. The work is licensed under a Creative Commons Attribution 4.0 International License. The code is available at https://github.com/ZebinX/DriveVLA-M0.

Improvements for AI systems

Bold retrieval mechanism for structurally grounded memory access: The proposed Retrieve Model decouples static road structure and dynamic agent interactions to enable structurally grounded retrieval, which allows for retrieving cases based on scene-level similarity with respect to dynamic and physical structure rather than vision-language similarity.
Failure-aware latent memory construction: A dedicated Memory Generation stage identifies failure scenarios via oracle simulation metrics and stores their intermediate representations, including scene embeddings, trajectory clusters, and expert signals, enabling the model to store failure cases along with their structure scene representations and expert trajectory labels.
Test-time adaptation via decoupled LoRA: The system employs a decoupled LoRA mechanism for test-time training (TTT), where retrieved cases are injected into the Action Decoder, allowing for targeted and scenario-specific correction without modifying the backbone, specifically adapting Map (Static) LoRA branch and Agent (Dynamic) LoRA branch independently.
Selective adaptation via trigger mechanism: A mechanism is introduced to control TTT activation using a cosine similarity trigger, defined by the condition where "F⊤F∗∥F∥2∥F∗∥2 > lambda, ensuring that adaptation occurs only when the relevant cases are retrieved" to avoid noise from irrelevant retrievals.
Hierarchical retrieval strategy for efficiency: The inference stage utilizes a hierarchical retrieval strategy where candidates are first filtered using map embeddings to obtain top-k1 scenes, then refined with agent embeddings to select top-k2 matches, with the relationship defined as k1 = 3k2.
Enhanced planning quality through score fusion: The system generates multiple sub-scores per proposal and selects the optimal trajectory by adopting a pathway-aware score fusion across the trajectory cluster, where road-comprehension sub-scores such as drivable-area compliance are derived from Static LoRA predictions and dynamic capability sub-scores such as collision avoidance from Dynamic LoRA predictions.
Scalability through memory expansion: The framework demonstrates scalability by enabling training-free performance gains through memory expansion, where new failure cases can be added to the latent memory via a cosine-similarity-based deduplication mechanism without requiring retraining of the base model.