MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models

arXiv:2610.21686 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models".

Dev: The gist: MAIA introduces a multimodal framework that anticipates interaction anomalies in contact-rich manipulation by modeling how robot–environment interactions evolve under planned actions,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Now we look at the title and authors for this MAIA paper, which is "MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models." It really tells you the whole story about what they’re trying to achieve.

Dev: The authors are Yi Qiao, Maolin Zheng, Haoran Sun, Lujia Dong, and Wenzhao Lian. They're working on how to use a multimodal world model to anticipate anomalies in contact-rich manipulation tasks.

Taro: That means they’re looking at situations where the robot is touching things—like grasping or pushing—and they want to predict when that interaction might go awry using different types of input, like vision and force data.

Rosa: Exactly. The paper uses a multimodal world model, which is this system that learns expected interaction dynamics from nominal executions and then predicts future visual and force responses under planned actions.

Dev: So it's not just looking at what’s happening now; it’s using the model to forecast the sequence of events that will follow a specific plan.

Taro: I wonder how much of that prediction is based on the vision versus the force feedback, since they are both inputs to this model.

Rosa: That’s a good question because they specifically build an action-force pathway into their world model, conditioning its predictive rollout on recent force history and future policy actions.

Dev: So it's not just a static prediction based on the current state; it’s conditioned by what the robot is currently doing and what the policy plans to do next.

Taro: It sounds like they are trying to build a system that understands the physical consequences of a sequence of actions in contact manipulation.

Rosa: Precisely. The authors introduce this framework, MAIA, as a way to formalize contact-rich anomaly anticipation as prospective action evaluation: predicting whether planned actions will cause the robot and environment interaction that deviates from its expected dynamics.

Dev: So it’s framing the problem not as detection, but as prediction of failure modes based on what happens next under a specific plan.

Taro: I think that shift from reactive monitoring to prospective evaluation is what makes this paper different in the contact-rich domain.

The paper's summary: Rosa: So let’s talk about the summary of MAIA, which is essentially saying they formulated contact-rich anomaly anticipation as prospective action evaluation to predict whether planned actions will cause a deviation from expected dynamics.

Dev: They instantiate this formulation in MAIA, a framework for Multimodal Anticipation of Interaction Anomalies. At its core, the vision-action-force world model learns expected interaction dynamics from nominal executions and predicts future visual and force responses under planned actions.

Taro: And they augment that model with a lightweight anomaly head to integrate those predictive representations to estimate whether an anomaly will occur and its temporal prevalence over the future horizon.

Rosa: That head estimates two key quantities, yt, which indicates whether any anomaly occurs, and rt, which measures its temporal prevalence. So it tells us if an anomaly happens and how likely it is to last over the future time period they are predicting.

Dev: This is crucial because conventional failure-oriented datasets don't provide the temporal supervision needed to train this kind of prediction effectively.

Taro: So they introduced CRIA, which is a real-world multimodal dataset for Contact-Rich Interaction Anomaly anticipation that has two thousand eight hundred fifty-two trajectories covering eighteen contact-rich manipulation tasks with frame-level anomaly labels <ref:2610.21686#pg2,a real-world multimodal dataset for Contact-Rich Interaction Anomaly anticipation>.

Rosa: This dataset provides the necessary ground truth for training their framework, and it covers a lot of interaction categories like misalignment, motion blockage, and trajectory deviation.

Dev: So the summary is that they use this model to predict if an action will cause a deviation from expected dynamics using vision, action, and force data to estimate anomaly occurrence over time.

The paper's improvements: Rosa: The improvements they point out are quite significant; first, MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings. They got bACC scores of zero point seven six five on seen tasks and zero point seven one zero on novel tasks, with median anticipation lead times of twenty-eight point zero and twenty-five point one steps, respectively.

Dev: And they show that this predictive capability leads to a huge improvement in task success from fifty-two point six percent to sixty-one point five percent when using these predictions for anomaly-aware action selection without retraining the underlying manipulation policy.

Taro: So we're talking about using the prediction to proactively choose the lowest-risk action for online execution, which is a huge step forward from just reacting after a failure occurs.

Rosa: And they also show limited-data adaptation within known categories; this means they can achieve a bACC of zero point seven four eight on novel tasks when adapted with limited task-specific data.

Dev: That’s the part where it gets really useful because it shows that the framework can transfer its predictive interaction representations across diverse contact regimes without needing massive amounts of new data for every single change.

Taro: So, joint adaptation with the anomaly head improves performance rapidly in novel-task adaptation, reaching a bACC of zero point seven six two with limited data, suggesting adapting the interaction representation is more effective than adjusting the anomaly head alone for novel-task transfer.

Rosa: And they also showed that integrating these predictions into action selection guides real-robot deployment to improve overall task success rates from fifty-two point six percent to sixty-one point five percent. That’s a direct link between prediction and improved operational performance in the physical world.

Conclusion: Dev: So to wrap up, MAIA introduces a framework for anticipating interaction anomalies in contact-rich manipulation via multimodal world models that uses prospective action evaluation to predict deviations from expected dynamics.

Rosa: It uses a vision-action-force data stream to estimate whether an anomaly occurs and its temporal prevalence over the future horizon using predictions on yt and rt.

Taro: The implication for me is that we have a system that can proactively choose actions based on predicted risk, which means we're moving toward systems that are more robust in unexpected situations.

Dev: And the paper’s finding that it boosts task success to sixty-one point five percent without retraining the underlying manipulation policy really highlights how valuable this predictive capability is for deployment.

Rosa: We’re looking at a system that can adapt its understanding of physical interactions to new contact regimes using limited data and still perform reliably in real-world scenarios.

Taro: It’s a solid step toward making robotic manipulation safer by having it predict risks before they happen, which is something I think we need to keep pushing for in the next phase.

Dev: So that’s our take on MAIA, anticipating interaction anomalies in contact-rich manipulation via multimodal world models.

Shanghai Jiao Tong University

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-16

Project page: https://joringell.github.io/MAIA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: MAIA introduces a multimodal framework that anticipates interaction anomalies in contact-rich manipulation by modeling how robot–environment interactions evolve under planned actions,

Key concepts

Vision–Action–Force World Model (VAF-WM): This model learns the expected physical dynamics of robot-environment interactions. It takes visual inputs, planned actions, and recent force history to predict what the environment will look like visually and how forces will react next. This allows the system to understand the 'nominal' or expected behavior during manipulation.
Anomaly Head: This is a small prediction component attached to the world model. It takes predictions from the VAF-WM (visual latents, interaction tokens, and future force sequences) and uses a temporal transformer to estimate two things: whether an anomaly will occur ($y_t$) and how likely it is over the next few steps ($r_t$)
CRIA Dataset: This is a real-world dataset of 2,852 trajectories covering 18 different contact-rich manipulation tasks. It includes frame-level labels indicating exactly when an anomaly starts and how it develops over time. This data is used to train MAIA to accurately predict interaction failures.

Terminology

World Model — 1× in this paper · explained in 7 episodes

Summary

The gist: MAIA introduces a multimodal framework that anticipates interaction anomalies in contact-rich manipulation by modeling how robot–environment interactions evolve under planned actions, enabling proactive anomaly-aware action selection and adaptation to novel tasks without retraining the underlying policy.

How it works

MAIA addresses the challenge of reactive monitoring by formulating contact-rich anomaly anticipation as prospective action evaluation, which involves predicting whether planned actions will cause the robot–environment interaction to deviate from expected dynamics. At its core, the framework utilizes a Vision–Action–Force World Model (VAF-WM) that learns nominal interaction dynamics and predicts future visual and force responses under planned actions. This model is extended with an action–force pathway that conditions its predictive rollout on recent force history and future policy actions, resulting in action-force context tokens C AF.

Key Components

The MAIA framework consists of two main parts: the Vision–Action–Force World Model (VAF-WM) and a lightweight anomaly head. The VAF-WM is initialized from a pretrained GE-Base backbone and augmented with an action–force pathway that conditions its predictive rollout on recent force history and future policy actions. This conditioning is achieved by encoding the future action sequence into action tokens, which query the temporal force representation through cross-attention, while a global summary modulates the resulting action-aligned features to produce action-force context tokens C AF.

The predictive anomaly head receives three predictive signals—predicted visual latents Zˆ V, interaction tokens Z I, and future force sequence Fˆ—from the VAF-WM. It then uses a temporal transformer to model dependencies across the prediction horizon, estimating whether an anomaly will occur and its temporal prevalence over the future horizon. This head estimates two key quantities: yt, which indicates whether any anomaly occurs, and rt, which measures its temporal prevalence.

Data and Training

To enable systematic evaluation, the framework is trained using CRIA, a real-world multimodal dataset for Contact-Rich Interaction Anomaly anticipation. CRIA comprises 2,852 trajectories spanning 18 contact-rich manipulation tasks and provides frame-level annotations of anomaly onset and evolution. The training is conducted in three stages: nominal interaction dynamics learning, predictive anomaly-head training, and limited-data adaptation to novel tasks.

Results and Contributions

Experiments demonstrate that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings. Specifically, MAIA achieved bACC scores of 0.765 on seen tasks and 0.710 on novel tasks, with median anticipation lead times of 28.0 and 25.1 steps, respectively. Furthermore, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. The framework also supports limited-data adaptation within known categories, achieving a bACC of 0.748 on novel tasks when adapted with limited task-specific data.

Ablation and Adaptation

Ablation studies show that adding force feedback significantly improves anomaly discrimination, as MAIA-VA improved bACC by 0.139 and 0.094 compared to MAIA-V. In novel-task adaptation, joint adaptation with the anomaly head improves performance rapidly, reaching a bACC of 0.762 with limited data, suggesting that adapting the interaction representation is more effective than adjusting the anomaly head alone for novel-task transfer. Finally, integrating predictions into action selection guides real-robot deployment to improve overall task success rates from 52.6% to 61.5%.

Limitations

Limitations include the finite-horizon forecasting and fixed-length history which may limit prediction reliability over longer horizons, and the lack of cross-embodiment and cross-sensor generalization due to experiments being conducted on a single robot platform and fixed force/torque sensing setup. Additionally, the framework focuses on anomaly anticipation rather than root-cause diagnosis or autonomous recovery, motivating future work in closed-loop control.

References

[1] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai et al., “π00: a vision-language-action model with open-world generalization,” arXiv preprint arXiv:2504.16054, 2025

[3] M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang et al., “Cosmos policy: Fine-tuning video models for visuomotor control and planning,” arXiv preprint arXiv:2601.16163, 2026

[4] T. Pang, H. T. Suh, L. Yang, and R. Tedrake, “Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models,” IEEE Transactions on robotics, vol 39, no 6 pp 4691–4711, 2023

[5] X. Cheng, E. Huang, Y. Hou, and M. T. Mason, “Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d,” in 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022 pp 2730–2736

[6] C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez et al., “Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies,” arXiv preprint arXiv:2503.08558, 2025

[7] Q. Gu, Y. Ju, S. Sun, I. Gilitschenski et al., “Safe: Multitask failure detection for vision-language action models,” Advances in Neural Information Processing Systems, vol 38 pp 40 041–40 076, 2026

[8] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” arXiv preprint arXiv:2304.13705, 2023

[9] N. He, S. Li, Z. Li et al., “Rediffuser: Reliable decisionmaking using a diffuser with confidence estimation,” in Forty-first International Conference on Machine Learning, 2024

[10] S. Zhou, B. Zhu, J. Yang et al., “Rc-nf: Robot-conditioned normalizing flow for real-time anomaly detection in robotic manipulation,” arXiv preprint arXiv:2603.11106, 2026

[15] H. Zhang, Y. Lu, B. Wang et al., “Foresight: Failure detection for long-horizon robotic manipulation with action-conditioned world model latents,” arXiv preprint arXiv:2606.23085, 2026

[19] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in 2017 IEEE international conference on robotics and automation (ICRA), IEEE, 2017 pp 2786–2793

[30] D. Sliwowski, S. Jadav, S.

Improvements for AI systems

No longer reactive monitoring, MAIA can anticipate interaction anomalies by modeling how robot–environment interactions evolve under planned actions. This allows for a shift from reactive detection of failures to proactive safeguarding by predicting whether planned actions will cause an interaction that deviates from expected dynamics.
Anomaly-aware action selection enables robots to Select the lowest-risk action for online execution, improving end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. This means the system can proactively choose safer motions during real-robot deployment based on predicted risk, rather than relying solely on post-hoc failure detection.
The framework can achieve limited-data adaptation within known categories, as MAIA learns transferable predictive interaction representations across diverse contact regimes and achieves a bACC of 0.710 on novel tasks using only 10 trajectories per task without tuning the core model. This allows for rapid deployment in new manipulation scenarios with minimal task-specific fine-tuning data.
The system can perform Anomaly Discrimination by explicitly modeling how visual and force responses evolve under planned actions, which provides physically grounded evidence for identifying anomalous interactions that might be missed by methods relying only on visual or action cues.
The robot can estimate the temporal prevalence of an anomaly, achieving an MAE of 0.28 on novel tasks, providing a measure of how likely an interaction is to become anomalous over the full future horizon. This helps in prioritizing interventions based not just on occurrence, but on expected duration or intensity.