MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes

arXiv:2610.21429 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes".

Dev: The gist The Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM) dataset introduces 31 scenes combining spatial audio rendering and physical interaction across diverse architectures and object layouts,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we’ve seen how MuSTMM sets up this challenge by combining audio rendering and physical interaction across thirty-one scenes with varied layouts <ref:2610.21429#pg1,audio rendering and physical interaction across 31 scenes with>. The core idea is that existing benchmarks only test layout variation in limited structures, so they needed something broader. Dev They claim MuSTMM addresses this by covering both structure variation—different room geometries—and layout variation—where the sounds and objects are placed relative to each other.

Taro: It matters because it pushes the limits of what we know about sound-triggered manipulation outside of controlled lab settings where you can see everything. Rosa Exactly. It brings together audio-visual navigation, like SoundSpaces or SAVi, with mobile manipulation tasks that only test physical interaction in a limited structure variation without the sound rendering component.

Dev: The paper introduces this by saying that STMM requires robots to respond to sound events by navigating to objects and doing physical interactions, but current evaluations only vary layouts within limited structures. Taro So the contribution is building a benchmark that covers both structural diversity and layout variation for unseen scenes.

Rosa: They have thirty point 11k episodes spanning those thirty-one scenes, which means they have a lot of varied scenarios to test these capabilities on. Dev And they've included this scene-disjoint split so you can evaluate single-event and multi-event tasks in unseen scenes using the held-out set.

Taro: It also mentions that they used SoundSpaces to precompute fourteen point thirty-six million binaural Room Impulse Responses, which is a lot of data for spatial audio rendering. Rosa That RIR data is what allows the system to accurately model how sound travels in those specific room geometries.

Dev: And they augment the navigation policy model with explicit spatial memory derived from depth observations to complement these acoustic cues with observed geometry. Taro So they are using what the robot sees about its immediate surroundings—the depth—to create a map that helps it understand where it is in relation to the sound source.

Rosa: This memory is projected into an egocentric Bird's-Eye View occupancy map, and fusing that with sound context creates a goal descriptor with target-related and structural priors. Dev That descriptor then guides navigation to the region needed for physical interaction, which is the whole point of this augmentation.

Taro: It’s about using what you see and what you hear together to make a better navigation decision in a scenario where nothing has been explicitly trained for that exact combination of sound and structure.

Conclusion: Rosa: So looking at the MuSTMM paper, it’s clear that they’ve put together a comprehensive setup for studying sound-triggered mobile manipulation. They used thirty-one scenes with different architectures and layouts to create a benchmark <ref:2610.21429#pg1>. Dev The authors are showing how combining spatial audio rendering with physical interaction can be tested in these diverse environments using this new dataset.

Taro: What this means practically is that we start moving past testing robots only in perfectly controlled lab settings where the environment is completely known. Rosa Exactly. It opens up the door to studying robots navigating and interacting with things when they encounter unexpected room geometries or object placements, even when they don't have prior training for that specific sound trigger.

Dev: The focus on unseen scenes using that scene-disjoint split is key because it tests the system’s ability to handle novel situations without having a perfect map of every possible scenario. Taro It suggests that adding explicit spatial memory, where the robot accumulates depth data into an occupancy map and fuses it with sound context, is a necessary step for better navigation in these complex scenarios.

Rosa: Even though they found improvements in navigation success rates—like that two point sixty percent boost—they also highlighted that navigation itself remains a major bottleneck when dealing with multi-event tasks. Dev So, the implication is that while we can get better at navigating to the target, figuring out what physical interaction to perform next is still a hard problem in unseen scenes.

Taro: It points toward future research focusing on how agents recover gracefully when navigation fails or when they have to perform a sequence of actions based on multiple sounds. Rosa So, MuSTMM provides this new testbed for studying STMM in unseen scenes, even if it shows that we still need to work on making the execution part smoother.

Hao Ju, Feng Yan, Zhihao Zhong, Heng Yang, Shaofei Huang, Zhedong Zheng

University of Macau · Enacta-AI

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 71/100

The gist: The gist The Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM) dataset introduces 31 scenes combining spatial audio rendering and physical interaction across diverse architectures and object

Key concepts

Sound-Triggered Mobile Manipulation (STMM): A task where a robot moves and interacts with objects only when it detects specific sounds. The goal is to navigate to the sound source and perform the correct physical action, making sound a primary way for the robot to perceive its surroundings and interact with the world.
Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM): A new dataset featuring 31 scenes with varied architectures and object layouts. It combines high-quality spatial audio rendering (Room Impulse Responses) with physical interaction tasks, providing a comprehensive benchmark for testing sound-triggered manipulation in complex, unseen environments.
Explicit Spatial Memory: An augmentation added to the robot's navigation policy that stores observed depth data from its movements. This data is projected into an egocentric Bird's-Eye View (BEV) map. This memory helps the robot understand the physical layout of a scene, complementing sound cues by providing geometric context for where it needs to go.
Goal Descriptor Fusion: A process where features from the sound context and the spatial memory map are combined. This fusion creates a 'goal descriptor' that gives the robot two crucial pieces of information: exactly where the target is located and what structures exist in that area, guiding accurate navigation to perform physical tasks.

Summary

The gist The Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM) dataset introduces 31 scenes combining spatial audio rendering and physical interaction across diverse architectures and object layouts, providing a benchmark for studying sound-triggered mobile manipulation in unseen scenes.

Introduction and Motivation

Sound events provide a natural trigger for robots to perceive their surroundings and interact with physical objects Existing benchmarks provide only partial challenges of different scene structures and object layouts Audio-visual navigation benchmarks support sound rendering across multiple scenes but do not include the physical interactions required by STMM Conversely, mobile manipulation benchmarks support physical interaction in a limited structure variation without sound rendering STMM brings these capabilities together but evaluates object-layout variation within limited scene structures This motivates a benchmark that combines structural diversity with both sound rendering and physical interaction. MuSTMM expands variation at two complementary levels: structure variation covers differences in scene architecture, including room geometry and spatial connectivity, while layout variation combines episode-specific placements of sounding targets and silent source-capable distractors with scenespecific arrangements of non-source background furniture.

MuSTMM Dataset Composition

MuSTMM consists of 30.11k episodes and supports sound rendering and physical interaction across 31 scenes. Structure variation is covered by 31 scenes from HSSD (Khanna et al., 2024), covering 17 room counts ranging from four-room scenes to scenes with 17 rooms and uses SoundSpaces (Chen et al., 2022) to precompute binaural Room Impulse Responses (RIRs) across the 31 scenes, yielding 14.36 million RIRs. Layout variation includes episode-wise placement of sounding targets and silent source-capable objects, along with 855 nonsound-source household furniture assets spanning categories such as chairs and cabinets. The dataset uses a scene-disjoint split, with 23 training scenes and 8 held-out test scenes, enabling evaluation of both single-event and multi-event tasks in unseen scenes.

Navigation Policy Model Augmentation

The Navigation Policy Model is augmented with explicit spatial memory to complement acoustic cues with observed geometry. This involves accumulating pose-aligned depth observations in a shared coordinate frame and projecting them into an egocentric Bird’s-Eye View (BEV) occupancy map. The map records occupancy and exploration status, and its features are fused with acoustic and temporal context to form a goal descriptor that provides both target-location and scene-structure priors. This fusion guides navigation to the target region required for subsequent physical interaction.

Policy Execution Framework

The framework consists of a Sound-triggered Task Planner and a set of learned, skill-specific policy models. The high-level planner predicts an ordered skill chain based on initial RGB and audio observations. Among the low-level policy models, the navigation policy model utilizes a Spatially augmented Goal Descriptor Branch that predicts the location of the sound source. This branch unprojects depth observations into world coordinates to form a spatial memory, which is then projected into an egocentric BEV occupancy map. The policy branch then processes this fused map feature with the spatio-temporal memory to predict navigation actions.

Performance and Ablation Results

Navigation Policy Model with explicit spatial memory achieves a navigation success rate of 75.28%, outperforming existing methods. Memory ablations show that combining explicit spatial memory with compressed spatio-temporal memory outperforms either component alone, highlighting their complementary roles in across-scene sound navigation. Quantitative results show that SSP achieves the best non-oracle performance in both planning and completion for object relocation in the single-event setting, with a CSR of 62.51%. The success rate of the navigation policy model is reported at 75.28% when comparing it to existing navigation policies like SAVi (15.68%) and MAGNet (71.47%).

Conclusion

MuSTMM introduces multiple scenes into sound-triggered mobile manipulation while supporting spatial audio rendering and physical interaction. The scene-disjoint protocol enables evaluation of single-event and multi-event tasks in unseen scenes. To improve navigation for subsequent interaction, the Navigation Policy Model was augmented with explicit spatial memory where depth observations are accumulated and represented as a BEV occupancy map. Features are fused with sound context to provide target-location and scene-structure priors. Navigation comparisons and memory ablations show that explicit spatial memory complements compressed spatio-temporal memory to improve sound-source navigation. Despite these gains, navigation remains a major execution bottleneck in unseen scenes, particularly for multi-event tasks.

AI Use Statement

In this work, we used generative AI tools (e.g., ChatGPT) to edit the research paper to improve readability, specifically for grammar and spelling checks We have not used generative AI tools for creating scientific figures or any plagiarism We have carefully reviewed all AI-assisted text revisions to supports their accuracy We take full responsibility for the final content of this work.

References

Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam Chang Chen, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. Trans4map: Revisiting holistic bird’s-eye-view mapping from egocentric images to allocentric semantics with vision transformers Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai et al. Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum.

Improvements for AI systems

The improved system can achieve a 3.81% success rate boost over existing methods in sound-triggered mobile manipulation by augmenting the Navigation Policy Model with an explicit spatial memory that complements acoustic cues with observed geometry. This allows the policy to generate a goal descriptor that provides both target-location and scene-structure priors, guiding navigation to the target region required for subsequent physical interaction.
The system can effectively handle unseen scenes by leveraging a scene-disjoint split protocol, enabling evaluation of both single-event and multi-event tasks in new environments. Furthermore, it can demonstrate superior navigation capability by utilizing the proposed memory augmentation, achieving a navigation success rate of 75.28% compared to baselines like SAVi (15.68%) and MAGNet (71.47%).
The system can robustly manage complex sound events in multi-event settings by employing a sound-triggered Task Planner that predicts an ordered skill chain, specifically handling primary sound source and deducing the secondary source from the mixed audio. This ensures the robot executes actions according to a specified priority, as seen in the qualitative result where it interacts with the primary source before the secondary one.
The navigation policy can be optimized by fusing visual and acoustic inputs for better goal estimation; specifically, vision observations provide rich geometry priors while binaural sound provides directional cues about the source. This fusion results in a goal descriptor that incorporates both target-related and structural priors for the navigation policy model.
The system can be extended to novel environments by incorporating an open-source data generator pipeline, which allows researchers to expand MuSTMM with additional scenes, assets, and task configurations. This ensures the benchmark remains versatile for studying soundtriggered mobile manipulation in unseen scenes.

Abstract

Sound-Triggered Mobile Manipulation (STMM) requires robots to respond to sound events by navigating to the associated objects and performing appropriate physical interactions. Existing STMM evaluations primarily vary object layouts within limited architectural structures, leaving challenges to unseen scenes insufficiently explored. We introduce Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM), a dataset that combines spatial audio rendering and physical interaction across 31 scenes with diverse architectures and object layouts. MuSTMM contains 14.36 million precomputed binaural Room Impulse Responses (RIRs) and a scene-disjoint split for evaluating single-event and multi-event settings in unseen scenes. We further augment the Navigation Policy Model with explicit spatial memory that complements acoustic cues with observed geometry. Specifically, we accumulate pose-aligned depth observations in a shared coordinate frame and project this memory into an egocentric Bird's-Eye View (BEV) occupancy map. Fusing map features with sound context produces a goal descriptor that provides both target-location and scene-structure priors, guiding navigation to the target region required for subsequent physical interaction. This policy achieves a 2.60% success rate boost over existing methods, while component ablations further support the effectiveness of spatial memory. MuSTMM and the navigation policy provide a testbed for the new challenge of studying sound-triggered mobile manipulation in unseen scenes. The code and dataset will be released upon acceptance.