MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes

summary

The gist

The gist The Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM) dataset introduces 31 scenes combining spatial audio rendering and physical interaction across diverse architectures and object

In short

MuSTMM introduces a dataset with 31 diverse scenes combining spatial audio and physical interaction for sound-triggered mobile manipulation. It augments navigation policies with explicit spatial memory, fusing depth observations into a Bird's-Eye View map to guide robots to targets based on both sound and scene structure. This method boosts success rates by leveraging visual geometry alongside acoustic cues in unseen environments.

Key concepts

Sound-Triggered Mobile Manipulation (STMM): A task where a robot moves and interacts with objects only when it detects specific sounds. The goal is to navigate to the sound source and perform the correct physical action, making sound a primary way for the robot to perceive its surroundings and interact with the world.
Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM): A new dataset featuring 31 scenes with varied architectures and object layouts. It combines high-quality spatial audio rendering (Room Impulse Responses) with physical interaction tasks, providing a comprehensive benchmark for testing sound-triggered manipulation in complex, unseen environments.
Explicit Spatial Memory: An augmentation added to the robot's navigation policy that stores observed depth data from its movements. This data is projected into an egocentric Bird's-Eye View (BEV) map. This memory helps the robot understand the physical layout of a scene, complementing sound cues by providing geometric context for where it needs to go.
Goal Descriptor Fusion: A process where features from the sound context and the spatial memory map are combined. This fusion creates a 'goal descriptor' that gives the robot two crucial pieces of information: exactly where the target is located and what structures exist in that area, guiding accurate navigation to perform physical tasks.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes

This episode discusses

MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes · Paper Radio

The paper

MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes · Read on arXiv

Hao Ju, Feng Yan, Zhihao Zhong, Heng Yang, Shaofei Huang, Zhedong Zheng

University of Macau · Enacta-AI

Sound-Triggered Mobile Manipulation (STMM) requires robots to respond to sound events by navigating to the associated objects and performing appropriate physical interactions. Existing STMM evaluations primarily vary object layouts within limited architectural structures, leaving challenges to unseen scenes insufficiently explored. We introduce Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM), a dataset that combines spatial audio rendering and physical interaction across 31 scenes with diverse architectures and object layouts. MuSTMM contains 14.36 million precomputed binaural Room Impulse Responses (RIRs) and a scene-disjoint split for evaluating single-event and multi-event settings in unseen scenes. We further augment the Navigation Policy Model with explicit spatial memory that complements acoustic cues with observed geometry. Specifically, we accumulate pose-aligned depth observations in a shared coordinate frame and project this memory into an egocentric Bird's-Eye View (BEV) occupancy map. Fusing map features with sound context produces a goal descriptor that provides both target-location and scene-structure priors, guiding navigation to the target region required for subsequent physical interaction. This policy achieves a 2.60% success rate boost over existing methods, while component ablations further support the effectiveness of spatial memory. MuSTMM and the navigation policy provide a testbed for the new challenge of studying sound-triggered mobile manipulation in unseen scenes. The code and dataset will be released upon acceptance.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes".

Dev: The gist The Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM) dataset introduces 31 scenes combining spatial audio rendering and physical interaction across diverse architectures and object layouts,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we’ve seen how MuSTMM sets up this challenge by combining audio rendering and physical interaction across thirty-one scenes with varied layouts <ref:2610.21429#pg1,audio rendering and physical interaction across 31 scenes with>. The core idea is that existing benchmarks only test layout variation in limited structures, so they needed something broader. Dev They claim MuSTMM addresses this by covering both structure variation—different room geometries—and layout variation—where the sounds and objects are placed relative to each other.

Taro: It matters because it pushes the limits of what we know about sound-triggered manipulation outside of controlled lab settings where you can see everything. Rosa Exactly. It brings together audio-visual navigation, like SoundSpaces or SAVi, with mobile manipulation tasks that only test physical interaction in a limited structure variation without the sound rendering component.

Dev: The paper introduces this by saying that STMM requires robots to respond to sound events by navigating to objects and doing physical interactions, but current evaluations only vary layouts within limited structures. Taro So the contribution is building a benchmark that covers both structural diversity and layout variation for unseen scenes.

Rosa: They have thirty point 11k episodes spanning those thirty-one scenes, which means they have a lot of varied scenarios to test these capabilities on. Dev And they've included this scene-disjoint split so you can evaluate single-event and multi-event tasks in unseen scenes using the held-out set.

Taro: It also mentions that they used SoundSpaces to precompute fourteen point thirty-six million binaural Room Impulse Responses, which is a lot of data for spatial audio rendering. Rosa That RIR data is what allows the system to accurately model how sound travels in those specific room geometries.

Dev: And they augment the navigation policy model with explicit spatial memory derived from depth observations to complement these acoustic cues with observed geometry. Taro So they are using what the robot sees about its immediate surroundings—the depth—to create a map that helps it understand where it is in relation to the sound source.

Rosa: This memory is projected into an egocentric Bird's-Eye View occupancy map, and fusing that with sound context creates a goal descriptor with target-related and structural priors. Dev That descriptor then guides navigation to the region needed for physical interaction, which is the whole point of this augmentation.

Taro: It’s about using what you see and what you hear together to make a better navigation decision in a scenario where nothing has been explicitly trained for that exact combination of sound and structure.

Conclusion: Rosa: So looking at the MuSTMM paper, it’s clear that they’ve put together a comprehensive setup for studying sound-triggered mobile manipulation. They used thirty-one scenes with different architectures and layouts to create a benchmark <ref:2610.21429#pg1>. Dev The authors are showing how combining spatial audio rendering with physical interaction can be tested in these diverse environments using this new dataset.

Taro: What this means practically is that we start moving past testing robots only in perfectly controlled lab settings where the environment is completely known. Rosa Exactly. It opens up the door to studying robots navigating and interacting with things when they encounter unexpected room geometries or object placements, even when they don't have prior training for that specific sound trigger.

Dev: The focus on unseen scenes using that scene-disjoint split is key because it tests the system’s ability to handle novel situations without having a perfect map of every possible scenario. Taro It suggests that adding explicit spatial memory, where the robot accumulates depth data into an occupancy map and fuses it with sound context, is a necessary step for better navigation in these complex scenarios.

Rosa: Even though they found improvements in navigation success rates—like that two point sixty percent boost—they also highlighted that navigation itself remains a major bottleneck when dealing with multi-event tasks. Dev So, the implication is that while we can get better at navigating to the target, figuring out what physical interaction to perform next is still a hard problem in unseen scenes.

Taro: It points toward future research focusing on how agents recover gracefully when navigation fails or when they have to perform a sequence of actions based on multiple sounds. Rosa So, MuSTMM provides this new testbed for studying STMM in unseen scenes, even if it shows that we still need to work on making the execution part smoother.

More episodes

← Home