MoSAT: Human Motion Generation from Spatial Audio and Textual Description
cs.GR, cs.CV, cs.RO, cs.SD
Submitted: 2026-09-20
Updated: 2026-09-20
License: http://creativecommons.org/licenses/by/4.0/
The gist: Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and
Terminology
Abstract
Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.
Sources
- Diffusion Posterior Sampling for General Noisy Inverse Problems
- LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environment
- Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking
- Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing
- Decoupled Weight Decay Regularization
- Representation Learning with Contrastive Predictive Coding
- Classifier-Free Diffusion Guidance
- Motion Flow Matching for Human Motion Synthesis and Editing
- InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling
- MotionHiFlow: Text-to-motion via hierarchical flow matching
- Flow Matching for Generative Modeling
- DeepSeek-V3 Technical Report
- Motion Anything: Any to Motion Generation
- Multisample Flow Matching: Straightening Flows with Minibatch Couplings
- Human Motion Diffusion Model
- Improving and generalizing flow-based generative models with minibatch optimal transport
Related papers
- SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control
- CADReasoner: Iterative Program Editing for CAD Reverse Engineering
- QuadLink: Autoregressive Quad-Dominant Mesh Generation via Point-Relation Learning
- DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches
- MotionPersona: Real-Time Locomotion Control across Personas, Bodies, and Styles
- GaussianBench: Physics-Fidelity Evaluation for Gaussian Scene Representations