Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation
cs.CL, cs.AI
Submitted: 2026-05-15
Updated: 2026-10-11
Comments: 16 pages, working in progress
Code: https://github.com/mjpost/sacrebleu
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedding or a prompted language name.
Terminology
Abstract
Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedding or a prompted language name. Such a signal, however, is learned separately for each language and transfers nothing between typologically related languages. In this paper, we present TypoCast, a frontend that conditions on the typological profile of the source language rather than on its label alone. A typology-informed hierarchical language encoding represents each language by its morphological type, constituent order and family, so that supervision is shared among related languages. A dual-route mixture-of-experts adapter routes inputs through a shared typological route together with a per-language residual, and a typology-aware prompt conveys the same profile to the decoder. Extensive experiments on CVSS-C demonstrate that TypoCast surpasses state-of-the-art speech-to-speech translation systems on French, German and Spanish to English, averaged over the three directions, and outperforms the identity-conditioned S2ST-Omni on every language of a seven-language inventory, with only 228M trainable parameters. Ablation studies further reveal that the conditioning signal acts during training and is dispensable at inference: a model trained without it is consistently worse, whereas removing it from a trained model changes BLEU by less than 0.2, and expert usage is consistent with the signal shaping how adapter capacity is divided among languages.
Sources
- CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
- Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models
- SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
- S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation
- GPT-4 Technical Report
- Qwen Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
- RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
- Qwen3 Technical Report
- IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
- CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
- FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
- ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
- VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering