Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
cs.LG, cs.AI, cs.CV
Submitted: 2026-10-07
Updated: 2026-10-08
Comments: Project: https://dominik-schnaus.github.io/unpaired-rosetta/, Code: https://github.com/dominik-schnaus/unpaired-rosetta
Code: https://github.com/bioptimus/releases
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data.
Terminology
Abstract
Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.
Sources
- Granite Embedding Models
- Qwen3-VL Technical Report
- Microsoft COCO Captions: Data Collection and Evaluation Server
- mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformations
- Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
- Brain-to-Text Decoding: A Non-invasive Approach via Typing
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry
- Exploiting Similarities among Languages for Machine Translation
- Vision Foundation Models for Computed Tomography
- Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Qwen3 Technical Report
- What Converges in the Platonic Representation Hypothesis? Structure over Geometry
- Jasper and Stella: distillation of SOTA embedding models
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks