MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
cs.CL, cs.AI
Submitted: 2026-10-08
Updated: 2026-10-08
Code: https://github.com/facebook/meta-encoder-eval
License: http://creativecommons.org/licenses/by/4.0/
The gist: System One models output constrained decisions and probability distributions rather than free-form text generation.
Terminology
Abstract
System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.
Sources
- Douyin Multimodal Embedding Model Technical Report
- TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens
- Reason to Contrast: A Cascaded Multimodal Retrieval Framework
- Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
- MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
- E5-V: Universal Embeddings with Multimodal Large Language Models
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Passage Re-ranking with BERT
- Representation Learning with Contrastive Predictive Coding
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering