MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference".
Jane: The gist The MHR-Router framework proposes a multimodal heterogeneous retrieval-enhanced adaptive routing method for Vision-Language Model inference that accurately dispatches queries to the most suitable VLM expert using local retrieval…
Tom: First, who's behind it and why it matters.
Title and authors: Tom: So we're talking about how MHR-Router improves things compared to what was out there before #pg6. It’s not just one clever trick, it’s a whole system designed for efficiency at scale #pg5.
Jane: Right, so the main improvement is this way it handles the expert selection. Instead of relying on some general routing strategy that might be pretty dumb, MHR-Router uses local retrieval evidence to make a choice based on what’s relevant right now #pg5.
Tom: The authors show that this method beats the best single expert by seven point three six percent and a matched retrieval-free multimodal router by one point two two percent across eight different visual abilities and eight different VLMs #pg3.
Jane: That's solid, but what really stands out is how it handles specific tasks. It shows improvements in five of the eight visual abilities, and it gets its biggest gains in fine-grained recognition and defect anomaly detection #pg3.
Tom: That’s because they are leveraging that local support and risk evidence from past cases, which means the router can adapt its decision based on the specific query you're asking #pg5.
Jane: It means we don't have to pre-label everything with ability or source-task information when we want to use this system at inference time #pg3. That’s a huge simplification for deployment.
Tom: Exactly. The system can select the optimal expert by looking at what has worked and what hasn't, which is much more flexible than sticking to a static profile #pg5.
Jane: It captures competence where a global accuracy score doesn't tell the whole story about how good an expert is for one specific question #pg4.
Tom: And from an engineering standpoint, they kept the feature extractor and the memory banks fixed while only optimizing the scoring parameters theta, which is much better than having to retrain everything every time you want to route a query #pg3.
Jane: That makes it much more practical for running this on a live system, because you aren't incurring massive training costs just to change the routing logic #pg5.
Tom: So what's the caveat? The paper flags that the success of this whole thing really depends on how well those success and failure memories are initially populated #pg5.
Jane: That’s true. If the memory bank is empty or full of bad data, the retrieval process won't be very helpful either, so building that initial memory bank is a big challenge for future work #pg3.
Tom: Exactly. The next step isn't just about getting the routing mechanism right; it’s about making sure those memories are accurate enough to drive that retrieval process effectively #pg5.
The paper's summary: Tom: So we're wrapping up MHR-Router, which is this multimodal heterogeneous retrieval-enhanced adaptive routing method for VLM inference #pg1. It’s essentially a smart system that decides which AI expert to use based on local evidence before the final model runs #pg5.
Jane: Right, so the main implication is that we can build systems that are much more flexible at inference time because they don't need fixed profiles for every single query #pg4.
Tom: The numbers show it outperforms the best single expert by seven point three six percent and a retrieval-free router by one point two two percent on six thousand queries #pg3. That's concrete performance data, which I always like to see #pg5.
Jane: It’s a huge win for anyone building an application because it means the system can choose the right tool based on what we ask, without needing explicit ability labels at inference time #pg3.
Tom: That flexibility is exactly what makes it interesting from an engineering side; we’re focusing on making the decision before the expensive VLM call happens #pg5.
Tom: So, to recap, MHR-Router uses a fixed multimodal representation and expert-specific success and failure memories to guide a scoring mechanism #pg5.
Jane: It’s a framework that fuses local retrieval evidence into an adaptive routing strategy for VLM inference #pg1.
Tom: Yeah, it shows we can get better accuracy by looking at localized evidence instead of relying on a general model profile #pg5.
Lu: I’m still thinking about how to make those success and failure memories really comprehensive, because that's where the real work is going forward #pg5.
Meng: For me, the next hurdle is making sure those memory banks are accurate enough so we don't end up routing a query to the wrong expert because of bad historical data #pg5.
Lalam: I'm confident this approach will be used more often in real-world systems because it actually adapts to the input, not just the system's default settings #pg4.
The paper's improvements: Tom: So we’re wrapping up MHR-Router, which is this multimodal heterogeneous retrieval-enhanced adaptive routing method for VLM inference #pg1. It’s essentially a smart system that decides which AI expert to use based on local evidence before the final model runs #pg5.
Tom: That’s it for MHR-Router. It’s a framework that fuses fixed multimodal features into a query representation and then uses expert-specific success and failure memories to guide the scoring #pg5.
Jane: So for the listener driving or cooking, this paper means that when you ask a complex visual question, the system doesn't just pick one model; it looks at what has worked before for each type of expert and chooses the best fit right there #pg3.
Tom: Exactly. It’s about capturing competence where global expert accuracy is a poor proxy for per-query competence #pg5.
Lu: The paper’s limitation is that the retrieval process relies on having those success and failure memories pre-populated, which means the quality of the routing depends entirely on how well those memories are built initially #pg2.
Jane: So they need a good way to populate that memory bank, which sounds like a challenge for future work in this area #pg3.
Conclusion: Tom: So we’re wrapping up MHR-Router, which is this multimodal heterogeneous retrieval-enhanced adaptive routing method for VLM inference #pg1. It’s essentially a smart system that decides which AI expert to use based on local evidence before the final model runs #pg5.
Jane: Right, so the main implication is that we can build systems that are much more flexible at inference time because they don't need fixed profiles for every single query #pg4.
Tom: The numbers show it outperforms the best single expert by seven point three six percent and a retrieval-free router by one point two two percent on six thousand queries spanning eight different visual abilities and eight different open-source VLMs #pg3. That's concrete performance data, which I always like to see #pg5.
Jane: It’s a huge win for anyone building an application because it means the system can choose the right tool based on what we ask, without needing explicit ability labels at inference time #pg3.
Tom: That flexibility is exactly what makes it interesting from an engineering side; we’re focusing on making the decision before the expensive VLM call happens #pg5.
Lu: I think this paper points to that representation learning can capture competence in a way that's more nuanced than just looking at overall model scores #pg4.
Meng: From a production view, the biggest thing is that we only have to optimize the scoring parameters theta, not the whole feature extractor or memory banks every time we want to route something #pg5. That keeps our deployment costs manageable #pg5.
Lalam: I'm pretty sure this direction is the right one because it directly addresses the need for query-dependent selection in complex multimodal tasks #pg4.
Tom: So, to recap, MHR-Router uses a fixed multimodal representation and expert-specific success and failure memories to guide a scoring mechanism #pg5.
Jane: It’s a framework that fuses local retrieval evidence into an adaptive routing strategy for VLM inference #pg1.
Tom: Yeah, it shows we can get better accuracy by looking at localized evidence instead of relying on a general model profile #pg5.
Lu: I’m still thinking about how to make those success and failure memories really comprehensive, because that's where the real work is going forward #pg5.
Meng: For me, the next hurdle is making sure those memory banks are accurate enough so we don't end up routing a query to the wrong expert because of bad historical data #pg5.
Lalam: I'm confident this approach will be used more often in real-world systems because it actually adapts to the input, not just the system's default settings #pg4.
Tom: That’s it for MHR-Router. It’s a framework that fuses fixed multimodal features into a query representation and then uses expert-specific success and failure memories to guide the scoring #pg5.
Jane: So for the listener driving or cooking, this paper means that when you ask a complex visual question, the system doesn't just pick one model; it looks at what has worked before for each type of expert and chooses the best fit right there #pg3.
Tom: Exactly. It’s about capturing competence where global expert accuracy is a poor proxy for per-query competence #pg5.
Lu: The paper’s limitation is that the retrieval process relies on having those success and failure memories pre-populated, which means the quality of the routing depends entirely on how well those memories are built initially #pg2.
Jane: So they need a good way to populate that memory bank, which sounds like a challenge for future work in this area #pg3.
Tom: Right, so the next step is making sure those memories are accurate and comprehensive enough to drive that retrieval process effectively #pg5.
Meng: I'm curious about how much data we need to feed into those memory banks before this system starts showing its real-world value #pg5.
Jane: That leads us perfectly into the next topic, but for now, that’s what MHR-Router does and why it’s interesting.
Wenjuan Zhu, Zheming Yang, Yunqing Hu, Qi Guo
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
cs.LG
Submitted: 2026-10-09
Updated: 2026-10-09
Project page: https://llava-vl.github.io
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: The gist The MHR-Router framework proposes a multimodal heterogeneous retrieval-enhanced adaptive routing method for Vision-Language Model inference that accurately dispatches queries to the most
Key concepts
- Multimodal Routing Representation
- This is the initial step where an image and a question are combined into a single query representation. It uses a fixed feature extractor to create an embedding by concatenating normalized image and text embeddings, allowing the router to decide which expert to use before any VLM inference happens.
- Heterogeneous Retrieval-Enhanced Adaptive Routing
- Instead of relying on static profiles, this stage uses expert-specific success and failure memories. For each expert, it retrieves local evidence (support from successful cases and risk from failed cases) to determine the most suitable expert for a specific query.
- Success and Failure Memories
- These are nonparametrically stored summaries for every expert. They partition the feature space into regions where an expert has previously succeeded or failed. This memory allows the router to find local support and risk evidence specific to the current input, enabling fine-grained routing decisions.
- Retrieval Statistics
- These are six calculated statistics derived from the retrieved support and risk evidence for each expert. These statistics summarize mean similarity, maximum similarity, and their differences. This structured data allows the system to efficiently evaluate and rank experts based on local performance evidence.
Summary
The gist The MHR-Router framework proposes a multimodal heterogeneous retrieval-enhanced adaptive routing method for Vision-Language Model inference that accurately dispatches queries to the most suitable VLM expert using local retrieval evidence.
Multimodal Routing Representation
The first stage of MHR-Router constructs a query representation that supports expert ranking without invoking any candidate VLM, motivated by the constraint that routing must be decided before inference. This design preserves two complementary sources of evidence: the visual content of the image and the intent expressed in the question.
The encoding of each query involves a frozen multimodal feature extractor fψ that produces the representation z i, where z i ∈ R d.
The frozen representation zi is constructed by concatenating an l2-normalized image embedding and an l2-normalized text embedding, as shown in the SigLIP or CLIP configuration. This setup is fixed for each experiment, ensuring that the parameters ψ remain frozen throughout scorer training.
Heterogeneous Retrieval-Enhanced Adaptive Routing
The second stage injects expert heterogeneity into routing through retrieval, where the query representation zi is modality-complete but expert-agnostic. To decide which expert is most likely to succeed, the router consults expert-specific evidence by storing it nonparametrically as success and failure memories.
For each expert m, ground-truth correctness partitions frozen training representations into success and failure memories: Pm =
Pm =
Pm =
Pm summarizes the region of the representation space in which expert em has previously succeeded, while Nm summarizes the region in which it has previously failed.
Independent cosine top-k retrieval forms R+i,m ⊂ Pm and R-i,m ⊂ Nm. These retrieve local support and risk evidence. The statistics calculated include mean similarity µ±i,m and maximum similarity ρ±i,m.
The six retrieval statistics are packed as ri,m = [µ+i,m; µ-i,m; Δµ+i,m; ρ+i,m; ρ-i,m; Δρ+i,m] ∈ R 6. This optimization allows only θ to be optimized, keeping the feature extractor and memory banks fixed.
Experimental Results
Experiments on 6,400 queries spanning eight visual abilities and eight open-source VLMs show that MHR-Router achieves 71.58±0.02% routing accuracy. This exceeds the development-selected best single expert by 7.36% and a matched retrieval-free multimodal router by 1.22%.
Table 1 compares fixed and group-level references with learned query-level routers, showing that MHR-Router achieves 71.58±0.02%, exceeding Best Single by 7.36% and the Multimodal Query Router by 1.22%.
MHR-Router improves five of eight abilities, matching counting and achieving its largest gains in fine-grained recognition (+21.60 pp) and defect anomaly detection (+8.54 pp). By leveraging complementary evidence from successful and failed cases, MHR-Router enables input-dependent expert selection without requiring ability or source-task labels at inference time. Experiments across eight abilities and eight experts demonstrate that MHR-Router outperforms the best single expert selected on the development set by 7.36% and the matched retrieval-free Multimodal Query Router by 1.
Improvements for AI systems
-
No reliance on static model profiles by utilizing
frozen multimodal features
to construct acomprehensive query representation.
This allows the router to operate without invoking candidate VLMs beforehand, which is crucial for real-time inference decisions. -
The system can perform heterogeneous modality routing by using
expert-specific success and failure memories as local evidence.
This enables the router to select an expert based onlocal retrieval evidence,
capturing fine-grained competence beyond global profiles. -
The improved system can achieve superior accuracy over traditional baselines, specifically exceeding the development-selected best single expert by
7.36%
and a matched retrieval-free multimodal router by1.22%.
-
By employing the shared scorer to adaptively select the optimal expert, the system can capture competence where
global expert accuracy is a poor proxy for per-query competence.
Sources
- Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
- MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
- An Effective Router for Vision-Language Model Selection
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Qwen2.5-VL Technical Report
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks