MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

arXiv:2610.22240 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents".

Jane: The gist: MMSearch-OPD introduces controllable data refinement and joint reinforcement learning with on-policy distillation to enable long-horizon multimodal deep search agents to handle complex, evidence-intensive investigations.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re looking at this paper today, "MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents." Basically, the problem they're tackling is that multimodal deep search is moving away from just answering simple questions to these long investigations that keep gathering evidence. But training agents for that keeps getting stuck because we don't have enough really hard training questions or good feedback on how the agent is actually processing things.

Jane: That makes sense, and it sounds like they are proposing MMSearch-OPD as a solution. What do you think the core idea is here? What's their big claim about how this system works?

Tom: The main thing they introduce is a combination of controllable data refinement with joint reinforcement learning and on-policy distillation. They call it the Refine pipeline, which takes existing simple multimodal questions and their reasoning chains and applies parameterized transformations to control things like search depth and what kind of cross-modal evidence is needed, all while reusing the original image question pairs.

Lu: What I find interesting about that Refine pipeline is how it's structured. They have four steps: Expand, Blur, Visualize, and Question Synthesis. It sounds like they are systematically making the search harder based on some knobs they control.

Meng: Expanding means increasing the chain depth by verifying those transitions against web evidence and adding exactly L evidence-supported hops, which is pretty concrete for an engineer to understand how to control complexity. But I wonder if that doesn't just make things brittle if you set L too high, right?

Jane: That's a valid concern, Meng. The Blur step sounds like it involves selecting M internal nodes and attaching W distinct, source-grounded facts to each node to resolve those nodes using indirect clues from those facts. It’s trying to force the agent to use more subtle connections instead of just direct links.

Lalam: From my perspective as a language model, I see that the Refine pipeline is essentially rewriting the search task itself based on specific requirements for depth and evidence before the agent even starts searching. It takes a seed question and uses those transformations to generate a refined question that demands more complex reasoning, which is what they call Question synthesis in step four.

Paper summary: Tom: Exactly. And it’s not just about making questions harder; they're doing joint reinforcement learning with on-policy distillation for training. They use answer rewards and lightweight behavior penalties to guide the RL part, and then they modulate a reward gap to adjust token-level teacher guidance on those imperfect student trajectories.

Lu: That teacher guidance modulation sounds like a smart way to handle the gap between what the perfect model knows and what their current student model is actually doing when it’s searching long sequences. They use an exponentially decaying distillation coefficient to reduce that dependence as training goes on, which seems like a pragmatic way to get better without getting stuck purely on the teacher's mistakes.

Jane: So, they aren't just teaching the agent by giving it right answers; they are using this feedback loop where the reward gap specifically shapes how much you rely on the teacher when you’re learning those long search trajectories. It’s a sophisticated way to handle training for tasks that take a long time to solve.

Meng: I'm thinking about deployment here, though. If we have these complex refinement pipelines, what does the cost look like? How much do we need to preprocess all this data for every possible combination of L, M, W, N, P and Q settings? That sounds like a lot of upfront engineering work before you even get to the training phase.

Tom: The authors address that by focusing on reusing existing image question pairs as seed assets for the Refine pipeline. They use configurable difficulty knobs like L and M to control the chain depth and disambiguation requirements, and they emphasize quality control checks on chain consistency and evidence provenance to make sure those added requirements are meaningful.

Lu: Reusing existing data is key, because they start with seeds collected from datasets like LiveVQA and FVQA, retaining questions that the data-generation model already answers correctly without tools. They then use those successful trajectories as a starting point for their refinement process.

Paper summary: Jane: That means they aren't starting from scratch with completely random hard questions; they’re building on what already exists, but applying these specific transformation rules to make the existing data harder in a structured way. It shifts the focus from just collecting more data to intelligently making the data demand more reasoning steps.

Tom: The results show clear gains from both parts of their approach. They report that MMSearch-OPD-9B reaches an average accuracy of sixty point two, which is an improvement of fifteen point three points over the corresponding Qwen3 point 5-9B agentic workflow, and they also see further improvements when joint training happens after the SFT stage, gaining another two point three point average gain on every benchmark after that.

Meng: A fifteen point jump is substantial, but I need to know if that comes from the refinement pipeline alone or if it’s truly synergistic with the RL and distillation setup. Because running this kind of complex training pipeline on real-world traffic sounds like it would be computationally intensive at scale.

Lalam: Based on what I see in my literature view, the combination of controllable data refinement and joint RL distillation seems to be the mechanism that allows for these sustained evidence acquisition capabilities in long-horizon tasks, which is what they set out to achieve with MMSearch-OPD.

Tom: So, to wrap up this part of the paper and MMSearch-OPD: it shows that by using controllable data refinement and joint reinforcement learning with on-policy distillation, we can enable long-horizon multimodal deep search agents to handle complex, evidence-intensive investigations. It really demonstrates the value of upgrading existing multimodal data to require sustained evidence acquisition and supports joint outcome optimization and selective teacher guidance for learning long-horizon search and reasoning over text and images.

Jane: It sounds like the title says it well: Easy Made Hard, Hard Made Learnable. They’re showing how you can take standard multimodal data and apply these parameterized transformations to systematically increase the search demands in a way that actually helps the AI learn those long-horizon search patterns. That’s a really practical application for anyone building agents today.

Conclusion: Tom: So, we've been looking at this paper called MMSearch-OPD, and now we’re coming to the conclusion about what it actually means for these long-horizon search agents.

Jane: It boils down to taking complex, evidence-heavy investigations—those multi-step searches—and making them trainable by using controllable data refinement and a joint training method.

Tom: The title itself says "Easy Made Hard, Hard Made Learnable." That’s the core idea for me. It means they took existing multimodal data and systematically made it demand more sustained evidence acquisition, which is what those long investigations actually require.

Lu: From my side, it suggests that we don't need to invent entirely new ways to generate impossibly hard questions from scratch. Instead, we can use parameterized transformations—like the Expand or Blur steps they describe—to take simple questions and increase their reasoning demands in a structured way.

Meng: But how does that translate to deployment? If you have these settings controlling depth and evidence needs, it sounds like a lot of pre-processing work just to set up the search space for training. What’s the real cost there?

Jane: The authors address that by reusing existing image question pairs as seed assets. They aren't starting from zero; they’re using what already exists, but applying these rules to make it harder in a controlled manner. That reuse is important for making this practical for production systems.

Lalam: I’m confident in the direction here because the mechanism—joint reinforcement learning with on-policy distillation—is specifically tuned to handle those imperfect, long search trajectories they are trying to train. It’s focused on optimizing that specific kind of complex behavior.

Tom: The results show gains from both stages, SFT and then joint training after that. They hit an average accuracy of sixty point two on the MMSearch-OPD-9B model, which is a fifteen point improvement over the Qwen3 point five agentic workflow they compared it to before those final joint optimizations.

Jane: So, for someone just listening, what this means is that we are getting better long-horizon search because we are making sure the AI has to keep gathering proof along the way, not just guessing the final answer.

Tom: Exactly. The implication is that upgrading existing multimodal data to require sustained evidence acquisition actually helps the AI learn those complex search patterns much better than just feeding it more raw data.

Jane: Next time, we'll look at how they actually set up that training loop with reward gaps and token guidance—that’s where the real learning happens.

Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei

Peking University · ×HKUST(GZ) · ×Tsinghua University · ☘The University of Hong Kong

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: MMSearch-OPD introduces controllable data refinement and joint reinforcement learning with on-policy distillation to enable long-horizon multimodal deep search agents to handle complex,

Key concepts

Refine Pipeline: This pipeline modifies existing simple multimodal questions by applying configurable operators like Expand, Blur, and Visualize. It systematically increases search difficulty by forcing the agent to verify transitions against external evidence or resolve internal nodes using specific facts, effectively making existing data harder for the model to solve.
Joint RL–OPD Training: This training method combines reinforcement learning (RL) with on-policy distillation (OPD). The RL part uses answer rewards and penalties to guide the agent's search behavior. OPD uses a teacher model's guidance, which is adjusted by a reward gap mechanism, to help the student model learn from imperfect long-horizon search paths.
Reward Gap Modulation: The reward gap ($\omega_i$) adjusts how strongly token-level guidance from the teacher is applied. It is calculated based on the difference between a target reward and an achieved reward. This mechanism penalizes inefficient tool usage or repetitive searching, ensuring the teacher's guidance is most impactful when the student model struggles with complex search steps.

Summary

The gist: MMSearch-OPD introduces controllable data refinement and joint reinforcement learning with on-policy distillation to enable long-horizon multimodal deep search agents to handle complex, evidence-intensive investigations.

How it works

MMSearch-OPD combines a Refine pipeline with joint RL–OPD training to improve long-horizon multimodal search capabilities through controllable data refinement and joint reinforcement learning (RL) and on-policy distillation (OPD) The Refine pipeline reuses existing simple multimodal questions and their candidate reasoning chains, applying configurable operators to control the intended evidence requirements while reusing seed assets

The Refine pipeline consists of several steps that modify the initial question based on desired search depth and cross-modal evidence needs

Expand (L) increases chain depth by verifying the extracted transitions against web evidence and appending exactly L evidence-supported hops
Blur (M, W) selects M internal nodes and attaches W distinct, source-grounded facts to each node to resolve nodes via indirect clues from facts
Visualize (N, P, Q) selects N nodes from the terminal node backward and extracts retrieved images and visual facts for each node
Question synthesis rewrites the question by replacing entity names with indirect clues based on Bi and Vi to require textual clues and image inspection

Agentic Training

The training process involves supervised initialization followed by joint RL–OPD training to optimize the student model on imperfect long-horizon search trajectories

Supervised initialization partitions the curated training questions into an SFT subset and a harder RL subset, where a larger model is fine-tuned on successful trajectories from both subsets and frozen as teacher πT
Joint RL and OPD for long-horizon search combines answer rewards and behavior penalties with reward-gap-weighted token guidance on imperfect student trajectories
The shaped reward gap adjusts guidance strength through ωi = [1 − Ri]+, where Ri = ci + Hi, penalizing inefficient tool usage and repetitive search behavior
Token feedback is calculated as ∆i,t = clip log πT(ui,t hi,t) - copd, copd, AOPD i,t = gi sg(ωi∆i,t), with exponential decay reducing teacher dependence as training proceeds

Data Curation and Difficulty Control

The paper addresses the challenge of constructing hard questions by reusing existing image–question pairs and applying transformations to increase search demands

Seed collection involves taking full training splits of LiveVQA and FVQA and retaining questions that the data-generation model answers correctly without tools, which provides candidate reasoning chains
Difficulty stratification is controlled via configurable difficulty knobs (L, M, W, N, P, Q) which control chain depth (Expand), disambiguation requirements (Blur), and visual evidence requirements (Visualize)
Quality control involves checking chain consistency and evidence provenance to ensure that the added requirements in Equation 8 are non-trivial

Results and Evaluation

Experiments show significant gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities

Overall performance shows that MMSearch-OPD-9B reaches 60.2 average accuracy, improving over the corresponding Qwen3.5-9B agentic workflow by 15.3 points
MMSearch-OPD improves performance across the evaluated search settings, gaining 12.8 points on average over the same 9B agentic workflow on MMSearch, LiveVQA, and BrowseComp-VL
Joint training beyond SFT further raises accuracy to 60.2 with improvements on every benchmark, with a 2.3-point average gain supporting joint online optimization after SFT

The introduction of MMSearch-OPD demonstrates the value of upgrading existing multimodal data to require sustained evidence acquisition and supports joint outcome optimization and selective teacher guidance for learning long-horizon search and reasoning over text and images.

Improvements for AI systems

textbfData Refinement via Refine Pipeline for Harder Tasks: Refine reuses existing multimodal questions and their candidate reasoning chains (Page 1). This pipeline applies Expand, Blur, and Visualize to control chain depth, disambiguation, and visual evidence requirements (Page 1), allowing the system to generate significantly harder long-horizon tasks by controlling evidence requirements through parameters like L (Expand), M/W (Blur), N/P/Q (Visualize) while reusing existing image–question pairs (Page 1).
textbfJoint Training via Reward-Conditioned Token Guidance: The system employs joint RL and OPD training that combines answer rewards and behavior penalties with reward-gap-weighted token guidance on imperfect student trajectories (Page 3). This mechanism uses the formula token feedback is ∆i,t = clip log πT(ui,t hi,t) - copd, copd (Equation 10) to guide the student model based on whether a trajectory reached its target reward (Within this subset [of trajectories], the shaped reward determines guidance strength through ωi = [1 − Ri] +).
textbfProgressive Teacher Influence Reduction: The training schedule uses an exponentially decaying distillation coefficient reduces teacher dependence as training proceeds (Page 1). This means the imitation pressure on the student model is gradually reduced over time, allowing the agent to move beyond the SFT teacher’s preferences as it learns from outcome-based optimization progresses (Page 3).
textbfControllable Search Difficulty: The system implements Difficulty Stratification Control knobs (Page 3) where refinement directly contributes to this by calculating a metric like DRefined, which sums requirements from Expand, Blur, and Visualize (Equation 8). This allows for explicit control over the designated chain’s evidence burden (Page 4), ensuring the agent is trained on tasks demanding sustained evidence acquisition.
textbfEconomical Search Behavior: The joint training optimizes for efficiency by incorporating behavior penalties such that Hi ≤ 0 penalizes inefficient tool usage and repetitive search behavior (Equation 9). This guides the agent to adopt increasingly economical search trajectories, as evidenced by the observation that later rollouts use fewer interactions and generated tokens without a corresponding decline in average answer quality (Page 8).

Abstract

Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.