SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation

arXiv:2610.21621 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation".

Dev: The gist The SPEAR framework transforms Vision-Language Models into efficient preference oracles via two synergistic mechanisms to enable practical VLM-guided robotic manipulation without manual reward engineering.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Now let's look at the title and who wrote this paper, "SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation". It’s by Zewen Wu, Ruizhe Liu, Pei Zhou, Feng Chen, and Yanchao Yang.

Dev: They are all researchers in the field. This framework essentially says we can transform these Vision-Language Models into much more efficient tools for preference learning without needing to manually engineer those complicated reward functions.

Taro: So it’s about building a system that lets robots learn from human demonstrations using these powerful visual models, but making sure the process is fast and doesn't rely on perfect rewards upfront.

Rosa: Precisely. The implication here is that we might be able to deploy VLM-guided manipulation in real-world scenarios much faster than before because we’re cutting down on the number of useless queries we have to run.

Dev: It means moving away from methods where you just throw a lot of random comparisons at the VLM and hope for the best, towards something more targeted and adaptive.

Taro: I wonder how much better this is when we consider what happens when the real world doesn't behave exactly as expected during those initial learning steps.

The paper's summary: Rosa: So, to summarize SPEAR, it introduces two synergistic mechanisms. First, distinguishability-aware query selection. This mechanism predicts how likely the VLM is to give a good answer before we even ask the question.

Dev: It uses multi-scale visual features and an online-adapted model to weigh the information gain against that predicted response likelihood. The goal is to only pick comparisons where the VLM can actually tell them apart, which cuts down on those abstentions we talked about earlier.

Taro: So it’s smart filtering based on what the visual data shows, rather than just asking everything and hoping for the best?

Rosa: Right. And second is the adaptive reward transition strategy. This handles that bootstrapping problem where our initial random policies make aimless motions, requiring us to use task-related rewards just to get them moving in a meaningful way.

Dev: Before we switch entirely to the preference-based learning, SPEAR uses VLM-synthesized warm-up rewards. This biases the early exploration toward behaviors that are actually relevant to the task, like getting closer or aligning something with its target.

Taro: So it uses the AI itself to guide us through the first phase of exploration before we rely on our learned preference model for refinement?

The paper's improvements: Rosa: Let’s talk about the actual performance gains they reported in "SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation". They show consistent superiority across tasks of varying difficulty, achieving an overall success rate of eighty-eight point nine percent at final evaluation <ref:2610.21621#pg1>.

Dev: That eighty-eight point nine percent is impressive because they outperformed the RL-VLM-F baseline by a margin of twenty-three point three percentage points <ref:2610.21621#pg1>. It also significantly reduced the wall-clock training time from about thirteen hours down to about nine hours when they hit a comparable success rate on contact-rich tasks.

Taro: So it’s not just that it works better, but the efficiency gains are real too, with query analysis showing SPEAR used nine hundred sixty total queries with a twenty-nine point six percent abstention rate, compared to one thousand six hundred forty queries and a fifty-five point five percent abstention rate for the baseline.

Rosa: And they confirmed that increasing visual distinguishability actually helps reliability; the average preference accuracy reached about ninety-two point three percent under conservative prompting, which shows that making the comparisons clearer leads to better feedback for the reward model <ref:2610.21621#pg1>.

Dev: The ablation studies also showed that combining distinguishability and information gain in their query acquisition mechanism gave them the best results for getting queries, achieving an eighty point seven percent final success rate in that part of their system.

Conclusion: Rosa: So to wrap up, SPEAR shows that by focusing on query efficiency through distinguishability-aware selection and using adaptive rewards to guide exploration early on, we can make VLM-guided preference learning practical without needing manual reward engineering.

Dev: The big implication is that this framework validates using VLMs as real preference oracles for scalable robot learning even when facing realistic computational constraints, specifically by managing query cost and bootstrapping challenges effectively.

Taro: From my side, what stands out is how they addressed the reliability of the VLM oracle itself by using a decomposed two-stage protocol and temperature scaling on the distinguishability predictor to keep things stable under distribution shift.

Rosa: It really does show that addressing both query efficiency and exploration bootstrapping allows for practical VLM-guided preference learning across diverse manipulation tasks, validated on a real Mobile ALOHA robot.

Dev: That’s the summary of SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation. It lays out a solid path forward for how we use these models in actual hardware and online learning scenarios.

Zewen Wu, Ruizhe Liu, Pei Zhou, Feng Chen, Yanchao Yang

HKU Musketeers Foundation Institute of Data Science · Department of Electrical and Electronic Engineering, The University of Hong Kong

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 8 pages, 4 figures. Accepted at IROS 2026. Project page: https://zevinwu.github.io/spear-project

Project page: https://zevinwu.github.io/spear-project

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The SPEAR framework transforms Vision-Language Models into efficient preference oracles via two synergistic mechanisms to enable practical VLM-guided robotic manipulation without manual

Key concepts

Distinguishability-Aware Query Selection
This mechanism reduces the number of queries by predicting whether a comparison is visually distinguishable before asking the VLM. It uses multi-scale visual features, like absolute differences and cosine similarity, combined with cross-attention to create a signature of visual difference across different scales. This ensures queries only target comparisons that are likely to yield an informative answer.
Adaptive Reward Transition
This strategy solves the exploration problem by using two phases. First, it uses VLM-synthesized warm-up rewards to guide the robot toward coarse task progress like alignment. Once exploration is complete, it switches to a preference-based reward for fine refinement, ensuring meaningful comparisons are made during training.
Preference Oracle
A preference oracle is a system that uses a model (like a VLM) to automatically judge which of two outcomes is better. SPEAR turns VLMs into this oracle by intelligently selecting the best comparison pairs and using adaptive rewards to guide the robot's learning process efficiently.
Mobile ALOHA Manipulation Tasks
These are real-world robotic manipulation tasks used for validation. SPEAR was tested on a physical Mobile ALOHA robot, allowing it to learn three manipulation tasks fully online through end-to-end interaction with the hardware, proving its practical applicability.

Summary

The gist The SPEAR framework transforms Vision-Language Models into efficient preference oracles via two synergistic mechanisms to enable practical VLM-guided robotic manipulation without manual reward engineering.

How it works

SPEAR addresses two fundamental inefficiencies in VLM-guided Preference-based Reinforcement Learning (PbRL): expensive queries that frequently yield abstentions on visually indistinguishable comparisons, and the bootstrapping paradox where initial random policies produce aimless motions requiring task-related rewards for meaningful comparison. The framework tackles these issues through two synergistic mechanisms:

  1. Distinguishability-Aware Query Selection: This mechanism aims to reduce abstention rates by targeting only informative and distinguishable comparisons. It achieves this by predicting VLM response likelihood before querying using multi-scale visual features and a calibrated distinguishability prediction model. The feature extraction involves global embeddings and relational features, such as absolute differences, element-wise products, and cosine similarity. These features are combined with local spatial patterns captured via cross-attention to yield a comprehensive signature of visual distinguishability across semantic and spatial scales. This ensures that queries target only answerable and informative comparisons, balancing resolving reward-model uncertainty with avoiding pairs the VLM cannot distinguish.

  2. Adaptive Reward Transition: To resolve the exploration bootstrapping challenge, SPEAR introduces an adaptive reward transition strategy. This involves a two-phase procedure: first, using VLM-synthesized warmup rewards to bias early exploration toward task-relevant behaviors. This warm-up reward is designed to encourage coarse task-related progress, such as approach or alignment. Once exploration saturates according to a generic plateau criterion, the policy transitions to the learned preference-based reward rθ for refinement. Relay exploration is also implemented to maintain distributional coverage after this transition. The framework was validated on a real Mobile ALOHA robot, learning three tasks fully online with end-to-end interaction.

Key Findings and Contributions

The study highlights several critical findings regarding the performance and efficiency of SPEAR:

"SPEAR demonstrates consistent superiority across tasks of varying difficulty, achieving an overall success rate of 88.2% at mid-training and 88.9% at final evaluation – outperforming the RL-VLM-F baseline by a margin of 23.3%"

The query efficiency analysis shows substantial savings in wasted comparisons; for instance, SPEAR used 960 total queries with a 29.6% abstention rate, compared to RL-VLM-F's 1640 queries with a 55.5% abstention rate. Furthermore, the wall-clock training time was reduced from 13.6 hours to 9.1 hours when reaching a comparable success rate.

The reliability diagnostics confirm that increasing visual distinguishability reduces abstention and improves accuracy, with the average preference accuracy reaching 92.3% under conservative prompting. The ablation studies confirm that the integrated weighting of distinguishability and information gain yields the best performance for query acquisition mechanisms, reaching 80.7% final success.

The paper concludes that SPEAR establishes VLMs as practical preference oracles for scalable robot learning without manual reward engineering. The contributions are threefold: proposing a distinguishability-aware query selection mechanism, introducing an adaptive reward transition strategy, and demonstrating substantial improvements across diverse manipulation tasks and on a real robot. The framework validates that addressing query efficiency and exploration bootstrapping enables practical VLM-guided preference learning under realistic computational constraints.

How it works

The framework operates by iterating through three steps: Exploration, Query Selection and Preference Prediction, and Reward Learning. The policy is trained with the surrogate reward model rθ while collecting candidate trajectories. Then, a sampling mechanism selects pairs for preference elicitation to form Dpref. Finally, the reward model rθ is updated via Bradley–Terry modeling using the collected preference dataset Dpref.

How it works

The core of query selection involves predicting VLM response likelihood using features ϕ(ξ). This prediction is modeled by a lightweight MLP Dψ that must adapt online as the policy explores new regions.

Improvements for AI systems

  1. What is improved: Query selection efficiency via Distinguishability-Aware Query Selection. This mechanism predict[s] VLM response likelihood before querying by combining global semantic embeddings with local spatial patterns to weight information gain with predicted distinguishability, ensuring queries target only answerable and informative comparisons.

  2. What the improved AI system can do: The system reduces wasted inference costs by avoiding queries on visually similar or indistinguishable trajectories, leading to substantially fewer wasted oracle queries and faster training times, as shown by reducing total query count from 1640 to 960 in specific comparisons.

  3. What is improved: Exploration bootstrapping via Adaptive Reward Transition. This strategy leverages VLM-synthesized rewards to jumpstart exploration before transitioning to preference-based refinement, allowing the policy to discover task-relevant behaviors early on when initial random policies produce aimless motions.

  4. What the improved AI system can do: The system overcomes the bootstrapping paradox by using a VLM-synthesized warm-up reward that encourages coarse task-related progress (e.g., approach, alignment, and initial contact) to guide early exploration toward semantic regions before switching to the learned preference reward model.

  5. What is improved: Robustness against VLM oracle fallibility through decomposed querying and reliability diagnostics. The paper adopts a decomposed twostage protocol that analyzes sub-trajectories independently and employs temperature scaling with parameter T on the distinguishability predictor to maintain reliable probabilities under distribution shift, leading to an average preference accuracy of 92.3% across VLM backbones.

  6. What the improved AI system can do: The resulting policy is more reliable because it explicitly mitigates positional bias and cross-image attribution errors, ensuring that the feedback used for reward learning is more accurate, even when using fallible VLM oracles.

Sources

Related papers