In-Context Robot Learning with VLM Agents

arXiv:2609.19138 · cs.CV, cs.RO · Submitted 2026-09-16 · Read on arXiv

cs.CV, cs.RO

Submitted: 2026-09-16

Updated: 2026-09-16

Comments: Project Page: https://cheng-haha.github.io/GPT-Policy GitHub Code: https://github.com/cheng-haha/GPT-Policy

Code: https://github.com/cheng-haha/GPT-Policy

Project page: https://cheng-haha.github.io/GPT-Policy

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• In-Context Learning — 16× in this paper · explained in 7 episodes
• Adaptation — 12× in this paper · explained in 1 episode
• Verification — 8× in this paper · explained in 2 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Foundation Model — 4× in this paper · explained in 3 episodes
• Foundation Models — 4× in this paper · explained in 7 episodes
• Planner — 4× in this paper · explained in 2 episodes
• Affordance — 3× in this paper · explained in 1 episode
• Generalization — 3× in this paper · explained in 13 episodes
• In-Context Learning (ICL) — 3× in this paper · explained in 6 episodes
• Vision-Language Models (VLMs) — 3× in this paper · explained in 6 episodes
• Vision-language models — 3× in this paper · explained in 2 episodes
• Adaptability — 2× in this paper · explained in 1 episode
• Embodied AI — 2× in this paper · explained in 2 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Action Flow — 1× in this paper · explained in 1 episode
• Active Exploration — 1× in this paper · explained in 2 episodes
• Active Perception — 1× in this paper · explained in 2 episodes
• Agentic System — 1× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Task Planning — 1× in this paper · explained in 2 episodes
• Vision-Language Model (VLM) — 1× in this paper · explained in 3 episodes

Sources

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Flamingo: a Visual Language Model for Few-Shot Learning
• $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Show-Harness: Just a VLM Agent Can Play Robots
• In-Context Imitation Learning via Next-Token Prediction
• RoboTTT: Context Scaling for Robot Policies
• OpenVLA: An Open-Source Vision-Language-Action Model
• Kimi K3: Open Frontier Intelligence
• Code as Policies: Language Model Programs for Embodied Control
• ASPIRE: Agentic /Skills Discovery for Robotics
• Octo: An Open-Source Generalist Robot Policy
• What Enables In-Context Behavior Prompting for Manipulation?
• MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
• RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models
• Instant Policy: In-Context Imitation Learning via Graph Diffusion
• ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
• In-Context Learning Enables Robot Action Prediction in LLMs

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models