GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model

The gist The main finding is that for short-to-medium horizon waypoint navigation, strong pretrained MLLM representations can be made actionable without training a new visual encoder or adding a continuous regression head.

GemNav Architecture and Adaptation

GemNav introduces a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for waypoint navigation using Low-Rank Adaptation (LoRA) on the language tower alone, with no auxiliary visual encoder and no continuous regression head <ref:2607.06882#pg2> The policy formulates navigation as multimodal sequence prediction in which observations and goal specifications map to discrete action tokens from the language-model head <ref:2607.06882#pg2> Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head <ref:2607.06882#pg2> The model is adapted via LoRA on all of the linear layers in the language tower (r=32, α=16, dropout 0) <ref:2607.06882#pg2> The vision tower remains frozen, with no LoRA targets, unfrozen parameters, or auxiliary encoder <ref:2607.06882#pg2>.

Discrete-Tokenization and Loss Function

GemNav introduces a discrete value tokenization that unifies continuous waypoint prediction and categorical stop signals (and) in a single 18-token sequence produced by the language-model head <ref:2607.06882#pg2>. This includes 64 value-bin tokens plus five role tokens, resulting in a tokenization of length 18 for N=8 waypoints <ref:2607.06882#pg2>. The target sequence for a trajectory sample is T (τ) =, vb(x1), vb(y1),..., vb(x8), vb(y8), <ref:2607.06882#pg2>. To restore the metric structure discarded by pure cross-entropy training, a soft-decoded auxiliary loss is introduced to recover the metric structure to the bin distribution and reduces validation displacement error by 23% over standard cross-entropy <ref:2607.06882#pg2>.

Training Protocol and Auxiliary Loss

The policy is supervised by token-level cross-entropy LCE on the target sequence (Eq. 2) <ref:2607.06882#pg2>. Stop and unreachable tokens are trained by the same cross-entropy as value bins so the model can refuse to plan when appropriate <ref:2607.06882#pg2>. The auxiliary loss Laux = 1/2 N Σ X N k=1 X a∈[x,y] aˆk − ak 2 is computed only on trajectory samples (stop and unreachable samples are skipped) <ref:2607.06882#pg2>. This term decodes through the existing value-bin tokens via Eq. 3, preserving the discrete-token interface without adding parameters <ref:2607.06882#pg2>.

Experimental Results and Performance

GemNav was evaluated on a single open dataset of approximately 8.7 hours, roughly three orders of magnitude smaller than competing training sets <ref:2607.06882#pg2>. The policy transfers zero-shot to four physically distinct unseen environments and stops within 0.25–0.42 m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse <ref:2607.06882#pg2>. The combined ADE is 0.49 m on SCAND, with a correctly predicted goal-reached signal on all stop samples (50/50) <ref:2607.06882#pg2>. In real-world deployment, GemNav succeeds in every pose-goal trial, achieving a 5/0/0 stop/partial/miss tally in each of the four environments <ref:2607.06882#pg2>.

Ablation Studies and Limitations

The paper reports that a two- or four-frame history window improves offline validation metrics but provides no deployment benefit over a single-frame policy, suggesting temporal context yields diminishing returns when the pretrained vision features are already strong <ref:2607.06882#pg2>. Furthermore, the distance-based goal sampler is shown to be superior to the legacy frame-based sampler because it deploys successfully across all four environments <ref:2607.06882#pg2>. The paper notes that cross-embodiment transfer remains untested as GemNav is trained on a single platform and deployed on one robot <ref:2607.06882#pg2>.

Conclusion

GemNav suggests that action representation, rather than further visual scaling alone, is an important design axis for adapting MLLMs to robot navigation <ref:2607.06882#pg2>. Future work will extend this study along three directions: evaluating cross-embodiment transfer across additional robot platforms, and improving pure image-goal stopping without relying on a goal coordinate <ref:2607.06882#pg2>.

--- Page 1 ---

The gist The main finding is that for short-to-medium horizon waypoint navigation, strong pretrained MLLM representations can be made actionable without training a new visual encoder or adding a continuous regression head.

GemNav Architecture and Adaptation

GemNav introduces a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for waypoint navigation using Low-Rank Adaptation (LoRA) on the language tower alone, with no auxiliary visual encoder and no continuous regression head <ref:2607.06882#pg2> The policy formulates navigation as multimodal sequence prediction in which observations and goal specifications map to discrete action tokens from the language-model head <ref:2607.06882#pg2> Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head <ref:2607.06882#pg2> The model is adapted via LoRA on all of the linear layers in the language tower (r=32, α=16, dropout 0) <ref:2607.06882#pg2>. The vision tower remains frozen, with no LoRA targets, unfrozen parameters, or auxiliary encoder <ref:2607.06882#pg2>.

Experimental Results and Performance

GemNav was evaluated on a single open dataset of approximately 8.7 hours, roughly three orders of magnitude smaller than competing training sets <ref:2607.06882#pg2>. The policy transfers zero-shot to four physically distinct unseen environments and stops within 0.25–0.42 m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse <ref:2607.06882#pg2>. The combined ADE is 0.49 m on SCAND, with a correctly predicted goal-reached signal on all stop samples (50/50) <ref:2607.06882#pg2>.

1. textbf Discrete-Token Action Representation for Navigation Enables Zero-Shot Transfer via Frozen MLLM Grounding: This approach allows discrete-token adaptation of frozen MLLMs can provide a data-efficient, deployable alternative for foundation model robot navigation, enabling policies to achieve zero-shot to four physically distinct unseen environments using only an 8.7-hour corpus with no auxiliary visual encoder or continuous regression head.

2. textbf Metric Structure Recovery via Soft-Decoded Auxiliary Loss: The introduction of a soft-decoded auxiliary loss recovers the metric structure that pure cross-entropy training discards, which has been shown to reduce validation displacement error by 23% over standard cross-entropy by penalizing distant bin errors more heavily, ensuring the model respects the physical meaning of waypoint tokens.

3. textbf Robust Goal Reaching via Pose and Coordinate Modalities: The system is designed to handle diverse goal specifications, allowing it to align metric waypoints and navigation-state decisions with the model’s native token interface when conditioned on ego, pose, and ego+pose goal modalities, successfully stopping in environments like the Warehouse within specific bounds (stops within 0.25m and 0.42m of the goal).

4. textbf Adaptive Temporal Context Management: While visual history improves offline metrics, real-world deployment benefits from a single-frame policy because adding short visual histories improves validation ADE but does not improve real-robot deployment, suggesting that past visual context encourages the policy to continue previous motion which can cause overshoot in closed-loop settings.

5. textbf Scalable Deployment via Distanced Goal Sampling: The use of a distance-based goal sampler fixes a critical failure mode where frame alignment limits trajectory length, as it ensures the model deploys successfully across all four environments, unlike legacy frame-based samplers which cause the policy to emit on every modality, effectively refusing to plan irrespective of the actual goal.