Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation

The gist The Safe-Night VLA framework integrates long-wave infrared thermal perception into a pre-trained vision-language backbone and couples it with a control barrier function safety filter to enable robots to see unseen states while enforcing geometric constraints for safe manipulation.

How it works

1. The Safe-Night VLA framework integrates LWIR thermal perception into a frozen VLM and couples it with a CBF safety filter integrates long-wave infrared thermal perception into a frozen VLM and couples it with a CBF safety filter.

2. The core perception engine, EAGLE 2.5 fuses a SigLIP-2 400M vision encoder with a Qwen3-1.7B large language model The core perception engine, EAGLE 2.5 fuses a SigLIP-2 400M vision encoder with a Qwen3-1.7B large language model.

3. The VLM backbone, comprising the vision encoder and language model, is strictly frozen to preserve pre-trained semantic representations The VLM backbone, comprising the vision encoder and language model, is strictly frozen to preserve pre-trained semantic representations.

4. Thermal and depth data are formatted as 3-channel pseudo-color images for the vision encoder Thermal and depth data are formatted as 3-channel pseudo-color images for the vision encoder.

5. An asymmetric data augmentation pipeline applies severe photometric perturbations exclusively to the RGB view to encourage attention to thermal and geometric representations An asymmetric data augmentation pipeline applies severe photometric perturbations exclusively to the RGB view to encourage attention to thermal and geometric representations.

Multimodal Input Processing

The system processes three synchronized camera views: RGB, thermal, and depth The system processes three synchronized camera views: RGB, thermal, and depth.

1. RGB (Irgb) provides global visual context RGB (Irgb) provides global visual context.

2. Thermal (Itherm) is cropped and aligned to the workspace, representing temperature gradients as 3-channel color intensities Thermal (Itherm) is cropped and aligned to the workspace, representing temperature gradients as 3-channel color intensities.

3. Depth (Idepth) maps are clipped and mapped to a Turbo colormap providing illumination-invariant geometric cues Depth (Idepth) maps are clipped and mapped to a Turbo colormap providing illumination-invariant geometric cues.

The thermal stream is used as a relative pseudo-color observation rather than calibrated radiometric thermometry The thermal stream is used as a relative pseudo-color observation rather than calibrated radiometric thermometry.

Safety Guarantee

To ensure safe execution with CBFs, the gripper actuation is decoupled from spatial motion using a jointspace runtime safety filter formulated as a strictly convex Quadratic Program (QP) To ensure safe execution with CBFs, the gripper actuation is decoupled from spatial motion using a jointspace runtime safety filter formulated as a strictly convex Quadratic Program (QP).

The QP solves directly for the safe joint displacement ∆qsafe to track the 6-DoF spatial intent uvla The QP solves directly for the safe joint displacement ∆qsafe to track the 6-DoF spatial intent uvla.

This formulation embeds a differential inverse kinematics solver, minimizing tracking error between physical robot motion and VLA Cartesian intent This formulation embeds a differential inverse kinematics solver, minimizing tracking error between physical robot motion and VLA Cartesian intent.

Experimental Validation

The framework was validated through real-world experiments on a Franka manipulator introducing a novel evaluation paradigm featuring temperature-conditioned manipulation, subsurface target localization, and reflection disambiguation We validate our framework through real-world experiments on a Franka manipulator introducing a novel evaluation paradigm featuring temperature-conditioned manipulation, subsurface target localization, and reflection disambiguation.

The results demonstrate that Safe-Night VLA outperforms RGB-only baselines and provides empirical evidence that foundation models can effectively leverage non-visible physical modalities for the safety-critical manipulation tasks Results demonstrate that Safe-Night VLA outperforms RGB-only baselines and provide empirical evidence that foundation models can effectively leverage non-visible physical modalities for the safety-critical manipulation tasks.

The policy actively grounds semantic tokens in thermal gradients rather than relying on dataset-induced spatial biases through cross-modal attention ablation studies The policy actively grounds semantic tokens in thermal gradients rather than relying on dataset-induced spatial biases through cross-modal attention ablation studies.

Failure Mode Analysis

Thermal input primarily resolves hidden physical states and cross-modal ambiguity, whereas depth and the runtime safety filter mainly improve execution robustness when illumination is degraded or the policy encounters modeled geometric constraints Thermal input primarily resolves hidden physical states and cross-modal ambiguity, whereas depth and the runtime safety filter mainly improve execution robustness when illumination is degraded or the policy encounters modeled geometric constraints.

Residual failures arise from recovery limitations and weak negative thermal contrast, where being safely stopped by the filter can still produce task-level failure Residual failures arise from recovery limitations and weak negative thermal contrast, where being safely stopped by the filter can still produce task-level failure.

The safety filter is a critical execution safeguard against geometrically unsafe OOD actions when RGB information is heavily attenuated under dim/night conditions The safety filter is a critical execution safeguard against geometrically unsafe OOD actions when RGB information is heavily attenuated under dim/night conditions.

Conclusion and Future Work

Safe-Night VLA addresses the fundamental limitations of RGB-only policies by integrating thermal perception with safety guarantees Safe-Night VLA addresses the fundamental limitations of RGB-only policies by integrating thermal perception with safety guarantees. Future work will focus on integrating real-time thermal point clouds to upgrade static geometric safeguards into dynamic, temperature-aware obstacle avoidance Future work will focus on integrating real-time thermal point clouds to upgrade static geometric safeguards into dynamic, temperature-aware obstacle avoidance. Furthermore, scaling this multimodal framework to SOTA, large-scale foundation models will be a key direction to unlock more robust manipulation in unstructured environments Furthermore, scaling this multimodal framework to SOTA, large-scale foundation models will be a key direction to unlock more robust manipulation in unstructured environments.

Safe-Night VLA addresses the fundamental limitations of RGB-only policies by integrating thermal perception with safety guarantees. Future work will focus on integrating real-time thermal point clouds to upgrade static geometric safeguards into dynamic, temperature-aware obstacle avoidance. Furthermore, scaling this multimodal framework to SOTA, large-scale foundation models will be a key direction to unlock more robust manipulation in unstructured environments.

--- Page 1 ---

Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation

Dian Yu∗, Qingchuan Zhou∗, Bingkun Huang, Majid Khadiv, Zewen Yang† Abstract— Current Vision-Language-Action (VLA) models rely primarily on RGB perception, preventing them from capturing modalities such as thermal signals that are imperceptible to conventional visual sensors Current Vision-Language-Action (VLA) models rely primarily on RGB perception, preventing them from capturing modalities such as thermal signals that are imperceptible to conventional visual sensors. Moreover, end-toend generative policies lack explicit safety constraints, making them fragile when encountering obstacles Moreover, end-toend generative policies lack explicit safety constraints, making them fragile when encountering obstacles. To address these limitations, we propose Safe-Night VLA, a multimodal manipulation framework that enables robots to see the unseen while enforcing geometric constraints for thermal-aware manipulation in unstructured environments To address these limitations, we propose Safe-Night VLA, a multimodal manipulation framework that enables robots to see the unseen while enforcing geometric constraints for thermal-aware manipulation in unstructured environments.

I. INTRODUCTION

State-of-the-art (SOTA) Vision-Language-Action (VLA) models have demonstrated impressive capability in grounding natural language into motor control within structured settings [1]–[4] State-of-the-art (SOTA) Vision-Language-Action (VLA) models have demonstrated impressive capability in grounding natural language into motor control within structured settings. As robot manipulation transitions toward unstructured real-world environments, however, policies increasingly face the dual challenge of the “unseen” As robot manipulation transitions toward unstructured real-world environments, however, policies increasingly face the dual challenge of the “unseen”. First, standard RGB sensors lack direct observability of intrinsic physical properties like surface temperature or sub-surface states, limiting the robot’s ability to perform thermodynamic reasoning First, standard RGB sensors lack direct observability of intrinsic physical properties like surface temperature or sub-surface states, limiting the robot’s ability to perform thermodynamic reasoning. Second, VLA models execute unpredictable actions when encountering out-of-distribution (OOD) scenes or workspace boundaries Second, VLA models execute unpredictable actions when encountering out-of-distribution (OOD) scenes or workspace boundaries.

Fig. 1. Multimodal perception comparison in downstream tasks Fig. 1. Multimodal perception comparison in downstream tasks. LWIR thermal observations (top row) are shown alongside RGB (middle row) and depth images (bottom row). Temperature-aware recognition: distinguishing thermally distinct yet visually indistinguishable objects for handling of hot vs.

1. Multimodal perception integration allows robots to see unseen physical states by integrating long-wave infrared thermal perception into a pre-trained vision-language backbone, enabling semantic reasoning guided by thermodynamic properties. This enables robots to distinguish between visually identical objects based on their surface temperature, such as hot versus cold items, which is impossible with RGB sensors alone.

2. Runtime safety enforcement via control barrier functions prevents unpredictable actions when encountering out-of-distribution scenes or workspace boundaries by bounding the VLA’s output space at the control level. This acts as a runtime geometric safeguard, intercepting policy hallucinations in OOD scenarios before they result in collisions.

3. The system can perform subsurface target localization by aligning surface thermal patterns with required subsurface states, detecting targets occluded beneath granular media through emitted thermal energy. This allows the robot to detect the surface “thermal bloom” produced when thermal energy diffuses through materials like crystal litter, even when the target is physically obscured.

4. The system can perform reflection disambiguation by leveraging the LWIR attenuation of common glass and silver mirrors, allowing it to distinguish between real objects and mirror-induced false targets. This is achieved because standard glass and silver mirrors are largely opaque to LWIR radiation, presenting a clear Single Box state instead of a visually deceptive reflection.

5. The system can execute constrained manipulation by solving a jointspace runtime safety filter formulated as a strictly convex Quadratic Program (QP) at each control step. This ensures safe execution by finding the safe joint displacement Deltaqsafe that tracks the desired Cartesian intent uvla while respecting collision constraints, guaranteeing deterministic workspace constraint enforcement during policy execution.

6. The system can provide robust manipulation in degraded environments by leveraging depth input to provide complementary geometric stabilization when RGB cues become unreliable under dim or night light conditions. This demonstrates that depth does not act as the primary semantic discriminator, but instead provides complementary geometric stabilization once RGB cues become unreliable.