VP-EVT: Active Visibility Preservation for Embodied Visual Tracking

The gist Embodied Visual Tracking (EVT) requires an agent to follow a designated target person in dynamic environments relying solely on egocentric visual perception, and VP-EVT treats target visibility as an explicit objective of motion control rather than a by-product of following

How it works

VP-EVT is a framework that treats target visibility as an explicit objective of motion control instead of a by-product of following Acting on this objective requires understanding both what obstructs the target and how robot motion can change that obstruction VP-EVT therefore uses language-guided perception to extract separate target and pedestrian observations When occlusion makes current observations incomplete, VP-EVT combines visual history with ego-motion to maintain estimates of their relative positions and motion Using these estimates and the current following waypoints, VP-EVT predicts whether continued following will sustain the obstruction When the predicted risk is low, the original following motion is retained; otherwise, a recurrent controller combines updated estimates with maneuver context from preceding decisions to coordinate viewpoint adjustment and the return to stable following

The framework comprises three complementary modules: (a) Target-Conditioned Visual Grounding, (b) Target–Pedestrian Temporal Belief, and (c) Plan-Conditioned Risk and Recurrent Control At decision time t, the system receives a causal RGB history It, a target description q, and robot ego-motion ∆Et First, visual grounding uses a frozen encoder and prompt-guided target and other-person maps to produce observations zTt and zPt Next, these observations, ego-motion, and prior estimates update a persistent target belief BTt and a short-term pedestrian belief BPt Finally, the beliefs are evaluated along the following waypoints Wt to anticipate persistent occlusion Standard following is retained when intervention is unnecessary; otherwise, a recurrent controller generates linear and angular velocity commands [v, ω] to adjust the viewpoint until stable following resumes

Target-Conditioned Visual Grounding

This module extracts visual evidence about both the designated target and potential occluders A frozen visual encoder extracts spatial features from a causal five-frame RGB history A target–pedestrian relation encoder uses the language prompt to produce a target map and an other-person map, retaining information under degraded observations [5], [6], [7], [8] For each map, the system retains its centroid, spatial spread, and confidence statistics Each observation head predicts a joint distribution over discretized bearing and distance Pet(i, j) = p(θi, dj It, q,Mt), e ∈ T, P These distributions are converted into local Cartesian positions using [x, y] = [d cos θ, d sin θ], with learned within-bin refinements These observations connect image-space grounding to robot-local spatial estimation

Target–Pedestrian Temporal Belief

Given the current observations zTt and zPt, the module maintains robot-local temporal beliefs that retain spatial estimates when visual evidence becomes unavailable For the target and pedestrian, indexed by e ∈ T, P, position and velocity are represented with a Gaussian belief xet = [xet, yet, vex,t, veyk,t]⊤ The target belief combines motion propagation with visual correction When target evidence is unavailable, the belief continues to propagate with increasing uncertainty until observations resume Coarse spatial belief encoding converts each filtered position distribution into a coarse robot-centric belief map on a polar grid discretized by bearing and distance These weights encode plausible target or pedestrian locations, providing a coarse spatial representation for subsequent risk prediction and control

Plan-Conditioned Risk and Recurrent Control

This module evaluates the following waypoints and selectively adjusts robot motion to preserve visibility Plan-conditioned occlusion risk predicts whether a pedestrian causes persistent occlusion based on the relative positions, motions, and current following waypoints The classifier is supervised by the duration of subsequent physical target loss in standard-following rollouts, distinguishing persistent obstruction from brief overlap or out-of-view failure When predicted risk is low, standard following continues; otherwise, a recurrent controller generates linear and angular velocity commands [v, ω] to adjust the viewpoint The recurrent controller uses a GRU to combine the previous state with the current context to obtain ht, and then predicts control increments ∆Ut A separate handoff predictor determines when standard following can resume once handoff criteria and visibility checks are satisfied across consecutive decisions

Evaluation and Results

OccEVT-Bench was introduced as a dedicated closed-loop benchmark for occlusion-aware embodied visual tracking that comprehensively quantifies visibility loss alongside tracking safety It measures target-loss persistence and accumulated visibility degradation alongside stable following recovery and collision safety, reporting physical-occlusion and out-of-view losses separately VP-EVT improves success rate by 121.4% relative to a competitive baseline and reduces visibility deficit and longest target-loss duration by 78.8% and 74.4%, respectively The results show that VP-EVT achieves a Risk SR of 77.5%, exceeding OA-VAT’s 35.0% by 42.5 percentage points Relative to OA-VAT, VP-EVT reduces VD by 78.8% and LTL by 74.4%, reaching 4.235 s and 3.442 s, respectively The loss decomposition reveals a further distinction, showing that VP-EVT reduces these values to 1.715 s and 1.747 s for POL and OOF, respectively

Conclusion

VP-EVT treats target visibility as a motion-control objective rather than a by-product of following Language-guided temporal beliefs support waypointconditioned occlusion prediction and selective recurrent control to preserve visibility and restore stable following Experiments show higher tracking success and reduced visibility degradation and continuous target loss, with comparable Safe-episode performance to the base follower Future work will improve intervention safety and robustness in denser crowds and diverse real-world settings The robot moves laterally around the pedestrian in the patterned shirt to reduce occlusion as it advances, and subsequently resumes following from a suitable position behind the target

--- Page 1 ---

VP-EVT: Active Visibility Preservation for Embodied Visual Tracking Yanci Wen, Tianle Zeng, Jianwei Peng, Weixi Situ, Hanjing Ye, and Hong Zhang Abstract— Embodied Visual Tracking (EVT) is an essential capability for intelligent robots in human–robot collaboration, requiring an agent to follow a designated target person in dynamic environments relying solely on egocentric visual perception. I. INTRODUCTION Embodied Visual Tracking (EVT) requires an agent to follow a designated moving target using egocentric visual observations [1], [2], [3], [4].

1. Bold header: Targeted Motion Control for Visibility Preservation

The system will treat target visibility as an explicit objective of motion control rather than a by-product of following. This allows the robot to proactively adjust its viewpoint to reduce obstructions, rather than simply continuing a motion that sustains occlusion, which is the core limitation addressed by VP-EVT.

2. Bold header: Language-Guided Target and Pedestrian Perception

The framework will utilize language-guided perception to extract separate target and pedestrian observations, using language prompts to generate distinct Target Map and Other-person Map. This separation allows for a more nuanced understanding of what obstructs the target versus where the target is located.

3. Bold header: Uncertainty-Aware Temporal Belief Maintenance

The system will maintain uncertainty-aware spatial and motion estimates even when direct observations are unavailable by integrating successive observations with ego-motion into target–pedestrian temporal beliefs. This ensures that positional estimates are not just propagated but corrected by visual evidence when it becomes available, preserving continuity during occlusion.

4. Bold header: Plan-Conditioned Risk Prediction

The system will predict the risk of persistent occlusion using the learned function rt = fr(B T t, B P t, W t, M t), evaluating whether continued following will sustain the obstruction. This predictive capability allows for proactive decision-making before a target loss becomes irreversible.

5. Bold header: Recurrent Visibility-Preserving Control

When risk is high, the system will generate linear and angular velocity commands [v, ω] to adjust the viewpoint until stable following resumes through a recurrent controller. This ensures that intervention is coordinated with maneuver context from preceding decisions to coordinate viewpoint adjustment and the return to stable following.

6. Bold header: Closed-Loop Evaluation via OccEVTBench

The evaluation will be conducted using OccEVTBench, a dedicated closed-loop benchmark for occlusion-aware embodied visual tracking that comprehensively quantifies visibility loss alongside tracking safety. This allows for the distinction between reduced persistent occlusion from eventual target recovery by reporting metrics like POL (physical-occlusion loss) and OOF (out-of-FOV loss) separately.

7. Bold header: Expert Demonstration Training

The system will be trained using OccEVT-Expert, an expert demonstration dataset containing 20,000 dynamic-occlusion episodes for training proactive visibility-preserving policies. This provides the necessary supervision to train the recurrent controller on complete expert maneuver sequences, ensuring that learned control actions are effective in real-world scenarios.