Illumination-Aware Active Perception for Spacecraft Inspection

arXiv:2610.22335 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Illumination-Aware Active Perception for Spacecraft Inspection".

Dev: The gist The proposed inspection-planning framework replaces geometric visibility with a learned measure of camera information derived from a classifier variational autoencoder (VAE) that operates directly on the captured image,…

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So to recap on "Illumination-Aware Active Perception for Spacecraft Inspection", the core idea is moving past simple geometric visibility checks which are flawed when illumination changes rapidly in space >

Dev: The paper claims that instead of relying on these geometric metrics which conflate visibility with actual task relevance, we should use a learned measure of camera information derived from a classifier variational autoencoder or VAE operating directly on the image >

Taro: The thesis is that the encoder model is trained to classify spacecraft surface components from rendered patches and its total latent variance reveals the varying illumination of those components >

Rosa: So, this learned illumination distribution replaces binary visibility checks with a continuous quantity where high variance means dark or occluded or specular, which is good for planning >

Dev: This allows them to replace geometric metrics with something that operates on image pixels directly from the onboard camera and use that as the inspection probability nu3 of s and a >

Taro: The overall goal is to identify components like solar arrays or radiators and assess their condition, whether nominal or degraded, without relying only on viewpoint geometry >

Rosa: It matters because without camera data, geometrical checks severely limit how well a planner can distinguish orbital viewpoints that actually give us information for inspection >

Dev: This means existing methods regularly admit false positives and false negatives during the inspection task because they don't account for illumination effects like eclipse or self-shadowing >

Taro: So this changes the way we think about active perception by using image data directly rather than just relying on precomputed geometric metrics >

Rosa: This is important because it makes the system more robust when dealing with the challenging lighting conditions inherent in orbital environments >

Dev: The results they show, comparing their approach to others like geometric planning with an illumination band and purely geometric planning, show a much better performance in terms of illumination quality >

Conclusion: Rosa: We're wrapping up this discussion on "Illumination-Aware Active Perception for Spacecraft Inspection" by discussing the big picture implications >

Dev: The authors are presenting this inspection-planning framework as a way to move toward more reliable autonomous inspection in space situations >

Taro: In simple terms, they are proposing that instead of just checking if you can see something from a certain angle, you use AI to learn what the lighting is doing to that object >

Rosa: So when we talk about this framework, it means the system uses image data as its primary source for decision making rather than just relying on geometric calculations >

Dev: It implies that for space situational awareness and proximity operations, we need a way to handle those tricky orbital environments where lighting is constantly shifting >

Taro: So what this paper suggests is that using learned perception metrics based on image variance gives you a much richer signal for planning than just counting line-of-sight intersections >

Rosa: The implication is that future autonomous systems in orbit won't need to rely on purely analytical models for inspection and can be more resilient to the unpredictable lighting >

Dev: It suggests that the next step involves extending this work into fault detection where this learned model could help differentiate between healthy parts and damaged ones without losing confidence in those observations >

Taro: It really points toward using uncertainty guided policy planners or other techniques that rank candidate views by predicted rendering uncertainty as a way to improve observation of a target >

Sagarika Rao Valluri, Benjamin Riviere

New York University · Mechanical and Aerospace Engineering · Computer Science

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: The gist The proposed inspection-planning framework replaces geometric visibility with a learned measure of camera information derived from a classifier variational autoencoder (VAE) that operates

Key concepts

Classifier Variational Autoencoder (VAE): A deep learning model trained to classify different spacecraft surface components from rendered patches. It is also used to learn the distribution of illumination for each component, allowing the system to understand how light affects the target.
Latent Variance: A measure derived from the VAE that quantifies uncertainty or variation in a component's appearance. High variance indicates poor visibility due to factors like darkness, occlusion, or specular reflection.
Image-Grounded Metric: A continuous inspection probability score calculated directly from the captured image pixels rather than relying on pre-defined geometric rules. This score accounts for complex lighting conditions and provides a more accurate assessment of whether a component is truly visible and informative.

Terminology

Active Perception — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed inspection-planning framework replaces geometric visibility with a learned measure of camera information derived from a classifier variational autoencoder (VAE) that operates directly on the captured image, resulting in better illumination than existing geometric and illumination-band approaches.

Problem Statement

Autonomous inspection of resident space objects is essential for space situational awareness and is an important tool to execute proximity operations in constrained orbital environments<ref:2610.22335#pg2> The primary objective of autonomous inspection is to maximize information gain of the target spacecraft in motion<ref:2610.22335#pg2> Without camera data, geometrical checks limit the planner’s ability to distinguish orbital viewpoints that yield photometrically informative measurements from those that do not, regularly admitting false positives/negatives during the inspection task<ref:2610.22335#pg2>

How it works

The proposed solution replaces geometrically computed metrics with a learned metric operating on image pixels directly from the onboard camera<ref:2610.22335#pg2> Specifically, a classifier variational autoencoder (VAE) is trained to classify target components from rendered imagery and learn corresponding illumination distributions<ref:2610.22335#pg2> During evaluation, the model’s latent variance is used for the inspection probability where it is high for dark, occluded, specular, or out-of-distribution patches and is low for well-resolved, in-distribution patches<ref:2610.22335#pg2> This learned metric replaces binary visibility checks with a continuous, image-grounded quantity<ref:2610.22335#pg2>

The inspection task is formulated as a Markov Decision Process (MDP) where the state space s is a combination of the physical state x and inspection z<ref:2610.22335#pg2> The reward R is defined as the expected change in detection of all the POIs<ref:2610.22335#pg2> The learned inspection probability ν3(s, a) is derived from the inverse encoder latent variance, which consumes the actual rendered image<ref:2610.22335#pg2>

Comparison to Baselines

The approach was evaluated by implementing a Guidance, Navigation, and Control architecture in a photorealistic physics simulator<ref:2610.22335#pg2> The evaluation showed that the closed-loop system equipped with the learned model captures images with a mean brightness of 0.57, compared with 0.04 and 0.02 for the geometric and illumination-band baselines<ref:2610.22335#pg2> This represents an increase of approximately 76 percentage points over the stronger baseline<ref:2610.22335#pg2> Furthermore, under the learned reward ν3, it reaches 190 POIs with ≈ 185 inspected within the first orbit, whereas the geometric baseline π1 only reaches 42 (2610.223357) and π2 attains only 100 POIs (6).

Qualitative Results

A qualitative analysis showed that ν1 conclusively observes POI’s placed on the target at t=5 secs even though the module is incomplete darkness, which are considered false positives<ref:2610.22335#pg8> Conversely, ν2 picks viewing angles to observe POI’s under better lighting conditions but rejects POI’s due to its conservative illumination band limits resulting in false negatives and also fails to account for self shadows<ref:2610.22335#pg8> The learned model ν3 overcomes the limitations of the geometric models by observing the POI’s under good lighting conditions and proceeding to also admit dim POI’s placed between modules<ref:2610.22335#pg8>

Conclusion

We present an illumination-aware active-perception framework that replaces geometric visibility with a learned, image-based detection score from a classifier VAE in a complete GNC system<ref:2610.22335#pg8> By evaluating information from the captured image directly, the framework remains accurate under harsh lighting conditions for complex structures in orbit and hence is a robust alternative to analytical models for orbital inspection<ref:2610.22335#pg8> Future work would extend to fault detection, where the learned model can differentiate between healthy components and broken modules while maintaining confidence of these observations over uncertainty<ref:2610.22335#pg8>

References

[1] W. Ailor, “Protecting the leo environment,” Journal of Space Safety Engineering, vol. 9, no. 3, pp. 449–454, 2022.

[6] Y. K. Nakka, W. Honig, C. Choi, A. Harvard, A. Rahmani, and S.-J. Chung, “Informationbased guidance and control architecture for multispacecraft on-orbit inspection,” Journal of Guidance, Control, and Dynamics, vol. 45, pp 1184–1201

[7] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on Learning Representations (ICLR), Banff, AB, Canada, 2014.

[8] NVIDIA, “Isaac Sim,” https://developer.nvidia.com/isaac/sim, 2024

[9] D. Malyuta, Y. Yu, P. Elango, and B. Ac¸ıkmes¸e, “Advances in trajectory optimization for space vehicle control,” Annual Reviews in Control, vol. 52, pp 282–315

[10] T. Lauinger and S. Ulrich, “Path planning for optimal coverage of orbiting space structures using lissajous curves,” in AAS/AIAA Space Flight Mechanics Meeting, Jan. 2025

[31] W. H. Clohessy and R. S. Wiltshire, “Terminal guidance system for satellite rendezvous,” Journal of the Aerospace Sciences, vol. 27, no. 9, pp 653–658

[34] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in International Conference on Learning Representations (ICLR), 2014.

[37] NASA, “3D Resources,” https://science.nasa.gov/3d-resources/, 2026

[4] D. Fourie, B. E. Tweddle, S. Ulrich, and A. SaenzOtero, “Flight results of vision-based navigation and control for autonomous spacecraft inspection of an unknown object,” Journal of Spacecraft and Rockets, vol. 51, no. 6, pp 2016–2026

[5] M. Sabatini, R. Volpe, and G. B. Palmerini, “Centralized visual based navigation and control of a swarm of satellites for on-orbit servicing,” Acta Astronautica, vol. 171, pp 323–334

[7] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on Learning Representations (ICLR), Banff, AB, Canada, 2014.

[8] NVIDIA, “Isaac Sim,” https://developer.nvidia.com/isaac/sim, 2024

[9] D. Malyuta, Y. Yu, P. Elango, and B. Ac¸ıkmes¸e, “Advances in trajectory optimization for space vehicle control,” Annual Reviews in Control, vol. 52

[10] T. Lauinger and S. Ulrich, “Path planning for optimal coverage of orbiting space structures using lissajous curves,” in AAS/AIAA Space Flight Mechanics Meeting, Jan. 2025<ref:2610.

Improvements for AI systems

Improve inspection metric by replacing geometric visibility with a learned image-grounded quantity: "We propose an inspection-planning framework that replaces geometric visibility with a learned measure of camera information derived from a classifier variational autoencoder (VAE) that operates directly on the captured image. This allows for detection based on illumination effects like eclipse, self-shadowing, and specular reflection," which are overlooked by existing methods.
Develop an inspection policy that maximizes illumination quality instead of just maximizing coverage: The learned metric's latent variance is used as the inspection probability where it is high for dark, occluded, or specularly saturated patches yield high variance and a low score; well-resolved, in-distribution patches yield a high score. This ensures the system prioritizes observing components under conditions that provide the most informative image data.
Enhance robustness against harsh orbital illumination: The proposed system achieves superior performance under varying lighting compared to baselines, showing a mean brightness of 0.57, with 81.6% of observations above 0.25, compared with 0.02 (3.0%) for geometric planning with an illumination band and 0.04 (6.0%) for purely geometric planning. This directly addresses the harsh and rapidly varying orbital illumination mentioned in the abstract by providing a more reliable measure of component inspection quality.
Implement a closed-loop GNC architecture leveraging image data: The system utilizes an inspection-based Guidance, Navigation, and Control stack where the decision layer selects viewpoints based on the learned metric to maximize the expected reward of Eq. 5, leading to a policy that nearly matches the illumination-band baseline on that baseline’s own reward (193 versus 197 POIs), while the reverse does not hold.
Enable component identification through classification: The VAE encoder is trained to classify spacecraft surface components from rendered patches, allowing the system to determine whether a surface element is a solar array, a radiator, or a docking interface in addition to merely confirming its presence.

Abstract

Harsh and rapidly varying orbital illumination introduces failure modes during spacecraft inspection. Existing planners rely on line-of-sight inspection metrics which perform occlusion checks against a known target and conflate geometric visibility with task-relevant information. These metrics overlook illumination effects including eclipse, self-shadowing, and specular reflection from spacecraft materials and tend to declare components as inspected despite the corresponding images lacking sufficient information. We propose an inspection-planning framework that replaces geometric visibility with a learned measure of camera information derived from a classifier variational autoencoder (VAE) that operates directly on the captured image. The encoder model is trained to classify spacecraft surface components from rendered patches, and its total latent variance is used to learn the varying illumination of said components, replacing binary visibility checks with a continuous, image-grounded quantity. We evaluate the approach by implementing a Guidance, Navigation, and Control architecture in a photorealistic physics simulator, and vary lighting and viewing geometries with multiple orbits. We show that, although all approaches successfully maximize their own objective, our approach results in better illumination: a mean brightness of 0.57, with 81.6% of observations above 0.25, compared with 0.02 (3.0%) for geometric planning with an illumination band and 0.04 (6.0%) for purely geometric planning.