Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix

The gist The study presents a novel approach leveraging Instruct Pix2Pix to develop prompting methodologies that generate realistic datasets with weather-based augmentations aiming to mitigate the impact of adverse weather on the perception capabilities of state-of-the-art object detection models.

Introduction and Motivation

Adverse weather conditions pose critical challenges for autonomous driving technology due to their impact on sensor reliability and overall safety. Object detection systems, such as Faster R-CNN and YOLOv10, face difficulties in collecting datasets that accurately represent these challenging scenarios. The authors address this by proposing a data augmentation technique based on a sequential prompting strategy using diffusion models, particularly the Instruct Pix2Pix model. This method aims to generate realistic weather conditions while preserving spatial and semantic consistency essential for building effective object models.

Background on Models and Data Challenges

Object detection models are categorized into one-stage models like YOLO [3] and two-stage region-based models like Faster R-CNN [2]. While these models offer distinct solutions, they struggle with unpredictable driving scenarios caused by adverse weather. Furthermore, collecting analogous data in different weather conditions is difficult, and for scarce data like videos prior to severe crashes, it is virtually impossible. Diffusion models have emerged as a powerful family of generative models capable of creating high-quality images by gradually adding noise and learning to reverse the process. The Instruct Pix2Pix model expands this capability by introducing image and text conditioning based on the Stable Diffusion framework.

Methodology for Data Augmentation

The core methodology involves a data augmentation strategy applied to two data frames: simulated and real-world. The process is outlined in Algorithm 1, which involves three main steps:

1. Prompt Application, where the Instruct Pix2Pix model is applied to every image in S using predefined prompts and hyperparameters for each target weather condition. In some cases, different prompts are required to be added sequentially.

2. Augmented Image Generation, which generates augmented images D′ containing the specified weather-induced modifications.

3. Filtering Step, which performs a manual filtering step on D′ to remove augmented images with missing objects due to model hallucinations and those exhibiting extreme or unrealistic perturbations. The final dataset is formed by integrating the remaining filtered images into the original dataset.

Experimental Setup and Results

Experiments were conducted in two environments: the CARLA simulator and real-world image datasets BDD100K and ACDC. The training details varied by model: Faster-RCNN used a ResNet-50 backbone with a learning rate of 1×10−4 for 10 epochs. YOLOv10 models (Nano and Medium) used pre-trained weights from COCO, training with a learning rate of 1 × 10−3 for 100 epochs. Performance was evaluated using a resampling method that draws 1,000 bootstrap samples of the same size as the original dataset for each weather condition to produce stable performance estimates.

In the simulated framework, data augmentation significantly improved robustness; for instance, in Table IV (Faster R-CNN), the Augmented approach yielded better results in all models compared to the Basic approach under adverse weather conditions. For real-world data, Faster R-CNN benefited most from these augmentations, showing a smaller performance gap across different conditions compared to a baseline trained strictly on clear-weather images. However, YOLO models produced very similar results in real-world evaluations, suggesting that the benefit was less pronounced for single-stage models.

Conclusion and Future Work

The study concludes that diffusion-based image augmentations can be a practical and powerful tool to enhance the robustness of object detection in adverse weather conditions. In the simulated environment, augmentations reduced the accuracy drop when models were evaluated under challenging weather conditions for both Faster-RCNN and YOLO models. For real-world data, Faster R-CNN demonstrated an increase in robustness under adverse conditions, achieving improvements such as a 0.07 mAP boost for pedestrian detection in night and rain settings compared to the Basic approach. Challenges remain regarding the manual filtering of hallucinated images, which suggests a need for future automation through methods such as [18]. Additionally, achieving a stronger benefit for single-stage YOLO models under real-world conditions may require more specialized training schemes. Finally, expansion to additional weather phenomena and refined hyperparameter tuning could further validate the approach.

ACKNOWLEDGMENT

This work has received funding from the European Union’s Chips Joint Undertaking (ChipsJU) framework, under grant agreement No 101139892 (EdgeAI-Trust), PCI2024-153417 financed by MICIU/AEI /10.13039/501100011033, and PLEC2023-010240 financed by MICIU/AEI / 1.

REFERENCES

[2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards realtime object detection with region proposal networks,” 2016.

[3] A. Wang et al., “YOLOv10: real-time end-to-end object detection,” 2024.

[4] T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: learning to follow image editing instructions,” 2023.

[5] International Standards Organization, ISO 26262.

[6] ——, ISO 26262. Road Vehicles — Safety of the Intended Functionality.

[7] ——, ISO/PAS 8800. Road Vehicles — Safety and Artificial Intelligence.

[8] J. Fernandez et al., “AI-FSM: Towards Functional Safety Management ´ for Artificial Intelligence-based Critical Systems,” in CARS@EDCC2024 Workshop - Critical Automotive applications: Robustness & Safety, Leuven, Belgium, Apr. 2024.

[9] J. Terven, D.-M. Cordova-Esparza, and J.-A. Romero-Gonz ´ alez, “A ´ comprehensive review of YOLO architectures in computer vision: from YOLOv1 to YOLOv8 and YOLO-NAS,” Machine Learning and Knowledge Extraction, vol. 5, no. 4.

[10] L. Yang et al., “Diffusion models: a comprehensive survey of methods and applications,” 2024.

[11] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” 2020.

[12] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “Highresolution image synthesis with latent diffusion models,” 2022.

[13] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: an open urban driving simulator,” 2017.

[14] M. Chaar, J. Raiyn, and G.

1. Data Augmentation Pipeline Enhancement: The methodology can be improved by automating Step 3, Filtering Step, to reduce human oversight while maintaining quality, as suggested by reference [18]. This automation would ensure that only high-fidelity augmented images are integrated into the training set.

2. Robustness Gap Mitigation: The system can achieve a smaller performance gap between default and adverse weather conditions in real-world evaluations by focusing on techniques that yield this improvement for Faster R-CNN, as observed in the results section of the paper. This suggests tailoring augmentation strategies specifically to address the weaknesses of different model architectures.

3. YOLO Model Specialization: For single-stage YOLO models, a more specialized training scheme should be investigated to achieve stable performance under default conditions, addressing the observation that YOLO-based models did not display the same level of benefit in real-world evaluations. This could involve modifying the learning rate or batch size specifically for these architectures.

4. Hallucination Reduction Strategy: Implement a more sophisticated mechanism within Step 3 to remove hallucinations beyond manual filtering, potentially by using classifier-free guidance adjustments during inference, as implied by the discussion on guidance scale in Section IV.B.

Enhancing the robustness of object detection systems under adverse weather conditions is crucial for the advancement of autonomous driving technology. This study presents a novel approach leveraging the diffusion model Instruct Pix2Pix to develop prompting methodologies that generate realistic datasets with weather-based augmentations aiming to mitigate the impact of adverse weather on the perception capabilities of state-of-the-art object detection models, including Faster R-CNN and YOLOv10. Experiments were conducted in two environments, in the CARLA simulator where an initial evaluation of the proposed data augmentation was provided, and then on the real-world image data sets BDD100K and ACDC demonstrating the effectiveness of the approach in real environments. The key contributions of this work are twofold: (1) identifying and quantifying the performance gap in object detection models under challenging weather conditions, and (2) demonstrating how tailored data augmentation strategies can significantly enhance the robustness of these models. This research establishes a solid foundation for improving the reliability of perception systems in demanding environmental scenarios, and provides a pathway for future advancements in autonomous driving.