Road Surface Friction Estimation for Winter Conditions Utilising General Visual Features

arXiv:2404.16578 · cs.CV · Submitted 2024-04-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Road Surface Friction Estimation for Winter Conditions Utilising General Visual Features".

Jane: Here's the gist of the paper. The gist: The proposed WCamNet model, which combines general visual features from a pretrained DINOv2 foundation model with a CNN processing pipeline, outperforms previous state-of-the-art models in predicting road surface friction from roadside camera images Problem Formulation and Dataset The task focuses on the application of deep learning for analyzing the road surface condition from a roadside camera image, formulated as a regression problem where the model predicts the road surface friction as a scalar value A large dataset was gathered from the national Finnish road infrastructure network consisting of roadside cameras and optical road friction sensors A total of 31 pairs of roadside camera stations and weather stations around Finland were utilized for data collection Data samples were gathered at approximately 20 minute intervals for roughly two months in February-March of 2023 The road friction sensors reported friction as a continuous value, grip factor, with a measurement range of 0.09-0.82 Lower values corresponded to slippery conditions, and higher values indicated better tyre grip Optical road friction sensor information from weather stations was used to label the images captured from the corresponding roadside camera station Model Architecture The proposed WCamNet model infuses general visual features from a foundation model with a CNN processing pipeline The major motivation for the model was that standalone CNN architectures lack attention mechanisms due to local feature extraction, consequently being unable to properly focus on the relevant sections of the images WCamNet applies a visual foundation model, DINOv2, to fuse scene understanding and attention for the CNN processing The moderately sized DINOv2-B model was chosen for the architecture During training of the network, the DINOv2 model remains frozen, while the weights of other layers are optimized When performing inference with the network, each RGB input image is resized to 602x602 and then fed to DINOv2 and a high-definition (HD) branch for parallel processing The final patch tokens produced by DINOv2 and the features from the HD branch are concatenated into a single tensor, which is fed through two consecutive standard residual squeeze-and-excitation (SE) blocks These SE blocks enable weighing the relevant extracted features, along with providing additional convolutional feature extraction The output of the consecutive SE blocks is pooled with global average pooling and fed through a fully connected layer with sigmoid activation Training and Testing The dataset was split with an approximate 50%-15%-35% train, validation, and test split, respectively A high portion of testing data was chosen to ensure that the models were well generalisable to different camera views WCamNet was trained for 15 epochs, and a Cosine Annealing learning rate scheduler with a warm restart every five epochs The CNN models, ResNet50, ResNet152 and VGG19 were trained for 30 epochs with a step-based learning rate scheduler reducing the learning rate to one tenth every ten epochs All models were evaluated with PyTorch implementations, trained with mean squared error loss function, applying stochastic gradient descent with a momentum of 0.9 Images fed to the networks were resized to 602x602 pixels and normalised based on the training set statistics Accuracy Results The acquired results demonstrate that WCamNet outperformed the other models in the accuracy benchmark VGG19 was able to reach the second lowest errors in the benchmark Compared to VGG19, WCamNet reached a 10% lower MAE and a 7% lower RMSE Ablation Study The ablation study showed that the design choices behind WCamNet are reasonable and improve performance Removal of SE blocks or the HD branch resulted in notable increase in errors, indicating that these elements are crucial for the performance of the model Discussion The presented WCamNet model was shown to improve the state of the art of camera-based road surface friction estimation in winter conditions The hybrid architecture combining general visual features from DINOv2 and a CNN processing pipeline was shown to affect performance favourably Although WCamNet was the most accurate computer vision model in the benchmark, the error values were still quite notable on an absolute scale Another factor affecting the absolute error is the spatial ambiguity in the image data Overall, the large quantity of the data used in this study should compensate for ambiguities in the data Future Research Directions To further enhance camera-based road surface friction estimation in winter conditions, other weather data could be included as input in the prediction models Fusion of camera images and weather information has been proposed in the past

5: , yet modern deep learning methods could likely demonstrate notable performance gains This type of data fusion could enable even higher accuracy in computer vision-based approaches, facilitating adoption of camera-based road monitoring methods in intelligent transportation applications Acknowledgment We acknowledge the funding provided by Henry Ford Foundation Finland, the computational resources provided by the Aalto Science-IT project, and the data provided by Fintraffic References (Selected)

1: C.-G.

Tom: Wallman and H.

Jane: Astr ˚ om, “Friction measurement methods and ¨ the correlation between road friction and traffic safety: A literature review,” 2001

7: K.

Tom: Simonyan and A.

Paper summary: Tom: So we've seen how they set up their problem: predicting road surface friction from a camera image in winter. The authors are proposing WCamNet, which uses a hybrid architecture combining general visual features from a pretrained visual transformer model with convolutional blocks.

Jane: That means the thesis is that this combination—getting the broad scene context from the transformer and the fine details from CNNs—is better for predicting that friction value than using either component by itself.

Lu: They specifically state that this combination of features is motivated because standalone CNN architectures often lack attention mechanisms due to their focus on local feature extraction, which can limit their ability to properly focus on the relevant sections of the images.

Meng: So, if you were looking at just a regular CNN, it might struggle to look at the whole picture and know where the critical icy patches are located without some kind of attention mechanism telling it what to focus on.

Tom: Exactly. WCamNet uses DINOv2 for that general visual feature fusion and then lets the convolutional blocks handle the fine extraction properties they need.

Jane: It seems like a very logical way to design a model when you have two different types of visual information—a global understanding and local texture detail—that both matter for this task.

Lu: And they mentioned that they benchmarked this against other CNN models, and in their tests, DenseNet201 showed the best performance, with VGG19 achieving the second best results.

Meng: So even with all those older models tested, WCamNet managed to get better than VGG19 on the accuracy benchmark. That’s a significant finding for this specific task.

Tom: It's significant because they reached a ten percent lower mean absolute error and a seven percent lower root mean square error compared to VGG19 in their evaluation. Those numbers are what we look at when comparing models, you know?

Jane: So, the paper claims WCamNet outperforms previous state-of-the-art models in this specific prediction task when evaluated on that dataset. That's what they’re claiming about the overall approach.

Lu: The authors are trying to move beyond just applying a transformer model or just a CNN by proposing this hybrid structure to get the best of both worlds for feature extraction properties.

Meng: What does that mean for deployment? Does it mean we can expect this to be easier to integrate into production systems, or is it still too heavy?

Tom: It means the architecture is designed to balance general features with fine extraction, aiming for a good trade-off in performance without necessarily making the system computationally impossible to run.

Jane: So the overall message is that integrating these different parts strategically can lead to better predictive accuracy for this kind of image analysis task.

Lu: And they built this on a large dataset from Finland, which is what gives their approach its grounding and relevance in real-world road conditions.

Meng: That data source makes sense because it grounds the research in conditions that are actually relevant for winter driving scenarios.

Tom: So so far, we’ve just covered what they are proposing and what they claim about outperforming previous models on this benchmark. Next up, we're going to look at their final thoughts on the paper, "Road Surface Friction Estimation for Winter Conditions Utilising General Visual Features".

Conclusion: Jane: So wrapping up the discussion on "Road Surface Friction Estimation for Winter Conditions Utilising General Visual Features", what’s the big picture here?

Tom: Essentially, this paper presents WCamNet as a hybrid architecture that successfully combines general visual features from DINOv2 with convolutional blocks to predict road surface friction. They showed that this hybrid structure leads to better predictive accuracy than previous models on their benchmark.

Lu: The main implication is the architectural design choice—combining global understanding with local feature extraction—is what drives the improved performance in this task.

Meng: From an engineering standpoint, it tells us that when designing a system, you can't just pick one component; you need to consider how different parts interact to get the best result.

Jane: So for someone listening who isn't reading the paper, it means that for predicting road friction in winter conditions from a camera image, using this specific hybrid method is what they found works well compared to other methods.

Tom: The authors are essentially confirming that their approach is superior to the existing state-of-the-art in terms of accuracy on their test set.

Lu: They did acknowledge that the spatial ambiguity in the image data is still a factor affecting the absolute error values, even with their large dataset.

Meng: So what they're admitting is that perfect prediction isn't achievable just by having more data; some inherent visual uncertainty remains regardless of how much you collect.

Jane: So for us, it means we need to keep looking at ways to reduce that remaining error, maybe by fusing in external weather data in the future.

Tom: That’s the path forward—looking at those next steps they suggest, like incorporating weather information into the prediction models for even higher accuracy.

Risto Ojala, Eerik Alamikkotervo

Aalto University

cs.CV

Submitted: 2024-04-25

Updated: 2024-04-25

DOI: 10.1109/ITSC58415.2024.10919639

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 73/100

The gist: The gist: The proposed WCamNet model, which combines general visual features from a pretrained DINOv2 foundation model with a CNN processing pipeline, outperforms previous state-of-the-art models in

Key concepts

WCamNet: A hybrid deep learning architecture that fuses general visual understanding from the DINOv2 foundation model with a standard CNN processing pipeline. It uses the transformer for scene understanding and CNN blocks for fine-tuned feature extraction, aiming to capture both broad visual context and specific surface details.
DINOv2 Foundation Model: A pre-trained visual transformer model used to provide general visual features to the WCamNet architecture. This model is frozen during training, allowing it to contribute scene understanding and attention mechanisms without being retrained, which helps the network focus on relevant image sections.
Road Surface Friction Prediction: The core task of the study is a regression problem where a deep learning model predicts a continuous scalar value representing road surface friction. Lower values indicate slippery conditions, while higher values suggest better tire grip, crucial for safe vehicle control.

Terminology

Transformer — 2× in this paper · explained in 2 episodes
Transformer Model — 2× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Feature Extraction — 1× in this paper · explained in 3 episodes

Summary

The gist: The proposed WCamNet model, which combines general visual features from a pretrained DINOv2 foundation model with a CNN processing pipeline, outperforms previous state-of-the-art models in predicting road surface friction from roadside camera images

Problem Formulation and Dataset

The task focuses on the application of deep learning for analyzing the road surface condition from a roadside camera image, formulated as a regression problem where the model predicts the road surface friction as a scalar value A large dataset was gathered from the national Finnish road infrastructure network consisting of roadside cameras and optical road friction sensors A total of 31 pairs of roadside camera stations and weather stations around Finland were utilized for data collection Data samples were gathered at approximately 20 minute intervals for roughly two months in February-March of 2023 The road friction sensors reported friction as a continuous value, grip factor, with a measurement range of 0.09-0.82 Lower values corresponded to slippery conditions, and higher values indicated better tyre grip Optical road friction sensor information from weather stations was used to label the images captured from the corresponding roadside camera station

Model Architecture

The proposed WCamNet model infuses general visual features from a foundation model with a CNN processing pipeline The major motivation for the model was that standalone CNN architectures lack attention mechanisms due to local feature extraction, consequently being unable to properly focus on the relevant sections of the images WCamNet applies a visual foundation model, DINOv2, to fuse scene understanding and attention for the CNN processing The moderately sized DINOv2-B model was chosen for the architecture During training of the network, the DINOv2 model remains frozen, while the weights of other layers are optimized When performing inference with the network, each RGB input image is resized to 602x602 and then fed to DINOv2 and a high-definition (HD) branch for parallel processing The final patch tokens produced by DINOv2 and the features from the HD branch are concatenated into a single tensor, which is fed through two consecutive standard residual squeeze-and-excitation (SE) blocks These SE blocks enable weighing the relevant extracted features, along with providing additional convolutional feature extraction The output of the consecutive SE blocks is pooled with global average pooling and fed through a fully connected layer with sigmoid activation

Training and Testing

The dataset was split with an approximate 50%-15%-35% train, validation, and test split, respectively A high portion of testing data was chosen to ensure that the models were well generalisable to different camera views WCamNet was trained for 15 epochs, and a Cosine Annealing learning rate scheduler with a warm restart every five epochs The CNN models, ResNet50, ResNet152 and VGG19 were trained for 30 epochs with a step-based learning rate scheduler reducing the learning rate to one tenth every ten epochs All models were evaluated with PyTorch implementations, trained with mean squared error loss function, applying stochastic gradient descent with a momentum of 0.9 Images fed to the networks were resized to 602x602 pixels and normalised based on the training set statistics

Accuracy Results

The acquired results demonstrate that WCamNet outperformed the other models in the accuracy benchmark VGG19 was able to reach the second lowest errors in the benchmark Compared to VGG19, WCamNet reached a 10% lower MAE and a 7% lower RMSE

Ablation Study

The ablation study showed that the design choices behind WCamNet are reasonable and improve performance Removal of SE blocks or the HD branch resulted in notable increase in errors, indicating that these elements are crucial for the performance of the model

Discussion

The presented WCamNet model was shown to improve the state of the art of camera-based road surface friction estimation in winter conditions The hybrid architecture combining general visual features from DINOv2 and a CNN processing pipeline was shown to affect performance favourably Although WCamNet was the most accurate computer vision model in the benchmark, the error values were still quite notable on an absolute scale Another factor affecting the absolute error is the spatial ambiguity in the image data Overall, the large quantity of the data used in this study should compensate for ambiguities in the data

Future Research Directions

To further enhance camera-based road surface friction estimation in winter conditions, other weather data could be included as input in the prediction models Fusion of camera images and weather information has been proposed in the past [5], yet modern deep learning methods could likely demonstrate notable performance gains This type of data fusion could enable even higher accuracy in computer vision-based approaches, facilitating adoption of camera-based road monitoring methods in intelligent transportation applications

Acknowledgment

We acknowledge the funding provided by Henry Ford Foundation Finland, the computational resources provided by the Aalto Science-IT project, and the data provided by Fintraffic

References (Selected)

[1] C.-G. Wallman and H. Astr ˚ om, “Friction measurement methods and ¨ the correlation between road friction and traffic safety: A literature review,” 2001

[7] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014

[30] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V.

Improvements for AI systems

Improve inference speed by replacing DINOv2-B with DINOv2-L backbone in WCamNet, as the larger model results in increased computational cost, suggesting a trade-off between higher accuracy and computational efficiency for real-time applications.
Enhance feature extraction robustness by incorporating weather data fusion, as this could provide valuable contextual information for determining the road surface friction to potentially enable even higher accuracy in computer vision-based approaches.
Increase spatial prediction resolution by moving beyond single ground truth labels, addressing the issue that a single ground truth reading was used to label each image, which is described as sub-optimal because the road surface condition can vary within different parts of a single image.

Abstract

In below freezing winter conditions, road surface friction can greatly vary based on the mixture of snow, ice, and water on the road. Friction between the road and vehicle tyres is a critical parameter defining vehicle dynamics, and therefore road surface friction information is essential to acquire for several intelligent transportation applications, such as safe control of automated vehicles or alerting drivers of slippery road conditions. This paper explores computer vision-based evaluation of road surface friction from roadside cameras. Previous studies have extensively investigated the application of convolutional neural networks for the task of evaluating the road surface condition from images. Here, we propose a hybrid deep learning architecture, WCamNet, consisting of a pretrained visual transformer model and convolutional blocks. The motivation of the architecture is to combine general visual features provided by the transformer model, as well as finetuned feature extraction properties of the convolutional blocks. To benchmark the approach, an extensive dataset was gathered from national Finnish road infrastructure network of roadside cameras and optical road surface friction sensors. Acquired results highlight that the proposed WCamNet outperforms previous approaches in the task of predicting the road surface friction from the roadside camera images.