Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation

arXiv:2601.06209 · cs.CV, stat.ML · Submitted 2026-01-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation".

Jane: Here's the gist of the paper. The gist: Active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets.

Tom: Active Learning aims to select informative samples from an unlabelled set of data, which is useful in cases where the amount of data is large and labelling is expensive such as in machine vision or medical imaging<ref:2601.06209#pg2>.

Jane: Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images<ref:2601.06209#pg2>.

Tom: This results, on the one hand, in a strong class imbalance in defect distribution and on the other hand, in a potential label shift caused by limited storage<ref:2601.06209#pg2>.

Paper summary: Jane: Wrapping up this discussion on "Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation," what's the main implication of this work for people who don't read the paper? What’s the big picture idea here?

Tom: The main implication is that active learning provides a practical, scalable path to building robust semantic segmentation models in machine vision <ref:2601.06209#pg14>.

Lu: It shows that it's not just about finding one perfect strategy but understanding how different strategies like entropy and core-set perform differently under varying conditions of imbalance and shift <ref:2601.06209#pg14>.

Jane: So, for the listener, if you have a data problem where you know there's imbalance and potential label shift, using an intelligent sampling method instead of just random selection is going to help your model get better results <ref:2601.06209#pg14>.

Meng: I think the authors are really showing that for deployment, active learning offers a way to improve performance beyond what you'd get from baseline methods like random sampling when things aren't perfectly balanced <ref:2601.06209#pg14>.

Tom: The core idea is that we can use active learning as a scalable pathway to get better models in areas with substantial class imbalance and plenty of unlabelled data <ref:2601.06209#pg14>.

Conclusion: Jane: So we’ve seen how this paper used simulated data to test active learning strategies under class imbalance and label shift challenges in image segmentation. Now, let's talk about what that title actually means for someone who just listens to the show.

Tom: The title is "Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation." It basically tells you they built a test bed where they could control exactly how imbalanced the defect class is and how much label shift there is by changing those parameters on their own.

Lu: They used two open-source datasets for this simulation, which are the potato diseases dataset and the severstal dataset. These are the specific sources they used to run their experiments ten.

Meng: The main thing they're pushing is not inventing a brand new way to pick samples, but instead creating a controlled protocol that fixes those imbalance and shift levels while letting you change them independently on your own ten.

Jane: So, what’s the big picture takeaway for someone who just wants to know if this research matters? It shows that when you have a lot of unlabelled data and labelling is expensive, active learning gives you a way to get better models than just picking random samples.

Tom: Exactly. We found that strategies like entropy sampling and core-set selection beat random sampling even when the defect class is very small or there’s some label shift present ten.

Lu: They also quantified exactly how much efficiency loss occurs under a strong label shift, which gives us a real look at those trade-offs between accuracy and speed ten.

Jane: So for the listener, it means that if you're dealing with image data where you know there's an imbalance and some uncertainty about the labels, using an intelligent sampling method instead of just picking randomly will help your model get better results.

Lalam: They demonstrate that the results indicate that active learning remains at least non inferior to random selection and often superior when the deployment prevalence is reasonably aligned ten.

Jane: The core idea is that we can use active learning as a scalable pathway to get better models in areas with substantial class imbalance and plenty of unlabelled data. It’s about making sure you spend your expert time labeling the most useful things first.

Tom: That’s right. We can use active learning as a scalable pathway to get better models in areas with substantial class imbalance and plenty of unlabelled data. But we still need to figure out how to make those sampling strategies work reliably on real traffic at scale, which is what we'll look at next.

Julien Combes, Alexandre Derville, Jean Francois Coeurjolly

Michelin · Laboratoire Jean Kuntzmann

cs.CV, stat.ML

Submitted: 2026-01-08

Updated: 2026-10-21

Code: https://github.com/JulienStats/WhenImbalanceComesTwice

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist: Active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets.

Key concepts

Active Learning (AL): A method to intelligently select the most useful unlabeled data points from a large pool for expert labeling. It saves time and money by focusing labeling efforts on samples that will provide the most information to improve a machine learning model.
Class Imbalance: A situation where some categories in the data are much more frequent than others. In this paper, it refers to defect distribution in images, where most images are 'free' and few contain defects, making it hard for models to learn the rare defect class.
Label Shift: A change in the underlying data distribution between when the model is trained and when it is used for prediction. This occurs here due to limited storage of acquired images, which changes how the model perceives new data points over time.

Terminology

Active Learning — 5× in this paper · explained in 6 episodes
Class Imbalance — 3× in this paper · explained in 3 episodes
Core — 2× in this paper · explained in 2 episodes
Semantic Segmentation — 1× in this paper · explained in 3 episodes

Summary

The gist: Active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets.

Background

Active Learning aims to select informative samples from an unlabelled set of data, which is useful in cases where the amount of data is large and labelling is expensive such as in machine vision or medical imaging<ref:2601.06209#pg2>. Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images<ref:2601.06209#pg2>. This results, on the one hand, in a strong class imbalance in defect distribution and on the other hand, in a potential label shift caused by limited storage<ref:2601.06209#pg2>.

Active Learning Framework

The paper focuses on pool-based AL where we consider two sets of data, a set where the data is labeled and another non-annotated one<ref:2601.06209#pg3>. The pool-based setting ensures to get the most relevant and informative data to train the model<ref:2601.06209#pg4>. The AL proceeds as follows:

At cycle j = 0,..., J, we extract a set of images S u j from the unlabelled dataset, i.e. a set of indices S u j subset of Du with size B called the budget<ref:2601.06209#pg5>.
We form the set S u j = X u i, i in S u j and these images are labeled by the expert and the whole is stored in S l j = (X u i, Y u i), i in S u j<ref:2601.06209#pg5>.
The model is then trained on D l j = D l(j-1) U S l j (with D(j-1) = ∅)<ref:2601.06209#pg5>.
The trained model at cycle j, say ˆf j, is directly used to score/predict every image from Du - U k k=0 S u k<ref:2601.06209#pg5>.
The predicted probabilities or scores (depending on the AL strategy) are then used to extract the next set of images S u j+1 from Du - U k k=0 S u k<ref:2601.06209#pg5>.

Active Learning Strategies Compared

The study compares three standard active learning selection strategies:

Random Sampling, which is the baseline<ref:2601.06209#pg8>.
Entropy Sampling, which computes the Shannon entropy for each predicted pixel and averages them over the full image to get a level of uncertainty for this image<ref:2601.06209#pg9>. This strategy is representative of latent-space exploitation, in which the selection process specifically targets the weaknesses of the model<ref:2601.06209#pg9>.
Core-Set Sampling, which uses the original Euclidean distance with L 2-normalized features obtained from global average pooling of the six multi-scale feature maps produced by the FPN encoder<ref:2601.06209#pg8>. This strategy focuses on exploration of the latent space, aiming to generate images that are highly diverse from one another<ref:2601.06209#pg9>.

Simulation Setting and Results

The simulation study uses two open-source datasets: the potato diseases dataset and the severstal dataset<ref:2601.06209#pg10>. The researchers control the levels of class imbalance by setting π u ranging from 0% to 100% and label shift sensitivity by controlling the difference between π u and π t<ref:2601.06209#pg10>. Key findings include:

Entropy and deep core-set based methods clearly outperform the naive random sampling

"The observed differences, with the naive random sampling method, are larger in the two following situations: (i) first, when datasets contain very few faulty images, i.e. for highly class-imbalanced unlabelled datasets; (ii) second, when the budget is moderate"

Finally, it is worth mentioning that the entropy based AL strategy provides the most stable results

Conclusion

The study demonstrates that AL improves label efficiency and inference performance over random sampling across a range of class-imbalance and label shift regimes, with the most pronounced benefits at moderate to high imbalance and very moderate label shift<ref:2601.06209#pg12>. Beyond F1-scores, AL-trained models exhibit greater stability<ref:2601.06209#pg14>. Overall, AL offers a practical, scalable pathway to robust semantic segmentation in machine vision, particularly in domains characterized by substantial class imbalance and abundant unlabelled data<ref:2601.06209#pg14>. The results indicate that AL remains at least non inferior to random selection and often superior when the deployment prevalence is reasonably aligned<ref:2601.06209#pg14>.

Data Availability

The two datasets used in this paper are open-source at the following URLs: The potato disease dataset can be downloaded here https://universe.roboflow.com/anup-kaygm/potato disease-binb3 and the severstal dataset can be downloaded here https://www.kaggle.com/c/severstal-steel-defect-detection. The code is available on this repository: https://github.com/JulienStats/.

References

The references listed are: 1 Aggarwal, U., Popescu, A., Hudelot, C.: Active Learning for Imbalanced Datasets. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1417–1426 (Mar 2020). https://doi.org/10.1109/WACV45572.2020.9093475 2 Anup: Potato disease Dataset (Jun 2024) 3 Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks 106, 249–259 (Oct 2018). https://doi.org/10.1016/j.neunet.2018.07.011 4 Choi, J., Yi, K.M., Kim, J., Choo, J., Kim, B., Chang, J.Y., Gwon, Y., Chang, H.J.: VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning (Dec 2020). https://doi.org/10.48550/arXiv.2003.11249 5 Elkan, C.: The foundations of cost-sensitive learning. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence - Volume 2. pp. 973–978. IJCAI’01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (Aug 2001) 6 Grishin, A.: Severstal: Steel defect detection (2019), https://kaggle.com/competitions/severstal-steel-defect-detection 7 Jadon, S.: A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). pp. 1–7 (Oct 2020). https://doi.org/10.1109/CIBCB48159. 8 Joshi, A.J., Porikli, F., Papanikolopoulos, N.: Multi-class active learning for image classification. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 2372–2379 (Jun 2009). https://doi.org/10.1109/CVPR.2009.5206627 9 Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (Jan 2017). https://doi.org/10.48550/arXiv.1412.6980 10 Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.

Improvements for AI systems

Adaptive labeling strategy selection: The improved system can dynamically choose between random sampling, entropy-based selection, or core-set selection based on real-time assessment of class imbalance and label shift levels in the unlabelled pool.
Robust performance under severe imbalance: The model will maintain higher F1 scores when facing highly class imbalanced unlabelled datasets compared to naive random sampling by preferentially selecting samples that address the minority defect class.
Resilience to data distribution mismatch: The system will exhibit greater stability and less degradation in performance when there is a significant difference between the unlabelled pool distribution and the test set distribution, such as when πu >> πt, compared to scenarios with label shift.
Enhanced uncertainty exploitation: By utilizing entropy sampling, the system can focus on regions of high uncertainty by computing average image entropy based on predicted probabilities, which is particularly effective at identifying samples containing larger defects.
Latent space diversity optimization: The core-set selection mechanism allows the system to explore the latent space of features by prioritizing highly diverse images, potentially leading to a more comprehensive representation of features within the model's capacity.