Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling

arXiv:2604.11007 · cs.CV · Submitted 2026-04-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling".

Jane: Here's the gist of the paper. The gist The proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an Open-Vocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the lack of training data.

Tom: Semantic segmentation of 3D point cloud scenes is a crucial task for various applications, including autonomous driving, robotics, augmented reality, and facility maintenance <ref:2604.11007#pg2>.

Jane: Existing data-efficient algorithms typically address only one or two of these challenges <ref:2604.11007#pg3>.

Tom: The paper proposes a data-efficient training framework specifically designed to address the three forms of data insufficiency: scarcity of scenes, scarcity of pointlevel annotations, and absence of 2D image sequences <ref:2604.11007#pg4>.

Title and authors: Tom: Let’s talk about what PLOVIS actually does in this paper. The authors introduce their specific method, Point pseudo-Labeling via Open-Vocabulary Image Segmentation-based Pseudo-Labeling, to solve that data insufficiency issue we just talked about.

Jane: So, they propose using an OpenVocabulary Image Segmentation model as a way to create these pseudo labels because they compensate for the lack of training data. It sounds like they are trying to bridge the gap between having three dee data and having those 2D images that segmentation models usually need <ref:2604.11007#pg1>.

Lu: They do this by creating 2D images directly from the training three dee point clouds, which actually gets rid of the need for having those separate, large sets of pre-existing 2D image sequences <ref:2604.11007#pg1>. That’s a big part of their data efficiency claim.

Meng: Creating those images from scratch sounds like it could be computationally heavy if you have to do it for every single scene you want to train on at scale. What about the cost of that rendering process?

Lalam: The process involves several steps: first, they use an OpenVocabulary Image Segmentation model as a pseudo label generator, then they render the point cloud into 2D images and use pixel-level zero-shot classification with that OVIS model to get those initial labels <ref:2604.11007#pg1,model as a pseudo label generator>.

Tom: And then they take each of those generated pseudo labels and pair them with a feature extracted by a pretrained Point Cloud Foundation model, which acts as an augmented training data source for their segmentation head.

The paper's summary: Jane: So, to summarize what they’re doing: PLOVIS takes three dee point clouds, turns them into 2D images on the fly using this OVIS model, gets pixel-level predictions from that model as pseudo labels, and then uses those pairs with pre-trained point cloud features for training <ref:2604.11007#pg1>.

Tom: The goal here is to get a lightweight segmentation head to train effectively without needing millions of perfectly annotated three dee scenes or millions of perfect point labels <ref:2604.11007#pg1>. They are trying to make the system flexible too, because they built it on top of independently pretrained PCF and OVIS models, meaning you could swap those out for stronger ones later.

Lu: What’s interesting from my side is that this modular design means it can potentially handle different kinds of point cloud data well because the OpenVocabulary Image Segmentation model has strong zero-shot capability. It’s not locked into one specific dataset structure.

Meng: That flexibility is good for research, but for me, the real question is how reliable those initial 2D renderings are when you’re dealing with real-world sensor noise or complex geometry <ref:2604.11007#pg1>. If the rendering introduces artifacts, those pseudo labels will be garbage.

Lalam: To make sure the training isn't just learning from noise, they introduce a two-stage filtering mechanism to clean up those initial pseudo labels before they even hit the main training loop.

The paper's improvements: Tom: Now let’s look at how they improve the process beyond just using OVIS as a label generator. They suggest a couple of key improvements in this framework.

Jane: The first major improvement is that they introduce a two-stage filtering mechanism to clean up the pseudo labels. They filter by confidence first, using the entropy of the pseudo label vector to remove things that seem uncertain or flat.

Lu: And then there’s a second stage of filtering based on loss values, where they compute training loss and use a two-component one-dimensional Gaussian Mixture Model clustering on those loss values to remove points that look confident but are likely wrong.

Meng: Filtering by confidence sounds like a good way to prune the bad guesses early on, which helps with the computational load when scaling up the pseudo label generation part. But what about that second stage? How do you define what makes a loss value indicative of an incorrect point?

Lalam: The authors say that this filtering removes noisy pseudo labels and results in a set denoted as D, which is then used for training the lightweight segmentation head. They are using these cleaned pairs to train against both human-annotated ground truth and the OVIS-based pseudo labels.

Conclusion: Tom: So, wrapping up on the findings of this paper, PLOVIS consistently outperforms existing methods when we look at mean Intersection over Union, or mIoU, and mean Accuracy across four different benchmark datasets like ScanNet and Sthree deeIS.

Jane: That’s a strong result because it shows that their approach to data-efficient training actually works better than standard fine-tuning strategies or even some of the state-of-the-art weakly supervised learning algorithms we discussed earlier.

Lu: The paper’s main contribution, as I see it, is proving that leveraging OVIS models for pseudo labels is an effective strategy for three dee point cloud segmentation when you are dealing with scarce training scenes and annotations <ref:2604.11007#pg1>.

Meng: For deployment at scale, the filtering mechanism and the way they balance true labels against those pseudo labels using the Class-wise Memory Bank seems like a solid setup, but I’d still need to see how stable that performance holds up when we move it off these controlled benchmarks onto real traffic data.

Lalam: I agree with Meng on the deployment aspect; while the filtering improves quality, they do acknowledge that they are focusing on closed-set semantic segmentation of indoor and outdoor three dee point clouds right now, meaning extension to different tasks or sensor types is something for future work <ref:2604.11007#pg1>.

Tom: So, in summary, PLOVIS shows that using OVIS-based pseudo labels is a viable path toward data-efficient training for three dee point clouds under scarcity conditions <ref:2604.11007#pg1>. That’s what we have here today with this paper on Data-Efficient Semantic Segmentation of three dee Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling <ref:2604.11007#pg1,Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image>.

Takahiko Furuya

University of Yamanashi

cs.CV

Submitted: 2026-04-13

Updated: 2026-10-21

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 73/100

The gist: The gist The proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an Open-Vocabulary Image Segmentation (OVIS) model as a pseudo label generator

Key concepts

Open-Vocabulary Image Segmentation (OVIS) Model: This model acts as a powerful pseudo label generator. It takes a 2D image derived from the 3D point cloud and uses text prompts to classify every pixel in that image, even for classes it wasn't explicitly trained on. This allows the system to create labels for many different categories without needing specific training data for each class.
Pseudo Label Generation: This is the process of creating temporary, estimated labels from unlabeled data. PLOVIS achieves this by rendering 3D points into 2D images and then feeding these images into the OVIS model to get pixel-wise predictions. These predictions serve as initial training signals for the segmentation model.
Two-Stage Pseudo Label Filtering: This mechanism cleans up noisy or incorrect pseudo labels. The first stage filters out uncertain labels by checking their confidence (low entropy). The second stage filters out confident but likely wrong points by analyzing the training loss, ensuring only reliable data is used for final training.
Class-wise Memory Bank (CMB): The CMB is a storage system that keeps diverse pairs of features and pseudo labels. When training, pairs are sampled from this bank in a class-balanced way. This ensures that the segmentation model learns effectively from rare classes, preventing bias toward more common classes when true annotations are scarce.

Terminology

Point Clouds — 3× in this paper · explained in 2 episodes
Pseudo-labeling — 3× in this paper · explained in 3 episodes
Semantic Segmentation — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Class Imbalance — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Generator — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an Open-Vocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the lack of training data.

Motivation and Problem Addressed

Semantic segmentation of 3D point cloud scenes is a crucial task for various applications, including autonomous driving, robotics, augmented reality, and facility maintenance <ref:2604.11007#pg2>. Existing data-efficient algorithms typically address only one or two of these challenges <ref:2604.11007#pg3>. The paper proposes a data-efficient training framework specifically designed to address the three forms of data insufficiency: scarcity of scenes, scarcity of pointlevel annotations, and absence of 2D image sequences <ref:2604.11007#pg4>. Training segmentation models often confronts three forms of data insufficiency in real-world application scenarios: Scarcity of scenes, Scarcity of annotations, and Absence of image sequences <ref:2604.11007#pg5>.

Proposed Algorithm: PLOVIS

The proposed algorithm is called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), which employs an OpenVocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the insufficiency of training data <ref:2604.11007#pg3>. PLOVIS creates 2D images for pseudo-labeling directly from training 3D point clouds, eliminating the need for 2D image sequences <ref:2604.11007#pg4>. The algorithm involves several key components:

Leveraging an OpenVocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the insufficiency of training data <ref:2604.11007#pg3>.
Creating 2D images for pseudo-labeling by rendering the point cloud into 2D images followed by pixel-level zero-shot classification using the OVIS model <ref:2604.11007#pg4>.
Pairing each pseudo label with a pointwise feature extracted by a pretrained Point Cloud Foundation (PCF) model [4], and the resulting pairs serve as augmented training data <ref:2604.11007#pg3>.
Introducing a two-stage filtering of pseudo labels combined with a class-balanced memory bank for effective training <ref:2604.11007#pg3>.

Pseudo Label Generation and Filtering

The process begins by rendering the 3D point cloud into a 2D image, where the camera position and field of view are randomly selected <ref:2604.11007#pg4>. The resulting pixelated, colored 2D image is then input to the OVIS model along with candidate class text prompts to produce a 3D tensor containing pixel-wise logits <ref:2604.11007#pg5>. These logits are converted into a pseudo label vector by using a temperature-scaled softmax function <ref:2604.11007#pg5>. The initial set of training pairs is defined as D! = (f, Gy) where M is the number of 3D points labeled by the OVIS model <ref:2604.11007#pg5>.

The two-stage filtering mechanism refines these initial pairs to remove noisy pseudo labels <ref:2604.11007#pg5>.

First stage: Filtering by confidence, which removes uncertain pseudo labels characterized by flat class probability distributions using the entropy of the pseudo label vector <ref:2604.11007#pg9>.
Second stage: Filtering by loss, which removes confident yet likely incorrect pseudo-labeled 3D points by computing training loss and applying a two-component 1D Gaussian Mixture Model (GMM) clustering on the loss values <ref:2604.11007#pg10>. The resulting set of cleaned pairs is denoted as D.

Training Strategy

The lightweight segmentation head is jointly trained using two types of supervision signals: human-annotated ground-truth labels (true labels) and OVIS-based pseudo labels <ref:2604.11007#pg13>.

Loss using true labels (L:/;9) is computed between the pointwise prediction y7 and the true label y, where only a few dozen 3D points are annotated per scene <ref:2604.11007#pg16>.
Loss using pseudo labels (L=49;?) is computed between the pointwise prediction y7 and the pseudo label vector yD, sampled from a Class-wise Memory Bank (CMB) <ref:2604.11007#pg16>.

The overall loss is defined as L = wL:/;9 + (1 − w)L=49;?, with a hyperparameter w balancing the two loss terms, set to 0.5 in this paper <ref:2604.11007#pg16>. The Class-wise Memory Bank (CMB) stores diverse pairs of pointwise features and pseudo labels, and training pairs are sampled from it in a class-balanced manner to compensate for the scarcity of true labels while mitigating training bias toward majority classes <ref:2604.11007#pg13>.

Experimental Results

Experiments on four benchmark datasets—ScanNet, S3DIS, Toronto3D, and Semantic3D—under realistic data-scarce conditions demonstrate that PLOVIS consistently outperforms existing methods including standard fine-tuning strategies and state-of-the-art weakly supervised learning algorithms <ref:2604.11007#pg6>.

Table 1 summarizes the segmentation accuracy of PLOVIS and the six competing algorithms across four datasets, showing that PLOVIS consistently achieves the highest mIoU and mAcc on all datasets <ref:2604.11007#pg13>. The key advantage of PLOVIS is its ability to generate a large number of diverse pseudo labels via the OVIS model and use them as supplementary supervision signals <ref:2604.11007#pg13>. Furthermore, the two-stage filtering mechanism improves the quality of pseudo labels by removing uncertain and potentially incorrect labels, thereby providing cleaner supervision <ref:2604.11007#pg16>. The class-wise evaluation reveals that PLOVIS achieves substantial improvements on minority classes, indicating successful pseudo label generation by the OVIS model and class-balanced training with CMB <ref:2604.11007#pg13>.

Conclusion

PLOVIS consistently outperforms all competing algorithms when evaluated in mIoU and mAcc, confirming the effectiveness of its approach to data-efficient training <ref:2604.11007#pg13>. The work demonstrates that leveraging OVIS-based pseudo labels is an effective strategy for data-efficient 3D point cloud segmentation under the condition where training scenes and annotations are scarce <ref:2604.11007#pg13>. Future work should explore restoring renderings into more photorealistic images using image generation models prior to OVIS inference and systematic prompt engineering to boost the accuracy of zero-shot pseudo-labeling by the OVIS model <ref:2604.11007#pg17>. The current PLOVIS framework focuses on closed-set semantic segmentation of indoor and outdoor 3D point clouds, with potential for extension to other tasks and sensor modalities <ref:2604.11007#pg17>.

--- Page 2604.

Improvements for AI systems

Improve data efficiency by training segmentation models on a few dozen sparsely annotated 3D point cloud scenes, rather than requiring large-scale, densely annotated datasets as in existing methods like full fine-tuning. This allows for effective training under the condition of scarcity of scenes while maintaining high accuracy.
Implement a two-stage pseudo label filtering mechanism to remove noisy pseudo labels by first filtering based on confidence (low entropy) and then based on loss values, which are estimated to be likely incorrect. This process ensures that the training set contains only reliable feature-label pairs, leading to cleaner supervision for the segmentation head.
Utilize a class-balanced memory bank to store diverse feature-label pairs sampled in a way that compensates for class imbalance in pseudo labels. This strategy mitigates the training bias toward majority classes by ensuring that minority classes receive sufficient attention during training, as described by the paper's goal to mitigat[e] the training bias toward majority classes.
Enhance zero-shot learning capability by systematically improving prompt engineering for the OpenVocabulary Image Segmentation (OVIS) model. This involves incorporating contextual descriptions, attribute-rich class definitions, or learned prompt optimization to boost the accuracy of pseudo-labeling when using simple text prompts.
Extend the framework's applicability by applying it to diverse tasks and modalities beyond closed-set semantic segmentation, such as instance and panoptic segmentation, and point clouds acquired from different sensor modalities like LiDAR and RGB-D.

Abstract

Semantic segmentation of 3D point cloud scenes is a crucial task for various applications. In real-world scenarios, training segmentation models often faces three concurrent forms of data insufficiency: scarcity of training scenes, scarcity of point-level annotations, and absence of 2D image sequences from which point clouds were reconstructed. Existing data-efficient algorithms typically address only one or two of these challenges, leaving the joint treatment of all three unexplored. This paper proposes a data-efficient training framework specifically designed to address the three forms of data insufficiency. Our proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an open-vocabulary image segmentation model as a pseudo label generator to compensate for the lack of training data. PLOVIS creates 2D images for pseudo-labeling directly from training 3D point clouds, eliminating the need for 2D image sequences. To mitigate the inherent noise and class imbalance in pseudo labels, we introduce a two-stage filtering of pseudo labels combined with a class-balanced memory bank for effective training. The two-stage filtering mechanism first removes low-confidence pseudo labels, then discards likely incorrect pseudo labels, thereby enhancing the quality of pseudo labels. Experiments on four benchmark datasets, i.e., ScanNet, S3DIS, Toronto3D, and Semantic3D, under realistic data-scarce conditions (a few tens of training 3D scenes, each annotated with only <100 3D points) demonstrate that PLOVIS consistently outperforms existing methods including standard fine-tuning strategies and state-of-the-art weakly supervised learning algorithms. Code will be made publicly available.