JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas

arXiv:2603.06168 · cs.CV · Submitted 2026-03-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas".

Jane: Here's the gist of the paper. The gist: JOPP-3D presents an open-vocabulary segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding.

Tom: How it works JOPP-3D converts RGB-D panoramic images into their corresponding wide field-of-view tangential perspectives and 3D point clouds, then uses these modalities to extract and align foundational vision-language features The pipeline involves a joint treatment of panoramas and 3D point clouds in JOPP-3D This allows natural language querying to generate semantic masks on both input modalities.

Jane: The methodology is structured into three major components.

Title and authors: Tom: So we're looking at JOPP-three dee today <ref:2603.06168#pg1>. It’s an open vocabulary segmentation framework that uses both panoramic and point cloud data together to understand scenes guided by language.

Jane: That sounds really ambitious. So, what’s the main idea behind bringing those two different types of data together?

Tom: Basically, they take RGB-D panoramic images and turn them into tangential perspectives along with three dee point clouds <ref:2603.06168#pg1>. Then they use both of those to pull out and align foundational vision-language features. That way you can ask questions in natural language and get semantic masks on both the panorama and the three dee data <ref:2603.06168#pg1>.

Lu: The core idea is treating panoramic images and their three dee point cloud reconstructions as equivalent inputs to the same pipeline, which is what they show in Figure one of JOPP-three dee: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas <ref:2603.06168#pg2> <ref:2603.06168#pg1,JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas>.

Meng: So, if I'm thinking about deployment, what does that mean practically? Are we talking about a massive processing load here?

Jane: Well, the pipeline has three main parts. First is tangential decomposition where they project the panoramic image and depth onto the icosahedron faces to make tangential decompositions and a unified three dee point cloud representation called Pthree dee <ref:2603.06168#pg1>.

Tom: That’s the first step. They set a field-of-view to one hundred degrees in their system, which they say ensures a wide coverage and aspect ratio per tangent view, and this two-view redundancy helps reduce boundary fragmentation.

Lu: Then they move on to three dee instance extraction and semantic alignment where they use pre-trained or unsupervised instance proposal backbones like Maskthree dee or SAMthree dee to extract instance-level three dee masks from Pthree dee <ref:2603.06168#pg1>.

Jane: And then the next step is associating each instance mask, let’s call it Mj, with a corresponding d-dimensional semantic embedding, e three dee j in R d that lets us reason openly about the vocabulary <ref:2603.06168#pg1>.

Tom: They do this by projecting the three dee points in Mj into each tangential perspective image I t using the tangential pose <ref:2603.06168#pg1>. Then they pick K tangential images with the highest number of three dee to 2D matched pixel coordinates, calculated as topk(K)

P uk,vk Mt j(u, v): for each mask Mj <ref:2603.06168#pg1>.

Lu: The final aggregated embedding e three dee j is computed as the normalized average of its top-K CLIP feature vectors <ref:2603.06168#pg1>. This is how they get that semantic representation from the visual data.

Title and authors: Jane: After getting those three dee semantic segmentations, they can project these back into the panoramic image domain to get the segmentation for each pixel <ref:2603.06168#pg1>.

Tom: They do this by transforming depth correspondence by placing the panoramic camera at the same poses as in Eq. (a) and (b). They associate a semantic label based on its nearest three dee semantic neighbor N(X p(u, v)) = arg min((xi,yi,zi)∈Pthree dee X p(u,v)—(xi,yi,zi)two> to each pixel <ref:2603.06168#pg1>.

Meng: So what are the results looking like when you actually test this stuff? Is it just theoretically sound or does it work on real data?

Jane: They evaluated JOPP-three dee on the Stanford-2D-three dee-s and ToF-three hundred sixty datasets, reporting both closed vocabulary mIoU and open vocabulary mIoU <ref:2603.06168#pg1,s and ToF-360 datasets>.

Tom: The quantitative results are pretty strong. On the Stanford-2D-three dee-s dataset, JOPP-three dee(u) achieved a new best performance with eighty point nine percent mIoU and eighty-seven point zero percent mAcc, compared to OpenMaskthree dee thirty-three.

Lu: And for the open vocabulary aspect, JOPP-three dee achieves a new state-of-the-art performance of seventy point one percent mIoU and seventy-four point six percent Open mIoU on that dataset too.

Jane: Similarly, on the ToF-three hundred sixty evaluation dataset, JOPP-three dee(u) shows clear improvements over other models tested there as well <ref:2603.06168#pg1>.

Tom: The synergy between the three dee pipeline and the panoramas is significant because the panoramas provide tangential decomposition for visual feature extraction and enable language-guided semantic alignment <ref:2603.06168#pg1>.

Lu: Conversely, the three dee pipeline synergizes back by producing predictions at native panorama resolution efficiently through semantic label propagation <ref:2603.06168#pg1>. This allows for full-resolution semantic segmentation of panoramas, which are more interpretable and useful for downstream visual applications.

Meng: So it solves the problem of having inconsistent labels across different viewpoints because it maintains a single canonical three dee prediction <ref:2603.06168#pg1>?

Jane: Exactly, that’s one of the big strengths. It allows you to query for things like "ceiling" or "dustbin" and get answers consistently across both input modalities.

Tom: The ablation study confirms that each component contributes substantially to the final performance. Removing the SAM Masking step, for example, leads to poor class distinction because CLIP embeddings are computed directly on crops that often contain multiple objects within large instances.

Lu: And removing depth-based correspondence leads to a consistent loss of performance across all categories compared to the full model. It shows how much the alignment step is crucial.

Jane: So if you’re building something, what would be the biggest hurdle for someone trying to use JOPP-three dee right now <ref:2603.06168#pg1>?

Title and authors: Meng: The paper mentions that one challenge is that some datasets rely on generic labels because annotating capacity is limited. For instance, in Stanford-2Dthree dee-s, objects like wall decorations or dustbins are all lumped into a single "clutter" category <ref:2603.06168#pg1>.

Tom: That makes sense for an open vocabulary approach, but it means the model has to work harder to distinguish those fine details without explicit labeling for them.

Lu: The paper also points out that the quality of instance proposals affects performance most when you compare Maskthree dee with their adaptation SAMthree dee forty-five, yielding moderate gains <ref:2603.06168#pg3>.

Jane: It’s a good warning because it tells us that we can’t just swap out the segmentation backbone and expect a massive jump in performance.

Tom: The authors also noted that for deployment, they have to deal with computational efficiency. They said that per panoramic capture of four thousand ninety-six by two thousand forty-eight pixels, the full pipeline averages around four point eight minutes per image.

Meng: Four point eight minutes sounds slow for a real-time application. What about inference speed once it’s trained?

Jane: For a single query in the three dee domain, they say you need roughly one point seven seconds to retrieve the corresponding semantic masks <ref:2603.06168#pg1>. And when normalizing compute relative to performance gain, the model achieves around zero point five four GPU plus CPU hours per percentage point.

Lu: That’s quite a lot of compute if you're running inference frequently on large panoramas, but it shows the cost of that deep integration.

Tom: The conclusion is that JOPP-three dee introduces a unified framework that bridges panoramic imagery with three dee point clouds, achieving superior performance compared to existing panoramic-only or point cloud-only approaches <ref:2603.06168#pg1>.

Jane: It really shows how powerful combining these modalities can be for scene understanding, and it sets a new bar for what’s possible when we combine language with three dee scenes <ref:2603.06168#pg1>.

Tom: I think the future work they point toward is finding a more reliable two-dimensional to three-dimensional instance lifting strategy that could allow a higher JOPP-three dee(u) performance and move towards complete zero-shot inference <ref:2603.06168#pg1>.

Lu: That’s the next big goal, moving away from relying on pre-trained 2D views for three dee lifting <ref:2603.06168#pg1>.

Meng: From a production standpoint, if we can get that zero-shot inference working smoothly, that would make it much more usable for real applications.

Jane: JOPP-three dee is a really important step toward scene understanding that connects open vocabulary reasoning between panoramic and three dee spaces <ref:2603.06168#pg1>. That’s where we'll leave it for today.

The paper's summary: Tom: So JOPP-three dee is essentially taking two different types of visual data—panoramic images and point clouds—and fusing them to let a language query generate masks on both inputs at the same time.

Jane: That’s the main takeaway, right? It’s not just picking one modality or the other; it’s building this unified framework that treats both panoramas and three dee reconstructions as equal starting points for scene understanding.

Lu: The core mechanism is this joint treatment, where they use the panorama to get tangential views and then align those with the three dee point cloud to extract semantic features that enable open-vocabulary reasoning.

Meng: So if I’m thinking about what this means for deployment, it sounds like a lot of complex processing just to get that initial alignment right.

Jane: It is intensive, but the results show that once you have those aligned features, you can actually get full-resolution segmentation back onto the panoramic image. They use depth correspondence to project the three dee meanings back onto every single pixel in the panorama.

Tom: And here’s a big win: they maintain consistency. Because they anchor everything to a single canonical three dee prediction, you don't get those jarring label changes you sometimes see when you switch between different camera views or point cloud methods.

Lu: The methodology is structured around three parts: tangential decomposition, then instance extraction and semantic alignment using tools like SAMthree dee or Maskthree dee, and finally the three dee to panoramic semantic extraction.

Jane: And that third step is where they use CLIP feature vectors from the top K most matching tangential images to create a robust d-dimensional embedding for every three dee instance. That’s how they achieve that open-vocabulary capability, letting you ask for things like "dustbin" or "wall."

Tom: The quantitative results are what really tell the story here. On Stanford-2D-three dee-s, JOPP-three dee gets eighty point nine percent mIoU and eighty seven point zero percent mean accuracy, which is a new high compared to OpenMaskthree dee <ref:2603.06168#pg1>.

Jane: That seventy point one percent mIoU on the ToF-three hundred sixty dataset is also significant. It shows that this joint approach actually outperforms existing methods when you’re dealing with these complex indoor scenes.

Meng: I see the results, but what about the practical stuff? The paper mentions that for a single query in the three dee domain, they need about one point seven seconds to get those masks. That sounds like it might be too slow for some real-time applications I’m looking at right now.

Tom: It is slow for raw computation, but they normalize the compute cost relative to the performance gain, and they get around zero point five four GPU plus CPU hours per percentage point improvement. That shows the efficiency of the model itself.

Jane: But there are limitations too. The paper points out that a big challenge is when datasets rely on very generic labels, like lumping all small items into one "clutter" category in Stanford-2Dthree dee-s, which means the system has to work harder to distinguish those fine details <ref:2603.06168#pg1>.

Lu: And they also showed that the quality of the instance proposals matters a lot. If you switch from Maskthree dee to their adaptation SAMthree dee, you see moderate gains, which tells us that how well we find those initial three dee masks is a big bottleneck in this system.

Tom: It confirms what we saw in the ablation study: removing the depth-based correspondence kills performance across the board, so that alignment step is essential.

Jane: So JOPP-three dee bridges two major domains—panoramas and three dee point clouds—and shows that this unified approach really opens up a new way to do language guided scene understanding.

Lu: And the future work they suggest is finding a better strategy for lifting instances from 2D to three dee, because right now, they aren't quite at complete zero-shot inference yet <ref:2603.06168#pg1>.

Meng: That’s what I’m interested in—if we can get that reliable lifting strategy, it could move this kind of scene understanding into much more practical settings.

The paper's improvements: Tom: So JOPP-three dee isn't just about getting good numbers; they’re proposing ways to make this framework even better, focusing on making it work without needing massive amounts of training data for every single new scene.

Jane: What does that mean in plain English? It means the authors are trying to find a way for the AI to be more general, so it doesn't just learn about the exact objects it saw in its training set, but understands concepts like "ceiling" or "dustbin."

Lu: They’re looking at improving that two-dimensional to three-dimensional lifting strategy. Right now, they rely on pre-trained 2D views to guess what the three dee structure is, and they want a method that can do that better without needing entirely new training sets for every single new object class <ref:2603.06168#pg1>.

Meng: From a production standpoint, if you can move toward zero-shot inference where the model just gets good at guessing things it has never seen before, that would drastically reduce the need for constant retraining when we deploy updates.

Jane: Exactly. They want to move past just getting high performance on known scenes and towards a system that is truly open-vocabulary, which is what they call zero-shot inference.

Tom: The implication there is huge for application development. It means you could build tools that understand brand new environments or objects without needing custom datasets for every single thing you want it to recognize.

Lu: They are also working on making the instance extraction more robust, specifically addressing those issues we saw in the ablation study where switching back to Maskthree dee gave them moderate gains. They’re trying to make that alignment step much more reliable across different input types.

Jane: So, they aren't just looking at one tweak; they are looking at the whole pipeline—the way they decompose the panorama, how they align the three dee points, and how they generate those semantic embeddings.

Tom: The idea is that by improving each piece individually while keeping them integrated, you get a bigger performance boost than if you just tried to tune one component in isolation.

Meng: It sounds like a lot of engineering work to ensure all these parts talk to each other correctly at scale. What’s the next hurdle for them then?

Lu: The authors also highlighted that they need to find a more reliable way for the 2D-to-three dee instance lifting strategy, because that’s currently where they are feeling the most uncertainty in terms of achieving top performance <ref:2603.06168#pg1>.

Jane: So, the big challenge isn't necessarily the core idea—fusing panoramas and point clouds—but making sure that 2D information accurately translates into reliable three dee geometry without needing a massive retraining effort for every new object <ref:2603.06168#pg1>.

Tom: That’s it. The paper is setting a clear direction: focus on making that lifting strategy more robust so they can move closer to true zero-shot reasoning in the future.

Conclusion: Tom: So to wrap up, JOPP-three dee is this framework that successfully merges panoramic imagery and three dee point clouds to create a unified system for open vocabulary semantic segmentation across both modalities.

Jane: It really shows how powerful it is when you combine visual data from different perspectives, allowing language to query things you couldn't find in just one type of input.

Lu: The implication is that we’re building tools that can understand complex indoor scenes with much greater flexibility than models trained only on standard point clouds or 2D images <ref:2603.06168#pg1>.

Meng: I still think the computational cost is high, but for the level of scene understanding it achieves across both inputs simultaneously, it seems like a necessary trade-off right now.

Lalam: I am confident in this result. The framework’s ability to generate consistent semantic masks across different viewpoints is something I see as very strong evidence of its utility.

Jane: That consistency is a major win for applications, especially when you need reliable scene labels whether the input is a panorama or a point cloud.

Tom: Yeah, and they proved that this joint approach beats both panoramic-only and point cloud-only methods on standard benchmarks like Stanford-2Dthree dee-s <ref:2603.06168#pg1>.

Lu: It pushes the boundary on multimodal integration by showing that the synergy between tangential decomposition and three dee instance extraction yields solid results.

Meng: The caveat remains that the performance is heavily dependent on how good those initial instance proposals are; if you use a weaker model, you see a noticeable drop in quality.

Jane: So, JOPP-three dee gives us a very high-resolution semantic map of entire panoramic scenes by leveraging that three dee alignment strategy.

Tom: It’s a solid piece of engineering because it tackles the problem of inconsistent labels by forcing everything to agree on one canonical three dee prediction first.

Lu: And looking ahead, the authors are focused on improving that two-dimensional to three-dimensional instance lifting strategy so they can actually aim for complete zero-shot inference.

Meng: If they nail that lifting, it opens up a lot more use cases for deploying these kinds of scene understanding models in real environments.

Lalam: I agree with the direction. The focus on that lifting mechanism is where the most significant potential for open vocabulary reasoning lies within this work.

Jane: That’s a great point, and that sets the stage well for what we can expect from these kinds of multimodal scene understanding tools next.

Sandeep Inuganti, Hideaki Kanayama, Kanta Shimizu, Mahdi Chamseddine, Soichiro Yokota, Didier Stricker, Jason Rambach

German Research Center for Artificial Intelligence, DFKI, Germany · Ricoh Company, Ltd. Japan · Ricoh International B.V. - Niederlassung Deutschland

cs.CV

Submitted: 2026-03-06

Updated: 2026-10-20

Comments: Accepted at WACV 2027

Code: https://github.com/dfki-av/JOPP-3D

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 80/100

The gist: The gist: JOPP-3D presents an open-vocabulary segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding.

Key concepts

Tangential Decomposition: This process projects a panoramic image and its depth onto the faces of an icosahedron to create tangential views. By setting the field-of-view to 100 degrees, this method generates a unified 3D point cloud representation (P3D). This redundancy helps reduce boundary fragmentation in the resulting scene geometry.
3D Instance Extraction and Semantic Alignment: This step uses models like Mask3D to find individual 3D masks from the P3D data. Each instance mask is then associated with a semantic embedding by projecting its points back into tangential perspective images. The final semantic embedding is an average of top-K CLIP feature vectors, enabling open-vocabulary reasoning.
3D to Panoramic Semantic Extraction: This component projects the 3D semantic segmentation results back onto the panoramic image domain. It uses depth correspondence to determine a pixel's label by finding its nearest 3D semantic neighbor in the P3D cloud. This ensures full-resolution semantic segmentation of panoramas.
Open-Vocabulary Reasoning: The framework enables open-vocabulary reasoning by associating 3D instance masks with d-dimensional semantic embeddings. These embeddings allow the system to understand and generate masks for objects it hasn't been explicitly trained on, such as specific structures or rare classes, by leveraging language guidance.

Terminology

Point Clouds — 3× in this paper · explained in 2 episodes
Semantic Segmentation — 3× in this paper · explained in 3 episodes
Adaptability — 1× in this paper · explained in 1 episode
Point Cloud Data — 1× in this paper · explained in 2 episodes

Summary

The gist: JOPP-3D presents an open-vocabulary segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding.

How it works

JOPP-3D converts RGB-D panoramic images into their corresponding wide field-of-view tangential perspectives and 3D point clouds, then uses these modalities to extract and align foundational vision-language features The pipeline involves a joint treatment of panoramas and 3D point clouds in JOPP-3D This allows natural language querying to generate semantic masks on both input modalities.

The methodology is structured into three major components.

Tangential Decomposition: This process projects the panoramic image and depth onto the icosahedron faces to generate tangential decompositions and a unified 3D point cloud representation, P3D The field-of-view is set to 100◦ in our system to ensure a sufficiently wide coverage & aspect ratio per tangent view This two-view redundancy reduces boundary fragmentation.
3D Instance Extraction and Semantic Alignment: This step uses pre-trained or unsupervised instance proposal backbones, such as Mask3D or SAM3D, to extract instance-level 3D masks from P3D The next step is to associate each instance mask Mj with a corresponding d-dimensional semantic embedding e 3D j in R d that enables open-vocabulary reasoning. This is achieved by projecting the 3D points in Mj into each tangential perspective image I t utilizing the tangential pose We then choose K-tangential images with the highest number of 3D to 2D matched pixel-coordinates by computing topk(K)[P uk,vk Mt j(u, v)] for each 3D instance mask Mj. The final aggregated embedding e 3D j is computed as the normalized average of its top-K CLIP feature vectors.
3D to Panoramic Semantic Extraction: Once the 3D semantic segmentation is obtained, it is possible to project these semantics back into the panoramic image domain. This involves transforming depth correspondence by placing the panoramic camera at the same poses as in Eq. (a) and (b) We associate a semantic label based on its nearest 3D semantic neighbor N(X p(u, v)) = arg min((xi,yi,zi)∈P3D X p(u,v)—(xi,yi,zi)2> to each pixel.

Evaluation and Results

The framework is evaluated on the Stanford-2D-3D-s and ToF-360 datasets. The performance is reported using both closed-vocabulary mIoU (standard mIoU) and open-vocabulary mIoU (Open mIoU) For 3D semantic segmentation, the standard 3D mIoU and mean Accuracy (mAcc) are reported.

Quantitative results demonstrate superior performance compared to existing methods. For instance, on the Stanford-2D-3D-s dataset, JOPP-3D(u) achieves a new best performance with 80.9% mIoU and 87.0% mAcc In comparison to OpenMask3D [33], JOPP-3D achieves a new state-of-the-art performance: 70.1% mIoU and 74.6% Open mIoU. Similarly, on the ToF-360 evaluation dataset, JOPP-3D(u) shows clear improvements over other models.

Synergy and Strengths

The synergy between the 3D pipeline and panoramas is significant. Panoramas provide tangential decomposition for visual feature extraction and enable language-guided semantic alignment. Conversely, the 3D pipeline synergizes back by producing predictions at native panorama resolution efficiently through semantic label propagation. This allows for full-resolution semantic segmentation of panoramas, which are more interpretable and useful for downstream visual applications. Furthermore, the joint pipeline solves the problem of inconsistent labels by maintaining a single canonical 3D prediction across different viewpoints.

Ablation Insights

The ablation study confirms that each component contributes substantially to the final performance. Removing the SAM Masking step (w/o SAM Mask) leads to poor class distinction because CLIP embeddings are computed directly on crops that often contain multiple objects within large instances. Using full panoramic images instead of tangentially decomposed perspective views (w/o Tgt Decomp.) also degrades performance due to spherical distortions hindering pertinent CLIP embeddings. Replacing Mask3D [29] with our SAM3D [45] adaptation yields moderate gains, showing the limitations of JOPP-3D where the quality of instance proposals affects performance most. Finally, removing depth-based correspondence (w/o Depth Corr.) leads to a consistent loss of performance across all categories compared to the full model.

Conclusion and Future Work

JOPP-3D introduces a unified framework that bridges panoramic imagery with 3D point clouds. The work demonstrates that JOPP-3D achieves superior performance compared to existing panoramic-only or point cloud-only semantic segmentation approaches while uniquely performing both tasks jointly. A clear future work is to find a more reliable 2D→3D instance-lifting strategy that can allow a higher JOPP-3D(u) performance and move towards complete zero-shot inference. The model represents an important step towards scene understanding that connects open-vocabulary reasoning between panoramic and 3D spaces.

A. Implementation details for SAM3D

The original SAM3D [45] implementation assumes an input of RGB-D image sequences, where continuous merging is performed by pairing two adjacent frames with sufficient overlap. In contrast, our input consists of discretely captured panoramic RGB-D images, which do not guarantee strong overlap between any two panoramas. To address this, we treat all tangential perspectives generated from a single panorama as an adjacent group and perform sequential merging within this group. Both local and global merging follow an overlap-based criterion: two instance masks are merged if their intersectionover-union (IoU) exceeds a predefined threshold.

B. Additional qualitative analysis

A key challenge during the evaluation of JOPP-3D arises from a dataset’s reliance on generic labels due to limitations in annotating capacity. For example, in the Stanford-2D3D-s [2] dataset, a large variety of semantically meaningful objects – including wall decorations (e.g. clocks, posters), dustbins, and other small furniture are all collapsed into a single “clutter” category. This poses a disadvantage for an open-vocabulary approach such as JOPP-3D. In Table B.1, we provide rare-class retrieval on a total 922 clutter instances in S3DIS [1] The top rare-classes able to be retrieved by JOPP-3D are listed in Table B.1.

C. Computational Efficiency

Quantifying computational efficiency for JOPP-3D presents a unique scenario, as our method is entirely training-free. The majority of JOPP-3D’s compute cost arises from two inference components: (1) our language-guided semantic alignment pipeline, and (2) the depth-correspondence mechanism used for consistent 3D-to-panoramic semantic transfer. Per panoramic capture of 4096 × 2048 pixels, our full pipeline averages around 4.8 minutes / image. For a single query, we require roughly 1.7 seconds to retrieve the corresponding semantic masks in the 3D domain. The model achieves ≈ 0.54 GPU+CPU hours per percentage point when normalizing compute relative to performance gain.

References

[1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534–1543, 2016.

[2] Iro Armeni, Sasha Sax, Amir R.

Improvements for AI systems

Immediate improvement of scene understanding via JOPP-3D enables natural language querying to generate semantic masks on both input modalities, allowing users to query for arbitrary objects like ceiling or dustbin in both panoramic and 3D data simultaneously.
Development of a unified framework that bridges 2D-3D understanding by constructing 3D point clouds directly from panoramic images and enabling consistent semantic prediction across both modalities.
Implementation of a geometrically grounded panoramic decomposition that utilizes a polyhedral projection [31] that approximates the sphere using a regular polyhedron and project the panorama onto its F faces, which results in a wider field-of-view of 100 degrees with two-view redundancy reduces boundary fragmentation.
Enhanced open-vocabulary retrieval by utilizing instance embeddings derived from aggregated CLIP features, ensuring that the system can perform open-vocabulary reasoning and retrieve large structures like walls and floors, that instance-centric formulations struggle with.
Achieving full-resolution semantic segmentation of panoramas by using a depth correspondence strategy to propagate semantics back into the panoramic domain, overcoming the limitation where current methods report results at lower resolutions (e.g., 1024 × 512).
Ensuring cross-panorama consistency by using a depth-correspondence strategy that leverages overlapping depth regions between neighboring panoramic scenes to propagate semantic labels, thus maintaining scenelevel consistency in panoramic segmentation.

Abstract

Semantic segmentation across visual modalities such as 3D point clouds and panoramic images remains a challenging task, primarily due to the scarcity of annotated data and the limited adaptability of fixed-label models. In this paper, we present JOPP-3D, an open-vocabulary segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding. We convert RGB-D panoramic images into their corresponding wide field-of-view tangential perspectives and 3D point clouds, then use these modalities to extract and align foundational vision-language features. This allows natural language querying to generate semantic masks on both input modalities. Experimental evaluation on the Stanford-2D-3D-s and ToF-360 datasets demonstrates the capability of JOPP-3D to produce coherent and semantically meaningful segmentations across panoramic and 3D domains. Our proposed method achieves a significant improvement compared to the SOTA in open and closed vocabulary 2D and 3D semantic segmentation. Code: github.com/dfki-av/JOPP-3D