Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions

arXiv:2508.01664 · cs.CV · Submitted 2025-08-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Shape Distribution Matters".

Jane: Here's the gist of the paper. The gist ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework for amodal segmentation that learns a latent shape distribution and dynamically routes objects to tailored experts, leading to consistent performance improvements over state-of-the-art methods.

Tom: Amodal instance segmentation aims to predict complete object masks, covering both visible and occluded regions, which is challenging due to complex occlusions and extreme shape variation Existing approaches typically rely on a single model to predict amodal masks across all object categories and shapes, which often leads to generic predictions due to limited model capacity The authors propose ShapeMoE, a shape-specific sparse Mixture-of-Experts framework tailored for amodal instance segmentation, enabling flexible and efficient learning over highly diverse occluded shapes Shape Distribution Learning The method first designs a shape modeling approach to capture the shape distribution of each object by generating "shapespecific embeddings from its visible mask" which serve as input Assuming that object shapes follow a Gaussian distribution, the Shape Distribution Encoder ES estimates the mean µ and standard deviation σ of this distribution using two separate encoders, Eµ and Eσ The Gaussian parameters (µ and σ) provide a probabilistic representation of shape characteristics, which can be leveraged by the router to effectively guide expert selection Shape-Aware Sparse Routing The Shape-aware Sparse Router R takes the predicted Gaussian parameters µ and σ as input and samples a latent shape representation lo from the learned distribution N (µ, σ2) using a differentiable reparameterization method This latent shape representation is then used to compute expert selection scores s via s = W ∗ lo, where W is a trainable matrix To sparsify the routing, only the top-k selected experts are retained and assigned samples for segmentation, ensuring that "only one expert is activated per sample" Shape-Specialized Segmentation Expert The final component is a shape-specialized segmentation expert designed to handle diverse occlusion patterns in object shapes The authors analyze the decoder architecture and selectively convert the "hyper-network as the core shape-specialized module into multiple expert branches with only marginal additional parameters" This targeted design allows each expert to focus on distinct amodal shape patterns while sharing common visual features, thereby enhancing the model’s capacity to reason about occlusion and capture intrinsic shape diversity Training and Evaluation The overall loss function L comprises two components: the cross-entropy loss LCE used to supervise the amodal mask prediction, and a Coefficient of Variation Squared (CV 2) loss employed to encourage balanced expert selection Experiments on COCOA-cls, KINS, and D2SA show that ShapeMoE consistently outperforms state-of-the-art methods, especially in occluded region segmentation Ablation studies confirm that increasing the number of experts from 1 to 4 leads to consistent performance improvements across all datasets Furthermore, selecting only one expert per sample achieves the best overall performance on COCOA-cls and D2SA datasets The qualitative results demonstrate strong performance across various occlusion scenarios, successfully predicting complete amodal masks in both rigid and complex occluded cases ShapeMoE is a novel shape-specific sparse Mixture-of-Experts framework that explicitly models the latent shape distribution of each object and dynamically assigns it to a specialized lightweight expert via a Shape-Aware Sparse Router Extensive experiments demonstrate that ShapeMoE consistently outperforms state-of-the-art methods across multiple datasets, particularly in accurately segmenting heavily occluded regions The designed shape-aware expert routing paradigm can offer a promising direction for segmentation under complex occlusions and shape diversity.

Jane: Page 1 --- Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions Zhixuan Li1, Yujia Liu2,3,4, Chen Hui5, Jeonghaeng Lee6, Sanghoon Lee6, Weisi Lin18 College of Computing and Data Science, Nanyang Technological University, Singapore 2 School of Computer Science, Peking University, Beijing China 3 National Key Laboratory for Multimedia Information Processing Peking University Beijing China 4 National Engineering Research Center of Visual Technology Peking University Beijing China 5 Nanjing University of Information Science and Technology China 6 Department of Electrical and Electronic Engineering Yonsei University Korea Weisi Lin1 Abstract Amodal segmentation targets to predict complete object masks, covering both visible and occluded regions.

Tom: This task poses significant challenges due to complex occlusions and extreme shape variation, from rigid furniture to highly deformable clothing.

Title and authors: Tom: So we're looking at this paper called Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions. It sounds like they're tackling the problem of predicting whole object masks, covering both what you see and what’s hidden behind things, which is tricky because shapes vary so much.

Jane: That's right. It tackles the challenge of extreme shape variation, from simple objects to really complex ones that are heavily occluded. The authors are Zhixuan Li and the team at Nanyang Technological University and Peking University. It seems like they're proposing a specific way to handle this complexity instead of using one generic model for everything.

Lu: What's interesting about the title is the focus on "Shape Distribution Matters." This suggests they aren't just guessing or trying to patch things up; they are explicitly modeling how different object shapes are distributed in a latent space, which is a big conceptual step.

Meng: I wonder if this means they’re going to build something that’s actually robust when you throw in some weird occlusion, because right now, most of these models just guess and end up with incomplete masks or weird blobs.

Lalam: The paper seems to be proposing a sparse Mixture-of-Experts framework specifically designed to route objects based on their learned shape characteristics. It suggests that instead of one expert guessing everything, you send the object down the right path based on its shape.

The paper's summary: Tom: So, the core idea is this ShapeMoE framework. They first try to capture the shape distribution by generating these "shape-specific embeddings from its visible mask," which they assume follow a Gaussian distribution. Then they use two separate encoders, Emu and Esig, to estimate that mean mu and standard deviation sigma for each object's shape.

Jane: That’s where it gets specific. Instead of just looking at the mask once, they are encoding the shape into these Gaussian parameters mu and sigma. This gives them a probabilistic representation of the object's shape characteristics, which is then used to guide the routing mechanism later on.

Lu: The key mechanism here is that this latent shape space allows for clear discrimination between different shapes. They use a Shape-Aware Sparse Router that takes those predicted parameters mu and sigma and samples a latent shape representation lo from the learned distribution N of (mu, sigma two) <ref:2508.01664#pg1>.

Meng: So, they sample this latent shape representation lo, and then they use a trainable matrix W to compute expert selection scores s equals W times lo. That sounds like the actual routing decision making happening inside the model.

Lalam: And to keep it efficient, they sparsify this routing by only keeping the top-k selected experts for each sample and only activating one expert per sample, which is a sparse mechanism designed to handle diverse shapes efficiently.

The paper's improvements: Tom: The paper points out several key improvements over just using a naive Mixture of Experts. They show that instead of relying on simple softmax-based routing, they introduce this shape-aware routing to get better results for amodal segmentation.

Jane: And the architecture itself is shaped to be efficient. They designed the shape-specialized segmentation expert by converting only the hyper-network component from SAM2 into multiple expert branches with only marginal additional parameters. This means each expert focuses on a specific kind of occlusion pattern while sharing common visual features, which should help it reason about those shapes better.

Lu: They also balanced their training using two loss components. They use the cross-entropy loss for the mask prediction itself, but they also employ a Coefficient of Variation Squared CV2 loss to encourage balanced expert selection during training. This prevents one expert from just dominating and being ignored while others are left unused.

Meng: From a deployment standpoint, that balanced selection is important. If you have too many experts but only one gets used consistently, it’s wasted capacity for inference. The CV2 loss is the mechanism to make sure all the experts get some training attention and are actually useful when they are called upon.

Lalam: And on top of that, experiments on COCOA-cls, KINS, and D2SA show consistent improvements over state-of-the-art methods for occluded region segmentation. They even found that selecting only one expert per sample yields the best overall performance on the COCOA and D2SA datasets.

Conclusion: Tom: So, to wrap up, ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework where you model the latent shape distribution and dynamically assign objects to specialized experts based on that distribution. It's meant to solve the problem of generic predictions under heavy occlusion by giving each expert a specific job.

Jane: The implication is that for segmentation tasks, especially those with highly varied shapes, you don't need one monolithic model. You can have many specialized experts that are only called when their specific shape pattern matches what’s in front of you.

Lu: I think the explicit modeling of the Gaussian distribution allows them to capture geometric identity in a compact latent space, which is a significant step toward making segmentation more semantically meaningful and less reliant on brute force capacity.

Meng: From my side, it means that while the model has high potential capacity, the sparse routing mechanism keeps the actual computational cost manageable during deployment because you’re only running a few experts per image.

Lalam: ShapeMoE is consistently outperforming state-of-the-art methods across COCOA-cls and D2SA, showing that this shape-aware expert routing paradigm is a promising direction for handling complex occlusions and shape diversity.

Zhixuan Li, Yujia Liu, Chen Hui, Jeonghaeng Lee, Sanghoon Lee, *Weisi Lin

College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China · National Engineering Research Center of Visual Technology, Peking University, Beijing

cs.CV

Submitted: 2025-08-03

Updated: 2026-10-17

Comments: 10 pages, 4 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework for amodal segmentation that learns a latent shape distribution and dynamically routes objects to tailored experts,

Key concepts

Amodal Instance Segmentation: This task aims to predict the complete mask of an object, covering both visible and hidden parts. It is challenging because object shapes vary widely, from rigid items to highly deformable ones, making it hard to accurately predict occluded regions.
Shape Distribution Learning: The method first learns a probabilistic representation of an object's shape by generating Gaussian embeddings from its visible mask. This captures the key characteristics of the object's shape distribution, which is then used to guide expert selection during routing.
Shape-Aware Sparse Routing: This mechanism uses the learned shape parameters to sample a latent shape representation and calculate expert scores. Only the top-k experts are activated per sample, ensuring that only one specialized expert processes each object while maintaining high efficiency.

Terminology

Interpretability — 1× in this paper · explained in 15 episodes
Mixture-of-Experts (MoE) — 1× in this paper · explained in 10 episodes

Summary

The gist ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework for amodal segmentation that learns a latent shape distribution and dynamically routes objects to tailored experts, leading to consistent performance improvements over state-of-the-art methods.

Introduction and Problem

Amodal instance segmentation aims to predict complete object masks, covering both visible and occluded regions, which is challenging due to complex occlusions and extreme shape variation Existing approaches typically rely on a single model to predict amodal masks across all object categories and shapes, which often leads to generic predictions due to limited model capacity The authors propose ShapeMoE, a shape-specific sparse Mixture-of-Experts framework tailored for amodal instance segmentation, enabling flexible and efficient learning over highly diverse occluded shapes

Shape Distribution Learning

The method first designs a shape modeling approach to capture the shape distribution of each object by generating shapespecific embeddings from its visible mask which serve as input Assuming that object shapes follow a Gaussian distribution, the Shape Distribution Encoder ES estimates the mean µ and standard deviation σ of this distribution using two separate encoders, Eµ and Eσ The Gaussian parameters (µ and σ) provide a probabilistic representation of shape characteristics, which can be leveraged by the router to effectively guide expert selection

Shape-Aware Sparse Routing

The Shape-aware Sparse Router R takes the predicted Gaussian parameters µ and σ as input and samples a latent shape representation lo from the learned distribution N (µ, σ2) using a differentiable reparameterization method This latent shape representation is then used to compute expert selection scores s via s = W ∗ lo, where W is a trainable matrix To sparsify the routing, only the top-k selected experts are retained and assigned samples for segmentation, ensuring that only one expert is activated per sample

Shape-Specialized Segmentation Expert

The final component is a shape-specialized segmentation expert designed to handle diverse occlusion patterns in object shapes The authors analyze the decoder architecture and selectively convert the hyper-network as the core shape-specialized module into multiple expert branches with only marginal additional parameters This targeted design allows each expert to focus on distinct amodal shape patterns while sharing common visual features, thereby enhancing the model’s capacity to reason about occlusion and capture intrinsic shape diversity

Training and Evaluation

The overall loss function L comprises two components: the cross-entropy loss LCE used to supervise the amodal mask prediction, and a Coefficient of Variation Squared (CV 2) loss employed to encourage balanced expert selection Experiments on COCOA-cls, KINS, and D2SA show that ShapeMoE consistently outperforms state-of-the-art methods, especially in occluded region segmentation Ablation studies confirm that increasing the number of experts from 1 to 4 leads to consistent performance improvements across all datasets Furthermore, selecting only one expert per sample achieves the best overall performance on COCOA-cls and D2SA datasets The qualitative results demonstrate strong performance across various occlusion scenarios, successfully predicting complete amodal masks in both rigid and complex occluded cases

Conclusion

ShapeMoE is a novel shape-specific sparse Mixture-of-Experts framework that explicitly models the latent shape distribution of each object and dynamically assigns it to a specialized lightweight expert via a Shape-Aware Sparse Router Extensive experiments demonstrate that ShapeMoE consistently outperforms state-of-the-art methods across multiple datasets, particularly in accurately segmenting heavily occluded regions The designed shape-aware expert routing paradigm can offer a promising direction for segmentation under complex occlusions and shape diversity.

--- Page 1 ---

Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions Zhixuan Li1, Yujia Liu2,3,4, Chen Hui5, Jeonghaeng Lee6, Sanghoon Lee6, Weisi Lin18 College of Computing and Data Science, Nanyang Technological University, Singapore 2 School of Computer Science, Peking University, Beijing China 3 National Key Laboratory for Multimedia Information Processing Peking University Beijing China 4 National Engineering Research Center of Visual Technology Peking University Beijing China 5 Nanjing University of Information Science and Technology China 6 Department of Electrical and Electronic Engineering Yonsei University Korea Weisi Lin1

Abstract Amodal segmentation targets to predict complete object masks, covering both visible and occluded regions. This task poses significant challenges due to complex occlusions and extreme shape variation, from rigid furniture to highly deformable clothing. Existing one-size-fits-all approaches rely on a single model to handle all shape types, struggling to capture and reason about diverse amodal shapes due to limited representation capacity. A natural solution is to adopt a Mixture-of-Experts (MoE) framework, assigning experts to different shape patterns. However, naively applying MoE without considering the object’s underlying shape distribution can lead to mismatched expert routing and insufficient expert specialization, resulting in redundant or underutilized experts. To deal with these issues, we introduce ShapeMoE, a shape-specific sparse Mixture-of-Experts framework for amodal segmentation. The key idea is to learn a latent shape distribution space and dynamically route each object to a lightweight expert tailored to its shape characteristics. Specifically, ShapeMoE encodes each object into a compact Gaussian embedding that captures key shape characteristics. A Shape-Aware Sparse Router then maps the object to the most suitable expert, enabling precise and efficient shape-aware expert routing. Each expert is designed as lightweight and specialized in predicting occluded regions for specific shape patterns. ShapeMoE offers well interpretability via clear shape-to-expert correspondence, while maintaining high capacity and efficiency. Experiments on COCOA-cls, KINS, and D2SA show that ShapeMoE consistently outperforms state-of-the-art methods, especially in occluded region segmentation. The code will be released 1

Introduction Amodal instance segmentation (Li and Malik 2016) aims to predict complete object masks, including occluded regions. A key challenge lies in the wide variation of complete object shapes, making accurate segmentation under occlusion particularly difficult. Specifically, some objects exhibit rigid and regular geometries, while others present highly deformable and irregular structures. Such high variation in shape introduces significant complexity to the learning process. Amodal segmentation has broad real-world applicability (Lee, Jeon, and Le 2023; Follmann et al. 2019; Qi et al. 2019), especially in satellite-based climate monitoring (Awalludin et al. 2024; Muhadi et al. 2020; Hoeser, Bachofer, and Kuenzer 2020) and disaster analysis (Gupta, Watson, and Yin 2021; Pi, Nath, and Behzadan 2021), where occlusions from clouds or shadows often obscure critical surface areas like land, rivers, and infrastructure. Existing approaches (Tai et al. 2025; Zhan et al. 2024; Ozguroglu et al. 2024; Gao et al. 2023; Xiao et al. 2021; Zhan et al. 20: Amodal segmentation has broad real-world applicability (Lee, Jeon, and Le 20: Tai et al. 25 Amodal instance segmentation (Li and Malik 16) aims to predict complete object masks, including occluded regions. A key challenge lies in the wide variation of complete object shapes, making accurate segmentation under occlusion particularly difficult. Specifically, some objects exhibit rigid and regular geometries, while others present highly deformable and irregular structures. Such high variation in shape introduces significant complexity to the learning process. Amodal segmentation has broad real-world applicability (Lee, Jeon, and Le 23; Follmann et al. 19; Qi et al. 19), especially in satellite-based climate monitoring (Awalludin et al. 24; Muhadi et al. 20), and disaster analysis (Gupta, Watson, and Yin 21; Pi, Nath, and Behzadan 21), where occlusions from clouds or shadows often obscure critical surface areas like land, rivers, and infrastructure. Existing approaches (Tai et al. 25; Zhan et al. 24; Ozguroglu et al. 24; Gao et al. 23; Xiao et al. 21; Zhan atal 20): Existing approaches typically rely on a single model to predict amodal masks across all object categories and shapes. However, this one-size-fits-all strategy often leads to generic predictions due to limited model capacity, either failing to predict the precise geometry of occluded regions or generating implausible completions, making it difficult to handle complex occlusions and diverse object shapes accurately. A natural solution to this challenge is to leverage the Mixture of Experts (MoE) (Shazeer et al. 17; Riquelme et al. 21) framework, which utilizes multiple specialized experts to better capture the diverse shape variations of target objects. However, routers in existing MoE frameworks often rely on simple softmax-based gating networks that lack an understanding of fine-grained shape semantics. This limitation results in suboptimal routing, where samples are assigned to inappropriate experts specialized in certain shape patterns. Consequently, such mismatches between samples and experts lead to limited performance in the amodal mask prediction.

Improvements for AI systems

Shape-aware routing enables shape-specific expert assignment by sampling a latent shape representation lo from a learned Gaussian distribution N (mu, sigma2) and computing expert selection scores s = W times lo. This allows for precise and efficient shape-aware expert routing by matching the object's specific geometric characteristics to the most suitable specialized expert.
The system achieves high model capacity while maintaining computational efficiency through a sparse Mixture of Experts mechanism where only one expert is activated per sample. This means that instead of running all experts, only the necessary ones are engaged based on shape-aware routing, reducing inference cost significantly.
Shape-specialized segmentation experts are designed by replicating only the hyper-network component from SAM2 across K expert branches, as this is explicitly tied to shape reasoning and prediction. This targeted design allows each expert to focus on distinct amodal shape patterns while sharing common visual features, enhancing the model's capacity to reason about occlusion.
The loss function L balances mask prediction supervision with Coefficient of Variation Squared (CV2) loss (Shazeer et al. 2017) is employed to encourage balanced expert selection. This ensures that the model maintains uniform utilization of all experts during training, which prevents any single expert from being completely ignored, leading to more stable and effective routing.
The ShapeMoE framework can be used for amodal segmentation tasks—predicting complete object masks covering both visible and occluded regions—across diverse shapes like rigid furniture and highly deformable clothing with superior accuracy over state-of-the-art methods.