Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Shape Distribution Matters".

Jane: Here's the gist of the paper. The gist ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework for amodal segmentation that learns a latent shape distribution and dynamically routes objects to tailored experts, leading to consistent performance improvements over state-of-the-art methods.

Tom: Amodal instance segmentation aims to predict complete object masks, covering both visible and occluded regions, which is challenging due to complex occlusions and extreme shape variation Existing approaches typically rely on a single model to predict amodal masks across all object categories and shapes, which often leads to generic predictions due to limited model capacity The authors propose ShapeMoE, a shape-specific sparse Mixture-of-Experts framework tailored for amodal instance segmentation, enabling flexible and efficient learning over highly diverse occluded shapes Shape Distribution Learning The method first designs a shape modeling approach to capture the shape distribution of each object by generating "shapespecific embeddings from its visible mask" which serve as input Assuming that object shapes follow a Gaussian distribution, the Shape Distribution Encoder ES estimates the mean µ and standard deviation σ of this distribution using two separate encoders, Eµ and Eσ The Gaussian parameters (µ and σ) provide a probabilistic representation of shape characteristics, which can be leveraged by the router to effectively guide expert selection Shape-Aware Sparse Routing The Shape-aware Sparse Router R takes the predicted Gaussian parameters µ and σ as input and samples a latent shape representation lo from the learned distribution N (µ, σ2) using a differentiable reparameterization method This latent shape representation is then used to compute expert selection scores s via s = W ∗ lo, where W is a trainable matrix To sparsify the routing, only the top-k selected experts are retained and assigned samples for segmentation, ensuring that "only one expert is activated per sample" Shape-Specialized Segmentation Expert The final component is a shape-specialized segmentation expert designed to handle diverse occlusion patterns in object shapes The authors analyze the decoder architecture and selectively convert the "hyper-network as the core shape-specialized module into multiple expert branches with only marginal additional parameters" This targeted design allows each expert to focus on distinct amodal shape patterns while sharing common visual features, thereby enhancing the model’s capacity to reason about occlusion and capture intrinsic shape diversity Training and Evaluation The overall loss function L comprises two components: the cross-entropy loss LCE used to supervise the amodal mask prediction, and a Coefficient of Variation Squared (CV 2) loss employed to encourage balanced expert selection Experiments on COCOA-cls, KINS, and D2SA show that ShapeMoE consistently outperforms state-of-the-art methods, especially in occluded region segmentation Ablation studies confirm that increasing the number of experts from 1 to 4 leads to consistent performance improvements across all datasets Furthermore, selecting only one expert per sample achieves the best overall performance on COCOA-cls and D2SA datasets The qualitative results demonstrate strong performance across various occlusion scenarios, successfully predicting complete amodal masks in both rigid and complex occluded cases ShapeMoE is a novel shape-specific sparse Mixture-of-Experts framework that explicitly models the latent shape distribution of each object and dynamically assigns it to a specialized lightweight expert via a Shape-Aware Sparse Router Extensive experiments demonstrate that ShapeMoE consistently outperforms state-of-the-art methods across multiple datasets, particularly in accurately segmenting heavily occluded regions The designed shape-aware expert routing paradigm can offer a promising direction for segmentation under complex occlusions and shape diversity.

Jane: Page 1 --- Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions Zhixuan Li1, Yujia Liu2,3,4, Chen Hui5, Jeonghaeng Lee6, Sanghoon Lee6, Weisi Lin18 College of Computing and Data Science, Nanyang Technological University, Singapore 2 School of Computer Science, Peking University, Beijing China 3 National Key Laboratory for Multimedia Information Processing Peking University Beijing China 4 National Engineering Research Center of Visual Technology Peking University Beijing China 5 Nanjing University of Information Science and Technology China 6 Department of Electrical and Electronic Engineering Yonsei University Korea Weisi Lin1 Abstract Amodal segmentation targets to predict complete object masks, covering both visible and occluded regions.

Tom: This task poses significant challenges due to complex occlusions and extreme shape variation, from rigid furniture to highly deformable clothing.

Title and authors: Tom: So we're looking at this paper called Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions. It sounds like they're tackling the problem of predicting whole object masks, covering both what you see and what’s hidden behind things, which is tricky because shapes vary so much.

Jane: That's right. It tackles the challenge of extreme shape variation, from simple objects to really complex ones that are heavily occluded. The authors are Zhixuan Li and the team at Nanyang Technological University and Peking University. It seems like they're proposing a specific way to handle this complexity instead of using one generic model for everything.

Lu: What's interesting about the title is the focus on "Shape Distribution Matters." This suggests they aren't just guessing or trying to patch things up; they are explicitly modeling how different object shapes are distributed in a latent space, which is a big conceptual step.

Meng: I wonder if this means they’re going to build something that’s actually robust when you throw in some weird occlusion, because right now, most of these models just guess and end up with incomplete masks or weird blobs.

Lalam: The paper seems to be proposing a sparse Mixture-of-Experts framework specifically designed to route objects based on their learned shape characteristics. It suggests that instead of one expert guessing everything, you send the object down the right path based on its shape.

The paper's summary: Tom: So, the core idea is this ShapeMoE framework. They first try to capture the shape distribution by generating these "shape-specific embeddings from its visible mask," which they assume follow a Gaussian distribution. Then they use two separate encoders, Emu and Esig, to estimate that mean mu and standard deviation sigma for each object's shape.

Jane: That’s where it gets specific. Instead of just looking at the mask once, they are encoding the shape into these Gaussian parameters mu and sigma. This gives them a probabilistic representation of the object's shape characteristics, which is then used to guide the routing mechanism later on.

Lu: The key mechanism here is that this latent shape space allows for clear discrimination between different shapes. They use a Shape-Aware Sparse Router that takes those predicted parameters mu and sigma and samples a latent shape representation lo from the learned distribution N of (mu, sigma two) <ref:2508.01664#pg1>.

Meng: So, they sample this latent shape representation lo, and then they use a trainable matrix W to compute expert selection scores s equals W times lo. That sounds like the actual routing decision making happening inside the model.

Lalam: And to keep it efficient, they sparsify this routing by only keeping the top-k selected experts for each sample and only activating one expert per sample, which is a sparse mechanism designed to handle diverse shapes efficiently.

The paper's improvements: Tom: The paper points out several key improvements over just using a naive Mixture of Experts. They show that instead of relying on simple softmax-based routing, they introduce this shape-aware routing to get better results for amodal segmentation.

Jane: And the architecture itself is shaped to be efficient. They designed the shape-specialized segmentation expert by converting only the hyper-network component from SAM2 into multiple expert branches with only marginal additional parameters. This means each expert focuses on a specific kind of occlusion pattern while sharing common visual features, which should help it reason about those shapes better.

Lu: They also balanced their training using two loss components. They use the cross-entropy loss for the mask prediction itself, but they also employ a Coefficient of Variation Squared CV2 loss to encourage balanced expert selection during training. This prevents one expert from just dominating and being ignored while others are left unused.

Meng: From a deployment standpoint, that balanced selection is important. If you have too many experts but only one gets used consistently, it’s wasted capacity for inference. The CV2 loss is the mechanism to make sure all the experts get some training attention and are actually useful when they are called upon.

Lalam: And on top of that, experiments on COCOA-cls, KINS, and D2SA show consistent improvements over state-of-the-art methods for occluded region segmentation. They even found that selecting only one expert per sample yields the best overall performance on the COCOA and D2SA datasets.

Conclusion: Tom: So, to wrap up, ShapeMoE introduces a shape-specific sparse Mixture-of-Experts framework where you model the latent shape distribution and dynamically assign objects to specialized experts based on that distribution. It's meant to solve the problem of generic predictions under heavy occlusion by giving each expert a specific job.

Jane: The implication is that for segmentation tasks, especially those with highly varied shapes, you don't need one monolithic model. You can have many specialized experts that are only called when their specific shape pattern matches what’s in front of you.

Lu: I think the explicit modeling of the Gaussian distribution allows them to capture geometric identity in a compact latent space, which is a significant step toward making segmentation more semantically meaningful and less reliant on brute force capacity.

Meng: From my side, it means that while the model has high potential capacity, the sparse routing mechanism keeps the actual computational cost manageable during deployment because you’re only running a few experts per image.

Lalam: ShapeMoE is consistently outperforming state-of-the-art methods across COCOA-cls and D2SA, showing that this shape-aware expert routing paradigm is a promising direction for handling complex occlusions and shape diversity.