MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation

summary

Video file (mp4)

The gist

The gist The MUSE benchmark provides a realistic evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design by focusing on complex, editable boundary

In short

MUSE is a new benchmark for evaluating Text-to-CAD models by testing their ability to create complex, multi-component assemblies rather than just single parts. It pairs practical designs with detailed specifications and checks generated models across three stages: code execution, geometric validity, and design intent alignment using specialized rubrics. Findings show a sharp failure cascade where success in one stage does not guarantee correct assembly structure or manufacturability.

Key concepts

Design Specification (S)
This structured input defines the target design goal. It breaks down a high-level idea into physical components, valid parameter ranges, and a manufacturing plan. It serves as the ground truth that the generated CAD model must satisfy in terms of topology and intended function.
Three-Stage Protocol
Evaluation happens in three sequential steps: first, checking if the generated code runs successfully (Code Validity); second, verifying geometric correctness like watertightness (Geometric Validity); and third, assessing how well the model meets engineering goals using a rubric judge (Design-Intent Alignment).
Rubric-Based VLM Judge
This is an AI tool used in the final stage to score models. It is trained to create specific 'Pass/Fail' criteria for features like manufacturability and joint design by referencing the Design Specification and engineering knowledge tables, ensuring the evaluation is grounded in real-world constraints.
Failure Cascade
Experiments show that performance degrades significantly as you move through the stages. Models that succeed at generating executable code often fail later when trying to achieve correct assembly structure or functional consistency, highlighting that synthesis alone is insufficient for engineering design.

Terminology used across episodes

This episode discusses

The paper

MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation · Read on arXiv

Hong Kong Polytechnic University · Curvature Flow Co., Limited

Large language models (LLMs) have recently advanced text-driven 3D generation, yet generating CAD models that satisfy practical design requirements remains challenging. Existing benchmarks focus primarily on single-part CAD models and geometric similarity, providing limited assessment of assembly structure and interface geometry. To address this gap, we introduce MUSE, a Text-to-CAD benchmark for generating multi-component, editable boundary representation (B-Rep) assemblies. MUSE pairs practical design instances with structured Design Specifications and evaluates generated outputs across four dimensions: code executability, geometric validity, structural and interface fidelity, and functional consistency. We further introduce Structure and Interface Evaluation (SIE), which combines reference Physical Assembly Graphs with direct B-Rep analysis to assess component correspondence, connectivity, interface types, and interface clearances. Experiments on proprietary and open-weight LLMs show that high execution success does not necessarily translate into correct assembly structure or interface geometry. Together, MUSE provides a benchmark and evaluation framework for studying multi-component Text-to-CAD generation beyond shape similarity. Our project website, including the leaderboard, dataset, and code, is available at.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation".

Jane: The gist The MUSE benchmark provides a realistic evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design by focusing on complex,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at the whole MUSE benchmark paper, the authors are pushing Text-to-CAD beyond simple shape matching by focusing on complex B-Rep assemblies and evaluating them against actual engineering needs like functionality and manufacturability.

Jane: They achieved this using a three-stage protocol: code check, geometric check, and then design intent alignment guided by rubrics from a Vision-Language Model judge. The key result is that there's a definite drop in performance as you move through those stages, especially at the final intent alignment step.

Lu: What this implies for the field is that we need to focus on coupling geometric constraints with engineering requirements during generation, because synthesizing code isn't enough anymore.

Meng: From a production standpoint, it suggests that before we deploy Text-to-CAD for complex products, we have to build evaluation frameworks that test manufacturability and assemblability directly, not just visual appeal.

Tom: I think the title of "MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation" captures it well—it’s about making sure the AI understands how parts connect physically.

Jane: It’s a move away from just generating plausible three dee shapes toward generating designs that are actually ready for fabrication and assembly <ref:2605.28579#pg1>. That's where the real engineering value lies.

Lu: The work provides a framework for studying multi-component Text-to-CAD generation by moving beyond shape similarity metrics to assess the actual structure and interface geometry of generated outputs.

Meng: So, the message is that we need better benchmarks that test if an AI can handle the full engineering pipeline, from initial concept to a potentially buildable part.

Conclusion: Tom: So we're wrapping up on the MUSE benchmark paper, which is all about testing Text-to-CAD not just for shapes, but for how things actually fit together in a real assembly.

Jane: It seems like they’ve really narrowed the focus down to that complex B-Rep stuff and pairing it with structured design specs instead of just looking at single parts.

Lu: Yeah, by using those Design Specifications—that whole D, G, Omega, M thing—they’re trying to force the AI to think about the entire manufacturing plan upfront.

Meng: I'm still wondering how robust this is when you take it out of a lab and try to run it on actual production data; these kinds of structured evaluations are nice but don't tell you if it survives scale.

Lalam: My internal literature review confirms that the core idea here is moving beyond just code generation to ensuring the generated geometry respects the engineering rules laid out in that specification.

Tom: Exactly, and the paper shows a clear drop-off in performance from getting executable code to actually having correct geometric validity, which is something I always look for when training models.

Jane: And even after getting decent code and geometry checks, they still struggle with the final part—design intent alignment—which includes things like manufacturability and how well it can actually be put together.

Lu: The authors are essentially saying that the challenge isn't just writing a script; it’s making sure that script produces an assembly structure that makes sense under real-world constraints.

Tom: So for anyone working on Text-to-CAD, this suggests we need to build these kinds of rigorous, multi-stage evaluation frameworks before we can truly trust the AI to handle complex product designs.

Jane: It definitely points toward a future where benchmarks have to test the whole pipeline—from code execution right through to functional consistency in the final assembly.

Meng: If we can get that kind of fidelity across stages, it might actually help us start deploying these models on more critical industrial tasks where errors are expensive.

More episodes

← Home