MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation

The gist The MUSE benchmark provides a realistic evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design by focusing on complex, editable boundary representation assemblies and assessing functionality, manufacturability, and assemblability rather than just shape matching.

MUSE Benchmark Overview

MUSE introduces a Text-to-CAD benchmark focused on complex, editable boundary representation (B-Rep) assemblies by pairing practical design instances with structured Design Specifications to evaluate generated models through a three-stage protocol. Each design instance is defined by a structured Design Specification which decomposes a high-level design goal into physical assembly graph, valid parameter ranges and a manufacturing plan. To support engineering-grounded evaluation, each design instance is paired with program-generated CAD drawings for visual review. All design instances are carefully reviewed by human designers.

Design Specification and Validation Protocol

MUSE utilizes a systematic, topdown Design Specification S, which consists of design description [10], valid parameter space, a manufacturing plan, and an assembly plan. This specification is defined as S = ⟨D, G, omega,M⟩. A CAD model C is topologically valid with respect to G if Φ(C) ∼= G. Semantic validity requires constraining parameters within a valid parameter space omega to preserve intended design semantics. The evaluation protocol proceeds in three stages, starting with Stage 1: Code Validity, where the generated CadQuery script is executed and checked for successful CAD model construction and STEP file export. Stage 2 involves Geometric Validity checks, including watertightness, manifold topology, self-intersection free verification, and overlap free checks. Stage 3 is Design-Intent Alignment using design-specific rubrics to assess functionality, manufacturability, and assemblability.

Evaluation Rubric Generation

Stage 3 assesses design intent alignment using a rubric-based visual language model (VLM) judge. The construction of the design-specific rubric is done by a Vision-LLM Rubric Generator, which takes input from the Design Specification S, reference engineering views, and expert examples to dynamically write a rigorous 0--1 Pass/Fail evaluation rubric for an AI-generated 3D model. This rubric generates six binary sub-criteria: Assembly readiness, Joint design, Tolerance, Functional adaptation, Usage stability, and Manufacturability. The judge is constrained to use the provided Knowledge Base tables for material selection and manufacturing methods when writing the rubric.

Experimental Findings

Experiments on MUSE reveal a clear failure cascade from executable code to valid geometry and finally to design-intent alignment, showing that performance degrades sharply at each step. Across both closed-source and open-source models, the final scores for functionality, manufacturability, and assemblability remain low even for the strongest models. For instance, Table 2 shows that while some models achieve high Code Check success rates (e.g., GPT-5.5 at 77.36%), the Design Intent Alignment scores drop significantly to around 38–40% across all models.

Model Performance Comparison

Closed-source models generally outperform open-source models across all evaluation stages, with GPT-5.5 achieving the strongest overall performance in the final score metric. Table 3 shows that for closed-source models, the average scores for functionality, manufacturability, and assemblability are around 30–40%. However, even these top performers struggle with fine-grained criteria; they achieve only about 19–21% on functionality and manufacturability. The core challenge identified is that the core challenge is therefore not just code synthesis, but coherent assembly generation under coupled geometric and engineering constraints.

Limitations

MUSE has two current limitations; first, the CAD models in the benchmark have not been physically manufactured, instead they are validated by professional designers which may not capture all real-world manufacturing issues. Second, the evaluation does not yet model the full physical assembly process; for example, the Connectable criterion checks geometric feasibility of joints but not assembly order. Future work includes extending the benchmark to more complex assemblies and incorporating physics-aware evaluation.

The MUSE benchmark provides a realistic evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design by focusing on complex, editable boundary representation assemblies and assessing functionality, manufacturability, and assemblability rather than just shape matching. The gist The MUSE benchmark provides a realistic evaluation framework for advancing Text-to-CAD from geometric generation toward true engineering design by focusing on complex, editable boundary representation assemblies and assessing functionality, manufacturability, and assemblability rather than just shape matching.

How it works

1. A structured Design Specification S = ⟨D, G, omega,M⟩ is defined to decompose a high-level design goal into physical assembly graph, valid parameter ranges and a manufacturing plan.

2. The evaluation follows a three-stage protocol: Stage 1 (Code Validity), Stage 2 (Geometric Validity checks like Watertightness and Self-Intersection Free), and Stage 3 (Design-Intent Alignment using design-specific rubrics).

3. For each stage, the generated model is assessed against criteria derived from the Design Specification S, such as checking if the visual inferred node-edge topology is semantically consistent with the target Component Assembly Graph for Assembly Readiness.

4. The VLM judge is trained to generate rubrics by cross-referencing the Design Specification with knowledge tables for material and manufacturing methods, ensuring that the evaluation criteria are grounded in engineering constraints.

Key Findings on Failure Cascade

The experiments reveal a clear failure cascade from executable code to valid geometry and finally to design intent alignment. This indicates that the core challenge is therefore not just code synthesis, but coherent assembly generation under coupled geometric and engineering constraints.

Model Performance Summary

Closed-source models consistently outperform open-source models across all stages. The final scores for closed-source models are generally higher, with GPT-5.5 achieving the strongest overall performance in the final score metric.

Limitations

MUSE has two current limitations; first, the CAD models in the benchmark have not been physically manufactured, instead they are validated by professional designers which may not capture all real-world manufacturing issues. Second, the evaluation does not yet model the full physical assembly process; for example, the Connectable criterion checks geometric feasibility of joints but not assembly order.

--- Page 1 ---

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation Xiaoyu Dong1 Zhi Li2∗ Xiao-Ming Wu1∗ 1The Hong Kong Polytechnic University 2Curvature Flow Co., Limited Hong Kong SAR Abstract Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design. Existing benchmarks focus primarily on generating single-part CAD models and evaluate them using geometric similarity metrics that fail to capture functionality, manufacturability, and assemblability. To address this gap, we introduce MUSE, a Text-to-CAD benchmark focused on complex, editable boundary representation B-Rep assemblies. MUSE pairs practical design instances with structured Design Specifications and evaluates generated models through a three-stage protocol: code check, geometric check, and design-intent alignment. The final stage uses design-specific rubrics to assess functionality, manufacturability, and assemblability. To enable scalable evaluation, we use a rubric-based visual language model VLM judge and validate its reliability through human annotation. Experiments on closedsource and open-source LLMs reveal a clear failure cascade from executable code to valid geometry and finally to engineering-ready design.

1. Improve Text-to-CAD systems by enforcing a strict failure cascade detection mechanism, explicitly identifying when performance degrades from code executability to valid geometry and finally to design-intent alignment, as revealed in experiments on closed-source and open-source models. This allows for targeted interventions at the specific stage where the model fails, such as improving geometric validity checks or design specification adherence.

2. Implement a rubric-based Visual Language Model (VLM) judge that is specifically trained to align with human annotation across fine-grained sub-criteria, achieving a strongest sub-criteria-level correlation (0.713) with human judges, thereby providing a more reliable automatic evaluation for functionality, manufacturability, and assemblability.

3. Enhance the design specification synthesis process by using the System Prompt CAD Design Specification Generator to force LLMs to strictly adhere to the Reference Tables when selecting Material, Manufacturing Method, and Connection Method based on inferred logic like analyzing boolean operations in CadQuery code. This ensures that generated designs are grounded in real-world engineering constraints rather than hallucinated methods.

4. Develop a three-stage evaluation protocol that mandates sequential progression: Stage 1: Code Validity, Stage 2: Geometric Validity, and Stage 3: Design-Intent Alignment. This prevents models from being rewarded for superficial shape matching when they fail earlier, explicitly addressing the finding that geometric validity does not translate to functional, manufacturable, or assemblable designs.

5. Create a dynamic rubric generation system using the Vision-LLM Rubric Generator prompt template to produce task-specific rubrics that dynamically weight criteria (e.g., 70% for must-have function) based on the Design Specification's goals, ensuring the evaluation moves beyond shape matching toward practical design quality.

Large language models (LLMs) have recently advanced text-driven 3D generation, yet generating CAD models that satisfy practical design requirements remains challenging. Existing benchmarks focus primarily on single-part CAD models and geometric similarity, providing limited assessment of assembly structure and interface geometry. To address this gap, we introduce MUSE, a Text-to-CAD benchmark for generating multi-component, editable boundary representation (B-Rep) assemblies. MUSE pairs practical design instances with structured Design Specifications and evaluates generated outputs across four dimensions: code executability, geometric validity, structural and interface fidelity, and functional consistency. We further introduce Structure and Interface Evaluation (SIE), which combines reference Physical Assembly Graphs with direct B-Rep analysis to assess component correspondence, connectivity, interface types, and interface clearances. Experiments on proprietary and open-weight LLMs show that high execution success does not necessarily translate into correct assembly structure or interface geometry. Together, MUSE provides a benchmark and evaluation framework for studying multi-component Text-to-CAD generation beyond shape similarity. Our project website, including the leaderboard, dataset, and code, is available at.