Edit4D: Editable 4D Scene Generation with Textual Control

The gist Edit4D introduces an editable text-to-4D scene generation framework that constructs 4D scenes from separate background and dynamic character representations, enabling flexible object-level editing and controllable camera motion.

How it works

Edit4D operates through a stage-specific generation pipeline where a user instruction is decomposed into prompts for the background scene, dynamic character, and composition text using a large language model The framework then generates these components in parallel: A background scene is generated as a 3D Gaussian representation using text- and camera-conditioned priors and it then constructs a dynamic 3D character via video-guided mesh reconstruction and Gaussian binding The final stage involves composing these components through text-guided segmentation and contact-aware optimization to obtain an editable 4D scene.

Editable 4D Scene Representation

The paper represents a 4D scene using Gaussian splatting defined by parameters G = (µi, ri, si, αi, ci) The goal is to generate a time-varying Gaussian representation conditioned on a user instruction by defining G(t) = Gbg ⊕ (M Nobj i=1 Gobj i(t)). This component-wise formulation naturally supports object-level editing, such as modifying the appearance, motion, or spatial placement of individual objects and allows for novel-view rendering via Ick,t = R(G(t), ck).

Stage 3: Scene–Character Composition

The composition stage automates the placement of the dynamic character into the background scene using a text-guided approach. This involves obtaining a 3D Gaussian mask M3D by selecting Gaussians whose projections fall within a 2D segmentation mask M2D derived from text and filtering candidates based on depth map Dbg to ensure spatial localization. To achieve physically plausible placement, the method introduces two complementary losses: a penetration loss Lpenetration which penalizes background scene points located inside the character mesh and a nearest-neighbor contact loss Lcontact which restrict[s] the optimization to a down-sampled subset S consisting of the top α% mesh vertices xM that are closest to the target object along the y-axis.

Global Scene Refinement

A global scene refinement is performed to improve appearance harmonization and rendering consistency by jointly optimizing the background scene, canonical character Gaussians, and the deformation network This refinement involves optimizing both latent-space and image-space consistency terms Lrefine = Et,ck,τ,ϵh γ(τ) λ1zck,t − zˆck,t2 +λ2Ick,t − ˆIck,t2. Furthermore, the method estimates cast shadows based on estimated scene illumination to produce temporally consistent shadows following the character motion.

Evaluation and Editability

Edit4D achieves state-of-the-art performance against recent 4D generation methods while supporting controllable camera motion Quantitative results show that Edit4D achieves state-of-the-art performance across the VBench metrics, obtaining strong text alignment, temporal consistency, and aesthetic quality. Furthermore, in an editability study, Edit4D achieves higher editing success and background preservation while reducing editing time. This efficiency is attributed to the asset based representation, which reuses components instead of regenerating the entire 4D scene.

Ablation and Limitations

Ablation studies confirm that constraints like initialization, contact loss, penetration loss, and global refinement are crucial for robust composition and quality The paper notes a limitation where the target object should be sufficiently visible and spatially localized in the input scene for text-guided segmentation to work effectively. Additionally, errors from intermediate prediction modules can propagate, meaning imperfect dynamic character reconstruction may introduce temporal or geometric inconsistencies.

Future Work

Future directions include incorporating more diverse editable assets into the component-wise representation and introducing interactive control over scene–character composition where users can refine placement after generation. Improving target localization under occlusion through stronger 3D scene understanding or target completion is also suggested as a path to increase robustness in composition. This work connects text-driven 4D generation with practical graphics-based editing workflows.

1. Bold Header: Enables object-level editing through component separation

The framework constructs 4D scenes from separate background and dynamic character representations, which enables independent editing, flexible control, and scene recomposition within a unified 4D space. This allows for modifications to the background or character without requiring the regeneration of the entire scene.

2. Bold Header: Automates scene–character placement

The system introduces a text-guided composition method that automatically places a dynamic character into a background scene, which localizes target region from text and optimizes character position with contact and penetration constraints, reducing the need for manual placement. This capability allows users to specify targets via text, and the system handles the complex geometric placement through optimization.

3. Bold Header: Ensures visual quality under novel-view changes

The method maintains better visual quality under novel-view camera changes compared to previous methods because it constructs the 4D scene from explicit background and dynamic 3D components, maintaining better visual quality under novel-view camera changes. This provides reliable synthesis for dynamic camera control by ensuring geometric consistency across different viewpoints.

4. Bold Header: Preserves unedited scene content during editing

Edit4D achieves higher editing success and background preservation than Free4D for both editing tasks while requiring substantially less editing time, because it reuses the existing scene assets and better preserves the unedited background content across viewpoints. This efficiency is achieved through its asset based representation, which reuses components instead of regenerating the entire 4D scene.

5. Bold Header: Supports character appearance modification

The modular 4D representation allows for character appearance modification, where one can construct a 3D mask for 'cloth' using our masking method and update the color values within the masked region while preserving the rest of the scene content. This enables flexible object-level editing on dynamic elements.

6. Bold Header: Provides physically plausible contact interactions

The composition stage uses contact-aware optimization with a penetration loss to prevent artifacts, ensuring physically plausible scene–character composition. This includes a mechanism to avoid implausible penetration while allowing stable contact between instances, as demonstrated by the optimization of the global translation and rotation of the character.

7. Bold Header: Improves fine-grained visual details via global refinement

The global scene refinement stage jointly optimize[s] the background scene, canonical character Gaussians, and the deformation network, which improves appearance harmonization between independently generated components while improving fine-grained visual details. This stage is crucial for achieving high perceptual quality by correcting residual inconsistencies.

8. Bold Header: Facilitates controllable camera motion

By using a representation where the rendered frame is defined as Ick,t = R(G(t), ck), Edit4D supports controllable camera motion, allowing users to define explicit camera trajectories for novel-view synthesis and dynamic scene visualization.