8.54D Spatial Generation

Text-to-4D generation specializes the 4D sampler of Eq. (8.63) to a textual condition $\mathbf{c}=\mathbf{x}$ encoded by the text encoder $\phi$ of Eq. (8.2),

Because language fixes only semantics and leaves geometry and motion grossly under-determined, the sampler must hallucinate both a plausible 3D shape and its temporal evolution from the prompt alone, the source of the semantic-to-motion ambiguity examined in §8.5.1.4.

Two strategies recur throughout this section, implicit distillation and explicit supervision, implemented across various 4D representations (Figure 8.23). The more common implicit strategy leverages Score Distillation Sampling (SDS) to indirectly optimize the geometry and motion of a 4D representation, using pre-trained diffusion models as frozen priors [2267, 2268]. Formally, it applies a score-distillation update to per-scene parameters $\theta$ [1741],

where $\mathbf{x}_{\theta}=\mathrm{Render}(\hat{\mathbf{r}}^{1:T}_{\theta},\pi)$ is a differentiable rendering at camera $\pi$, $s$ indexes diffusion noise levels, and $\epsilon_{\phi}$ is a frozen 2D/video diffusion prior. The update differentiates through the renderer while treating the prior noise residual as the supplied gradient signal; this lifts the static score-distillation objective of §8.2.1.2 to a temporal sequence. In contrast, explicitly supervised methods initialize the 4D generation from text-to-3D or text-to-video outputs and use these generated assets as direct supervisory signals [2275, 191, 2277, 2270, 2007, 2278]. This “generate-then-reconstruct” strategy first synthesizes target frames $\hat{\mathbf{y}}$ with a fast generator $\mathcal{G}_{\text{2D}}$ and then fits the 4D asset to them,

replacing score-distillation guidance with explicit pixel targets; reconstruction may itself require per-scene optimization, so a speed advantage depends on the implementation. The two strategies are realized across mesh, NeRF, and 3DGS representations.

Mesh-based Methods. Meshes offer explicit geometric control, which suits precise text-driven shape manipulation and animation. CT4D [2276] implements a three-stage “Generate-Refine-Animate” pipeline in which a mesh is generated from text, its texture refined, and its vertices animated; TextMesh4D [2279] deforms a generated mesh through a per-face Jacobian deformation field under video-diffusion priors; and AvatarCLIP [2171] generates a static avatar zero-shot and animates it by registering motion to an SMPL skeleton.

NeRF-based Methods. NeRFs represent continuous geometry and complex temporal variation well. MAV3D [2267] introduced multi-stage SDS optimization that refines texture, geometry, and motion with priors from both T2I and T2V diffusion models; 4D-fy [2280] blends 3D-aware, image, and video score signals in a hybrid SDS; and Dream-in-4D [2268] first generates a static asset with a multiview diffusion model and then adds dynamics with a video diffusion model.

Gaussian Splatting-based Methods. 4D Gaussian Splatting (4DGS) [2272, 191] renders efficiently and differentiably, which speeds up text-driven dynamic scene synthesis. Early methods followed the two-stage implicit pipeline of Eq. (8.65): AYG [2270] generates a static 3D Gaussian model from text via SDS and adds dynamics under a T2V prior, and Comp4D [2278] uses an LLM to parse a complex prompt into several dynamic objects and their trajectories for a compositional scene. Later works moved to explicit supervision (Eq. (8.66)): PLA4D [2273] aligns the 4D Gaussians pixel by pixel with generated video targets, and 4Real [2281] uses generated multiview video frames as the reference for photorealistic scene-level generation. For digital humans, AvatarGO [2282] generates 4D human–object interactions zero-shot by animating a text-to-3D human with the SMPL-X skinning function.

Text-to-4D, like the other subfields of this section, has no standardized benchmark, because large-scale dynamic 3D data with text descriptions is hard to capture and annotate. Training and evaluation instead adapt static 3D repositories (Table 8.12): Diffusion4D [2283] filtered Objaverse and Objaverse-XL [1764, 1765] down to 54K high-quality animated assets, and SV4D [2284] built ObjaverseDy by adaptive sampling and viewpoint adjustment. These are synthetic renderings and lack the multimodal annotations, text in particular, needed to train and evaluate text-conditioned models robustly.

The challenge specific to text-to-4D is semantic-to-motion ambiguity: a prompt such as “a dancing dog” admits endlessly many movements, styles, and trajectories, and methods that lean on generic T2V priors struggle to produce diverse, fine-grained motion that matches a nuanced description. One response uses LLMs for motion planning, as Comp4D [2278] does when it decomposes a prompt into a structured spatio-temporal plan for several interacting objects; another targets human–object interaction [2282], moving toward story-driven dynamic scenes.

Image-to-4D generation specializes the 4D sampler of Eq. (8.63) to a single-image condition $\mathbf{c}=I$,

where the source view at $t=0$ is fixed by $I$ while every unseen viewpoint and time step must be inferred. Both the unseen 3D geometry and the entirely absent temporal dynamics have to be inferred from one 2D view, which requires combining the spatial cues of the input with temporal priors from large video models [2288, 2289, 2287].

Both strategies of §8.5.1.2 apply, and most methods use 4D Gaussian Splatting as the representation [2288, 2286].

Implicit Optimization with Diffusion Priors. These methods instantiate Eq. (8.65), typically combining a multiview diffusion model for geometric consistency with a video diffusion model for temporal coherence. Animate124 [2289] first generates a static NeRF from the image and then refines its dynamics with a video-diffusion SDS loss, and Diffusion² [2286] reconciles the scores of heterogeneous priors (multiview and video) through Variance-Reducing Sampling, improving both multiview and temporal consistency of the resulting 4D Gaussians.

Explicit Supervision with Generated Data. The slow, iterative nature of SDS motivates the two-stage pipeline of Eq. (8.66), in which an image-to-video model first generates an intermediate video that then supervises the 4D reconstruction directly. DreamGaussian4D (DG4D) [2288] generates a static 3D Gaussian model and learns a deformation field to animate it, using the generated video for texture refinement and dynamic supervision, and EG4D [2287] avoids SDS entirely by chaining image-to-video and image-to-multiview models into coherent image sequences that guide a fast reconstruction.

Unified and Tuning-free Generation. GenXD [2290] trains one model on both 3D and 4D data, using multiview-temporal modules that separate camera motion from object motion, and Free4D [190] builds a 4D scene from a single image without fine-tuning any model, distilling pretrained image-to-video and multiview priors into a consistent 4D representation.

Specialized Methods for Human Generation. Dynamic human avatars form a sub-domain in which strong body priors raise fidelity. Human4DiT [2291] uses a hierarchical 4D transformer that integrates human-specific features, camera parameters, and temporal signals to generate 360-degree human animations from a single image, and Disco4D [2292] models the body with SMPL-X while representing clothing separately with Gaussian splats, allowing detailed animation of clothed humans.

With no image–4D pairs, methods are evaluated on images from Objaverse [1764] or on in-the-wild photographs chosen by the authors, which makes comparison across papers unreliable. The metric set is that of §8.5.1.3, applied to the source view and to pseudo-ground-truth novel views where available.

Inferring a complete shape and a plausible motion from one viewpoint is extremely ill-posed, which shows up as geometric inconsistency in unseen views (e.g., , the “Janus problem”) and as unnatural or repetitive motion, since every dynamic detail must be hallucinated. Domain-specific priors are the main remedy: Human4DiT [2291] and Disco4D [2292] regularize geometry and motion with parametric body models and reach much higher fidelity, at the cost of generality.

Video-to-4D generation specializes the 4D sampler of Eq. (8.63) to a monocular video condition $\mathbf{c}=\mathbf{v}^{1:T}$,

so that re-rendering $\hat{\mathbf{r}}^{1:T}$ from the input viewpoint reproduces $\mathbf{v}^{1:T}$ while novel views remain geometrically consistent. Unlike image-to-4D (Eq. (8.67)), the temporal dynamics are now observed and the residual ambiguity is purely geometric: the unseen off-axis structure.

Most methods densify the monocular input into multi-view supervision and reconstruct against it (Eq. (8.66)), some add score distillation (Eq. (8.65)), and the newest predict the 4D asset feed-forward.

Mesh-based Methods. Meshes are interpretable and efficient to animate. DreamMesh4D [2293] obtains a coarse mesh and drives it with a sparse deformation graph and linear blend skinning (LBS), reaching strong spatio-temporal consistency more cheaply than implicit volumetric methods, and V2M4 [2294] reconstructs an animated, textured mesh asset directly from a monocular video.

NeRF-based Methods. NeRF-based methods model the continuous volumetric content of a video well, and they overcome the single view by generating data. Consistent4D [2269] densifies the monocular input through temporal and multiview interpolation before optimizing a NeRF with SDS, 4Diffusion [2295] fine-tunes a video diffusion model with a learnable motion module on synthetic multiview data so that it produces consistent multiview video to guide generation, and SV4D [2284] completes missing views and frames with a diffusion model and uses the resulting multiview video to supervise the reconstruction.

Gaussian Splatting-based Methods. 4DGen [2296] grounds 4D Gaussian generation in the input video with spatial-temporal pseudo labels; Efficient4D [2297] converts the video into a multiview image matrix with a multiview-consistent generator (SyncDreamer) for rapid reconstruction; SC4D [2298] first fits sparse control Gaussians for coarse shape and motion and then drives dense Gaussians through LBS; and L4GM [2299] predicts the 4D Gaussians of a whole clip in a single feed-forward pass, trained on rendered animated objects.

Video-to-4D has the one dedicated evaluation set in this section, released with Consistent4D [2269]: synthetic and real monocular videos with training and test splits. The metrics are those of §8.5.1.3, with reconstruction fidelity measured at the input view and novel-view quality measured where synthetic ground-truth multiview video exists.

The video constrains motion but not the unseen 3D structure, which leads to geometric distortion, depth ambiguity, and a “painted-on” look in novel views. Explicit geometric structure is the main remedy: DreamMesh4D [2293] and SC4D [2298] constrain the reconstruction with meshes or sparse control points under LBS, which improves both coherence and efficiency, and combining such structure with the appearance modeling of neural representations is the obvious next step.

Here the condition is itself a 3D representation $\mathbf{c}=\mathbf{r}_{0}\in\mathcal{R}$, so only its temporal evolution must be generated. We therefore specialize Eq. (8.63) to a deformation operator that takes the static asset and an optional auxiliary driving signal $\mathbf{c}_{\text{aux}}$ (text, audio, or a motion sketch) to a sequence,

with the first frame pinned to the input asset. This places 3D-to-4D in the same family as the editing operators of §8.2.5 and §8.4.4: it transforms an existing representation rather than synthesizing one from scratch.

Methods realize the deformation operator $\mathcal{D}_{\theta}$ of Eq. (8.69) either with explicit mesh structure, increasingly through learned rigging, or with neural and physics-informed deformation fields.

Mesh-based Methods. Meshes give explicit structure and precise control over deformation. Motion3DGAN [2301] generates 3D facial animation from a static face mesh by first predicting the displacement of facial landmarks for a target expression and then deforming the full mesh, with a GAN keeping the sequence temporally coherent. A recent line makes arbitrary meshes animatable by predicting the rig itself: MagicArticulate [2302] and RigAnything [2303] predict skeletons and skinning weights feed-forward, Puppeteer [2304] rigs a mesh and then animates it by fitting the rig to videos generated from renderings of the mesh, and AnimateAnyMesh [2305] generates text-driven mesh animation in a single forward pass with a rectified-flow model over a dynamic-mesh latent.

NeRF-based Methods. ElastoGen [2306] generates physically grounded elastodynamic behavior: it rasterizes a static 3D model and then predicts and refines deformation fields iteratively with recurrent and 3D convolutional modules, so that the motion obeys physical principles.

No benchmark with ground-truth dynamic sequences exists; methods are evaluated on a few individual assets, often faces or generic objects, mostly by user study and with the metrics of §8.5.1.3 where a reference animation is available. Physical plausibility, specific to this subfield, can be measured for physics-informed methods such as ElastoGen [2306] as deviation from a high-fidelity simulator or as violation of conserved quantities such as energy and momentum.

Even with the geometry given, the space of plausible animations of a static model is unbounded, so without guidance the task is unconstrained, and current methods specialize to narrow categories such as faces. Physical priors constrain it (ElastoGen [2306] grounds motion in elastodynamics), learned rigging makes arbitrary assets animatable, and pairing the asset with text, audio, or motion sketches turns open-ended generation into controllable animation.

We generalize the single-condition cases of Eqs. (8.64)–(8.69) to a set of $K$ conditions, each mapped by its own encoder $\kappa_{k}$ and fused into the shared embedding:

which recovers a single-modality sampler when $K=1$. Users can then specify not only what is generated (semantics) but how it behaves (motion path, physical behavior, artistic style), which complex applications such as character animation and driving simulation require.

Methods fuse the encoders $\kappa_{k}$ of Eq. (8.70), usually combining score distillation (Eq. (8.65)) with explicit supervision (Eq. (8.66)) tailored to each condition.

Motion and Trajectory Control. TC4D [2307] lets users define a spline trajectory that guides both the global rigid transformation and the local deformation of an object, with text supplying the semantics. For human animation, skeletons are the natural condition: Beyond Skeletons [2308] integrates text and image conditions with the skeleton to refine motion, and STAR [2309] uses skeleton-aware SDS and motion retargeting so that avatar movement is both visually coherent and anatomically accurate.

Physics-Informed Control. Sync4D [2310] and Phy124 [2311] integrate differentiable Material Point Method (MPM) simulation, so that the generated content follows realistic dynamics such as elasticity and collisions under video or text guidance.

User-Centric and Artistic Control. Generative Rendering [2312] uses a pre-existing dynamic mesh to guide a pretrained text-to-image model, giving precise control over the style and detail of the rendered output, and sketch-based interfaces [2313] let users draw the dynamics they want.

Complex, Application-Specific Control. MagicDrive3D [2274] generates dynamic driving scenes from 2D bounding boxes, Bird’s-Eye-View (BEV) maps, agent trajectories, and text together, producing tightly controlled traffic scenarios.

No benchmark exists; building one would require 4D assets annotated with several aligned conditions (e.g., , a clip with a description, a skeletal motion sequence, and a trajectory map). Text adherence is scored with the CLIP Score of §8.5.1.3, while adherence to sketches or trajectories relies on task-specific errors (e.g., , distance from the specified trajectory) or on user studies, which remain the only holistic judge of how well a combination of inputs was followed.

Semantic guidance from text, geometric constraints from skeletons, and rule-based priors from physics can contradict one another, and keeping each respected without one overpowering the rest while preserving fidelity is the open problem. Domain-specific stacks such as MagicDrive3D [2274] sidestep it by fixing the set of inputs for one application, and interactive interfaces such as sketches [2313] point toward iterative creation in which several control channels are adjusted in turn.

Data Scarcity and Generative Priors. Large, annotated 4D datasets are the field’s main missing ingredient: real dynamic 3D data with multi-view consistency and multimodal annotation (text, motion capture) is expensive and technically hard to collect, which has largely ruled out end-to-end training. The field has therefore converged on distilling pretrained text-to-image, text-to-video, and text-to-3D models, the de facto way to guide 4D synthesis.

From Score Distillation to Explicit Supervision. Score distillation [1741] converges slowly, is costly and unstable, and produces multi-faced geometry and flickering textures. The generate-then-reconstruct pipeline of DreamGaussian4D [2288] and EG4D [2287], which first generates target videos and then reconstructs against them, is replacing per-prompt SDS, and feed-forward predictors such as L4GM [2299] and AnimateAnyMesh [2305] take the next step by removing per-scene optimization altogether.

4D Gaussian Splatting as the Default Representation. NeRF-based works rendered well but trained and rendered slowly. 3D Gaussian Splatting [121] and its temporal extension offer real-time, high-fidelity rendering with efficient gradient-based optimization, and their explicit, point-based form is easier to edit and to couple with physical simulation, so 4DGS has become the representation of choice for current methods [191, 2288, 2270].

Consistency, Control, and Physical Plausibility. Keeping geometry, appearance, and motion coherent across all viewpoints and over long horizons remains the central difficulty, and control is moving beyond single prompts toward fine-grained manipulation of motion, appearance, and interaction. Physical plausibility is the newest goal: integrating differentiable simulators, as PhysSplat [2275] and Phy124 [2311] do, makes generated dynamics more realistic and predictable. Progress on all three is hard to measure while evaluation relies on small, author-curated test sets and on image and video metrics that ignore cross-view and long-range consistency; 4D-aware benchmarks and metrics are a prerequisite for the rest.