8.8Summary and Outlook

This chapter has surveyed spatial reconstruction and generation across seven subfields that, taken together, span an axis of increasing output structure and dimensionality: 2D images (§8.1), 3D objects and scenes (§8.2), video (§8.3), embodied motion (§8.4), 4D space-time content (§8.5), world models (§8.6), and interpretable symbolic programs (§8.7). Despite their different output domains, the subfields are unified by a single conditional-generation backbone, established in §8.1.1: an encoder $\kappa$ maps a condition $\mathbf{c}$ into a latent code, a generator $G_{\theta}$ samples structure from that code, and a decoder or renderer turns the sampled structure into a concrete artifact (Eq. (8.1), Eq. (8.2)). What distinguishes the subfields is not this backbone but the output space and the decoder that closes it: a pixel decoder for images, a volumetric or splatting renderer for 3D, a spatio-temporal decoder for video, a kinematic sampler for motion, a time-indexed 3D representation for 4D, a state-transition predictor for world models, and an exact, non-learned program executor for symbolic generation. Table 8.14 consolidates this view.

A shared backbone, specialized by output space. The progression from §8.1 to §8.7 is best read not as seven disjoint problems but as one generative abstraction instantiated against progressively richer codomains, and each section casts its own subtasks as conditional specializations of that common sampler. The symbolic case (§8.7) is the limiting instance in which the decoder degenerates to an exact program executor, trading the expressiveness of a learned renderer for interpretability and verifiable control.

The 2D prior as a reusable engine. The image generators of §8.1 recur throughout the chapter not as terminal systems but as priors to be distilled or lifted: 3D synthesis distills frozen 2D diffusion through score distillation and multi-view consistency (§8.2), video models inflate text-to-image backbones along the temporal axis (§8.3), and 4D pipelines compose both (§8.5). A single advance in the 2D substrate therefore propagates outward across every higher-dimensional subfield, which concentrates research leverage but also propagates the substrate’s failure modes.

Physical plausibility and fragmented evaluation. Across subfields, the recurring failure mode is not low visual fidelity but violated physics and geometry, manifest as multi-view inconsistency in 3D, temporal drift and flicker in video, foot-skating and interpenetration in motion, and dynamic implausibility in 4D. Evaluation remains correspondingly fragmented: each subfield has evolved its own metric tuples, and all are migrating toward learned and vision-language-model judges as scalar similarity scores prove insufficient for spatially structured output. A unified, physically grounded evaluation protocol that spans these subfields remains an open goal.

Outlook. These observations point toward convergence rather than further fragmentation. World modeling (§8.6) already acts as a conceptual capstone that absorbs reconstruction, generation, and dynamics into a single predictive substrate, while programmatic and symbolic generation (§8.7) supplies the controllability and interpretability that purely neural pipelines lack; their integration, neural generators steered and verified by symbolic structure, is a natural trajectory for the field. The capabilities surveyed here also form the generative substrate for the applications of Part IV, where embodied agents, digital twins, and interactive environments consume generated spatial content rather than producing it for its own sake.