8.23D Spatial Reconstruction and Generation

3D-Native Generators. The earliest 3D generators instantiate $G_{\theta}^{\text{3D}}$ as a sampler trained directly on a 3D representation, by denoising score matching on voxels, SDFs, or triplanes or, in earlier work, by an adversarial fit to a category-specific 3D corpus $p_{\text{data}}^{\text{3D}}(\mathbf{r})$,

where the noise predictor $\epsilon_{\theta}^{\text{3D}}$ operates directly on $\mathbf{r}\!\in\!\mathcal{R}$ with no 2D intermediary, so the supervision signal scales with the size of the 3D corpus rather than with 2D image–text pairs. Because 3D data is heterogeneous, these generators are themselves diverse, having been instantiated on signed distance fields [1722], voxel grids [1723, 1724, 1725], point clouds [1726, 1727], view sets [1728, 1729], neural radiance fields [278, 1730, 1731, 1732, 1733], triplanes [1734, 1735, 1736], and 3D Gaussian Splatting [1737, 1738, 1739]. The scarcity and monotony of 3D corpora remain the bottleneck: even with sophisticated architectures, generated geometry and texture lack fidelity and stay confined to a handful of canonical categories, which is what pushed the field, around 2022–2023, toward paradigms that use the much larger 2D corpora as an indirect supervision signal for 3D structure.

2D-Prior Distillation (SDS Family). The second paradigm sidesteps the 3D-data bottleneck by realizing $G_{\theta}^{\text{3D}}$ as a per-shape optimization that distills gradients from a pretrained 2D generator (a CLIP model [143] or a 2D diffusion model [185]) trained on web-scale image data [1670]. Given the frozen 2D noise predictor $\epsilon_{\phi}$ of Eq. (8.2) and a differentiable renderer $\mathrm{Render}(\mathbf{r}_{\theta},\pi)$ that images a 3D representation $\mathbf{r}_{\theta}\!\in\!\mathcal{R}$ under camera pose $\pi$, score distillation updates only the per-shape 3D parameters $\theta$ via the gradient

where $w(t)$ is a step-dependent weighting and $\mathbf{y}_{t}^{\pi}$ is the rendered view perturbed by the diffusion forward process. The contrast with Eq. (8.10) is sharp: the 2D prior of §8.1 is held fixed and no 3D corpus is consumed, while the trainable parameters live in the representation of a single asset, hence the per-shape optimization cost paid at inference. A line of work [1740, 1741, 1742, 1743] applies score-distillation sampling (SDS) or its variants to a NeRF or mesh; DreamFusion [1741] obtains zero-shot text-to-3D by optimizing a NeRF whose multi-view renderings the 2D diffusion prior scores, and GaussianDreamer [1744] moves the same recipe onto a 3DGS representation. These methods need no 3D training data, but because the optimization is rerun for every prompt they are slow, prone to over-saturated color, and subject to the “multi-face” (Janus) artifact in which the 2D prior fails to disambiguate the front and back of an object; variational score distillation [1745] mitigates the over-saturation but not the per-prompt cost.

Multi-View Generation and Lifting. The third paradigm realizes $G_{\theta}^{\text{3D}}$ as a two-stage pipeline: a fine-tuned 2D diffusion model generates a small set of geometrically consistent views, and a separate reconstructor lifts them to a 3D asset, so that Eq. (8.9) factorizes as

where the view-conditional 2D sampler $p_{\theta}(\cdot\mid\cdot)$ is a fine-tuned variant of Eq. (8.2) that additionally consumes a reference image or text $\mathbf{y}_{\text{ref}}$ and a target camera pose $\pi_{v}$, and $\mathcal{F}_{\psi}$ is a (typically feed-forward) multi-view-to-3D reconstructor. Two differences from Eq. (8.11) follow: the 2D prior is fine-tuned rather than frozen, and the 3D representation appears only inside the deterministic lift $\mathcal{F}_{\psi}$, with no per-shape inner loop at inference, which is what lets this paradigm run one to two orders of magnitude faster than SDS at the price of depending on the view-consistency of the synthesized $\hat{\mathbf{y}}_{v}$. The first step was to make a pretrained diffusion model pose-aware. Following fine-tuning recipes such as ControlNet [1693] and LoRA [1746], several works [1747, 1748] fine-tune for cross-view consistency; Zero-1-to-3 [1749] opened the open-world single-image setting by training a diffusion model to synthesize a novel view conditioned on an input view and a relative camera pose; MVDream [1750] and ImageDream [1751] fix a world-camera coordinate convention from which geometry can be recovered; Magic123 [1752] combines the Zero-1-to-3 prior with the 2D prior of Stable Diffusion; One-2-3-45 [1753] lifts Zero-1-to-3 views to a mesh; and Consistent123 [1754] and MVD-Fusion [1755] add boundary and depth priors. Attention then turned to consistency across the generated views: SyncDreamer [1756] correlates features across views with a 3D-aware attention, CAT3D and Free3D [1757, 1758] model the multi-view distribution jointly, and Zero123++ [1759] tiles six surrounding views into a single grid image, a trick inherited by One-2-3-45++ [1760], Instant3D [1761], InstantMesh [1762], and AR-1-to-3 [1763].

Feed-Forward Reconstruction and 3D-Native Latent Diffusion. The fourth group amortizes generation across object-scale 3D corpora rather than optimizing each asset against a frozen 2D prior, in two instantiations. A feed-forward reconstructor trains $G_{\theta}^{\text{3D}}$ end-to-end to map the conditioning embedding $\kappa(\mathbf{c})$ directly to a 3D representation $\hat{\mathbf{r}}\!\in\!\mathcal{R}$, supervised by differentiable rendering against $V$ ground-truth views $\{\mathbf{y}_{v},\pi_{v}\}$ drawn from the corpus,

A 3D-native latent diffusion model instead learns a 3D autoencoder pair $(E^{\text{3D}},D^{\text{3D}})$ with $\mathbf{z}^{\text{3D}}\!=\!E^{\text{3D}}(\mathbf{r})$, trains a denoising model on the compact latent, and decodes via $\hat{\mathbf{r}}=D^{\text{3D}}(\hat{\mathbf{z}}^{\text{3D}})$,

which is the 2D LDM objective Eq. (8.5) of §8.1.1 with the autoencoder moved from pixels to $\mathcal{R}$. Both variants avoid the per-shape optimization of Eq. (8.11), but only the reconstructor uses a single prediction pass; latent diffusion still samples iteratively. The unlocking factor was the arrival of Objaverse and Objaverse-XL [1764, 1765]. LRM [1766] and its follow-ups [1767, 1768, 1769] show that large transformers trained purely with rendering losses capture object geometry and texture in detail and generalize to unseen images, and a parallel line introduces compact 3D-native latent codes [1770, 1771, 1772, 1773] whose distribution a latent diffusion model can learn; TRELLIS [1721], CLAY [1774], HiGS-RGBN [1775], and Hunyuan3D [1776] build on this to bring generated assets close to production grade while keeping the open-vocabulary reach that the SDS family pioneered.

Two design choices cut across the four paradigms. The representation $\mathcal{R}$ (mesh, SDF, voxel, triplane, NeRF, 3DGS, or structured latent) is largely independent of the paradigm: SDS has been run on NeRFs and on 3DGS, feed-forward reconstructors output triplanes, meshes, or Gaussians, and latent-diffusion stacks decode to almost any explicit representation through a learned VAE. So is the conditioning modality: the same backbone takes text, a single image, or sparse views by changing only the encoder $\kappa$ and the camera-pose embedding, and production systems chain them (image-to-3D conditioned on a text-to-image preview). The succession of paradigms itself tracks the accumulation of supervision at each level, from 3D-native corpora to 2D image–text pairs to Zero-1-to-3-style multi-view priors to Objaverse-XL-scale 3D, with the most recent industrial systems sitting in the fourth paradigm while inheriting the multi-view-consistency tricks of the third.

Training Corpora. Object-scale 3D corpora drive the feed-forward and latent-diffusion paradigms; Table 8.2 lists the ones most commonly used, and Figure 8.6 shows representative Objaverse-XL assets. They span the classical CAD repository ShapeNet [1777], category-bounded real-world scans with accurate geometry and PBR textures (Google Scanned Objects [1778], OmniObject3D [1779] with dense multi-view imagery), the open-vocabulary web-scraped Objaverse and Objaverse-XL [1764, 1765] that unlocked the fourth paradigm, and G-Objaverse [1780], which adds text-driven synthesized assets between the two.

Evaluation Benchmark Suites. Recent benchmarks have moved to multi-dimensional assessment beyond plain text–image alignment. GT23D-Bench [1781] curates a high-fidelity dataset and a suite of 3D-aware metrics spanning alignment and visual quality; MATE-3D [1782] provides designed prompt categories with large-scale human annotations across several perceptual dimensions and introduces HyperScore as a learned evaluator; T23D-QA [1783] adopts a question-answer protocol that scores geometry, texture, and alignment independently; and Gen3DEval [1784] fine-tunes a vision-language model for 3D quality assessment, scoring text fidelity, appearance, and surface quality from rendered surface normals. Together with the 3D-relevant subsets of the 2D suites collected in §8.1.1 (Table 8.1), they form the de facto evaluation protocol for 3D object synthesis.

Constraint-specific Metrics. Because an asset is judged from many viewpoints and under different downstream uses, evaluation reports complementary metrics along three axes. Text-3D alignment renders multi-view images from the asset and measures text–image similarity with CLIP-like encoders, aggregated across views (mean, worst-case, or view-consistency variants); pure CLIP alignment can be misled by texture “shortcuts” and does not guarantee geometric correctness, which motivates the 3D-aware alignment scores of GT23D-Bench [1781] and learned evaluators such as HyperScore [1782]. Geometry quality is measured against ground-truth shapes, where they exist, by Chamfer Distance (CD) [1785], F-score at varying thresholds, normal consistency, and volumetric IoU for watertight meshes; for text-to-3D without ground truth it falls back on multi-view depth and normal consistency, silhouette consistency, and learned 3D-quality predictors. Appearance and multi-view consistency are measured by perceptual distances (LPIPS) between rendered views and references where available, by view-consistency measures (variance across views under fixed lighting and camera trajectories), and by human studies, which recent benchmarks collect as multi-dimensional ratings to calibrate the automatic metrics [1782, 1783].

The paradigm succession of §8.2.1.2 shows up in two measurable quantities. Generation time has fallen by about three orders of magnitude: DreamFusion [1741] needs roughly 1.5 hours per asset and Magic3D [1742] about 40 minutes, whereas LRM [1766] reconstructs a shape in about 5 seconds and InstantMesh [1762] completes image-to-3D in around 10 seconds. Image-to-3D geometry is now routinely scored on Google Scanned Objects [1778] with Chamfer Distance (lower is better) and F-score at $\tau{=}0.2$ (higher is better), where InstantMesh [1762] reaches CD $\approx 0.18$ and FS $\approx 0.88$, ahead of TripoSR [1786] ($0.217$ / $0.843$), CRM [1768] ($0.252$ / $0.787$), and LGM [1767] ($0.345$ / $0.671$). Coarse text-to-3D alignment, by contrast, is saturated: CLIP-style scores no longer separate the strongest systems, so Hunyuan3D 2.0 [1787] reports ULIP- and Uni3D-based image- and text-to-shape similarity backed by a study in which 50 participants rated 300 unselected results, and learned evaluators (HyperScore on MATE-3D [1782], T23D-QA [1783]) score geometry and texture on separate axes. The reconstruction metrics carry their own caveat: CD and F-score saturate on coarse geometry and can reward visually plausible but physically thin assets, so reported gains are credible mainly where geometric, perceptual, and human ratings agree. Commercial text- and image-to-3D services now ship these capabilities with PBR export to game engines, alongside research systems such as Meta 3D Gen [1788] and the open-vocabulary 3D-native-latent systems CLAY [1774], TRELLIS [1721], and Hunyuan3D [1776, 1787].

Multi-View and 3D Consistency under Open-Vocabulary Prompts. Enforcing consistent geometry and texture across viewpoints while preserving fine detail remains difficult in all four paradigms, and the failures have names. The Janus artifact renders both the front and the back of an object as a frontal face, most acutely in SDS pipelines, where the 2D prior cannot disambiguate camera direction; view drift makes adjacent views of a multi-view stack disagree on geometry, a residual concern in Zero-1-to-3-style cascades that SyncDreamer’s cross-view attention and Zero123++’s tiled six-view trick only partly remove; and texture seams appear where neighboring views’ color and lighting cannot be merged into a globally coherent UV map, a persistent weakness of feed-forward reconstructors. These failures motivate 3D-native backbones with structured latents [1721] that share parameters across views by construction, and benchmarks that score multi-view consistency rather than a single rendered view [1782].

Representation Trade-offs between Fidelity, Editability, and Deployability. Even a consistent asset must land somewhere a downstream tool can consume, and the choice of representation is not neutral. Meshes with UV/PBR textures are what game engines and simulators want but lose detail to discretization; radiance fields and Gaussians give high visual fidelity and fast rendering but require conversion, which introduces its own failures: topology breakage during marching-cubes-style meshing, baked illumination that fuses lighting into albedo when converting from a NeRF or 3DGS, and texture bleeding across UV seams. The response is unified latent representations that decode to several formats [1721], so that a single generator serves diverse downstream requirements and the choice of $\mathcal{R}$ is deferred to inference.

Scalable, Reliable Evaluation and Human-Aligned Metrics. These representation choices complicate evaluation, because the metric panel a paper reports tends to depend on which format its generator targets, and the automatic metrics remain imperfect for geometry plausibility and asset usability. CLIP-style alignment is fooled by texture shortcuts that ignore the underlying shape; Chamfer Distance saturates on coarse geometry while penalizing semantically irrelevant fine detail; and LPIPS on rendered views cannot distinguish a physically plausible asset from a visually plausible cardboard cut-out. Multi-dimensional evaluators trained on large human-annotation sets (e.g., , HyperScore on MATE-3D [1782]) and diagnostic benchmarks (e.g., , GT23D-Bench [1781], T23D-QA [1783]) that score alignment, geometry, and texture on independent axes are the current answer, so that progress on one axis is no longer masked by averaging.

Data Scaling and Asset Readiness. Underlying consistency, representation, and evaluation alike is the supply of training data and the industrial scale at which it can be exploited. Large-scale corpora such as Objaverse-XL [1765] and industrial-scale systems such as Hunyuan3D [1787, 1789] and Meta 3D Gen [1788] point toward high-resolution, engine-ready assets with controllable materials, relightable textures, and post-processing for watertightness and topology. The question has shifted from whether a paradigm can be trained at scale, which the fourth paradigm has settled, to which slice of the 3D-asset distribution the available corpora still under-represent: articulated objects, semi-transparent or refractive materials, and topologically non-trivial assemblies are visibly thinner in current open corpora than in production pipelines, and closing this gap is the explicit target of recent industrial releases.

Layout and Scene-graph Conditioned Methods. A major line of research generates 3D environments from high-level structural priors such as layouts $\mathbf{L}$ or scene graphs $\mathbf{G}_{\text{in}}$, instantiating Eq. (8.15) by materializing the structured tuple as an explicit object set,

where the joint over the $N$ slots and their poses is factorized autoregressively (a transformer over object slots) or as a denoising diffusion over the slot tensor. SceneFormer [1790] explored autoregressive modeling of room layouts and furniture placement, and DiffuScene [1791] introduced diffusion over object arrangements, improving realism and diversity. Recent work adds language and visual reasoning for compositional control: LayoutGPT [1703] plans the layout with an LLM before neural scene construction, and LayoutVLM [1792] optimizes layout consistency under multimodal alignment. Scene-graph approaches (GraphDreamer [1793], CommonScenes [1794], EchoScene [1795]) use relational reasoning to keep placements plausible, and InstructScene [1796] adds natural-language conditions for controllable scene creation. Beyond indoor settings, CityCraft [1797] scales layout synthesis to the city level, and GALA3D [1798] couples layout-guided diffusion with Gaussian-based rendering for photorealistic multi-object environments.

View-to-3D Representation Synthesis. The second paradigm trades the explicit structured set for a continuous 3D representation, recovering the scene as a radiance field or 3D Gaussian collection from sparse calibrated views; the scene tuple collapses into a single field,

where $\mathcal{F}_{\psi}$ is either a deterministic feed-forward regressor or a generative sampler over the field. This scales the multi-view-and-lift recipe of §8.2.1.2 from one asset to a full scene by replacing the per-object reconstructor with a scene-scale one. Deterministic regressors such as PixelNeRF [1799], PixelSplat [1800], DepthSplat [675], and MVSplat [1801] / MVSplat360 [1802] stay strictly faithful to the input views and are reconstruction methods, treated in §8.2.3.2. The generative variant samples the field instead: DreamScene [1803] introduces a diffusion prior over Gaussian fields for text-conditioned scene creation, and Generative 3DGS [1804] generates 3D Gaussians for scenes with video priors.

View-to-View Synthesis (Regression and Generative). The third paradigm skips the explicit 3D representation altogether and formulates novel view synthesis as a direct conditional 2D-to-2D map from observed to target views,

where $\pi^{*}$ is the target camera pose and no intermediate 3D representation is materialized. Regression-based approaches learn a deterministic map: SRT [1805] encodes the scene into latent tokens that support interpolation and extrapolation across viewpoints, and LVSM [1806] predicts novel views with a large feed-forward transformer and minimal 3D inductive bias. Generative approaches model the conditional distribution over unseen views, which permits stochastic synthesis under ambiguity and large viewpoint changes: Photoconsistent-NVS [1807] and SEVA [1808] use diffusion to improve diversity and robustness along extended camera trajectories, and CausNVS [1809] generates views autoregressively and causally, modeling inter-view dependencies explicitly to stay coherent across complex viewpoint transitions.

Video-model Based Scene Synthesis. The fourth paradigm adds temporal coherence to the viewpoint variety of the view-to-view family by exploiting large video generation models as spatiotemporal priors; the scene tuple is materialized as a frame sequence,

treating the scene as the implicit 3D structure of a coherent video and the camera trajectory $\{\pi_{v}\}$ as an explicit conditioning channel. Where the feed-forward reconstructors of §8.2.1.2 output a static asset in one pass, the video paradigm trades static 3D for temporally coherent multi-frame output that mirrors the navigability requirement of a scene more directly. CameraCtrl [1810] and MotionCtrl [1811] condition video generation on camera trajectories directly; ViewCrafter [1812] and Gen3C [1813] add geometric guidance by unprojecting observed views into an intermediate 3D representation and reprojecting it to the target viewpoints, which improves pose fidelity; and CAT4D [1814] and 4DiM [1815] train jointly on posed static multi-view data and on temporal video data, so that viewpoint variation and temporal dynamics are modeled together for 4D scene synthesis.

Training Corpora. Table 8.3 lists the corpora in use. Layout-conditioned methods rely overwhelmingly on 3D-FRONT [1816], usually combined with its companion 3D-FUTURE [1817] furniture library for instance-level geometry. View-based methods (P2 / P3) draw on real-world scans of residential interiors (ScanNet [448], Matterport3D [340]), on photorealistic synthetic corpora with ground-truth depth and normals (Hypersim [664]), and on Replica [339] for compact high-fidelity indoor benchmarks. Video-model-based methods (P4) inherit the web-scale supervision of their underlying video prior, far larger than any 3D corpus but without explicit 3D supervision, supplemented by camera-pose-annotated subsets distilled from ScanNet, RealEstate10K [734], or hand-labeled trajectories.

Evaluation Benchmark Suites. Layout-conditioned methods report the metrics introduced with ATISS on 3D-FRONT (object-category frequency, pairwise-relation accuracy, and FID on rendered top-down or perspective views). View-to-3D and view-to-view methods adopt the NeRF-era panel of PSNR / SSIM / LPIPS on held-out views of ScanNet, RealEstate10K [734], or DL3DV-10K [735]. Video-model-based methods add FVD and a camera-trajectory-adherence metric, the mean translation and rotation error between requested and recovered camera poses.

Constraint-specific Metrics. Four axes mirror the scene tuple of Eq. (8.15): layout adherence (object-count agreement, pairwise-relation accuracy on $\mathbf{G}$, and bounding-box IoU between requested and synthesized placements); geometric fidelity where ground truth exists (Chamfer Distance, F-score, and depth and normal consistency against held-out views); photometric quality (PSNR / SSIM / LPIPS against ground-truth views, FID / KID against held-out scene distributions, and FVD for video outputs); and camera-pose adherence, most relevant for P4 (translation and rotation error of poses recovered by structure-from-motion or a learned pose regressor). A paper reports the subset of axes its paradigm can be scored on, which is why comparison across paradigms remains difficult.

Cross-Paradigm Inconsistency. The four paradigms solve subtly different problems, and each leaves a gap when used alone: a layout-conditioned generator can satisfy the symbolic prior $\mathbf{G}_{\text{in}}$ yet produce a non-traversable static scene, and a view-to-3D generator can produce a navigable Gaussian field that ignores the requested object semantics. The gaps surface as layout-adherence drift, where the generated NeRF or 3DGS no longer satisfies the input layout once the user roams outside the trained view envelope; object drop, where small or thin objects from $\mathbf{L}$ are silently omitted during continuous-field synthesis; and geometry flattening, where a multi-object composition collapses into a single textured slab when the 2D or video prior dominates. Hybrids such as GALA3D [1798] and ViewCrafter [1812] chain a layout or geometry prior with a Gaussian or video renderer to keep both halves honest, and future systems will more likely compose the four paradigms than choose among them.

Scaling Beyond Indoor Rooms. Almost all paradigms remain anchored to furnished indoor rooms because that is where the training corpora live: 3D-FRONT [1816] alone supplies the layout supervision for the dominant P1 methods, and ScanNet, Matterport3D, and Hypersim dominate P2 / P3. The result is an outdoor and urban scale gap that CityCraft [1797] is among the few systems to address, a category imbalance with many furniture classes but few terrain, vegetation, or industrial ones, and an open-world domain shift under which methods trained on residential interiors fail on industrial sites, excavations, or invented environments. Web-scraped open-vocabulary scene corpora and hybrid synthetic-real pipelines that blend procedurally generated environments with captured scans are the active responses, both for breadth and to control the long tail of the scene-class distribution.

Camera-Trajectory Coherence under Long-Horizon Synthesis. The video paradigm (§8.2.2.2) excels at short camera moves but degrades on long, unbounded trajectories that require persistent world memory. Long trajectories expose temporal blur, where textures shimmer or drift between frames; scene amnesia, where objects reappear with different geometry when the camera revisits a location; and pose-conditioning collapse, where the model learns to ignore the camera signal in favor of the marginal video distribution. ViewCrafter [1812]’s unprojection and reprojection scaffold, Gen3C [1813]’s 3D cache, and CAT4D [1814]’s joint multi-view and video training all inject an explicit 3D inductive bias back into the video prior, but the trade-off between the video prior’s perceptual quality and the geometric prior’s rigidity remains an open design tension.

Evaluation. A P1 layout-adherence score and a P3 PSNR cannot be ranked against each other, no single dataset evaluates layout faithfulness, geometric fidelity, photometric quality, and pose adherence jointly, and no human-aligned evaluator analogous to HyperScore [1782] has been validated on scene outputs.

Trends: Toward 4D and World Models. Scene synthesis targets static geometry conditioned on camera poses, but the same formulation is being extended toward 4D synthesis and action-conditioned world modeling: Dreamer [147], GameNGen [1818], Diamond [1819], RTFM [1820], and Genie3 [157] move from static view or video synthesis toward open-ended environment generation in which spatial structure, temporal evolution, and interaction are modeled together. From the standpoint of this subsection, scalable, temporally coherent video generation with explicit viewpoint control is the foundation those systems build on; they are treated as world models in §8.6.

Iterative 3D Reconstruction. The classical paradigm casts reconstruction as a per-scene optimization: there is no learned $\theta$ shared across scenes; the 3D representation $\mathbf{r}$ itself is the optimization variable, and Eq. (8.21) is minimized from scratch for each new input,

Classical image-based reconstruction is rooted in Structure-from-Motion (SfM) [1822, 107, 1823, 115], which estimates camera poses from unposed images, followed by Multi-View Stereo (MVS) [1824, 112, 1179, 113, 1825, 1826], which recovers dense geometry. Decades of refinement have made SfM + MVS a mature and widely used pipeline (see [112] for a comprehensive treatment), yet it still struggles with textureless surfaces, reflective objects, and large illumination changes.

A more recent line represents the scene as a continuous function parameterized by a neural network: signed distance functions [117, 1827, 288, 1828], occupancy fields [118, 1829], and neural radiance fields [119, 271, 1830, 1831]. The original formulations were expensive to train and render and limited in quality; PlenOctrees [1832], Plenoxels [289], Instant-NGP [120], RefNeRF [1833], TensoRF [1834], and ZipNeRF [272] are among the many follow-ups that improved efficiency and fidelity, but all of them still require a large number of input images per scene. Sparse-view variants add auxiliary information such as regularization [1835], depth priors [1836, 1837], point clouds [1838], and surface priors [1839], or distill a 2D diffusion prior into the per-scene representation, the SDS recipe of §8.2.1.2 applied to reconstruction from one or a few images [1840, 1841, 1842]. 3D Gaussian Splatting [121] then made per-scene optimization fast, training within minutes and rendering in real time, with subsequent work adding depth regularization, anti-aliasing, mesh extraction, and surface alignment [1843, 290, 1844, 1845]. All of these systems, however, work well only over nearby views or a small scene and must be re-optimized from scratch for every new input, which motivates the amortized paradigm.

Feed-forward 3D Reconstruction. The feed-forward paradigm amortizes the per-scene optimization of Eq. (8.22) across a learned prior: a network $G_{\theta}^{\text{rec}}$ is trained once on a large multi-view corpus, then deployed in a single forward pass on unseen inputs, minimizing

where $\mathcal{D}$ is a representation-specific distance (Chamfer Distance for point maps, a rendering loss for radiance fields) and $(\mathbf{O},\mathbf{r}_{\text{gt}})$ are observation–asset pairs drawn from a 3D corpus. This is the reconstruction-side analogue of the view-to-3D paradigm of §8.2.2.2: sparse calibrated views are mapped feed-forward to a continuous scene representation, but the supervision is a reconstruction loss against ground-truth 3D rather than a rendering loss against held-out views alone.

Early feed-forward work applied 2D networks to individual 3D representations, for depth estimation [55, 1846], voxel reconstruction [1847, 1377], point-cloud reconstruction [1848, 132], and mesh deformation [285, 1849], but remained limited to single objects or small datasets. Learned components then replaced individual stages of the SfM + MVS pipeline, keypoint detection and description [539, 1850, 540], image matching [541, 1851], pose estimation [723, 625, 1852], and depth prediction [133, 625], proving more robust and accurate than their hand-designed counterparts and being integrated into the classical pipeline [1853, 116, 622]. DUSt3R [122] and MASt3R [728] then reconstructed large scenes from unposed images end to end: similar in spirit to MVS but without explicit keypoint detection, matching, and pose estimation, the model predicts point maps in the reference view directly and recovers camera poses in post-processing, and follow-ups [1854, 731, 732, 1855, 123] have improved quality and efficiency at scale.

Beyond geometry alone, a parallel line recovers geometry and appearance together through differentiable rendering. Scene Representation Networks [183] and Light Field Networks [1856] introduced ray-based querying for novel-view synthesis and were followed by epipolar attention [1857, 1858], large transformers [1806, 1859], and integration with pretrained generative models [1758, 1757]; PixelNeRF [1799] pioneered pixel-aligned features for NeRF-based reconstruction and inspired 3D volume representations [1860, 1861], feature-matching fusion [1862], and large transformers [1863, 1766]. 3D Gaussian Splatting [121] has since become the representation of choice for feed-forward reconstruction, in PixelSplat [1800], Splatter Image [1864], MVSplat [1801], Flash3D [1865], LatentSplat [1737], Splatt3R [1866], GS-LRM [1867], NoPoSplat [1868], and DepthSplat [675]. These methods synthesize novel views with high appearance fidelity, but the views remain confined to small camera changes within the observed envelope, which motivates the generative paradigm below.

Complete 3D Reconstruction. A harder regime is to reconstruct a complete scene from an observation $\mathbf{O}$ that covers only part of it, which requires inferring both visible and occluded regions; the operator shifts from a deterministic map to a generative posterior over the unobserved regions,

which shares the formal structure of the synthesis sampler of Eq. (8.9) with the conditioning replaced by a strong observation constraint $\mathbf{O}$.

Complete reconstruction has been studied extensively for single objects [1869, 1771, 1870, 1774, 1871, 1721, 1786, 1776, 1787, 1872] but remains under-explored for large scenes. Classical scene-scale methods fuse depth maps from multiple views into a volumetric representation [109, 114, 392, 334, 170], but they are sensitive to depth errors and produce noisy, incomplete reconstructions. A newer line fine-tunes large pretrained diffusion models [185, 1873] for the task, conditioning on controlled camera trajectories [1811, 1874, 1875, 1757, 1876, 1808] or on auxiliary signals [1877, 1812, 1802, 1447]; these methods synthesize observation-consistent views but leave the underlying geometry implicit. A small but growing body of work addresses geometry directly: WVD [1878] and Bolt3D [1879] generate a hybrid RGB + point-map output that recovers geometry and appearance rather than novel views alone, and LaRI [1880] reconstructs a complete scene from a single image, still limited to small indoor scenes.

Two design choices cut across the three paradigms. The density of the observation decides which paradigm applies: a dense calibrated multi-view set calls for per-scene optimization, a sparse uncalibrated capture for a feed-forward map, and a single image for a generative posterior, since progressively more of the scene must be inferred rather than observed. The output representation $\mathcal{R}^{\text{rec}}$, by contrast, is largely independent of the paradigm and driven by the downstream consumer: a point map (DUSt3R), a mesh (classical MVS post-processing), a NeRF (per-scene optimization, PixelNeRF), a 3DGS field (3DGS, PixelSplat), or a hybrid RGB + point map (Bolt3D, WVD).

Training Corpora. Synthetic corpora such as Objaverse [1764] supply large and diverse 3D assets, and MegaSynth [1881] and Zeroverse [1882] explore procedurally generated data for scalable supervision. Real-world static corpora include ARKitScenes [669] and ScanNet++ [939] for indoor capture, dynamic corpora such as KITTI-360 [662] capture outdoor scenes with temporal variation, and TartanAir [666] contains both static and dynamic scenes, which permits evaluation across both settings.

Evaluation Benchmark Suites. Iterative methods (P1) are evaluated on DTU [1174] and Tanks-and-Temples [1173] for geometric fidelity and on the NeRF-era benchmarks LLFF [1883], MipNeRF360 [1831], and RealEstate10K [734] for novel-view quality. Feed-forward methods (P2) adopt the same panel together with ScanNet [448] subsets that supply ground-truth point maps for direct geometry supervision. Complete-reconstruction methods (P3) have no standardized suite and typically report on the precursor benchmark of their underlying 2D or video prior.

Constraint-specific Metrics. Four axes mirror the rendering-versus-geometry decomposition of Eq. (8.21). Photometric quality is measured by PSNR, SSIM [1884], and LPIPS [1885], covering pixel-level fidelity, structural similarity, and perceptual consistency of novel views. Camera-pose adherence uses Relative Translation Accuracy (RTA), Relative Rotation Accuracy (RRA), and the Area Under the Curve (AUC) that summarizes pose accuracy across angular thresholds. Geometric fidelity of point maps and point clouds is measured by Accuracy (precision), Completeness (recall), and Chamfer Distance, where Accuracy reflects geometric precision, Completeness measures surface coverage, and Chamfer Distance aggregates both. Depth quality for monocular estimation uses the absolute relative error $|y-\hat{y}|/y$ and the inlier ratio $\delta\!<\!T$ (the fraction of predictions within a multiplicative threshold of the ground truth), with median scaling or least-squares fitting applied beforehand because monocular depth is scale-ambiguous; multi-view depth reports Accuracy, Completeness, and Overall, the mean of the first two and equivalent to Chamfer Distance.

The paradigm succession of §8.2.3.2 is visible in two quantities. Per-scene optimization has moved from NeRF [119], which trains for hours and renders at a few frames per second, to 3D Gaussian Splatting [121], which trains a scene in minutes and renders in real time (${\geq}30$ fps at 1080p); the feed-forward paradigm removes per-scene optimization altogether, with DUSt3R / MASt3R [122, 728] and VGGT [123] recovering point maps, depth, and camera poses in a single sub-second forward pass and Fast3R [732] processing over a thousand images at once. On pose, learned reconstructors now rival classical pipelines: VGGT [123] reports a camera-pose AUC@30 of 85.3 on RealEstate10K in pure feed-forward mode and 93.5 with bundle-adjustment refinement. Two caveats apply. There is no cross-paradigm leaderboard, since P1 is scored on novel-view PSNR / SSIM / LPIPS, P2 adds geometric Accuracy / Completeness / Chamfer Distance and pose AUC, and P3 inherits whichever protocol its 2D or video prior used; and on raw depth-map accuracy feed-forward models still trail tuned MVS pipelines, with quality degrading under free-viewpoint extrapolation (§8.2.3.4), so the strongest numbers hold inside the observed view envelope. Phone-based NeRF and Gaussian-Splatting capture apps and the open-source Nerfstudio stack have brought per-scene reconstruction to commodity devices, while classical photogrammetry suites remain the production baseline for metric accuracy.

Limited Modality in Datasets. Most reconstruction and view-synthesis datasets cover a narrow band of modalities. Widely used benchmarks such as RealEstate10K [734] and MipNeRF360 [1831] are RGB-only, without depth, LiDAR, or semantic annotation, and even Objaverse-XL [1765], with 10.2M objects, is primarily synthetic mesh data without real-world sensor modalities. Merging datasets of different modalities is the usual workaround, at the cost of domain shift and annotation inconsistency, and the gap is widest for dynamic scenes: datasets with synchronized RGB, depth, optical flow, and 3D motion remain far fewer than static ones, and most dynamic datasets record either camera motion or object motion rather than both with a full sensor suite. The open problem is a scalable, modality-rich corpus that combines the diversity of Objaverse [1764] with the multi-sensor completeness of ScanNet++ [939], collected under one protocol so that aligned RGB, depth, and semantic signals supervise the same model.

Reconstruction Accuracy. Feed-forward models still trail traditional MVS pipelines [1886, 1821] on raw depth-map accuracy, because MVS exploits known camera parameters and a hypothesized depth set to build cost volumes from which depth is predicted. Whether feed-forward reconstructors [732, 123] learn an approximation of such cost volumes implicitly in their attention layers is an open question; injecting explicit camera parameters or other priors into the backbone, for instance through Diffusion Transformers [1645], is one route to closing the gap.

Free-viewpoint Rendering. The accuracy gap is most visible at free-viewpoint rendering, where the model must produce novel views far from the input camera distribution. Disocclusions, geometric uncertainty, and limited generalization conspire: regions never observed come out blurred, ghosted, or geometrically wrong, because current methods rely on local consistency and cannot infer plausible content for occluded areas, and view-dependent effects and complex light transport demand reasoning beyond interpolation. Hallucinating missing detail while keeping novel views consistent is exactly the generative regime of §8.2.3.2.

Long Context Input. Reconstruction from many views meets a hard computational ceiling: methods for geometry reasoning and novel-view synthesis rely on full attention, whose compute grows quadratically with the number of tokens; exact-attention kernels such as FlashAttention-2 [1887] remove the quadratic memory of the attention matrix but not the quadratic compute. VGGT [123], for example, needs about 21 GB of GPU memory for 100 input frames and about 41 GB for 200, so inference over long sequences and training on many views quickly exceed a single GPU. Recurrent alternatives such as CUT3R [731] integrate new views incrementally while maintaining a memory state, which keeps inference memory near constant, but they forget earlier views and degrade as the number of inputs grows. Reasoning efficiently over hundreds or thousands of views while keeping memory and compute manageable remains open, with recurrent and sparse-attention architectures the main candidates.

Panoramic Image Formats. The projection $\Pi$ fixes how the sphere is laid out for a 2D network and which distortions the generator must absorb. Equirectangular projection (ERP) maps the sphere onto a rectangle in latitude–longitude coordinates; it stretches content near the poles but is continuous and directly compatible with standard 2D pipelines and convolutional networks, which has made it the dominant format for learning-based synthesis. Cubemap projection represents the sphere as six perspective faces (front, back, left, right, top, bottom), removing polar distortion at the cost of discontinuities along face boundaries, and is the native format of real-time rendering and VR / AR systems, where each face is processed as an ordinary perspective image. Tangent images and perspective patches decompose the panorama into overlapping tangent-plane projections so that models trained on ordinary perspective images can be reused. Fisheye and dual-fisheye projections capture 180–220 degrees per lens with radial distortion; most consumer 360-degree cameras capture two hemispheres this way and stitch them, so dual-fisheye is the native acquisition format even when methods operate on ERP.

Panoramic Image Synthesis. Two task formulations differ in their conditioning: text-to-panorama generation, which leans on the prior of a pretrained generator and is judged on semantic alignment and diversity, and image-conditioned panorama generation, which expands an NFoV image or an incomplete panorama by inpainting or outpainting and must preserve the content, style, and panoramic distortion of its input. Formally, this family is the base case of Eq. (8.25) with the lifting operator disabled: it samples a single static omnidirectional image and stops,

where the generative work is carried out under a chosen projection $\Pi\in\{\text{ERP},\text{cubemap},\text{tangent}\}$ that dictates how spherical distortion is handled; Figure 8.10(a) depicts this static-panorama pathway.

Early approaches [1888, 1889] relied on GANs [1646] and were constrained by limited training data and mode collapse; diffusion models [184, 185] brought diversity and fidelity, and current methods divide by projection format and architecture. The three strategies below are three realizations of the sampler of Eq. (8.26), differing in how the spherical output is factorized before being mapped onto the sphere,

where $\Pi^{-1}$ lifts the generated representation onto the sphere and $\mathcal{S}$ is a spherical stitch-and-blend operator. The first two cases synthesize the panorama directly (and so require panoramic training data), whereas the third reuses a frozen perspective generator $p_{\theta}^{\text{2D}}$ and pushes the cross-view-consistency burden into $\mathcal{S}$.

$\blacktriangleright$ Equirectangular projection (ERP) methods. The direct-ERP case generates on the latitude–longitude grid and must handle polar distortion and the wrap-around seam architecturally. For text-to-panorama generation, DiffPano [1890] fine-tunes a diffusion model with LoRA on ERP, SMGD [1891] guides the diffusion process with spherical manifold convolutions that respect geometry across the sphere, and PAR [1892] replaces diffusion with autoregressive modeling to avoid the mismatch between ERP and the noise assumptions of standard diffusion. The seam is treated at the denoising or decoding stage: PanoDiff [1893] uses geometric augmentation when completing a panorama from unregistered NFoV images, PanoDiffusion [1894] generates depth jointly and rotates the latent circularly to enforce left–right continuity, StitchDiffusion [1895] re-denoises the wrap-around region at every step together with global cropping, and PanFusion [1896] pairs an ERP branch with a multi-view perspective branch to draw on both representations. For image-conditioned generation, OmniDreamer [1897] introduced mask-based outpainting from NFoV images with GANs, AOG [1898] and ODIS [1899] explore alternative conditioning for inpainting and outpainting, and PanoDecouple [1900] separates distortion guidance from content completion to counter the “visual cheating” in which a model buys image quality at the cost of geometric accuracy.

$\blacktriangleright$ Cubemap-based methods. The second case of Eq. (8.27) generates six perspective faces, which removes polar distortion but moves the problem to the face boundaries. CubeDiff [1901] introduced the cubemap representation for text-to-panorama generation, and DreamCube [1902] synchronizes the six planes to suppress discontinuous seams and inconsistent color tone; overlapping the fields of view of adjacent faces reduces seams further but duplicates generation in the overlap and lowers the effective resolution. JoPano [1903] models the faces jointly with a DiT and a Joint-Face Adapter, applies Poisson blending along the boundaries, and introduces the seam metrics (Seam-SSIM, Seam-Sobel) used in §8.2.4.3.

$\blacktriangleright$ Perspective image-based methods. The third case leaves a perspective text-to-image model untouched and pushes the consistency burden into the stitch operator $\mathcal{S}$, with or without panorama-specific training. Training-free methods apply a pretrained perspective model directly: MultiDiffusion [1692] denoises overlapping patches independently and blends them, PanoGen [1904] outpaints recursively from an initial view until the sphere is covered, DiffCollage [1905] casts the panorama as a factor graph whose factor nodes are portions and whose variable nodes are their overlaps so that portions are generated in parallel rather than autoregressively, PanoFree [1906] iterates warping and inpainting over multi-view images, and SphereDiff [1907] moves MultiDiffusion onto a spherical latent with dynamic spherical sampling and distortion-aware weighted averaging so that quality holds up to the poles. These methods ride on advances in perspective generation but struggle with global consistency across the full sphere. Training-based methods fine-tune on panoramic data: MVDiffusion [1908] generates eight perspective views with consistent geometry and reprojects them into a closed-loop panorama, and TanDiT [1909] decomposes the panorama into a structured grid of tangent-plane images that a Diffusion Transformer generates jointly in one denoising pass under spatial attention.

Panoramic Video Generation. Extending synthesis over time adds temporal coherence to omnidirectional consistency. Formally, this family extends the static generator of Eq. (8.26) over time, sampling a sequence under a joint spatial-and-temporal coherence constraint,

where $\mathcal{W}^{\text{t}}$ enforces frame-to-frame motion consistency on top of the per-frame 360-degree wrap-around continuity; Figure 8.10(b) depicts this temporally-extended pathway. The methods share this generator but differ in the conditioning argument $\mathbf{c}$ supplied to Eq. (8.28), which separates them into three input modalities,

where $\mathbf{x}$ is a text prompt, $P_{0}$ a static panorama anchor reused as the first frame, and $(I_{t},\mathcal{M}_{t})$ an NFoV frame paired with its visibility mask. The difficulty grows down the list: as the conditioning becomes spatially sparser, the model must hallucinate progressively more of the sphere while preserving the temporal coherence of Eq. (8.28).

$\blacktriangleright$ Text-to-panoramic video generation. 360DVD [1910] adapts a pretrained text-to-video model to the ERP domain through adapter modules, keeping wrap-around consistency across frames but struggling to balance text alignment against spherical geometry and to vary style under several control conditions. TiP4GEN [1911] adopts a dual-branch panoramic-and-perspective architecture in the spirit of PanFusion and lifts the generated video into a geometry-aligned 4D Gaussian scene. PanoWan [1912] and PanoDiT [1913] fine-tune large video diffusion transformers on panoramic video data and gain motion quality from scale, and PanoLora [1914] treats the task as a projection adaptation from perspective views that LoRA can learn with little data. Training-free alternatives carry the static-panorama machinery into time: SphereDiff [1907] extends its spherical latent to video, and DynamicScaler [1915] denoises through a seamless rotating window with an offset-shifting scheme and Global Motion Guidance, which yields arbitrary output resolution at constant memory. Text alone, however, offers little control over layout and motion, which motivates the richer conditioning below.

$\blacktriangleright$ Image-to-panoramic video generation. Animating a static panorama anchors the video spatially and asks for plausible omnidirectional motion that respects the input geometry. 4K4DGen [1916] animates user-selected regions with an image-to-video model under overlapping windows, at the cost of fixed-window artifacts and limited motion. ViewPoint [1917] introduces a panorama representation that keeps global spatial continuity while Pano-Perspective attention transfers a perspective video prior, and HoloTime [1918] reconstructs the result as 4D Gaussian Splatting so that the animated scene can be explored under viewpoint changes. GenEx [1919] generates a 3D-consistent imaginative environment from a single image as a panoramic video stream with loop consistency over long trajectories, which makes it usable for embodied exploration. For explicit control, PanFlow [1920] conditions on synthetic optical flow as a scene-agnostic motion signal and warps noise spherically so that camera and object motion can be controlled separately while the seam stays consistent; OmniDrag [1921] adds drag-style interaction at scene and object level through a spherical motion estimator; and PanoWorld-X [1922] reprojects ERP features onto the sphere inside a Sphere-Aware Diffusion Transformer to follow diverse camera trajectories.

$\blacktriangleright$ Perspective-to-panoramic video generation. Expanding an NFoV video to a full 360-degree video requires coherent hallucination in space and time at once, under visibility masks that change with camera motion. VidPanos [1923] frames the task as space-time outpainting and uses a generative video model to fill the regions a panning camera never observed; Imagine360 [1924] combines a global panoramic branch with local perspective priors and an elevation-aware design for the moving masks; and Beyond the Frame [1925] adds geometry-aware modules and motion-aware conditioning to improve realism and motion consistency in the outpainted regions.

Panoramic 3D Reconstruction and Generation. Reconstruction bypasses the generator of Eq. (8.25): rather than generating $\hat{P}$, it takes observed posed panoramas and applies the lifting operator directly,

where $\Lambda$ is realized either as a per-scene optimization (iterative) or as an amortized network (feed-forward), mirroring the split of §8.2.3; Figure 8.10(c) depicts this reconstruction pathway. Generation then composes the two directions, feeding a synthesized panorama into the same $\Lambda$.

$\blacktriangleright$ Iterative optimization-based panoramic reconstruction. Realizing $\Lambda$ of Eq. (8.30) as a per-scene optimization,

re-solved from scratch for each scene, these methods fit a radiance field or an explicit representation to multi-view panoramas, the panoramic analogue of §8.2.3.2. NeRF-based work includes 360Roam [1926] for geometry-aware indoor reconstruction, 360FusionNeRF [1927] for novel views from a single panorama under geometric and semantic supervision, EgoNeRF [1928] with spherical grids for egocentric capture, and OmniNeRF [1929], which processes raw dual-fisheye input and its lens distortion. Gaussian-splatting alternatives are faster: 360GS [1930] projects Gaussians onto spherical tangent planes, ODGS [1931] rasterizes on tangent planes with elevation-aware densification, OP43DGS [1932] corrects the projection bias of the local affine approximation across camera models, OmniGS [1933] exploits omnidirectional optical properties, and Seam360GS [1934] calibrates dual-fisheye input by jointly optimizing translation vectors and learnable angular-distortion embeddings. Scene4U [1935] reconstructs from a single panorama in depth layers: an open-vocabulary segmenter and an LLM decompose it semantically, diffusion inpainting restores occluded regions, and 3D Gaussians are fitted per layer.

$\blacktriangleright$ Feed-forward generalization-based panoramic reconstruction. Amortizing $\Lambda$ of Eq. (8.30) into a trained network applied in a single forward pass,

where $\Lambda_{\psi}$ is trained once over a panoramic corpus (the analogue of §8.2.3.2), trades the per-scene cost of Eq. (8.31) for a learned prior that reconstructs from sparse panoramas without scene-specific training. PanoGRF [1936] pioneered feed-forward reconstruction of wide-baseline panoramas with depth guidance; PanoSplatt3R [1937] extends DUSt3R [122] to unposed wide-baseline panoramas through RoPE rolling, a rotary positional embedding rolled across attention heads so that ERP periodicity and correspondences across the seam are modeled; and PanSplat [1938] builds spherical 3D Gaussian pyramids on a Fibonacci lattice to reach 4K novel views with feed-forward inference and cubemap rendering for memory efficiency.

$\blacktriangleright$ Panorama-based 3D generation and world modeling. This family realizes the full composition of Eq. (8.25): it first generates the panorama and then lifts it,

so that the synthesized omnidirectional image of Eq. (8.26) becomes the structured intermediate fed to the reconstruction operator of Eq. (8.30); Figure 8.10(d) depicts this generate-then-lift pathway. Text $\rightarrow$ panorama $\rightarrow$ 3D pipelines generate the omnidirectional image with a diffusion model [1890, 1901] before reconstructing a NeRF, Gaussian, or mesh scene. LayerPano3D [1939] parses the panorama into hierarchical depth layers and reconstructs each with 3D Gaussian Splatting, DreamScene360 [1940] fits panoramic Gaussians for free-roaming text-conditioned scenes, Matrix-3D [1941] couples trajectory-guided panoramic video generation with a dual-pathway reconstruction that conditions on scene-mesh renders, and HunyuanWorld [1942] treats the panorama as a 360-degree world proxy for text- and image-conditioned explorable scenes with mesh export and disentangled object representations. The panoramic video generators GenEx [1919] and PanoWorld-X [1922] (above) turn the same intermediate into an action- or trajectory-conditioned observation space, which is where panorama synthesis meets the world models of §8.6.

Training Corpora. For static synthesis, Structured3D [1943] (synthetic indoor scenes with controlled geometry) and SUN360 [1944] (real-world indoor and outdoor panoramas) are the standard benchmarks for distortion handling and seam artifacts, in the splits adopted by JoPano, PanoDecouple, and DreamCube [1903, 1900, 1902]; Matterport3D [340], with RGB-D panoramas, camera poses, and mesh geometry across 90 buildings, is the most widely used corpus for both generation and reconstruction [1938, 1939, 1942, 1908, 1896], and Upright360 [1939] filters Matterport3D, web, and Blockade Labs panoramas with GeoCalib [1945] to near-zero pitch and roll for realistic indoor synthesis. Panoramic video with geometric annotation has been the bottleneck. 360+x [1946] and WEB360 [1910] are small real-world collections (the latter captioned for text conditioning), 360-1M [1876] scales to over a million videos for world-model pretraining, PanFlow [1920] adds optical flow and camera poses on motion-rich sequences, PANOVID [1912] aggregates these with Imagine360 [1924] and public VR videos but lacks poses and depth, and Matrix-Pano [1941] fills that gap with Unreal Engine 5 renders carrying depth maps and camera trajectories; embodied work renders action-conditioned observations in Unreal Engine directly [1919, 1922]. For reconstruction, Replica [339] and Residential [1947] serve as synthetic indoor benchmarks at baselines of about 1.0 m and 0.3 m respectively [1938], 360Loc [1948] supplies posed panorama sequences used for 4K fine-tuning [1938], and the scarcity of posed real panoramic video led PanSplat [1938] to capture its own dual-fisheye footage with poses from visual SLAM.

Evaluation Metrics. Panorama evaluation combines general image and video quality metrics with panorama-specific measures of distortion, seam consistency, and spherical topology.

$\blacktriangleright$ Image fidelity and semantic alignment. FID measures distributional similarity between generated and real panoramas [1903, 1939, 1900, 1909]; PanFusion [1896] shows that FAED (Fréchet Auto-Encoder Distance), computed in the latent space of an auto-encoder trained on the target data, tracks panorama quality more closely than Inception features do. Text-to-panorama generation reports CLIP-Score between prompt and output [1942, 1903, 1939, 1909], image-conditioned tasks report CLIP-I between input and output [1942], and no-reference quality is scored with BRISQUE and NIQE [1942, 1939, 1941], complemented by aesthetic scorers and human preference studies, for instance the Average User Ranking protocol of LayerPano3D [1939]. Layout-conditioned synthesis measures adherence by 3D and 2D IoU against layouts recovered with HorizonNet [864], following PanFusion [1896].

$\blacktriangleright$ Panorama-specific consistency. Projection-induced distortion and boundary discontinuities call for metrics of their own. JoPano [1903] scores cubemap seams with Seam-SSIM (SSIM between adjacent edge pixels of neighboring faces) and Seam-Sobel (gradient discontinuity along the seam); PanoDecouple [1900] introduces Distort-CLIP, a CLIP variant trained to detect panoramic distortion, so that the “visual cheating” of models that improve image quality while degrading geometry becomes measurable; and loop consistency across the 360-degree wrap-around is assessed by PSNR in overlapping regions [1908] or across the left–right boundary [1920, 1912].

$\blacktriangleright$ Reconstruction and novel-view synthesis. Because pixel density on the sphere is non-uniform, WS-PSNR (Weighted Spherical PSNR) weights pixels by latitude and replaces plain PSNR for equirectangular outputs [1938, 1936], complemented by SSIM and LPIPS [1939, 1938, 1941].

$\blacktriangleright$ Video quality and spatiotemporal dynamics. Panoramic videos are often projected to several perspective views (e.g., , four views at azimuths $\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\}$ with a $90^{\circ}$ field of view) so that human raters and perspective-domain tools can be used [1942, 1939, 1924]. Q-Align [1949], a large multimodal model trained to score visual quality on discrete text-defined levels, is increasingly used for images and videos alike [1942, 1924]; FVD [1950], computed in the feature space of an inflated 3D ConvNet, is the standard distributional metric for video [1920, 1912]; VBench [1951] adds imaging quality, aesthetic quality, motion smoothness, and subject consistency over projected views; and trajectory-conditioned generation is scored by frame-to-frame consistency and by the range of camera motion it sustains [1922]. Two limits remain. Ground truth is scarce for dynamic panoramic video with depth and camera annotation, so evaluation falls back on projected perspective views or user studies; and world-level properties such as loop consistency over long trajectories and action-conditioned spatial coherence have no dedicated benchmark yet.

Unique Challenges in Panoramic Synthesis. Modeling a complete omnidirectional environment imposes constraints that NFoV synthesis never meets. The first is distortion-aware learning: ERP stretches content near the poles, so visual information is non-uniformly distributed over the image and convolutional kernels designed for translation invariance on the plane struggle with the varying distortion, which motivates spherical manifold convolutions and latitude-aware architectures [1891, 1900]. The second is seamless boundary consistency: wrap-around coherence across the longitudinal boundary must be preserved to avoid visible seams, typically through circular padding or latent rotation during denoising [1920, 1912]. The third is scaling structured generation to 4K and beyond while keeping global geometric and semantic consistency over the whole sphere, which pushes toward memory-efficient attention and hierarchical spherical representations [1938, 1915].

From Panorama to World Models. The clearest trend is the move from static spherical snapshots toward interactive 3D world modeling, in which the panorama is the interface through which an embodied agent reasons about a complete environment rather than a fragment of one. The systems that lift a generated panorama into an explorable world (HunyuanWorld [1942], Matrix-3D [1941]) and World Labs’ Marble [158] show what panoramic priors make possible, and temporal panoramic modeling supplies the 4D half: generating temporally coherent panoramic video requires motion dynamics that respect spherical topology and occlusion boundaries [1920, 1922]. Integrated with interactive systems, such panoramic world simulations become a basis for embodied interaction, long-range spatial reasoning, and environment simulation in robotics and immersive media.

Representation Innovations. Efficiency and fidelity both hinge on the omnidirectional representation. Efficient omnidirectional representations such as the Fibonacci-lattice Gaussian pyramids of PanSplat [1938] render at high fidelity without the redundancy of uniform grids, and multi-modal panoramic fusion, which binds 360-degree RGB to depth, semantic labels, and geometric proxies such as surface normals [1939, 1935], makes the synthesized environment both more realistic and more editable, closing part of the gap between 2D generative priors and full 3D scene intelligence.

2D-lifted Iterative Editing. A dominant paradigm inherits from instruction-based NeRF editing [1953, 1954, 1955, 1956]: the method iterates between rendering views from the current scene, applying a 2D diffusion-based editor to them, and updating the 3D representation to match the edited guidance under geometric regularization. Formally, this family instantiates $E_{\theta}$ of Eq. (8.34) as a per-scene optimization that drives the rendered views toward the output of a frozen 2D editor $\mathcal{E}_{\text{2D}}$ (the InstructPix2Pix-style operator of §8.1.3),

so that the 2D edits are lifted into the 3D representation and the $\arg\min$ is re-solved from scratch for every scene; Figure 8.12(a) depicts this render-edit-update loop. Instruct-NeRF2NeRF [1957], EditNeRF [1958], Free-Editor [1959], and CTRL-D [1960] lift instruction-driven 2D edits into the radiance field this way, and with the rise of 3D Gaussian Splatting (3DGS) [121] the same principle moved to Gaussian primitives: Instruct-GS2GS [1961] iteratively edits the training images with InstructPix2Pix [1711] and re-optimizes the 3DGS to reflect the instruction while preserving 3D structure [1962, 1963]. The recipe is simple and borrows the full strength of 2D editors, but it is slow, prone to view-dependent artifacts, and only as good as the stability of the 2D editor and the coverage of the rendered guidance views.

Multi-view-consistent Guidance. Because independently edited views can disagree (inconsistent textures, hallucinated geometry), recent work [1964] enforces multi-view consistency at the guidance level rather than relying on the 3D update to reconcile conflicting edits. Formally, rather than editing each view independently as in Eq. (8.36), this family makes the guidance images mutually consistent before the lift,

where $\mathcal{W}_{u\to v}$ warps view $v$ into view $u$ using geometry or depth derived from the current 3D estimate, penalizing cross-view disagreement; the consolidated $\{\hat{I}_{v}\}$ then drive the 3D update of Eq. (8.36), as in Figure 8.12(b). GaussCtrl [1965] edits several rendered views in a coupled fashion rather than sequentially, improving multi-view consistency for text-driven 3DGS editing [1962, 1966], and VcEdit [1962] integrates the 3DGS into the image-editing process itself with consistency modules that counter cross-view disagreement and mode collapse. A complementary line transfers a single-view edit to other views via geometry: Gomel and Wolf [1967] extracts diffusion attention features from a reference edit, warps them across views using depth from the Gaussian splats, and injects the warped features to propagate a coherent edit. These methods preserve 3D coherence more robustly but still struggle with large non-rigid shape changes or severe occlusion, where reliable correspondence and geometry are hard to come by.

3D-native Primitive Editing. To reduce reliance on unstable 2D guidance and enable interactive use, 3D-native approaches operate directly on explicit scene primitives. Formally, this family bypasses the rendering loop of Eq. (8.36) entirely: a grounding or segmentation operator $\mathcal{G}$ selects the target subset of primitives and a direct geometric transform $\mathcal{T}$ applies the edit,

acting on explicit Gaussians or mesh facets so that the unedited primitives are preserved by construction: the identity-preservation term of Eq. (8.35) is satisfied structurally rather than by a penalty; Figure 8.12(c) shows this pathway. 3DSceneEditor [1968] runs a fully 3D pipeline, combining instance segmentation and zero-shot grounding to localize the target and then performing Gaussian-level addition, deletion, relocation, recoloring, and replacement with strong controllability in real time, and InterGSEdit [1969] adds user-in-the-loop key-view selection and a geometry-consistent attention prior for iterative, preference-guided 3DGS editing with control over the edit degree. More broadly, GaussianEditor and 3DSceneEditor [1970, 1968] impose higher-level structure on the Gaussians, grouping them by object identity or semantic coherence (typically through zero-shot grounding or segmentation [1971, 1972] followed by anchoring) so that an edit can be formulated at the level of an instance rather than of individual primitives, which localizes the target more precisely and protects the background.

LLM-integrated Editing. Large language models have been integrated into 3D editing pipelines not merely as text encoders but as high-level reasoning and control modules. Formally, the LLM decomposes the instruction into an ordered program of geometry-aware operators applied to the asset,

so that high-level reasoning (attribute inference, edit planning, dependency resolution) is separated from the low-level operators $o_{k}$, often the primitive transforms of Eq. (8.38), that it dispatches; Figure 8.12(d) depicts this plan-then-execute pathway. ShapeLLM-Omni [1973] casts 3D editing as language-grounded reasoning in which the LLM performs shape understanding, attribute inference, and edit planning before invoking geometry-aware operators, and LL3M [1974] extends this to multimodal 3D manipulation requiring compositional understanding of geometry, appearance, and spatial relations. A complementary line [1975] uses the LLM to guide or parameterize image-to-3D or text-to-3D backbones in a feed-forward setting rather than a per-scene optimization, and LLMs also serve as coordination agents for global consistency [1976, 1614]: Xia et al. [1966] shows that reasoning over object identities, relations, and edit dependencies reduces the conflicts that arise in purely diffusion- or optimization-based pipelines when several objects and attributes must be edited at once.

Efficient and Training-free Editing. A final trend reduces test-time cost through feed-forward reconstruction or reuse of pretrained models [1977, 1978]. These strategies are distinct: a training-free editing pipeline can still involve iterative diffusion inference. The amortized feed-forward case evaluates the editing operator of Eq. (8.34) directly,

eliminating the per-scene $\arg\min$ of Eq. (8.36) by amortizing inference into a trained network. Training-free alternatives instead reuse pretrained components, potentially through single-view guidance, mesh proxies, or iterative native-3D latent editing; Figure 8.12(e) distinguishes these routes from the per-scene optimization loop in panel (a). Free-Editor [1959] edits a NeRF zero-shot from text without scene-specific retraining, countering multi-view style inconsistency with a “single-view editing” scheme and an Edit Transformer; on the scene-operation side (insert, replace, delete), FreeEdit [1979] uses a mesh proxy to perform training-free edits guided by foundation models at near real-time rates in complex indoor scenes. These methods improve usability but typically trade away edit fidelity and coverage, especially for fine-grained geometry, and depend strongly on robust grounding and localization in 3D.

Evaluation Metrics. Evaluation considers geometric fidelity, appearance quality, and semantic alignment with the instruction. Geometry is measured by Chamfer Distance (CD) [1975], lower being better, and by F-score [1966], which captures surface overlap at a distance threshold. Texture and appearance are assessed by comparing renderings of the edited representation against target or inpainted images with SSIM [1884], LPIPS [1885], and DreamSim [1981] [1982]. Visual–text alignment is scored with CLIP- or DINO-based embedding similarity [1983], and GPTEval3D [212] uses GPT-4V to compare multi-view renderings of edited shapes on text alignment, 3D plausibility, and texture detail. To measure the edit relative to the input rather than in isolation, directional CLIP (CLIP${}_{\text{dir}}$) [1963] checks that the change in image embedding, computed over multi-view renderings of input and output, follows the change in text embedding.

Edit cost is the clearest axis separating the five families. Instruct-NeRF2NeRF [1957] takes on the order of 30–90 minutes per edit; GaussianEditor [1970] brings this under 20 minutes by operating on explicit Gaussians; PrEditor3D [1977] edits a shape in a few minutes, and VoxHammer [1978] performs precise local edits training-free in a native 3D latent space. On quality the field reports DINO-I and CLIP-T on Edit3D-Bench [1978], directional CLIP [1963], and GPT-4V judgment [212]; the 3D-native and feed-forward families lead on locality and speed, while 2D-lifted methods keep an edge on open-vocabulary appearance edits where a strong 2D editor does the work. The speed of feed-forward editing comes at a cost in fine-grained geometry control and a dependence on robust 3D grounding, taken up below.

Geometry-level, not merely appearance, editing. Most current methods edit color and texture well but treat geometry as fixed: precise, instruction-driven changes to shape, including thin, transparent, or high-frequency structures, remain unreliable, and edits that should alter the underlying surface often only repaint it. Closing this gap requires editors that operate on geometry and appearance jointly rather than deferring geometry to a frozen reconstruction.

Large non-rigid and structural edits. Adding, removing, or substantially deforming structure while preserving the identity of the rest of the asset is still difficult: drag-driven, as-rigid-as-possible Gaussian editors [1984] handle moderate deformation, but large non-rigid motion and topology changes frequently lose original detail. Robust structural and compositional editing, including relation-aware multi-object edits, remains an active frontier.

Material–lighting disentanglement and relightable editing. Because most representations bake illumination into appearance, an edit that changes an object’s material or color is not guaranteed to stay consistent when the scene is relit. Relightable representations that decompose geometry, BRDF, and incident light [1985] are a prerequisite for edits that survive relighting and for inserting edited content into new lighting, yet stable material–light decomposition under editing is not solved.

The consistency bottleneck has moved to 3D grounding. Now that native-3D editing handles appearance coherence, the limiting factor is localization: methods such as VoxHammer [1978] are highly sensitive to 3D mask accuracy and degrade rapidly when the target region is imprecisely grounded. Robust open-vocabulary 3D grounding and segmentation [1971, 1972], ideally without user-provided masks [1966], is therefore the new gate on edit quality, and it is most acute in dynamic scenes, where the edited region must be tracked with temporal identity preserved across frames [1960].

Two pressures cut across all four. Evaluation remains immature: the benchmarks of §8.2.5.3 are small (Edit3D-Bench annotates 100 source models and 300 edits) and no metric panel is yet standard, so progress on the harder edits above is difficult to measure reliably. And the strongest forward trend is unification, bringing generation, segmentation, grounding, and editing into a single structured pipeline [1986] with interfaces that go beyond text to reference images [1983], masks [1987], sketches [1955], and object-level operations, so that an agent can specify and verify a geometry-, material-, and relighting-consistent edit rather than only a repaint.