4.23D

RGB-D images are a popular 2.5D representation that captures both photometric and geometric information from a single viewpoint. This data format consists of two co-registered images: a standard RGB color image providing appearance and texture details, and a corresponding depth map where each pixel value encodes the distance from the sensor to the corresponding point in the scene. Formally, it can be defined as a function mapping a 2D image grid $\Omega$ to a 4D space combining color and depth:

where the first three channels represent RGB color and the fourth channel encodes the distance from the sensor to a point in the scene. In practice, RGB-D data is stored as a tensor $I\in\mathbb{R}^{H\times W\times 4}$.

The primary advantage of RGB-D data lies in its efficiency and accessibility. It is captured by low-cost sensors such as Microsoft Kinect or Intel RealSense [260], and its image-like grid structure enables the direct application of powerful 2D convolutional neural networks (CNNs) to tasks such as semantic segmentation and object detection [261, 262, 263]. This fusion of color and depth provides a rich, multi-modal input for scene understanding. However, a significant limitation is its view-dependent nature; an RGB-D image represents only a partial view of the 3D world, suffering from occlusions and providing no information about surfaces hidden from the camera. Despite this, its real-time acquisition and processing capabilities make it indispensable in robotics for navigation [264] and grasping [265, 266, 267], in augmented reality (AR) for scene interaction, and as a foundational input for 3D reconstruction pipelines that fuse multiple frames into a complete model [114].

The voxel grid is a foundational 3D representation that discretizes space into a uniform grid of volumetric elements, or voxels (a portmanteau of “volume” and “pixel”). Analogous to a pixel in 2D, a voxel grid can be modeled as a function mapping discrete 3D coordinates to a set of attributes:

where $D,H,W$ are the grid dimensions and $\mathcal{A}$ can represent occupancy, color, density, or material properties, thereby enabling a detailed volumetric description of an object.

A key advantage of this representation is its ability to model complex geometries and internal structures, which are often challenging for surface-based methods like polygon meshes to capture. The conceptual simplicity of voxels, which can be seen as the three-dimensional extension of 2D pixels, facilitates straightforward implementation and manipulation. Consequently, voxel grids are natively suited for applications involving inherently volumetric data, such as medical imaging (e.g., , MRI and CT scans) [268], where biological tissues are naturally captured as a 3D array of values. They are also widely adopted in computer graphics and gaming for tasks like procedural generation and creating destructible environments, as popularized by games like Minecraft.

Point clouds offer a direct and unstructured 3D representation, defined as a set of points in 3D space:

Each point $p_{i}$ consists of its geometric coordinates $(x_{i},y_{i},z_{i})$ and an optional $C$-dimensional attribute vector $\mathbf{a}_{i}$ encoding properties such as color, intensity, or surface normals. They are the native output of active 3D scanning technologies like LiDAR (Light Detection and Ranging) and structured-light scanners [269], as well as passive methods like photogrammetry [115].

The primary strength of point clouds lies in their raw and unstructured nature; they represent geometry without imposing any explicit topological connectivity (unlike meshes) or spatial discretization (unlike voxels). This makes them a faithful and often highly precise representation of the captured data. However, this very lack of inherent structure poses significant challenges for downstream processing. The unordered and potentially sparse distribution of points makes tasks like feature extraction, segmentation, and reconstruction non-trivial, necessitating specialized algorithms and deep learning architectures (e.g., , PointNet [132] and its successors [270]) that are designed to operate directly on irregular point sets.

A mesh is a canonical surface representation in computer graphics, defining an object’s boundary with a set of geometric and topological primitives. Formally, a triangle mesh is a tuple $M=(V,F)$, where

Here, $V$ is a set of vertices representing points in 3D space, and $F$ is a set of faces (typically triangles) that define connectivity between vertices.

The key characteristic of a mesh is its explicit topological structure, which encodes the connectivity between these elements. This structure strikes an effective balance between descriptive power and computational efficiency. Its prevalence is due to several key advantages: meshes can represent complex, smooth surfaces with arbitrary fidelity through adaptive tessellation (i.e., , varying the polygon density); moreover, this representation is natively supported by modern Graphics Processing Units (GPUs), enabling highly efficient real-time rendering. The explicit connectivity also facilitates the straightforward computation of essential surface properties like normals and curvature, which are critical for realistic lighting and physical analysis. However, as a purely surface-based representation, standard meshes are ill-suited for modeling volumetric or internal structures, and representing objects with complex or changing topologies can be algorithmically challenging.

Implicit representations define 3D geometry not as a discrete set of primitives (e.g., , points or polygons), but as the level set of a continuous function:

The surface $\mathcal{S}$ is implicitly defined as the set of points where the function evaluates to a specific value, typically zero:

A particularly powerful and popular form of this is the Signed Distance Function (SDF). An SDF maps any spatial coordinate $\mathbf{x}$ to the closest distance to the surface, where the sign indicates whether the point is inside ($f(\mathbf{x})<0$) or outside ($f(\mathbf{x})>0$) the object. This formulation inherently guarantees watertight, closed surfaces and provides a smooth, continuous description of the geometry, making it resolution-agnostic. A key advantage of implicit fields is their ability to elegantly handle complex and dynamic topologies; operations like merging, splitting, or creating holes do not require complex and error-prone re-meshing procedures. This makes them ideal for physical simulations, such as fluid dynamics and shape manipulation.

This paradigm has experienced a major resurgence with the rise of neural fields or neural implicit representations. Instead of storing the function in a grid, a neural network is trained to approximate it [118] (e.g., , DeepSDF [117]). This approach is memory-efficient and can learn continuous, high-fidelity shape priors from data. Extensions such as Neural Radiance Fields (NeRF) further generalize implicit fields to jointly model geometry and appearance, and are discussed in detail in Section 4.2.6. To visualize or use implicit surfaces in traditional rendering pipelines, an explicit mesh representation is often extracted via algorithms like Marching Cubes [105].

Neural Radiance Fields (NeRF) represent a paradigm shift in novel view synthesis and implicit 3D scene representation [119]. The core innovation is to represent a static scene as a continuous function with a 5D input domain (3D location $+$ 2D viewing direction), parameterized by a simple Multi-Layer Perceptron (MLP). This neural network, denoted as $F_{\Theta}$, maps a 3D spatial coordinate $\mathbf{x}=(x,y,z)$ and a 2D viewing direction $\mathbf{d}=(\theta,\phi)$ to a volume density $\sigma$ and an emitted color $\mathbf{c}=(r,g,b)$. A crucial design choice is that density $\sigma$ depends only on the location $\mathbf{x}$, while the color $\mathbf{c}$ depends on both location and viewing direction $(\mathbf{x},\mathbf{d})$. This allows NeRF to capture complex, non-Lambertian surface effects such as specular reflections and translucency.

To render an image from a novel viewpoint, rays are cast through the scene, and their colors are computed by integrating the predicted color and density values along each ray using principles of classical volume rendering and numerical quadrature. By optimizing the network’s weights to minimize the photometric reconstruction error between rendered and ground-truth images, NeRF can learn a highly detailed and coherent representation of the scene from only a sparse set of posed input images. The resulting state-of-the-art photorealism has established NeRF as a foundational technique, inspiring a vast body of follow-up research aimed at improving its speed, scalability, and applicability to dynamic scenes [120, 271, 272, 273].

3D Gaussian Splatting (3DGS) has recently emerged as a groundbreaking alternative to neural radiance fields, achieving state-of-the-art quality for novel view synthesis while enabling real-time rendering [121]. Unlike the implicit representation of NeRF, 3DGS represents a scene as a set of $N$ anisotropic 3D Gaussian primitives, each parameterized by:

• •

  Position $\bm{\mu}\in\mathbb{R}^{3}$: the 3D center of the Gaussian;

• •

  Covariance $\bm{\Sigma}\in\mathbb{R}^{3\times 3}$: shape and orientation, allowing thin planes through to volumetric wisps;

• •

  Color: Spherical Harmonics (SH) [274] coefficients capturing view-dependent appearance;

• •

  Opacity $\alpha\in[0,1]$: transparency.

The density contributed by a single primitive at a query point $\mathbf{x}\in\mathbb{R}^{3}$ is

$\bm{\Sigma}$ is decomposed into a scaling matrix $\mathcal{S}$ and rotation matrix $\mathcal{R}$, such that

with per-primitive scaling and rotation stored as $s\in\mathbb{R}^{N\times 3}$ and quaternions $r\in\mathbb{R}^{N\times 4}$. The key to 3DGS’s real-time performance lies in the differentiable splatting rasteriser introduced together with the representation [121], which extends earlier differentiable rasterization work on meshes [275]. Instead of costly per-pixel ray marching, 3DGS projects (or “splats”) these 3D Gaussians onto the 2D image plane, creating 2D Gaussian splats. These splats are then sorted and alpha-blended in a highly-parallelized, feed-forward process that is exceptionally well-suited for modern GPU architectures. Differential splatting applies a viewing transform $W$ and Jacobian $J$ to compute the transformed covariance $\bm{\Sigma}^{\prime}=JW\bm{\Sigma}W^{T}J^{T}$, enabling novel view rendering. Each pixel color $\mathcal{C}$ is obtained by blending the $N$ overlapping primitives along the ray:

where $c_{i}$ and $\alpha_{i}$ denote the color and opacity of the $i$-th primitive. The entire model, initialized from a sparse point cloud (typically from Structure-from-Motion), is optimized end-to-end against training images. The optimization process also includes an adaptive density control mechanism to dynamically add, split, or prune Gaussians to best represent the scene’s geometry. This combination of an explicit, flexible primitive with a fast, differentiable rasterizer has set a new standard for performance and quality in 3D scene reconstruction.

To harness the complementary strengths of different 3D representations, hybrid methods have emerged as a powerful paradigm. These approaches combine two or more representations, integrating the high-fidelity and continuous nature of implicit functions with the computational efficiency and explicit structure of primitives like meshes or voxels. This section reviews two influential hybrid techniques: Deep Marching Tetrahedra (DMTet) and Tri-plane.