4.34D

RGBD videos $\mathcal{V}=\{(I_{t},D_{t})\}_{t=1}^{T}$, with $I_{t}\in\mathbb{R}^{H\times W\times 3}$ and $D_{t}\in\mathbb{R}^{H\times W}$, are a straightforward extension of RGBD images (see Section 4.2.1) to the temporal domain. Equivalently, they can be stored as a 4D tensor $\mathcal{V}\in\mathbb{R}^{T\times H\times W\times 4}$, with the last channel holding depth. RGBD videos can be captured using depth sensors, such as Microsoft Kinect or Intel RealSense, which provide synchronized RGB and depth data. These videos are commonly used in applications such as motion capture, gesture recognition, and augmented reality. There are also many learning-based methods [293, 294, 295] that directly predict consistent RGBD videos from monocular video sequences $\mathcal{I}=\{I_{t}\}_{t=1}^{T}$. However, such a representation may struggle to handle complex dynamic scenes with significant occlusions or topology changes. If we want to further reconstruct a unified 4D scene in a canonical space, we may also need to provide or estimate camera poses $\mathcal{P}=\{P_{t}\}_{t=1}^{T}$ for each frame, as well as point trajectories over time to establish temporal correspondences, e.g., via joint pose-trajectory optimization in recent video-based SfM systems [296].

Point clouds are another popular representation for 3D scenes (see Section 4.2.3), and they can be extended to 4D by incorporating temporal information. A 4D point cloud can be represented as $\mathcal{P}=\{(\mathbf{p}_{i}(t),\mathbf{c}_{i}(t))\}_{i=1}^{N}$, where $\mathbf{p}_{i}(t)\in\mathbb{R}^{3}$ is the 3D position of point $i$ at time step $t$, and $\mathbf{c}_{i}(t)\in\mathbb{R}^{3}$ is the corresponding color. To map an uncalibrated, continuous video sequence $\mathcal{I}=\{I_{t}\}_{t=1}^{T},I_{t}\in\mathbb{R}^{H\times W\times 3}$, to a 4D point cloud representation, it generally has two types of representations: (1) a sequence of point clouds $\mathcal{P}=\{\mathcal{P}_{t}\}_{t=1}^{T}$ in a fixed coordinate system, where each $\mathcal{P}_{t}=\{(\mathbf{p}_{i}(t),\mathbf{c}_{i}(t))\}_{i=1}^{H\times W}$ is an independent point cloud at time step $t$; (2) a unified point cloud $\mathcal{P}=\{(\mathbf{p}_{i}(t),\mathbf{c}_{i}(t))\}_{i=1}^{N}$, where $N$ is the total number of points across all time steps, and each point $i$ has a trajectory over time. The first representation is easier to obtain, as it can be directly derived from the depth maps estimated from each frame using monocular depth estimation methods [296]. However, it may suffer from temporal inconsistencies and a lack of point correspondences across frames. The second representation, in contrast, requires explicit temporal correspondences (either ground-truth tracks, optical-flow lifting [255], or jointly optimized point trajectories [296]), and is therefore harder to acquire, but it offers persistent point identities and consistent trajectories that are useful for downstream 4D reconstruction, motion analysis, and dynamic NeRF/3DGS initialization.

We can extend the mesh representation (see Section 4.2.4) to 4D by representing a sequence of 3D meshes over time. A 4D mesh is a sequence $\{\mathcal{M}_{t}\}_{t=1}^{T}$ with $\mathcal{M}_{t}=(\mathbf{V}_{t},\mathbf{F}_{t})$, where $\mathbf{V}_{t}\in\mathbb{R}^{N\times 3}$ and $\mathbf{F}_{t}\in\mathbb{N}^{M\times 3}$ are the vertices and faces of the mesh at time $t$, respectively. To reconstruct a 4D mesh using deep learning, researchers typically sample a set of points on the mesh surface at the original time step $t=0$, and then learn a deformation field to model the motion of these points over time [297]. Then, the deformed points at each time step can be used to reconstruct watertight manifold meshes using Marching Cubes [105] or Poisson Surface Reconstruction [298].

As discussed in Section 4.2.6, NeRF [119] is a continuous, volume-based 3D representation that maps a 3D position $\mathbf{x}\in\mathbb{R}^{3}$ and a 2D viewing direction $\mathbf{d}\in\mathbb{S}^{2}$ to a color $\mathbf{c}\in\mathbb{R}^{3}$ and a volume density $\sigma\in\mathbb{R}_{\geq 0}$ via a learned function $F_{\Theta}:(\mathbf{x},\mathbf{d})\rightarrow(\mathbf{c},\sigma)$. This function is typically implemented using a neural network, which is trained on a set of input images and their corresponding camera poses. To extend NeRF to dynamic scenes, we can introduce a temporal dimension $t$ to the input of the function, resulting in a 4D representation $F_{\Theta}:(\mathbf{x},\mathbf{d},t)\rightarrow(\mathbf{c}_{t},\sigma_{t})$. This allows the model to capture changes in the scene over time, enabling the rendering of novel views at different time steps. In practice, the temporal dimension can be represented in various ways. For example, it can be trained to directly take a continuous time variable as input, and output displacement $\Delta\mathbf{x}(t)$ to deform the 3D position $\mathbf{x}(t)=\mathbf{x}(0)+\Delta\mathbf{x}(t)$ at time step $t$ [187]. Alternatively, the deformation can be parameterized using a dense $\text{SE}(3)$ field [299], which models the rigid transformations (rotation and translation) of points in the scene over time. Other approaches [300] may also incorporate additional latent codes to represent different time steps, enabling the model to capture more complex temporal variations.

Given a static scene representation $\mathcal{G}=\{(\alpha_{i},\bm{\mu}_{i},\bm{\Sigma}_{i},c_{i})\}_{i=1}^{G}$, which contains $G$ 3D Gaussians [121] (see Section 4.2.7), where $\alpha_{i}\in[0,1]$ is the opacity, $\bm{\mu}_{i}\in\mathbb{R}^{3}$ is the mean (3D position), $\bm{\Sigma}_{i}\in\mathbb{R}^{3\times 3}$ is the covariance matrix, and $c_{i}:\mathbb{S}^{2}\rightarrow\mathbb{R}^{3}$ is the color function that maps a viewing direction to an RGB color, we extend it to a deformable 3D Gaussian Splatting representation for dynamic scenes by letting both the mean and the covariance vary in time: $\mathcal{G}(t)=\{(\alpha_{i},\bm{\mu}_{i}(t),\bm{\Sigma}_{i}(t),c_{i})\}_{i=1}^{G}$ [191]. The mean is parameterized by a deformation field, $\bm{\mu}_{i}(t)=\bm{\mu}_{i}(0)+\Delta\bm{\mu}_{i}(t)$, where $\bm{\mu}_{i}(0)$ is the initial position at $t=0$ and $\Delta\bm{\mu}_{i}(t)$ is the time-dependent offset. The covariance is decomposed in the same way as in the static case, $\bm{\Sigma}_{i}(t)=\mathcal{R}_{i}(t)\mathcal{S}_{i}(t)\mathcal{S}_{i}(t)^{T}\mathcal{R}_{i}(t)^{T}$, with the rotation $\mathcal{R}_{i}(t)$ and scaling $\mathcal{S}_{i}(t)$ themselves outputs of the deformation field at time $t$. The opacity $\alpha_{i}$ and color function $c_{i}$ are typically kept constant over time. In practice, this means that the appearance of the scene does not change significantly over time, while the latest work [301] also explores time-varying colors and opacities. This representation allows for efficient rendering of dynamic scenes by accounting for the temporal changes in the positions and shapes of the 3D Gaussians.