4.12D

An image can be modeled as a discrete sampling of a continuous visual signal over a 2D spatial domain. Formally, a digital image is represented as a function

where $\Omega$ denotes the image grid, $C$ is the number of channels (e.g., , $C=1$ for grayscale, $C=3$ for RGB), and each pixel value encodes intensity or color information at a discrete spatial location [233, 234]. In practice, images are stored as finite tensors $I\in\mathbb{R}^{H\times W\times C}$, where $H$ and $W$ denote image height and width.

A video extends this representation by introducing a temporal dimension. Given a sequence of frames sampled over time, a video can be written as

or equivalently as a 4D tensor $V\in\mathbb{R}^{T\times H\times W\times C}$, where $T$ denotes the number of frames [234]. This formulation preserves spatial structure within each frame while enabling the modeling of temporal dynamics across frames. From a computational perspective, images and videos are treated as structured arrays of numerical values that can be efficiently processed using linear algebra operations and convolutional kernels. This regular grid structure enables weight sharing, local receptive fields, and efficient parallel computation, forming the basis of CNNs and their spatiotemporal extensions [235, 236]. In modern deep learning systems, image and video tensors may undergo normalization, resizing, patch embedding, or tokenization before being fed into neural architectures, including CNNs and vision transformers [142, 237]. For video inputs, spatiotemporal extensions either inflate 2D kernels into 3D convolutions [238, 239] or extend self-attention across space and time, as in the family of video transformers [240, 241].

A segmentation map is a semantic representation that assigns a label or a mask to each pixel, thereby partitioning visual data into meaningful regions such as objects, object parts, or background categories [234, 242]. For a single image defined on $\Omega\subset\mathbb{R}^{2}$, segmentation can be written as a mapping

where $\mathcal{L}$ is a set of labels. In semantic segmentation (SS), $S(u,v)$ denotes the class label of pixel $(u,v)$. In instance segmentation (IS), the output is a set of masks $\{M_{k}\}_{k=1}^{K}$ with associated class labels $\{\ell_{k}\}$, where each $M_{k}:\Omega\rightarrow\{0,1\}$ corresponds to an object instance [127, 131]. Panoptic segmentation (PS) unifies SS and IS by predicting a single, non-overlapping labeling of pixels that distinguishes both “stuff” and countable object instances [243].

These tasks extend naturally to videos. Given a sequence of frames $\{I_{t}\}_{t=1}^{T}$, video semantic segmentation (VSS) predicts per-frame semantic labels $\{S_{t}\}$, while video instance segmentation (VIS) predicts instance masks with temporal consistency across frames, often requiring instance association (tracking) over time [242]. Video panoptic segmentation (VPS) further combines VSS and VIS, producing temporally consistent panoptic outputs and is commonly evaluated with metrics that account for both segmentation quality and tracking consistency [242]. Closely related settings include video object segmentation (VOS) [244], which segments foreground objects across time under semi-supervised or unsupervised protocols, and referring segmentation, which predicts masks grounded by natural language expressions in images (RIS) [245] or videos (RVOS) [246]. Segmentation maps are typically formed by dense prediction models that produce either hard labels or per-pixel class probabilities $P(u,v,\ell)$ (and, for instance-level tasks, a set of object masks and scores). A key advantage of segmentation is its explicit and interpretable structure: it converts low-level pixels into object- and region-level representations that support downstream reasoning and decision-making, and it can be fused naturally with geometry (e.g., , depth, normals) and language in multimodal systems [242]. Consequently, segmentation maps are central to applications such as autonomous driving, medical image analysis, robotic manipulation, visual editing, and human-computer interaction.

A depth map is a geometric representation that augments a 2D image with per-pixel distance information, describing the spatial layout of a scene from a given viewpoint [234]. Formally, given an image domain $\Omega\subset\mathbb{R}^{2}$, a depth map is a scalar function

where $D(u,v)$ denotes the distance from the camera center to the first visible surface along the viewing ray corresponding to pixel $(u,v)$. Depending on the sensing modality and formulation, this value may represent metric depth, inverse depth, or disparity [112]. Depth maps can be obtained through multiple mechanisms, including stereo vision via triangulation [112], active sensors such as structured-light or time-of-flight cameras [247], and learning-based monocular depth estimation [55]. When combined with known camera intrinsics, a depth map enables the back-projection of image pixels into 3D space, thereby serving as a bridge between 2D image representations and 3D geometry. A key advantage of depth maps lies in their compact, image-aligned structure, which allows them to be processed using standard convolutional architectures and easily fused with the RGB appearance information. Compared to volumetric representations, depth maps are memory-efficient while still encoding meaningful geometric cues. As a result, they are widely used in applications such as 3D reconstruction, novel view synthesis [119], robotic navigation, and autonomous driving [173].

A surface normal map is a geometric representation that encodes local surface orientation at each pixel of an image. Given an image domain $\Omega\subset\mathbb{R}^{2}$, a surface normal map assigns to each pixel a unit vector

where $\mathbf{N}(u,v)=(n_{x},n_{y},n_{z})$ represents the direction of the surface normal in camera or world coordinates, and $\mathbb{S}^{2}$ denotes the unit sphere. Surface normal maps can be derived from depth maps or 3D geometry via local differentiation, or directly estimated from RGB images using learning-based methods [248]. Classical approaches to normal estimation rely on photometric stereo or geometric reconstruction [249], while modern methods employ CNNs to regress surface normals from monocular images [250]. Regardless of the estimation procedure, surface normal maps remain image-aligned and encode rich local geometric information that complements depth and segmentation cues. Surface normal maps can capture fine-grained shape and orientation details that may be ambiguous or noisy in raw depth estimates. Because surface normals are invariant to absolute depth scale, they provide robust geometric constraints for scene understanding. Consequently, surface normal maps are widely used in applications such as 3D reconstruction, object pose estimation, scene layout analysis, realistic rendering, and as auxiliary supervision in multi-task learning [250].

An edge map is a structural representation that highlights salient intensity or color discontinuities in an image, which often correspond to object boundaries and significant geometric transitions [251]. Given an image $I:\Omega\rightarrow\mathbb{R}^{C}$, an edge map is typically constructed by measuring local image gradients, for example,

where $\nabla I$ denotes the spatial gradient operator. Classical edge detection methods rely on hand-crafted operators such as Sobel, Laplacian of Gaussian, and the Canny detector [251], while modern learning-based approaches directly predict the edge probabilities using convolutional neural networks [252]. Edge maps may be binary, indicating the presence or absence of an edge, or real-valued, encoding edge strength, orientation, or confidence. The primary advantage of edge maps is their ability to significantly reduce visual complexity while preserving essential geometric structure. This abstraction makes them relatively robust to changes in illumination and appearance. Consequently, edge maps are widely used in image segmentation, contour detection, shape analysis, line drawing, and as intermediate cues for higher-level tasks such as object recognition and depth estimation [234].

An optical flow map is a motion representation that encodes the apparent 2D displacement of pixels between consecutive frames in a video sequence [253]. Given two image frames $I_{t}$ and $I_{t+1}$, the optical flow is defined as a vector field

where $(f_{x},f_{y})$ describes the horizontal and vertical displacement of pixel $(u,v)$ from time $t$ to $t+1$. Classical optical flow estimation relies on assumptions such as brightness constancy and spatial smoothness, leading to variational formulations [253, 254]. More recently, deep neural networks have been proposed to directly regress dense flow fields from image pairs or short video clips [128, 255]. Regardless of the estimation method, the resulting optical flow map remains image-aligned, preserving spatial correspondence with the input frames. The main advantage of optical flow maps is their ability to capture rich dynamic information about scene motion, including object motion, camera motion, and motion parallax, while remaining computationally efficient. As a result, optical flow is a core representation in video analysis and robotic perception tasks such as motion segmentation, action recognition, video interpolation, and visual odometry.

A 2D keypoint set is a sparse image representation that encodes the image-plane locations of a finite set of semantically or geometrically meaningful points, such as corners, joints, or landmarks [234]. Unlike dense maps that assign a value to every pixel, keypoint representations summarize visual content with a small number of salient points, trading pixel-level coverage for compactness and geometric explicitness. Formally, given an image defined on a domain $\Omega\subset\mathbb{R}^{2}$, a keypoint representation is a set of $K$ points

where each keypoint corresponds to a specific semantic or geometric entity. In learning-based formulations, keypoints are often represented implicitly as confidence or heat maps:

with the keypoint location obtained by finding the maximum or expected value of $H_{k}$ [256]. 2D keypoints can be obtained using classical feature detectors, such as Harris corners, SIFT, and SURF [257, 111], or through modern deep neural networks that directly regress keypoint locations or predict dense heat maps [258, 256]. Depending on the task, keypoints may represent generic interest points, object-specific landmarks, or articulated structures such as the human body joints. The detection task that produces such a set, including its extension to landmark detection and human pose estimation, is discussed in detail in Section 6.1.3 of Chapter 6. A key advantage of 2D keypoint representations is their sparsity and compactness: they significantly reduce the dimensionality of visual data while preserving critical structural information. This makes keypoints computationally efficient, robust to background clutter, and well-suited for geometric reasoning and correspondence matching. Human pose estimation is a specialized and highly structured instance of 2D keypoint detection that focuses on localizing a predefined set of anatomical landmarks of the human body, such as joints and facial keypoints, in images or videos [259]. In the 2D setting, the task aims to estimate the image coordinates of body joints, typically represented as a set of keypoints $\mathcal{P}=\{(x_{j},y_{j})\}_{j=1}^{J}$, where $J$ denotes the number of joints in a human skeletal model. These keypoints are not independent: they are constrained by the underlying human body structure, which is often modeled using kinematic trees, graphical models, or learned structural priors.