6.1Spatial Entity Recognition

Given a 2D image $I$, 2D object detection identifies instances of predefined object classes and localizes each one in the image plane. The output is a set of detections

where $c_{i}$ is the predicted class, $b_{i}=(x_{i},y_{i},w_{i},h_{i})$ a 2D bounding box (center and size, or corner coordinates), and $s_{i}\in[0,1]$ a confidence score. In short, the task answers what objects are where. Figure 6.1 summarizes the 2D task and its detector families.

3D object detection keeps the same set-of-detections output but places each box in metric 3D space. The input is a LiDAR point cloud $\mathcal{P}=\{\mathbf{p}_{j}\}_{j=1}^{M}$ with $\mathbf{p}_{j}\in\mathbb{R}^{3}$ (often with an intensity channel), one or more calibrated images, or both, and each box becomes

a 3D center, the physical length, width, and height, and a heading angle $\theta_{i}$ about the vertical axis; indoor benchmarks often use axis-aligned boxes and drop $\theta_{i}$. Because the box is expressed in the sensor or ego frame rather than the image plane, a 3D detection already carries metric position and heading, which is why driving and robotics stacks consume it directly.

Table 6.1 summarizes the benchmarks that shaped the field. PASCAL VOC [444] fixed the evaluation protocol and remained the reference through the early deep-learning era despite its modest scale. MS COCO [440], with complex everyday scenes and segmentation masks alongside boxes, then became the standard for both detection and segmentation. Open Images [442], Objects365 [441], and LVIS [443] scale the vocabulary to hundreds and then over a thousand categories, with LVIS deliberately emphasizing rare classes, which turned it into the canonical long-tail and open-world benchmark.

Evaluation follows the precision–recall framework. A detection $(\hat{c},\hat{b},\hat{s})$ is correct when $\hat{c}$ matches the ground-truth class and $\hat{b}$ overlaps the ground-truth box $b$ above an Intersection over Union (IoU) threshold

Sorting the detections of a class by confidence traces a precision–recall curve whose area is the Average Precision (AP); the mean Average Precision (mAP) averages AP over classes. Benchmarks differ chiefly in the IoU threshold: PASCAL VOC uses a single $\mathrm{IoU}=0.5$ (AP₅₀), whereas COCO averages over ten thresholds from $0.5$ to $0.95$ to obtain its primary AP and reports AP₅₀ and AP₇₅ as diagnostic slices. COCO further stratifies by object size into AP_S (area $<32^{2}$ pixels), AP_M ($32^{2}$–$96^{2}$), and AP_L ($>96^{2}$), which exposes the persistent difficulty of small objects.

3D detection is benchmarked mainly on driving data, KITTI [173], nuScenes [445], and the Waymo Open Dataset [446], and indoors on SUN RGB-D [447] and ScanNet [448]; Omni3D (Table 6.1) pools image-based 3D annotations from both. The IoU above is computed between 3D boxes or their BEV footprints: KITTI reports 3D and BEV AP at IoU $0.7$ for cars and $0.5$ for pedestrians and cyclists, and indoor benchmarks report mAP at 3D IoU $0.25$ and $0.5$. nuScenes matches detections by center distance on the ground plane ($0.5$, $1$, $2$, and $4$ m) instead of IoU and folds translation, scale, orientation, velocity, and attribute errors into its nuScenes Detection Score (NDS); Waymo weights AP by heading accuracy (APH).

Table 6.2 reports COCO val2017 results across the three families. On the canonical ResNet-50 backbone, AP has climbed from $37.4$ for Faster R-CNN to $53.1$ for RT-DETR over roughly a decade, with much of the recent gain attributable to transformer-based set prediction (DETR $\rightarrow$ Deformable DETR $\rightarrow$ DINO) and improved denoising-style query initialization. The AP_S column highlights the well-documented small-object weakness of vanilla DETR ($20.5$); deformable attention and denoising progressively close this gap (Deformable DETR $26.4$, DINO $32.0$, RT-DETR $34.8$). On the efficiency frontier, the modern YOLO line (YOLOv8-x at $53.9$, YOLOv10-l at $53.2$) matches RT-DETR in accuracy at substantially higher throughput, so the traditional accuracy–speed divide between CNN one-stage detectors and transformer set prediction has largely closed.

The accuracy–efficiency trade-off remains unresolved. The transformer-based detectors that top COCO and open-vocabulary leaderboards are still expensive at inference: even with deployment stacks such as TensorRT, their latency and parameter counts rule out most mobile and embedded targets, while the lightweight real-time detectors that fit those budgets trail them in accuracy. The same tension recurs for every recognition task in this section. Open-world detection degrades on the hardest instances. Open-vocabulary detectors extend recognition beyond a fixed label set, yet still miss small objects and mislabel rare categories, precisely the long-tail regime that motivated LVIS. Detection inside multimodal large language models is still immature. Recent systems fold detection into MLLMs as a grounding primitive that ties image regions to language, but even the strongest closed-source models post markedly lower numbers than specialist detectors on standard benchmarks [424, 425]. 3D detection is bounded by depth. For camera-only detectors, localization error is dominated by depth error, which grows with distance [449]; LiDAR measures depth directly but its returns thin out with range, so distant and small objects are seen through a handful of points.

Semantic Segmentation: Given an input image $I\in\mathbb{R}^{H\times W\times 3}$, semantic segmentation assigns a class label to every pixel, producing a dense label map

where $C$ is the number of classes; instances of the same class are not distinguished.

Instance and Panoptic Segmentation: Instance segmentation detects and segments each object instance separately. The output is a set of labeled masks

where $m_{i}\in\{0,1\}^{H\times W}$ is the binary mask of the $i$-th instance and $c_{i}$ its class; masks may overlap, and only countable thing classes are covered. Panoptic segmentation keeps the same output form but requires the $K$ masks to form a non-overlapping partition of the image that covers thing instances and stuff regions alike, each stuff class as a single mask.

Video Segmentation: Given a video clip $V\in\mathbb{R}^{T\times H\times W\times 3}$, video segmentation predicts temporally consistent masks across frames, output as a set of mask tubes

each tracking an object or region over time, with semantic, instance, and panoptic variants as above.

3D Segmentation: Given a point cloud $\mathcal{P}=\{\mathbf{p}_{j}\}_{j=1}^{M}$ with $\mathbf{p}_{j}\in\mathbb{R}^{3}$ and optional color or intensity, 3D semantic segmentation labels every point,

and 3D instance and panoptic segmentation predict point masks $m_{i}\in\{0,1\}^{M}$ under the same set form and overlap constraints as their image counterparts; voxels or mesh vertices take the place of points for other 3D representations. The definitions carry over unchanged. What changes is the input: the points are unordered, unevenly spaced, and sparse, and that is where 3D methods depart from image methods.

Figure 6.3 summarizes the method families surveyed below.

Table 6.3 summarizes the main benchmarks. PASCAL VOC [444] anchored the classical semantic-segmentation era with 20 foreground classes and no instance labels. MS COCO [440], with semantic, instance, box, and panoptic annotations over complex object-centric scenes, is the shared testbed for all three modern task types. Cityscapes [501] and Mapillary Vistas [503] specialize to street scenes, the latter stressing long-tail classes and diverse geographies, and ADE20K [492] targets holistic scene parsing over 150 object and stuff classes. The 3D rows of the table cover indoor RGB-D scans (NYU-D V2, SUN RGB-D, ScanNet) and outdoor LiDAR (SemanticKITTI), and S3DIS [507] is a further indoor standard.

Segmentation evaluation reuses IoU (Section 6.1.1) with masks in place of boxes. For semantic segmentation the headline metric is the mean Intersection over Union (mIoU),

where $C$ is the number of classes and $\mathrm{TP}_{k}$, $\mathrm{FP}_{k}$, $\mathrm{FN}_{k}$ count the true-positive, false-positive, and false-negative pixels for class $k$. Because the average is unweighted, mIoU is sensitive to rare classes, which is one motivation behind ADE20K and Mapillary Vistas; pixel accuracy is dominated by frequent classes (e.g., , sky, road) and is less informative.

For instance segmentation, the COCO mask-AP protocol matches predictions to ground truth by mask IoU and averages AP over IoU thresholds $0.5$–$0.95$. For panoptic segmentation, Panoptic Quality [243] is $\mathrm{PQ}=\mathrm{SQ}\times\mathrm{RQ}$, the average IoU of matched segments times the F1 score over matched, false-positive, and false-negative segments, treating things and stuff uniformly. For video object segmentation, the DAVIS $\mathcal{J}\&\mathcal{F}$ metric averages region similarity ($\mathcal{J}$, mask IoU) and contour accuracy ($\mathcal{F}$, computed on mask boundaries). 3D semantic segmentation reports the same mIoU computed over points or voxels, and 3D instance segmentation on ScanNet reports AP at mask IoU thresholds averaged over $0.5$–$0.95$, with AP₅₀ and AP₂₅ as looser slices.

Table 6.4 reports results across these four protocols. The clearest trend is on semantic segmentation, where ADE20K mIoU has roughly doubled in under a decade, from FCN-8s’s $29.4$ to Mask2Former’s $57.7$; the largest single jumps track two shifts, first to transformer backbones (SETR, SegFormer, Swin-L) and then to universal query-based formulations. That second shift is what unifies the table: MaskFormer is the first entry to report semantic mIoU and panoptic PQ within one framework, and Mask2Former extends the same architecture to all three tasks with competitive results on each. Instance segmentation followed a partly separate path, climbing from Mask R-CNN ($35.4$) and HTC ($38.4$) on R50-FPN to $50.1$ once Mask2Former is paired with a Swin-L backbone, which confirms that transformer backbones lift mask quality at the instance level as well, not only semantic accuracy.

The cost of self-attention at high resolution, over long video, and on dense 3D data sharpens the accuracy–speed tension noted for detection, and two problems are specific to segmentation. Long-video, multi-object consistency is hard. Holding spatial precision and temporal coherence together remains difficult in cluttered scenes, particularly for video panoptic segmentation and segmentation-based multi-object tracking, where objects enter, leave, and occlude one another over long horizons. Segmentation is moving toward the open world. The field is shifting from fixed label sets to class-agnostic, open-vocabulary, and multimodal settings, in which language and other auxiliary signals are folded in to improve spatial reasoning and adaptability, a direction that SAM, X-Decoder, and OMGSeg already anticipate.

Given an input image $I$, a parametric model $f_{\theta}$ estimates a set of point locations

where each $\mathbf{p}_{i}$ is the image coordinate of a salient visual or structural element. Landmark detection fixes a predefined semantic identity for each point (a particular eye corner, a particular joint); the two terms are otherwise used interchangeably.

3D keypoint detection predicts $\mathbf{p}_{i}\in\mathbb{R}^{3}$ instead. From images, the depth of each point must be inferred; from depth maps or point clouds $\mathcal{P}$, it is observed, and the detector chooses or regresses points on the measured surface. For human pose, 3D joints are usually expressed in camera coordinates relative to a root joint (the pelvis), which leaves global depth unconstrained, as a single image cannot fix it.

The task varies along three axes. The input modality sets what geometry is directly available: most detectors work on 2D images (RGB, grayscale, or infrared), where keypoints must be inferred from appearance alone, whereas point clouds expose spatial structure directly; RGB-D or RGB-infrared fusion adds robustness in clutter and poor lighting, and video supplies temporal continuity through motion blur and transient occlusion. The prediction target ranges from planar image coordinates, which underpin most benchmarks, to 3D keypoints required in robotics and autonomous driving, to 4D trajectories tracked over time. The application scenario fixes what the points mean. Research has concentrated on humans: body joints for pose estimation, facial landmarks for expression recognition and identity verification, and finger joints for gesture-based interaction and sign language recognition [508]. Keypoints are also defined on everyday items, industrial parts, and vehicles for manipulation, inspection, and driving (e.g., , monocular 3D detection from vehicle keypoints [509]), on animals for behavior analysis (multi-animal DeepLabCut [510]), and on anatomical structures in medical imaging, where universal landmark detectors cover several modalities and anatomies [511].

Classical pipelines combined local feature detectors with statistical shape models that handled occlusion and deformation through explicit geometric constraints. Moravec [512], Harris [257], and Shi–Tomasi [513] corners were followed by SIFT [111], SURF [514], FAST [515], BRIEF [516], and ORB [517], which improved scale invariance, speed, and robustness. For faces and hands, active shape models [518], active appearance models [519], and constrained local models with regularized mean-shift fitting [520] provided the parametric frameworks.

Learning-based detection began with supervised descent for face alignment [521] and, in 2014, with DeepPose [258], which regressed human joints directly with a deep network. Heatmap regression then dominated through 2020: Tompson et al. [522, 523] coupled convolutional networks with graphical models, convolutional pose machines [524] and stacked hourglass networks [256] refined heatmaps over repeated stages, and HRNet [525] kept a high-resolution branch throughout the network for precise localization.

Multi-person estimation splits into two strategies. Top-down methods detect each person first and then estimate keypoints inside the box, which is accurate but scales with the number of people; bottom-up methods predict all keypoints at once and then group them, which is faster but makes grouping the hard part. Part Affinity Fields [526], consolidated as OpenPose [527], are the canonical bottom-up design; YOLO-Pose [528], RTMPose [529], and RTMO [530] push real-time multi-person estimation across both paradigms.

Vision transformers and foundation models made keypoint detection general-purpose: ViTPose [531] set simple transformer baselines for human pose in 2022, and Sapiens [532] in 2024 provides a single foundation model for human-centric vision tasks.

3D human pose estimation grew out of these 2D detectors. Lifting methods take detected 2D keypoints as input and regress the 3D joints: a fully connected network on 2D joints alone was a strong baseline [533], temporal convolutions over 2D pose sequences (VideoPose3D [534]) and spatio-temporal transformers (MotionBERT [535]) use motion to reduce the depth ambiguity, and DiffPose [536] casts lifting as diffusion. Direct methods predict 3D from pixels, as in the volumetric heatmaps of Pavlakos et al. [537]. With a calibrated multi-view rig most of the ambiguity disappears, and learnable triangulation [538] fuses per-view heatmaps into a 3D estimate. Recovering a full body mesh rather than a joint set is taken up with human motion in Section 8.4.

Classical corner detectors were in turn replaced by learned ones: SuperPoint [539] learns detection and description jointly under self-supervised homographic adaptation, DISK [540] selects keypoints with a reinforcement-style objective, and SuperGlue [541] and the detector-free LoFTR [542] recover dense correspondences directly. These serve as correspondence rather than semantic points and are the workhorses of the structure-from-motion, visual-odometry, and SLAM pipelines in the spatial-localization section that follows.

On point clouds, keypoints mostly serve registration and pose estimation. Intrinsic Shape Signatures [543] select points whose local neighborhood has a distinctive eigenvalue structure; USIP [544] learns, without labels, detectors that stay stable under rigid transformation, and D3Feat [545] predicts a detection score and a descriptor for every point, the 3D analogue of SuperPoint. Category-level 3D keypoints are a different kind: Suwajanakorn et al. [546] discover keypoints that are consistent across instances of a category through a relative-pose objective, and kPAM [547] uses such keypoints as the interface for robotic manipulation. Keypoint-based 6D object pose, which predicts the 2D projections of 3D model keypoints and solves PnP, belongs to pose estimation (Section 6.2.5).

Table 6.5 reports COCO Keypoints (val2017) results for the CNN, transformer, and real-time families.

Keypoint datasets are mostly embedded in broader pose and action benchmarks rather than built in isolation; Table 6.6 lists the most used. Human body datasets are the most mature and trace the field’s expansion: MPII [548] and COCO Keypoints [440] fixed in-the-wild 2D evaluation, PoseTrack [549] added temporal association in video, Human3.6M [550] supplies controlled multi-view 3D ground truth for 2D-to-3D lifting, DensePose [551] replaces sparse joints with dense surface correspondences, and CrowdPose [552] stresses crowded, occluded scenes. Face datasets moved from large pose variation (AFLW [553]) and high-density annotation (HELEN [554]) to occlusion (COFW [555]), the standardized 68-point protocol of 300-W [556] and its large-pose synthetic extension 300-W-LP [557], and the 98-point, hard-condition WFLW [558]. Hand datasets contend with fine articulation and self-occlusion through multi-view capture (CMU Panoptic HandDB [559]), synthetic 3D labels (RHD [560]), real 3D pose and shape (FreiHAND [561]), and interacting hands (InterHand2.6M [562]). Animal-Pose [563] and AP-10K [564] carry the human recipe to animals across species, while ApolloCar3D [565] and KeypointNet [566] define keypoints on rigid vehicles and 3D object categories.

Metrics measure the distance between predicted and ground-truth points, normalized for scale. The End-Point Error (EPE) is the raw Euclidean distance. The Normalized Mean Error (NME) divides it by a reference length:

where $\hat{\mathbf{x}}_{i}$ and $\mathbf{x}_{i}$ denote the predicted and ground-truth locations of the $i$-th keypoint, and $d$ is a task-dependent normalization factor (e.g., , inter-ocular distance for faces). The Percentage of Correct Keypoints (PCK) instead thresholds the error,

where $\alpha$ controls the tolerance and $s$ is the reference scale, giving an accuracy rate that is easy to compare across methods; the Cumulative Error Distribution (CED) curve and its Area Under the Curve (AUC) summarize behavior across thresholds.

For large-scale benchmarks, Object Keypoint Similarity (OKS) incorporates object scale and keypoint-specific uncertainty,

Here $s$ denotes the object scale (the square root of the segmented area), $\kappa_{i}$ is a per-keypoint normalization constant calibrating the fall-off, and $v_{i}\in\{0,1,2\}$ is the ground-truth visibility flag for keypoint $i$, where $v_{i}=0$ indicates unlabeled, $v_{i}=1$ indicates labeled but occluded, and $v_{i}=2$ indicates labeled and visible. OKS plays the role of IoU in computing average precision and recall on MS COCO [440]. For 3D human pose, the Mean Per-Joint Position Error (MPJPE) averages the Euclidean joint error in millimeters after aligning the root joints; PA-MPJPE applies a Procrustes alignment of scale, rotation, and translation first, and 3DPCK thresholds the error at $150$ mm, as on MPI-INF-3DHP [567]. Point-cloud keypoint detectors are scored by repeatability under rigid transformation and, downstream, by registration recall on benchmarks such as 3DMatch [568].

Robustness in unconstrained scenes is still the first problem [569, 570]: occlusion, extreme pose, and non-rigid deformation leave partial or ambiguous configurations, and symmetric parts (left versus right limbs, the fingers of a self-occluded hand) are routinely swapped. Generalization is the second [571, 572, 573]: keypoint definitions differ across datasets even for the same object class, so models trained on one benchmark transfer poorly to another, and the gap widens for animals, objects, and anatomical structures, where annotation is scarce and demographic or species bias enters the training data. The accuracy–speed tension discussed for detection applies here as well [574, 575]: high-precision models are too heavy for embedded targets, and lightweight ones lose accuracy precisely under occlusion and fast motion.

Formally, scene classification learns a mapping $f:I\mapsto y\in\mathcal{Y}$ from an image $I$ to a label $y$ in a predefined category set $\mathcal{Y}$, or, in the multi-label case, to a subset $Y\subseteq\mathcal{Y}$ covering several facets of the scene [582] (“urban” for area, “parking lot” for place, “night” for condition). Beyond the single-/multi-label split, the task is instantiated by domain, with indoor [583] and outdoor [584] scenes, remote-sensing imagery [585, 586], and driving contexts such as highways and tunnels [580], and by modality, extending from still images to video [587] and acoustic scenes [588].

Figure 6.5 summarizes the progression from hand-crafted global descriptors through deep features to vision–language models.

Early methods summarized the whole image into one discriminative vector of low-level texture, color, and gradient statistics. The Spatial Envelope model [589] describes a scene by a few perceptual dimensions (naturalness, openness, roughness, expansion, ruggedness), a descriptor commonly called GIST. Bag-of-Visual-Words (BoVW) [590] treats the image as a histogram of quantized local features such as SIFT [111], discarding most spatial information [591]; Spatial Pyramid Matching (SPM) [592] restores coarse layout by computing BoVW histograms over a pyramid of sub-regions and concatenating them.

Deep networks learn scene features end to end. CNNs [125, 593] such as AlexNet [125], VGG [594], ResNet [23], and DenseNet [595], pre-trained on ImageNet [439, 125], proved to be strong global feature extractors for scenes, and Places-CNN [596] re-trained them on the large-scale Places benchmark to establish the standard transfer recipe. Vision Transformers [142] match or exceed CNNs with a class-token embedding that attends over all patches, in the manner of BERT [42].

A separate line models layout explicitly, since a stadium is recognized by its global oval while a classroom differs from an office by the mid-scale arrangement of desks and blackboard. Multi-scale pooling over the CNN feature hierarchy (FPN-style [597]) captures structure at several scales, and scene graphs [598] make objects and their relations explicit, so that propagating over them with graph convolutional networks [599] yields layout-aware scene features [600].

Vision–language models (VLMs) turn scene classification into open-vocabulary, zero-shot matching. Contrastive models such as CLIP [143] embed the image and the textual names of all candidate classes in one space and pick the most similar class [601], needing no fine-tuning beyond prompt design [602]; the recipe extends to 3D point clouds [368]. Autoregressive multimodal large language models (MLLMs; LLaVA [145], Qwen-VL [603], Gemma 3 [604]) instead generate the label as text in response to an instruction [605]. With rich semantics available for free, the research focus has shifted from one global label toward compositional understanding of what is inside the scene [606].

Table 6.7 lists the standard benchmarks. The small early sets, Scene 15 [592] and the indoor MIT 67 [583], were superseded by SUN 397 [607, 608], which fixed a near-exhaustive category list, and by Places [596, 609], which scaled the same categories to millions of images across its Places205, Places88, Places365-Standard, and Places365-Challenge subsets. NYU Depth v2 [504] and SUN RGB-D [447] add aligned depth and a scene category per image for indoor RGB-D classification.

Scene classification is evaluated primarily with top-1 accuracy (the fraction of images for which the highest-scoring predicted class matches the ground truth) and top-5 accuracy (the fraction of images where the ground-truth class is among the top five predictions), with the latter especially useful on long-tail benchmarks such as Places-365 and SUN 397. For multi-label settings, the mean Average Precision (mAP) and class-wise F1 are reported instead, mirroring the protocol used in multi-label image recognition.

Table 6.8 reports results on the two most-cited scene benchmarks. Supervised CNNs on Places-365 cluster in a narrow band of roughly $53$–$55\%$ top-1, with diminishing returns from AlexNet to ResNet, while CLIP reaches $63.2\%$ top-1 on SUN 397 zero-shot [143], on par with supervised scene classifiers without any target-set training.

Closed-set accuracy on Places-365, SUN 397, and MIT 67 is now recovered zero-shot by CLIP [143] and by MLLMs (LLaVA [145], Qwen-VL [603], Gemma 3 [604]), so “what kind of place is this” is no longer the frontier. The task has climbed a granularity ladder, from one label to attribute sets to grounded, compositional description across image, video, and 3D, and the open problems sit at that richer end, where MLLMs are weaker than their coarse-label accuracy suggests.

Fine-grained and long-tail discrimination. MLLMs still trail dedicated classifiers and CLIP-style models on fine-grained and long-tail categories, where the distinction between, e.g., , a “bistro” and a “brasserie” turns on subtle cues [610].

Compositional and spatial grounding. The natural output is no longer a label but a description whose claims can be grounded in specific objects and layouts [606]. MLLMs describe fluently but ground weakly: objects are referenced vaguely rather than by region or box, and the persistent failure is structural rather than semantic, since models name the constituents of a scene correctly yet err on relative layout, depth ordering, orientation, and cross-view or 3D consistency. The cause is representational: 2D patch tokens are optimized for semantic alignment with language, not geometry, so fine spatial cues are lost once flattened into the language stream, and spatial-cognition benchmarks show even strong MLLMs degrading sharply as tasks move from naming toward layout and 3D reasoning [611, 612].

Hallucination and faithfulness. Free-form description reintroduces a failure mode that closed-set classification never had: MLLMs assert objects, attributes, or relations absent from the image, and their reasoning traces rarely point to verifiable spatial evidence [612].