6.2Spatial Localization and Position Estimation

Given an image $I$ and optional auxiliary observations $\mathcal{V}$ (a second view, a multi-view set, a sparse sensor measurement, or neighboring video frames), a depth estimator predicts a depth map

where $D(u)$ is the distance along the optical axis from the camera center to the surface seen at pixel $u$. The task splits along three axes.

Sensing. Active methods emit light (structured light, time-of-flight, LiDAR) and measure its return, providing reliable metric depth even on textureless surfaces but tying accuracy to the sensor and degrading under strong ambient light or at long range. Passive methods infer depth from ordinary images alone; they need no special hardware and have become the dominant research line, at the cost of sensitivity to texture and illumination.

Scale. Metric depth gives distances in physical units, which 3D reconstruction and navigation require. Relative depth recovers only pixel ordering, or depth up to an affine transform; it cannot support metric measurement but is easier to supervise and generalizes better because heterogeneous datasets can be mixed without a consistent scale.

Density and Input Configuration. Classical pipelines produced sparse depth at feature points or semi-dense depth along high-gradient edges; dense per-pixel prediction is now the norm. The input configuration fixes how much geometry is available: active sensing measures depth directly; stereo recovers it from the disparity between two calibrated views; multi-view estimation enforces consistency across many views, which improves robustness to occlusion and matching ambiguity at the price of computation and known camera poses; monocular estimation has no geometric constraint at all and relies on learned statistical, structural, and semantic cues, which is why it is ill-posed in scale yet the most widely applicable setting.

Table 6.9 groups the main datasets into Type I, which ship explicit depth maps for direct supervision, and Type II, which require depth to be derived from LiDAR, stereo, or SfM. Three waves are visible. Indoor RGB-D collections (NYU Depth v2 [504], TUM RGB-D [652], SUN RGB-D [447], later ScanNet [448] and Matterport3D [340]) and the LiDAR-based KITTI [653, 654] fixed the indoor and outdoor protocols that still anchor the field, with Make3D [38] as the laser-scanned precursor. Relative-depth supervision from internet photos (DIW [655], ReDWeb [656], MegaDepth [657]) and synthetic data (Scene Flow [619]) then relaxed the need for metric sensors, while ETH3D [658], iBims-1 [659], ApolloScape [660], and Cityscapes stereo pairs [501] served precision evaluation and self-supervised training. The current wave is driven by autonomous driving and rendering: multi-sensor fleets (nuScenes [445], Waymo Open [446], Argoverse 2 [661], KITTI-360 [662], DDAD [663]) supply millions of LiDAR-calibrated frames, photorealistic synthetic sets (Hypersim [664], Virtual KITTI 2 [195], UrbanSyn [665], TartanAir [666]) supply exact ground truth even for transparent surfaces and hard lighting, and DIODE [667], BlendedMVS [668], ARKitScenes [669], Mannequin [670], and WildRGB-D [671] cover high-precision scans, handheld LiDAR, and dynamic or cluttered capture.

Depth is evaluated with pixel-wise regression metrics. The root mean squared error and its logarithmic variant,

measure absolute deviation, the log form damping the dominance of far-range errors. Because an error of fixed size matters more at close range, relative errors normalize by the true depth,

and Abs Rel is the primary metric on most benchmarks. For monocular prediction, whose scale is ambiguous, the scale-invariant log error discounts a global scale shift,

Threshold accuracy reports the fraction of pixels within a multiplicative tolerance,

Ordinal metrics, which score only the pairwise depth ordering, are used where supervision is relative; they track perceived layout but say nothing about metric precision.

On KITTI, supervised regressors such as BTS [628] and DPT [630] reach Abs Rel around $0.06$ with $\delta_{1.25}$ near $0.96$, while the self-supervised Monodepth2 [180] reaches Abs Rel near $0.10$ without any ground-truth depth. On NYU Depth v2, ZoeDepth [632] and Depth Anything [140] reach $\delta_{1.25}$ of $0.96$ and $0.98$ after fine-tuning, and UniDepth [672] matches them zero-shot, which makes transfer a credible alternative to per-benchmark training. Generative models such as Marigold [636] match or exceed regression on scale-invariant metrics while producing sharper boundaries, so recent gains come more from representation and data scale than from architectural depth. Boundary sharpness, temporal stability, and latency are still not evaluated in a standard way.

Monocular depth is ill-posed: distinct 3D scenes project to the same image, so a single view fixes depth only up to scale and predictions drift across scenes. Scene content adds its own failures. Moving objects break the static-world assumption behind self-supervised and multi-view methods, and in video the result is flicker between frames and scale drift over time. Transparent and reflective materials and textureless regions give weak or misleading appearance cues; thin structures and sharp boundaries lose detail; low light, adverse weather, and underwater imagery degrade models trained on clean data. These scene effects compound into a domain gap: models are tied to the geometry and depth distribution of their training data, the indoor–outdoor divide is especially wide, real depth labels are scarce and noisy, and synthetic data still transfers imperfectly.

Current work attacks these problems from several sides. Depth foundation models [140, 672, 632] supply strong zero-shot priors, and Video Depth Anything [295] adds temporal modules to one of them for consistent video depth, while video depth diffusion (DepthCrafter [294], ChronoDepth [673], RollingDepth [674]) pursues the same goal generatively. Coupling depth with 3D scene representations (DepthSplat [675]) improves spatial consistency, language cues help anchor metric scale [676], and lightweight architectures such as MonoLENS [677] and LMDepth [678] target the real-time budgets of driving, robotics, and AR/VR.

Let $I\in\mathbb{R}^{H\times W\times 3}$ be an image, $\mathcal{V}$ optional auxiliary observations (neighboring frames, stereo or multi-view images, depth, intrinsics, or poses), and $\Omega$ the set of valid pixels. The goal is to estimate

where each $\hat{\mathbf{n}}_{u}$ is a unit vector in the camera frame or in a gravity- or world-aligned frame. Supervision comes from depth sensors, fitted meshes, rendered assets, multi-view reconstruction, or indirectly from photometric and intrinsic-image losses, and camera metadata matters because a pixel’s viewing ray depends on focal length and field of view. Monocular prediction from one RGB image is the canonical benchmark setting; RGB-D variants use measured or predicted depth as input or supervision through the differential relation between depth gradients and normals, multi-view and video variants enforce viewpoint or temporal consistency, object-centric variants target CAD-like objects, hands, faces, and bodies, and panoramic variants predict under equirectangular or fisheye geometry, where pinhole assumptions break.

Surface normal estimation has moved from physics-based inverse rendering to supervised dense prediction and, most recently, to foundation-model and diffusion-based geometry estimation. Figure 6.7 summarizes the method families; the last two below are grouped as one family in the figure.

Table 6.10 shows that normal supervision is usually inherited from another representation rather than annotated directly. RGB-D datasets (NYU Depth v2 [504], SUN RGB-D [447], ScanNet [448]) supply measured depth from which local planes or mesh normals are computed, which fixes the indoor evaluation regime but also means the labels are only as good as the depth sensor and are dominated by man-made indoor structure. Scanned and synthetic collections address coverage and label quality from opposite sides: Matterport3D [340], Taskonomy [682], and Omnidata [683] scale supervision across many environments and tasks, whereas Replica [339] and Hypersim [664] give clean rendered normals with physically meaningful lighting at the price of a synthetic-to-real gap. DIODE [667] pushes toward indoor–outdoor generalization, yet outdoor normals remain far less standardized than outdoor depth because sparse LiDAR is a poor match for dense orientation supervision; intrinsic-image sets such as IIW [689] and MPI Sintel Intrinsics [690] constrain shading and reflectance for inverse-rendering formulations without being normal benchmarks themselves. The most visible gap is temporal: although ScanNet and other RGB-D videos contain sequences, there is no widely adopted video normal benchmark, so temporal stability is reported qualitatively or through method-specific warping losses, even though flicker often hurts reconstruction and AR consumers more than a small rise in mean angular error.

Evaluation is angular because normals are unit vectors. With $\hat{\mathbf{n}}_{i},\mathbf{n}_{i}\in\mathbb{S}^{2}$ the predicted and ground-truth normals at valid pixel $i$ and $N$ the number of valid pixels, the per-pixel error in degrees is

summarized by its mean (sensitive to boundary noise and invalid-depth artifacts), its median (robust to a few severe failures), occasionally its RMSE, and by threshold accuracies

which expose changes in the fraction of practically usable pixels that a small shift in mean error can hide. Training objectives usually minimize the cosine loss $1-\hat{\mathbf{n}}_{i}^{\top}\mathbf{n}_{i}$, monotonic in angle for unit vectors, but benchmarks report angles for interpretability. Beyond per-pixel angle, joint depth–normal models are checked for agreement between predicted normals and normals differentiated from depth, reconstruction-oriented work asks whether normal discontinuities align with object boundaries and whether large planes stay coherent, and video methods report temporal consistency by warping predictions between adjacent frames with flow or pose,

after rotating camera-frame normals into a common frame. Protocol details move rankings: depth-derived normals are unstable at discontinuities, papers mask invalid pixels differently, and some evaluate in the camera frame while others rotate into a gravity-aligned frame, so the table below records which paper produced each number.

Table 6.11 makes two things visible. Within one protocol, camera-aware discriminative models lead: DSINE roughly halves the median error of the 2015 baseline, and Metric3Dv2 reaches $13.3^{\circ}$ mean error zero-shot, so projective inductive bias and large camera-diverse pretraining still pay off even with strong backbones. Diffusion-based estimators trail these numbers on NYU Depth v2 under the StableNormal protocol, but they trade accuracy on clean indoor scenes for coherence in textureless or ambiguous regions and for a usable uncertainty, which the single-frame benchmark does not reward; the post-GeoWizard line has accordingly shifted the target from single-frame accuracy toward stable, usable geometry, StableNormal controlling diffusion variance and NormalCrafter [687] enforcing temporal coherence.

An RGB image alone does not determine shape, lighting, and reflectance: a dark pixel may be a tilted surface, a dark material, a cast shadow, or a specular highlight, and learned models resolve the ambiguity with priors rather than removing it, which is why estimators look confident on common indoor layouts and fail systematically on rare materials, unusual lighting, and out-of-distribution scenes. The persistent failure modes follow from this. Reflective and transparent surfaces produce gradients caused by the environment rather than the surface. Textureless and weakly lit regions give little evidence, and networks hallucinate planar sheets. Thin structures and depth discontinuities mix receptive fields and inherit depth-derived label noise, blurring normals around chair legs, plants, railings, and silhouettes. Wide-field and panoramic cameras break models trained under narrow pinhole assumptions, which is what makes ray-aware designs such as DSINE [684] necessary. Dynamic and deformable objects flicker because most supervision comes from static scans or rigid synthetic scenes.

Dataset bias compounds the problem: canonical benchmarks are indoor and sensor-derived, so outdoor surfaces, hands, faces, deformable and transparent objects, and egocentric manipulation stay underrepresented, and no accepted benchmark jointly measures cross-domain generalization, temporal stability, and downstream reconstruction quality. TransNormal [688] is a corrective for one slice of the non-Lambertian regime (glass, liquid, mirror, metal, wet roads), which remains far from solved. The directions now taking shape answer these gaps directly: geometric foundation models that merge depth, normals, segmentation, material, and calibration into one dense-prediction backbone, following Omnidata [683] through Metric3Dv2 [685] and GeoWizard [637]; diffusion distillation and variance control that keep the coherence and uncertainty of generative models while recovering real-time, deterministic inference [686]; temporally consistent, reconstruction-coupled normals [687]; and physically grounded models for non-Lambertian materials, where data scale is no substitute for modeling refraction and reflection.

The input is one or more images or a video together with a specification of what to measure: the height of an object, the distance between two objects, or the distance an object travels. The output is a length in physical units, or a displacement over a time interval, from which average speed is read off (speed estimation).

Every vision-based route to a metric length passes through the camera model, shown in Figure 6.8. Under the pinhole model a world point $P$ with homogeneous coordinates $\mathbf{P}_{w}$ projects to the pixel

where the extrinsics $(R,\mathbf{t})$ move the point into the camera frame and the intrinsic matrix $K$ (focal length, principal point, pixel aspect) maps metric image-plane coordinates to pixels. Real lenses deviate from this model; Brown’s model [691] corrects the dominant radial term as

with a tangential term for sensor–lens misalignment, and camera calibration [692] recovers $K$ together with the distortion coefficients. Without calibration, every pixel-to-length conversion inherits an unknown scale.

Beyond calibration, geometric pipelines rely on a small set of recurring intermediate quantities, illustrated in Figure 6.9: vanishing points, where parallel world lines converge and from which camera orientation follows; the horizon line, which fixes the camera’s tilt relative to the ground; and a ground-plane homography, which lets lengths on a dominant plane such as a road or floor be measured once a single reference length is known. Stereo pipelines add rectification, which aligns the two image planes so that correspondences lie on the same scanline, and disparity matching. Learning-based pipelines keep object detection (Section 6.1.1) as the front end and replace the explicit geometry with regression of depth, 3D center, and dimensions from region features, optionally conditioned on a dense depth map and refined over time when video is available.

Vision-based height and distance estimation is organized along two axes: the sensor configuration (stereo versus monocular), which decides whether metric depth is available by triangulation, and the inference paradigm (geometry-based versus learning-based), which decides whether scene geometry is solved from projective constraints or inferred from data. Figure 6.10 lays out the resulting families; it draws the uncalibrated case as a third branch for readability, but it is best read as the weakly constrained regime of monocular or stereo inference in which some geometric information is missing and must be supplied by priors.

Table 6.12 summarizes the benchmarks, which fall into three groups by sub-task. Object-size estimation is evaluated on the indoor RGB-D sets CAMERA25 and REAL275 [710], where category-level dimensions are scored jointly with pose on controlled tabletop scenes. Sensor-to-object distance is rarely benchmarked on its own; it is evaluated through the 3D-detection annotations (box centers, sizes, velocities) of the driving datasets KITTI [653], nuScenes [445], Waymo Open [446], and Argoverse 2 [661], which test metric localization at scale across multi-camera rigs and scenes, with BDD100K [711] as a monocular video source. Speed and displacement estimation is dominated by fixed-camera traffic video: AI City Challenge 2018 [712], BrnoCompSpeed [713], UTFPR [223], and QMUL [714], plus the UAV-mounted Beihang videos [715]; these are the only benchmarks that score a physically meaningful motion quantity rather than static geometry.

Each sub-task is scored with the same triplet: an absolute error in physical units, a relative error for cross-range comparison, and a threshold accuracy for downstream binary decisions. For object size, the metric is the mean absolute error in physical units (centimeters for human height, millimeters for category-level pose),

with the mean absolute percentage error $\mathrm{MAPE}=\tfrac{1}{N}\sum_{i}|\hat{s}_{i}-s_{i}|/s_{i}\times 100\%$ alongside for cross-class comparison; when size is scored jointly with pose, the 3D IoU of predicted and ground-truth boxes is thresholded, the NOCS protocol [710] reporting $\mathrm{Acc}@\tau$ at $\tau\in\{5\%,\,10\%,\,20\%\}$ IoU and at $\{5,\,10\}$ cm translation. Inter-object distance reuses MAE, RMSE, and MAPE on distances $\hat{d}_{i}$ and the depth-style ratio accuracy $\delta_{\tau}$ of §6.2.1 with $\tau\in\{1.10,\,1.25\}$; because stereo error grows as $\Delta Z\propto Z^{2}/(fB)$, KITTI-style protocols [653] stratify by range (near $<10$ m, mid $10$–$50$ m, far $>50$ m) so that far-field degradation is not averaged away. Speed estimation reports MAE, RMSE, and MAPE in km/h, but the dominant device is a percent-within-tolerance band: AI City Challenge 2018 [712] fixes $\pm 3$ km/h and combines it with detection recall into its S1 score, and BrnoCompSpeed [713] adds the median absolute error because single-frame outliers from motion blur or detection jitter distort the mean. Two protocol details change rankings: per-vehicle averaging (BrnoCompSpeed) weighs trajectories differently from per-frame averaging matched to track IDs (AI City), and monocular and uncalibrated methods are sensitive to the assumed focal length, principal point, and reference length, so their reported accuracy is only meaningful together with the calibration assumptions.

Performance is sub-task specific. Category-level size recovery on REAL275 [710] remains far from millimeter accuracy in cluttered scenes. On BrnoCompSpeed [713], calibration-aware pipelines reach low single-digit km/h errors on the original sessions but degrade when transferred to unseen camera placements, exposing the brittleness of fixed-camera homography assumptions, and on the AI City 2018 speed track [712] the leading entries pair strong detection-and-tracking front ends with multi-frame speed regression, so the S1 score reflects detection recall as much as raw speed accuracy. Across all three sub-tasks the bottleneck is calibration, of the camera and of the scene-plane reference, rather than the regression backbone.

Calibration sensitivity is the dominant open problem: small errors in intrinsics or in the scene-plane reference propagate non-linearly into every recovered length, and methods that treat focal length and reference size as latent variables, or refine calibration online from multi-view consistency, generalize across camera setups where fixed-calibration pipelines fail. The second gap is coverage: benchmarks are confined to indoor tabletop size estimation and vehicle-traffic speed, leaving pedestrian speed in crowds, package dimensions in logistics, and indoor inter-object distance essentially unmeasured. Fusing visual estimates with LiDAR, radar, or inertial sensing would lift robustness in low light and under occlusion, but the vision-only setting treated here still lacks a benchmark that scores metric accuracy jointly with calibration uncertainty.

Let $\mathbf{T}_{t}=[\mathbf{R}_{t}|\mathbf{t}_{t}]\in SE(3)$ denote the pose of camera $t$ with respect to a world frame. Given images $\{I_{t}\}$ (monocular, stereo, RGB-D, multi-view, or video), optional intrinsics $\{\mathbf{K}_{t}\}$, optional inertial readings, and sometimes a prior map or database, the goal is to estimate relative transformations $\Delta\mathbf{T}_{i\rightarrow j}=\mathbf{T}_{j}\mathbf{T}_{i}^{-1}$, an absolute pose $\mathbf{T}_{q}$ for a query image, or a full trajectory together with sparse or dense scene geometry, and occasionally the intrinsics themselves when calibration is unknown. The variants differ in what is assumed known. Monocular VO is scale ambiguous without external priors; stereo and RGB-D VO recover metric scale from a known baseline or measured depth; visual-inertial odometry uses IMU measurements to stabilize scale and fast rotation; SLAM adds mapping and loop closure to online odometry; SfM recovers poses and structure from unordered collections; map-based relocalization assumes an existing scene representation and places a query in it; and map-free relocalization and feed-forward reconstruction skip explicit map construction by predicting relative pose or globally aligned point maps directly from images.

Figure 6.11 summarizes five method families, ordered from explicit geometry to learned geometric priors to feed-forward scene-level geometry.

Table 6.13 shows three kinds of benchmark serving three different questions. Trajectory benchmarks (KITTI Odometry [173], TUM RGB-D [652], EuRoC MAV [733]) measure drift under a fixed sensor setup and cover a narrow slice of deployment: urban driving, small rooms, micro-aerial-vehicle flights. Relocalization benchmarks (7-Scenes [724], Cambridge Landmarks [723]) ask whether a query can be placed in a known scene and favor methods with scene-specific maps, retrieval, or scene-coordinate regression, so they are not comparable to online VO. Large internet and video corpora (MegaDepth [657], RealEstate10K [734], ScanNet [448], DL3DV-10K [735]) serve mostly as training data for generalizable geometry models, which need diverse baselines, unknown intrinsics, unordered images, dynamic content, and dense reconstruction targets.

Evaluation is correspondingly layered. With $\mathbf{T}_{i}$ and $\hat{\mathbf{T}}_{i}$ the ground-truth and estimated poses, the absolute trajectory error measures global discrepancy after a rigid or similarity alignment $\mathbf{S}$,

and the relative pose error measures local motion over an interval $\Delta$,

KITTI reports the same idea as translational and rotational drift averaged over fixed-length path segments. Whether scale is aligned before computing ATE, or scale drift is scored separately, decides whether monocular, stereo, RGB-D, and visual-inertial methods can be compared at all. Pairwise matching and feed-forward pose methods report rotation error and translation-direction error in degrees, often aggregated as the area under the accuracy curve (AUC) below angular thresholds; relocalization reports the fraction of queries within thresholds such as $(5\,\mathrm{cm},5^{\circ})$ indoors or meter-level outdoors, or median translation and rotation error; and feed-forward geometry models add depth error, point-map error, Chamfer distance, registration rate, or view-synthesis quality, since their cameras are judged by whether they support a usable 3D representation.

Table 6.14 puts the two regimes side by side. On trajectory benchmarks the lesson is robustness as much as accuracy: the classical systems that are highly accurate when they track lose tracking on a fraction of EuRoC flights and on most TUM freiburg1 sequences, whose rolling shutter, motion blur, and fast rotation defeat feature matching, whereas DROID-SLAM’s learned dense correspondence inside a bundle-adjustment loop tracks every sequence at centimeter-level error. On multi-view pose the feed-forward models have overtaken the COLMAP pipeline by a wide margin and keep improving on unseen data (RealEstate10K), with VGGT ahead of DUSt3R-style pairwise regression at a fraction of the runtime, and a short bundle-adjustment refinement still adding several points, which is the hybrid pattern the challenges below return to. The two blocks are not comparable to each other, and that is the point: a method optimized for low drift under a calibrated rig and one optimized for zero-shot sparse-view geometry are judged by different questions.

Geometric ambiguity sets the floor. Monocular cameras cannot recover metric scale without priors; pure rotation, planar scenes, and very small baselines make epipolar geometry ill-conditioned; rolling shutter and unknown intrinsics violate the assumptions built into many solvers. Learned models can hide these failures behind priors but cannot restore missing information, which is why visual-inertial fusion, calibration, and depth priors remain necessary even with large geometry models. Deployment adds its own: dynamic agents corrupt the static-world assumption behind classical SLAM and bundle adjustment; textureless rooms, repetitive corridors, glass, water, and specular surfaces yield unreliable correspondences; appearance shift across time of day, weather, season, and camera hardware breaks place recognition; and long-horizon operation still needs loop closure, map maintenance, and uncertainty handling, none of which a feed-forward prediction supplies on its own.

The strongest trend is the convergence of SLAM, reconstruction, and foundation geometry. DROID-SLAM [95] showed that optimization can be learned without abandoning geometric structure, and the feed-forward models above predict dense geometry, cameras, and correspondences together; the near-term frontier is hybrid, with feed-forward geometry supplying initialization, dense priors, and generalization and optimization enforcing metric consistency, loop closure, and uncertainty-aware correction. Two further shifts matter for spatial intelligence. Pose estimation must become dynamic-scene aware, estimating camera and object motion jointly as MegaSaM [296] and MonST3R [730] begin to do, rather than discarding moving content as outliers; and camera pose should be judged by downstream usefulness, whether it supports stable depth and normal prediction, reconstruction, embodied navigation, AR anchoring, or world-model memory. As pose becomes one output of larger geometric perception systems, the boundary between VO, SLAM, scene reconstruction, and spatial memory will keep blurring.

The input is an RGB image, an RGB-D image, or a short multi-view burst of a scene containing one or more rigid objects, together with object-specific priors that depend on the setting: a CAD mesh or a small reference image set per object (instance-level and model-free), or a category label with a canonical frame (category-level), optionally with camera intrinsics and an upstream 2D detection or mask. The output is a pose $\mathbf{T}_{o}\in SE(3)$ per detected object in the camera frame, augmented in the category-level setting with a metric scale $s\in\mathbb{R}^{+}$ (9-DoF pose). For symmetric objects the prediction is properly an equivalence class of poses under the object’s symmetry group, which the ADD-S and BOP metrics below account for.

Figure 6.12 schematizes the task and the three method-family trajectories, which recent methods often straddle.

Table 6.15 lists the recurring benchmarks. Instance-level evaluation grew from LINEMOD [766], the texture-rich entry point that introduced the ADD metric, through its occlusion-heavy re-annotation LINEMOD-Occluded [767], which modern methods report as the more informative of the two, to the video setting of YCB-Video [737] and the texture-less, symmetric industrial parts of T-LESS [768], where symmetry handling decides the ranking; HomebrewedDB [769] serves as an out-of-distribution test for methods trained on the BOP core data, and SPEED [770] stretches the task to spacecraft under scale and illumination ranges no tabletop set covers. NOCS-REAL275 with its synthetic companion CAMERA [710] is the category-level standard. The BOP Challenge [771, 772, 773] consolidates the instance-level datasets under one protocol and has been the field’s headline leaderboard since 2017; its 2023 edition separates a classical CAD-known track from a novel-object track, formalizing the model-free shift. Modern papers report at minimum on LINEMOD-Occluded, YCB-Video, and a BOP-core subset, novel-object methods add the BOP unseen-object track, and category-level methods center on REAL275.

Each metric answers to a particular failure mode of pose error, and the symmetry-aware variants exist because many objects admit several valid poses. The Average Distance of Model Points (ADD) is the standard for asymmetric objects: with predicted pose $(\hat{\mathbf{R}},\hat{\mathbf{t}})$, ground truth $(\mathbf{R},\mathbf{t})$, and mesh vertices $\mathcal{M}$,

and a pose counts as correct when $\mathrm{ADD}<0.1\,d$ for object diameter $d$, the ADD-0.1d threshold introduced with LINEMOD [766]. For symmetric objects ADD-S replaces direct vertex correspondence with nearest-neighbor matching,

under the same threshold; the two are reported together so that symmetric and asymmetric objects are diagnosed separately. For AR overlay the relevant error is in image space, so the 2D projection error averages the pixel distance between mesh vertices projected under the two poses, with 5 px as the usual threshold. Category-level evaluation on REAL275 counts a prediction correct when

and adds 3D box IoU at $50\%$ and $75\%$ to score pose and size jointly. The BOP Challenge [771] averages recall over three symmetry-aware errors, each thresholded at several operating points: the Visible Surface Discrepancy (VSD), which scores only the visible object surface and is therefore robust to occlusion, the Maximum Symmetry-aware Surface Distance (MSSD), the worst-case surface deviation over all valid symmetries, and its image-space analogue MSPD,

averaged over the BOP core datasets. ADD and ADD-S remain the default for LINEMOD-style instance-level work, BOP-AR for cross-dataset comparison, and projection error for AR; the metrics persist side by side because no single norm captures both strict pose error and the equivalence-class structure that VSD and MSSD make explicit.

Table 6.16 reports LINEMOD-Occluded, the most widely used instance-level benchmark under the BOP protocol. Among RGB methods scored by ADD(-S)-0.1d, explicit geometry steadily closes the gap over direct regression: PoseCNN’s 24.9 rises to 40.8 with per-pixel keypoint voting (PVNet) and to 62.2 with geometry-guided dense coordinate regression (GDR-Net). On the symmetry-aware BOP-AR, the render-and-compare CosyPose (63.3) and the dense-correspondence SurfEmb (75.8) are now matched or exceeded by model-free systems with no per-object training, FoundationPose reaching 78.8 against MegaPose’s earlier 58.3; this is the 2023–2024 shift that made novel-object pose a deployment target rather than a research curiosity. The gap between the two families is widest on the symmetric objects of T-LESS, where losses that model symmetry (BOP-AR-aware objectives, ADD-S optimization) generalize better than those assuming a single asymmetric pose.

The onboarding bottleneck has shifted from pose to detection. Moving to novel objects relocated the limiting stage rather than removing it. On the BOP Challenge 2024 [764], 6-DoF localization of unseen objects has nearly caught up with the seen-object regime (the strongest training-free entry, a FreeZe variant [763], trails the best seen-object method by a few points of average recall), whereas detection of unseen objects still lags its seen-object counterpart by roughly a third and is now the largest error source in the novel-object stack. The open problem is zero-shot detection and segmentation of a never-seen instance in clutter, together with the cost of onboarding it: BOP 2024 added a track in which an object is registered from a short reference video, Any6D [774] registers from a single RGB-D anchor image, and how few views, or whether a single image or a text prompt, suffice is the defining frontier question. The same challenge pivoted toward high-resolution AR/VR headset recordings, a signal that the benchmark-realism gap, including articulated and deformable objects, open-vocabulary targets named rather than modeled (POPE [775]), and egocentric capture, is now a first-order concern.

Absolute metric scale remains ambiguous for model-free targets. The true size of an unseen object is under-determined from RGB alone, since scale and depth trade off along the viewing ray. Category-level 9-DoF methods absorb this by regressing size in a canonical frame [710], and model-free systems lean on a depth sensor or a known CAD extent; when neither is available, scale is the residual unknown separating a plausible pose from a usable one, which Any6D [774] makes explicit by coupling pose recovery with scale alignment, and monocular depth priors inherit the affine ambiguity of §6.2.1.

Accuracy trades sharply against latency. The methods that top the novel-object leaderboard are the least suited to closed-loop use: on BOP 2024 [764] the most accurate unseen-object method runs in tens of seconds per image, close to an order of magnitude slower than both the fastest seen-object pipelines and the practical sub-second entries, which buy speed with accuracy. Heavy segmentation, render-and-compare, and diffusion priors (Diff9D [765]) are the cause, which is why one-correspondence template matching (GigaPose [760]), distillation, and selective invocation of expensive components have become load-bearing for robotic and AR loops at frame rate.

Symmetry and geometric ambiguity are irreducible. Symmetric and texture-less parts of the kind that fill T-LESS [768] admit a continuous or discrete family of photometrically indistinguishable poses, and any loss assuming a unique target diverges on them; transparent and specular surfaces violate the brightness-constancy and depth-sensing assumptions behind correspondence and RGB-D fusion; rotation parameterization adds degeneracies that continuous 6D or axis-angle encodings reduce but do not remove; and heavy occlusion, the regime of LINEMOD-Occluded [767], starves the estimator of the evidence needed to resolve any of them. Generative multi-hypothesis pose (Diff9D [765]) predicts a distribution instead of a point, but turning that distribution into a calibrated confidence a planner can trust in safety-critical manipulation and driving is still largely unaddressed.

Across these problems, pose pipelines increasingly consume foundation-model priors as plug-in modules (promptable segmentation for zero-shot detection, monocular depth for inference-time geometry) rather than training pose-specific backbones, and the boundary between pose estimation, 6-DoF tracking, and neural reconstruction is dissolving, as FoundationPose [759] folds estimation and tracking into one loop and the reconstruction discussion of Chapter 8 pursues further. Promptable, real-time, uncertainty-aware pose is the target the next generation of methods converges on.