6.4Spatio-Temporal Perception

Given consecutive frames $I_{t}$ and $I_{t+1}$ (optionally with a stereo partner or several views, and with intrinsics and relative poses when known), scene flow estimation returns a field $\mathbf{v}(\mathbf{x})\in\mathbb{R}^{3}$ that displaces the surface point imaged at pixel $\mathbf{x}$ in frame $t$ to its position at $t+1$, in camera or world coordinates. In the stereo setting this is equivalently a triple $(d_{t},d_{t+1},\mathbf{f})$ of disparity in both frames plus optical flow, which is how the KITTI benchmark represents and scores it. In image-only pipelines, structure and motion are inferred from appearance, parallax, and temporal correspondence. Geometry-aware refinement modules may instead take depth estimated upstream or measured by RGB-D sensors; point-cloud-only methods are outside the scope here.

Methods differ along two axes: what they observe (stereo, the classical and still dominant setting; monocular video, the self-supervised frontier; multi-view rigs) and what supervises them (synthetic ground truth, photometric and geometric consistency, or view synthesis). Figure 6.21 groups them into five families along these axes.

Table 6.25 lists the image-based benchmarks. Synthetic sets provide dense, exact ground truth at scale (FlyingThings3D as the standard pretraining corpus; Sintel and Spring for film-grade motion, with Spring at HD resolution and super-resolved ground truth; TartanAir for environment diversity), while KITTI Scene Flow 2015 [955], built on the KITTI suite [173], is the only real-world benchmark and carries sparse LiDAR-derived labels on 200 training and 200 test scenes. LiDAR-based scene-flow splits of nuScenes [445], Waymo Open [446], and Argoverse 2 [661], and the event-camera flow of DSEC [941], serve the point-cloud and event communities and are not image-based benchmarks in the sense used here.

KITTI scores scene flow in image space. A pixel’s disparity in frame $t$ (D1), its disparity in frame $t{+}1$ (D2), and its optical flow (Fl) are each counted as outliers when the error exceeds both 3 px and 5% of the ground-truth value, and a scene-flow outlier (SF) is any pixel that fails at least one of the three; the benchmark reports outlier percentages over all pixels and separately over background and foreground. Point-level evaluation on synthetic data instead uses the 3D end-point error

with $\hat{\mathbf{v}}_{i}$ and $\mathbf{v}_{i}$ the predicted and true motion of point $i$, together with accuracy ratios at fixed thresholds (the fraction of points with error below 0.05 m or 5%, and below 0.1 m or 10%) and an outlier ratio (error above 0.3 m or 10%), conventions inherited from point-cloud scene flow.

Table 6.26 collects KITTI Scene Flow 2015 test results for the explicit-output families. RAFT-3D more than halves the SF-all error of the best classical method (PRSM), and the foreground column, where rigidly moving vehicles meet non-rigid motion, is where the learned methods gain most. The monocular rows are a different regime: Mono-SF, with a learned depth prior, sits at 23% SF-all, and Self-Mono-SF without any ground truth at 49.5% (33.9% after fine-tuning), the price of giving up stereo. The implicit-field family has no row because it reports rendering quality (PSNR, LPIPS) on its own scene sets rather than motion error.

The supervision landscape is lopsided. Every dense-ground-truth benchmark in Table 6.25 is synthetic and stereo, so monocular methods are scored only against KITTI’s sparse labels, and models pretrained on FlyingThings3D or Sintel still lose accuracy on real driving footage after fine-tuning, most visibly on foreground motion where synthetic random-object trajectories differ from vehicle kinematics. All current benchmarks also emphasize rigid or random-object motion; dense ground truth for non-rigid humans, hands, and animals does not exist at scale, which leaves the family of implicit-field methods that handle such motion best without a quantitative yardstick.

That yardstick problem is the second challenge. Explicit-output methods [948, 949, 951] are scored by KITTI’s outlier rates; implicit-field methods [188, 187, 191] by PSNR and LPIPS of rendered views. No protocol evaluates both on the same input, so the two paradigms are compared only qualitatively, and whether a motion field is good depends on whether scene flow is the end (the KITTI leaderboard) or a means (reconstruction, novel-view synthesis, physical reasoning). The cost profiles diverge as well: RAFT-3D needs several GRU iterations per frame, variational methods minutes per frame, and implicit fields minutes to hours per scene.

Foundation priors are reshaping the monocular line. Depth Anything [140] and DINOv2 [958] supply the per-pixel depth and semantic features that Self-Mono-SF had to learn from scratch, and SAM 2 [482] segmentation together with the object-pose estimators of Section 6.2.5 makes an object-centric formulation (a per-object SE(3) trajectory plus residual non-rigid motion) practical, a bridge to the tracking problem of Section 6.4.2. As implicit-field scene flow merges into 4D reconstruction, with 4D Gaussian Splatting [191] the early example, evaluation is likely to follow the downstream task rather than the KITTI leaderboard.

Given a frame sequence $\{I_{t}\}_{t=1}^{T}$ (a video, or RGB-D, LiDAR, or multi-view streams), a tracker returns a set of trajectories $\{\mathcal{T}_{k}\}$, each $\mathcal{T}_{k}=\{(t,b_{t}^{k})\}$ a time-indexed series of states $b_{t}^{k}$ with one identity $k$ held fixed over the sequence. The state is a bounding box in the basic setting and a segmentation mask, a 3D box, or a set of keypoints in the variants.

The variants, illustrated in Figure 6.22, differ in how the targets are specified and what state is tracked. Single object tracking (SOT) [963, 964] follows one target given its first-frame box; multiple object tracking (MOT) [965, 966, 711] tracks every instance of given categories, typically pedestrians and vehicles, under identity constraints; MOTS [967] and video object segmentation replace boxes with masks; 3D-MOT [968, 429, 445, 446] outputs world-coordinate 3D boxes from LiDAR, RGB-D, or multi-view input; pose tracking [549, 969, 970] and landmark tracking [971] follow skeletal joints or facial keypoints; and panoptic tracking combines pixel-level semantic classes for stuff with temporally consistent instance identities for things. Category-specific trackers for pedestrians [972], vehicles [973], or animals [974] specialize any of these.

Figure 6.23 organizes the methods into four families around image-plane box and mask tracking (SOT, MOT, MOTS, VOS). Pose, landmark, and panoptic tracking reuse the same propagate-or-associate machinery with a different state representation and are covered at the benchmark level in Table 6.27.

Table 6.27 groups the benchmarks by what is tracked. The SOT line runs from short, category-agnostic clips (OTB100, TC128, the VOT challenges) to large, long-term, category-diverse training sets (TrackingNet, LaSOT, GOT-10k, whose test classes are disjoint from training) and to condition-specific tests for aerial footage (UAV123), high frame rate (NFS), and language initialization (TNL2K). MOT benchmarks are pedestrian-only in MOT16/17 and MOT20 and multi-class driving footage in BDD100K; the 3D-MOT counterparts are the nuScenes and Waymo tracking tasks. DAVIS and YouTube-VOS supply mask-level ground truth, the latter with categories unseen at training time; the RGB-D sets add depth and have grown from tens of sequences (PTB, STC) to over a thousand (RGBD1K); and pose tracking moved from single-person upper-body clips (VideoPose2.0, Poses in the Wild, ICDPose) to multi-person full-body sequences with person identities (PoseTrack18, CMU Panoptic).

Each line has its own metrics. SOT uses the success plot: with $\mathrm{SR}(\tau)$ the fraction of frames whose predicted-box IoU exceeds $\tau$, the headline number is its area under the curve,

reported with normalized precision (the fraction of frames whose center error, divided by the target size, is below a threshold) on LaSOT and TrackingNet, and with average overlap (mean IoU over frames) and $\mathrm{SR}_{0.5}$, $\mathrm{SR}_{0.75}$ on GOT-10k; the VOT challenges use expected average overlap (EAO), which folds accuracy and failure rate into one expectation [1015]. MOT scores detection and identity jointly. MOTA [1031] charges false negatives, false positives, and identity switches against the number of ground-truth objects,

and so is dominated by detection; IDF1 is the F1 score of identity-level true positives, $\mathrm{IDF1}=2\,\mathrm{IDTP}/(2\,\mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN})$, and emphasizes identity preservation; HOTA [1032], $\sqrt{\mathrm{DetA}\cdot\mathrm{AssA}}$ averaged over localization thresholds, balances the two and has become the primary MOT metric. VOS reports the Jaccard index $\mathcal{J}=|M\cap G|/|M\cup G|$ of predicted and true masks and the contour F-measure $\mathcal{F}$ from boundary precision and recall [244], averaged as $\mathcal{J}\&\mathcal{F}$.

Table 6.28 reports the methods of §Methodology on the benchmarks each line treats as standard. Within SOT the step from a discriminative head over correlation features (DiMP) to attention-based fusion (TransT, STARK) is worth about ten points of LaSOT AUC, and the one-stream design of OSTrack adds six more over TransT. On MOT17 the detect-then-associate trackers lead, and ByteTrack’s gain over FairMOT comes from association alone; the end-to-end trackers (CenterTrack, TrackFormer, MOTR) trail on MOT17 but MOTR leads ByteTrack by 6.5 HOTA on DanceTrack [1000], where appearance cues are weak. VOS accuracy on DAVIS-17 has moved from the mid-80s (XMem, DeAOT) to above 90 $\mathcal{J}\&\mathcal{F}$ with SAM 2, which is also the first method trained on a corpus far larger than the benchmarks themselves.

Identity under occlusion remains the central failure. Motion models and appearance embeddings bridge short occlusions, but long occlusions and dense crowds still produce identity switches and fragmented tracks [997, 1001], and memory-based propagation drifts or forgets over long videos even when each step is accurate [989]. HOTA and IDF1 measure the outcome but not robustness to occlusion length or the calibration of track confidence, so the metrics reward the easy part of the problem.

Trackers are also tuned to a few curated benchmarks and transfer poorly to UAV footage, thermal and event imagery, adverse weather, and multi-camera surveillance; foundation-model trackers such as OneTracker [1006] and the RGB-X benchmarks UASTrack [1007] and UNTrack [1008] address this directly but inherit the biases of their pretrained backbones. Long streams add memory and compute limits on top of drift.

The accuracy of Transformer trackers comes at a cost that keeps them off embedded hardware, while the lightweight correlation-filter line gives up robustness; and the unified models that would amortize that cost across tasks meet negative transfer between objectives with different demands, mask precision against identity association in particular [1003]. Language and segmentation priors widen what a tracker can be asked to do, at the price of prompt sensitivity that current benchmarks do not measure.

Temporal Action Localization (TAL) takes an untrimmed video $V=\{x_{t}\}_{t=1}^{T}$, where $x_{t}$ may be a visual, audio, or multimodal feature, and detects all instances of $K$ predefined action classes, returning $\mathcal{A}=\{a_{i}\}_{i=1}^{N}$ with $a_{i}=(t_{i}^{\mathrm{start}},\,t_{i}^{\mathrm{end}},\,c_{i},\,s_{i})$: start and end times, a class label $c_{i}\in\{1,\ldots,K\}$, and a confidence $s_{i}$ (the $(t_{s},t_{e},c)$ tuples of Section 6.3.4).

Temporal Sentence Grounding (TSG), also called natural language video localization or video moment retrieval, pairs $V$ with a sentence $q$ and returns a ranked list of moments $\mathcal{M}(q,V)=\{m_{j}\}_{j=1}^{M}$, $m_{j}=(\tau_{j}^{\mathrm{start}},\,\tau_{j}^{\mathrm{end}},\,r_{j})$, with boundaries and a relevance score $r_{j}$; usually one segment is the answer. TAL is thus an exhaustive detection task over a closed label set, TSG a targeted retrieval task guided by open-ended language.

Both are studied under several settings, which Figure 6.25 lays side by side. Offline methods see the whole video; online methods infer causally over the streamed prefix and are judged on low-latency detection of action starts [1036, 1037]. Supervision ranges from frame-accurate boundaries to video-level labels only (weakly supervised TAL via multiple-instance learning, Section 6.3.4; weakly supervised TSG by aligning the two modalities during training [1038]), with self-supervised video pretraining [1039, 1040] and open-vocabulary or zero-shot variants [1041] reducing the labeled data needed. TSG additionally varies in scope, from a single video [1034, 1033] to corpus-level retrieval that first finds the video and then the moment (VCMR) [1042], and in output, with QVHighlights [1043] asking for a saliency ranking of highlight clips alongside the grounded moment.

Figure 6.24 traces both tasks along the same path from hand-designed pipelines to end-to-end Transformers.

The TAL row is the detection lineage of Section 6.3.4: sliding windows over hand-crafted features such as improved dense trajectories [1044], then two-stage proposal-plus-classification systems (R-C3D [1045], SSN [883], TAL-Net [1046]) with dedicated boundary-sensitive proposal generators (BSN [884], BMN [1047], G-TAD [1048]), then anchor-free dense detectors and query-based set prediction. Three developments go beyond that treatment. Dense single-stage detectors were scaled by ActionFormer [1049], a multi-scale Transformer encoder over the feature pyramid, and TriDet [1050], which replaces self-attention with a convolutional scalable-granularity-perception module and a trident boundary head; both still regress boundaries at every position and rely on non-maximum suppression. DETR-style set prediction removes anchors and suppression: RTD-Net [1051] applies a relaxed Transformer decoder to proposal generation and TadTR [1052] decodes the full action set end to end [1053, 1054]. And masked video pretraining [1039, 1040] now supplies the backbones that these heads are fine-tuned on.

TSG began with cross-modal matching. CTRL [1034] embeds the sentence and candidate clips in a shared space and ranks clips, with boundary regression on top; MCN [1033] does the same for DiDeMo with global video context. Later models reason over the candidates jointly: MAN [1055] refines moments iteratively through a graph over proposals, 2D-TAN [1056] scores every (start, end) pair on a two-dimensional temporal-adjacency map so that neighboring candidates inform each other, and VSLNet [1057] drops proposals altogether and predicts the start and end positions as span extraction, in analogy to reading comprehension. Moment-DETR [1058] brought set prediction to grounding, decoding moments and saliency scores directly from an encoder–decoder, while TVR [1042] extended grounding to corpus scale with a coarse-to-fine search over subtitles and frames.

Table 6.29 lists the benchmarks; the TAL rows are those of Section 6.3.4, retained for comparison. The TSG benchmarks move from a closed domain (TACoS [1059], cooking) to open-domain sentences over indoor activity (Charades-STA [1034]), Flickr videos with multiple references per query (DiDeMo [1033]), and dense paragraph-level captions (ActivityNet Captions [1035]); YouCook2 [1060] adds procedural structure, TVR [1042] multimodal (subtitle) context and corpus-level retrieval, and QVHighlights [1043] saliency labels for joint highlight detection.

Temporal IoU between a predicted and a ground-truth segment is the primitive for both tasks, and TAL is scored by mAP at fixed tIoU thresholds (per-threshold on THUMOS14, averaged over $[0.5{:}0.95]$ on ActivityNet) and by AR@AN for proposal quality [1062], as defined in Section 6.3.4. TSG uses retrieval metrics instead. For queries $\mathcal{Q}$, ranked predictions $\hat{m}_{j}$, and reference segments $G(q)$,

is the fraction of queries for which some top-$n$ prediction overlaps a reference by at least $\theta$ (commonly $n\in\{1,5\}$, $\theta\in\{0.3,0.5,0.7\}$), and $\mathrm{mIoU}=\frac{1}{|\mathcal{Q}|}\sum_{q}\max_{g\in G(q)}\mathrm{tIoU}(\hat{m}_{1},g)$ scores the top prediction alone; QVHighlights adds mAP over tIoU thresholds [1043]. Recall at a threshold binarizes overlap and ignores the quality of the ranking, which motivated ranking-sensitive alternatives such as AxIoU [1063], and annotator disagreement about where a described moment begins and ends puts a ceiling on how precisely any of these can be measured.

Table 6.30 gives R@1 results for the TSG methods above; TAL numbers are in the table of Section 6.3.4. Relational reasoning over candidates (2D-TAN) and span prediction (VSLNet) roughly doubled CTRL’s R@1 at tIoU 0.5 on Charades-STA and raised it by half again at 0.7, and Moment-DETR adds a further step on Charades-STA without proposals or suppression; its QVHighlights numbers are the reference point for joint grounding and highlight detection.

Boundaries are soft. Actions span sub-second gestures to multi-minute activities, which no single temporal resolution serves, and annotators disagree about where an instance starts and ends; a sentence adds its own ambiguity about which part of an activity it describes. The tIoU-thresholded metrics treat these fuzzy boundaries as exact, so part of the measured error is irreducible [1063], and probabilistic boundary modeling with calibrated confidences is the natural response.

The two tasks are converging. A closed label set limits TAL, while TSG is bounded by the vocabulary of its query sets; text prompts in place of class labels [1041] and set prediction for moments [1058] point to a single architecture that grounds both categories and sentences. Doing this over minutes of video, across camera viewpoints (egocentric footage in particular), with audio, narration, or noisy subtitles as further evidence [1042, 1043], and causally for streaming use [1036] is where the current systems fall short: memory-efficient long-context encoders and corpus-scale coarse-to-fine retrieval are the two ingredients most actively developed.