6.5Multimodal Spatial Perception

The input is an audio stream $A$ (mono, binaural, or multichannel, optionally ambisonic [326]) time-aligned with a video $V$ (RGB, depth, or multi-view) of the same interval; the audio may contain zero, one, or several sources, and the video may show silent objects that look like sounders. The output localizes the sounding entity in a form that depends on the variant: a 2D heatmap or mask, a face or body box with a speaking state, an event class with its temporal extent and direction of arrival (DOA, azimuth and elevation) and optionally distance [1065], a 2D location with depth, or a 3D point or trajectory in camera or world coordinates. When the source is off screen or silent, the correct output is a null prediction rather than the nearest plausible object.

Five variants are in common use. Visual sound source localization (VSSL) highlights the sounding region of a frame or clip as a class-agnostic heatmap, in single- and multi-source settings and increasingly with negative or off-screen audio as a robustness test. Active speaker detection (ASD) decides who is speaking and localizes the talker as a face track, often under streaming constraints [1064]. Audio-visual sound event localization and detection (AV-SELD) couples what and when with where: event class, temporal extent, DOA, and in recent benchmarks distance [1065]. The 2.5D variant adds depth or range to a 2D localization from monocular depth [140], stereo, or echoes [328]; and 3D DOA estimation returns a spatial bearing, whereas source-position or trajectory estimation additionally uses range or triangulation constraints from calibrated cameras, arrays, or egocentric rigs [681].

Figure 6.26 maps the five variants onto four method families.

Table 6.31 pairs each variant with its benchmarks and metrics; there is no shared benchmark across variants because their outputs differ.

VSSL is scored by thresholding the localization map and computing its IoU with the annotated region (cIoU, the fraction of test images above 0.5) and the area under the IoU-threshold curve (AUC) [1066]; the extended VGG-SS and Flickr protocols add clips whose audio has no visible source and report average precision over the decision to localize at all [1067, 1070]. ASD uses detection-style mAP over face tracks with speaking labels [1080]. SELD combines detection (F-score, error rate) with localization (DOA error, localization recall) [1075], with distance error added in STARSS23 [1065]; 2.5D inherits the depth metrics of Section 6.2.1, and 3D settings report angular and positional error.

Table 6.32 collects the two variants with a stable leaderboard. On VSSL the negative-audio protocol changes the ranking: LVS leads the earlier methods on the standard Flickr test but drops to 9.8 AP when silent clips are mixed in, whereas SLAVC holds both. ASD on AVA-ActiveSpeaker has moved from 79.2 mAP for the dataset baseline to above 93 with explicit temporal and inter-speaker modeling, and the remaining differences among top systems are about latency and streaming rather than accuracy. AV-SELD has no such table: each DCASE edition changes the data and the metric, and the STARSS23 distance extension is not comparable to the DOA-only tracks that SELDnet and ACCDOA were evaluated on.

Co-occurrence is not causation. A model that has learned which objects usually make which sounds will localize a guitar whether or not it is playing, and the negative-audio protocols [1067, 1070] exist because the standard tests rewarded exactly this; the open problem is grounding on the acoustic evidence itself, which matters most for off-screen and silent cases where the right answer is no localization.

Going from 2D saliency to metric 3D requires semantics and geometry together. Depth foundation models [140] (Section 6.2.1) and self-supervised visual features [958] now supply priors that AV-SL pipelines previously learned from scratch, and STARSS23’s joint event, time, DOA, and distance protocol [1065] is the first benchmark that scores all four at once; multi-object objectives for overlapping events are the current frontier there.

Multiple simultaneous sources, identity through occlusion, and on-screen/off-screen transitions over long videos [1069, 1081] are the AV-SL form of the identity problem of Section 6.4.2 and the pose-continuity problem of Section 6.2.5. Streaming use adds causal encoders, low-latency fusion, calibration drift, and graceful fallback when one modality is missing [1064, 1072].

The input is a scene $S$ (an image, a video, or a 3D environment) and a query $Q$, from a bare noun phrase (“the red cup”) to a sentence with attributes, relations, actions, or an implicit goal (“the person who is waving”, “something to open the box with”). The output $\mathcal{R}(Q,S)$ is the referred region in the form the variant requires: a box, a mask, a set of regions (possibly empty), a per-frame mask track, or a 3D box. Two formulations recur through the methods below: ranking a set of candidate regions $r$ by a cross-modal score, or regressing the region directly,

with $\phi$ and $\psi$ the text and region encoders and $\mathcal{C}(S)$ a proposal set.

Figure 6.27 lays out the five variants, which differ in scene modality, output granularity, and query complexity.

Referring expression comprehension and segmentation (REC / RES). One image, one expression, one target, returned as a box (REC) or a mask (RES). RefCOCO and RefCOCO+ [1082] (short phrases, the latter without location words) and RefCOCOg [1083] (longer descriptions), all on MS COCO [440] images collected through ReferItGame [1088] or Mechanical Turk, have been the standard benchmarks for a decade; Flickr30k Entities [1089, 1090] grounds every entity mention of a caption at once.

Generalized REC / RES (GREC / GRES). An expression may match zero, one, or several objects (“the red carrots” versus “the unicorn”), which single-target methods cannot represent. gRefCOCO [1084], Ref-ZOM [1091], PhraseCut [1092], and OmniLabel [1093] evaluate this setting, and MaskGroups-HQ [1094] generalizes the prompt itself to arbitrary groups of masks specified by text and reference masks. The output is a set, so the metric must be as well (§Datasets and Metrics).

Reasoning grounding. Some queries require inference beyond noun matching: “the loser” needs a judgment about the people shown, “turn on the TV” needs the remote. CLEVR-Ref+ [1095] diagnoses this on rendered scenes with ground-truth reasoning programs; ReasonSeg [1085] poses implicit, multi-step queries on real images with mask output, extended by MMR [1096] (194K implicit queries, multi-target and part-level), ReasonSeg-Inst [1097] (instance level), and MC-Bench [1098] (instances across image pairs). Ref-Adv [1099] keeps the classic REC format but makes expressions minimally sufficient, negation-heavy, and paired with same-category distractors.

Video referring and reasoning grounding. The referent must be found in every frame through motion, appearance change, and occlusion, and the query may name it by what it does (“the man waving his hand”). Refer-Youtube-VOS [246], Refer-DAVIS17 [1100], A2D- and JHMDB-Sentences [1101], MeViS [1102], ReVOS [1103], and ReasonVOS [1104] add language to existing segmentation and actor-action datasets; Ref-VPS [1105] extends the referent from objects to processes such as smoke spreading or waves breaking, which have no object boundary at all. Tracking machinery is usually folded in to carry identity across frames [1106].

3D and embodied grounding. The scene is a point cloud or a set of views, and the query refers to an object in 3D (“the chair next to the window”). ScanRefer [373] requires detecting the referent as a 3D box and Multi3DRefer [1107] allows zero to many targets; ReferIt3D [374] provides the object instances and scores selection among them; Instruct3D [1108] and Multi3DRes [1109] ask for point-level masks from instruction-style queries. Embodied grounding makes the output an action: an agent told “go pick up the red cup on the right” must identify the cup and reach it [201].

Figure 6.28 organizes the methods into eight families, (A) to (H). They are trajectories rather than eras: the oldest paradigm has resurfaced at foundation-model scale.

Table 6.33 lists the benchmarks by variant. Annotation cost rises steeply from a 2D box to a per-frame mask to a 3D box, so image-level REC/RES corpora dominate in size and maturity, video referring sets are an order of magnitude smaller, and 3D corpora smaller still. Within the image benchmarks a generational split has opened. RefCOCO/+/g now serve mainly as training corpora: specialists and generalist MLLMs report Acc@0.5 above 90% on them, an audit found label errors in 14%, 24%, and 5% of RefCOCO, RefCOCO+, and RefCOCOg [1137], and much of the score survives shuffling the word order of the expression [1099]. The evaluation-oriented replacements are built to expose what saturation hides: Ref-L4 [1137] (cleaned labels, long expressions, a 365-category vocabulary), Ref-Adv [1099] (shortcut-suppressed, where the same models fall to roughly 60%), HC-RefLoCo [1138], GigaGrounding [1139], and GroundingME [1140], which scores discrimination, spatial reasoning, limited visibility, and the rejection of ungroundable queries, and on which the best model resolves 45% of samples while most score zero on rejection.

Every metric here is built on the IoU of predicted and reference regions (Section 6.4.2 gives the mask form). REC reports Accuracy@0.5, the fraction of expressions whose box reaches IoU $\geq 0.5$, with stricter thresholds and the threshold-averaged mAcc [1137] increasingly added; phrase grounding, where several proposals may be returned, reports Recall@$k$. For set-valued outputs (GREC / GRES) the prediction is compared with the ground-truth set through per-expression F1 over boxes or masks [1091] and a no-target accuracy for expressions with no referent [1084], and OmniLabel’s AP${}_{\text{descr}}$ computes average precision over free-form descriptions including negative ones [1093]. Segmentation reports two aggregates that differ in weighting: gIoU (also called mIoU) averages per-sample IoU and so weights every expression equally, whereas cIoU accumulates intersection and union over the whole set before dividing and so weights large objects more; reasoning and referring segmentation papers report both [1085], often with Precision@$X$ for the distribution of mask quality and a boundary IoU for edge accuracy [1122]. Video referring segmentation uses the $\mathcal{J}\&\mathcal{F}$ composite of Section 6.4.2.

Table 6.34 reports the two variants on which the families can be compared. On REC the step from proposal ranking and modular reasoning (MAttNet) to end-to-end regression (TransVG) and then to grounded pre-training (MDETR, Grounding DINO) is about 14 points of RefCOCO Acc@0.5, and the generalist MLLMs (Qwen2.5-VL, InternVL 2.5) sit at or above the best specialist without benchmark-specific training. On RES the grounding MLLMs with a mask decoder have passed the end-to-end specialists (CRIS to Sa2VA is eleven cIoU points on RefCOCO), and Sa2VA’s video columns show the same model carrying over to referring video segmentation; on ReasonSeg, LISA-7B reaches 52.9 gIoU after fine-tuning where open-vocabulary segmenters stay below 30 [1085]. The tool-augmented and generative families are evaluated zero-shot on subsets of these benchmarks and are not directly comparable here. What the table cannot show is cross-variant breadth: methods strong on image REC routinely degrade on video, reasoning, and 3D grounding, and no single architecture leads all five variants.

Ambiguity and interaction. “The guy on the right” depends on the viewpoint, and “the red cup” presumes one salient red cup. People resolve this through shared context or a follow-up question; grounding models make an arbitrary choice or fall back on a dataset bias such as the largest instance. Recognizing that an expression matches several entities, abstaining under uncertainty, and asking for clarification are outside almost every current benchmark, which is one-shot (one query, one image), while real use is a dialogue with memory of what was grounded before.

Reasoning and commonsense. “The tool needed to open the box” cannot be resolved from the image alone, and adversarial evaluation shows that much of current REC accuracy rests on shortcut matching rather than reasoning [1099]. LLM-based models carry the knowledge but may answer “a key” when none is visible, and vision-only models lack it entirely; reasoning-driven segmentation [1085] and agentic active perception [1133] are the two bridges. Video adds temporal and causal structure (“the man who picks up the phone and then leaves”, “the car that caused the accident”) that current video grounding, which attends mostly to appearance and simple motion, does not capture.

Open world and 3D. Open-vocabulary detectors and CLIP-based models have blind spots on rare objects, fine-grained distinctions (viola versus violin), proper nouns, and figurative references (“the blue crown” for a cap), and per-category analyses of large multimodal models show a strong bias toward common training categories [1137]. Description-conditioned pre-training [1116] and generative pre-training [1105, 1136] are the routes to long-tail coverage that do not require labeling it. Functional references (“the driver’s seat”) also imply spatial context that only 3D understanding resolves, and the 3D-language work surveyed above [1141, 201] will grow with augmented reality and robotics.

Efficiency and evaluation. The strongest systems are multi-model tool chains or LLMs with vision adapters and do not fit AR glasses or robots; compact models [1123, 1122] and non-autoregressive selection [1094] show that accuracy does not require scale, but the accuracy-efficiency frontier is unmapped. Evaluation has the saturation problem described under §Datasets and Metrics, and the new benchmarks, each with its own protocol, risk a cycle of leaderboard chasing; a single protocol spanning image, video, 3D, and dialogue, scoring rejection and clarification as well as localization, does not yet exist.