7.5 Spatial Reasoning Benchmarks · Spatial Intelligence Handbook

Early spatial-reasoning benchmarks positioned the model as a detached observer. CLEVR [1358] and VSR [1479] ask for relative geometric relations from a static third-person viewpoint, and their 2D inputs leave depth ambiguous, a weakness that 3DSRBench [1583], MMSI-Bench [1588], DSI-Bench [1591], and STI-Bench [1590] now probe directly. SQA3D [1620], OpenEQA [1563], and EmbSpatial-Bench [1587] instead place the model inside the scene, so that it must self-localize before it can answer, and ScanQA [41], ScanRefer [373], and Multi3DRefer [1107] ground queries in point clouds and RGB-D data, where the answer is a coordinate rather than a relation. The reasoning demanded has deepened in parallel: IntPhys [1198] and Physion [680] test whether a model expects gravity, collision, and friction to hold over time, and OpenEQA and RealWorldQA [1473] ask functional questions (“Can I navigate through this gap?”, “Where would I find a cloth to clean a spill?”) in which geometry matters only through what it affords.

Data provenance splits the field. Simulators, the sources behind SpatialVLM [229], AI2-THOR [1621], and Habitat [204], yield unlimited, perfectly annotated data; real-world collections such as ScanNet [448] and OpenEQA supply the clutter, occlusion, and sensor noise that simulation does not reproduce. The resulting practice is sim-to-real: pre-train on large synthetic corpora for general spatial logic, then fine-tune on the scarcer real data. Annotated 3D data remain scarce and fine-grained metric estimation remains hard for large models; both limits recur throughout the table below.

7.5.1Comparative Analysis#

Table 7.6 compiles 61 spatial-reasoning benchmarks across four categories (physical-focused, rooted in §7.1.2; vision-based, overlapping with §7.3.1; 3D-focused, rooted in §7.3.6 and §7.3.8; and robot-environment, rooted in §7.3.7 and §7.4). A narrower 15-row subset focused specifically on agentic spatial reasoning is provided in Table 7.5 of §7.3.7; this subset filters Table 7.6 to image-and-video benchmarks that exercise agentic deployments.

Table 7.6Spatial-reasoning benchmarks surveyed in this chapter, grouped by what they test. Number is the evaluation-set size where the source paper documents one (train / test when both are given, scene counts for simulators,

\sim

for approximate figures). Perspective marks explicitly designed ego (first-person) and exo (third-person) settings; Undeclared marks attributes the source paper does not state. Data Source and Task Type follow the source papers.

Benchmark	Number	Modality	View Type		Perspective		Data Source	Task Type
Benchmark	Number	Modality	Single	Multi	Ego	Exo	Data Source	Task Type
$\bullet$ Physical Focused Spatial Reasoning
IntPhys [1198]	15k / 4.3k	3D	✓	$\times$	$\times$	✓	Synthetic	VoE plausibility
IntPhys 2 [1622]	1,416	3D	✓	$\times$	$\times$	✓	Synthetic	VoE plausibility
Physion [680]	$\sim$ 1.2k	3D	✓	$\times$	$\times$	✓	Synthetic	Physical prediction / choice
Physion $++$ [1337]	$\sim$ 10k	3D	✓	$\times$	$\times$	✓	Synthetic	Physical prediction / choice
Physical Concepts Dataset [1623]	300k	3D	✓	$\times$	$\times$	✓	Synthetic	classification
Physics-IQ [1624]	396	Video	✓	$\times$	$\times$	✓	Real	Generative physical prediction
SPhyR [1625]	1,296	Image	✓	$\times$	Undeclared		Synthetic	Physical prediction / infill
$\bullet$ Vision-Based Spatial Reasoning
CLEVR [1358]	853k	Image	✓	$\times$	$\times$	✓	Simulation	VQA
VSR [1479]	10,972	Image	✓	$\times$	$\times$	✓	Real	VQA
What’s Up [1470]	$\sim$ 8,000	Image	✓	$\times$	$\times$	✓	Real	VQA
RocketScience [1626]	482	Image	✓	$\times$	$\times$	✓	Real	VQA
SpaRE [1627]	3.4M	Image	✓	$\times$	$\times$	✓	Real/Sim	VQA
RealWorldQA [1473]	765	Image	✓	$\times$	✓	$\times$	Real	VQA
3DSRBench [1583]	2,772	Image	✓	$\times$	$\times$	✓	Real/Sim	VQA
InternSpatial-Bench [1584]	6,008	Image	✓	$\times$	$\times$	✓	Real	VQA
OmniSpatial [1474]	$\sim$ 8,400	Image	✓	$\times$	✓	✓	Real	VQA
Spatial457 [1585]	23,752	Image	✓	$\times$	Undeclared		Real	VQA
All-Angles Bench [1586]	$\sim$ 2,100	Image	$\times$	✓	Undeclared		Real	VQA
EmbSpatial-Bench [1587]	3,640	Image	✓	$\times$	✓	$\times$	Real	VQA
MMSI-Bench [1588]	1,000	Image	$\times$	✓	Undeclared		Real	VQA
MindCube [1472]	21,154	Image	$\times$	✓	Undeclared		Simulation	VQA
BLINK [1628]	3,807	Image	✓	$\times$	$\times$	✓	Real	VQA
NuScenes-SpatialQA [1629]	3.3M+	Video	✓	✓	✓	✓	Real	VQA
VSI-Bench [227]	5,130	Video	✓	$\times$	✓	$\times$	Real	VQA
VLM4D [1589]	$\sim$ 1,800	Video	✓	$\times$	✓	✓	Real	VQA
STI-Bench [1590]	2,064	Video	✓	$\times$	✓	✓	Real	VQA
DSI-Bench [1591]	$\sim$ 1,700	Video	✓	$\times$	Undeclared		Real	VQA
SITE [1565]	8,068	Video	✓	$\times$	Undeclared		Real/Sim	VQA
RoboSpatial [783]	3M	Image/3D	✓	$\times$	✓	$\times$	Real/Sim	VQA
OSD [789]	Undeclared	Image	✓	$\times$	$\times$	✓	Real	VQA
SpatialVLM [229]	2B	Image	✓	$\times$	$\times$	✓	Real/Sim	VQA
ProVision [1630]	65k	Image	✓	$\times$	$\times$	✓	Simulation	VQA
WhatIsWhere [1631]	6,000	Image	✓	$\times$	$\times$	✓	Real	VQA
SURDS [1632]	9,250	Image	✓	$\times$	✓	$\times$	Real	VQA
EarthSpatialBench [1633]	325k+	Image	✓	$\times$	$\times$	✓	Real	VQA
PureSpace [1634]	1,200	Image	✓	$\times$	Undeclared		Simulation	VQA
SAT [1635]	175k / 150	Image	$\times$	✓	✓	✓	Simulation	VQA
SpatiaLQA [1636]	9,605	Image	✓	$\times$	$\times$	✓	Real	VQA
Q-Spatial++ [1459]	101	Image	✓	$\times$	$\times$	✓	Real	VQA
$\bullet$ 3D Focused Spatial Reasoning
ScanQA [41]	41,363	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D VQA
SQA3D [1620]	33,460	3D	✓	$\times$	✓	$\times$	Real (ScanNet)	3D VQA
MMScan QA [1637]	295K	3D+Image	✓	✓	$\times$	✓	Real (EmbodiedScan)	3D VQA
OpenEQA [1563]	1,632	Video/3D	✓	$\times$	✓	$\times$	Real (HM3D, ScanNet)	Embodied QA (EM / A splits)
Space3D-Bench [1638]	1,000	3D	✓	$\times$	✓	$\times$	Real(Replica)	3D QA
Scan2Cap [1639]	39,775	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D Captioning
MMScan Captioning [1637]	172k	3D+Image	✓	✓	$\times$	✓	Real (EmbodiedScan)	3D Captioning
ScanRefer [373]	51,583	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D Grounding
Multi3DRefer [1107]	42,735	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D Grounding
SR3D [374]	83,572	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D Grounding
NR3D [374]	41,503	3D	$\times$	✓	$\times$	✓	Real (ScanNet)	3D Grounding
SpaceEra++ [1640]	925k	Video/3D	$\times$	✓	✓	$\times$	Synthetic	3D VQA
Ego3D-Bench [1641]	8,600+	Image	$\times$	✓	✓	$\times$	Real	VQA
SpatialScore [1642]	28,093 (Hard: 1,400)	Image/3D	$\times$	✓	Undeclared		Real/Sim	VQA
iVISPAR [1581]	300	3D Env	✓	$\times$	$\times$	✓	Simulation	Interactive reasoning
$\bullet$ Robot
AI2-THOR [1621]	120 (Scenes)	3D Env	$\times$	✓	✓	$\times$	Simulation	Nav/Manip
TDW [1643]	Infinite	3D Env	$\times$	✓	✓	✓	Simulation	Physical Sim
Habitat 2.0 [204]	100 (Scenes)	3D Env	$\times$	✓	✓	$\times$	Simulation	Navigation
iGibson 2.0 [1344]	15 (Scenes)	3D Env	$\times$	✓	✓	$\times$	Simulation	Interactive Nav
Habitat 3.0 [98]	211 (Scenes)	3D Env	$\times$	✓	✓	$\times$	Simulation	Social Nav
R2R [1415]	21,567	Image/Path	$\times$	✓	✓	$\times$	Real	Navigation
RxR [1644]	126,069	Image/Path	$\times$	✓	✓	$\times$	Real	Navigation

A modality migration runs from the top of the table to the bottom: the physical-focused benchmarks rely on synthetic 3D renderings, the vision-based block is image- and video-heavy with growing 3D extensions, the 3D-focused block is built on point clouds and RGB-D scans, and the robot block has moved entirely to 3D environments. The Perspective axis is imbalanced: a large fraction of vision-based benchmarks carry Undeclared ego/exo annotations, the physical-focused benchmarks lean exo (a third-person observer judging physical plausibility), and the robot block is entirely ego; the field is migrating from spectator-style observation to situated cognition faster than benchmark annotations record it. The Data-Source spectrum is bifurcated between synthetic or simulated (top block and robot block) and real (middle blocks), with the Real/Sim benchmarks (SpaRE, 3DSRBench, RoboSpatial, SpatialVLM, SITE) as the bridge that the sim-to-real training practice of the opening relies on.

7.5.2Future Directions#

Unified Spatial-Cognition Benchmark. No single benchmark in Table 7.6 exercises a model along all of perception fidelity, symbolic precision, memory persistence, and long-horizon action conditioning simultaneously. A unified benchmark covering all four §7 paradigms (§7.1 rule-based, §7.2 learning-based, §7.3 foundation-model-driven, §7.4 hybrid) would close the evaluation-attribution challenge flagged in §7.4 §Challenges and Trends.

Cross-Modal Evaluation. The current axes (Modality, View, Perspective, Data Source, Task Type) under-specify the cross-modal demand of contemporary spatial reasoning. Future benchmarks should explicitly probe modality-to-modality transfer (e.g., , 2D-trained models evaluated on 3D queries; text-conditioned policies evaluated on video-grounded tasks), so that the modality-mismatch challenge flagged in §7.3.6 (text-vs.-image-vs.-3D retrieval) becomes a first-class evaluation axis rather than a per-paper case study.

Embodied Long-Horizon Coverage. The Robot block in Table 7.6 covers navigation thoroughly but under-represents long-horizon manipulation and multi-room interactive tasks. The §7.3.7 15-row subset (Table 7.5) further restricts attention to image-and-video benchmarks; the embodied long-horizon suites ALFWorld and VirtualHome (§7.3.4) and cross-embodiment evaluation on Open X-Embodiment (§7.3.2) are the nearest existing coverage, and standardizing on them would close the long-horizon-spatiotemporal-coherence challenge flagged in §7.3.7 §Challenges and Trends.

From Descriptive to Interactive Spatial Reasoning. The Task Type column of Table 7.6 is dominated by visual question answering: the overwhelming majority of benchmarks ask a model to describe static spatial relations (“where is the object”), and only the small Robot block evaluates reasoning that is conditioned on the agent’s own actions. Yet the trajectory traced in this chapter, from passive observation to situated interaction, calls for benchmarks that test interactive spatial reasoning: predicting how an action changes the spatial state, judging affordances and reachability (“can I fit, reach, or grasp this”), and reasoning counterfactually about manipulation outcomes. Such evaluation couples the action-conditioned reasoning of §7.2.5 with the embodied hybrid pipelines of §7.4, extending the descriptive benchmarks surveyed here toward the manipulation- and affordance-grounded settings that embodied deployment ultimately demands.