7.3Foundation Model-driven Spatial Reasoning

Spatial reasoning, which embodied AI [56, 1445], autonomous driving [1446, 790], and spatial content generation [1447, 1448] all depend on, is cast here as inference over learned representations rather than explicit geometric computation: a pre-trained model $\mathcal{M}_{\theta}$ answers a spatial query $q$ from context $c$,

where the context may be absent (a text-only LLM), an image or image set $I$ (a VLM), or an image together with auxiliary evidence such as a scene graph, region features, or depth, and the output $y$ is a relation, a metric estimate, an orientation, or a navigation plan. For text-only LLMs the spatial prior is built from statistical patterns in massive corpora [15, 1449, 1450]: such models infer that “sky” is above “ground”, but the reasoning is ungrounded, with no mapping to physical reality or visual signal, and therefore prone to physically impossible inferences. VLMs ground the same prior by aligning the LLM with a visual encoder, and spatial competence then extends beyond recognizing objects and their intrinsic attributes to reasoning about relations among objects and to spatial extrapolation, inferring unseen states or running mental simulations from what is perceived [1451, 229].

Interventions against the spatial deficiencies of foundation models fall into four levels, in increasing order of invasiveness: prompt engineering leaves the model untouched, training strategies update its weights, data-centric augmentation changes what it learns from, and model-centric enhancement alters the architecture.

Evaluation has moved from general VQA to benchmarks that probe spatial capability along four levels [1451]. Intrinsic-attribute benchmarks test perceptual precision on single objects, with EgoOrientBench [1467] evaluating object orientation from an egocentric viewpoint and GeoMeter [1468] metric distance and dimension, and they show that even state-of-the-art models are weak at reference-free perception. Relational benchmarks, the most populated class, test the parsing of spatial relations through vision-centric questions such as relative position (CV-Bench [1469], What’s Up [1470]), and this is where general-purpose VLMs peak. Deep-spatial-logic benchmarks go beyond visible relations: SRBench [1471] evaluates mental rotation and navigation, and MindCube [1472] the construction of an internal 3D mental model from multiple views. Comprehensive suites such as RealWorldQA [1473] and OmniSpatial [1474] cover all levels at once; a wider coverage map of these static probes appears in Table 7.5 of §7.3.7.

Three obstacles limit what foundation models can be made to do spatially. Mainstream vision encoders such as CLIP optimize for global semantic alignment and capture coarse semantics at the expense of fine-grained geometric detail [1475, 1476], and the contrastive objective compounds this by inducing the “bag-of-concepts” behavior that under-supervises structural relations among objects [1477, 1478]. Training data with precise metric annotation (quantitative distance, 3D coordinates) is scarce, and existing datasets rarely support the extrapolation tasks that require reasoning beyond direct observation [1479, 1480]. Spatial reference is ambiguous: “to the left” may mean the viewer’s left (egocentric), the object’s left (allocentric), or another agent’s left, and VLMs seldom switch frame of reference from context, which misaligns interpretation in embodied use [1479, 1481].

Two directions respond to these. Reinforcement learning with process rewards such as GRPO moves training from static VQA toward interaction, letting the model acquire spatial logic by trial and error in simulation or the real world [1457]. World-model-informed pretraining attacks the 2D-to-3D gap directly, learning physical regularities (gravity, collision) alongside image-text correspondence from synthetic corpora with exact geometric ground truth [202, 1461, 229]; the use of such world models at inference time is the subject of §7.3.8.

A VLA model extends the reasoning core of §7.3.1 with an action head: given a visual observation $o_{t}$, a language instruction $\ell$, and the robot’s proprioceptive state $s_{t}$, it emits an executable action chunk

where each action encodes end-effector increments, gripper commands, joint targets, or pose waypoints that a controller converts into motion [1482, 1483]. Spatial reasoning under this paradigm is embodied grounding: language tokens must be anchored not only to visual features but to the three-dimensional geometry, affordances, and constraints of the workspace, so that the generated actions respect depth, stability, and contact rather than merely naming where objects are.

VLA methods for spatial reasoning fall into four classes by their action-generation and reasoning paradigm.

Evaluation rests on two complementary resources: generalization-oriented benchmarks in simulation and large-scale real-world embodied datasets. VIMA-Bench [1495] measures spatial generalization through a four-level protocol that runs from random object placement to fully novel task compositions; ALFRED [1307] poses long-horizon household tasks that combine high-level goals with low-level instructions; and CALVIN [1496] and LOTA-Bench [1497] test language-conditioned manipulation across environments, scoring generalization to unseen objects and configurations. Among datasets, Open X-Embodiment [1498] is the most comprehensive, integrating 22 robot datasets from 21 institutions and 527 skills to enable cross-platform generalization; BridgeData [1499] spans 71 tasks in 10 environments and shows that joint training on limited out-of-domain data raises success rates substantially; and RH20T [1500] adds vision, force, and audio across 147 tasks, suited to one-shot learning and experience-based policy transfer. Across these suites performance falls sharply along every generalization axis (L1 to L4 in VIMA-Bench, seen to unseen splits in ALFRED, single to multi-step chains in CALVIN), while data scale and embodiment diversity are the most reliable lever for cross-platform transfer and multimodal sensing (force, audio) becomes decisive where vision alone underdetermines the contact configuration.

The generalization drop above reflects a semantic–physical misalignment: VLA systems inherit their competence from LLMs and VLMs that excel at semantic abstraction and cross-modal alignment but have had little exposure to physical interaction, so instructions are often interpreted correctly and executed incorrectly. The same gap appears as pseudo-interaction: actions are generated from statistical co-occurrence in the training data rather than from an understanding of environment dynamics or causality, so when the environment is unfamiliar or changes quickly the model cannot revise its strategy from sensory feedback.

Two further limits are practical. Long-horizon consistency is unsolved for autoregressive and diffusion policies alike: over extended action sequences the spatial representation drifts or hallucinates, and existing embodied datasets, including Open X-Embodiment, are dominated by tabletop manipulation and grasping, so environmental diversity and large unstructured scenes are under-represented. Inference latency compounds across autoregressive decoding of large Transformers, and for high-frequency tasks such as dynamic grasping or mobile navigation the compute and memory demands remain prohibitive on embedded platforms.

The directions that follow are a unified token stream over vision, language, and action that lets VLA systems grow into proto world models (the world-model view is developed in §7.3.8); active perception and causal reasoning, in which the agent probes its environment and updates its policy from feedback, as the answer to pseudo-interaction; and virtual-real data synthesis (high-fidelity simulation, synthetic generation, multi-robot data sharing) as the route to embodied data at the scale that text corpora provided for language models.

The task is to answer a spatial query $q$ about a scene or environment, given context $c$ that may be purely textual, visual, 3D, temporal, or embodied. A CoT-style reasoner augments the answer with an intermediate trace

where each $t_{\ell}$ records a partial spatial commitment: a relation, coordinate estimate, subgoal, viewpoint change, object binding, or predicted action effect. The trace is useful only insofar as it improves the grounded answer $\hat{y}$ or makes the failure mode inspectable; many apparent “reasoning steps” are not grounded in metric or perceptual evidence and merely rationalize a final guess, so the central question is not whether a model can produce more text, but whether the trace helps preserve spatial invariants across transformations, viewpoints, time steps, and actions.

CoT, Tree-of-Thought (ToT) [1502], and Graph-of-Thought (GoT) [1503] differ mainly in the topology of the intermediate trace: CoT is a single path through partial spatial commitments; ToT branches over alternative hypotheses or subplans and uses a value function to prune; GoT allows shared sub-derivations, such as reusing a computed room layout or object graph across several queried relations. All three search over partial reasoning states with a proposal mechanism, an evaluator $V$, and a halting rule $H$,

and differ in whether the search structure is a path, a tree of alternatives, or a graph of reusable sub-results, and in whether the trace stays verbal or becomes executable through programs (Program-of-Thought, PoT [1504]) or environment-changing actions (Chain-of-Action, CoA [1505]). The use cases below organize how these traces are deployed.

Evaluation mirrors the three use cases. Spatial puzzles, relative-position QA, and textual maze navigation are evaluated with prompt-based CoT on the suites of [1506, 1507]; visual question answering and scene-grounded compositional queries are evaluated under CoT with program execution (§7.3.4), where the rationale’s correctness is mediated by program output rather than by the rationale string; and long-horizon planning with sensory feedback is evaluated through CoT-augmented navigation (NavCoT [1510], NavR1 [1511]), CoT-augmented instruction following (ThinkBot [1512], EmbodiedGPT [1513]), visual- and action-trace supervision for VLA models [1514, 1515], and scene-graph-augmented embodied reasoning [1516].

The transfer of CoT to spatial intelligence is uneven. Coordinate hallucination persists under simple textual CoT: even with ToT-style beam expansion, prompt-based planners mistrack absolute positions across multi-step chains in maze and grid-navigation problems [1507]. Ungrounded supervision can make things worse: textual CoT applied without a structured or rationale-grounded training signal degrades spatial VLM performance, because the model is forced to verbalize a reasoning trace it does not possess [1517, 1501], especially when the prompt rewards plausible explanation length over geometric correctness. Search cost scales super-linearly in solution depth for ToT and GoT (and in the branching factor for ToT), which limits real-time or on-device deployment. Trace–action mismatch arises in embodied settings when a VLA model produces a coherent verbal rationale while its action head follows a different latent policy; action-aware CoT aligns rationales with action tokens and visual predictions [1515], but reliable attribution between the visible trace and the executed behavior remains unresolved.

Grounded rationale supervision pairs CoT with explicit spatial signals (scene graphs [1516], depth maps, 3D coordinate frames, action traces, or layout-coordinate traces for spatial generation [1508]) so the rationale is verifiable rather than free-form. A complementary line makes the trace execution-grounded: PoT-style traces compile and run against a geometric engine or vision–language tool, with execution outputs serving as a truth signal, the natural bridge from CoT to code / DSL generation in §7.3.4; for embodied agents the parallel bridge runs from CoT to CoA and the agentic orchestration loops of §7.3.7.

Given a high-level, often ambiguous, natural-language query $q$ and access to an API $\mathcal{A}$ (a robot’s control interface, a programming-language interpreter, or a set of vision-language functions), the LLM acts as a high-level planner that emits a program and offloads execution to the external system $\mathcal{E}$,

where $P$ is written in a general-purpose language such as Python or in a domain-specific language and composed of calls that manipulate objects, query spatial relations, or control an agent, and the output $y$ is a change in the environment’s state (a robot moving an object), an answer to the query, or both.

Code/DSL generation for spatial reasoning is typically evaluated under three benchmark families that mirror the two-mode split of the §Methodology above. For 2D scene understanding, methods are evaluated on compositional visual question answering (GQA [839]), referring expression comprehension (RefCOCO / RefCOCO+ / RefCOCOg [1082, 1083, 1088]), and image-editing benchmarks. For 3D scene understanding, evaluation has converged on 3D referring expression and 3D visual grounding benchmarks (e.g., , ScanRefer [373], Multi3DRefer [1107], ScanQA [41]), which exercise both the perception-API composition and the long-tail of object-attribute reasoning. For embodied planning, the canonical evaluation suites are ALFWorld [1520] and VirtualHome [1521] (long-horizon household tasks) and Minecraft (open-world goal acquisition, as exemplified by Voyager [150]); the headline metric in this family is task-success rate over multi-step goals rather than single-step accuracy. Across all three families, the recurring methodological observation is that code-generation methods inherit the accuracy of their underlying perception or control modules, so benchmark performance tracks the module suite more tightly than it tracks the LLM backbone.

The interpretable trace comes with three recurring limits. A fixed, human-authored DSL (ViperGPT’s find / compute_depth / verify_property) restricts the model to a predefined operation set, so spatial concepts outside the API are systematically out of reach; VADAR’s dynamic API synthesis [1518] is a response but does not solve the coverage problem. Brittle execution with silent failure follows from the dependence on perceptual modules (OWL-ViT [422], OpenScene [39], LERF [369]): an upstream error propagates through the program and yields a plausible but wrong answer with no diagnostic signal. Sample-inefficient debug loops limit self-refinement agents such as Voyager [150] and Code as Policies [1366], whose LLM calls per task scale poorly in high-dimensional or stochastic environments.

Two directions follow. PoT-style execution-grounded rationales are being folded back into chain-of-thought training, with the generated program serving as both rationale and verification signal, and learnable DSLs such as LEFT [348] replace fixed perception with differentiable logic programs whose components are trainable, pointing toward a DSL that adapts under task supervision rather than being hand-engineered.

A tool-augmented reasoning system is formally defined as a tuple $(\mathcal{M}_{\theta},\mathcal{T},\pi_{\phi})$, where $\mathcal{M}_{\theta}$ denotes the reasoning core, $\mathcal{T}=\{T_{1},\dots,T_{k}\}$ is a catalogue of external tools, and each tool $T_{i}:\mathcal{I}_{i}\rightarrow\mathcal{O}_{i}$ is exposed only through a structured input–output interface. The orchestration policy $\pi_{\phi}$ governs tool selection, argument construction, response parsing, retry / fallback behavior, and feedback integration. Given an input $x\in\mathcal{X}$, the system maintains a time-ordered reasoning history $h_{t}$ (initially $h_{0}=\varnothing$) and proceeds by the iterative update

where $s_{t}$ is the intermediate reasoning state, $(i_{t},q_{t})$ is the selected tool-and-query pair, $r_{t}$ is the returned result, and $\|$ denotes appending to the interaction history. The loop halts when the policy or model emits a STOP decision, after which an output decoder $g$ produces the final answer $y=g(h_{T})$. Both the reasoning core and the tool internals are black-box; what is exposed, optimized, and evaluated is the interface history and the policy that controls it.

Two design choices interact: the tool catalogue determines which black-box capabilities are available, and the orchestration policy determines how they are sequenced, checked, and folded back into the model’s spatial answer. The catalogue roles below are complementary, not sequential.

Evaluation asks whether the orchestrator selects useful tools and converts their outputs into better spatial decisions. For execution-augmented tool use the canonical suites are SayCan [1311] (LLM-grounded household manipulation), ALFWorld [1520] and VirtualHome [1521] (long-horizon text-and-embodied environments), and Voyager-on-Minecraft [150] (open-world skill accretion through code-as-policy). For spatial-reasoning-specific tool use, SpaceTools [1547] targets the metric gap between VLM perception and physical manipulation, and Cambrian-S [1546] introduces the spatial-supersensing framework that exercises tool augmentation from perception (Stage I) to predictive world modeling (Stage IV). Validation- and memory-augmented reasoning are evaluated on the memory and agentic benchmarks of §7.3.6 and §7.3.7, and predictive tools on those of §7.3.8, where the central question is whether imagined evidence improves the final answer enough to justify its cost.

The recurring limitations originate in the LLM–tool interface rather than in any individual tool. The exploration bottleneck is combinatorial: each tool adds an action dimension and tool composition multiplies them, so naive reinforcement learning over tool use is inefficient; Double Interactive Reinforcement Learning (DIRL) [1547] warm-starts from expert demonstrations before open exploration, but the scaling of the action space itself is unresolved. The metric gap moves rather than disappears: VLMs excel at semantic perception but fail at the precise metric estimation manipulation requires, and offloading geometry to depth estimators and pose solvers lets the LLM reason over measurements instead of hallucinations [1547, 1522], yet query formulation, output parsing, and error attribution at the interface become the new precision bottleneck. Interface calibration compounds this: tool outputs arrive with heterogeneous units, confidence scores, coordinate frames, and failure modes, and an orchestrator that cannot judge whether a returned depth map, solver result, retrieved memory, or imagined rollout is trustworthy can perform worse than a direct model response.

Learned orchestration is one direction: policies are moving from hand-written call rules toward imitation- or reinforcement-learned controllers that decide which tool is worth its latency and uncertainty budget [1547]. A parallel shift is the System-1 / System-2 architectural split, in which the neural backbone handles fast, intuitive perception (System 1) while the toolset handles slow, deliberative, verifiable reasoning (System 2) [1548], achieving both flexibility and rigor.

Where §7.3.5 treats memory as one black-box tool among others, here the external memory itself is the central learning target. Formally, a retrieval- & memory-augmented system is the triple $(\mathcal{P},\mathcal{M},\mathcal{R})$, where $\mathcal{P}$ is a reasoning core (an LLM, VLM, or embodied planner), $\mathcal{M}$ is an external memory store, and $\mathcal{R}:\mathcal{Q}\times\mathcal{M}\to\mathcal{C}$ maps a query $q_{t}$ and the current memory state to a relevant context fragment $c_{t}$. Given an instruction $x$, an observation stream $o_{t}$, and a reasoning history $h_{t}$ (initially $h_{0}=\varnothing$), the system proceeds by the iterative update

where $\mathrm{Query}_{\mathcal{P}}$ extracts the planner’s information need, $\mathcal{R}$ returns the retrieved context, and $\mathrm{Write}$ inserts, revises, or invalidates memory entries using the newest observation and decision state. The decomposition brings three concrete advantages: scalability, by letting agents operate over large environments while fetching only task-relevant subgraphs or vectors; personalization, through targeted retrieval of user profiles, historical preferences, and episodic interactions; and explainability, since decisions can be traced to the specific memories, rules, or map fragments that were retrieved.

Three families of memory architecture instantiate the $(\mathcal{P},\mathcal{M},\mathcal{R})$ triple at different levels of abstraction and scale.

Progress in memory-augmented spatial reasoning requires separating memory-specific tests from broad spatial probes.

The recurring limitations originate in the Planner-Memory-Retriever interface rather than in any individual component. Query formation under unbounded context is the central bottleneck, not memory size alone: the act of deciding what to retrieve requires compressing the current task, viewpoint, uncertainty, and history into a finite representation, and memory paging and scheduling alleviate but do not remove the need for a learned retrieval policy. Write/update consistency is where append-only memories become dangerous: agents need mechanisms for reconciling conflicting observations, invalidating stale object locations, and representing uncertainty when the world may have changed since the last visit; EmbodiedBench-style protocols make the issue visible, but robust update semantics remain open. Integration mechanics are unresolved between in-context learning, which concatenates retrieved text to the prompt (simple but token-inefficient), and cross-attention fusion, which injects retrieved 3D features into attention layers (3DLLM-Mem [1560]; expressive but architecture-coupled), with no clear winner on the fidelity–modularity trade-off. Modality mismatch has no general theory: RAG-over-text (NavRAG [1549]) discards visual nuance, RAG-over-images (VisRAG [1550]) loses structural priors, and when to retrieve text, images, or 3D representations remains task-dependent.

Memory is shifting from static maps to dynamic semantic belief states: systems store probabilities, semantic labels, and interaction histories rather than coordinates, enabling agents to reason under uncertainty about where an object likely resides given commonsense and past observation, as RoboMemory [1558] typifies. Graph- and vector-based retrieval are converging into hybrid architectures that use vector search to identify graph entry nodes and topological traversal for neighborhood reasoning. Most abstractly, meta-cognitive memory such as Meta-Memory [1568] generates task-specific “cognitive maps” and retrieves strategies rather than raw data, pointing toward learnable retrieval policies and bridging into the agentic orchestration of §7.3.7.

Whereas §7.3.5 treats tools as black-box operators and §7.3.6 treats external memory as the central learning target, agentic spatial reasoning composes both into a closed-loop cycle in which the orchestration policy itself becomes the learning target. Formally, an agentic spatial reasoning system is the tuple $(\mathcal{P},\mathcal{S},\mathcal{A},\mathcal{F},\pi)$, where $\mathcal{P}$ is a planning core (an LLM or VLM), $\mathcal{S}$ is the spatial state space (perception, internal map, history), $\mathcal{A}$ is the action space (sensorimotor commands or sub-goals), $\mathcal{F}$ is a reflection / critic function that scores proposed actions or rationales, and $\pi$ is the orchestration policy. Given an instruction $x$, the system maintains a history $h_{t}$ (initially $h_{0}=\varnothing$) and proceeds by the iterative perceive–reason–act–reflect cycle

where $\mathrm{Perceive}$ extracts the current spatial observation, $\pi$ samples a proposed action, $\mathcal{F}$ supplies a reflection signal (verbal critique, planner-critic verdict, or value estimate), and the history accumulates all four. This formulation makes explicit that “agentic” is not a new capability but rather a structural promotion: the orchestration policy $\pi$ becomes the central learning target rather than the planner $\mathcal{P}$ or any individual tool.

Four orchestration problems organize the methods: grounding observations into executable state, closing the feedback loop, coordinating across agents, and using action and world-model modules without reducing agency to any one of them.

Evaluating agentic spatial reasoning requires benchmarks that go beyond static visual question answering: a useful protocol tests whether the agent can gather missing observations, choose tools or actions, use memory across long horizons, and revise its plan after feedback, and static spatial probes should be read as component diagnostics rather than direct evidence of closed-loop agency. Two protocols exercise the loop directly: iVISPAR [1581] evaluates VLMs as agents in an interactive sliding-tile setting with visual 2D, visual 3D, and text inputs, where the answer depends on multi-step state transitions rather than one-shot recognition, and TIR-Bench [1582] requires agents to use tools (zoom, crop, rotate) to manipulate images while solving spatial puzzles. Memory-centered settings (OpenEQA [1563], §7.3.6) and open-3D probes such as Open3DVQA [1567] test what the loop must retain, and the broad visual-spatial probes (SpatialEval [1450], VSI-Bench [227], OmniSpatial [1474], Spatial-DISE [1566]) identify weak spatial primitives along the axes of static vs. dynamic content, single vs. multi-view, ego vs. exo perspective, and seen vs. unseen queried information; Table 7.5 maps the coverage.

Table 7.5 should be read as a coverage map, not as a leaderboard for agentic systems. Most entries evaluate static answer generation or localized visual-spatial primitives; only interactive protocols such as iVISPAR and tool-use settings directly exercise action selection and feedback. Claims about agentic spatial reasoning should therefore report loop-level metrics (observation gathering, action choice, recovery after failure, memory update, and final task success) rather than single-turn accuracy alone.

Boundaries are collapsing between language and action (VLAs), between robot bodies (cross-embodiment), and between perception and imagination (world models), and two difficulties gate progress toward generalist spatial agents. The sim-to-real fidelity gap: high-fidelity simulators (predictive world models, generative video) accelerate training but introduce systematic biases that physical execution exposes, and the cost–precision frontier for transfer remains open. Long-horizon spatiotemporal coherence: across multi-hour episodes the agent’s internal map drifts, and episodic memory and reflection reduce but do not eliminate the drift, because the bottleneck is the finite context window of the planning core.

Explicit orchestration over observation–state–action–reflection tuples is one direction: VLA and WLA systems are beginning to expose intermediate subgoals, action intents, and predicted states that a loop can inspect rather than hiding all control inside a single forward pass. A second is mental simulation via generative world models: DreamerV3-style, Genie-style, VLWM-style, and WorldPilot-style models let agents verify spatial logic in imagination before acting, shifting the paradigm from “trial-and-error” toward “trial-in-imagination” while keeping the agentic focus on when to simulate, when to act, and when to revise.

The task addressed in this subsection is spatial reasoning itself: given a query $q$ about a scene together with the available context $c$ (a textual description, one or more observations, or a scene specification), a reasoner must produce a spatial answer $y$, either an inferred spatial property $o$ (e.g., , a relation, a structure, or a physical attribute) or a plan / policy $\pi$ that acts on the scene. What distinguishes the methods surveyed here is how they arrive at $y$: rather than mapping $c$ to $y$ in a single feed-forward pass, the reasoner consults a world model as an auxiliary instrument that can imagine how the scene would look or evolve, and reasons over that imagined evidence. Concretely, the world model is a pretrained simulator exposing a distribution $\mathbb{P}(s_{0},s_{1},\dots,s_{T}\mid a_{0},a_{1},\dots,a_{T-1},c)$ over world states $s_{t}$, or over their partial observations $\operatorname{proj}(s_{t})$ when full states are unavailable, optionally conditioned on an action sequence $\left\{a_{t}\right\}_{t=0}^{T-1}$ (null when the model is action-free). A world-model-based reasoner treats $\mathbb{P}$ as a queryable tool: given the query, it (i) conditions $\mathbb{P}$ on the context $c$ and, when planning, on a candidate action sequence; (ii) draws predictive feedback $f$ from $\mathbb{P}$, a rollout of future observations, future states, or action-conditioned trajectories, or a dynamics-aware latent; and (iii) applies a readout $g$ that maps the feedback back to the spatial answer, $y=g(c,f)$. The world model supplies the imagined evidence $f$, but producing the spatial answer $y$ remains the objective, and the fidelity of $f$ bounds how much the world model can help.

We treat the world model as a pretrained instrument and organize methods by the form of predictive feedback it returns to the reasoner. Two families emerge: generative feedback, in which the model is rolled out to synthesize explicit futures that the reasoner inspects, and latent-representation feedback, in which the model’s dynamics-aware embedding is consumed directly without decoding; the understanding-versus-planning distinction cuts across both.

Evaluation follows the usage mode. Planning-oriented use is scored on the embodied suites of §7.3.2 and on long-horizon household environments such as ALFWorld [1520] and VirtualHome [1521], where the question is whether policies extracted from imagined rollouts transfer to the real environment. Understanding-oriented use is scored on the static spatial probes that single views underdetermine, VSI-Bench [227] and MindCube [1472] among them, under test-time-scaling protocols that vary how many imagined views the reasoner is allowed and whether a verifier selects among them [1603, 1596]. The world model itself is assessed on video-generation quality (FVD, FID) together with task-specific physics-consistency probes, and surveys of general world models track the field’s shift from 2D toward 3D-consistent simulation [1604], while recent analyses argue that advancing MLLM spatial reasoning demands purpose-built recipes and data rather than scale alone [1522]. No single benchmark currently exercises a world model along perception fidelity, action conditioning, and long-horizon spatial memory at once, a gap the challenges below make explicit.

Reading the world model as a tool clarifies where the difficulties lie: some are inherited from the instrument itself, and are properly the concern of §8.6, while others are specific to how a reasoner queries it. The most active questions of 2025–2026 are no longer whether a world model can imagine, but whether its imagined evidence can be trusted and whether consulting it is worth the cost.

Imagination is not a free lunch. Systematic probes report that world-model imagination improves spatial reasoning in only a minority of cases, and that adding imagined views yields non-monotonic, sometimes negative accuracy gains at steep token and latency cost [1596, 1603]; the sheer amount of imagination is an unreliable proxy for accuracy [1522], which recasts “when and how much to imagine” as a first-class test-time-scaling problem, with adaptive gating [1596] an early answer and a predictive account of which queries need the world model still missing. Faithfulness is the companion problem: a world model can render fluent futures that are physically or relationally wrong, so likelihood under $\mathbb{P}$ is necessary but not sufficient, and when the imagined evidence lacks fidelity verification buys little, since heuristic verifiers give limited calibration gains and an information bottleneck sets in, as a reasoner cannot recover a signal the imagination never contained [1603]. Grounding imagined evidence to the actual scene and calibrating trust in it remain open; physics-consistency objectives and geometry-grounded latents [1602] are partial responses.

The reasoner can be no more reliable than the simulator it queries. Even state-of-the-art models such as Genie 3 [157], despite markedly longer-horizon consistency, still drift in spatial memory, and consistency mechanisms [1598, 1605] narrow but do not close this gap, with hallucination and long-horizon error compounding over rollout length; raising this ceiling is the province of world-model construction (§8.6), and the reasoning methods surveyed here can only mitigate its symptoms.

The feedback forms also trade off along different axes, with no principled account of which a given query should use: pixel rollouts are inspectable but expensive and prone to appearance drift; state or 4D rollouts are geometrically consistent but require structured simulators; latent feedback is cheap but opaque; action rollouts commit early to a policy; and most systems commit to one form a priori. Extending world-model-based reasoning beyond pixels (e.g., , to forces [1606], tactile, or audio modalities) is gated by the absence of internet-scale pre-training data in those modalities, a constraint that also surfaces in the multi-modal aspects of §7.2.6.

Two shifts define the current frontier. From external tool to trained-in reasoning: rather than invoking a frozen simulator at inference time, recent work trains the reasoner’s use of the world model, whether by reinforcement learning that supervises an internal state-estimation and transition-modeling loop [1601] or that rewards geometrically consistent active imagination [1597], or by distilling the world model’s imagined dynamics directly into a vision-language model so that no simulator need be queried at test time [1607]. From pixels to predictive latents: the imagined branch is increasingly consumed as a latent process whose hidden states guide reasoning, rather than as a faithfully rendered video [151, 155, 1602]. Both shifts move the field away from ever-larger, faithfully-rendered pixel simulators and toward cheaper, more trustworthy, query-efficient feedback that a reasoner can afford to consult, and can be trained to consult well.