Most of the following gaps are invisible in single-step evaluations and appear only when a system must integrate many views, revise its memory, act over many steps, or operate beyond room scale.
Models recognize scenes without relating views to one another. Multimodal models can name nearly every object in a room yet cannot reliably say where the objects are; Chapter 6 found them strong on contextual and functional relations and weak on distances, sizes, and angles (Section 6.3.1). Given several views of a room, they rarely notice that one was captured after an object moved [3011], and over routes spanning several streets they answer by recognizing landmarks and matching text, with little evidence of path integration [3012]. These failures admit more than one explanation: the model may hold no account of the scene against which a later view could be checked, or it may hold spatial information that it cannot associate across viewpoints, update, or use in a metric computation. Current benchmarks do not separate these cases (Section 16.2).
SLAM registers each new view against a map and treats disagreement as a signal to correct one or the other (Section 6.3.5). Whether registering views in a common frame before language is involved (Section 5.5), for instance with feed-forward models that regress point maps from image collections (Section 6.2.4), closes the gap for multimodal models is unsettled, and metric estimates must in any case survive unknown or wrong camera calibration, the main open problem even for specialist pipelines (Section 6.2.3).
Memory breaks when it has to be revised. Section 15.3 argued that persistence requires revising beliefs, not only retaining observations. The gap is that few learned systems implement a write policy deciding what enters the world state, with what confidence, and when an entry is revised or removed. Work on spatial memory has concentrated on recall, through longer contexts, better retrieval, and richer memory structures (Section 7.3.6), whereas the reported failures concern revision: outdated beliefs kept after the world changes [231], append-only memories that accumulate contradictions, and agents that drift as their context fills (Section 7.3.7). Applications make the requirement concrete: a manipulator must keep an object’s pose current while its own gripper occludes it (Section 9.1), and a home assistant must track object locations over weeks as the household moves things (Section 13.1). SLAM makes a version of these decisions when it selects keyframes, merges landmarks, and discards map regions that no longer match (Section 6.3.5); we are not aware of a learned counterpart for the richer world states that applications now require.
Pixel prediction is mismatched with the needs of control. Video world models predict pixels, while a controller needs the quantities that decide whether an action succeeds, such as contact, mass, and friction, which are often invisible in images (Section 7.1.2). A pixel loss spends most of a model’s capacity on lighting and texture, pixel predictions drift over long horizons (Section 8.3.6), and a model can score well on tasks with incorrect dynamics [3008]. The objective rewards appearance; the use requires outcomes. World-action models that predict compact encodings of future observations match generative ones at much lower cost [3016], following the predictive-embedding principle of the JEPA family (Section 8.6.1), and a further step is to predict physical quantities directly, estimated through pushing, lifting, or probing. Whether such predictions hold under intervention is the question of Section 15.4; the gap here is that these quantities are rarely prediction targets at all.
Correct perception does not carry through to completed actions. Chapter 7 described the gap between spatial judgment and action as instructions interpreted correctly and executed incorrectly (Section 7.3.2). On VABench, models that judge spatial relations well still fail many manipulation tasks and almost all long-horizon ones, and degrade markedly on unfamiliar shapes [3005]; generalist robot policies improve rapidly, yet their spatial representations drift over long action sequences (Section 7.3.2). Small errors compound over many steps, and in contact-rich tasks a single misjudged grasp can end an episode, so the gap is one of execution rather than judgment and grows with the horizon. An agent must keep observing and correcting until the task is complete, as the gains from letting a model look again suggest (Section 15.5); this is the third open tension of Chapter 3, the integration of spatial reasoning into long-horizon embodied agents. Closing it requires training and evaluating the act–observe–correct loop as a whole, with benchmarks that reward recovery as well as first-attempt success.
Room-scale results say little about larger spaces. Most results in this handbook concern single rooms or tabletops. At street and city scale, models revert to landmark recognition even over a single kilometer [3012], geospatial models trained on one region or sensor degrade on another (Section 11.2), aerial and ground robots must align views from very different heights (Section 9.4), and skills learned on one robot body transfer poorly to another (Section 9.1). Scale changes more than the size of the numbers: the dominant representation shifts from local metric geometry to routes and topology and then to georeferenced coordinates, the sensing from egocentric views to aerial imagery, most of the state from seen to remembered or inferred, and the rate of change from seconds to years.
Geographic information systems handle scale by nesting reference frames, placing an object in a room, the room in a building, and the building in a georeferenced map, each level with its own coordinates and precision. Learned spatial models rarely represent such nesting. We expect nested frames to be part of the solution, together with benchmarks that pose the same question at several scales (Section 16.2), and we regard cross-scale transfer as a challenge in its own right.
Each of these gaps appears where one part of the loop depends on another’s output, which is why evaluating each part on its own misses them.