The same physical environment can be represented through images, point clouds, occupancy grids, scene graphs, neural fields, or the latent state of a learned model, and Part II surveyed each of these forms (Chapter 4). They differ less in how much information they hold than in which aspects of the environment they preserve, which they abstract away, and how they organize what remains. Scene graphs typically emphasize entities and their relations, Gaussian representations primarily encode appearance and geometry, and a latent state retains what its training objective rewarded; each can be extended with further structure, but none is informative in isolation from the questions and operations it is meant to support.
That spatial knowledge and spatial reasoning are separable is shown by large models that acquire the former without the latter. A transformer trained only on Othello move sequences forms an internal representation of the board (Section 4.4), and language models encode the geographic positions of cities in their activations without ever being shown a map [3006]. The same models remain well below human performance on perspective taking, mental rotation, and building a model of a place from a sequence of views [3007], and chain-of-thought prompting, which helps on arithmetic and code, tends to degrade their accuracy on visual spatial benchmarks [1501]. Performance recovers when the model is allowed to draw a coarse cognitive map, render an imagined view, or call a geometric tool before answering (Sections 7.3.5 and 7.3.8). Encoding spatial information and computing with it are therefore distinct requirements, and reasoning performance depends partly on whether the representation in use supports the operations the task demands. This does not show that language is unsuited to spatial reasoning; it suggests that a representation in which distances and viewpoints are described rather than held tends to accumulate error each time they are manipulated.
Spatial tasks differ not only in the information they require but in the computations they demand of it. Judging a relative position needs a reference frame and a relation; metric reasoning needs distances, angles, and a scale; changing viewpoint needs a rigid or projective transformation applied to 3D structure; navigation needs topological connectivity and traversability; manipulation needs local geometry together with contact constraints and material properties. A representation suited to semantic relations can be insufficient for metric transformation, and a detailed geometric reconstruction can still omit the properties on which physical interaction depends, a gap Section 11.1 summarized as geometry is not behavior. The effectiveness of a representation is therefore a matter of its compatibility with the downstream operations. The recurring finding of Part III that hybrid systems are limited by their weakest interface (Section 7.4) is an instance of this: the interface is where one representation’s operations are asked to run on another representation’s content.
These observations suggest a task-relative criterion of sufficiency. Spatial-state sufficiency depends on the queries, predictions, and actions it must support; a representation is sufficient for a family of tasks if every decision the family requires can be made from it as well as from the world itself. Information irrelevant to one task may be essential to another, so sufficiency cannot be judged from the representation alone. This creates a tension between compact states optimized for an immediate objective and general-purpose states intended to remain useful across tasks that are not yet known. A latent world model can succeed at the tasks it was trained for while modeling the environment’s dynamics unfaithfully [3008], which satisfies sufficiency for those tasks and gives no assurance for others; a dense geometric reconstruction, by contrast, serves many queries but remains silent about physical properties that are not visible.
Whether a finite shared state can preserve the distinctions required by an open-ended range of tasks, or whether a spatial system should instead maintain adaptable representations whose detail and structure depend on downstream demands, remains open. Shared spatial latents read by many task heads, task-specific representations maintained in parallel, and interfaces through which modules request information at the precision they need (Section 7.4) represent three answers, and the available evidence does not yet favor one, particularly when future tasks are unknown.
Evaluating a spatial representation therefore requires examining the information it preserves and the operations it supports, rather than its expressiveness in the abstract. Reconstruction quality and recognition accuracy are the usual proxies, and both are insufficient: a photometrically accurate replica can fail a metric tolerance check and encode nothing about which drawers open (Section 11.1), and a correct answer can rest on language priors rather than on a correct belief about the scene. Sufficiency has to be tested across tasks, by asking whether a representation fixed for one purpose supports queries, transformations, and action predictions it was not built for. Whether a state that is sufficient at one moment remains so as the world changes and objects leave the field of view is a separate question, taken up next.