16.4Future Directions

Persistence and rendering can be separated. Interactive video models remember a place by regenerating it, which ties memory to the context window and to the renderer. A factorized design would keep what must persist, such as layout, object identities, and their uncertainty, in an explicit layer, and use a learned layer to predict change and render what the agent needs to see, with $\Psi_{\mathcal{S}}$ promoting confirmed predictions into the explicit layer and revising it when observations contradict it. The alternatives are a latent state with learned memory, as in models that factorize memory inside the generator [3013], and a growing context with retrieval. The hypothesis is that at equal compute the factorized design stays consistent over longer horizons and more changes; it is refuted if a latent-only model matches it on revisit and change protocols while passing the faithfulness tests of Section 15.4. The decisive experiment is an agent that leaves a building, returns hours later, detects what changed, and acts accordingly; for the factorized variant, the open question is the interface that decides when a prediction enters the explicit layer and when an entry is overridden.

The medium of spatial reasoning can be learned per query. Section 15.2 found that spatial answers improve when a model can work with maps, imagined views, or geometric tools. The choice of medium can be learned per query from whether it led to a correct answer and at what cost, so that simple questions are answered in language and a map or simulation is invoked only when needed, with the intermediate steps later distilled into the model (Section 7.3.8). The comparison is with the same model reasoning in text alone and with a policy that always invokes the tool, on map building and perspective taking, with image-removed controls; the hypothesis fails if the learned selector does not beat the always-tool policy on cost at equal accuracy.

The update policy can be trained from experience. Passive data leave the update policy of Section 15.3 largely untrained (Section 15.5). It can be trained directly, by rewarding observations that add information to the map and penalizing beliefs kept after the evidence has changed, with learning inside a world model lowering the cost by trying alternatives in imagination first (Section 7.3.7); benchmarks that read out a model’s belief (Section 16.2) supply the training signal as well as the evaluation. The target is an agent that builds as accurate a world state from its own exploration as from a curated set of views, and the scripted strategies that currently outperform foundation models [231] are the baseline to beat.

Verification is feasible where part of the state is explicit. Neural networks are difficult to verify, whereas objects, poses, and programs can be checked against constraints. The Algebraic World Model turns exploration into hypotheses checked by machine-verified proofs [3022], and uncertainty can be verified once its calibration is matched to the policy actually executed [3017]. The alternative is to verify latent states indirectly, through probes and faithfulness tests [3008], which is cheaper and yields statistical rather than logical guarantees. In either case the obstacle is perception, since any guarantee about the state inherits the errors of its estimate from sensor data. A machine-checked property, such as a bound on collision probability, holding for a learned agent over a deployment-scale domain would change the situation of Section 16.3; no current system provides one.

A shared world state needs the machinery of version control. When several agents and people maintain one world state, they need a common reference frame, stable object identities, a record of changes, a procedure for resolving conflicts, and permissions on who may read and write what. Version control solved the analogous problems for shared code, and multi-agent systems that build cognitive maps already record changes as atomic commits and cross-check one another [3023]. Embodied teams add difficulty, since ground and aerial robots must first align very different viewpoints (Section 9.4) and city-scale and planetary digital twins supply the georeferenced frames into which local maps must fit (Sections 11.1 and 11.2); permissions are also where the privacy requirements of Section 15.5 can be enforced. A suitable test is a team of heterogeneous robots and a person who maintain one consistent world state of a building over several days, each able to query how the others currently perceive it.

A theory of spatial representation has a concrete agenda. Section 15.2 proposed sufficiency relative to tasks and actions as the criterion for a representation. Turning it into a theory requires identifiability results stating which world-state variables are recoverable from a given set of modalities and actions, extending the sensor-specific analyses of Chapter 6; sufficiency results stating which variables must be preserved for a task family under every action the agent may take, of which equivariance to rigid motion (Section 7.2.3) is the one case already worked out; faithfulness tests deciding, for any architecture, whether a latent state’s dynamics follow the environment’s [3008]; and a specification of which parts of a representation must be explicit for inspection and verification. Such results would let the world-centered property required by Chapter 2 be tested for any system, regardless of architecture.