Spatial competence is not exhausted by interpreting observations or predicting future states. In an unfamiliar environment an agent must decide where to look, what to investigate, and when to act under uncertainty, and models that perform well on provided evidence can perform poorly when responsible for gathering it. Theory of Space makes the comparison directly: foundation models asked to explore until they knew where objects were did worse when they collected the observations themselves than when the same observations were supplied, and explored less systematically than simple scripted strategies [231]. On VABench, models that judge spatial relations well still fail many manipulation tasks and almost all long-horizon ones, and the gap narrows when they may move the camera and look again [3005]. These results locate part of the weakness in deciding what to do, rather than only in recognizing what is seen.
Section 15.1 asked which properties can be made identifiable by acting; the question here is which action to choose. Information gathering should be guided not only by uncertainty but by the consequences of that uncertainty for the decision at hand. An observation that sharpens a reconstruction may be worth less than one that settles whether a passage is traversable or a grasp is safe, and the value of a view depends on the goals, costs, and risks of the actions that follow it. Exploration policies learned in simulation formalize this for navigation objectives (Section 7.2.5); the counterpart for the richer states of Sections 15.2 and 15.3, where the quantity to resolve may be an object’s identity, a person’s belief, or a door’s state, is largely untrained. Reporting accuracy as a function of an interaction budget, the number of views, moves, or queries a model may spend, would place passive and active evaluation on one scale and reward agents that reach correct decisions with few observations.
Passive data, simulation, and real-world interaction provide complementary forms of experience. Internet video, the largest source, supports perceptual and physical priors and already enables robot planning after modest fine-tuning (Section 8.6); simulation and generated worlds permit controlled exploration without risk (Section 7.2.5); real interaction alone exposes the consequences of the agent’s own decisions, which passive video does not contain and on which both the update policy of Section 15.3 and the predictor of Section 15.4 depend. Each source carries its own distortions. The simulation gap is widest where contact, materials, lighting, and sensor noise matter (Sections 9.1 and 11.3), and experience generated inside a world model is useful only within the model’s range of validity. Learning inside a world model, with rollbacks to earlier decision points and correction of the model from real transitions [3018], lowers the cost of real interaction without removing the need for it. How to combine the three sources without transferring incorrect assumptions from simulated or observational distributions into real-world action remains unresolved, and existing robot datasets, concentrated on tabletop success, offer little of the failures and recoveries such a combination requires (Section 7.3.2). The consequences of an action, moreover, are not confined to the agent’s own observations: in a shared space, acting also changes what other agents observe, believe, and do.
Different agents observe and understand the same environment differently, and a shared physical world does not imply shared knowledge of it. A system must relate its own reference frame to those of others, since a location on the user’s left may lie on the robot’s right and people switch between egocentric and allocentric descriptions without noticing (Section 7.3.1). It must distinguish the physical state from what a person believes about it, since a user who left the kitchen before the kettle was moved still believes it is on the stove. It must anticipate how its own actions change what others perceive and do. Mental World Modeling maintains joint physical and mental states for this reason and finds both necessary to predict human decisions (Section 8.6.3.2) [2352]. People, on this view, belong in the world state on the same footing as walls and doors, with positions, viewpoints, and beliefs treated as quantities to be estimated.
In human environments, spatial actions are inherently interactive. Robots that model people only as moving obstacles freeze in dense crowds once every path appears blocked, unable to anticipate that people will make room (Section 13.1), and an assistant’s motion, speech, and help change what the people around it do next, so a plan that ignores this is planning against a stale state. Persistent spatial memory carries a second consequence: a world state that includes people records where they live, what they own, and how they spend their days, and the design choices of what to store, at what detail, and for how long matter more than the access controls applied afterward (Sections 10.1 and 13.1). Such systems also act on their users’ own spatial abilities, which weaken under habitual reliance on navigation aids and strengthen with practice [3019]. Reliable spatial agency must therefore account for other agents’ responses and for the consequences of maintaining and using spatial knowledge about them.
Agents that acquire the information they need, act despite incomplete knowledge, and revise their world models from the consequences of interaction would address each of the preceding questions in operational form: action changes what is identifiable, exposes which information the representation lacked, supplies the observations that keep the state current, and tests the predictions the model made. Spatial intelligence, on this view, is a closed loop in which perception, representation, prediction, and action continually constrain and improve one another. A usable model of physical space is one that remains open to new evidence, explicit about its limits, and corrected by what its own actions reveal.