The preface asked how the field can measure genuine progress without mistaking benchmark saturation for it. Chapter 15 described the properties a usable spatial model should have; this section asks how to measure them. Most benchmarks compared in Section 7.5 pose questions about a single image or video, and few require the model to act. This format admits shortcuts, which models have repeatedly exploited: models that nearly solved synthetic visual reasoning datasets failed on novel combinations of familiar elements and on real images (Section 7.2.1), and much of a referring-expression score survives when the words of the expression are shuffled (Section 6.5.2).
Measured against the loop of Chapter 2, current evaluation covers some components far better than others. Perception is evaluated frame by frame, through detection, segmentation, depth, and tracking accuracy, and rarely by whether its uncertainty is calibrated or its output stable over time. Reasoning is evaluated by answer accuracy on static questions, rarely by whether the answer rests on a correct belief about the scene. Prediction is evaluated by visual quality, rarely by correctness under actions absent from training. Action is evaluated by task success in simulation, rarely by recovery after failure. The update is rarely evaluated on its own, and neither is the effect a system has on the people who use it.
Score the belief behind the answer. A correct answer may come from language priors, regularities in the answer format, or a correct model of the scene, and accuracy alone cannot tell them apart, nor separate the explanations left open for the first gap of Section 16.1. Benchmarks can instead read out a model’s belief about the scene and score it directly, as Theory of Space does at each step of an exploration [231], or probe hidden states, as was done for the Othello model of Section 4.4. A cheaper variant asks several questions about the same scene and checks that the answers are consistent with a single layout. Controls that remove the image or shuffle the input are inexpensive and should be standard.
Gathering evidence is a separate ability. Static benchmarks supply the evidence; deployed systems must collect it. Deciding where to look, when to move, and when to ask is a skill in its own right, and the active–passive gap of Section 15.5 shows that current models lack it. We propose reporting accuracy as a function of an interaction budget, the number of views, moves, or queries a model may use, which places passive and active evaluation on one scale and rewards models that reach correct answers with little effort. In deployment the budget has a direct cost in time and energy.
Spatial failures unfold over time. Most benchmark questions refer to a single moment, while many failures in Section 16.1 emerge only over time. Streaming protocols show the model only the video seen so far, so it cannot use future frames to answer about the past [3015]; revisit protocols move the camera away and back to check whether a place is rendered consistently or changed correctly [3010]; change protocols alter the scene behind the agent and test whether its belief follows. None is yet standard, and long-horizon tasks remain rare even among embodied benchmarks (Section 7.5).
The same question should be asked at several scales. Benchmarks cluster at room scale and are often built on a few indoor scan collections, while surgery, orbital operations, and underwater robotics have benchmarks of their own that rarely share protocols with them (Sections 12.1 and 11.3 and Chapter 14). Posing the same spatial question at the scale of a table, a building, and a city would show where competence stops transferring and whether a model represents nested frames; a model that answers well about a desk and poorly about a building has learned the regularities of tabletop scenes, not a general spatial ability.
Generated worlds should be judged by what an agent can do in them. Generated and reconstructed worlds are still evaluated mainly by appearance, which Chapter 8 showed to be misleading. Video metrics do not register physically impossible motion (Section 8.3.6), rendered-view similarity cannot distinguish a solid object from a cut-out that matches the training views (Section 8.2.1), no standard protocol exists for the physical correctness or long-term stability of world models (Section 8.6.4), and vision-language judges share the blind spots of the models they evaluate (Section 8.8). A more informative test asks whether an agent that plans or trains in the generated world succeeds in the real one, complemented by the interventional tests of Section 15.4.
Benchmarks should borrow construct validity from psychometrics. Spatial benchmarks adopted tasks from psychometrics, from mental rotation to perspective taking (Chapter 2), but not its standards. A psychometric test reports the distribution of human performance, establishes reliability across repeated testing, and checks construct validity through agreement with other tests of the same ability and divergence from tests of different ones; most AI benchmarks report at most a single human score. A benchmark whose questions can be answered from text alone does not measure spatial ability, and automatically generated benchmarks, now common, need such checks all the more.