Unlike the visual and geometric representations of the previous sections, which encode space explicitly through pixels, points, or surfaces, language represents space symbolically, naming entities and stating the relations among them (e.g., , “the chair is next to the table” or “the store is across the street from the park”) and thereby capturing relational, topological, and functional structure that metric representations leave implicit. The question this section pursues is not how language describes space, which is self-evident, but whether models trained only on language come to encode it. Answering that requires first understanding how language representations are learned at all, so the section moves from the evolution of language representation learning, from masked-language pretraining to large autoregressive models, to the more surprising finding that such sequence-only training induces latent, spatially structured world models rather than mere text statistics. The scope here is language in isolation; how language is aligned with visual, 3D, and audio modalities is deferred to Chapter 5.
4.4.1The Evolution of Language Representation Learning#
Language representation learning has evolved rapidly over the past decade, from masked-language pretraining (starting with BERT [42] and continued by RoBERTa [302], XLNet [303], and T5 [304]) to autoregressive large language models trained purely on next-token prediction, exemplified by GPT-2 and GPT-3 [305, 43]. Across this progression, a recurring observation is that representation quality scales with model size, data size, and the sophistication of the pretraining objective [306], and that pretrained representations encode richer structure than the surface text statistics they were trained on [307, 308]. This last property, that language-only training produces representations carrying structure beyond text, is what makes language representations load-bearing for spatial intelligence, as the next subsection makes explicit.
4.4.2Language Representation Learning Induces Spatial Intelligence#
Although LLMs are optimized for sequence prediction, recent work suggests that they can develop structured internal representations that resemble world models [309, 310, 311]. One particularly pertinent demonstration of this on spatial intelligence is Othello-GPT, a transformer trained solely on sequences of legal moves from the board game Othello [312].
Despite receiving no prior knowledge of the board layout or game rules, Othello-GPT achieves near-perfect accuracy in predicting legal moves (actions in a physical space). Probing experiments reveal that nonlinear classifiers can reliably reconstruct the entire board state from hidden activations [313, 307, 314]. Crucially, interventional studies show that manipulating internal representations of specific tiles alters the model’s predictions in a way consistent with counterfactual board states, providing causal evidence of spatial encoding [315, 316].
These findings align with broader research indicating that LMs acquire implicit representations of abstract or grounded concepts. For example, models have been shown to encode syntactic structure [308], color spaces [311], and even logical state-tracking [309]. Work on chess transcripts similarly shows that LMs can track board positions without explicit supervision [317]. Compared to systems like AlphaZero, which are built with strong inductive biases and explicit access to board states [318, 319], Othello-GPT highlights the surprising extent to which sequence-only training can lead to emergent spatial reasoning.
In sum, it has been demonstrated that spatially structured representations can emerge in LMs trained only on language-only sequences. This suggests that autoregressive sequence prediction is sufficient, in certain domains, to induce latent world models that encode spatial relations and can causally drive action predictions.