Spatial Intelligence
PDF
Handbook

Spatial Intelligence

Foundations, Architectures, Methodologies, Applications and Challenges

How AI systems construct, maintain, and manipulate representations of the physical world, and use them to perceive, reason, predict, and act. One framework, five parts and 17 chapters, from the cognitive map to world models.

Read online Download PDF Awesome list★ Star
Parts
5
Chapters
17
Figures
163
References
3,023
The handbook drawn as one concentric world: representation at the centre, the perception–reasoning–generation loop on the inner ring, applications along the coast, the five eras as stepping stones and the open questions on the horizon.

What is spatial intelligence?

The ability of an AI system to construct, maintain and manipulate spatial representations of the physical world that support perception, reasoning, prediction and embodied action.

The full definition, Section 2.2 →
Environmentℰgeometric · physical ·causal constraintsobjects & scenesrooms & buildingscities & terrainAI system𝒮Perception𝑓perc: 𝒪 → ℛ𝒮extracts structurepersonshelftablerobotAction𝑓act: ℛ𝒮 → 𝒜acts on the worldWorld-centered spatialrepresentationℛ𝒮geometrictopologicalshelfdoortablepersoncupsymbolicon(cup, table)in(table, room)near(cup, sofa)abstractobject · scene · relationℛ𝒮𝑡+1 = Ψ𝒮(ℛ𝒮𝑡, 𝑎𝑡, 𝑜𝑡+1)updated through interactionReasoning𝑓reason: ℛ𝒮 → ℛ𝒮elaborates the statesofashelfcup reachablePrediction𝑓pred: ℛ𝒮 × 𝒜 → ℛ𝒮advances the state𝑡𝑡+1𝑡+2observationstaskput the cupon the shelfactions

Five eras of representing space

Each era is defined by what it took space to be, and each left a problem for the next.

Five parts, one path from representation to action

IPart I

Introduction and Background

  1. 1Spatial Intelligence: A Rising AI Paradigm
  2. 2A Definition in Motion: Revisiting Spatial Intelligence
  3. 3Scope and Structure of This Handbook
IIPart II

Spatial-aware Representation: Building Blocks of Space

  1. 4Single-Modal Spatial Representation
  2. 5Cross-Modal Spatial Representation
IIIPart III

Spatial Intelligence: From Perception to Generation

  1. 6Spatial Perception
  2. 7Spatial Reasoning
  3. 8Spatial Reconstruction and Generation
IVPart IV

Applications

  1. 9Spatial Intelligence for Embodied Systems
  2. 10Spatial Intelligence for Virtual Realities
  3. 11Spatial Intelligence for Geo-spatial Systems
  4. 12Spatial Intelligence for Biomedicine and Natural Sciences
  5. 13Spatial Intelligence for Intelligent Assistance
  6. 14Spatial Intelligence for Others
VPart V

Discussions and Futures

  1. 15Foundations: What Spatial Intelligence Requires
  2. 16Research Gaps and Future Directions
  3. 17Conclusion

Explore the field

Everything the handbook cites, compares and builds on, organised for searching and browsing.

3,023LiteratureEvery work the handbook cites, with the sections that cite it. 576Datasets & benchmarksCollected from the comparison tables across the chapters. 124Tools & platformsSimulators, libraries and open-source code, linked to where the book uses them. 104MilestonesRepresentative works by field and year, across the five eras. 163FiguresAll figures of the handbook in one gallery, each linked to its context. A–ZGlossary & notationTerms, abbreviations and symbols, with the sections that define them.

Developed by 32 researchers from 15 institutions

  • University of Oxford
  • National University of Singapore
  • Nanyang Technological University
  • University of Rochester
  • The Chinese University of Hong Kong
  • University of Macau
  • Stanford University
  • Imperial College London
  • Monash University
  • University of Cambridge
  • University of California, Los Angeles
  • Harvard University
  • Microsoft Research
  • Columbia University
  • University of California, Merced
Meet the team →

Citing the handbook

If the handbook is useful in your work, please cite it. Every section can also be cited on its own with Cite this section in the reader; numbers online follow the PDF, so “Section 7.2” or “Figure 6.6” mean the same in both.

@book{wu2026spatial,
  title     = {Spatial Intelligence: Foundations, Architectures, Methodologies,
               Applications and Challenges},
  author    = {Wu, Shengqiong and Zhao, Yu and Ju, Tianjie and Xu, Jundong and
               Li, Bobo and Yang, Jingkang and Tian, Shulin and Zhang, Daoan and
               Zhao, Yanguang and Ju, Xuan and Zhao, Haoyu and Zhou, Yuchen and
               Li, Xiangtai and Zheng, Zhedong and Hsu, Joy and Kong, Xin and
               Li, Ruining and Zhang, Cheng and Li, Chengzu and Duan, Jiafei and
               Zhou, Yu and Li, Kenneth and Zheng, Chuanxia and Cui, Lei and
               Wei, Furu and Liu, Ziwei and Feiner, Steven K. and Luo, Jiebo and
               Cai, Jianfei and Yan, Shuicheng and Yang, Ming-Hsuan and
               Fei, Hao},
  year      = {2026},
  url       = {https://spatial-agi.github.io}
}