Spatial Intelligence
PDF

Contents

Five parts, 17 chapters and 64 sections, numbered as in the PDF. The bar below is the book at scale: each block is a chapter, sized by its pages.

1234567891011121314151617
Part I · 32 pagesPart II · 24 pagesPart III · 249 pagesPart IV · 123 pagesPart V · 27 pages
PrefaceNotationGlossaryHow to read this handbook (Section 3.3)
IPart IIntroduction and Background
1Spatial Intelligence: A Rising AI Paradigm16 pp. · p. 24
  1. 1.1Why Spatial Intelligence? Why Now?p. 24
    1.1.1 AI Paradigm Shift1.1.2 AI Methodological Shift1.1.3 Modality Shift1.1.4 Disciplinary Convergence
  2. 1.2Historical Evolution and Milestonesp. 28
    1.2.1 Era 1: Symbolic and Cognitive Spatial Computing (1950s–1980s)1.2.2 Era 2: Geometric and Probabilistic Spatial Modeling (1980s–2012)1.2.3 Era 3: Learning-based Spatial Modeling (2012–2019)1.2.4 Era 4: Large Foundation Spatial Modeling (2019–2023)1.2.5 Era 5: World-Centered Spatial Intelligence (2023–Present)
2A Definition in Motion: Revisiting Spatial Intelligence11 pp. · p. 40
  1. 2.1The Landscape of Spatial Intelligencep. 40
    2.1.1 Beyond A Single Dimension2.1.2 Disciplinary Perspectives2.1.3 Toward A Unified View
  2. 2.2Defining A Spatial Intelligence Frameworkp. 46
    2.2.1 Core Elements2.2.2 Formal Characterization2.2.3 Compatibility with Existing Research and Paradigms
3Scope and Structure of This Handbook5 pp. · p. 51
  1. 3.1The Research Pillars of Spatial Intelligencep. 51
  2. 3.2Goal of the Handbook: Charting a Research Ecosystemp. 52
  3. 3.3How to Read This Handbookp. 53
IIPart IISpatial-aware Representation: Building Blocks of Space
4Single-Modal Spatial Representation17 pp. · p. 56
  1. 4.12Dp. 56
    4.1.1 Image and Video Representations4.1.2 Segmentation Maps4.1.3 Depth Maps4.1.4 Surface Normal Maps4.1.5 Edge Maps4.1.6 Optical Flow Maps4.1.7 2D Keypoints
  2. 4.23Dp. 60
    4.2.1 RGB-D Images4.2.2 Voxel Grids4.2.3 Point Clouds4.2.4 Meshes4.2.5 Implicit Representations4.2.6 Neural Radiance Fields4.2.7 3D Gaussian Splatting4.2.8 Hybrid Methods
  3. 4.34Dp. 65
    4.3.1 RGBD Video4.3.2 Point Cloud4.3.3 Mesh4.3.4 NeRF4.3.5 Deformable 3D Gaussian Splatting
  4. 4.4Languagep. 68
    4.4.1 The Evolution of Language Representation Learning4.4.2 Language Representation Learning Induces Spatial Intelligence
  5. 4.5Audiop. 69
  6. 4.6Domain-Specific Representationsp. 70
5Cross-Modal Spatial Representation7 pp. · p. 73
  1. 5.1Overviewp. 73
  2. 5.22D-textp. 74
  3. 5.33D-textp. 75
  4. 5.4Audio-textp. 76
  5. 5.5Vision-visionp. 77
IIIPart IIISpatial Intelligence: From Perception to Generation
6Spatial Perception80 pp. · p. 80
  1. 6.1Spatial Entity Recognitionp. 80
    6.1.1 Object Detection6.1.2 Segmentation6.1.3 Keypoint and Landmark Detection6.1.4 Scene Classification
  2. 6.2Spatial Localization and Position Estimationp. 96
    6.2.1 Depth Estimation6.2.2 Surface Normal Estimation6.2.3 Height and Distance Estimation6.2.4 Visual Odometry and Camera Pose Estimation6.2.5 Pose Estimation
  3. 6.3Spatial Structure Constructionp. 119
    6.3.1 Spatial Relation Extraction6.3.2 Scene Graph Generation6.3.3 Layout Estimation and Room Parsing6.3.4 Action Detection6.3.5 Mapping and SLAM
  4. 6.4Spatio-Temporal Perceptionp. 137
    6.4.1 Scene Flow Estimation6.4.2 Object Tracking6.4.3 Temporal Localization
  5. 6.5Multimodal Spatial Perceptionp. 150
    6.5.1 Audio-Visual Spatial Localization6.5.2 Vision-Language Grounding
7Spatial Reasoning69 pp. · p. 160
  1. 7.1Rule-based Symbolic Spatial Reasoningp. 160
    7.1.1 Geometric and Algebraic Reasoning7.1.2 Physical Reasoning7.1.3 Topological and Graph-based Reasoning7.1.4 Constraint-based Spatial Solving7.1.5 Rule-based and Programmatic Reasoning7.1.6 Causal Reasoning
  2. 7.2Learning-based Spatial Reasoningp. 176
    7.2.1 Implicit Neural Spatial Reasoning7.2.2 Explicit Structured Neural Reasoning7.2.3 Geometry-aware Neural Spatial Reasoning7.2.4 Differentiable Optimization Layers for Spatial Reasoning7.2.5 Reinforcement Learning with Spatial Inductive Biases7.2.6 Multi-agent Spatial Reasoning
  3. 7.3Foundation Model-driven Spatial Reasoningp. 199
    7.3.1 LLM/VLM-centric Spatial Reasoning7.3.2 VLA-based Spatial Reasoning7.3.3 Chain-of-Thought for Spatial Reasoning7.3.4 Code/DSL Generation for Spatial Reasoning7.3.5 Tool-augmented Reasoning7.3.6 Retrieval- & Memory-augmented Spatial Reasoning7.3.7 Agentic Spatial Reasoning7.3.8 World-Model-based Spatial Reasoning
  4. 7.4Hybrid Spatial Reasoningp. 222
    7.4.1 Task Definition7.4.2 Methodology7.4.3 Benchmarks7.4.4 Challenges and Trends
  5. 7.5Spatial Reasoning Benchmarksp. 226
    7.5.1 Comparative Analysis7.5.2 Future Directions
8Spatial Reconstruction and Generation100 pp. · p. 229
  1. 8.12D Spatial Generationp. 229
    8.1.1 Text-to-image Generation8.1.2 2D Spatial-aware Image Generation8.1.3 2D Spatial Inpainting & Editing
  2. 8.23D Spatial Reconstruction and Generationp. 244
    8.2.1 3D Object Synthesis8.2.2 3D Scene Synthesis8.2.3 3D Scene Reconstruction8.2.4 3D Panorama Synthesis8.2.5 3D Scene Editing
  3. 8.3Video Spatial Generationp. 273
    8.3.1 Text-to-Video Generation8.3.2 Spatial Controlled Video Generation8.3.3 Temporal Controlled Video Generation8.3.4 Other Controlled Video Generation8.3.5 Video Editing8.3.6 Summary and Critical Challenges and Future Directions
  4. 8.4Motion Generationp. 291
    8.4.1 Human Motion Generation8.4.2 Animal Motion Generation8.4.3 Vehicle Motion Planning8.4.4 Motion Editing
  5. 8.54D Spatial Generationp. 303
    8.5.1 Text-to-4D Generation8.5.2 Image-to-4D Generation8.5.3 Video-to-4D Generation8.5.4 3D-to-4D Generation8.5.5 Multi-conditional 4D Generation8.5.6 Summary, Overarching Challenges and Future Directions
  6. 8.6World Modelingp. 312
    8.6.1 Representation World Model8.6.2 Generative World Model8.6.3 Mental World Modeling8.6.4 Discussions and Challenges
  7. 8.7Programmatic and Symbolic Spatial Generationp. 322
    8.7.1 Task Definition8.7.2 Methodology8.7.3 Benchmarks8.7.4 Challenges and Trends
  8. 8.8Summary and Outlookp. 325
IVPart IVApplications
9Spatial Intelligence for Embodied Systems49 pp. · p. 329
  1. 9.1Robotic Manipulationp. 329
    9.1.1 Introduction9.1.2 System Overview9.1.3 Pivotal Task-1: Grounding Action in Spatial Anchors9.1.4 Pivotal Task-2: Inferring Functional Action Possibilities9.1.5 Pivotal Task-3: Encoding Spatial Reasoning in Generalist Models9.1.6 Pivotal Task-4: Satisfying Geometric and Logical Constraints9.1.7 Established Tools and Systems9.1.8 Challenges
  2. 9.2Autonomous Drivingp. 339
    9.2.1 Introduction9.2.2 System Overview9.2.3 Pivotal Task-1: Perception and Detection9.2.4 Pivotal Task-2: Localization and Mapping9.2.5 Pivotal Task-3: Decision Making and Control9.2.6 Established Tools and Systems9.2.7 Challenges
  3. 9.3Wearable AI Systemp. 348
    9.3.1 Introduction9.3.2 Wearable Devices9.3.3 System Overview9.3.4 Pivotal Task-1: Embodied Motion Sensing and Pose Reconstruction9.3.5 Pivotal Task-2: Behavioral Understanding and Intention Modeling9.3.6 Pivotal Task-3: Safe Collaboration and Embodied Interaction9.3.7 Established Tools and Systems9.3.8 Challenges
  4. 9.4Aerial Agentsp. 358
    9.4.1 Introduction9.4.2 System Overview9.4.3 Pivotal Task-1: Ground-Air Collaborative Perception and Visual Geolocalization9.4.4 Pivotal Task-2: Remote Sensing Detection and Environmental Understanding9.4.5 Pivotal Task-3: Autonomous Aerial Navigation and Decision Making9.4.6 Established Tools and Systems9.4.7 Challenges
  5. 9.5Warehouse and Industrial Logisticsp. 368
    9.5.1 Introduction9.5.2 System Overview9.5.3 Pivotal Task-1: Perception and Recognition9.5.4 Pivotal Task-2: Localization and Mapping9.5.5 Pivotal Task-3: Reasoning and Planning9.5.6 Pivotal Task-4: Manipulation and Control9.5.7 Pivotal Task-5: Human-Robot Collaboration and Safety9.5.8 Established Tools and Systems9.5.9 Challenges
10Spatial Intelligence for Virtual Realities14 pp. · p. 378
  1. 10.1AR/VR/XRp. 378
    10.1.1 Introduction10.1.2 System Overview10.1.3 Pivotal Task-1: Spatial Tracking, Mapping, and Registration10.1.4 Pivotal Task-2: Egocentric Human Sensing and Spatial Interaction10.1.5 Pivotal Task-3: Spatial Display and Real-Time Rendering10.1.6 Established Tools and Systems10.1.7 Challenges
  2. 10.2Entertainmentp. 386
    10.2.1 Introduction10.2.2 System Overview10.2.3 Pivotal Task-1: World Reconstruction and Spatial Mapping10.2.4 Pivotal Task-2: Embodied Performance Capture and Avatar Control10.2.5 Pivotal Task-3: Spatially-Aware Content Authoring and Virtual Production10.2.6 Established Tools and Systems10.2.7 Challenges
11Spatial Intelligence for Geo-spatial Systems25 pp. · p. 392
  1. 11.1Digital Twinsp. 392
    11.1.1 Introduction11.1.2 System Overview11.1.3 Pivotal Task-1: High-Fidelity Twin Construction11.1.4 Pivotal Task-2: Physics-Based Modeling and Behavioral Simulation11.1.5 Pivotal Task-3: Real-Time Synchronization and Predictive Operation11.1.6 Established Tools and Systems11.1.7 Challenges
  2. 11.2GeoAIp. 400
    11.2.1 Introduction11.2.2 System Overview11.2.3 Pivotal Task-1: Precision Agriculture11.2.4 Pivotal Task-2: Urban Planning, Logistics, and Transport11.2.5 Pivotal Task-3: Environmental Monitoring11.2.6 Pivotal Task-4: Water Resources Management and Precipitation Forecasting11.2.7 Pivotal Task-5: Disaster Management11.2.8 Pivotal Task-6: Healthcare11.2.9 Established Tools and Systems11.2.10 Toward Geospatial Foundation Models and GeoLLMs11.2.11 Challenges
  3. 11.3Aerospace Engineeringp. 410
    11.3.1 Introduction11.3.2 System Overview11.3.3 Pivotal Task-1: Satellite Pose Estimation11.3.4 Pivotal Task-2: Spacecraft Detection, Segmentation, and Parts Recognition11.3.5 Pivotal Task-3: Operator Agents11.3.6 Pivotal Task-4: Planetary Robotics11.3.7 Established Tools and Systems11.3.8 Challenges
12Spatial Intelligence for Biomedicine and Natural Sciences17 pp. · p. 417
  1. 12.1Medicine and Clinical Healthcarep. 417
    12.1.1 Introduction12.1.2 System Overview12.1.3 Pivotal Task-1: Volumetric Medical Image Perception12.1.4 Pivotal Task-2: Anatomical Reconstruction, Modeling, and Surgical Planning12.1.5 Pivotal Task-3: Image-Guided Intervention and Surgical Navigation12.1.6 Established Tools and Systems12.1.7 Challenges
  2. 12.2Natural Sciencep. 425
    12.2.1 Introduction12.2.2 Physics12.2.3 Chemistry12.2.4 Life Science12.2.5 Challenges
13Spatial Intelligence for Intelligent Assistance9 pp. · p. 434
  1. 13.1Assistantp. 434
    13.1.1 Introduction13.1.2 System Overview13.1.3 Pivotal Task-1: Managing the Physical Environment13.1.4 Pivotal Task-2: Co-habiting and Collaborating13.1.5 Pivotal Task-3: Empowering Vulnerable Users13.1.6 Established Tools and Systems13.1.7 Challenges
14Spatial Intelligence for Others9 pp. · p. 443
  1. 14.1Architecture, Engineering, and Constructionp. 443
  2. 14.2Sports Analytics and Performancep. 444
  3. 14.3Cultural Heritage and Archaeologyp. 445
  4. 14.4Marine and Underwater Systemsp. 447
  5. 14.5Retail and Smart Spacesp. 448
  6. 14.6Outlookp. 449
VPart VDiscussions and Futures
15Foundations: What Spatial Intelligence Requires7 pp. · p. 452
  1. 15.1Spatial Observability and Identifiabilityp. 452
  2. 15.2Representation and State Sufficiencyp. 453
  3. 15.3Persistent World States and Belief Updatingp. 454
  4. 15.4Predictive Validity and Physical Groundingp. 456
  5. 15.5Situated and Interactive Spatial Intelligencep. 457
16Research Gaps and Future Directions6 pp. · p. 459
  1. 16.1Research Gaps along the Loopp. 459
  2. 16.2Benchmarking Gapsp. 461
  3. 16.3Deployment Bottlenecksp. 462
  4. 16.4Future Directionsp. 463
17Conclusion14 pp. · p. 465
AcknowledgementsReferences (3,023)Author contributions