2.2 Defining A Spatial Intelligence Framework · Spatial Intelligence Handbook

Given the inherently hybrid and interdisciplinary nature of spatial intelligence, it becomes necessary to revisit and refine what we mean by “spatial intelligence” in a precise and unified manner. Rather than treating spatial intelligence as a loosely defined collection of capabilities, we aim to articulate a coherent conceptual framework that can accommodate its diverse manifestations across systems, tasks, and environments. In this section, we therefore propose an explicit definition of spatial intelligence and analyze its core properties in a structured way.

This definition emphasizes several essential aspects. First, spatial intelligence is grounded in representations of the world rather than isolated sensory inputs. Second, it is functional rather than static, supporting perception, reasoning, prediction, and action. Third, it is inherently situated, meaning that spatial intelligence only manifests meaningfully within an environment where tasks are defined and executed. In the following, we unpack this definition by examining its key attributes.

2.2.1Core Elements#

2.2.1.1Parties#

At a minimum, spatial intelligence involves two fundamental parties: an AI system and the physical world or environment in which it operates. Neither can be meaningfully defined without the other.

AI System. The AI system refers to any computational entity capable of perception, reasoning, and decision-making. This includes, but is not limited to, large language models, multimodal foundation models, world models, embodied agents, and robotic systems. Crucially, spatial intelligence is not tied to a specific architecture. Instead, it characterizes a class of capabilities that may be realized through different model families, training paradigms, and system designs [230, 231].

Physical World or Environment. The environment denotes the external context in which the AI system is situated. This typically refers to the physical world governed by geometric, physical, and causal constraints. In some cases, the environment may also include social or human-centric settings, where spatial structure interacts with social norms, human intent, and collective behavior [232]. Spatial intelligence arises precisely from the interaction between the AI system and such structured environments.

2.2.1.2Spatial Representation#

A core component of spatial intelligence lies in the ability to form, maintain, and manipulate world-centered spatial representations. These representations serve as structured world models that are compositional, generalizable, and reusable across tasks.

Spatial representations may vary along several dimensions:

•

Granularity. Representations can operate at different spatial scales, including object-centric representations, scene-centric representations, and relational representations capturing relative geometry or topology.
•

Modality. Spatial information may be derived from heterogeneous sensory channels, including pixels, grid maps, point clouds, voxel grids, language descriptions, audio signals, spectral measurements, radar inputs, as well as tactile and force feedback.
•

Level of Abstraction. Spatial representations may be geometric, such as metric coordinates and surfaces, topological, such as connectivity graphs, or semantic and symbolic, such as object categories and relational predicates.
•

Concrete versus Abstract. Representations may directly encode observable properties of the physical world or serve as abstract internal models that summarize, compress, or reinterpret spatial structure for reasoning and planning.

Importantly, spatial intelligence does not require committing to a single representational form. Instead, it demands the ability to align and translate across multiple representational levels in a world-centered manner.

2.2.1.3Dynamic Interaction#

Spatial intelligence is not a static property acquired through passive observation. Rather, it emerges through continuous and dynamic interaction with the environment. This interaction includes navigation, manipulation, exploration, and embodied perception. An intelligent system must ground its spatial understanding in embodied experience, for example by learning the structure of a room through movement, or improving depth perception through active viewpoint selection. Interaction also extends to socially situated behaviors. Maintaining appropriate interpersonal distance, coordinating actions with humans, or navigating shared spaces all require spatial understanding informed by social context. As a result, spatial representations must be executable and continuously updated through perception-action feedback loops. We further decompose dynamic interaction into several functional components.

Perception. Perception refers to the extraction of spatial structure from multimodal sensory inputs. This includes recognizing objects and scene elements, estimating distances, identifying spatial layouts, and localizing entities within an environment.

Reasoning and Planning. Spatial reasoning involves inference over spatial relations and dynamics. This includes trajectory prediction, path planning, occlusion reasoning, spatial causality, temporal change modeling, relational inference, and logical or commonsense reasoning about space. Such reasoning spans both low-level problems, such as shortest-path computation, and high-level abstractions, such as understanding how rearranging objects alters the affordances of a space.

Predicting. Prediction in spatial intelligence operates along two axes. One concerns forecasting future spatial states or events in a manner consistent with physical laws and causal structure. The other concerns generative capabilities, such as producing spatially aware images, videos, 3D or 4D scenes, and motion trajectories.

Embodied Action. Embodied action denotes the capacity to interact with physical or simulated environments while continuously pursuing goals. This includes pose adjustment, navigation, grasping, and object manipulation. Typical scenarios include domestic robots and autonomous driving systems. Such action requires tight integration of perception, reasoning, planning, and prediction, while aligning internal spatial models with task objectives, human intent, and social context.

2.2.2Formal Characterization#

To make the definition more precise, we provide a lightweight formalization that captures the core components of spatial intelligence without committing to a specific implementation or learning paradigm.

Let $\mathcal{E}$ denote the environment in which an AI system is situated, representing the physical or simulated world governed by spatial, physical, and causal constraints. Let $\mathcal{S}$ denote the AI system or agent. We assume that tasks are defined with respect to the environment and are drawn from a task space $\mathcal{T}$ .

World-Centered Spatial Representation. We define a world-centered spatial representation as a system-induced mapping from the environment to an internal state space:

\Phi_{\mathcal{S}}:\mathcal{E}\rightarrow\mathcal{R}_{\mathcal{S}},

(2.1)

where $\mathcal{R}_{\mathcal{S}}$ denotes the internal spatial representation space maintained by the AI system $\mathcal{S}$ . The representation $\Phi_{\mathcal{S}}(e)\in\mathcal{R}_{\mathcal{S}}$ may take various forms, including geometric maps, scene graphs, implicit fields, symbolic layouts, or learned latent embeddings. Crucially, $\mathcal{R}_{\mathcal{S}}$ is not a raw sensory space, but a structured and system-dependent representation space that supports compositionality, generalization, and reuse across tasks.

Functional Use of Spatial Representations. Spatial intelligence is not defined by representation alone, but by how representations are functionally manipulated by the AI system. We characterize this through a family of system-conditioned operators acting on $\mathcal{R}_{\mathcal{S}}$ :

\mathcal{F}_{\mathcal{S}}=\{f^{\mathcal{S}}_{\text{perc}},f^{\mathcal{S}}_{% \text{reason}},f^{\mathcal{S}}_{\text{pred}},f^{\mathcal{S}}_{\text{act}}\},

(2.2)

corresponding to perception, reasoning, prediction, and embodied action. Each operator is defined relative to the system $\mathcal{S}$ :

$\displaystyle f^{\mathcal{S}}_{\text{perc}}$	$\displaystyle:\mathcal{O}\rightarrow\mathcal{R}_{\mathcal{S}},$	(2.3)
$\displaystyle f^{\mathcal{S}}_{\text{reason}}$	$\displaystyle:\mathcal{R}_{\mathcal{S}}\rightarrow\mathcal{R}_{\mathcal{S}},$	(2.4)
$\displaystyle f^{\mathcal{S}}_{\text{pred}}$	$\displaystyle:\mathcal{R}_{\mathcal{S}}\times\mathcal{A}\rightarrow\mathcal{R}% _{\mathcal{S}},$	(2.5)
$\displaystyle f^{\mathcal{S}}_{\text{act}}$	$\displaystyle:\mathcal{R}_{\mathcal{S}}\rightarrow\mathcal{A},$	(2.6)

where $\mathcal{O}$ denotes observations and $\mathcal{A}$ denotes the action space. The two representation-to-representation operators play distinct roles: $f^{\mathcal{S}}_{\text{reason}}$ elaborates the current world state by inferring relations and constraints already implicit in it, whereas $f^{\mathcal{S}}_{\text{pred}}$ advances the state forward in time, rolling out how the world would evolve under a hypothesized action (with a null action recovering passive forecasting).

Task-Oriented Intelligence. Given a task $t\in\mathcal{T}$ , spatial intelligence manifests as the ability of the AI system $\mathcal{S}$ to select and execute actions based on its internal spatial representation:

\pi_{\mathcal{S}}:(\mathcal{R}_{\mathcal{S}},t)\rightarrow\mathcal{A},

(2.7)

such that actions are grounded in the system’s current world representation and lead to task completion within the environment. This formulation highlights that spatial intelligence is inherently situated and agent-dependent: the same environment may induce different behaviors depending on the internal representation and decision mechanisms of $\mathcal{S}$ .

Dynamic Interaction and World State Evolution. Finally, spatial intelligence requires representations to be dynamically updated through interaction. We model the evolution of the internal world representation of system $\mathcal{S}$ as:

\mathcal{R}^{\mathcal{S}}_{t+1}=\Psi_{\mathcal{S}}(\mathcal{R}^{\mathcal{S}}_{% t},a^{\mathcal{S}}_{t},o_{t+1}),

(2.8)

where $a^{\mathcal{S}}_{t}\in\mathcal{A}$ is the action selected by system $\mathcal{S}$ and $o_{t+1}\in\mathcal{O}$ is the subsequent observation. It emphasizes that spatial representations are executable, interactive, and continuously revised by the system, rather than static encodings of the environment.

This formalization clarifies that spatial intelligence is not tied to a specific modality, algorithm, or architecture. Instead, it is defined by the existence of a system-induced, world-centered spatial representation space $\mathcal{R}_{\mathcal{S}}$ and the ability of the AI system $\mathcal{S}$ to manipulate, update, and exploit this representation to support perception, reasoning, prediction, and embodied action within an environment.

2.2.3Compatibility with Existing Research and Paradigms#

The proposed framework is intentionally designed to be inclusive rather than prescriptive. Rather than introducing a new task taxonomy or replacing existing formulations, it provides a unifying lens through which a wide range of spatial intelligence-related tasks, domains, and research traditions can be consistently described. From this perspective, most existing work can be understood as instantiating or emphasizing only a subset of the core elements defined in our framework, often driven by domain-specific constraints, data availability, or evaluation protocols.

A large body of research in computer vision, graphics, and extended reality focuses primarily on the representation layer of spatial intelligence. Tasks such as 3D reconstruction, neural rendering, scene synthesis, video generation, and emerging 4D spatiotemporal modeling concentrate on learning expressive spatial representations $\mathcal{R}_{\mathcal{S}}$ , often with little or no emphasis on action, task execution, or long-term interaction. Within our framework, these approaches can be viewed as developing increasingly powerful instantiations of world-centered spatial representations, while largely abstracting away the agent $\mathcal{S}$ and the functional operators that connect representation to behavior. Their contributions lie in advancing the expressiveness, continuity, and generative capacity of spatial representations, even if they remain largely offline or perception-centric.

Robotics, embodied AI, and embedded systems emphasize a different subset of the framework, placing primary focus on action grounding and closed-loop interaction. Vision-Language-Action models, navigation agents, and manipulation systems operationalize the mapping from internal representations to actions, often under real-time and physical constraints. However, in many cases, the underlying spatial representations remain task-specific, shallow, or tightly coupled to particular embodiments. From the perspective of our framework, these systems instantiate the functional operators and dynamic interaction components, while relying on relatively limited or implicit world representations. This explains why many embodied systems exhibit strong short-horizon competence but struggle with long-term consistency, world editing, or cross-task transfer.

Classical and modern SLAM systems provide a complementary example. SLAM explicitly targets the construction and maintenance of a world-centered spatial representation under uncertainty, typically in the form of metric maps, pose graphs, or hybrid geometric-topological structures. Within our framework, SLAM can be understood as a specialized instantiation of the representation update process, with strong guarantees on geometric consistency and state estimation. At the same time, SLAM systems often remain agnostic to high-level reasoning, semantic abstraction, or task-conditioned decision making, highlighting again that they occupy a specific slice of the broader spatial intelligence spectrum rather than the full stack.

Viewed through this unified lens, existing spatial intelligence research does not fragment into incompatible paradigms, but instead forms a landscape of partial realizations. Different communities emphasize representation learning, perception, reasoning, prediction, or action to varying degrees. Our framework provides a common conceptual structure that makes these emphases explicit, clarifies their limitations, and reveals how advances in one area can be systematically integrated with others. In this sense, the framework is not a replacement for existing approaches, but a scaffold for understanding how they collectively contribute to, and fall short of, world-centered spatial intelligence.