1.1Why Spatial Intelligence? Why Now?

AI is undergoing a foundational paradigm shift, i.e., , from task-specific models toward world-centric general intelligence. Traditional AI systems were designed to solve narrowly defined problems, such as image classification, text generation, or speech recognition, each trained and optimized within a well-constrained input-output space. However, the emergence of foundation models and large language models (LLMs) has reshaped expectations: AI is now envisioned to generalize across domains, modalities, and tasks in open-world settings. This ambition hinges on a deeper capability, the ability to understand and model the world itself, often described as building a “world model” [4].

Spatial Intelligence emerges as a central pillar enabling this capability. It encompasses the perception and representation of objects, scenes, spatial layouts, motion, and physical interactions, and it serves as a carrier of world-generalizable knowledge that supports reasoning and transfer across tasks and modalities. Fei-Fei Li has made this stake explicit: “Our dreams of truly intelligent machines will not be complete without spatial intelligence” [5]. On her account, any AI system that aims to serve in physical environments must possess a rich understanding of spatial structure and dynamics.

Moreover, this transformation also signals a shift in the boundary of context: traditional AI focused on self-contained intelligence, where the model reasons primarily over its own input context. Today’s systems, in contrast, must be understood as individuals embedded in the world¹, agents whose perception and action are grounded in broader spatial, temporal, and social environments. Beyond the technical evolution, this amounts to a philosophical deepening of AI’s cognitive scope. Li captures the underlying continuity in evolutionary terms: spatial intelligence is, in her words, what “links perception with action”, the capacity that turned seeing into surviving and, eventually, into understanding and doing [7].

Without accurate spatial modeling of the world, therefore, there can be no robust reasoning, decision-making, or interaction. A wide range of emerging tasks, including robotic manipulation, embodied navigation, video understanding, and virtual human interaction, demand spatial grounding as a prerequisite, which makes spatial intelligence a load-bearing requirement for any AI system expected to act in, adapt to, and help within the real world.

Historically, from the era of classical machine learning [8, 9] to the deep learning revolution [10], and now to the LLM wave [11, 12], progress in AI has largely followed a data-centric trajectory: larger models trained on increasingly vast static corpora. While this paradigm has delivered remarkable achievements, it also imposes an inherent limitation, in that AI systems are still framed as static input–output function approximators rather than as agents that can perceive, reason, and act within the real world. The recent surge of LLMs, such as GPT [11], PaLM [13], and LLaMA [14], has only amplified this trajectory. Although these models exhibit stunning linguistic fluency, symbolic manipulation, and emergent generalization capabilities [15, 16], they remain intrinsically disembodied and spatially ungrounded. They can generate poetic descriptions of a cup on the table, yet they cannot perceive the cup, interact with it, or build a physical model to explain its behavior. This exposes the core limitation of the current paradigm: LLMs might operate on surface-level linguistic statistics without accessing the structured constraints, causal mechanisms, and physical affordances of the real world.

Fei-Fei Li has repeatedly emphasized this disconnect. Linguistic competence, she argues, must not be mistaken for understanding: for all their capability, today’s language models “remain wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded” [5]. Without spatial intelligence, encompassing physics, motion, and the transformative nature of actions, AI systems cannot develop a grounded, interactive model of reality. Her insight exposes a deeper methodological gap: true intelligence must ground its internal representations in embodied space, action, and physical experience, and text alone cannot supply that grounding.

Yann LeCun extends this critique at the architectural level, famously dismissing LLMs as “an off-ramp, a distraction, a dead end” on the path toward human-level intelligence. In his view, autoregressive language models do not, and cannot, construct models of how the world works; they continue text through pattern completion rather than through prediction grounded in the dynamics of an external world [17]. His proposed alternative, an architecture organized around a learned predictive world model, points squarely toward systems capable of predictive modeling, causal reasoning, and dynamic world interaction.

Rich Sutton, from a reinforcement learning (RL) perspective, presses the same point on the learning process itself. Sutton’s “bitter lesson” famously argued that general methods which scale with computation win out over hand-engineered knowledge [18]; notably, he does not regard today’s LLMs as the vindication of that argument. In his recent view, systems trained once on static human data and then frozen are cut off from the very thing that drives intelligence, namely continual learning from one’s own experience [19]. Silver and Sutton argue that the field must move from the “era of human data” into an “era of experience”, in which agents learn predominantly from acting in the world and observing consequences. Learning from static text captures correlations in what humans have already written; understanding the world requires action, consequence prediction, and causal discovery.

Taken together, these critiques, arising independently from visual cognition, neural architectures, and reinforcement learning, signal a profound methodological shift in progress: AI must evolve from a purely LLM-centric paradigm to one of world-integrated intelligence. It must move beyond static, offline training toward continual, interactive, and adaptive learning, and it must outgrow shallow pattern matching in text to engage with the physical, spatial, and causal structures that govern real environments. More than an algorithmic upgrade, this demands a redefinition of what intelligence means: agents capable of perceiving the world, predicting its dynamics, interacting within it, and learning continuously from such experiences [20, 21, 22].

Human intelligence is inherently multimodal. Any meaningful pursuit of AGI must therefore embrace multimodality as a foundational principle. The trajectory of multimodal AI is undergoing a significant shift, from early-stage modality separation toward deep modality integration. In earlier AI systems, perception and modeling are often confined to a single modality [23], such as text-only language models, speech recognition engines, or visual classification systems. However, with the rise of foundation models, the research paradigm is shifting towards hybrid-modality [24, 25] and even any-to-any cross-modal understanding and generation [26]. That is, both input and output can now be flexibly composed across arbitrary modalities (e.g., , audio-based question answering, video synthesis, image-conditioned dialogue) [27, 28, 29]. Simultaneously, the very nature of modality is expanding in dimensionality and diversity. Traditional sensory modalities, e.g., , vision, language, and audition, represent only a subset of how humans perceive the world. With the rapid advancement of sensor technology, future AI systems are expected to be equipped with a broader range of high-dimensional and multi-channel sensory inputs [30, 31, 32, 33], including spectrograms, millimeter-wave radar, infrared thermography, tactile feedback, gustatory signals, and olfactory landscapes. These emerging modalities are especially vital in domains such as Medicine [34], Robotics [35], Environmental Monitoring [36], and AR [37], offering a more human-aligned perceptual basis for multimodal AI systems. This multimodal expansion necessitates a unified modeling capability that operates at the level of holistic world understanding.

Among all modalities, vision remains the most central to human perception. One of the most critical trends in modality development is the transition from static imagery to video, and from 2D representations to richer 3D and even 4D (spatiotemporal) structures [38, 39]. Traditional image-based recognition tasks operate on single-frame static inputs. However, real-world environments demand continuous understanding, prediction, and generation across space and time. This drives the growing importance of tasks such as 3D reconstruction, temporal modeling, trajectory prediction, video-based question answering, and dynamic world state tracking [40, 41]. The increasing richness of modalities goes well beyond a technological extension; it is a necessary foundation for building AGI. To rival or surpass human intelligence, AGI must be equipped with a comparably rich sensorium, enabling it to perceive, reason, and act effectively in dynamic, multimodal environments. Thus, the integration and extension of modalities are not ends in themselves, but serve the larger ambition of world modeling, i.e., , the central focus of this work.

This point has been underscored by Fei-Fei Li in her discussion of spatial intelligence. Tracing the evolutionary arc from the first light-sensing organisms to primates, she observes that “nature has created this virtuous cycle of seeing and doing powered by spatial intelligence” [7]. Such capability acts as a bridge between cognition and the physical world, a necessary scaffold for grounding AI in embodied reality; without it, meaningful interaction with the real world is out of reach. Where traditional intelligence centers on language or symbolic manipulation, spatial intelligence is tightly coupled with physical structure, three-dimensional relationships, and dynamic evolution, making it the cornerstone of embodied, interactive intelligence. From the perspective of modality development, then, spatial intelligence marks the turning point where multimodal intelligence evolves from surface-level perception to deep, structured understanding, well beyond a mere byproduct of increasing sensory complexity. The focus of AI research must accordingly shift from modeling and aligning modalities to modeling the structured world that those modalities collectively describe, the path toward a closed-loop paradigm from perception to world modeling, and from world modeling to causal reasoning.

Modern AI research is undergoing a profound disciplinary evolution, centered around modality enrichment and spatiophysical coupling. This paradigm shift reflects a convergence of traditionally siloed domains, spanning Computer Vision (CV), Natural Language Processing (NLP), Audio Processing, Robotics, Geographic Information Systems (GIS), and Urban Simulation, toward a unified framework for modeling intelligent agents that are physically embedded in and interact with the real world.

In the vision domain, the research focus has gradually shifted from static image processing to dynamic scene understanding. These developments provide a perceptual foundation for large-scale world modeling. Language modalities have also evolved beyond static language modeling (e.g., , BERT [42], GPT [43]), toward multimodal semantic fusion tasks such as Visual Question Answering (VQA) [44], Vision-and-Language Navigation (VLN) [45], and embodied dialogue, which have become critical interfaces in the expansion of large language models into the multimodal regime. The audio modality has similarly moved beyond traditional speech recognition to include audio event detection, ambient sound understanding, and even 3D auditory modeling, all of which support spatial cognition and situational awareness in real-world interactive environments [46, 47].

Meanwhile, the rapid development of robotics and embodied AI is pushing the evolution of AI systems from “perception-understanding” pipelines to closed-loop “perception-action” systems. AI is outgrowing its existence as software on a computer, becoming embodied, real-time, and spatially grounded. In domains such as autonomous driving, robotic manipulation, and wearable AI systems, tasks increasingly require deep spatial reasoning capabilities for layout perception, path planning, and object manipulation [48, 49]. These tasks demand integration not only of vision and language capabilities but also of knowledge from control theory, motion planning, and state estimation, marking a shift toward truly multidisciplinary AI systems.

The expansion of modalities is also reshaping traditional disciplines through interdisciplinary integration. In GIS and urban computing, AI models are increasingly combined with spatial coordinates, map representations, and temporal population data, enabling applications in smart city planning, trajectory prediction, and disaster simulation. In Earth sciences and remote sensing, AI is employed for multimodal surface analysis, climate modeling, and environmental monitoring, pushing the frontier toward 3D Earth modeling and dynamic ecological understanding [50]. In architectural design and digital twin systems, AI supports full lifecycle 3D spatial modeling, simulation, and predictive analytics for complex structures and spaces.

Cognitive science and neuro-inspired AI (NeuroAI) are playing an increasingly pivotal role in this trend. Foundational research on spatial intelligence in human cognition, such as hippocampus-mediated cognitive maps, spatial memory, and embodied cognition theory, is gradually informing the design of AI spatial perception systems [51, 52]. As Fei-Fei Li puts it, spatial intelligence is “the scaffolding upon which our cognition is built”, the substrate that anchors intelligence to the physical world [5]. Thus, the path toward AGI must transcend abstract symbolic computation and advance toward grounded-in-reality agent intelligence. This trend reshapes the research trajectory of AI and, at the same time, redefines the underlying infrastructure of general intelligence itself, with spatial intelligence standing as a central pillar of the emerging world-centered era.