8.6World Modeling

Vision-oriented Modeling. A major line of representation world modeling builds latent spaces from visual observations: video frames are encoded into a compact latent space by self-supervised learning, and a transition model predicts future latent states from past latents and actions, which captures the dynamics of the visual world more efficiently than pixel-space prediction. Concretely, an encoder $E_{\phi}$ maps observations to latent states and a predictor $P_{\theta}$ is trained to minimize a prediction loss defined entirely in latent space,

where $\operatorname{sg}(\cdot)$ denotes the stop-gradient operator and $E_{\bar{\phi}}$ a slowly updated target encoder. This objective instantiates the transition of Eq. (8.71) with the predicted state $\hat{S}(t+1)$ taken to be an abstract embedding rather than raw pixels (Figure 8.24(a)), which is precisely what allows representation world models to capture dynamics efficiently while discarding perceptually salient yet predictively irrelevant detail.

The World Models architecture [4] decomposes the problem into a VAE-based vision model, an RNN-based memory model, and a lightweight controller; the vision and memory components learn latent dynamics from raw pixels without supervision, and the controller then learns its behavior inside the “dreamed” environment, which greatly simplifies policy learning. The Dreamer agents carry this idea into reinforcement learning at scale: Dreamer [147] learns a latent dynamics model from pixels and trains an actor and critic entirely on imagined latent trajectories, and DreamerV3 [152] applies a single configuration across a wide range of domains. MuZero [148] learns a model that predicts only the quantities planning needs (reward, value, and policy) and plans with it by tree search, mastering Atari, Go, chess, and shogi without being given the rules.

Yann LeCun’s autonomous intelligence framework proposes the Joint Embedding Predictive Architecture (JEPA) [17] as the core mechanism for constructing world models. JEPA learns latent representations and predictions through self-supervised objectives rather than pixel-level reconstruction, and Eq. (8.72) is its generic form. I-JEPA [151] first instantiated it on images, predicting the embeddings of masked target regions from a visible context block, and V-JEPA [2318] carried the same feature-prediction objective to video, predicting masked spatio-temporal embeddings in latent space with no pixel reconstruction, text, or negative examples, and so avoiding the cost and ambiguity of pixel prediction. V-JEPA 2 [155] turns this encoder into a world model. Pretrained on more than a million hours of internet video, it is extended with an action-conditioned predictor, V-JEPA 2-AC, post-trained on less than 62 hours of unlabeled robot video; deployed zero-shot on Franka arms, it picks and places objects by planning actions whose predicted embeddings reach a goal image.

The stop-gradient and slowly updated target encoder in Eq. (8.72) exist only to keep the representation from collapsing, and the most recent JEPA work removes them. LeJEPA [2319] shows that an isotropic Gaussian is the optimal embedding distribution for downstream prediction and enforces it with a single sketched regularizer (SIGReg), leaving one trade-off hyperparameter and no teacher–student machinery. LeWorldModel [2320] uses the same regularizer to train an action-conditioned JEPA world model end to end from raw pixels with only two loss terms; at about 15M parameters it trains on a single GPU, plans up to 48 times faster than world models built on foundation-model encoders, and its latents encode physical quantities well enough to flag physically implausible events. The principle has also reached language: VL-JEPA [200] predicts continuous embeddings of the target text instead of generating tokens, and LLM-JEPA [2321] adds an embedding-prediction objective to the next-token training of language models.

DINO-World [2322] instead freezes DINOv2 as a universal visual encoder, whose features are already semantic, geometric, and object-centric, learns a latent-space predictor from large-scale unlabeled video, and adds action-conditioned prediction to model how interventions change future visual states.

Hybrid World Modeling. Another important direction expands representation learning beyond vision by incorporating additional modalities such as LiDAR, camera trajectories, or multi-view geometric signals. These hybrid models aim to overcome the limitations of purely visual latent spaces by injecting explicit spatial structure, depth cues, and geometric consistency constraints. For instance, MUVO [1446] is a multimodal generative world model that learns jointly from synchronized camera and LiDAR streams. By integrating LiDAR-derived depth priors into the latent world representation, MUVO achieves more reliable 3D scene prediction, improved object motion modeling, and long-horizon forecasting in autonomous driving environments. UniWorld [2323] introduces a unified pretrained architecture that merges 3D object detection, motion prediction, and scene representation learning. Through joint training of BEV features, temporal dynamics, and multisensor context, UniWorld produces a structured latent state that simultaneously supports perception and trajectory forecasting. Aether [2324] proposes a geometry-aware unified world model where camera trajectories are treated as explicit action signals. This allows the model to learn action-conditioned scene transitions and perform predictive visual planning. By binding egomotion with geometric consistency constraints, Aether generates spatially coherent future predictions and supports counterfactual simulation within a 3D-aware latent space.

Language-oriented World Modeling. A final category examines language as a central medium for latent world representation. This raises an important question: can LLMs themselves be viewed as world models? From a formal perspective, LLMs indeed share structural similarities with classical world models. They maintain internal states, treat textual context as observations, accept prompts as actions that alter these states, and generate predictions that correspond to inferred future states. Models such as Dynalang [2325] demonstrate that natural language can serve as a universal intermediate representation for encoding environment states, predicting transitions, and expressing causal relations. LLMs also exhibit emergent knowledge of commonsense physics, intuitive causality, and human behavior derived from massive text corpora. However, their limitations remain clear: they are trained on symbolic descriptions rather than direct interaction with the physical world; their internal dynamics reflect linguistic statistics more than physical laws; and their world representations tend to be sparse, fragmented, and disconnected from geometric or temporal continuity. Thus, while LLMs can be interpreted as language-driven world models, their predictive capacity is primarily symbolic and semantic, rather than grounded in spatial or physical understanding.

Video-centric Generative World Models. Recent advances in video generation have renewed interest in the idea of a “world simulator”. Video models are intrinsically suited to encoding spatiotemporal structure, and when they produce sequences that are not only visually coherent but also consistent with basic physical intuition, they begin to resemble implicit world models. This became especially prominent with the release of OpenAI Sora [1991], whose generated videos demonstrate strong temporal consistency, long-horizon stability, and apparent physical plausibility, prompting the community to reconsider whether deep video generation might be converging towards a form of physical world modeling. Subsequently, Google DeepMind’s Genie 3 [157] extended video generation from passive “watchable” content to interactive experiences: the model updates the generated world in response to user actions, enabling agent-in-the-loop, closed-loop world simulation, complementing the action-conditioned scene generators surveyed in §8.2.2. World Labs’ RTFM [1820] further highlights real-time performance, achieving high frame-rate and low-latency prediction. Systems such as Wan [1990], Emu3.5 [206], OpenAI Sora 2 [2328], and Veo 3 [2329] continue this trajectory, gradually pushing video generation from mere visual fidelity towards world coherence, physical consistency, and interactivity. Action-conditioned diffusion has also turned games into testbeds: GameNGen [1818] simulates the game DOOM interactively in real time, and DIAMOND [1819] trains reinforcement-learning agents entirely inside a diffusion world model of Atari games, while NVIDIA’s Cosmos [156] offers pretrained video world foundation models intended to be post-trained as simulators for robotics and driving.

Video world models are particularly promising in autonomous driving. GAIA-1 [2330] proposes a generative world model tailored for driving scenarios, built around a spatiotemporal Transformer that generates future multi-view scenes based on past observations. By implicitly capturing traffic patterns, pedestrian behavior, scene geometry, and multi-agent motion, GAIA-1 can support closed-loop evaluation, behavior prediction, and synthetic data generation, thereby reducing the cost and risk of traditional real-world testing.

3D/4D Asset Generation as World Modeling. A complementary avenue is to build world models that operate directly on structured three- or four-dimensional representations, such as 3D geometry, occupancy fields, or 4D spatiotemporal volumes. Unlike video-based approaches that primarily model dynamics in pixel space, these methods explicitly parameterize geometry and temporal evolution, making it easier to represent interactive, spatially consistent worlds.

Genesis [2331] exemplifies this direction. It is not only a generative 3D/4D physics simulation platform, but also integrates a vision-language model-driven “world-building agent” that uses a tool API to create dynamic 4D environments (geometry plus dynamics). These generated worlds can be rendered into multiple modalities (RGB, depth, semantics, contact events, etc.), providing rich supervision for training embodied AI systems. Genesis thus combines a unified physics engine with generative agents for large-scale world construction.

Autonomous driving again serves as a natural application domain for 3D world models. DriveArena [2332] builds a closed-loop generative simulation platform, allowing planning modules to iteratively probe and learn within synthesized 3D driving environments. Its diffusion-model-based multi-agent scene generation can rapidly produce long, interactive driving episodes. OccWorld [2333] and OccSora [2334] adopt 4D occupancy grids as the core representation, predicting future 3D occupancy over time from continuous video and sensor streams, thereby providing a unified motion-and-structure world model. Copilot4D [2335] tokenizes LiDAR scans and forecasts future LiDAR point clouds with discrete diffusion, learning a world model from unlabeled sensor data alone.

Generating interactive 3D worlds brings generative world models closest to their conceptual ideal. Marble [158] from World Labs combines action-conditioned generation, geometric consistency constraints, and multimodal inputs, so that users can act within the generated world and observe the consequences. Its next-generation model, Atlas [2336], is an omni world model pretrained from scratch on text, images, video, and 3D: an autoregressive diffusion transformer that conditions every generated image and depth map on an explicit camera pose and grounds them all in a shared 3D context, so that the same model generates camera-controlled video of up to a minute, reconstructs scenes from a handful of images, and exports its output as point clouds or 3D Gaussian splats. World Labs reports that it outperforms specialized open-source reconstruction models, and that it will power future versions of Marble.

Dimensions of Mental State Representation. Beyond the belief-desire-intention core, a mental world model for embodied agents must span several further layers of internal state and social context. It has to read affect, recognizing emotional cues and how they shift a person’s risk tolerance and willingness to cooperate. It has to represent social dynamics: the relationships, roles, group structures, and cultural norms that shape behavior. And it has to interpret communication, both the verbal channel and the non-verbal one carried by tone, gesture, posture, and facial expression. These layers are what turn a bare goal estimate into a socially situated account of why a person acts as they do.

Human spatial reasoning is social as well as geometric, which ties mental modeling to spatial intelligence. To assist or collaborate with humans in homes, factories, classrooms, or vehicles, an agent must infer the intention behind a spatial action (e.g., , where a person intends to move next), anticipate trajectories from goals or habits, adapt navigation or manipulation to user preferences, and interpret spatial configurations that reflect what a person believes or misunderstands.

Joint State. The world state is the pair

The physical part collects the objects, the embodied characters, their spatial and contact relations, and the environmental conditions of the scene. The mental part holds an individual state $m_{t}^{i}$ for every agent $i$ in the set $\mathcal{I}$, optional states $m_{t}^{G}$ for groups $G\in\mathcal{G}$, the social attitudes and role relations $R_{t}^{\mathrm{ment}}$ between agents, and the atmosphere $\alpha_{t}$ of the setting. Each individual state extends the belief-desire-intention core introduced above,

with identity attributes, beliefs, attention focus, goals, intentions, emotions, dispositions, norms, and behavioral constraints. Beliefs, goals, and intentions are the BDI variables; emotions, norms, and the relations $R_{t}^{\mathrm{ment}}$ turn the affect and social-dynamics layers discussed above into explicit state variables.

Target-Specific Observation. The model reasons from the perspective of one target agent $\epsilon\in\mathcal{I}$, which sees only a rendering of the joint state:

where $\kappa_{t}^{\epsilon}$ is the target’s perceptual access (position, line of sight, hearing range, available modalities) and $\rho_{t}^{\epsilon}$ its social and cognitive perspective (role, prior beliefs, relationships to others, awareness of norms, Theory-of-Mind capacity). The mental part of the observation contains the target’s view of its own state and its estimates $\widehat{m}^{\,j,(\ell)}_{t,\epsilon}$ of every other agent $j$, nested up to a depth $L$; at the first level they are sampled as

which is the inference of Eq. (8.75), made stochastic and conditioned on the target’s own mind and social context. Spatial access enters the mental model directly through $\kappa_{t}^{\epsilon}$: what the target can see from where it stands bounds what it can believe, so a poor decision can follow from an object outside its view as much as from a mistaken belief.

Action and Coupled Transition. The target proposes $K$ candidate actions $\mathcal{A}_{t}^{\epsilon}=\Pi_{\epsilon}^{K}(o_{t}^{\epsilon})=\{a^{\epsilon}_{t,1},\dots,a^{\epsilon}_{t,K}\}$. Each action has a physical carrier and a mental content, $a_{t}^{\epsilon}=(a_{t}^{\epsilon,\mathrm{phy}},a_{t}^{\epsilon,\mathrm{ment}})$ (approaching someone, and meaning to comfort them), and the policy factorizes as $\Pi_{\epsilon}(a_{t}^{\epsilon,\mathrm{phy}},a_{t}^{\epsilon,\mathrm{ment}}\mid o_{t}^{\epsilon})=\Pi_{\epsilon}^{\mathrm{phy}}(a_{t}^{\epsilon,\mathrm{phy}}\mid o_{t}^{\epsilon})\,\Pi_{\epsilon}^{\mathrm{ment}}(a_{t}^{\epsilon,\mathrm{ment}}\mid a_{t}^{\epsilon,\mathrm{phy}},o_{t}^{\epsilon})$. The world model then predicts the next joint state with a staged transition,

The asymmetry is deliberate. An intention moves nothing until it is carried out physically, so the physical factor ignores $a_{t}^{\epsilon,\mathrm{ment}}$; the mental factor depends on both components, because the same reassuring words land differently when the speaker is approaching than when walking away. The physical factor still conditions on the mental state, since what objects, space, and visibility allow is joined by what other agents intend to do. Those other agents are not controlled: their actions $a_{t}^{-\epsilon}$ are marginalized under the simulator’s predictive distribution,

with $p(a_{t}^{-\epsilon}\mid s_{t})=\prod_{j\neq\epsilon}\Pi_{j}\big{(}a_{t}^{j}\mid\Omega_{j}(s_{t})\big{)}$ for simultaneous moves, so that every agent still acts on its own partial observation. The whole pipeline, $s_{t}\xrightarrow{\Omega_{\epsilon}}o_{t}^{\epsilon}\xrightarrow{\Pi_{\epsilon}}a_{t}^{\epsilon}\xrightarrow{T_{\theta}}s_{t+1}$, is Eq. (8.71) with a mental component added to the state and a perspective added to the observation.

Evaluating Candidate Futures. Simulating each candidate gives a branch $\hat{s}^{k}_{t+1}$, and the predicted decision is the branch of highest value,

In MENTIS, the training-free and fully inspectable baseline of MWM, the branch value $r_{k}$ is a weighted combination of normalized scores for physical plausibility, mental consistency, and social appropriateness, and a binary safety and legality veto sets it to zero. Each formal component maps to a module: a state parser recovers $\hat{s}_{t}$, an observation generator implements $\Omega_{\epsilon}$, an action parser structures the candidates, a world-state transitor implements $T_{\theta}$, and an evaluator assigns $r_{k}$. On situated decision scenarios presented as text, image, and sounding-video stories, experiments across eight LLM-based world models indicate that the explicit mental component is what makes a person’s actual decision predictable [2352].