10.1AR/VR/XR

Augmented, virtual, and mixed reality (collectively, extended reality or XR) form a continuum of media in which a head-worn display couples the user’s egocentric viewpoint to a computational world that is registered to, or substituted for, the physical surroundings. What distinguishes XR from screen-based media is that spatial intelligence is not an optional feature but a constitutive requirement: before any pixel can be drawn, the system must know where the headset is, what geometry surrounds it, and where the user is looking and reaching. XR is therefore the most demanding real-time deployment target for the spatial capabilities surveyed in Part III. The perception machinery (detection, depth, pose, reconstruction) of Chapter 6, the spatial reasoning and planning of Chapter 7, and the neural reconstruction and rendering of Chapter 8 all reappear here, but specialized to run egocentrically, on-device, and within a hard latency budget.

This section treats the XR medium itself as a spatial-computing platform: the runtime engine that localizes the device, maps and registers the world, senses the user’s body, and presents a composited result. The neighboring section on entertainment (§10.2) builds on top of this engine, treating entertainment as an application domain: its real-time AR/VR experiences and virtual production, together with the offline, high-fidelity reconstruction and performance capture that supply them with authored content; here we focus on the platform that makes such experiences run at interactive rates. Concretely, a headset must continuously answer three coupled questions, i.e., , where am I (6-DoF self-localization), what is around me (geometry and semantics), and how do I fuse virtual content into that reality so that it appears physically present, while simultaneously sensing the user and refreshing the display. Early systems answered the localization question with outside-in tracking, using external base stations or cameras (SteamVR Lighthouse, the original Oculus constellation), but the dominant modern paradigm is inside-out tracking, in which visual–inertial SLAM runs entirely on the device, as in Meta Quest, Apple Vision Pro, and Microsoft HoloLens. The three pivotal tasks below correspond to the stages of this loop.

As sketched in Figure 10.1, an XR device closes a tight real-time loop. Egocentric sensors first sense the device and the user (outward cameras and an inertial measurement unit for head pose; inward-facing cameras for eyes and outward- or downward-facing cameras for hands and body). The system then tracks and maps the environment, recovering a 6-DoF pose and a persistent geometric model, and registers virtual content into that model with correct anchoring, occlusion, and lighting. Finally, it renders and displays the composite back through the headset optics, after which the user moves and the loop repeats. The pipeline shares an end-to-end motion-to-photon budget; roughly $20$ ms at a display refresh of $90$ Hz or higher are the commonly cited targets, though the right values depend on device and application, and exceeding them degrades presence and comfort. The three pivotal tasks map onto this loop directly: Task-1 is the world-facing engine (track, map, register), Task-2 is the user-facing input (egocentric sensing and interaction), and Task-3 is the user-facing output (display and rendering).

The first task is the medium’s spatial backbone: it answers where the device is, what surrounds it, and how virtual content is pinned to the world so convincingly that the illusion holds. It proceeds in three steps: localize the device, model the environment, and register synthetic content into that model. Figure 10.2 sketches this progression from inside-out tracking to a persistent scene map and finally to anchored, occluded, and relit virtual content.

Inside-out 6-DoF tracking. The foundational layer estimates the headset’s six-degree-of-freedom pose at high rate by fusing visual features with inertial measurements. Feature-based visual SLAM with loop closure and relocalization remains the academic standard, exemplified by the ORB-SLAM lineage and its visual-inertial extension ORB-SLAM3 [736], while tightly-coupled visual–inertial odometry such as VINS-Mono [2644] underpins many mobile and headset stacks; direct (photometric) methods such as DSO [719] and the earlier LSD-SLAM [718] offer an alternative that does not depend on sparse keypoints. More recent learned dense systems, including DROID-SLAM [95] and transformer-based geometric matchers [728], trade computation for robustness in low-texture and fast-motion regimes. These methods are evaluated chiefly on EuRoC [733] and TUM-VI [935], with absolute trajectory error (ATE) as the headline metric, but the binding constraints in XR are drift accumulation, recovery from tracking loss (the kidnapped-device problem), and the power and thermal envelope of on-device inference.

Spatial mapping and scene understanding. The tracked stream is consolidated into a usable model of the environment: dense depth and mesh, planar surfaces (floor, wall, table), and semantic labels. Real-time volumetric fusion in the KinectFusion tradition [114] provides the geometric substrate, on top of which plane and mesh extraction expose the surfaces that gameplay and user interfaces attach to. (The offline, high-fidelity reconstruction of such scenes as navigable content is treated in §10.2 and Chapter 8; here the map is a means to localization and registration, not an end in itself.) Semantic and instance understanding is benchmarked on indoor datasets such as ScanNet [448] and Matterport3D [340], with mean intersection-over-union (mIoU) for segmentation, and increasingly on mobile-captured data with laser ground truth such as ARKitScenes [669] and high-quality reconstructions such as Replica [339] for depth and geometry evaluation.

Persistent and shared anchors. For content to survive across sessions and to be experienced jointly by co-located users, the map must be persistent and shareable. Cloud-hosted and peer-to-peer world maps let a device relocalize against a previously built model, and at city scale a Visual Positioning System anchors content to real landmarks in well-mapped areas. The open problems are relocalization under appearance change (lighting, season, rearranged furniture), the memory footprint of on-device maps, and the privacy implications of uploading scans of private spaces, a thread we return to in the challenges.

Registration and realistic compositing. Finally, virtual content must be registered both geometrically and photometrically so that it appears to occupy the scene. Geometric registration pins objects to anchors and planes in the SLAM frame; depth-based occlusion lets real geometry correctly hide virtual pixels, as in the per-pixel depth produced for mobile interaction by DepthLab and the ARCore Depth API [2645]; and lighting estimation predicts the dominant light direction and an ambient environment map from a single image so that virtual objects are shaded and shadowed consistently [2646]. Small errors here are perceptually unforgiving: a few centimeters of depth error at an object boundary, or a mismatched shadow, immediately reveals the composite as synthetic, which is why robust occlusion and real-time inverse lighting remain active research problems.

The second task closes the loop from the user’s body back into the system: it senses the hands, eyes, and posture from inside-out cameras and turns those signals into spatial intent. Low-level egocentric estimates of hands, gaze, and body are fused into selection, manipulation, and locomotion. As Figure 10.3 indicates, the same headset sensors that localize the device are repurposed to track the user, and the resulting estimates drive a gaze-and-gesture interaction model.

Hand tracking. Articulated 3D hand pose (commonly $21$ joints, often a full mesh) is regressed from headset cameras under heavy self-occlusion and truncation. The standard hand model is MANO [2647], and on-device pipelines such as MediaPipe Hands [2648] combine a palm detector with a landmark regressor to run in real time. Methods are benchmarked on FreiHAND [561] and the two-hand InterHand2.6M [562], with 3D mean per-joint position error (MPJPE) in millimeters as the headline metric; robust two-hand interaction and egocentric truncation remain the hard cases.

Eye and gaze tracking. Estimating 3D gaze from infrared eye images serves three roles: calibration, gaze as a pointing modality, and as the driver for foveated rendering (Task-3). Appearance-based estimation in the iTracker and GazeCapture tradition [2649] established that gaze can be regressed from commodity cameras, and headset-specific datasets such as OpenEDS [2650] provide near-eye infrared imagery with dense eye-region segmentation, with the later OpenEDS2020 track adding gaze prediction (angular error in degrees). The decisive interaction consequence is that gaze targets quickly but lacks a natural trigger, a long-standing difficulty known as the Midas-touch problem, which motivates the gaze-plus-pinch pattern discussed below.

Full-body pose from sparse signals. Driving a full-body avatar from a headset and two controllers is an underconstrained inverse problem, since the lower body is unobserved and must be synthesized. Recent methods infer plausible full-body motion from these three tracked points, including the transformer-based AvatarPoser [2651] and the diffusion-based AGRoL [2652], while a parallel line recovers pose from sparse body-worn inertial sensors, as in DIP [2441], TransPose [2444], and the physics-aware PIP [2452]. These models are trained on large motion corpora such as AMASS [2209] and evaluated with MPJPE and temporal jitter. This sparse, live setting is distinct both from the wearable-IMU body sensing of §9.3 (which instruments the body more densely) and from the studio performance capture that drives authored avatars in §10.2; the XR-specific challenge is plausibility from radically incomplete observation.

Multimodal interaction and locomotion. The recovered signals are fused into spatial actions. The foundational idea of combining a deictic gesture with a complementary modality dates to Bolt’s “Put-That-There” [2653], and the modern instantiation decouples targeting from confirmation: the eyes select while a hand pinch or a voice command commits, sidestepping the Midas-touch problem. Locomotion through a virtual space larger than the physical room is handled by redirected walking, which applies imperceptible rotational and translational gains to steer the user within a confined area [2654], with teleportation as the common non-natural fallback. Spatial user interfaces, in turn, must choose a reference frame (world-locked, body-locked, or head-locked) and increasingly anchor panels to scene semantics so that content lands on appropriate surfaces. Learning these interaction behaviors at scale is increasingly grounded in large egocentric corpora such as Ego4D [681], which supply the first-person video of hands, objects, and activities that egocentric-understanding models are trained and benchmarked on.

The third task presents the composited world back to the user within the perceptual budget that the previous two stages must respect. Three questions decide it: where to spend rendering effort, how to hide system latency, and how to display the result comfortably. Figure 10.4 previews this pipeline, from gaze-contingent shading through latency compensation to the optics that place the image before the eye.

Foveated rendering. Because acuity falls off sharply outside the fovea, rendering effort can be concentrated at the gaze point. Foveated 3D graphics established multi-resolution shading driven by eye tracking [2655], and subsequent work added temporal stability and contrast preservation to make the quality reduction imperceptible [2656]. Eye-tracked foveated rendering now ships on consumer headsets, materially reducing shading cost on thermally constrained hardware.

Latency compensation. Even an efficient renderer cannot meet the motion-to-photon budget on every frame, so the last stage warps the rendered image to the most recent head pose. Asynchronous time warp applies a rotational reprojection at display refresh, asynchronous space warp adds depth-aware frame extrapolation, and head-mounted optical-see-through systems use per-pixel late-stage reprojection [2657]. These techniques keep the world stable under fast head motion and are essential to comfort.

Passthrough versus optical see-through. Two display regimes trade off differently. Video passthrough mixed reality (as on Meta Quest 3 and Apple Vision Pro) renders the real world from cameras, granting full control over occlusion and relighting at the cost of camera-to-display latency, parallax, and limited dynamic range; optical see-through (as on HoloLens 2 and Magic Leap 2) presents the real world with zero latency at its true focal depth (though the virtual overlay still incurs render latency and a fixed focal plane) but cannot occlude real light or match real-world brightness. The choice propagates back into the registration problem of Task-1.

Depth-correct rendering and visual comfort. Conventional stereo displays fix the focal plane while vergence varies with content, producing the vergence-accommodation conflict and the eye strain that limits session length. Varifocal, light-field, and holographic near-eye displays aim to restore correct focus cues [2658], but remain largely at the prototype stage.

Neural rendering for telepresence. Finally, photoreal social presence requires driving a faithful avatar of a remote user from headset-mounted cameras that see only oblique crops of the face. The codec-avatar line, beginning with deep appearance models [2659], learns a person-specific decoder that is driven on-device, a direct specialization of the neural rendering of Chapter 8 to the constrained, real-time XR setting.

A mature ecosystem implements these tasks across mobile AR, standalone headsets, and the web, and it is organized into developer frameworks, cross-platform standards, hardware, and the companies that drive them.

SDKs and standards. At the device level, Apple’s ARKit¹ and Google’s ARCore² expose plane detection, anchors, depth, light estimation, and people occlusion, with ARCore’s Geospatial API localizing against world imagery for outdoor anchoring; Vuforia³ specializes in robust image and object targets for industrial AR. Cross-engine abstraction layers such as Unity AR Foundation⁴ and the Meta Presence Platform and Spatial SDK⁵ present the common subset (planes, anchors, meshes, occlusion, hand and body tracking) over multiple devices, while 8th Wall and WebXR⁶ bring markerless AR into the browser. These abstractions matter because XR content is expensive to author and must migrate across devices and sessions; this is the role of cross-platform standards, principally OpenXR⁷ as a unifying runtime API and glTF together with OpenUSD and USDZ⁸ as asset and scene-description formats. Without a shared runtime and shared asset formats, the ecosystem fragments into incompatible walled gardens.

Hardware. Current headsets divide along the passthrough versus optical-see-through axis. Apple Vision Pro⁹ (2024) and Meta Quest 3 and 3S¹⁰ use high-resolution video passthrough, with the Quest line the consumer volume leader; Microsoft HoloLens 2¹¹ (optical see-through, with production ended in 2024) and Magic Leap 2¹² target enterprise, and Varjo¹³ and Pico¹⁴ serve high-fidelity simulation and the Asian consumer market respectively. Lightweight true-AR glasses, such as Snap Spectacles¹⁵ and Meta’s Orion and Project Aria research prototypes, signal the all-day-glasses horizon toward which the field is converging.

Companies and platforms. Meta leads consumer volume and the most aggressive glasses research bet; Apple defines the premium spatial-computing tier and the USD content pipeline; Google supplies world-scale localization through ARCore and Android XR; Microsoft has pivoted from first-party HoloLens hardware toward OpenXR software and enterprise cloud services; Niantic¹⁶ operates a city-scale Visual Positioning System for persistent, multi-user AR; and Snap, Varjo, and ByteDance’s Pico round out the social-AR, high-fidelity, and consumer-standalone niches.

XR runs every spatial capability egocentrically and on-device under a perceptual real-time budget, and its open problems follow from that constraint: holding the motion-to-photon budget, fitting perception into the compute and thermal envelope of glasses, keeping tracking and registration robust in the wild, sharing persistent maps without compromising privacy, and making extended wear visually comfortable.