9.2Autonomous Driving

Autonomous driving sits at the intersection of spatial intelligence, transportation engineering, and human–machine interaction, and it is one of the most demanding real-world testbeds for the spatial capabilities surveyed in Part III. A self-driving vehicle must continuously close a perceive–localize–decide–act loop: it has to recognize and track the surrounding traffic scene, estimate its own pose within a metric map, reason about how nearby agents will move, and commit to safe control actions, all under tight latency and safety budgets. Advances in deep learning, multi-sensor fusion, and vehicle-to-everything (V2X) communication have steadily pushed the boundaries of what vehicles can perceive, predict, and decide, while fundamental questions around robustness, trust, ethics, and regulation remain open. Perception and reasoning themselves are treated in Chapter 6 and Chapter 7; what is specific to driving is that each capability must hold up at speed, in all weather, and with lives at stake.

Figure 9.6 summarizes the SAE J3016 automation levels that frame this domain. The SAE J3016 standard ¹ provides a widely adopted framework for defining automation levels, with most current production systems still operating at Level 2. Progress toward higher levels, especially full Level 5 autonomy, faces major technical [2410], ethical [2411], and financial [2412] challenges.

Deep learning has driven most of this progress, from Transformers [2413] and graph networks [2248] that model multi-agent interaction to lifelong learning for open-world adaptation [2414], but its opacity raises safety concerns compounded by sensor noise and the limited reach of vehicle-to-vehicle communication, and reaching Levels 4 and 5 depends as much on redundant hardware, large-scale data pipelines, and verifiable safety assurance as on model architecture.

Modern self-driving stacks realize the perceive–localize–decide–act loop as a modular pipeline, sketched in Figure 9.7. A front-end perception module converts raw, multi-modal sensor streams (camera, LiDAR, radar) into a structured model of the scene: what objects are present, where the drivable surface and lane geometry lie, and which agents are most safety-critical. A localization and mapping module places the ego-vehicle within a metric, typically HD-map-anchored, representation of that scene, so that perception outputs can be expressed in a stable world frame. A decision and control module then predicts how the scene will evolve and selects the steering, acceleration, and braking commands that are sent to the actuators and fed back into the environment, closing the loop. Each of the three stages is a distinct spatial-intelligence problem with its own representations, benchmarks, and failure modes, and they are the pivotal tasks below, in pipeline order.

Perception is the front-end of the pipeline: it turns raw camera, LiDAR, and radar streams into a structured, layered model of the scene that every downstream module consumes, reusing the detection, segmentation, and 3D scene-understanding machinery of Chapter 6 but specializing it to the driving Operational Design Domain (ODD). It is best read as a set of complementary structured representations, sketched in Figure 9.8: object detection localizes traffic participants, semantic and occupancy prediction labels the surrounding space, lane detection recovers the road geometry that constrains motion, and pedestrian recognition isolates the most safety-critical agents.

The base layer is object detection, which identifies and localizes obstacles and is the most direct determinant of collision safety [2415]. Its central practical constraint is the ODD: environmental factors such as poor weather degrade sensor visibility and model accuracy [2416], so recent work formalizes and iteratively validates the ODD through quantifiable environmental models and scenario catalogues, trusting a detector only within conditions it has demonstrably handled, though the absence of a standardized ODD specification still makes cross-system safety claims hard to compare. Robustness to adverse weather is therefore a first-class concern [2417, 2418, 2419]: rain, snow, fog, and glare corrupt LiDAR, radar, and camera signals alike, and because driving datasets under-represent these conditions, models trained on fair-weather data generalize poorly. Responses range from dedicated robust pipelines, such as multi-stage visibility enhancement coupled with deep CNN tracking [2420], to new sensing hardware, e.g., , frequency-modulated continuous-wave (FMCW) LiDAR and high-dynamic-range cameras that recover usable signal in glare at higher cost. Moving from the image plane into metric space, 3D object detection localizes and tracks objects in the vehicle’s surroundings [434], where the bird’s-eye-view (BEV) representation has become dominant because it yields a unified top-down map well suited to planning. BEVFormer [2421] fuses multi-camera and temporal cues with spatiotemporal transformers to approach LiDAR-level accuracy on nuScenes, while Fast-BEV [2422] and SparseBEV [2423] trade marginal accuracy for the lightweight, sparse, query-based computation needed for real-time on-vehicle deployment, together tracing the field’s balancing of accuracy, latency, and deployability.

Detections tell the planner where objects are but not which space is free; that dense layer is supplied by semantic segmentation and its 3D extension, occupancy prediction [2424, 2425]. The progression runs from 2D pixel labeling with FCN [127] and DeepLab [461] to volumetric reasoning: the Tri-Perspective View [2426] lifts RGB into a 3D representation rivalling LiDAR, SurroundOcc [2427] uses multi-camera 2D–3D attention to capture arbitrary shapes, and occupancy world models such as OccWorld [2333] extend this to forecasting how the 3D scene evolves, moving beyond the limits of bounding boxes. Because these dense predictors are data-hungry and domain-sensitive, robustness techniques such as the SePiCo pixel-contrast framework [2428] and GAN-based synthetic data [2429] are used to survive scene variability, occlusion, and weather shift. On top of the free-space layer, lane detection recovers the road geometry that defines the lateral reference for lane-keeping and planning; early CNN- and YOLO-style detectors trace markings well in clear conditions but are brittle on curved, worn, or occluded lanes, motivating successors that lift lanes into 3D and use attention over global context to infer occluded segments. Finally, pedestrian recognition focuses on vulnerable road users whose behavior can be difficult to predict: CNN-, YOLO-, and SSD-based detectors deliver the required precision and latency, color–thermal fusion handles low-light scenes, and spatial–temporal models couple detection with motion prediction to anticipate crossing behavior.

Across all four layers, perception is benchmarked on large-scale driving datasets, KITTI [173], the Waymo Open Dataset [446], and nuScenes [445] for 2D/3D detection, Cityscapes [501] and BDD100K [711] for segmentation, dedicated lane benchmarks for lane detection, and CityPersons [2430] for pedestrians, and scored with mean Average Precision (mAP) and the nuScenes Detection Score (NDS), mean Intersection-over-Union (mIoU), and F1/accuracy, respectively (Table 9.2). Increasingly these metrics are reported on adverse-weather and out-of-distribution splits, precisely to measure behavior within and beyond the declared ODD.

If perception answers what is around the vehicle, localization and mapping answer where the vehicle is and what the persistent structure of the world is. The two are tightly coupled: a metric, HD-map-anchored pose lets perception outputs be expressed in a stable world frame that planning can rely on, and the map itself is built and maintained from the same sensor streams. As Figure 9.9 shows, the problem decomposes into pose estimation against a known map (localization), the construction of that map (3D mapping and reconstruction), and the joint solution of both when no map exists (SLAM), with multi-sensor fusion the common thread that holds accuracy together when any single modality degrades.

Localization has moved from GPS-dependent positioning to learned matching of live camera or LiDAR observations against a pre-built map, reaching centimeter-level accuracy even in GPS-denied urban canyons; semantic cues such as roads and buildings further let a vehicle re-localize in changed environments without full remapping. Because no single sensor is reliable across occlusion, weather, and GPS dropout, sensor-fusion localization is the dominant paradigm: probabilistic estimators (particle filters, Kalman filters, and NDT scan matching) reconcile GNSS, IMU, and LiDAR, while learned vision–radar fusion compensates for modality-specific weaknesses in rain, fog, and glare. The recurring bottlenecks here are cross-sensor spatial/temporal synchronization and fault tolerance, which motivate Bayesian uncertainty weighting and online-adaptive fusion that recalibrates as conditions change.

3D mapping and reconstruction supply the persistent representation that localization matches against, turning LiDAR and stereo streams into metric models; for the place-recognition and loop-closure that keep maps globally consistent, transformer descriptors such as SeqOT [2431] match sequential LiDAR scans faster than the sensor frame rate, trained on richly annotated corpora such as ApolloScape [660]. The open problems are the cost of processing dense point clouds at city scale and the handling of occluded or incomplete returns, which favor incremental map updating over full re-mapping. When the map is unavailable, SLAM solves localization and mapping jointly, which is critical in GPS-denied corridors; modern driving-oriented vSLAM hybridizes classical geometry with learned features (for example, extending ORB-SLAM [393] with learned features, semantic labels, and learned depth to favor stable landmarks) and recovers from drift through loop closure with bundle adjustment.

These capabilities are evaluated on KITTI Odometry [173], Oxford RobotCar [1242], and ApolloScape [660] for long-horizon SLAM and place recognition, on nuScenes [445] and Waymo [446] for map-relative pose, and on Argoverse [2255] for the trajectory forecasting that localization feeds, e.g., , with the Motion Transformer [2247] (Table 9.2). The standard metrics, i.e., , Absolute and Relative Pose Error (ATE/RPE), recall@$N$ for loop closure, and minADE/minFDE for forecasting, are reported under GPS-denied, adverse-weather, and sensor-degradation conditions to stress exactly the robustness the fusion machinery is built to provide.

Decision making and control is the action end of the pipeline: it consumes the scene model from perception and the pose from localization, predicts how the scene will evolve, and commits to the steering, acceleration, and braking that close the loop, drawing on the spatial-reasoning and planning paradigms of Chapter 7. As Figure 9.10 indicates, two architectural philosophies dominate, differing in how much of the perceive-to-act mapping is learned end-to-end versus decomposed into interpretable modules, and they are increasingly converging.

The first philosophy is learning-based decision making, centered on (deep) reinforcement learning, which acquires driving policies through trial-and-error interaction and is well suited to the sequential, interactive nature of driving. Representative methods illustrate its design axes: the stochastic latent actor-critic (SLAC) [2432] learns compact latent states from high-dimensional images, PPO [1413] within a nonlinear control framework handles sharp turns and obstacle avoidance, and the DeepSet-Q architecture [2433], building on permutation-invariant set encoders [1372], ingests a variable number of surrounding agents for high-level lane-change decisions. Hybrids with explicit planning, such as Monte Carlo tree search guided by DRL or scene-representation transformers that forecast neighbor intentions, improve sample efficiency and situational awareness in dense traffic. The binding difficulty is that safety-critical episodes are rare in logged data, which pushes the field toward high-fidelity simulation for rare-event coverage and toward encoding hard safety constraints directly in the policy rather than trusting the reward signal alone. The second philosophy, end-to-end decision making, learns the mapping from sensors to trajectory or control directly, removing hand-designed interfaces between modules. Early instances learned state-to-action policies with DDPG [2434] in simulators such as TORCS, and multi-task backbones such as HybridNets [2435] share computation across detection and segmentation without yet closing the loop to control. Current systems are trained by imitation on large driving logs: TransFuser [2436] fuses camera and LiDAR features with transformer attention and predicts waypoints for a closed-loop agent, and UniAD [2437] chains tracking, mapping, motion forecasting, and occupancy prediction through query interfaces inside one network optimized for planning. Driving world models such as GAIA-1 [2330] generate future multi-view scenes conditioned on ego actions, giving such policies a learned simulator for training and evaluation. Opacity is the price of end-to-end training: attribution methods such as layer-wise relevance propagation (LRP) [2438] support interpretation but do not by themselves establish safety or regulatory compliance, which is why hybrid designs keep inspectable scene or trajectory intermediates together with explicit constraint checks, whose guarantees extend only as far as the modeled constraints.

Closed-loop driving policies are evaluated chiefly in simulation, i.e., , the CARLA urban simulator [2326] (NoCrash, Longest6), the nuPlan planning benchmark [2439], and highway-interaction simulators built on HighD/INTERACTION (Table 9.2), because rare, safety-critical events cannot be exercised safely on public roads. The headline metrics are the CARLA Driving Score (route completion penalized by infractions), collision and intervention rates, and route completion, complemented by comfort proxies such as jerk and, for the open-loop forecasting sub-problem, minADE/minFDE.

The systems on the road today populate a deployment ladder whose rungs are the SAE levels of Figure 9.6, but what separates one rung from the next is rarely a single breakthrough in perception accuracy. It is, instead, two coupled engineering choices: how wide an Operational Design Domain (ODD) the system claims, and where legal responsibility for the driving task sits. Three tiers are in genuine production, i.e., , supervised Level 2 assistance, certified Level 3 “eyes-off” automation, and driverless Level 4 robotaxis, and, as the following progression shows, moving up the ladder consistently means adding sensing and compute redundancy while narrowing the domain in which the autonomy guarantee holds.

Every tier of the deployment ladder works because it limits where it operates, and general, trustworthy autonomy requires lifting those limits. Four threads, each cutting across perception, localization, and decision, stand in the way: justifying life-critical decisions (safety and explainability), dependable behavior across the long tail of road, weather, and traffic conditions (resilience and robustness), adaptation to genuinely novel situations from little prior data (unfamiliar road scenarios), and continual learning on deployed vehicles without eroding mastered behavior (enhanced autonomy in complex scenarios).