9.3Wearable AI System

Wearable devices have become a central sensing modality for embodied and spatial intelligence, delivering continuous, fine-grained measurements of human motion, physiology, and physical interaction. Because they are worn on or integrated into the body, they access proprioceptive and kinesthetic cues, joint rotations, muscle activations, contact forces, and locomotion patterns, which exteroceptive sensors such as RGB-D cameras or LiDAR cannot capture directly. We use the term broadly, covering inertial measurement units (IMUs), surface electromyography (EMG) sensors, smart textiles, haptic interfaces, and wearable cameras: body-mounted technologies that together model human dynamics and embodied interaction in real-world settings. It is this egocentric, body-centric vantage point that makes wearables valuable for spatial intelligence, letting a system reason about how a person moves through, reads, and acts upon their surroundings, and it has driven rapid adoption across health monitoring, sports analytics, rehabilitation [2440], industrial safety, and human–robot collaboration.

The signals wearables provide are especially valuable because they are anticipatory. Joint kinematics and muscle activity reveal what a person is doing and, crucially, what they are about to do next, which is what underpins intent recognition, action forecasting, and safe coordination in shared spaces. Wearable IMU arrays, for example, recover whole-body skeletal motion in cluttered or occluded settings where optical capture breaks down [2441, 2442], and EMG surfaces pre-activation patterns that precede any visible movement [2443].

These devices are often bracketed with head-mounted XR hardware, yet their role in spatial intelligence differs in kind. XR headsets integrate sensing and display for immersive visualization and mixed-reality interaction, leaning on outward-facing cameras for SLAM and environment mapping. The wearables treated here look inward instead, supplying the biomechanical and physiological state, high-frequency, continuous, and on-body, that an XR headset does not observe.

The spatial-intelligence capabilities that wearables ultimately serve, pose estimation, action recognition, localization, and trajectory prediction, are formulated in Part III (spatial perception in Chapter 6, spatial reasoning in Chapter 7). This section does not re-derive them; it concentrates on the body-centric sensing layer, the devices that acquire embodied signals, the systems assembled around them, and the tasks, methods, and failure modes that surface once spatial intelligence must be grounded in continuous on-body measurement, spanning long-horizon activity and gait analysis, intention prediction for human–robot collaboration, and real-time safety in industrial and logistics settings.

Body-centric sensing spans a diverse device ecosystem, summarized in Table 9.3.

Taken together, the devices above form the front end of a single body-centric perception-to-action loop that defines the wearable AI system as a whole. Raw on-body signals are first sensed (inertial, neuromuscular, tactile, and egocentric-visual streams), then modeled into geometric and biomechanical state (joint kinematics, body pose, contact forces), interpreted and predicted into higher-level behavioral and intentional descriptions (activities, gestures, short-horizon motion forecasts), and finally fed back to the wearer or to a collaborating agent through action channels (haptic cues, exoskeleton assistance, safety alerts, or shared-control commands). The loop then closes: the consequences of feedback alter the wearer’s motion, which the sensors observe anew.

Figure 9.11 schematizes this loop and situates the three pivotal tasks that organize the remainder of this section as complementary stages of it. Pivotal Task-1 (Embodied Motion Sensing and Pose Reconstruction) occupies the sense-to-model stage, turning body-centric signals into stable geometric descriptions of motion. Pivotal Task-2 (Behavioral Understanding and Intention Modeling) occupies the interpret-and-predict stage, mapping reconstructed motion and neuromuscular cues to recognized activities and anticipated actions. Pivotal Task-3 (Safe Collaboration and Embodied Interaction) closes the loop at the action stage, converting interpreted human state into coordinated, safe, and assistive responses in shared physical environments. Because all three stages must run under the power, compute, and reliability constraints of body-worn hardware, lightweight on-device inference is a cross-cutting concern that recurs throughout.

The first stage of the wearable loop turns raw body-centric signals into a geometric description of how the wearer moves. It is a bottom-up reconstruction problem: per-segment inertial and biomechanical measurements must yield joint angles, segment velocities, and ultimately a full-body pose that remains stable over long horizons and in settings where exteroceptive capture fails. This task subsumes full-body skeletal pose estimation, fine-grained limb tracking, real-world locomotion and gait analysis, and body-centric localization in GPS-denied spaces; the underlying pose-estimation formulations are developed in Chapter 6, so the emphasis here is on reconstruction from on-body sensing. Figure 9.12 illustrates this reconstruction stack from segment-level signals to full-body pose.

From signals to kinematics. Reconstruction begins with kinematic and dynamic modeling, which provides the physical grounding for interpreting inertial and pressure-based signals. Forward and inverse kinematics infer joint angles and segment orientations from inertial measurements, compensating for drift and sensor noise through optimization-based filtering and biomechanical constraints. Classical recursive estimators such as extended and unscented Kalman filters incorporate kinematic priors and temporal smoothness to improve stability in real-world settings [164, 2451], while articulated-body dynamics and parametric body models such as SMPL [338] enforce anatomical plausibility for complex motions such as rapid locomotion or limb coordination. Multi-IMU fusion against musculoskeletal models extends these ideas to full-body pose estimation in challenging environments where optical motion-capture systems fail [2441, 2444, 2452], and recent systems fold in physics-based dynamics and inter-sensor ultra-wideband ranging to tighten accuracy and suppress jitter [2453].

Learning-based reconstruction and sensor fusion. Where analytic models struggle with sparse or noisy sensing, data-driven approaches learn the mapping from raw inertial streams to pose directly [2441, 2444], and sensor fusion integrates heterogeneous modalities for drift-compensated estimates. IMU-and-vision fusion resolves ambiguities in inertial tracking by leveraging environmental context from egocentric cameras [681, 2448], while drift-compensation techniques, including zero-velocity updates and constraint-based corrections, stabilize long-horizon wear. For body-centric localization, sparse-sensor systems couple inertial pose estimation with simultaneous localization and mapping, so that the wearer’s global trajectory is recovered together with body pose rather than left to drift freely [2454]; visual-inertial odometry that fuses the same inertial signals with first-person imagery offers a complementary route through GPS-denied indoor spaces, assessed on dedicated benchmarks [935].

Benchmarks and evaluation. The kinematic, learning-based, and fusion methods above are measured against the same yardsticks: large-scale motion-capture corpora such as AMASS [2209], which provides a canonical surface-and-skeleton reference for full-body pose, alongside dedicated inertial-pose datasets for sparse-IMU reconstruction [2445, 2441], while body-centric localization is assessed separately on visual-inertial odometry benchmarks [935]. Reconstruction quality is reported as mean per-joint position error (MPJPE) and joint-angle error, and localization as absolute trajectory error and accumulated drift (Table 9.4). Read against the needs of real deployment, these numbers reveal a clear gap. On curated benchmarks, sparse-IMU systems now recover root-relative full-body pose to within roughly five centimeters of optical ground truth [2452], but global translation still drifts over long sequences unless it is anchored by SLAM or ranging [2454], and even the root-relative accuracy falls off in the wild, where fast or unseen motions and loose sensor placement push errors past the tolerance that downstream control or clinical gait analysis can absorb. The pose the next stage consumes is thus dependable on average yet brittle at exactly the fast, atypical movements that matter most.

Once motion has been reconstructed, the second stage of the loop interprets it: recognizing what the wearer is doing and anticipating what they will do next. Continuous neuromuscular and inertial cues feed activity recognition and segmentation and then short-horizon intention prediction, even in daily-life settings where visual systems face severe occlusion or field-of-view limits. Because the recognition and reasoning formulations themselves are treated in Chapter 7, the focus here is on how high-frequency wearable streams are modeled to support behavioral inference. Figure 9.13 sketches the path from multimodal time-series and body-graph representations to recognized activity and predicted intent.

Sequence models for wearable time-series. The availability of large-scale wearable recordings has driven deep models tailored to high-frequency temporal streams from IMU, EMG, and multimodal sensors. Recurrent networks, temporal convolutional networks, and Transformers are widely applied to gait classification, gesture recognition, and activity segmentation [2455]: temporal convolutions capture multi-scale dependencies for long-horizon pattern detection [2456], while attention-based architectures learn complex correlations across sensor channels [141]. Because labeled wearable data are scarce and expensive, self-supervised pretraining on unlabeled motion recordings has become the dominant route to transferable representations [2457]: masked-reconstruction encoders such as LIMU-BERT learn compact IMU features from raw streams [2458], and multimodal contrastive objectives such as IMU2CLIP align inertial signals with video and language, pointing toward wearable foundation models that generalize across tasks and users [2459].

Graph-based body representations. Because human anatomy is structured as a kinematic tree, graph neural networks offer a natural representation for wearable-derived motion [599], encoding relational dependencies between joints and limbs directly in graph space. Such formulations fuse IMU placements across body segments, propagate temporal features along skeletal edges, and remain robust to missing or corrupted sensor channels; combining graph convolution with temporal attention jointly models spatial structure and temporal evolution for activity recognition and biomechanical analysis [2460].

Neuromuscular cues and multimodal fusion for intent. Beyond recognizing observed motion, EMG-based wearables detect subtle neuromuscular activations that precede visible movement [2443], providing predictive cues for early intention estimation and responsive human-robot collaboration. Fusing EMG with inertial data sharpens fine-grained gesture and anticipatory-command decoding [2447], while fusing IMU with egocentric video grounds activity recognition in environmental context [913, 681]. A recent milestone shows how far this can scale: a surface-EMG wristband trained across more than a thousand users decodes gestures, clicks, and handwriting from neuromuscular signals alone and, critically, generalizes to new wearers without per-user calibration [2461], addressing the inter-user variability that has long limited EMG interfaces. Physiological and pressure cues additionally support inference of fatigue, balance, and abnormal gait over extended periods.

Benchmarks and evaluation. Taking the reconstructed motion of the previous stage as input, the sequence, graph, and fusion models above are evaluated on wearable human-activity-recognition corpora [2462, 2463], EMG gesture datasets [2443], and egocentric activity datasets such as Ego4D [681] and EPIC-KITCHENS [2464], with recognition reported as accuracy and macro-F1 and intention prediction as anticipation accuracy and prediction lead time (Table 9.4). Here the gap to maturity is one of generalization rather than peak score. Balanced locomotion corpora such as PAMAP2 are close to saturated, yet the same architectures score markedly lower on cluttered, null-dominated sets such as Opportunity and lose accuracy again under cross-user and cross-environment evaluation; reliable intention forecasting, meanwhile, still extends only to short, well-rehearsed horizons. This falls well short of the open-world, anticipatory understanding that the safe-collaboration stage below depends on, where the system must read unfamiliar users in unfamiliar settings.

The third stage closes the loop, converting interpreted human state into coordinated, safe, and assistive responses in shared physical environments. Contact- and force-level sensing of human–environment interaction supports real-time inference of effort and hazard, which in turn drives feedback for collaboration and shared autonomy. The collaboration and shared-control reasoning that this task builds on is developed in Chapter 7 and, for the manipulation setting specifically, in Section 9.1; here the emphasis is on the wearable feedback and safety layer. Figure 9.14 depicts this closing stage, from contact sensing to coordinated safety response.

Sensing interaction and effort. Haptic and tactile wearables capture contact forces, surface interactions, and manipulative behaviors, supplying the signals needed to model human–environment interaction and object handling [2465, 2446]. Exoskeletons equipped with torque and force sensors use these signals to infer user effort and detect hazardous conditions, adjusting assistive strategies accordingly [2450]; probabilistic fusion that integrates biomechanical priors and uncertainty estimates is central to robustness in such safety-critical settings.

Coordination, safety, and feedback. In industrial and shared-autonomy scenarios, on-body inertial and proximity sensors contribute to safety monitoring by detecting collision risks, unsafe postures, and sudden motion anomalies, while haptic channels return spatial alerts and force feedback that close the sensorimotor loop [2449]. Combined with the intention forecasts of Pivotal Task-2, these signals form the basis for predictive safety mechanisms and shared-control frameworks that anticipate human motion rather than merely reacting to it.

On-device constraints. Because feedback must be timely, the action stage is where the wearable system’s power, memory, and compute limits bite hardest. Model compression (pruning, quantization, and distillation) [2466] and edge-optimized architectures [2467] enable real-time inference on wearable microcontrollers or smartphone-class processors, and on-device adaptation mitigates inter-user variability without relying on remote servers, keeping safety-critical computation local and low-latency.

Benchmarks and evaluation. Closing the loop, the interaction-sensing, safety, and on-device methods above are assessed on human-object-interaction datasets such as HOI4D [2465] and, for assistive control, on exoskeleton and teleoperation testbeds, with metrics spanning contact- and force-estimation error, collision-avoidance and safety-violation rates, and intention-anticipation lead time (Table 9.4). Evaluation here is the least mature of the three stages: it remains fragmented across hardware-specific testbeds with few shared protocols, so reported gains are hard to compare and seldom stress real safety guarantees. Current systems perform convincingly in instrumented demonstrations, but closed-loop, certified safety under the full variability of human partners, the bar that genuine shared autonomy must clear, is still an open target rather than a measured result. The robustness, generalization, and safety gaps that recur at each stage of the loop are what the challenges below take up directly.

Commercial wearables have converged on the head and the wrist, and what separates current products is less their sensors than where perception runs and whether anything is shown back to the wearer. Full augmented-reality headsets are covered with the XR platforms in Section 10.1; here the focus is on everyday-wearable glasses, neural input bands, and the research platforms and chips behind them.

AI glasses without a display. By volume, the dominant wearable AI device is a pair of ordinary-looking glasses carrying a camera, microphones, open-ear speakers, and an IMU, with a multimodal assistant running on the paired phone or in the cloud. Meta’s line with EssilorLuxottica¹⁰ defines the category: the second-generation Ray-Ban Meta (2025) records 3K video from a 12-megapixel camera and runs up to eight hours per charge, and the Oakley Meta Vanguard adapts the design to sport with a 122-degree camera. Display-less models made up 96% of AI-glasses shipments in the first half of 2026, almost all of them Meta’s¹¹, while Xiaomi’s AI glasses (2025) and Alibaba’s Qwen-based Quark AI glasses¹² (November 2025) lead a fast-growing Chinese segment. Google and Samsung previewed Gemini-based Android XR glasses¹³ with Gentle Monster and Warby Parker in May 2026, with the first display-less models due in the fall. Spatially, these devices answer questions about what the wearer is looking at one camera frame at a time; most keep no persistent map of the surroundings, so the continuous body and scene tracking of Pivotal Tasks 1 and 2 is still largely absent from shipping products.

Display glasses and neural input. Adding a display makes the glasses an interface, and input then becomes the hard problem. Meta Ray-Ban Display¹⁴ (September 2025) puts a small monocular display, with a 20-degree field of view, in the right lens and ships with the Meta Neural Band, a surface-EMG wristband that decodes finger gestures and handwriting; it is the commercial form of the generic neuromotor interface described in Pivotal Task-2 [2461]. In waveguide display glasses, Rokid¹⁵ led shipments in the first half of 2026, ahead of Meta and Even Realities¹⁶, whose glasses favor a minimal text display over cameras.

Research platforms and on-glass compute. Much of the egocentric data behind the methods of this section was recorded on research glasses rather than consumer products. Meta’s Project Aria [2469]¹⁷ provides glasses with RGB, tracking, and eye-tracking cameras, IMUs, and spatial microphones, together with machine-perception services for trajectories, eye gaze, and hand tracking; its second generation (2025) adds a heart-rate sensor and a contact microphone in the nosepad, alongside on-device perception. The platform underlies Aria Digital Twin [2448]; Ego-Exo4D [2470], which pairs egocentric Aria recordings of skilled activities with synchronized exocentric cameras over 1,422 hours; and Nymeria [2471], which adds full-body ground-truth motion from a motion-capture suit to 300 hours of everyday activity, the kind of supervision the pose reconstruction of Pivotal Task-1 needs. Ego4D [681] gathered its 3,670 hours from several head-mounted cameras, and remains the main benchmark for egocentric activity understanding. On the compute side, Qualcomm’s Snapdragon AR1+ Gen 1¹⁸ (2025) runs a one-billion-parameter language model on the glasses without a phone, a first step toward the on-device inference that the latency and power constraints of Pivotal Task-3 call for.

Wearable sensing is coupled to the dynamics and variability of the human body in a way that external sensing is not, and that coupling is the source of its hardest problems. The four below track the loop of the System Overview: signal quality at the sense stage, robustness under partial observation, fusion across heterogeneous modalities, and generalization across users.