13.1Assistant

An intelligent assistant is a generalist embodied agent that shares the user’s own private, unstructured world and acts on the user’s behalf. Unlike the domain agents of the preceding sections, which are optimized for a bounded operational envelope (a road network, a warehouse aisle, an operating theater), the assistant must function inside the open-ended geometry of a home, an office, or a sidewalk, in the continuous physical presence of the person it serves. Its defining spatial challenge is therefore not a new perception, reasoning, generation, or memory primitive: those capabilities are surveyed in Chapter 6 (perception), Chapter 7 (reasoning), Chapter 8 (generation), and Section 7.3.6 (retrieval- and memory-augmented spatial reasoning). The challenge here is rather to build and maintain a persistent, personalized, and queryable spatial-semantic model of the user’s environment, to ground human intent onto that model, and to act on it legibly and safely while a person occupies the same space.

Three properties separate the assistant from the embodied agents discussed elsewhere in this part. The environment is the user’s own, so the assistant cannot assume an engineered layout and must instead generalize across the long tail of real homes. It is also personalized: the same instruction (“tidy up,” “take me to the exit”) resolves to different geometric goals for different users, so the spatial model must absorb individual preference. And because the human is always co-present, otherwise routine motion becomes a social act that must be predictable, comfortable, and safe. These three demands (an unengineered space, an individual user, and a person sharing that space) recur across every regime in which the assistant operates.

Across these settings, every assistant closes the same loop. It perceives the environment and the user, represents what it perceives in a persistent spatial-semantic memory, grounds a natural-language or gestural instruction onto that memory, reasons about a goal-directed plan, acts on the physical or informational world, and observes the human’s feedback, which re-grounds the next interaction. The persistent memory is the part that distinguishes an assistant from a single-shot perception or manipulation system: because the assistant lives in one place over long horizons, it can accumulate a map of where objects belong, where the user tends to be, and how the user prefers tasks to be performed. Long-horizon mobile-manipulation benchmarks such as EMMOE instantiate one slice of this loop (language-conditioned planning and action in open environments) but still leave persistent personalization, memory update, and human feedback as broader assistant-level requirements [2945]. Figure 13.1 schematizes the loop, with the spatial-semantic memory at its center and the human present at every stage.

The assistant is deployed in three escalating regimes of human-proximity, which organize the remainder of this section. In the first, the assistant acts on the space while the human is a beneficiary at a distance: it tidies, fetches, and searches. In the second, the assistant acts alongside the human as a partner that must move and manipulate in shared space without intruding. In the third, the assistant acts for a vulnerable human, standing in for an impaired capability under hard safety, privacy, and accessibility constraints. The three regimes share the loop above but place progressively more weight on modeling the person rather than the place; Table 13.1 collects benchmarks for all three.

The most autonomous regime asks the assistant to maintain the physical state of a human environment with little real-time supervision. A persistent model of the dwelling supports two complementary operations: grounding instructions for physical action, and retrieving remembered evidence to answer queries. New observations update the model, including after an action changes the environment. Figure 13.2 separates these action and recall branches around their shared home memory.

Representing the home. Everything downstream depends on a queryable model of the space, which extends the scene-understanding methods of Chapter 6 into a persistent, semantically indexed memory. The memory architectures themselves are treated in Section 7.3.6; here they are the infrastructure that lets an assistant know not only where surfaces are, but what and whose. The 3D scene graph [341] introduced a hierarchical structure that unifies objects, rooms, and cameras with the underlying geometry, and Hydra [1554] brought such graphs into real-time, loop-closure-optimized robot perception so that the memory stays current as the assistant moves. To make the memory addressable by free-form language, ConceptFusion [2946] builds an open-set, multimodal 3D map that answers text, image, or click queries, and ConceptGraphs [1555] fuses foundation-model outputs across views into a captioned, language-queryable 3D scene graph suitable for planning. This open-vocabulary memory is what lets an assistant act on objects it was never explicitly trained to recognize.

Acting on the space: tidy-up and rearrangement. Restoring order is the canonical management task, framed by Batra et al.[2947] as bringing an environment from an arbitrary configuration into a target spatial state. The benchmark substrate includes the Visual Room Rearrangement (RoomR) challenge [2948], which asks an agent to observe a goal arrangement and then reset one to five displaced objects, and Housekeep [2949], which removes the explicit goal and instead judges placements against human preferences over nearly two thousand objects. Method work has moved from fixed rules to learned commonsense priors. TIDEE [2950] tidies a never-before-seen room from pixels using a visuo-semantic out-of-place detector and an associative memory of where receptacles tend to be, and TidyBot [2951] makes the placement preference-aware, using few-shot summarization with a large language model to generalize a particular user’s tidying rules from a handful of examples. The shift these systems mark, from a universal notion of tidiness to one conditioned on the resident, is exactly the personalization property that defines the assistant.

Acting on the space: open-vocabulary mobile manipulation. Tidying a real home requires picking and placing arbitrary objects at arbitrary locations, the task formalized as open-vocabulary mobile manipulation by HomeRobot [2952], which pairs a simulation and real benchmark with a software stack for the Hello Robot Stretch. Notable deployment evidence comes from OK-Robot [2953], which combines a phone-scanned dense vision-language map with off-the-shelf navigation and grasping to achieve zero-shot pick-and-drop across ten real homes, and from Mobile ALOHA [2954], which shows that low-cost whole-body teleoperation plus imitation learning can acquire dexterous bimanual household skills. These systems lean directly on the manipulation primitives of Section 9.1, increasingly instantiated as generalist vision-language-action policies (e.g., , OpenVLA [99], $\pi_{0}$ [100]) that fold perception, grounding, and control into one model; the assistant contribution is to drive them in unstructured homes from open-vocabulary goals rather than in an engineered cell. Commercial and pre-commercial examples illustrate the same pressure from different directions: mapping vacuum robots such as iRobot’s Roomba, which build semantic room maps for targeted cleaning,¹ represent the mature narrow end, while emerging home-humanoid platforms such as 1X’s Neo² and Figure³ show how the field is attempting, so far without a general-purpose result, to generalize mobile manipulation into domestic settings.

Searching and recalling the model. The dual of acting on the space is querying it: an assistant is, among other things, a search engine for the physical world that answers “where did I leave my keys?” Object-goal navigation supplies the active half, with goal-oriented semantic exploration (SemExp) [1416] building an episodic semantic map and a policy that exploits object co-occurrence to find a target efficiently rather than searching blindly. Embodied question answering [1404] couples navigation with answering, and its open-vocabulary successor OpenEQA [1563] spans both an episodic-memory setting, in which the assistant must recall what it has already seen, and an active-exploration setting, over more than a hundred real environments. For wearable assistants that record the user’s day, the same recall problem becomes natural-language search of long egocentric video, the episodic-memory task defined by Ego4D [681]. This is the application-facing counterpart of the memory-augmented reasoning systems in Section 7.3.6.

As the distance between assistant and human shrinks, spatial intelligence must become social intelligence. The assistant must model the human as a co-located agent with intent, viewpoint, and comfort zones, not as a moving obstacle or a bare command source, and act predictably and collaboratively. Running through every part of this task is the reconciliation of reference frames: the assistant’s own egocentric view, the human’s egocentric view, and a shared allocentric map must be kept aligned.

Socially-aware navigation. Moving through a populated space is not collision avoidance; it is adherence to the unwritten contracts of proxemics. The classical account is the social force model [2955], which casts pedestrian motion as attraction to a goal under repulsion from others, operationalizing Hall’s anthropological zones of personal space. Data-driven successors learn the interaction directly: Social LSTM [2249] couples per-person recurrent predictors through a social-pooling layer to forecast jointly plausible trajectories, and deep reinforcement-learning planners such as the attention-based crowd-robot model [2956] weigh neighbors to navigate dense crowds while avoiding the “freezing robot” failure that purely reactive planners suffer [2957]. Progress is measured on human-demonstration corpora such as SCAND [2958], which records hours of socially compliant teleoperation for imitation and inverse reinforcement learning. The trend is toward anticipatory planning, in which the assistant infers the human’s intended path and yields proactively, and toward learnable cost functions that adapt passing distances to individual or cultural preference rather than enforcing one universal margin.

Grounded instruction following and spatial dialogue. A collaborating assistant must resolve spatial language against what it sees, including the deictic references (“put it next to the other bowl,” “the one over there”) that pervade human speech. Vision-and-language navigation, introduced with the Room-to-Room benchmark [1415], established the problem of following natural-language route instructions in photorealistic buildings; ALFRED [1307] extended it to long-horizon household tasks with irreversible state changes, and TEACh [1308] cast the task as a two-agent dialogue in which a commander and follower must build shared spatial reference over multiple turns. Resolving a referring expression to a specific object in a cluttered 3D scene is studied directly by ScanRefer [373] and ReferIt3D [374], whose same-class distractors force the model to rely on spatial relations rather than appearance. These benchmarks build on the reasoning capabilities of Chapter 7; the assistant requirement is to maintain a spatial working memory that links current observations to earlier instructions so that deixis can be resolved.

Shared autonomy and collaboration. When full autonomy is brittle, the assistant acts as a third hand under shared control. The defining formulation is shared autonomy via hindsight optimization [2959], in which the robot infers a distribution over the user’s spatial goal and assists toward it without committing to a single confident prediction, formalizing the earlier policy-blending account of arbitration between human and robot intent [2960]. Equally important is making the assistant’s own motion readable: legible motion [2639] shapes a reaching trajectory so a human can quickly infer its intended target, which is what allows close-quarters collaboration to feel safe. Physical handovers, where giver and receiver must coordinate in space and time [2961], are the elementary collaborative act on which assistive manipulation is built.

Perspective-taking and egocentric grounding. Many assistants observe the world from a wearable, first-person viewpoint, so the frame-alignment problem becomes explicit. Ego4D [681] and its successors anchor egocentric perception at scale, and dedicated capture platforms such as Project Aria⁴ supply the calibrated multi-sensor first-person stream such assistants run on, while spatial-VQA backbones such as SpatialVLM [229] give a vision-language model the quantitative 3D reasoning (distances, sizes, directions) needed to answer spatial questions about the user’s scene. The hardware and pose-estimation side of wearable sensing is treated in Section 9.3, and the augmented-reality runtime that renders guidance into the user’s view is treated in Section 10.1; here the contribution is the reasoning that converts a first-person observation into a collaborative action in the human’s frame. Embodied-AR applications make this alignment problem explicit when directions, warnings, or manipulation cues must be registered into the user’s view [2962]. Figure 13.3 illustrates the elements of this regime.

The highest-impact regime uses spatial intelligence to compensate for a sensory, cognitive, or motor impairment, acting as a prosthesis for the user’s perception or agency. The assistant translates the environment into a queryable, navigable, and actionable representation and delivers it through a channel the user can still use (audio, haptics, or physical action). Because the loop closes around a vulnerable person in their own home, safety, privacy, and personalization are design constraints from the start.

Assistance for the blind and visually impaired. For blind and low-vision users, spatial intelligence turns an unknown and hazardous environment into a queryable one. The oldest strand is sensory substitution, exemplified by the vOICe [2963], which sonifies the spatial structure of an image into sound. The modern strand is conversational visual question answering: VizWiz [2964] introduced a deployed system in which a blind user photographs their surroundings and asks a spoken question, and the VizWiz Grand Challenge [2965] turned the resulting real-world distribution, including blurry and unanswerable images, into the reference benchmark for assistive VQA. Indoor wayfinding is addressed by smartphone navigation aids such as NavCog [2966], which deliver turn-by-turn audio guidance from accurate beacon-based localization, and by three-dimensional spatial audio that places points of interest around the user. Commercial assistive applications include Microsoft’s Seeing AI for scene and text narration⁵, Be My Eyes for volunteer- and model-mediated visual assistance⁶, and wearable systems from OrCam⁷, Envision⁸, and biped.ai⁹ for scene description and obstacle warning. These tools move from passive obstacle warning to active interrogation, in which the user asks “is there a free seat?” or “read the sign on the door” and the assistant retrieves the answer.

Cognitive and mobility assistance. For users with cognitive decline or limited mobility, spatial assistants provide external scaffolding for the activities of daily living. Ambient assisted living systems such as the CASAS smart home [2967] instrument a dwelling to track the resident’s location and activity, learning the spatio-temporal patterns whose violation (a fall, a missed meal) signals a need to intervene; wearable accelerometry supplies the canonical fall-detection paradigm [2968].

Where monitoring gives way to physical action, the assistant becomes a shared-control effector. Smart wheelchairs [2969] and wheelchair-mounted assistive arms use collaborative control to enforce spatial safety zones, for example forbidding a chair from driving down a staircase, so that the user retains mobility without constant supervision; the goal-arbitration that underlies such control is the same shared-autonomy machinery introduced above, here cast as a decision-theoretic problem [2970]. Assistive feeding [2971] shows how fine spatial reasoning (where to acquire a bite, how to transfer it to the user’s mouth) becomes a caregiving act. The platform lineage runs from Robots for Humanity [2972], which let a man with quadriplegia perform self-care through a teleoperated PR2, to the affordable Stretch mobile manipulator [2973], with physics-based simulation of caregiving tasks consolidated in Assistive Gym [2974]. Commercial assistive hardware in this space includes the Kinova JACO arm¹⁰ and the Obi feeding device¹¹. This regime overlaps the clinical setting of Section 12.1, but the boundary is clear: there the focus is diagnosis and surgical intervention, whereas here it is daily living in the home. Figure 13.4 summarizes the compensation principle.

Deployed assistants have so far approached the persistent, personalized spatial model of this section from opposite ends: software agents know a great deal about the user but nothing about the user’s space, while home robots occupy the space but act with limited autonomy.

Personal agents. The most widely used assistants now act on a user’s behalf in the digital world. OpenAI’s ChatGPT agent¹² (July 2025) runs on its own virtual computer, browsing and filling in forms across multi-step tasks and asking permission before consequential actions. Meta’s Muse¹³ (September 2026), billed as a personal agent for everyday users, likewise runs in an isolated virtual machine and handles errands such as shopping and travel booking, and its request for access to email, calendars, and payments made privacy the main point of public debate around its launch. Voice assistants are being rebuilt on the same models: Amazon’s Alexa+¹⁴, generally available in the United States since February 2026, adds generative dialogue to the smart-home control that Alexa already provided, and Apple’s Siri¹⁵ is being rebuilt on Google’s Gemini models, adding personal context and on-screen awareness. For spatial intelligence, these agents matter mainly as the layer that holds personal context and preference; apart from smart-home control, none of them maintains a model of the physical home.

Camera-grounded assistants. A second line brings the physical world into the conversation through a camera. Gemini Live’s camera mode¹⁶, the product form of Google’s Project Astra, lets a user point a phone at the surroundings and ask about what it sees, and Meta AI on camera glasses does the same from the wearer’s viewpoint (Section 9.3). These systems answer spatial questions about the current view, and the assistive applications of Pivotal Task-3 build on the same capability, but they keep at most a short history of what they have seen, so the long-horizon spatial memory this section calls for remains outside them.

Home robots. Embodied assistants remain the least mature tier. Beyond mapping vacuums, the home humanoids introduced under Pivotal Task-1 are moving toward sale: 1X opened preorders for NEO in October 2025, at $20,000 or a monthly subscription, with first deliveries promised for 2026 and with remote operators teleoperating tasks that its onboard autonomy cannot yet complete, and Figure’s third-generation humanoid, Figure 03 (2025), is designed for homes and driven by its Helix vision-language-action model. The reliance on teleoperation shows how far these products are from the autonomous, open-vocabulary mobile manipulation of Pivotal Task-1, and it raises the in-home privacy question discussed below.

The three regimes share a common obstacle: the assistant must be reliable in the one setting that resists engineering, namely a real person’s private and unstructured life. The challenges below are ordered from the perceptual frontier (generalization, memory) through the human frontier (personalization, social safety) to the deployment frontier (privacy, compute, trust), and each cuts across all three pivotal tasks above. Governance benchmarks for embodied agents have begun to measure the deployment end directly, testing whether a system stays controllable, recoverable, and auditable under perturbation rather than only whether it completes tasks [2975].

Generalization to unstructured homes. An assistant cannot be retrained per household, yet homes vary without bound in layout, lighting, and contents. Zero-shot results such as OK-Robot [2953] are encouraging but remain far from the reliability a daily aide requires, and closing the gap depends on the open-vocabulary representations and sim-to-real transfer surveyed in Part III rather than on more in-domain data.

Long-horizon spatial memory. Because the assistant persists in one place, it must remember spatial state over days and weeks: where objects were last seen, which routines the user follows, what was already done. Real-time scene-graph memories [1554] and episodic-memory benchmarks [1563] begin to address this, but the general memory mechanisms are treated in Section 7.3.6; the assistant-specific unsolved problem is maintaining a consistent, updatable, and forgetting-resistant model over long deployments in a private home.

Personalization and preference learning. The assistant’s geometric goals are user-specific, so it must learn an individual’s placement rules, comfort distances, and routines from sparse interaction. Few-shot preference summarization [2951] and learnable social cost functions are early steps, but acquiring rich preferences without burdening the user remains an open problem.

Social compliance and safety. Operating in continuous contact with people turns motion into a social and safety-critical act. Anticipatory, legible navigation and motion [2957, 2639] and shared-control safety zones [2970] mitigate risk, yet guaranteeing comfort and physical safety across the diversity of real human behavior is far from settled.

Privacy of persistent in-home perception. An assistant that continuously perceives a private home raises acute privacy concerns, the more so for assistive cameras worn by users who cannot themselves audit what is captured. The concern becomes sharper when assistance depends on remote operation or cloud annotation, because the raw spatial stream itself may reveal the user’s routines and vulnerabilities. Persistent spatial memory must be reconciled with selective perception, on-device processing, and user control over what is retained.

On-device compute and trust. Wearable and home platforms are power- and thermally constrained, forcing the heavy perception and reasoning stacks of Part III onto efficient, often on-device, inference. Compounding this, assistance for vulnerable users is safety-critical, so calibrated reliability and appropriate user trust, neither over- nor under-reliance, are prerequisites for deployment rather than refinements of it.