9.1Robotic Manipulation

Robotic manipulation has advanced rapidly, driven by strides in perception, learning, and control. Yet a significant gap remains between competent laboratory systems and robust, general-purpose robots capable of operating in unstructured settings. A central bottleneck is that manipulation is fundamentally spatial: every decision (where to reach, how to establish contact, how to route a trajectory through clutter, or how to use a tool) hinges on perceiving, representing, and reasoning about three-dimensional structure under uncertainty. Manipulation is thus a direct test of the spatial intelligence developed throughout Parts I–III: it demands that systems go beyond semantic recognition to build actionable 3D world models, ground tasks in geometric constraints, predict contact outcomes, and plan behaviors that respect the physical laws of the environment. The perception, reasoning, and generation capabilities that such systems presuppose are surveyed in Chapter 6, Chapter 7, and Chapter 8; this section focuses not on re-deriving those capabilities but on how they are composed, grounded, and constrained for physical manipulation.

A manipulation system maps raw multimodal observations and a task instruction into executable, constraint-satisfying actions, and this section organizes that mapping into four strategic pivots, each a fundamental shift in how systems bridge perception and physical action. (i) Grounding Action in Spatial Anchors moves beyond raw pixels to vision priors and sparse spatial tokens (keypoints, dense correspondences) that stay consistent across viewpoints and instances; (ii) Inferring Functional Action Possibilities shifts from object classification to extracting affordances and tool-use capabilities, bridging perception with contact-rich execution; (iii) Encoding Spatial Reasoning in Generalist Models transitions from single-task policies to foundation models (VLMs/VLAs) that integrate explicit 3D representations and reasoning traces for cross-embodiment generalization; and (iv) Satisfying Geometric and Logical Constraints advances from simple trajectory generation to constrained optimization and neuro-symbolic planning (TAMP), ensuring actions respect strict safety limits and long-horizon logical structure. The four pivots are complementary and mutually reinforcing rather than strictly sequential: anchors and affordances structure the perceptual input, generalist models supply broad priors, and constraint-based planners check feasibility under the modeled geometric, physical, and logical assumptions. Figure 9.1 situates them as complementary capabilities within a closed-loop perception-to-action system, and the remainder of this section treats them in this order.

This pivot reframes visuomotor learning from operating over raw pixels to reasoning through explicit spatial anchors (sparse semantic keypoints, dense correspondences, point tracks, or affordance-based contact points) that compress high-dimensional observations into spatial tokens tied to object- or task-centric reference frames and designed to remain comparatively stable across viewpoint, instance, and clutter. As Figure 9.2 indicates, the approach follows a three-step logic: large-scale vision priors first produce the anchors, policies are then conditioned on them, and the payoff is read off as a generalization gap between seen and unseen instances.

The dominant methodology leverages large-scale vision priors to generate, propagate, or retrieve these anchors, then trains standard visuomotor learners on the resulting structured inputs. A common strategy prescribes anchor points in a few reference frames and propagates them across large datasets via off-the-shelf correspondence models: P3-PO [2369] propagates manually specified keypoints using dense features from DINO [2370], DINOv2 [958], or diffusion-based representations [2371, 2372, 2373] (these visual representations are surveyed in Chapter 4). Relatedly, Robo-ABC [2374] retrieves affordance contact points from an external memory and transfers them to novel scenes via dense semantic correspondence, while Track2Act [2375] learns long-term point tracks from Internet-scale video and uses them to infer object-centric transformations that serve as open-loop manipulation plans. Once anchors are obtained, policy learning conditions action prediction directly on them, through keypoint-conditioned behavior cloning (KALM [2376]), diffusion policies in object-relative frames (AffordDP [2377]), and shared human–robot 3D keypoint spaces (Point Policy [2378]). Across these variants, anchoring actions to spatial structure reduces perceptual ambiguity and improves robustness under distribution shift.

Because the central claim is out-of-distribution generalization, evaluation pairs high-fidelity simulation with real-robot deployment rather than relying on either alone. In simulation, affordance-centric methods such as AffordDP [2377] run on IsaacGym [2379] over articulated objects from GAPartnet [2380] whose functional parts (handles, knobs, levers) vary widely in geometry, while imitation suites such as Robomimic [2381] and RoboTwin [224] isolate trajectory tracking and dual-arm coordination, with P3-PO [2369] using them for controlled spatial-anchoring ablations. Real-world suites then stress in-the-wild generalization on articulated (drawers, microwaves), deformable (towel folding), and tool-use (sweeping) tasks, as in Point Policy [2378] and Track2Act [2375], and a zero-shot cross-category protocol (train on one knife, test on spatulas or novel knives) is formalized by Robo-ABC [2374] to measure how well retrieved anchors transfer without extra supervision. Performance is reported as Task Success Rate and, to isolate the contribution of anchoring, a Generalization Gap between seen and unseen instances. Spatially grounded policies outperform pixel baselines under shift in these studies: P3-PO reports large gains on cluttered, novel-object scenes [2369], and Robo-ABC improves both affordance-retrieval accuracy and real-robot success without heavy robot-data collection, binding demonstrations to reusable spatial structure more effectively than classic one-shot imitation [2382].

This pivot shifts from general object recognition to extracting object affordances, i.e., , the specific action possibilities implied by an object’s functional properties, and uses them as the intermediate representation that bridges raw perception and contact-rich execution. Whereas Pivotal Task-1 treats contact points as viewpoint-stable anchors for conditioning a policy, affordances here are studied as functional possibilities that specify what action an object supports. As Figure 9.3 shows, the thread runs from how affordances are represented, through the actions they drive (grasping, placement, tool use), to outcome-oriented benchmarks that score achieved function rather than imitated trajectory.

To capture functional properties beyond whole-object categorization, prior work builds object-centric representations (part-level segmentation [2383], dense visual descriptors [335], and keypoint-based encodings [547, 2384, 337]) that directly support action synthesis, and more recent vision–language models predict spatial affordance points directly from open-vocabulary prompts (RoboPoint [1494], MOKA [2385]). These representations drive learning-based 6-DoF grasping [2386, 2372, 2387, 2275, 2388, 2389] and stable object placement [2390, 2391, 267, 2388, 2392] by grounding generation in functional structure rather than raw geometry, and a specialized strand further models tool–behavior relationships, either learning tool–action–effect pairs [2393] or predicting specific affordances such as pulling [2394], so as to infer end-effector poses that achieve the intended functional outcome.

Benchmarks for affordance methods emphasize understanding object functionality over geometric matching: the property under test is functional generalization, not trajectory imitation. For grasping, YCB-based real-world evaluations [1184], GraspNet-1Billion [1183], and simulation suites such as Dex-Net [336] score 6-DoF stability and contact validity across novel instances and viewpoints; placement benchmarks such as RLBench [1346] and ManiSkill [2395] test stable stacking and insertion in cluttered, articulated scenes, and M2T2 [2388] explicitly evaluates valid placement poses conditioned on object geometry and scene context. For articulated objects and tools, Where2Act [337] and PartNet-Mobility [2396] score the prediction of actionable parts and interaction parameters for pulling, pushing, and rotating, while tool-use tasks judge whether the inferred end-effector pose produces the intended causal effect (drawing, hooking, pulling) rather than reproducing a specific trajectory. Evaluation therefore centers on outcome-oriented success: stable grasps that lift and hold unseen objects under pose, scale, and appearance variation; placement success, pose accuracy, and post-release stability under clutter and occlusion; and, for articulated and tool interactions, whether the predicted parameters yield the desired functional result.

This pivot moves from specialized, single-task policies toward generalist robotic foundation models. Existing Vision–Language–Action (VLA) models such as RT-2 [97] and OpenVLA [99] show strong semantic understanding but often lack the spatial common sense needed for precise manipulation or robust generalization to novel viewpoints, so the central challenge is to strengthen spatial grounding during pre-training, enabling explicit reasoning about 3D geometry and generalization across environments and embodiments [229]. The general foundation-model and spatial-reasoning machinery these policies inherit is treated in Chapter 7; here we examine only its manipulation-specific adaptation, which Figure 9.4 groups into three families: explicit 3D representations, view-invariant representations, and grounded reasoning traces.

The first family builds explicit 3D representations: models such as 3D-LLM [1141] and 3D-VLA [201] bridge point clouds and large language models, with 3D-VLA adding a 3D generative pre-training objective that lets the model reason about action outcomes in a consistent geometric space, effectively “imagining” future states. The second groups view canonicalization and spatial action encoding as distinct mechanisms: OG-VLA [2397] projects multi-view observations into a point cloud and re-renders canonical orthographic views to reduce sensitivity to camera placement, while SpatialVLA [2398] uses Ego3D position encoding for observations and Adaptive Action Grids to discretize spatial movement actions; adapting to new setups can involve re-discretizing the grids. The third injects spatial reasoning through grounded reasoning traces: “thinking” models such as Emma-X [2399], MolmoAct series [2400, 2401], closely related to TraceVLA [1485], replace direct pixel-to-action regression with intermediate, grounded outputs (2D motion trajectories or relative depth maps) that explicitly condition the final action prediction.

These approaches are evaluated under method-specific zero-shot and adaptation protocols across embodiments, viewpoints, and environments, with key metrics including multimodal goal-generation quality, robustness to extrinsic camera perturbations, and skill transfer across robot morphologies such as from a fixed-base Franka arm to a mobile manipulator [1577, 2402]. The reported gains point the same way: canonical orthographic views keep grounding consistent under viewpoint change (OG-VLA), spatially aligned action grids share 3D structure across setups through discretized movement (SpatialVLA), and intermediate visual or depth-aware reasoning traces act as a regularizer on action prediction (Emma-X, MolmoAct).

This pivot moves planning from heuristic trajectory generation toward formal constrained optimization that enforces geometric feasibility, fundamental physical laws (non-penetration, friction cones, force limits), and high-level logical task structure. By casting manipulation as multi-objective optimization, these methods seek to satisfy modeled hard constraints while optimizing soft task objectives, spanning both high-frequency control and long-horizon decision-making. Figure 9.5 organizes the thread along three fronts (symbolic–geometric planning, contact and trajectory optimization, and constraint-aware generative planning) before synthesizing the roles constraints play in learning-based systems.

Methodological progress has moved from purely symbolic planners toward neuro-symbolic, learning-augmented optimization frameworks that unify discrete logic with continuous physics. Classical Task and Motion Planning (TAMP) and Logic-Geometric Programming (LGP) [1267] provide the foundation by jointly reasoning over symbolic task structure and geometric feasibility, and recent systems augment them with Large Language Models and Vision-Language Models for scalability in open-ended domains; for example, VIZ-COAST [2403] uses the spatial reasoning of VLMs to predict refinement failures before expensive geometric optimization, substantially reducing search complexity on long-horizon problems. At the level of physical interaction, contact-implicit optimization discovers contact schedules without explicit mode enumeration but is traditionally too slow for real-time control; GPU-parallelized systems such as SPaSM [2404] compile constraint evaluation and gradient computation into CUDA kernels, accelerating joint optimization of object placements and robot trajectories. Beyond deterministic constraints, learning-based and generative models capture uncertainty and complex structure: differentiable physics engines [2405] let gradients flow through physical interactions, stochastic implicit neural signed distance functions [2406] model geometric uncertainty directly within the planning manifold, and diffusion-based planners such as JM2D [2407] couple model-free diffusion with model-based constraint optimization during sampling, rather than relying only on a sequential plan-then-filter interface.

Such systems are benchmarked on multi-stage re-grasping under uncertainty, forceful tool use, and tight-tolerance sequential interaction, since what they promise is reliability over long, contact-rich horizons; suites such as the Functional Manipulation Benchmark (FMB) [2408] scoring not only task completion but constraint satisfaction under partial observability, external disturbance, and compounding execution error. In these systems constrained optimization plays three roles: it acts as a teacher, generating physically valid trajectories that supervise imitation and offline reinforcement learning; as a safety shield, where model-based projections map unconstrained neural or diffusion policy outputs onto constraint-satisfying sets at execution time [2409]; and as a structural prior that prunes physically infeasible regions of the action space and curbs compounding error in long-horizon tasks. Together these roles keep learned and generative planners grounded in geometric and physical reality, reducing physically implausible outputs and supporting interpretable, feasible plans under the modeled assumptions; real-world execution still depends on state estimation, model accuracy, solver convergence, and disturbance handling.

Table 9.1 consolidates the principal benchmarks, representative methods, and headline evaluation focus across the four pivots, situating the spatially grounded methods surveyed above within a common evaluation landscape.

Progress in spatially grounded manipulation has been inseparable from a maturing ecosystem of simulators, open software stacks, shared datasets, and, increasingly, commercial robot platforms that serve as common development and deployment substrates.

Simulation and physics engines. GPU-parallel engines such as NVIDIA IsaacGym and the newer Isaac Sim / Isaac Lab¹ [2379], articulated-object simulators such as SAPIEN [2396] (with its PartNet-Mobility assets), and general-purpose physics engines including MuJoCo [1187], PyBullet², and the recent Genesis³ provide high-throughput, contact-rich environments, while ManiSkill [2395], RLBench [1346], Robomimic [2381], and RoboTwin [224] package standardized task suites, demonstrations, and evaluation protocols that have become de facto testbeds for imitation and policy learning.

Open software stacks and datasets. On the software side, the Robot Operating System (ROS 2) with the MoveIt motion-planning framework⁴ remains the de facto middleware for real-robot integration, and Hugging Face’s LeRobot⁵ packages datasets, pretrained policies, and training pipelines for low-cost reproduction. Large cross-embodiment efforts such as Open X-Embodiment⁶, which aggregates demonstrations and the RT-X policies across dozens of robot platforms, together with open generalist checkpoints such as Octo [1577] and OpenVLA [99] and NVIDIA’s GR00T humanoid foundation models⁷, expose pretrained weights and fine-tuning interfaces that lower the barrier to transferring foundation-model policies onto new morphologies.

Commercial and humanoid platforms. On the deployment side, a rapidly growing set of companies is moving spatially grounded manipulation out of the laboratory. Boston Dynamics⁸ (the electric Atlas and the Stretch warehouse robot), Tesla Optimus⁹, Figure AI¹⁰, 1X Technologies¹¹, Agility Robotics¹² (the Digit humanoid, piloted in logistics), Physical Intelligence¹³ (the $\pi_{0}$ generalist policy), and Sanctuary AI¹⁴ each pair learned manipulation with humanoid or mobile-manipulator hardware, while firms such as Covariant¹⁵ target warehouse pick-and-place at commercial scale. These platforms are the clearest evidence that the spatial-intelligence capabilities surveyed above are crossing from benchmark tasks into economically deployed systems.

Together these tools convert the representation and reasoning machinery of Chapter 7 into reproducible, physically grounded manipulation pipelines; their principal limitation remains the fidelity gap between simulated and real contact dynamics, which the following discussion revisits.

Despite rapid progress across all four pivots, bringing spatial intelligence to robust, general-purpose manipulation remains bounded by a set of intertwined difficulties that cut across perception, policy, and planning. They cluster into four threads: keeping a persistent and calibrated internal model of the scene; closing the loop between contact physics and active sensing; transferring skills across embodiments while sourcing the data to do so; and meeting the reliability budget of a deployed system.

Persistent, calibrated world models. The first requirement is a persistent 3D belief: an actionable, temporally consistent world model that survives heavy occlusion, articulation, deformation, and sensor drift, and that is maintained at control rates rather than reconstructed from scratch each frame. Persistence alone is not enough if the model cannot say how much it trusts itself. Calibrated uncertainty asks that distributional beliefs over geometry, contact state, friction, and articulation propagate through perception, policy, and planner, so that a decision reflects confidence rather than a bare point prediction and a low-confidence grasp can trigger caution instead of failure.

Contact modeling, sensing, and active perception. Manipulation is ultimately about contact, yet most spatial models still reason about collision-free geometry rather than the physics of touch. Progress on contact modeling and sensing hinges on tightly integrating vision with force and tactile feedback to reason about stick–slip, compliance, and incipient failure. Because contact is often unobservable until it happens, this couples naturally to information-seeking control: interleaving manipulation with active perception and probing (touch-to-disambiguate, look-while-acting) inside tractable belief-space planners. And because contact is precisely where simulators diverge from reality, sim-to-real for contact stays open, demanding better material, friction, and compliance models together with fast system identification for on-robot adaptation.

Cross-embodiment transfer and data at scale. Skills learned on one platform rarely move cleanly to another. Cross-embodiment and cross-scene transfer requires aligning action spaces and spatial tokens across grippers, cameras, and environments without hand-tuned frames or retraining. Sustaining that transfer is a data problem as much as an algorithmic one: data and evaluation at scale calls for “long-tail” data engines that cover rare geometries, materials, and tools with counterfactual and safety labels, and for benchmarks that measure calibration, persistence, recovery, and long-horizon competence rather than success rate alone.

Systems reliability. Finally, a deployed manipulator must hold together as a system: meeting latency budgets, absorbing calibration drift, detecting out-of-distribution geometry, and enabling safe fallback and continual learning without catastrophic forgetting. Closing these gaps will require tighter couplings among 3D world models, affordance predictors, language-grounded objectives, and constraint-aware planners, so that spatial representations are not only expressive but actionable, calibrated, and safe in unstructured settings.