12.1Medicine and Clinical Healthcare

Anatomy is a three-dimensional arrangement of organs with shape, volume, and topology; disease is localized, so that a lesion is a position and an extent in the body frame rather than a label on an image; and intervention is fundamentally an act of navigation, guiding an instrument to a target through tissue that must not be harmed.

The perception machinery of Chapter 6 (detection, segmentation, depth, pose) and the reconstruction and generation of Chapter 8 reappear here, specialized to volumetric data, to registration across modalities and time, and to a regulatory setting in which a wrong answer harms a patient. The unit of analysis is the individual patient and the individual procedure: the molecular and cellular scale belongs to the natural sciences (§12.2), the generic digital twin, of which the patient-specific medical twin below is a specialization, to §11.1, and the population scale of spatial epidemiology to the GeoAI of §11.2.

Figure 12.1 draws clinical spatial intelligence as a closed loop around the patient: multimodal imaging (computed tomography, magnetic resonance, ultrasound, endoscopy, pathology) is acquired and the anatomy perceived, a patient-specific 3D model is built and a plan formed, the plan is executed under image guidance, and the outcome is imaged anew. Registration connects the image, patient, and instrument frames throughout this pipeline; the recurring difficulty is keeping the virtual model aligned with the physical patient as tissue moves and deforms. The three pivotal tasks map onto the loop directly (Task-1 perceives, Task-2 models and plans, Task-3 acts), and Table 12.1 collects their benchmarks.

The first task perceives the anatomy. Medical perception is intrinsically volumetric: it converts a voxel grid into delineated 3D anatomical objects with measurable shape and volume, localizes disease in the body frame, and aligns observations across modalities and time. Figure 12.2 illustrates three complementary outputs: segmented organs, a localized candidate finding, and a cross-modal alignment; these tasks need not form a serial pipeline.

Volumetric segmentation. Segmentation is the spatial core of medical imaging, and its history is a move from two-dimensional slices to full volumes. The encoder-decoder U-Net [174] founded the area, and its volumetric successors, the 3D U-Net [268] and the V-Net [2835] (which introduced the now-ubiquitous Dice loss to handle voxel class imbalance), moved it into three dimensions. The self-configuring nnU-Net [2836] adapts preprocessing, architecture, and training to each dataset and remains the baseline that is hard to beat, while transformer encoders such as UNETR [2837] and Swin-UNETR [2838] add global context. At the whole-body scale, TotalSegmentator [2839] delineates more than a hundred anatomical structures in a single CT, functioning as a spatial atlas in routine use. These methods are evaluated by the Dice similarity coefficient for volumetric overlap and by boundary-geometry metrics, the 95th-percentile Hausdorff distance and the average symmetric surface distance, that capture shape errors which overlap alone misses [2840].

Medical foundation models. The recent shift mirrors the rest of vision: from per-task specialists to promptable foundation models that segment arbitrary anatomy from spatial prompts. The Segment Anything Model [138] catalyzed the move, and MedSAM [2841] adapted it by fine-tuning on more than a million medical image–mask pairs. Because anatomy is volumetric, the decisive step is into 3D: SAM-Med3D [2842] replaces the slice-wise encoder with a fully three-dimensional one, and SegVol [2843] supports both spatial and textual prompts over hundreds of anatomical categories. Much of this work is built on the open MONAI framework [2844], whose medical-aware 3D transforms and model zoo have become standard infrastructure.

Detection and landmark localization. Detection localizes a finding in the body frame, and landmark localization fixes canonical anatomical coordinates; both are spatial tasks distinct from whole-image classification. Lung-nodule detection is anchored by the LIDC-IDRI reference database [2845] and the LUNA16 benchmark [2846], which standardized evaluation on the free-response operating characteristic (sensitivity against false positives per scan). Anatomical landmarks can be found by treating localization as a sequential spatial search, as in the reinforcement-learning agent of Ghesu et al. [2847] that navigates the volume to a target.

Registration as spatial alignment. Registration brings two images into a common coordinate frame and is foundational for multimodal fusion, atlas-based labeling, longitudinal tracking, and surgical planning. Learning-based methods reframe it as predicting a dense deformation field in a single forward pass: VoxelMorph [2848] established the unsupervised CNN formulation, and TransMorph [2849] added a transformer backbone. These build on classical diffeomorphic and intensity-based methods, principally the symmetric diffeomorphic SyN [2850] and the intensity-based toolbox elastix [2851], and are measured by the target registration error between corresponding landmarks. Segmentation and detection are benchmarked on generalist suites such as the Medical Segmentation Decathlon [2852], the brain-tumor benchmark BraTS [2853], the abdominal multi-organ datasets AMOS [2854] and BTCV [2855], the kidney-tumor KiTS19 [2856], and the cardiac ACDC [2857].

The second task turns the perceived anatomy into a patient-specific 3D model that can be measured, simulated, and planned upon, bridging from pixels to clinical decisions. Figure 12.3 previews this progression from a segmentation to an anatomical mesh, a patient-specific simulation, and a surgical plan.

3D anatomical reconstruction. A segmentation becomes a usable model once it is turned into geometry. The classical route extracts an isosurface mesh by Marching Cubes [105], optionally constrained by a statistical shape model that encodes anatomical priors as a learned distribution over landmarks, an idea founded by Active Shape Models [518] and surveyed for medical use by Heimann and Meinzer [2858]. Deep methods deform a template mesh directly from the volume, as in Voxel2Mesh [2859] and the whole-heart reconstruction of Kong et al. [2860]. Implicit neural representations supply a continuous alternative, with DeepSDF [117] and Occupancy Networks [118] as the substrate (Chapter 8) and medical adaptations such as the neural attenuation fields of NAF [2861] reconstructing volumes from sparse-view cone-beam CT.

Patient-specific modeling and the medical digital twin. A geometric model becomes a twin when it is endowed with patient-specific physiology and kept faithful to a particular patient, a medical specialization of the generic digital twin of §11.1. What is clinically specific is that the model is built from the patient’s own imaging and physiological signals, governed by mechanistic physiology rather than CAD, and bound by regulatory credibility. Cardiac twins are the most mature instance: the vision was set out by Corral-Acero et al. [2694], the mechanistic backbone is the multiscale cardiac modeling reviewed by Niederer et al. [2862], and a personalized virtual heart built from scar imaging has been used to guide ablation of ventricular tachycardia [2863]. Hemodynamic twins compute fractional flow reserve from coronary CT by simulating blood flow [2864], and the paradigm extends to the brain, where a virtual epileptic patient estimates the epileptogenic zone to guide surgery [2865]. Such models also underwrite in-silico trials, in which virtual patient cohorts partially substitute for physical ones [2866], a route that regulators have begun to accept, as in the VICTRE imaging trial [2867]. The broader landscape is surveyed in recent reviews of digital twins for health [2695].

Surgical and treatment planning. The patient model is finally used to plan. Open platforms such as 3D Slicer [2868] assemble segmentation, registration, and 3D modeling into a planning workflow, and real-time finite-element engines such as SOFA [2869] support interactive surgical simulation and rehearsal on deformable tissue. In radiotherapy, planning is increasingly cast as a learning problem of predicting an achievable dose distribution from anatomy, standardized by the OpenKBP challenge [2870]. Across these settings the plan is a spatial object, a trajectory, a dose map, or a resection boundary, defined on the patient’s own anatomy.

The third task brings the plan into the operating room and acts within the patient’s live coordinate frame. The central problem is patient-to-image registration under real-world deformation and latency: the preoperative model must be aligned to the physical patient, and instruments tracked within that frame, in real time. Figure 12.4 previews registered preoperative anatomy and augmented-reality guidance alongside the complementary reconstruction of a deforming endoscopic scene.

Surgical navigation. Navigation aligns the preoperative model to the patient and tracks instruments within that frame, typically by optical tracking, which is sub-millimeter but requires line of sight, or electromagnetic tracking, which tolerates occlusion but is perturbed by metal. The defining difficulty is deformation: in neuronavigation, the brain shifts by up to centimeters once the skull is opened, invalidating the rigid preoperative registration, which is why intraoperative imaging and learning-based compensation are active concerns.

Augmented-reality guidance. Rather than divert the surgeon’s gaze to a screen, augmented reality overlays the segmented anatomy directly on the surgical field, a use of the XR runtime treated in §10.1. The approach is most mature in spine surgery, where the Augmedics xvision system, the first FDA-cleared augmented-reality navigation platform, projects the patient’s CT-derived spine onto a transparent head-mounted display for pedicle-screw placement [2871].

Endoscopic spatial perception. Minimally invasive surgery presents a hard perception problem: a monocular, moving camera views a deforming, specular, poorly lit scene. Surgical workflow and instruments are recognized from video, as standardized by the Cholec80 dataset and the EndoNet architecture for joint tool and phase recognition [2872]. Geometry is recovered by self-supervised depth and visual odometry, benchmarked by the SCARED challenge [2873] and the EndoSLAM dataset [2874], and the deforming surgical scene is reconstructed in 3D by dynamic neural rendering, as in EndoNeRF [2875], a clinical specialization of the neural reconstruction of Chapter 8.

Surgical robotics. Robotic systems execute the plan within the patient frame; their manipulation and control are the subject of §9.1, and the concern here is the spatial perception and autonomy they enable. The da Vinci Research Kit [2876] is the standard open research platform, and autonomy is commonly graded on a six-level scale from teleoperation to full autonomy [2877]. Most deployed systems remain teleoperated, but the frontier of autonomous spatial action is exemplified by the Smart Tissue Autonomous Robot, which performs supervised-autonomous soft-tissue suturing using 3D tissue tracking [2878] and has been extended to in-vivo intestinal anastomosis [2879].

Open-source frameworks. The research stack runs from voxels to a planned model: 3D Slicer¹ and MITK² provide segmentation, registration, and planning environments over the lower-level ITK and SimpleITK³ engines; MONAI⁴ supplies 3D-native deep-learning components and nnU-Net⁵ the standard segmentation pipeline; and the web viewer OHIF⁶ renders and annotates volumes in the browser.

Navigation and surgical-robotics vendors. The clinical stack carries the model from preoperative registration to constrained in-OR action. Navigation systems from Brainlab⁷, Medtronic (StealthStation)⁸, and Stryker⁹ register and track instruments, and Augmedics¹⁰ adds an augmented-reality head-mounted overlay. Surgical robots span teleoperated soft-tissue systems (Intuitive da Vinci and Ion¹¹, Medtronic Hugo, CMR Surgical Versius¹²) and CT-planned, haptically constrained orthopedic systems (Stryker Mako, Zimmer Biomet ROSA¹³).

Imaging-AI and clinical platforms. A deployed layer turns spatial findings into care. Aidoc¹⁴ triages critical findings on CT, Viz.ai¹⁵ alerts stroke teams to large-vessel occlusions, HeartFlow¹⁶ computes non-invasive fractional flow reserve from coronary CT, PathAI¹⁷ quantifies biomarkers on whole-slide pathology, and Tempus¹⁸ fuses imaging with genomic and clinical data.

Every capability in the clinical loop must be accurate, robust to deformation, and trustworthy enough to act on a patient. The open problems follow from those demands: generalizing across scanners and sites, obtaining labels under privacy constraints, registering deforming tissue during surgery, earning regulatory and clinical trust, and meeting real-time budgets while fusing imaging with non-imaging evidence.