8.4Motion Generation

Let a motion be a temporal sequence of body poses $\mathbf{m}^{1:T}=(\mathbf{m}^{1},\dots,\mathbf{m}^{T})$, where each frame $\mathbf{m}^{t}\in\mathcal{P}$ is a pose in a chosen representation space $\mathcal{P}$ and $\mathcal{M}=\mathcal{P}^{T}$ is the resulting motion space. Writing $\phi$ for the text encoder of Eq. (8.2) and $\kappa:\mathcal{C}\!\to\!\mathcal{C}^{\prime}$ for an encoder of a driving signal $\mathbf{c}\in\mathcal{C}$ (an action label, a text description, a music or speech track, or a scene), human motion generation learns a conditional sampler

This is the conditional-sampler template of Eq. (8.2) instantiated over a temporal pose sequence rather than an image latent: with no driving signal ($\kappa=\varnothing$) it reduces to unconditional motion synthesis, and the choice of $\mathbf{c}$ indexes the five conditioning settings surveyed in §8.4.1.2.

Figure 8.18 illustrates the pose representations that instantiate $\mathcal{P}$. Broadly, representations fall into keypoint-based and rotation-based categories, and the two are convertible: joint rotations map to spatial keypoints via forward kinematics (FK), and keypoints recover joint rotations via inverse kinematics (IK).

Methods for human motion generation are organized along two axes. The first is the learning paradigm: regression-based techniques learn a deterministic mapping from conditions to a single target motion, whereas generative approaches model the full conditional distribution of motion using GANs, VAEs, normalizing flows, or diffusion models (the generative paradigms themselves are detailed in §8.1.1). Formally, a regression model realizes Eq. (8.54) as a point estimate,

which is simple and stable but collapses the inherently one-to-many condition–motion relationship onto an averaged trajectory. Generative models instead sample $\hat{\mathbf{m}}^{1:T}\!\sim\!p_{\theta}(\mathbf{m}^{1:T}\mid\kappa(\mathbf{c}))$; the now-dominant diffusion instantiation trains a denoiser $G_{\theta}$ to recover the clean motion at every noise level $n$,

which preserves diversity and supports flexible conditioning, at higher sampling cost. The second axis is the conditioning modality $\mathbf{c}$, which indexes the five settings below; each setting instantiates the same Eq. (8.54) and has migrated, broadly, from regression and adversarial models toward diffusion. Figure 8.19 previews this two-axis organization.

Human motion datasets group naturally by conditioning modality; Table 8.10 gives their representation, scale, and condition. Text- and action-conditioned corpora dominate, led by the language-annotated KIT Motion-Language [2206] and HumanML3D [2207], BABEL [2208]’s sequence- and frame-level labels over the AMASS [2209] aggregate, and the action taxonomies of UESTC [2210], NTU-RGB+D 120 [2211], and HumanAct12 [2158]. Audio-conditioned resources split into music-to-dance sets (AIST++ [2178], AIOZ-GDANCE [2180], and several genre-specific collections) and co-speech gesture sets (Trinity [2212], BEAT [2213], PATS [2198]), many built by fitting parametric body models to in-the-wild video. Scene-conditioned datasets (PROX [2214], GRAB [2215], HUMANISE [2216], CIRCLE [2205]) capture human–scene and human–object interaction with synchronized geometry, from Vicon and IMU rigs to synthetic engines such as GTA-IM [2199]. Across all three groups, the cost of high-quality capture forces a trade-off between scale and annotation fidelity.

Data scarcity is the most basic constraint: high-quality motion capture is expensive and slow, so dataset scale trades against annotation fidelity, and heterogeneous pose representations and conditioning signals fragment the data that exist; fusing heterogeneous sources through weak supervision and multimodal foundation models is the main response. Semantic alignment is the second. Motion encodes culture, communication, and context, not only mechanics, and a model that fits joint trajectories while missing their relation to language, music, or environment produces motion that is kinematically clean but perceptually empty; semantic priors from language and foundation models help, but how to measure the result is unresolved. That is the evaluation problem: every objective metric above has blind spots, none replaces a user study, and the field still needs metrics that are both interpretable and better correlated with human perception. Controllability matters for practical use as much as in image generation, and joint masking, attribute prompts, and style conditioning are moving toward editing tools with textual, spatial, and temporal handles (§8.4.4). Interactivity remains underexplored: most methods animate one person in a static scene, and socially coupled groups (conversation, group dance) and motion grounded in dynamic, actionable scenes need both new corpora and new models.

Animal motion generation extends the human sampler of Eq. (8.54) with an explicit species (morphology) descriptor. Let $s\in\mathcal{S}$ index a species and let $\mathcal{P}_{s}$ be its species-dependent pose space (its skeleton topology and joint count depend on $s$), so that $\mathcal{M}_{s}=\mathcal{P}_{s}^{T}$ is the corresponding motion space. With $\kappa$ encoding a driving signal $\mathbf{c}\in\mathcal{C}$ (an action class, a text description, an anchor pose, or past motion), the task is to learn

Restricting $\mathcal{S}$ to the single human morphology recovers Eq. (8.54) exactly; the open difficulty is that $s$ ranges over a heterogeneous, long-tailed set of skeletons for which paired text–motion data are scarce, so the species condition must do double duty as both a generation control and a regularizer that shares statistical strength across morphologies.

Almost all animal motion generation is text-driven, and the methods share a common response to the data-scarcity and morphology problems of Eq. (8.57): transfer representations learned from abundant human motion, then specialize them to a species through feature modulation. Concretely, given a backbone feature $\mathbf{h}$ at some layer, species-aware modulation applies a FiLM-style affine transform conditioned on $s$,

where $\gamma(s),\beta(s)$ are predicted from the species descriptor, allowing one shared network to adapt its internal representation to diverse anatomical forms. The methods differ in how they obtain the shared representation and how strictly they enforce cross-species plausibility, as Figure 8.20 summarizes.

OmniMotionGPT [2236] tackles limited animal data by jointly training motion autoencoders on human and animal corpora and aligning their latent encodings to text via CLIP embeddings; it relies most directly on transfer, and at inference it still depends on human-motion priors, which prevents truly text-only animal generation. Motion Avatar [2237] uses SinMDM [2238] for data augmentation in its ZooGen pipeline (the route by which the Zoo-300K dataset is built) and extends the masked-motion modeling of MoMask [2176] to animals, adding an LLM planner that orchestrates joint motion and 3D-avatar mesh generation; it makes little use of species-specific structure, which is what Eq. (8.58) addresses. AniMo [2239] makes that remedy central, injecting species-aware modulation into a joint-aware spatiotemporal tokenizer and learning, through masked modeling, a mapping from text plus species attributes to motion tokens; its AniMo4D dataset (78,149 sequences over 114 species) yields markedly better morphological and behavioral fidelity than transfer-only baselines. X-MoGen [2240] pushes toward a single model spanning humans and animals: it embeds motion into a shared space with graph-based VAEs and latent autoencoders, generates via masked modeling, and adds a morphological-consistency module that enforces skeletal plausibility across species, supported by the UniMo4D corpus (${\sim}115$ species, ${\sim}119$K sequences). OmniMotionGPT and Motion Avatar still lean on human priors, whereas AniMo and X-MoGen promote species morphology to a first-class, explicitly regularized conditioning variable, so the four together trace a progression from transfer toward native cross-species modeling.

Animal motion resources fall into two groups (Table 8.11). Capture and pose-estimation corpora supply ground-truth articulated motion but no language: RGBD-Dog [2241] and 3DDogs-Wild [2242] pair RGB-D or RGB video with marker-based mocap for canine pose, and DeformingThings4D [2243] provides synthetic 4D sequences with dense mesh-deformation annotations across humanoid and animal categories. Text-to-motion corpora drive recent progress: AnimalML3D [2236] is the first text–animal-motion dataset, and the field then scales sharply through Zoo-300K [2237] (built by the ZooGen pipeline with SinMDM and LLM annotators), AniMo4D [2239] (normalized to a 30-joint cross-species skeleton), and UniMo4D [2240], which merges AniMo4D with HumanML3D and KIT-ML. Even the largest text-conditioned sets remain smaller and more morphologically fragmented than their human counterparts of §8.4.1.3, which is what forces the transfer-and-modulate methodology above.

Physical realism is the most basic gap: most methods do not simulate ground-contact forces, quadruped balance, or the fluid dynamics of swimming and flight, so artifacts the human setting has largely learned to suppress reappear, and differently for every body plan. Behavioral diversity is harder than for humans because one instruction such as “walk stealthily” realizes into different gaits, postures, and speeds across species, so the conditional distribution to be captured is far wider while the data to capture it are scarcer. Unseen species and radical morphologies remain out of reach: species-aware modulation (Eq. (8.58)) and morphological-consistency constraints generalize across the skeletons seen in training, but insects and winged or aquatic creatures violate the shared-skeleton, fixed-topology assumptions on which current methods rest. AniMo and X-MoGen already move toward treating morphology as a variable rather than a template, and topology-flexible representations are the direction this points.

Let a trajectory be a sequence of agent states $\bm{\tau}^{1:H}=(\mathbf{s}^{1},\dots,\mathbf{s}^{H})$ over a horizon $H$, where each state $\mathbf{s}^{t}$ collects position, heading, and velocity. The driving signal is now a structured scene context $\mathbf{c}=(\mathbf{s}^{-P:0},\,\mathcal{G},\,\mathcal{A})$ comprising the agent’s own past states, an HD-map / lane graph $\mathcal{G}$, and the observed histories of neighboring agents $\mathcal{A}$. With $\kappa$ encoding this context, vehicle motion planning learns

the conditional-sampler template of Eq. (8.54) with the pose sequence replaced by a state trajectory and an added kinematic-feasibility constraint $f_{\text{dyn}}$ (e.g., , a bicycle model) that distinguishes this setting from articulated-body generation. Dropping $\mathcal{A}$ recovers single-agent forecasting; supplying a goal or reward in place of $\mathbf{c}$ turns prediction into planning.

Prediction methods vary along two axes: how the scene context is encoded and how the multimodal future is decoded. Early encoders rasterize the scene into a bird’s-eye-view image, but the field has converged on vectorized encoders that represent lanes and agent tracks as polylines: VectorNet [2244] and LaneGCN [2245] build graph representations of the HD map, and transformer encoders such as Wayformer [2246] and the Motion Transformer (MTR) [2247] attend jointly over map and agents; graph-attention predictors [2248] model spatial and temporal interaction in the same spirit. On the decoding side, the defining property is multimodality, since a single context admits many plausible futures, so models predict a set of $K$ modes with probabilities,

rather than the single point estimate of Eq. (8.55), which here would average distinct maneuvers (turning vs. going straight) into an infeasible mean. The earliest learned predictors modeled social interaction directly in agent-coordinate space, Social-LSTM [2249] through pooled recurrent states and Social-GAN [2250] through adversarial sampling, and Trajectron++ [2251] added dynamic feasibility to a graph-structured latent-variable model. Goal-based decoders make the modes explicit: DenseTNT [2252] samples dense candidate endpoints on the map and scores trajectories conditioned on them, exposing the maneuver structure that Eq. (8.60) captures only implicitly. Generative decoders replace the fixed mixture with a learned sampler: MotionDiffuser [2253] denoises whole multi-agent trajectory sets jointly.

Planning turns the same machinery on the ego vehicle. ChauffeurNet [2254] learns the ego policy by imitating expert driving and augments the demonstrations with synthesized perturbations and worst cases, so that the policy stays robust when it drifts off the data manifold. Figure 8.21 situates these families along the encode/decode axes.

Progress is measured on large real-world driving logs that pair agent tracks with HD maps: nuScenes [445], Argoverse and Argoverse 2 [2255, 661], and the Waymo Open Motion Dataset [218], the last two offering the scale and interaction density that current transformer models require. Evaluation centers on the multimodal-aware displacement metrics that match Eq. (8.60): minimum average and final displacement error over the $K$ predicted modes (minADE_K, minFDE_K), miss rate (the fraction of scenes where no mode lands within a tolerance of the ground truth), and the soft mean-average-precision of the Waymo protocol, which jointly scores accuracy and the calibration of mode probabilities $\pi_{k}$. Off-road and collision rates further test the feasibility constraint $f_{\text{dyn}}$ that purely displacement-based scores ignore.

A planner must assign trustworthy probabilities to the futures it enumerates, not merely list them: an over-confident missed mode of Eq. (8.60) is a safety failure rather than a metric error, so calibrated multimodality is the first open problem. Feasibility and safety are the second: displacement metrics reward trajectories a vehicle cannot physically execute, so feasibility has to be enforced inside the model ($f_{\text{dyn}}$) rather than checked afterward, and the long tail of near-collision scenarios is exactly where data are scarcest. The third is closed-loop distribution shift: models trained to imitate logged, open-loop futures drift once their own predictions feed back into the scene, the difficulty ChauffeurNet’s worst-case synthesis was built to expose. Diffusion- and transformer-based scene-level samplers that forecast all agents jointly and run closed-loop are the current direction, and they connect directly to the controllable-video and world-model lines of §8.3 and §8.6, in which a driving world model both predicts and is conditioned on the ego plan.

Given a source motion $\mathbf{m}^{\text{in}}\!\in\!\mathcal{M}$, an edit region $\mathbf{M}\!\in\!\{0,1\}^{T\times J}$ over time $T$ and joints $J$, and an edit control $\mathbf{c}$ (a text instruction, target keyframes, a desired root trajectory, or a target joint), motion editing learns an operator

which lifts the image-editing operator of Eq. (8.8) into the motion domain. As in image and video editing, Eq. (8.61) encodes two competing objectives, editability (the output must realize the control $\mathbf{c}$ inside $\mathbf{M}$) and preservation (it must match $\mathbf{m}^{\text{in}}$ outside $\mathbf{M}$ and remain a plausible, smooth motion), and methods differ chiefly in how they trade these off.

Because a pretrained motion diffusion model already samples plausible motion, most editors realize Eq. (8.61) without retraining, by conditioning the sampler of Eq. (8.56) rather than learning a dedicated $E_{\theta}$. The simplest mechanism is masked infilling: at every denoising step the observed entries are overwritten with the (noised) source while only the masked entries are generated,

which directly enforces preservation by construction. This is the basis of motion in-betweening, where $1-\mathbf{M}$ marks given keyframes: robust in-betweening [2256] predates the diffusion era, and CondMDI [2257] casts flexible keyframe-conditioned in-betweening as masked diffusion, while PriorMDM [2258] reuses a frozen motion-diffusion prior for editing and composition tasks such as joining and in-filling. A second family handles spatial controls that masking alone cannot express (a target root path, or a specific joint at specific times) through classifier-style guidance: GMD [2259] and OmniControl [2260] steer the denoiser toward arbitrary spatial constraints at any joint and time. A third, optimization-based family treats the initial diffusion noise itself as the free variable, solving $\min_{\mathbf{z}}\,\mathcal{L}_{\text{edit}}(\mathcal{D}(\mathbf{z}))$ so that the decoded motion satisfies an arbitrary differentiable criterion: DNO [2261] shows this single mechanism serves editing, in-betweening, and refinement as one universal motion prior. Finally, semantic editors take a natural-language instruction in place of a geometric mask: MotionFix [2262] learns text-driven edits from paired source–target motions, and LLM-driven approaches [2263] decompose an instruction into a program of primitive edits applied iteratively. Figure 8.22 groups these four mechanisms by how they obtain $E_{\theta}$, paralleling the editing taxonomies of §8.1.3 and §8.3.5.

Motion editing inherits the corpora of §8.4.1.3 (HumanML3D [2207] and BABEL [2208] over AMASS), used in editing protocols where keyframes, joints, or trajectories are held out as the constraint, together with the dedicated MotionFix [2262] benchmark of paired source–edit–instruction triples that supplies the ground truth most editing tasks lack. Evaluation must report the editability–preservation pair jointly: editability through control adherence (instruction-conditioned R-Precision, target-joint or trajectory error) and overall realism (FID, foot-skating), and preservation through the deviation from the source outside the edit region. As in video editing, single scalars correlate weakly with perceived edit quality, so reporting a tuple, and increasingly a learned or VLM-style judge of instruction compliance, is becoming standard.

The central tension is the editability–preservation trade-off of Eq. (8.61): pushing harder on a control propagates unwanted change through the kinematic chain (editing one arm disturbs balance and foot contact), so a good editor must respect the body’s global coordination, not just the masked entries. This is compounded by sparse-control ambiguity: a single keyframe or a root path under-determines the full-body motion, and the model must fill the gap plausibly, which the guidance and noise-optimization families address with differing robustness. Underlying both is an evaluation gap sharper than in synthesis: because the “correct” edit is rarely unique, ground-truth-based metrics under-reward valid alternatives, motivating the paired-triplet and judge-based protocols above. The trend mirrors the rest of §8.4 and §8.3: from per-mechanism, training-free editors toward unified models that generate, control, and edit motion within a single conditional sampler, increasingly driven by natural-language instructions rather than geometric masks.