8.12D Spatial Generation

Generative Adversarial Networks. GAN-based T2I models instantiate $G_{\theta}$ as a feed-forward generator $G_{\theta}(\mathbf{n},\phi(\mathbf{x}))\!\to\!\hat{\mathbf{z}}$ driven by an auxiliary noise vector $\mathbf{n}\sim\mathcal{N}(0,I)$; the conditional distribution $p_{\theta}(\mathbf{z}\mid\phi(\mathbf{x}))$ is defined only implicitly, as the pushforward of $\mathbf{n}$ through $G_{\theta}$. Training follows the two-player game of Goodfellow et al. [1646], pitting the generator against a discriminator $D_{\psi}$,

in which $D_{\psi}$ scores both photorealism and text-image consistency. The result is sharp high-frequency detail and single-pass inference, bought at the price of unstable training and mode collapse. Within this framework, StackGAN [1647] generated a low-resolution image first and refined it with a second GAN conditioned on both the text and the coarse image; AttnGAN [1648] let the generator attend to individual words while rendering different regions, with an auxiliary image-text matching loss (DAMSM) that reduced attribute errors such as wrongly colored objects; XMC-GAN [1649] replaced that loss with cross-modal contrastive learning and improved multi-object prompts. By 2021 diffusion models had overtaken these systems on COCO, chiefly because they train stably and cover more of the image distribution, and research attention moved on; the GAN emphasis on high-frequency detail survives in hybrid and adversarially fine-tuned samplers.

Autoregressive Models. Autoregressive (AR) models discretize the latent space: the autoencoder $E$ is realized as a vector-quantized variational autoencoder (VQ-VAE) whose output is a sequence of discrete tokens $\mathbf{z}=(z_{1},\dots,z_{N})\in\mathcal{V}^{N}$ over a learned codebook $\mathcal{V}$, and $G_{\theta}$ is factorized in causal raster order,

trained by maximum likelihood on the next-token cross-entropy. T2I thus becomes sequence-to-sequence translation, and the transformer that performs it inherits the compositional competence of language models: DALL-E [1650] modeled text and image tokens as a single sequence and showed zero-shot generalization across prompt types. The cost is decoding latency, one forward pass per token, which grows with resolution. Parti [1651] scaled the transformer to 20B parameters, generated at $256\times 256$, and reached $1024\times 1024$ with a convolutional super-resolution stage; Muse [1652] abandoned strict sequential order for masked token prediction, filling in many tokens per step over a handful of rounds, and matched diffusion quality on several benchmarks at a fraction of the sampling time. AR generation remains attractive where images are produced jointly with text (interleaved documents, storyboards) and a single sequence model handles both.

Diffusion Models. Diffusion models realize $G_{\theta}$ as iterative denoising rather than a single feed-forward pass. A fixed forward process corrupts the clean latent $\mathbf{z}_{0}=E(\mathbf{y})$ along a schedule $(\alpha_{t},\sigma_{t})_{t=0}^{T}$, $\mathbf{z}_{t}=\alpha_{t}\mathbf{z}_{0}+\sigma_{t}\epsilon$ with $\epsilon\sim\mathcal{N}(0,I)$ and $\mathbf{z}_{T}\!\approx\!\mathcal{N}(0,I)$; a noise-prediction network $\epsilon_{\theta}$ learns to invert it under the denoising score-matching objective

which is the diffusion-specific realization of the training objective implicit in Eq. (8.2). At inference the model starts from $\mathbf{z}_{T}\sim\mathcal{N}(0,I)$, applies $\epsilon_{\theta}$ for $t=T,\dots,1$, and decodes $\hat{\mathbf{y}}=D(\hat{\mathbf{z}}_{0})$; each step corrects a small error, which is why the final samples are both detailed and faithful. Running this process in the latent space of Eq. (8.1) rather than in pixels, as Latent Diffusion Models [185] did (e.g., , $4\times 64\times 64$ instead of $3\times 512\times 512$ for a 512px image), cut memory and compute enough to train on commodity hardware and to release Stable Diffusion openly. Conditioning the denoiser on embeddings from a pretrained language-image model, as in GLIDE [1653] and DALL$\cdot$E 2 [1654], improved prompt alignment, and classifier-free guidance [1655] supplied a single knob that trades diversity against fidelity by extrapolating the conditional prediction away from the unconditional one. The remaining weakness is sampling cost, dozens of sequential network evaluations per image; higher-order ODE samplers and the distillation methods below address it. Diffusion or flow-matching cores now underlie most leading systems, with the native multimodal autoregressive stacks discussed under proprietary systems the main exception.

Flow Matching and Rectified Flow. Rather than a fourth paradigm, flow matching is diffusion reformulated in continuous time. It parameterizes $G_{\theta}$ as a learned velocity field $v_{\theta}(\mathbf{z}_{t},t,\phi(\mathbf{x}))$ instead of a noise predictor: training picks the straight interpolation $\mathbf{z}_{t}=(1-t)\mathbf{z}_{0}+t\mathbf{z}_{1}$ between a clean latent $\mathbf{z}_{0}=E(\mathbf{y})$ and a Gaussian sample $\mathbf{z}_{1}\!\sim\!\mathcal{N}(0,I)$ and regresses the velocity,

and inference integrates the probability-flow ODE $\mathrm{d}\mathbf{z}_{t}/\mathrm{d}t=v_{\theta}(\mathbf{z}_{t},t,\phi(\mathbf{x}))$ from $t=1$ back to $t=0$. The formulation was introduced concurrently by Lipman et al. [1656] (Flow Matching) and Liu et al. [1657] (Rectified Flow); Albergo and Vanden-Eijnden [1658] gave the unifying stochastic-interpolant view in which diffusion is the special case of a noisier interpolation schedule. Three consequences matter in practice. Straight-line interpolations yield nearly straight ODE trajectories, so matched quality is reached in noticeably fewer steps (genuine one-step generation still needs a reflow or distillation stage); the objective is a single regression with no noise-schedule design and no per-schedule guidance recalibration; and the loss scales smoothly with model size and resolution. Stable Diffusion 3 [1659] scaled the recipe to T2I on a DiT-style transformer backbone and reported the best GenEval scores among open models together with favorable human preference on Parti-prompts; the Flux family followed the same recipe. Flow matching composes with every refinement below as readily as a noise-predictor core does and has become the default continuous-time formulation in 2024-onwards systems.

Backbone Evolution: From U-Net to Diffusion Transformers. Early diffusion models used the U-Net for its multiscale feature fusion and locality bias. The Diffusion Transformer (DiT) of Peebles and Xie [1645] replaced it with a stack of transformer layers over latent patch tokens, giving every denoising step a global receptive field, and showed that FID improves monotonically with depth, width, and token count along predictable scaling laws, reaching FID 2.27 on class-conditional ImageNet at $256^{2}$. Attention cost grows quadratically with resolution, so hybrid CNN-transformer designs and efficient attention remain active; the DiT lineage nonetheless underlies SD3 and most 2024-onwards systems, and it points toward a single transformer that generates several modalities, differing only in tokenization and training data.

Sampling Acceleration and Distillation. Fifty or more denoising steps rule out interactive use and low-power deployment. Salimans and Ho [1660] distilled a $T$-step teacher into a $T/2$-step student and iterated the halving to reach $1$–$4$ steps. Consistency Models [1661] train a network to map a noisy latent at any timestep directly to the clean image, either from scratch with a consistency loss or by distilling a pretrained diffusion model, and Latent Consistency Models [1662] applied the idea to Stable Diffusion’s latent space, matching $50$-step quality in $2$–$4$ steps. All of these are post-training procedures that leave the architecture untouched, so any strong backbone can be made fast after the fact.

Proprietary and Commercial Systems. Leading systems have migrated to proprietary, industrially scaled stacks of three kinds. Native multimodal models such as OpenAI’s GPT Image 1.5 [1663] and Google’s Gemini 3 Pro Image (Nano Banana Pro) [1664] run a unified transformer over interleaved text and image tokens and emphasize semantic reasoning and multi-turn editing; latent-diffusion iterations such as Imagen 4 [1665] scale the LDM recipe with stronger text encoders and curated data; preference-tuned systems such as Midjourney v7 [1666] and FLUX.2 [max] [1667] fine-tune strong priors on human-preference data (FLUX.2 on the rectified-flow recipe of §8.1.1.2; Midjourney’s architecture is undisclosed). Each reduces to one of the three paradigms plus some of the refinements just listed; because architectural and training detail is withheld, public benchmarking and reproduction still rest on the open SD / DiT stack.

Training Corpora. Corpus scale has driven much of T2I progress. Early work fit fine-grained sets with ten captions per image, CUB-200-2011 (11,788 bird images) [1668] and Oxford-102 Flowers (8,189 images) [1669]; MS-COCO Captions [440] (123,287 images, five captions each) became the standard general-scene corpus and, for years, the evaluation set as well, a conflation that the suites below were designed to end. Current models train on web-scale image-text pairs: LAION-5B [1670] releases 5.85 billion CLIP-filtered pairs, and proprietary corpora of similar scale, less documented, underlie DALL-E 2 and Parti. Scale brought duplication, demographic bias, unsafe content, and licensing disputes with it, and filtered subsets such as LAION-Aesthetics and curation pipelines are now part of every training recipe.

Evaluation Benchmark Suites. An evaluation suite is a curated prompt set plus a verification protocol: prompts are adversarial or capability-stratified, reference images are usually absent, and scoring is automatic (an object detector, a VQA model, or an LLM/MLLM judge) rather than reconstruction-based. Table 8.1 lists the main suites since 2022. The earliest, DrawBench [1671] and PartiPrompts [1651], are human-pairwise prompt sets released with Imagen and Parti; HRS-Bench [1672] and HEIM [1673] are holistic, scoring 13 and 12 axes respectively; TIFA [1674], GenEval [1675], and T2I-CompBench [1676] introduced automatic verification by VQA, object detection, and BLIP-VQA plus UniDet, and the latter two are now reported alongside FID and CLIPScore in most papers. For spatial intelligence the salient suite is VISOR [1677], which isolates two-object relations (left/right, above/below) over $25{,}280$ prompts and finds that otherwise strong models frequently render the wrong configuration. DPG-Bench [1678] targets dense, long compositional prompts, GenAI-Bench [1679] pairs compositional prompts with VQAScore judging, and WISE [1680] asks whether generated images agree with real-world facts (flag colors, landmarks, anatomy), a knowledge axis orthogonal to compositional binding.

Evaluation Metrics. Metrics fall into four tiers. Distributional metrics compare the generated and real image populations and say nothing about any individual prompt: the Fréchet Inception Distance (FID) [1681] is the Wasserstein-2 distance between Gaussian fits of Inception-V3 features of generated and real images, the Inception Score rewards images that are individually confidently classified yet collectively diverse, and generative precision/recall [1682, 1683] separate quality from coverage through neighborhood tests in feature space. Alignment metrics score one image against its prompt; CLIPScore [1684], the cosine similarity between the CLIP embeddings of $\hat{\mathbf{y}}$ and $\mathbf{x}$, is cheap and reference-free but inherits CLIP’s insensitivity to attribute binding, counting, and spatial relations. Learned preference models, HPSv2 [1685], ImageReward [1686] (fit to $137$K human comparisons), and PickScore [1687] (fit to the ${\sim}500$K real-user comparisons of Pick-a-Pic), regress a scalar that correlates with average user preference, bundling realism, alignment, and aesthetics; they have become the default automatic ranking while inheriting their raters’ biases and degrading out of distribution.

The newest tier uses a vision-language model as judge. TIFA [1674] has a language model write verification questions from the prompt and a VQA model answer them against the image, the score being the fraction answered correctly; VQAScore [1688] collapses this to a single yes/no probability under a fine-tuned CLIP-FlanT5 model and reports the best human correlation on compositional suites; VIEScore [1689] and LLMScore [1690] prompt a multimodal LLM with a structured rubric and return per-axis scores with brief rationales, and direct prompting of frontier VLMs with custom rubrics has become routine for ad-hoc capability probes. Judge metrics track human preference on compositional and spatial prompts far better than CLIPScore, at the cost of a frontier-model call per image, sensitivity to rubric wording, and drift across judge-model versions that makes numbers from different papers hard to compare. No single tier suffices, and current papers report a tuple (FID or IS, CLIPScore, one or two preference scores, one judge score) on several suites plus a small human study; cross-metric disagreement is read as a warning rather than averaged away.

Conditioning Modalities. Dense pixel-level scaffolds supply a value at every location, which makes them the easiest channels to follow and the most restrictive on creativity. Instance and semantic masks drive region-conditioned synthesis, with MultiDiffusion [1692] offering a training-free protocol that samples jointly over overlapping mask windows; edges, sketches, and line art, popularized by the canny and soft-edge variants of ControlNet [1693] and the edge head of T2I-Adapter [1694], fix only the location of intensity discontinuities and leave texture and appearance to the prompt; depth and normal maps, served by ControlNet’s depth variant, impose coarse occlusion ordering and surface orientation without requiring a metric reconstruction. Coordinate-level scaffolds drop to per-region or per-point supervision. Boxes with labels are the most studied channel, a set of $(\text{box},\text{label})$ tuples fixing location and rough scale, with LayoutDiffusion [1695], LayoutLLM [1696], and the open-set GLIGEN [1697] as representatives; 2D keypoints (typically human pose) are sparser still, a handful of joint coordinates, injected through ControlNet or handled by GLIGEN as a special case of grounding. Discrete symbolic structure carries no pixel- or coordinate-level information at all, and the model must map it onto the image plane itself. Scene graphs reach back to the sg2im line of Johnson et al. [1698] and re-emerged in the diffusion era through LLM-emitted layouts, as in LayoutLLM, where the graph is first decoded into a coordinate-level layout; region phrases and referring expressions bind a text span to a box, mask, or free-form polygon and are served by GLIGEN’s grounded cross-attention and by phrase-grounding heads attached to SD and SDXL [1699].

Injection Mechanisms. Five mechanism families cover the literature; production pipelines routinely stack a side branch for dense geometric channels, an adapter for style or a reference image, and inference-time guidance to sharpen adherence on adversarial prompts.

Side-branch controllers. The backbone weights stay frozen and an auxiliary branch mirroring the encoder half of the base U-Net is trained, connected through zero-convolution layers whose weights are initialized to zero so that the augmented model reproduces the base model exactly at the start of training and degrades gracefully if the branch is poorly tuned. ControlNet [1693] is the prototype and the first widely adopted retrofit pathway for non-text structural control: the condition (a canny edge map, an OpenPose skeleton, a depth or normal map, a segmentation mask, or a scribble) is encoded by the side branch in parallel with the noisy latent and injected into the base U-Net through additive residual connections at each scale. One branch is trained per modality on labeled image–condition pairs, which produced a library of plug-in controllers for SD 1.5 and SDXL [1699] that compose with prompt conditioning at inference time without retraining the backbone; most later designs either shrink this recipe or fold it into a unified adapter library.

Lightweight adapters. Where the side branch spends capacity (half a U-Net per modality), T2I-Adapter [1694], developed concurrently with the same goal of plug-in control over a frozen backbone, spends as little as possible: a few residual blocks per modality whose multi-scale features are added directly into the corresponding encoder stages. Several adapters can be combined at inference, an image conditioned simultaneously on an OpenPose skeleton, a depth map, and a color palette with a scalar weight on each, so the adapter is preferred for multi-channel fusion and fast iteration, and the side branch where maximum single-channel fidelity matters.

Grounded cross-attention. Side branches and adapters run an external pathway alongside the backbone; the next two families intervene inside its attention stack. GLIGEN [1697] inserts a new gated self-attention layer between the existing self-attention and cross-attention of each U-Net transformer block; the layer attends jointly over the visual tokens and a set of grounding tokens that encode each $(\text{phrase},\text{box})$ pair, a learned scalar gate controls how strongly the grounding signal enters the residual stream, and only the new layer is trained, so the network falls back to ungrounded behavior when no constraint is supplied. The problem it solves is complementary to the side branch: not how to inject a dense pixel-level scaffold, but how to introduce open-set symbolic grounding (any category named in text, localized by a box, keypoints, or a reference image) into a frozen T2I backbone, with several grounding channels interleaved within one mechanism.

Layout-aware cross-attention. LayoutDiffusion [1695] is the dense-supervised cousin of grounded attention: rather than gating a new layer onto a frozen backbone, it co-trains a layout-aware cross-attention block end to end for layout-to-image (L2I) generation specifically. A structured layout, a set of $(\text{box},\text{class label})$ tuples, becomes a sequence of region tokens fused with the noisy latent at every denoising step, and an explicit object–scene factorization models global scene context separately from per-object features so that object identity, scale, and pairwise relations survive in cluttered scenes. It reports stronger layout IoU and relation satisfaction than ControlNet or T2I-Adapter on multi-object COCO benchmarks, at the cost of being specialized to box-and-label input rather than the broader modality library.

Inference-time guidance. The fifth family adds no parameters and modifies only sampling. BoxDiff [1700] is the canonical training-free spatial controller: it shapes the cross-attention maps so that activations of box-localized prompt tokens concentrate inside their boxes while the backbone stays untouched. MultiDiffusion [1692] fuses overlapping mask windows at sampling time, and composing scores from several conditional models or tuning the classifier-free guidance scale per region are related strategies. Because nothing is retrained, these methods suit rapid prototyping, backbones whose weights are unavailable for fine-tuning, and per-prompt sharpening on top of the trained mechanisms above.

Training Corpora. Spatial-aware models are typically trained on corpora that pair images with rich structural annotations: COCO Captions [440] for layouts and captions; ADE20K [492] ($20$K images, $150$ classes) for semantic and instance masks; Visual Genome [796] for object boxes, attributes, and pairwise relations supporting scene-graph conditioning; RefCOCO / RefCOCO+ / RefCOCOg [1082] for phrase grounding; and PhraseCut [1092] for region-prompt mask supervision. Additional structure (canny edges, depth, normals, OpenPose skeletons) is typically derived on the fly from these base corpora by running pretrained predictors over each image, producing the paired image–condition data on which ControlNet-style and adapter-style pipelines are trained.

Evaluation Benchmark Suites. Most of the suites surveyed in §8.1.1 (Table 8.1) include axes that probe spatial-aware generation directly and serve as its primary out-of-the-box benchmarks. GenEval [1675] exposes a position subset that scores whether two objects appear in a specified relation (left of, above); T2I-CompBench [1676] provides explicit spatial and non-spatial relationship categories with BLIP-VQA and UniDet scoring; HRS-Bench [1672] includes a spatial skill among its thirteen; GenAI-Bench [1679] offers compositional prompts on which spatial relations are a frequent failure of CLIP-based scoring; and WISE [1680] measures knowledge-grounded location accuracy. Beyond these, the L2I literature reports task-specific evaluations on COCO layouts: detection-based AP for box-localized objects alongside FID and CLIPScore.

Constraint-specific Metrics. Layout adherence is box IoU between the conditioning layout and the boxes recovered by a frozen detector on $\hat{\mathbf{y}}$, often with per-object center distance; mask adherence uses mask IoU and boundary $F$-score against the conditioning mask; edge and sketch adherence report recall (sometimes precision) of edges re-extracted from $\hat{\mathbf{y}}$; geometry adherence reports depth RMSE and normal angular error where ground-truth geometry exists; pose adherence reports keypoint reprojection error or PCK; and relational adherence reports relation-satisfaction accuracy and per-entity object AP for scene-graph and phrase-grounded settings. These numbers do not aggregate: averaging layout IoU with mask IoU has no operational meaning, since the two live on different scales and reward different shape regimes, and most papers report a different subset of them, which is the main obstacle to cross-paper comparison in this subfield.

Variants by Task Type. The tasks differ in what populates $\mathbf{c}$ and $\mathcal{M}$ and in whether ground-truth pixels exist for evaluation. Reconstruction-style completion carries no semantic intent: hole filling repairs scratches and occlusions from $(I,\mathcal{M})$ alone, and object removal reconstructs the background behind a specified instance. Constraint-driven local synthesis adds intent inside $\mathcal{M}$: text-guided local editing follows an instruction scoped to a region (“make the left cup red”), structure-guided inpainting takes edges, sketches, or semantic maps as a scaffold for shape and boundary, and geometry-guided inpainting conditions on depth or normals to respect occlusion ordering. Object-level edits treat the region as an object whose identity or embedding in the scene is the unit of change: insertion with harmonization of color, grain, lighting, shadows, and reflections; attribute and appearance editing that changes material or style while preserving identity (garments, products, logos); and exemplar-guided transfer that drives content or style from a reference patch. Interactive workflows replace declarative constraints with actions taken at edit time: point- or handle-based reshaping, and multi-round, multi-region sessions in which stability across rounds is itself the requirement. The metrics appropriate to each are collected in §Benchmarks.

Editing Mechanisms. Masked latent denoising. The most direct realization of Eq. (8.8) runs the diffusion sampler only inside $\mathcal{M}$ while clamping the latent outside $\mathcal{M}$ to a noise-perturbed copy of $\mathbf{z}^{\text{in}}$ at every step. RePaint [1705] does this with a pretrained, unconditional DDPM and no retraining, replacing the known-region pixels with a forward-noised sample of the input at each reverse step and adding a resampling schedule that re-injects noise periodically so the in-fill can re-harmonize with its context over several rounds. Blended Diffusion [1706] made the same scheme text-driven by interleaving a CLIP-guided gradient step that steers the content inside $\mathcal{M}$ toward a target prompt with the context-clamping step outside it, the first widely cited text-guided local editor. LaMa [1707] predates the diffusion line and is instead a feed-forward inpainter whose fast Fourier convolutions give every layer an image-wide receptive field, a Fourier-domain inductive bias that stays robust on the very large holes that defeat purely local CNN backbones. Region-aware noise schedules and classifier-free guidance are standard add-ons, and ControlNet-inpaint and the inpainting variants of T2I-Adapter (§8.1.2.2) compose with masked denoising by supplying the inside-$\mathcal{M}$ condition.

Inversion + rewrite. A complementary strategy first recovers the noise trajectory that the model would have used to generate $I$ (inversion) and then re-runs the reverse process under an edited prompt while reusing or swapping the attention maps of the original (rewrite). Prompt-to-Prompt [1708] reuses or reweights source cross-attention maps during generation under the edited prompt, which preserves spatial structure while changing selected semantic content; for a real image an inversion step first supplies a suitable noise trajectory. Imagic [1709] jointly optimizes the text embedding and lightly fine-tunes the model to reconstruct $I$, then interpolates the embedding toward the target prompt, preserving identity-level features (faces, garments) under non-rigid edits. Null-Text Inversion [1710] fixes the practical failure that plain DDIM inversion drifts away from $I$ under classifier-free guidance, by optimizing the null-text embeddings used in the inverse pass so that exact reconstruction holds at ordinary guidance scales. The family needs no full backbone fine-tune for Prompt-to-Prompt or Null-Text and dominates attribute, style, and structure-preserving real-image edits.

Instruction-tuned editing. A third family avoids inversion entirely by retraining a diffusion editor on paired (before, instruction, after) triplets, mapping $I$ directly to $\hat{I}$ from a natural-language instruction with no explicit mask. InstructPix2Pix [1711] built its corpus by pairing GPT-3-generated edit instructions with Prompt-to-Prompt edits of Stable Diffusion samples and fine-tuned Stable Diffusion on the triplets, so that at inference the user supplies only $(I,\text{instruction})$ and the editor localizes the change itself. MagicBrush [1712] replaced the synthetic data with about 10K manually annotated real-image edits across multi-turn sessions, yielding both a stronger supervised editor and the standard instruction-editing test set. Emu Edit [1713] trained a single model on a heterogeneous mixture of recognition (segmentation, detection) and generation (insertion, removal, style change) tasks with learned task embeddings that route each input, and leads its own benchmark on precision and instruction following. Instruction editors are the most user-friendly mechanism, no mask and no inversion, at the cost of large supervised retraining.

Exemplar / reference-guided editing. A visual exemplar, an example image or patch encoding the desired content or style, replaces the textual instruction and enters through an auxiliary cross-attention pathway beside the standard text conditioning. Paint-by-Example [1714] fine-tunes Stable Diffusion to fill $\mathcal{M}$ with the content of an exemplar whose identity is encoded by a CLIP image encoder, with strong exemplar augmentation (cropping, color jitter) at training time to prevent copy-paste behavior and force semantic rather than pixel-level transfer. IP-Adapter [1715] generalizes this into a plug-in reference channel: a lightweight decoupled cross-attention head consumes a CLIP image embedding of the exemplar and is inserted into a frozen SD or SDXL backbone, so that any number of reference images can be combined with text prompts and with structural adapters such as ControlNet at inference time. It is the standard route for subject personalization and style transfer when the desired content is hard to describe in words.

Drag-based latent optimization. The most interactive family takes user-supplied (source point, target point) pairs and iteratively optimizes the latent so that the features at each source point migrate to the corresponding target while a locality regularizer pins the latent outside the user-marked region. DragGAN [1716] introduced the recipe on a StyleGAN backbone: a motion-supervision loss tracks the source feature and pulls it toward the target by gradient descent in the $\mathcal{W}^{+}$ latent space, and a point-tracking step re-localizes the moved feature at each iteration to prevent drift. DragDiffusion [1717] ported the recipe to the LDM latent space, running a per-timestep latent-update loop together with an identity-preserving LoRA fine-tune of the U-Net so that both the dragged subject and the unmarked context survive. Both are inference-time optimizers with latency proportional to the number of drag iterations, and no other family yet offers the same directness of spatial control.

Read across the five families, the mechanisms track the generative substrate: feed-forward inpainters such as LaMa gave way to inversion-based diffusion editors (Prompt-to-Prompt, Imagic), which in turn yielded to supervised instruction-tuned editors (InstructPix2Pix, Emu Edit) as paired data became available at scale. Orthogonal to that migration, the structural conditioning interfaces of §8.1.2 compose with every mechanism here, so a production pipeline pairs a mechanism with a structural channel rather than choosing one: Blended Diffusion with ControlNet-depth for geometry-aware text-guided edits, or IP-Adapter with DragDiffusion for exemplar-driven drag edits.

Training Corpora. Inpainting draws on Places2 [609] and Paris StreetView for scene-level masks, DIV2K for high-resolution patches, and the COCO / ADE20K mask supervision of §8.1.2 [440, 492]. Constraint-driven local synthesis reuses the structural channels of the previous subsection, RefCOCO / RefCOCO+ / RefCOCOg [1082] and PhraseCut [1092] for referring-expression masks, SA-1B masks [138], and COCO edge and depth predictions, with the convention that the mask is supplied or derived from the expression; geometry-guided editing adds depth and normal supervision from NYUv2 [504], Replica [339], and synthetic corpora, and insertion with harmonization uses composite datasets such as iHarmony4. Instruction editing has its own data ecology: the synthetic InstructPix2Pix corpus of Prompt-to-Prompt-generated triplets, the manually annotated MagicBrush triplets [1712], and the multi-task Emu Edit training mixture [1713].

Evaluation Benchmark Suites. The holistic suites of §8.1.1 (Table 8.1) carry over in part: T2I-CompBench [1676] and GenAI-Bench [1679] score the attribute and compositional fidelity that edited images must also satisfy, and HEIM [1673] includes alignment and quality axes that apply to edits. The dedicated suites are EditBench [1718], released with Imagen Editor, a curated set of text-guided inpainting prompts over object, attribute, and scene categories that serves as the standard evaluation for text-driven mask inpainting; the MagicBrush test set [1712], $1{,}053$ manually annotated single- and multi-turn triplets, which is the standard for instruction-tuned editors; and the Emu Edit benchmark [1713], which extends instruction editing to per-task subsets (insertion, removal, style change, color change, global change). Drag-based methods evaluate on small custom test sets curated by Pan et al. [1716] and Shi et al. [1717]; a community-wide drag-editing benchmark has not yet emerged.

Constraint-specific Metrics. Reconstruction quality (PSNR / SSIM / LPIPS against ground truth, with FID / KID for diverse-completion settings) is defined only where held-out content exists. Region adherence uses mask-restricted CLIPScore or region-level VQAScore to ask whether the edit honors the prompt inside $\mathcal{M}$; edit locality measures $\ell_{1}$ or LPIPS outside $\mathcal{M}$ against $I$, lower being better; harmonization is scored by color-statistic divergence, shadow-direction consistency, and the rate at which an off-the-shelf forgery detector flags the composite; identity preservation uses feature-cosine similarity over face or landmark regions for people and CLIP-image similarity for products and characters; and instruction following is paired accuracy, whether $\hat{I}$ honors the instruction better than $I$, judged by human raters or by a VLM over a structured rubric, as is now standard on MagicBrush and Emu Edit. Change inside $\mathcal{M}$ and preservation outside it are reported together; a method that wins on one and loses on the other has not solved the task.