We define four axes that structure this cross-modal representation learning problem, which we use to organize the remainder of the chapter. The first axis is the alignment strategy, which we categorize into: contrastive alignment, generative alignment, and programmatic alignment. Contrastive alignment learns a shared embedding space by pushing together paired samples and pulling away negative samples. Such alignment scales to weakly paired data and enables retrieval and coarse grounding, but often needs additional spatial supervision to understand fine-grained relations grounded to specific visual regions. In contrast, generative alignment predicts across modalities (e.g., , captioning from vision or language-conditioned decoding into images, 3D, or video), which gives rich supervision and coverage, but also tends to drift from true underlying geometry unless it is further constrained, for example, by multi-view consistency. Finally, programmatic alignment aligns visual entities to explicit symbols (e.g., , attributes, relation graphs, or concepts in domain-specific languages), which define spatial semantics and naturally interface with program-based, verifiable reasoning, but often requires stronger supervision and trades off expressiveness for structure.
The second axis is spatial modality, which ranges from view-aligned 2D to world-aligned 4D. Here, two-dimensional tokens (e.g., , pixels, patches) are expressive and broadly applicable, but rely on some form of geometric structure to capture faithful spatial relationships. Three-dimensional tokens (e.g., , points, meshes, implicit fields, Gaussian primitives) better encode metric scale and rigid-body invariance (i.e., SE(3) consistency: rotations and translations of the scene do not change the underlying semantics), so that linguistic queries are grounded in world coordinates rather than in image projections. However, they are less ubiquitous and often rely on partial and noisy reconstructions of geometry. Extending 3D through time yields 4D traces (e.g., , tracks), that bind actions to expressive motion and change, but significantly increase computational complexity and require reliable alignment data, while being sensitive to complex semantics such as identity switches. These different spatial modalities correspond to and build upon the single-modality representations of Chapter 4.
The third axis is the language interface, from free-form text to structured interfaces. Free-form text (e.g., , natural language) enables open-vocabulary understanding for referring expressions, retrieval, and instruction following, but comes with corresponding ambiguity in terms of frames of reference, scale, and units. In contrast, structured interfaces (e.g., , programs) reduce such ambiguity by explicitly describing spatial semantics, enabling integration with symbolic reasoning, but are less flexible in capturing the full range of natural language descriptions.
The fourth axis is the training regime, which defines both the level of supervision and coverage. Paired supervision, for example, from captioned videos or referring expressions in 3D scenes, gives strong alignment, but is expensive and can be biased toward frequent compositions and co-occurrences. Weakly paired internet data, for example, from alt-text or transcripts, enable diversity, but, due to low signal-to-noise ratios, often yield noisier spatial grounding.
Below, we organize the chapter by cross-modal pair, using spatial modality (which pair) and alignment strategy (which objective) as the primary organisational axes; the language interface and training regime axes recur as cross-cutting attributes referenced throughout. We give an overview of 2D-text alignment (Section 5.2), 3D-text alignment (Section 5.3), audio-text alignment (Section 5.4), and vision-vision alignment (Section 5.5); each follows the scope and axes defined here.