The most developed cross-modal setting pairs images with language. Its defining tension is that a text-aligned 2D representation lives in the image projection, not the world: appearance and layout sit on the pixel grid, yet the relations a caption asserts (left of, above, behind) are only implicit there. How faithfully those relations survive alignment turns on two choices: the granularity at which the image is represented, and the objective that binds it to text, and this section is organized around them. Models often fall into three broad families of visual representations, across different levels of granularity. First are global embeddings that compress an entire image into a single vector paired with a sentence embedding. Here, dual-encoder approaches such as CLIP [143] and ALIGN [344] show that large-scale contrastive pretraining over noisy web data yields strong zero-shot classification and retrieval performance. Second, many works have proposed local tokens tied to image subregions, in which vision-language models, often transformers (e.g., , ViLBERT [24], LXMERT [25], UNITER [345], OSCAR [346]) input region features extracted by bottom-up detectors (e.g., , Faster R-CNN [172]) into cross-modal attention blocks, enabling word-region alignment, and hence fine-grained reasoning over object proposals. A third class of methods makes relations explicit; here, models extract intermediate representations that are directly aligned with spatial predicates such as “left-of” or “on.” NSCL [347] and LEFT [348] learn hybrid representations that explicitly bind named visual concepts with latent features. This representational spectrum, from global to region to relation-based, forms the basis for the objectives below.
Contrastive retrieval serves as a common objective for large-scale pretraining, as it learns a shared metric space, without requiring dense and expensive supervision. As representative examples, CLIP [143] and ALIGN [344] maximize agreement between paired image and text embeddings, enabling robust zero-shot transfer; LiT [349] shows that contrastively tuning only the text encoder on top of a locked image encoder can further improve such transfer; and CoCa [350] fuses contrastive learning with captioning in a single encoder-decoder architecture to combine discriminative and generative strengths. Another line of work adapts contrastive alignment to more fine-grained regions. RegionCLIP [351] lifts global contrast to region-text pairs, FILIP [352] proposes late token-token interaction to align words with patches efficiently, X-VLM [353] learns multi-grained alignment such that entities and attributes can be localized and described consistently across granularities, and ALBEF [354] conducts image-text contrast with cross-modal fusion and momentum-distilled pseudo-targets to stabilize learning on noisy web data.
In addition, masked modeling with textual conditioning has been shown to provide a complementary self-supervised signal that encourages cross-modal consistency. LXMERT [25] and UNITER [345] pretrain with masked language modeling, masked region modeling, and image-text matching to build word-region alignment. VLMo [355] generalizes this idea with a mixture-of-modality experts, and BEiT-3 [356] performs masked data modeling over images, text, and image-text pairs in a single multiway transformer.
With more labels, generative objectives can be applied for this representation learning problem, and also complement self-supervised ones. Here, methods teach models to produce language conditioned on images, intuitively, to generate tokens that act as spatially grounded explanations. As representative examples, SimVLM [357] shows that a prefix language-modeling objective trained on weak supervision can improve performance on captioning and visual question answering, BLIP [358] adds caption bootstrapping to denoise web supervision, and GIT [359] simplifies a scalable image-encoder and text-decoder architecture.
In a more fine-grained manner, grounding objectives can directly optimize spatial correspondence between language and 2D pixels, points, or boxes. Here, text-conditioned detection and phrase grounding unify detection and language. MDETR [360] learns a DETR-style [136] detector modulated by text, GLIP [361] pretrains on both detection and grounding corpora and self-generates boxes from image-text pairs to scale supervision, and Grounding DINO [362] augments strong detection backbones with grounded pretraining for open-set detection. At the pixel level, DenseCLIP [363] adapts image-text matching to pixel-text matching for dense prediction, and GroupViT [364] shows that segmentation structure can appear from text supervision alone.
Across these categories, we see that 2D-text representations can emerge from diverse approaches, for example, differing in whether spatial structure is implicit (e.g., , in attention over patches) or explicit (e.g., , in region tokens), and whether supervision is weakly paired and contrastive, generative, or explicitly grounded. Models that learn 2D vision-language representations now increasingly combine all contrastive, captioning, grounding, and masked objectives to solve a wide range of downstream tasks, from (more semantic) referring expression comprehension to (more visually grounded) localization.