Moving from the image plane to the scene itself removes the projection gap that limits 2D-text alignment: language is now bound to geometry expressed in world coordinates, so that metric scale and the invariance of meaning under rotation and translation become properties the representation can actually enforce. The difficulty shifts accordingly, from recovering relations that pixels hide to obtaining and aligning 3D structure that is often partial, noisy, and expensive to annotate. As before, the representation families and the alignment objectives acting on them jointly organize the section. Two complementary families of representations have emerged in this domain. First, object-centric and scene-centric encoders learn world-aligned tokens, then align them to language via contrastive or generative objectives. ULIP [365] instantiates this paradigm by aligning point-cloud features to an image-text space, scaling supervision through synthetic triplets. OpenShape [366] extends this direction further with curated 3D corpora and hard negative mining to support open-world 3D retrieval, and OpenScene [39] co-embeds dense features for 3D scenes with text and image pixels in CLIP feature space. Additional projection strategies have also been explored, for example, adapting 2D vision-language models to 3D by rendering depth or multi-view images and transferring CLIP’s alignment to point clouds; PointCLIP [367] and CLIP2Point [368] do so and enable adapters for cross-modal transfer.
A second family binds language to implicit scene fields. Here, LERF [369] optimizes a dense 3D language field inside NeRF [119], by rendering CLIP embeddings across views and enables open-vocabulary queries that return relevancy volumes. CLIP-Fields [370] similarly distills 2D vision-language cues into 3D implicit maps to support semantic search and view localization. In addition, replacing NeRF with 3D Gaussian Splatting [121] yields efficient language fields; for example, LangSplat [371] combines CLIP features with Gaussians for fast, precise open-vocabulary querying. These methods build on advances in explicit 3D primitives and real-time rendering to maintain high-fidelity representations while training efficiently.
In this 3D-text domain, objectives also follow the representation choices. Contrastive alignment remains common, for example, ULIP trains with InfoNCE [372] over point clouds-images-text, while OpenShape scales both data and hard-negative mining. Projection-based approaches transfer contrastive objectives to depth, and multi-view renders to align point clouds with text labels. Grounded objectives enforce explicit 3D correspondence between language and geometry. 3D referring expression datasets and models (e.g., , ScanRefer [373], ReferIt3D [374], 3D-VisTA [375], BUTD-DETR [376], NS3D [377], InstanceRefer [378]) localize targets from language descriptions and learn to understand complex relations dependent on viewpoints.
Here, geometry-aware representations are critical for aligning language with the physical world. Ideal 3D-text representations should respect SE(3) structure (e.g., , rotations and translations should not spuriously change semantics), yet cross-modal alignment should still encode view-dependent language (e.g., , “the chair to the left of the table”). To this end, priors from equivariant architectures, such as the SE(3), Transformer [379] and Vector Neurons [380], can be integrated into 3D-text models. Implicit fields enable different options; while NeRF-based language fields are not strictly equivariant, multi-view consistency and cross-view regularization can improve their spatial coherence. Alternatively, Gaussian splatting uses a canonical world frame and enables efficient supervision during language-field optimization. Overall, these representation choices and objectives give rise to a wide range of approaches for 3D-text alignment, from tri-modal contrastive pretraining to implicit or splat-based language fields.