A common approach to improving vision-language alignment is to first align vision with vision, intuitively, enforcing geometric consistency such that visual features are anchored to a persistent world model before combining with language. Concretely, the goal of vision-vision representation learning is to fuse visual inputs (e.g., , multiview RGB) into a set of world-aligned scene tokens, such that they are invariant to viewpoint changes and can be later queried by language. Here, we discuss three broad families of approaches: lifting 2D features into 3D scene tokens, projecting these 3D tokens back into 2D for supervision, and using audiovisual correspondence to add constraints that further anchor the world model.
Lifting 2D to 3D follows two complementary directions. The most classical and canonical approach first estimates camera poses and geometry with structure-from-motion (SfM) and simultaneous localization and mapping (SLAM), then adds semantics to the generated map. Here, commonly used frameworks include COLMAP for SfM [115], KinectFusion and ElasticFusion for dense RGB-D mapping [114, 392], and ORB-SLAM2 for SLAM [393]. Learning-based reconstructions build upon this direction by back-projecting per-view features into a 3D volume and predicting TSDF or occupancy directly from posed images, for example, as in NeuralRecon [394]. Though components may differ, all of these approaches output view-agnostic scene tokens that can later be guided by linguistic supervision and queries.
Projecting 3D back to 2D provides self-supervision. Differentiable rendering in neural fields anchors world-aligned tokens to multiview observations, and enforces cross-view consistency [119]. Distilling 2D foundation features into 3D fields adds semantic knowledge without manual, expensive 3D labels. Here, works typically transfer 2D features into a co-registered 3D feature volume, which can be rendered to any view [395, 396]. In addition, distillation is not limited to 2D language targets alone; for example, ImageBind [397] proposes a multisensor feature target (image-text-audio-depth-thermal-IMU), which enables transfer of cross-sensor semantics even when there are few explicit labels. Although these distillations are vision-vision processes, they are explicitly designed for language alignment later. The key benefit is that now vision-language alignment is conducted on a geometry-consistent representation, rather than directly on view-specific features.
Audio-visual alignment can further constrain this vision-vision world model; self-supervised objectives have been designed to enforce consistency between audio and visual motion, which allows models to localize sound sources and disambiguate co-occurring events [398, 399, 400]. Such cues provide directional or timing semantics that help anchor linguistic queries to specific world-aligned entities, especially in cases with multiple candidate entities. Across this variety of approaches, vision-vision representation learning methods all aim to learn a persistent world model that is geometrically and temporally consistent before adding any linguistic supervision, therefore making downstream cross-modal alignment more data-efficient and spatially faithful to the physical world.