Sound aligns to language along a different primary dimension than vision does. What an acoustic signal encodes first is time, the sequence of events as they unfold, and only secondarily, and coarsely, does it carry spatial attributes such as direction of arrival or distance. Audio-text representation learning therefore aims for embeddings that are open-vocabulary and temporally grounded, with spatial awareness added where the labels permit. The section again moves from the representation granularity to the objectives that align it with text. One such direction is to embed audio-text representations at a coarse level, for example, with global encoders mapping an entire audio clip and a full sentence into a joint space, enabling zero-shot retrieval. Here, CLAP [381] and LAION-Audio [382] scale contrastive language-audio pretraining with curated datasets, and show strong transfer across retrieval benchmarks. In addition, AudioCLIP [383] extends CLIP to a tri-modal image-text-audio setting, to leverage visual-text semantics, and Wav2CLIP [384] distills CLIP’s alignment into an audio encoder when audio-text pairs are scarce.
Beyond global representations, a line of work that aims to extract representations at a more fine-grained scale learns temporally local structure, for example, by operating on spectrogram patch tokens with transformer architectures. The Audio Spectrogram Transformer (AST) [385] and PaSST [386] show that attention over time-frequency patches captures long-range temporal dependencies, and HTS-AT [387] further returns frame-level maps that are useful for event localization. These tokenizations make it straightforward to bind subclip semantics to text (e.g., , “a door slams, then footsteps echo”). Self-supervised masked objectives, e.g., , SSAST [388], MAE-AST [389], and Audio-MAE [390], additionally pretrain such backbones by reconstructing masked patches, while BEATs [391] adds an iterative tokenizer-encoder scheme that encourages high-level semantic abstraction.
Across advances in representations and objectives, an emergent and increasingly common paradigm is to first pretrain an audio backbone with masked modeling, then align embeddings to text via contrastive losses, and finally, optionally add a captioning head to increase supervision. In tasks where spatial semantics is important, incorporating spatial labels, such as direction-of-arrival, helps refine the audio-text representation space. By leveraging these approaches, models can learn to align predicates such as “left-of” or “far” with meaningful acoustic features. These methods enable models to learn embeddings that are not only open-vocabulary and temporally grounded, but also (coarsely) spatially aware.