Audio-Visual Source Localization

Module C: Computer vision for spatial audio. Lesson 5 of 20, 45 min.

When a picture shows a trumpet, a barking dog and a passing bus, a human viewer knows almost instantly which pixels are making which sound. Teaching a machine to make that same link, with no human labels, is the problem of audio-visual source localization. It is the first computer vision building block for any system that wants to place sounds where they appear on screen, and this lesson covers the research that made it work, the maths behind a localization heatmap, and where these models fail.

Why vision belongs in a spatial audio pipeline

A spatial mix is a set of decisions about where each sound lives. With music stems those decisions come from the producer, from the arrangement and from inference over the audio itself (see Spatial Inference). When there is a picture, the picture adds a second, very strong constraint: if the singer stands on the left of the frame, the voice should usually come from the left. The core lesson on Immersive Video Sound shows that principle from a practitioner's side. This module looks at the machine learning underneath it.

The core idea is simple to state. Sound and vision describe the same events, so they are naturally paired: every video frame comes with the audio recorded at the same moment. That pairing is free supervision. A network can learn to tell whether an image and an audio clip belong together, and once it can do that, it can also be asked which part of the image best matches the audio. The answer is a heatmap over the frame, and the peak of that heatmap is a candidate direction for an audio object.

Over the rest of the lesson we will cover:

• Self-supervised correspondence: networks trained only on the fact that frames and audio co-occur (Look, Listen and Learn; Objects that Sound; temporal alignment).
• Localization heatmaps: how patch-level image embeddings and a clip-level audio embedding combine into a cosine-similarity map, and how hard negatives sharpen it.
• Separation conditioned on pixels: Sound of Pixels and Looking to Listen, which go from "where is the sound" to "give me the sound of that region".
• Shared embedding spaces and datasets: CLIP, ImageBind, AudioSet and VGGSound.
• Failure modes: off-screen sources, silent objects, and the ventriloquism effect, which is both a model failure and a perceptual gift.

The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.