Depth, Tracking and Moving Sound from Video
Module C: Computer vision for spatial audio. Lesson 6 of 20, 30 min.
Abstract
A localization heatmap is not yet a trajectory. This lesson builds the full chain from video to audio-object motion: monocular depth, detection, segmentation and multi-object tracking, Kalman smoothing, and the geometry that maps pixels to azimuth, elevation and distance. It closes with 360 video, learned binaural and Ambisonic generation, and neural acoustic fields.
By the end you can
- Convert pixel positions and depth into listener-relative azimuth, elevation and distance using a pinhole camera model and field of view
- Compare SORT, DeepSORT and ByteTrack and mitigate identity switches that would make audio objects jump
- Implement a constant-velocity Kalman filter that produces smooth, occlusion-tolerant object-audio trajectories
- Design perceptually plausible distance mappings using level, direct-to-reverberant ratio and spectral cues
Knowing which pixels make a sound is only half the job. To drive an audio object you also need to know how far away the source is, which object it is from one frame to the next, and how to turn a moving box on a flat image into a smooth path around a listener. This lesson builds that chain, from monocular depth and tracking to the geometry that yields azimuth, elevation and distance, and then looks ahead to 360 video and neural scene representations.
From pixels to positions: the pipeline at a glance
The previous lesson ended with a localization heatmap. A renderer for Dolby Atmos (ADM BWF), Eclipsa Audio (IAMF) or binaural cannot use a heatmap directly; it wants a trajectory, a position per object over time. The core lesson on Immersive Video Sound describes this from the mixing chair. Here we look at the computer vision stack underneath.
A practical pipeline has five stages, each with its own failure mode:
| Stage | Question it answers | Typical models or tools | Main failure |
|---|---|---|---|
| Detection | What objects are in this frame, and where? | YOLO family | Missed or duplicated boxes |
| Segmentation | Exactly which pixels belong to the object? | SAM, SAM 2 | Mask leaks into background |
| Tracking | Is this the same object as last frame? | SORT, DeepSORT, ByteTrack | Identity switches |
| Depth | How far is each pixel from the camera? | MiDaS, Depth Anything | Unknown scale, flicker |
| Geometry and smoothing | Where is it relative to the listener? | Pinhole model, Kalman filter | Wrong field of view, jitter |
Two points shape the rest of the lesson. First, every stage is noisy, and visual noise becomes audible motion: a box that wobbles by a few pixels becomes a voice that shimmers left and right. Second, hearing is far less precise than a camera. Azimuth blur is a few degrees at best in front of the listener and much larger to the sides and above (see How We Hear in 3D). A good pipeline spends its effort on stability and plausibility, not sub-pixel accuracy.
The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.
Key readings
- Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., & Koltun, V. (2022). Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3).
- Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., & Wang, X. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. European Conference on Computer Vision (ECCV).
- Kalman, R. E. (1960). A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1), 35-45.
- Gao, R., & Grauman, K. (2019). 2.5D Visual Sound. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Zahorik, P., Brungart, D. S., & Bronkhorst, A. W. (2005). Auditory Distance Perception in Humans: A Summary of Past and Present Research. Acta Acustica united with Acustica, 91(3), 409-420.