Psychoacoustic Models Meet Deep Learning

Module H: Advanced psychoacoustics and AI. Lesson 15 of 20, 40 min.

Every spatial audio system makes a bet about human hearing. A perceptual codec bets that you will not notice the quantization noise it hides under louder sounds. A binaural renderer bets that your brain will accept someone else's ear filters. A neural codec trained with a mel-spectrogram loss bets that errors the loss ignores are errors you cannot hear. This lesson takes those bets seriously. It covers the classic psychoacoustic models that engineers have relied on for forty years, the perceptual metrics and losses that bring those models into machine learning, and a newer line of research, led in part by the Laboratory for Computational Audition at MIT, that uses deep neural networks as scientific models of hearing itself.

Why psychoacoustics is the hidden layer of audio AI

Psychoacoustics is the quantitative study of the relationship between physical sound and perception. Its core findings are compact enough to fit in a few rules, but those rules quietly shape almost every audio product:

• Frequency resolution is not uniform. The cochlea behaves like a bank of overlapping band-pass filters whose bandwidth grows with centre frequency. Two tones inside one filter interact strongly; two tones in distant filters are processed almost independently.
• Loud sounds hide quiet ones. A masker raises the threshold of audibility for other sounds near it in frequency (simultaneous masking) and, more weakly, just before and after it in time (temporal masking).
• Loudness is not level. Perceived loudness depends on frequency, bandwidth and duration, so two signals with the same RMS level can differ greatly in loudness.
• Two ears are better than one. Interaural differences let the auditory system localize sources and pull a target out of noise, an advantage known as binaural unmasking.

Each rule has an engineering consequence. Perceptual codecs allocate bits per critical band and push quantization noise below the masking threshold. Loudness normalization standards weight frequencies according to perception. Spatial renderers rely on interaural cues and spectral cues described in the core lesson How We Hear in 3D.

Machine learning adds a twist. A neural network trained to minimize a loss learns whatever the loss rewards. If the loss is a plain waveform mean squared error, the model will spend capacity on phase details that are often inaudible and may neglect spectral detail that is clearly audible. If the loss is perceptually informed, the model's errors move to where listeners are least sensitive. In other words, psychoacoustics is the hidden layer that decides what "good" means for an audio model.

The research direction also runs the other way. Since 2018, a series of studies has trained deep networks on natural auditory tasks and then asked whether the networks behave like people. When they do, the networks become testable models of hearing; when they do not, the failures reveal what the models are missing. For spatial audio this matters directly: a network that localizes sounds like a human could, in principle, predict front-back confusions or externalization failures before anyone runs a listening test.

The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.