Audio Tokens and Neural Codecs

Module B: Training your own audio AI. Lesson 3 of 20, 30 min.

Every modern audio language model, from speech continuation systems to text-to-music generators, rests on one quiet design decision: how to turn a continuous waveform into a sequence of discrete symbols a Transformer can predict. This lesson traces that decision from sample-level autoregression to residual vector quantization and today's neural codecs. By the end you will understand what an "audio token" actually is, why there are several kinds, and why spatial and multichannel audio make tokenization harder than it looks.

Why audio needs tokens

Language models are next-symbol predictors. Text arrives pre-segmented into subwords, but a 48 kHz stereo file produces 96,000 numbers every second, while musical structure unfolds over seconds or minutes. Modelling samples directly is brutally expensive, because attention cost grows with sequence length.

An audio tokenizer solves this by learning a compressed, discrete representation: a small number of integer codes per short frame of audio, chosen from a learned vocabulary (a codebook). Once audio is a sequence of integers, the full machinery of language modelling applies: cross-entropy training, sampling, prompting, and conditioning on text.

There are two broad families of tokens you will meet throughout this module:

• Acoustic tokens come from neural codecs such as SoundStream, EnCodec and DAC. They are trained to reconstruct the waveform faithfully, so they capture timbre, recording conditions and fine detail.
• Semantic tokens come from self-supervised encoders such as wav2vec 2.0 or HuBERT, typically by clustering their internal features. They discard much acoustic detail but capture longer-range, more linguistic or structural content.

Systems like AudioLM combine the two in a hierarchy, while MusicGen shows that a single well-organised stream of acoustic tokens can be enough for music. For a practitioner building spatial audio tools, the core question is the same: which representation preserves the information your downstream task needs, at a sequence length your model can afford? If you want a refresher on why spatial cues are so fragile to begin with, revisit How We Hear in 3D before continuing.

The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.