Exploring and Fine-Tuning Audio Models on Hugging Face
Module I: Build and ship your own audio model. Lesson 18 of 20, 30 min.
Abstract
The Hugging Face Hub offers a vast catalogue of audio models and datasets, but choosing well requires judgement about licences, training data, evaluation and engineering fit. This lesson teaches deliberate navigation of the Hub, critical reading of model cards, inference and fine-tuning with transformers and datasets, and responsible publishing with model cards and Gradio Spaces. It ends by connecting Hub models to MLflow and Databricks for governed serving.
By the end you can
- Search the Hub by task, library and licence, and shortlist audio models using a structured decision checklist
- Evaluate a model card for licence, gating, training data, evaluation fit, limitations and engineering constraints
- Run audio pipelines and fine-tune an audio classifier with transformers, datasets and Trainer at the correct sampling rate
- Publish a model with a complete card and Gradio Space, and register it in Databricks with the MLflow transformers flavour
The Hugging Face Hub hosts a very large number of public models, datasets and demo apps, and audio is one of its fastest-moving areas. For a researcher or product team, that abundance is both an opportunity and a trap. The right pretrained encoder can save months of work; the wrong one can bring an incompatible licence, undisclosed training data or a benchmark score that does not survive contact with real music. This lesson teaches you to navigate the Hub deliberately: how to search and filter, how to read a model card critically, how to run and fine-tune audio models with the transformers and datasets libraries, how to publish your own model responsibly, and how to bring Hub models into the Databricks pipeline from the previous lesson.
Navigating the Hub with intent
The Hub has three main kinds of repository:
- Models: weights, configuration and a model card (the
README.md), versioned with Git. - Datasets: data files or loading scripts plus a dataset card.
- Spaces: hosted demo apps, usually built with Gradio or Streamlit, that let you try a model in the browser.
Search by task, not by name
The model listing can be filtered by task. For audio, the most useful are:
| Task filter | What it does | Example use in spatial audio |
|---|---|---|
| Audio classification | Labels a clip | Stem role, instrument, scene type |
| Automatic speech recognition | Speech to text | Dialogue detection, subtitles, search |
| Audio-to-audio | Transforms audio | Enhancement, separation |
| Text-to-audio / text-to-speech | Generates audio from text | Sound design drafts, narration |
| Feature extraction | Returns embeddings | Similarity search, downstream classifiers |
| Zero-shot audio classification | Labels with free-text classes | Tagging without training data |
Further filters narrow by library (transformers, diffusers, others), language, licence and dataset. Sort by most downloads to find widely tested baselines and by trending to find new releases. Neither is a quality measure: popular models may be old, and trending models may be unproven.
A short map of audio model families
These families are well documented on the Hub and form a good starting toolkit:
- Whisper (OpenAI; Radford et al., 2023): multilingual speech recognition trained on large-scale weakly supervised data. Repositories such as
openai/whisper-large-v3. - wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021): self-supervised speech encoders, for example
facebook/wav2vec2-base-960handfacebook/hubert-base-ls960. - Audio Spectrogram Transformer (Gong, Chung and Glass, 2021, from MIT CSAIL): a transformer over spectrogram patches, with an AudioSet-fine-tuned checkpoint at
MIT/ast-finetuned-audioset-10-10-0.4593. - CLAP (LAION; Wu et al., 2023): contrastive language-audio pretraining for zero-shot classification and text-audio retrieval, for example
laion/clap-htsat-unfused. - EnCodec (Meta; Défossez et al., 2023): a neural audio codec producing discrete tokens, for example
facebook/encodec_24khz. Covered in depth in Audio Tokens and Neural Codecs. - MusicGen (Meta; Copet et al., 2023): text-conditioned music generation over EnCodec tokens, for example
facebook/musicgen-small. - Stable Audio Open (Stability AI): latent diffusion for sound effects and short music clips,
stabilityai/stable-audio-open-1.0, released under a custom licence and gated behind acceptance of its terms.
Most of these are mono or stereo models. Very few public models output true spatial formats such as Ambisonics or object metadata, which is exactly why fine-tuning and pipeline design, covered in Fine-Tuning a Spatial Scene LLM, remain valuable skills.
The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.
Key readings
- Wolf, T., Debut, L., Sanh, V., et al. (2020). Transformers: State-of-the-art natural language processing. EMNLP 2020: System Demonstrations.
- Mitchell, M., Wu, S., Zaldivar, A., et al. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229.
- Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.
- Gong, Y., Chung, Y.-A., and Glass, J. (2021). AST: Audio Spectrogram Transformer. Interspeech 2021.
- Wu, Y., Chen, K., Zhang, T., et al. (2023). Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. ICASSP 2023.