Designing Listening Tests: MUSHRA, ABX and Statistics

Module J: Research methods and capstone. Lesson 19 of 20, 30 min.

Every claim that one spatial render is "more immersive", "more natural" or "transparent to the original" is a claim about human perception, and only a properly designed listening test can support it. This lesson teaches the standardized methods that reviewers expect, the spatial attributes worth measuring, and the statistics that turn a spreadsheet of ratings into a defensible conclusion. You will finish by analysing MUSHRA data in Python the way a journal referee would want to see it.

Why listening tests decide the argument

Objective metrics are useful for regression testing, but none reliably predicts how listeners judge envelopment, externalization or the plausibility of an upmixed scene. When you compare binaural renders, upmix strategies or a 5.1.4 bed against its ADM source, the final evidence is a controlled perceptual experiment. A weak experiment produces a confident number that collapses at a reviewer's first question.

A good test makes four decisions explicitly:

• What question is asked. Is the difference detectable at all, how large is the impairment relative to a reference, or which version do people prefer?
• Which method answers it. Detection calls for ABX or forced choice, small impairments for ITU-R BS.1116, intermediate quality for MUSHRA, and overall acceptability without a reference for ACR and mean opinion scores.
• Who listens, and under what conditions. Expert or naive listeners, headphones or loudspeakers, calibrated playback level, matched loudness.
• How the data are analysed. The model, the correction for multiple comparisons and the effect size should all be chosen before any data exist, ideally in a preregistration.

If you have not yet done the Critical Listening lab, skim it first. Your listeners will need the same vocabulary of timbre and space.

The rest of this lesson, its lab and self-check are part of Academy Advanced, a paid course. See who has access.