# How to Test Spatial and Immersive Audio: BS.1116, MUSHRA, ABX/2AFC and ITU-T P.800 ACR Explained

> Which listening test should you use for spatial and immersive audio? A practical guide to ITU-R BS.1116, MUSHRA (BS.1534), ABX and 2AFC, and ITU-T P.800 ACR, with scales, statistics, sample sizes and a step-by-step protocol for headphones and 3D loudspeaker rooms.

Web page: https://spatial9.ai/blog/spatial-audio-listening-tests-bs1116-mushra-abx-p800-acr
Author: Spatial9 Research Team
Published: 2026-11-21. Updated: 2026-11-21.

## Key takeaways
- Pick the method by the size of the difference: ABX or 2AFC to detect it, BS.1116 to grade small impairments, MUSHRA to rank several intermediate systems, P.800 ACR for everyday opinion.
- BS.1116 uses a known reference plus a hidden reference and reports a Subjective Difference Grade from 0 to minus 4.
- MUSHRA rates many systems at once on a 0 to 100 scale with a hidden reference and anchors. For spatial tests, add a spatial anchor such as a mono fold-down.
- ABX needs enough trials: 12 of 16 or 15 of 20 correct for p below 0.05, and short tests often miss real but small differences.
- P.800 ACR gives a Mean Opinion Score from many naive listeners with no reference, and P.808 adapts it for online testing.
- For spatial audio, match loudness with BS.1770, report the loudspeaker layout or headphones, HRTF and head tracking, and rate attributes such as localisation, externalisation and envelopment.

Quick answer: To test spatial and immersive audio, pick the method by the size of the difference you expect. Use ABX or 2AFC to prove a difference can be heard at all. Use ITU-R BS.1116 to grade small impairments with expert listeners. Use MUSHRA (ITU-R BS.1534) to rank several systems of intermediate quality against a hidden reference and anchors. Use ITU-T P.800 ACR to collect a Mean Opinion Score from many ordinary listeners with no reference. For spatial work, also match loudness, fix and report the playback (loudspeaker layout or headphones, head tracking, HRTF), and ask about spatial attributes, not only "quality".

Every immersive audio claim eventually meets the same question: did people actually hear it?

A renderer sounds wider. A new codec sounds "transparent". A head-tracked binaural mix feels more natural. Those are hypotheses. A listening test is how you turn them into evidence that a reviewer, a standards body or a client will accept.

The four methods in this guide are the backbone of perceptual audio research. They were not invented for spatial audio, but they are the ones you will see in almost every paper, codec evaluation and product claim about 3D sound. Choose the wrong one and you either miss a real effect or publish a number that means nothing.

[Image: Match the method to the question. Tiny differences need a detection test. Small ones need BS.1116. Several systems of medium quality fit MUSHRA. Everyday opinion fits P.800 ACR.]


## The one question to answer before choosing a method

Before you open any test software, write one sentence: what decision will this test inform?

- "Can listeners hear any difference between the original and the encoded file?" That is a detection question. Use ABX or 2AFC.
- "How much worse is the encoded file, when the difference is small?" That is a small impairment grading question. Use BS.1116.
- "Which of these five renderers sounds best, and by how much?" That is a multi-system comparison of intermediate quality. Use MUSHRA.
- "What would ordinary people think of this experience on their own devices?" That is an absolute opinion question. Use P.800 ACR.

Spatial audio adds a second layer: quality of what? Overall quality, localisation accuracy, envelopment, externalisation, timbre and source width are different things. A system can win on envelopment and lose on timbre. Decide which attributes matter before you choose the scale.


## ITU-R BS.1116: grading small impairments

ITU-R BS.1116 is the method for detecting and grading small impairments, the kind that only trained listeners hear in a quiet, controlled room. It is the reference method when a codec or processing chain is meant to be close to transparent.


### How a BS.1116 trial works

Each trial is double-blind, triple-stimulus with a hidden reference. The listener gets three buttons:

- Reference, always the original and labelled as such.
- A and B, one of which is the hidden reference and the other the system under test, in random order.

The listener grades A and B against the known reference on a continuous five-grade impairment scale from 1.0 to 5.0, with labels from "Imperceptible" (5.0) to "Very annoying" (1.0). One of the two must get 5.0, because the listener believes it is the hidden reference.

[Image: What the listener actually sees. BS.1116 grades impairment, MUSHRA rates quality from 0 to 100, ACR uses five categories and ABX is a forced choice.]


### The result: the Subjective Difference Grade

Results are reported as the Subjective Difference Grade (SDG): the grade given to the system under test minus the grade given to the hidden reference. An SDG of 0 means "no audible difference". An SDG of minus 4 means "very annoying". Because the listener can mistake the hidden reference for the test signal, a positive SDG on a trial is possible and is a useful sign that the difference was not heard.


### What BS.1116 demands

- Expert listeners, trained on the specific artefacts, and post-screening to remove listeners who could not reliably tell the reference from the test signal.
- A controlled listening room with specified background noise, reverberation and loudspeaker setup, or high-quality headphones.
- Short excerpts, typically around 10 to 20 seconds, chosen to be critical, meaning the material most likely to reveal the impairment.
- Around 20 listeners is often enough once they are expert and screened.


### When to use it for spatial audio

Use BS.1116 when your spatial system should be almost indistinguishable from a reference: an Eclipsa Audio (IAMF) encode against its uncompressed source, a higher-order Ambisonics decoder against a known-good one, or a binaural render against the same render at a higher bitrate. If most of your systems are clearly impaired, BS.1116 wastes listener time. Move to MUSHRA.


## MUSHRA (ITU-R BS.1534): comparing several systems

MUSHRA stands for MUltiple Stimuli with Hidden Reference and Anchor. It is defined in ITU-R BS.1534 and is designed for intermediate quality, where the differences are clearly audible and you want to rank several systems in one go.


### How a MUSHRA trial works

On one screen, the listener sees a labelled reference and a set of unlabelled stimuli, each with its own slider. Among the stimuli are:

- the hidden reference, which should be rated near 100,
- one or more anchors, deliberately degraded versions that calibrate the low end of the scale. The standard anchor is a 3.5 kHz low-pass version of the reference, with a 7 kHz low-pass version as a mid anchor,
- the systems under test.

Each stimulus is rated on a continuous 0 to 100 quality scale divided into five bands: Excellent, Good, Fair, Poor and Bad. Listeners can switch between all stimuli freely and in sync, which makes MUSHRA very efficient: one trial gives you a rating for every system.


### Post-screening

Listeners who rate the hidden reference below 90 for more than 15% of the test items are typically excluded. This is the main defence against listeners who are guessing or not listening carefully.


### Anchors for spatial audio

A 3.5 kHz low-pass anchor tells you about timbre, not about space. For spatial tests, many researchers add a spatial anchor as well, such as a mono fold-down or a strongly narrowed image, so the bottom of the scale means "no spatial impression" rather than "dull sound". Say exactly which anchors you used. Anchors shape the scale, and results with different anchors are hard to compare.

[Image: An illustrative MUSHRA result. The hidden reference sits near 100, the anchors at the bottom, and the systems under test in between.]


### Reading a MUSHRA chart correctly

MUSHRA results are usually shown as means with 95% confidence intervals. Two things trip people up:

- Overlapping intervals do not prove "no difference". Each listener rated every system, so the data is paired. Test the per-listener differences with a repeated-measures model, not by eyeballing error bars.
- MUSHRA scores are relative. A "Good" score of 70 means good compared with this reference and these anchors. It is not an absolute quality and should not be compared with a 70 from another test.


### When there is no reference

Immersive audio often has no single "correct" version. Is the 7.1.4 mix the reference for a binaural render, or the producer's intent? When no reference exists, ITU-R BS.2132 describes a multi-stimulus method without a given reference. Use it, or run an attribute-based MUSHRA-style test where listeners rate a defined attribute such as envelopment, rather than pretending one version is perfect.


## ABX and 2AFC: is there any difference at all?

Sometimes you do not need a grade. You need to know whether a difference can be heard at all. That is what ABX and two-alternative forced choice (2AFC) tests are for.


### ABX

The listener hears A, hears B, and then hears X, which is secretly either A or B. They must say which. A single trial says almost nothing, since anyone has a 50% chance of guessing right. Over many trials, the number of correct answers tells you whether the listener is doing better than chance.


### 2AFC and its relatives

In 2AFC, the listener hears two stimuli and must choose the one with a property, for example "which one is wider?" or "which one comes from above?". It is the workhorse of psychoacoustics, used to measure thresholds such as the smallest audible change in elevation or source width. Related designs include triangle tests (which of three is the odd one out) and same/different tests.


### The maths: how many trials?

ABX results follow a binomial distribution. With a 5% significance level, a one-sided test needs:

| Trials | Correct answers needed (p < 0.05) |
| 10 | 9 or more |
| 16 | 12 or more |
| 20 | 15 or more |
| 25 | 18 or more |
| 50 | 32 or more |
| 100 | 59 or more |

The table hides the real trap: statistical power. If a listener can truly hear the difference 70% of the time, a 16-trial ABX catches it less than half the time. A "not significant" result from a short ABX test is not evidence that no difference exists.

[Image: More trials, more power. A listener who is right 70% of the time needs about 30 trials before a test is likely to catch them.]


### When to use ABX or 2AFC for spatial audio

- Checking whether a lossy immersive encode is transparent before you run a full BS.1116 test.
- Measuring thresholds: the smallest audible head-tracking latency, the smallest audible change in a source position, the point where an HRTF swap becomes noticeable.
- Validating a claim quickly, such as "listeners can tell head-tracked from static binaural".

Always balance the order, randomise which of A and B is X, and decide the number of trials before you start. Stopping when the score looks good is a classic way to fake significance.


## ITU-T P.800 ACR: the Mean Opinion Score

ITU-T P.800 comes from telephony. Its best-known method is Absolute Category Rating (ACR): the listener hears one stimulus at a time, with no reference, and rates it on a five-point category scale:

- 5: Excellent
- 4: Good
- 3: Fair
- 2: Poor
- 1: Bad

The average of those ratings is the Mean Opinion Score (MOS). P.800 also defines Degradation Category Rating (DCR), which compares a stimulus with a reference, and Comparison Category Rating (CCR), which rates the second of a pair on a scale from minus 3 to plus 3. For online and crowdsourced listening, ITU-T P.808 adapts ACR to remote participants with built-in quality checks.


### Why ACR matters for immersive audio

BS.1116 and MUSHRA tell you what experts hear in perfect conditions. ACR tells you what ordinary listeners think on their own headphones and phones. For streaming, gaming, social video and music apps, that is often the question that decides a product.

ACR has known weaknesses. The categories are coarse, people avoid the ends of the scale, and the score depends on the range of quality in the test. A MOS of 4.1 in one test and 4.1 in another are not automatically the same. Always include a few known reference conditions so results can be anchored.


## Side-by-side comparison

|  | ITU-R BS.1116 | MUSHRA (ITU-R BS.1534) | ABX / 2AFC | ITU-T P.800 ACR |
| Question | How small is the impairment? | How do several systems rank? | Is there a difference at all? | What do people think of it? |
| Quality range | Near transparent | Intermediate | Any, especially tiny | Any, everyday |
| Reference | Known plus hidden | Known plus hidden, with anchors | A and B are both known | None |
| Scale | Continuous 1.0 to 5.0 impairment | Continuous 0 to 100 | Right or wrong | Five categories, MOS |
| Listeners | Expert, screened | Expert, screened | Expert or trained | Many, naive |
| Typical size | About 20 | About 15 to 20 after screening | Many trials per person | Dozens to hundreds |
| Main output | Subjective Difference Grade | Mean score with 95% CI | Correct count, p-value | Mean Opinion Score |
| Best spatial use | Codec and decoder transparency | Renderers, HRTFs, upmixers | Thresholds, quick claims | Consumer experience |


## How to test spatial and immersive audio properly

The method is only half of the test. In spatial audio, these details decide whether a result is trusted.

[Image: Six steps that keep a spatial listening test honest, from one clear question to pre-planned analysis.]


### 1. Match loudness first

Louder sounds better, and wider often sounds louder. Before any test, loudness-match every stimulus using ITU-R BS.1770 integrated loudness, ideally within about 0.5 LU, and check by ear. Unmatched loudness is the single most common reason a spatial system "wins" for the wrong reason.


### 2. Fix the playback and report it

- Loudspeakers: use a standard layout from ITU-R BS.2051, such as 4+7+0 (often called 7.1.4), calibrated per channel for level and delay. Report the room.
- Headphones: report the model, the equalisation, the HRTF used (generic, selected or individual) and whether head tracking was on. Head tracking changes externalisation and front-back confusion dramatically, so never mix tracked and untracked sessions in one result.


### 3. Rate attributes, not just "quality"

For spatial audio, overall quality hides the interesting part. Consider separate scales or separate tests for:

- Localisation accuracy: is the sound where it should be?
- Externalisation: does it sound outside the head?
- Envelopment and spaciousness: does it surround the listener?
- Apparent source width and depth.
- Timbre and colouration: did the spatial processing change the tone?
- Front-back and up-down confusions.

Vocabularies such as the Spatial Audio Quality Inventory (SAQI) give consistent definitions for these attributes, which makes results comparable across studies.


### 4. Train listeners on what they will hear

Let every listener hear all conditions, including the anchors, before the real test. Explain each attribute with an example. Untrained listeners often rate "different" as "worse", which punishes any spatial system that changes the image at all.


### 5. Keep it blind and randomised

Randomise the order of trials and the position of each system on screen. Hide system names. In head-tracked tests, keep the visual scene the same across conditions, because a picture strongly changes where people hear a sound.


### 6. Plan the statistics before the test

Decide the number of listeners, the screening rule and the analysis before collecting data. For MUSHRA and BS.1116, a repeated-measures ANOVA or a linear mixed model with listener as a random factor is standard. When the data are far from normal, use non-parametric tests such as Wilcoxon signed-rank or Friedman. Report effect sizes and confidence intervals, not only p-values.


## A practical recipe for a typical spatial study

Here is a sequence that works well for comparing spatial renderers or immersive encodes:

- Pilot with ABX on the hardest excerpt. If expert listeners cannot reliably tell the systems apart, stop: the difference is not audible.
- Run MUSHRA with a hidden reference, a 3.5 kHz anchor and a spatial anchor, on 6 to 10 critical excerpts of 10 to 20 seconds each, with 15 to 20 screened expert listeners.
- If the best systems are close to the reference, run BS.1116 on just those to grade the remaining small impairments.
- Validate with ACR online with many listeners on their own headphones, to check that the expert ranking holds in the real world.

That chain gives you detection, ranking, fine grading and real-world opinion, each answered by the method designed for it.

You can try an informal version of the idea right now: switch between the stereo and spatial versions of the same passage and notice what changes, not just whether you like it.


## Where perceptual tests meet objective metrics

Listening tests are slow and expensive, so the industry also uses objective predictors such as PEAQ (ITU-R BS.1387) for coding quality, and research metrics for spatial quality. At Spatial9 we use the Intent Fidelity Score to check that a spatial mix still matches its creative intent at scale. Metrics like these are only credible when they are calibrated against listening tests like the ones in this guide. A metric that disagrees with MUSHRA is the metric that is wrong.

If you are new to the formats being tested, start with what spatial audio is, binaural vs spatial audio and headphone vs speaker spatial audio.


## Frequently asked questions


### What is the difference between BS.1116 and MUSHRA?

BS.1116 grades small impairments, close to transparency, with a known reference and two unlabelled stimuli per trial. MUSHRA compares several systems of intermediate quality at once, with a hidden reference and anchors, on a 0 to 100 scale. Use BS.1116 when differences are subtle and MUSHRA when they are clearly audible.


### Is MUSHRA suitable for spatial audio?

Yes, it is widely used for comparing spatial renderers, HRTFs, upmixers and immersive codecs. Add a spatial anchor such as a mono fold-down, match loudness, report the playback system and consider rating spatial attributes such as envelopment or localisation, not only overall quality.


### How many listeners do I need?

For BS.1116 and MUSHRA, around 15 to 20 expert listeners after post-screening is common. For ACR, use many more, often dozens to hundreds, because naive listeners vary more. For ABX, the number of trials per listener matters most: plan at least 20 to 30 trials to have a good chance of detecting a real but small difference.


### What is a good MOS score?

On the P.800 ACR scale, 4 means "Good" and 5 "Excellent". In practice, scores above about 4 are considered high quality, but a MOS only has meaning within its own test. Always compare MOS values from the same test with the same reference conditions.


### What does a Subjective Difference Grade of minus 1 mean?

It means listeners rated the system about one grade below the hidden reference on the BS.1116 impairment scale, roughly "perceptible but not annoying" on average. An SDG close to 0 means the difference was not reliably heard.


### Can I run these tests online?

ACR is well suited to online testing, and ITU-T P.808 defines how to do it with crowdsourced listeners. MUSHRA and ABX can run in a browser, for example with the open-source webMUSHRA tool, but you lose control over the headphones and the room. For spatial audio, online tests are best for consumer opinion, while expert grading belongs in a controlled setup.


### Should head tracking be on during spatial listening tests?

It depends on the question, but it must be consistent and reported. Head tracking usually improves externalisation and reduces front-back confusion, so tracked and untracked results should never be mixed in one condition.


## Sources

- ITU-R BS.1116, Methods for the subjective assessment of small impairments in audio systems
- ITU-R BS.1534, Method for the subjective assessment of intermediate quality level of audio systems (MUSHRA)
- ITU-T P.800, Methods for subjective determination of transmission quality
- ITU-T P.808, Subjective evaluation of speech quality with a crowdsourcing approach
- ITU-R BS.2132, Subjective assessment of audible differences of sound systems using multiple stimuli without a given reference
- ITU-R BS.2051, Advanced sound system for programme production
- ITU-R BS.1770, Algorithms to measure audio programme loudness and true-peak audio level
- ITU-R BS.1387, Method for objective measurements of perceived audio quality (PEAQ)
- webMUSHRA, open-source browser-based listening test software

