Moving Camera, Moving Sound: How Audio Should Follow Picture in Immersive Video
Spatial9 Team · · 7 min read
Key takeaways
- Decide what each sound is anchored to first: the world, the screen or the listener's head.
- Pan on-screen sources gently and scale their movement to the size of the picture, especially on phones and laptops.
- Keep dialogue mostly centred and keep ambience beds stable across cuts and camera moves.
- In 360 and VR video, scene sounds are world-locked in the ambisonic mix while narration and score stay head-locked.
- Use movement when it tells the viewer something new, such as an approach, an exit or an off-screen location.
- The embedded A/B is a binaural headphone illustration in two-channel AAC; Spatializer for Video is in early access.
Immersive audio gives you more places to put a sound, and every camera move tempts you to use them. A pan, a whip, a cut to a reverse angle: each one seems to ask the soundtrack to swing round too. Sometimes it should. Often it should not. This guide sets out the decisions behind a soundtrack that feels attached to the picture without making the viewer seasick.
Three ways a sound can be anchored
Before any panning decision, decide what each sound is anchored to.
| Anchor | What it means | Typical use |
|---|---|---|
| World-locked | The sound has a fixed place in the scene. When the view turns, the sound moves the opposite way relative to the viewer. | 360 and VR scene sounds, establishing geography, off-screen sources with a clear location |
| Screen-locked | The sound is placed relative to the frame of a flat video. It follows what is on screen, not the space behind it. | Dialogue, on-screen action in normal YouTube or streaming video |
| Head-locked | The sound stays fixed relative to the listener's head, whatever the picture does. | Narration, score, interface sounds, inner voice |
In a flat video the viewer cannot turn the camera, so the camera is effectively their head. "World-locked" then means a sound moves across the speakers when the camera pans past it. In a 360 video the viewer controls the view, so world-locking is done by the player, which rotates the sound field on the device. YouTube's spatial audio format for 360 video reflects exactly this split: a first-order ambisonic sound field that turns with the view, plus an optional head-locked stereo pair that does not. Our guide to 360 degree video audio covers that format in detail.
Panning to picture without overdoing it
Panning to picture means placing an on-screen source where it appears in the frame. It is the most natural link between sound and image, and also the easiest to exaggerate.
The reason is geometry. A television, laptop or phone covers only a small angle of the viewer's field of view, while an immersive mix can place sounds all the way round. If a car crosses the frame from left to right and its sound sweeps from hard left to hard right, the audio travels much further than the picture does. On a phone held at arm's length the mismatch is obvious.
Some practical rules:
- Scale movement to the screen. Let an on-screen source move within the front of the mix, roughly as wide as the picture, not across the whole room.
- Pan the source, not the bed. Move the sound that belongs to the moving object. Leave the ambience where it is.
- Use the sides and rear for entrances and exits. A car that leaves frame can continue past the viewer. That is where width and surround earn their keep.
- Use height sparingly. Aircraft, rain and birds overhead can be convincing. Constant activity above the listener quickly feels artificial.
Keeping dialogue anchored
Dialogue is the one element viewers must never lose. A long-standing convention in film and television mixing is to keep dialogue mostly in the centre, the loudspeaker position defined at 0 degrees in multichannel layouts such as ITU-R BS.775. There are good reasons:
- Viewers look at faces, and faces are usually near the centre of the frame.
- Speech that jumps sides at every reverse shot draws attention to the edit.
- A centred voice survives downmixing to stereo and mono, and stays clear on small speakers.
You can still give dialogue a sense of place. A small offset toward a speaker standing at the edge of frame, a touch of room reflection from the space they are in, or a voice that moves off-screen when a character walks away are all fair. What to avoid is a voice that swings each time the camera cuts between two people in conversation.
Cuts, POV shots and fast camera moves
Cuts
A cut changes the viewer's position instantly. Changing the whole sound field at the same moment underlines the edit and can feel like a jolt. Most of the time:
- Keep the ambience bed continuous across cuts within the same location.
- Re-place only the sources whose on-screen position really changed, and only if it matters.
- Let sound lead or trail picture at scene changes. Editors call these J-cuts and L-cuts, and they smooth the transition.
POV shots
A point-of-view shot asks the viewer to be someone. The soundtrack can support that by bringing close sounds such as breathing, footsteps or clothing nearer and more head-locked, while the world around stays put. When the shot returns to an objective angle, return the mix gently too.
Whip pans and handheld camera
A whip pan rotates the view fast. Spinning the whole sound field just as fast is rarely pleasant, especially on headphones. Let the moving source smear or carry over, and keep the bed stable. With shaky handheld footage, do not attempt to track every wobble; anchor sounds to the scene and let the picture move.
When movement helps, and when it distracts
Movement helps when it tells the viewer something:
- Where a source is, especially when it is off-screen.
- That something is approaching, passing or leaving.
- That the viewer has moved into a new space.
- A deliberate moment of spectacle, such as a fly-by or a reveal.
Movement distracts when it tells them nothing new:
- Dialogue that pans with every shot.
- Ambience that rotates with every camera move.
- Constant motion that competes with the picture for attention.
- Wide movements on small screens, where they do not match the image.
A useful test: watch a scene with the picture hidden. If the sound moves a lot but you cannot say why, the viewer will not know either.
An illustration on headphones
The player below is a short A/B of the Spatial9 demo video. Use it to compare the original stereo soundtrack with the Spatial9 spatial version. Use headphones.
Watch the before and after video example (best on headphones).
Please treat it as a headphone illustration. Both versions are two-channel AAC files, and the spatial version is a binaural render for headphones, not a native multichannel or Eclipsa Audio deliverable. It shows the kind of placement and movement this article describes. It is not a measurement, and on speakers it will sound different. Our article on binaural vs spatial audio explains why a binaural file cannot adapt to other playback systems.
A checklist for each scene
- Decide the anchor for every element: world, screen or head.
- Centre the dialogue, then allow only small, motivated offsets.
- Match on-screen panning to the size of the picture on your target device.
- Keep beds stable across cuts and camera moves within a location.
- Choose a few moments where movement carries meaning, and let them breathe.
- Check the mix on headphones, on speakers and folded down to stereo. Our guide to headphone vs speaker spatial audio explains what changes between them.
Where Spatial9 fits
Spatializer for Video is designed to give a video soundtrack immersive placement that follows the action, with export in Eclipsa Audio for YouTube. It is in early access, so its behaviour may change before general release. If you already have a finished immersive mix as an ADM master, the ADM to Eclipsa Audio converter is free. For a broader overview of making YouTube sound bigger, read how to make YouTube videos sound cinematic.
Scope of this guide
This is a practical guide to mixing decisions, based on established conventions and public format documentation. It does not report listening tests or audience research, and none of its advice is a measured result. Rules of thumb here are starting points; genre, picture and the director's intent can and should override them.
Frequently asked questions
Should sound always follow the camera?
No. Follow the camera for on-screen sources when it helps the viewer locate them. Keep ambience, dialogue and music stable unless a change is motivated by the story.
What is the difference between world-locked and head-locked audio?
World-locked sound stays fixed in the scene, so it moves relative to you when you turn. Head-locked sound stays fixed relative to your head. In 360 video, scene sounds are world-locked and narration or score is usually head-locked.
Why does my dialogue sound like it jumps between shots?
It is probably being panned to each speaker's position. Pull it back toward the centre and keep any offsets small and consistent across the conversation.
Is panning to picture different on a phone?
Yes in practice. The picture covers a smaller angle, so the same audio movement feels bigger relative to the image. Check your mix on the screen size your audience uses most.
Sources
- Use spatial audio in 360-degree and VR videos, YouTube Help.
- Spatial Audio RFC, Google spatial-media project on GitHub (head-locked stereo alongside ambisonics).
- ITU-R BS.775: Multichannel stereophonic sound system with and without accompanying picture, International Telecommunication Union.
- ITU-R BS.2051: Advanced sound system for programme production, International Telecommunication Union.
- Fundamental concepts of spatial audio, Resonance Audio documentation.
- Immersive Audio Model and Formats (IAMF) specification, Alliance for Open Media.