Voice Auralisation in XR (2025)

In face-to-face conversations, our voice radiates differently depending on the direction we face. When virtual humans become part of everyday mixed-reality communication, should these effects also be reproduced? And if so, how accurately do we need to do it before the interaction starts feeling natural? This project was conducted by Pablo Abehsera Morell during his PhD thesis as part of the Sonicom project.

Pablo during a green-screen recording session

The question was approached with 3 questions, that led to 3 studies: 1) how well can we reconstruct voice directivity from limited room measurements, 2) can listeners notice missing directional acoustic cues when seeing a visible virtual speaker, and 3) what happens when two people hold a conversation while hearing it through different virtual acoustics.

Study 1: Reconstructing source directivity

Measuring a room impulse response for every possible orientation of a speaker is tedious. For a mixed-reality application, where a virtual talker may turn their head at any time, this quickly becomes a problem. If we had just one omnidirectional measurement of a room, could we still reconstruct the energy variations of a rotating directional source?

Anechoic voice directivity measurement session

Pablo measured binaural room impulse responses in 3 rooms with different acoustic behaviours, using both an omnidirectional loudspeaker and a rotating directional loudspeaker. From there, he developed a method to modify the omnidirectional response so that it behaved more like the measured directional source.

RIR measurement setups in the 3 rooms (studio, conference room, amphitheatre)

The method worked by splitting the impulse response into time regions — direct sound, early reflections, and late reverberation — and applying frequency-dependent gain changes to each region.

Directivity projection principle

Measured rotating impulse response

Reconstructed rotating impulse response

Voice auralisation with measured IRs

Voice auralisation with reconstructed IRs

Participants were then asked to compare auralisations based on the measured rotating source with auralisations generated using the reconstruction method.

associated paper

Study 2: Seeing the source rotate

If a virtual person visibly turns while speaking, do listeners care whether the reverberation changes accordingly? For this study, we needed a person speaking and rotating in a virtual room. So we recorded ourselves in front of a green screen, speaking, singing, whistling, etc. The same 360° camera was then used to capture the target room. After removing the green screen from the videos, the recorded speakers were composited into the room panorama. Once displayed in a Quest headset, participants saw what looked like a 360° video of someone talking in front of them.

Participant viewpoint during the experiment (mono sound)

On the audio side, we used binaural room impulse responses measured with a rotating Genelec loudspeaker placed where the virtual speaker would appear. Several auralisation strategies were then compared with varying degrees of source directivity reconstruction quality.

Simulated directivity (centre of room)

Simulated directivity (near wall on your left)

Unlike the first study, participants did not compare each rendering against an explicit reference. They simply rated how natural the acoustics felt while watching the person move and speak.

associated paper

Study 3: Mismatched acoustics

If I hear our conversation as if we were in a dry studio, while you hear the same conversation as if we were in a reverberant hall, does that mismatch affect how naturally we interact?

Acoustic mismatch principle

We did not need full AR avatars for this one. Instead, we built the equivalent of a controlled video call between two acoustically isolated rooms. Each participant saw the other on a screen and heard the full conversation — their own voice and the other participant's voice — rendered in a virtual acoustic condition.

Teleconferencing setup

Both participants did not necessarily hear the same acoustic. One could perceive the discussion in a dry room while the other heard it in a highly reverberant space.

Anechoic

Room 1

Room 2

Room 3

To keep the conversation natural, we displayed discussion prompts on the screen: from low-stakes topics to slightly more serious debates. The virtual rooms were generated in real time using the IRCAM Spat reverberation tools in Max. We then analysed both subjective ratings and objective speech behaviour: speech level, pauses, turn-taking, overlaps, interruptions, and speaking rate.

Participants during the experiment

associated paper