In face-to-face conversations, our voice radiates differently depending on the direction we face. When virtual humans become part of everyday mixed-reality communication, should these effects also be reproduced? And if so, how accurately do we need to do it before the interaction starts feeling natural? This project was conducted by Pablo Abehsera Morell during his PhD thesis as part of the Sonicom project.
The question was approached with 3 questions, that led to 3 studies: 1) how well can we reconstruct voice directivity from limited room measurements, 2) can listeners notice missing directional acoustic cues when seeing a visible virtual speaker, and 3) what happens when two people hold a conversation while hearing it through different virtual acoustics.
Measuring a room impulse response for every possible orientation of a speaker is tedious. For a mixed-reality application, where a virtual talker may turn their head at any time, this quickly becomes a problem. If we had just one omnidirectional measurement of a room, could we still reconstruct the energy variations of a rotating directional source?
Pablo measured binaural room impulse responses in 3 rooms with different acoustic behaviours, using both an omnidirectional loudspeaker and a rotating directional loudspeaker. From there, he developed a method to modify the omnidirectional response so that it behaved more like the measured directional source.
The method worked by splitting the impulse response into time regions — direct sound, early reflections, and late reverberation — and applying frequency-dependent gain changes to each region.
Participants were then asked to compare auralisations based on the measured rotating source with auralisations generated using the reconstruction method.
If a virtual person visibly turns while speaking, do listeners care whether the reverberation changes accordingly? For this study, we needed a person speaking and rotating in a virtual room. So we recorded ourselves in front of a green screen, speaking, singing, whistling, etc. The same 360° camera was then used to capture the target room. After removing the green screen from the videos, the recorded speakers were composited into the room panorama. Once displayed in a Quest headset, participants saw what looked like a 360° video of someone talking in front of them.
On the audio side, we used binaural room impulse responses measured with a rotating Genelec loudspeaker placed where the virtual speaker would appear. Several auralisation strategies were then compared with varying degrees of source directivity reconstruction quality.
Unlike the first study, participants did not compare each rendering against an explicit reference. They simply rated how natural the acoustics felt while watching the person move and speak.
If I hear our conversation as if we were in a dry studio, while you hear the same conversation as if we were in a reverberant hall, does that mismatch affect how naturally we interact?
We did not need full AR avatars for this one. Instead, we built the equivalent of a controlled video call between two acoustically isolated rooms. Each participant saw the other on a screen and heard the full conversation — their own voice and the other participant's voice — rendered in a virtual acoustic condition.


Both participants did not necessarily hear the same acoustic. One could perceive the discussion in a dry room while the other heard it in a highly reverberant space.
To keep the conversation natural, we displayed discussion prompts on the screen: from low-stakes topics to slightly more serious debates. The virtual rooms were generated in real time using the IRCAM Spat reverberation tools in Max. We then analysed both subjective ratings and objective speech behaviour: speech level, pauses, turn-taking, overlaps, interruptions, and speaking rate.