Oculus Project (WIP)

In collaboration with the Facebook Reality Labs, the overall project investigates how far we need to push binaural audio rendering quality in VR for players to get life-like reactions for various types of games. The first two experiments will focus on the benefits of binaural rendering on players accuracy and efficiency: first assessing the impact of HRTF individualization, second that of training on a given set of HRTFs.

Phase 1: Get everything up and ready

The headset just arrived

CAD of the VR Room

VR Room install: check

Phase 2: HRTF individualization experiment

In-game video of the HRTF experiment

This is a short footage of a subject running the HRTF individualization experience. The point here was to assess the impact of individualized HRTF in a VR game where, needless to say, the whole design was built around target localization. HRTF individualization can roughly be seen an adaptation of the audio rendering to players “listening profile”. With individual rendering comes i.e. an improved capacity for audio source localization in the VR scene. The drawback is that establishing a player's profile is a pain at the moment (for the player at least). The goal of the experiment was to check whether this pain is worth it, and if so for which “level” of gameplay (no real need for high speed and accurate localization when everything is moving so slow you can randomly shoot and still get away with it).

Phase 3: HRTF training experiment

The objective is to design a VR experience / game to accelerate HRTF selection and learning. The first design proposed was based on a harvesting mechanism, coupled with throwing things here and there to score points. The harvesting forced players to pay attention to audio-visual targets of known position in space (hear it spawn, check where it is, continue playing while it ripens, harvest it before it goes bad). The throwing things, along with a few other game-play induced player moving audio sources around their head, was supposed to further help with the learning (proprioception).

HRTF training experiment v1 storyboards

HRTF training experiment v1 let's play

The first design was finally discarded. Beta tests showed that not only participants had a hard time understanding the game mechanics, but that playing it did not result in any substantial HRTF-wise improvement (i.e. audio localization accuracy). A second version was designed, switching gears: if the first implementation was a game turned into a learning experience, the second was a learning experience made into a game. We listed all the known problems of non-individualized binaural rendering (localization-wise), and created learning scenarios to expose and overcome them one by one. This new design capitalizes on skill learning itself as an incentive, presenting participants with a tool that we ourselves would want to use to quickly learn a new set of ears.

HRTF training experiment v2 let's play

Phase 4: Work in progress

HMD upgrade: Quests 1 and 2 are here