In VR gaming, the gap between what is seen and what is heard hinders the ‘being there’ experience. Even when jumping into the VR world with a special device, like an HMD, users still experience the virtual world with their own previous experiences stored in their brain. That’s why it would be unrealistic if one were to find themselves in a large church with high ceilings and marble walls, but would hear anechoic, dry sounds instead of long echoes. The moment something unnatural in a VR setting is noticed, the experience becomes limited and recognized as virtual rather than real.
The key to achieving this realism in sound is reverberation. Reverberation, or room impulse response (RIR), is a measured dataset that contains the room information that affects how sound is heard, such as the geometry and materials of the room or source-to-listener position. Listen from this video how the same audio signal is heard differently when different reverberations are applied like a filter.
Why reverberation is important in VR gaming
If you have played a good VR game, sound objects must have been spatialized and rendered relative to your point of view (POV). For example, when you look to the right, sound objects are heard relatively to the left. In addition to this mere direction change, what makes the sound experience realistic is the reverberation.
When a plain mono audio signal is filtered through a certain reverberation effect, you will hear sound as if the audio is coming from a specific room where that reverberation was measured. Filtering reverberation can make your steps heard as if you are in a wooden house, long tunnel or carpeted floor. Re-synthesizing the reverberation basically means reproducing the auditory scene.
Reverberation generally consists of direct sound, early reflection and late reverberation, and each plays a role in creating a believable VR experience. Direct sound maintains sound path information from a source to a receiver, as well as the respective time delay and level attenuation information. Early reflection is determined by wall structure, materials and obstacles in the room space. Both direct sound and early reflection deliver the direction of individual sound objects and its reflection. Both direct sound and early reflection contribute to the directional cue, which is a powerful storytelling tool in the gameplay. The difference between them is whether it contains the specific room information. Direct sound comes straight from a sound source to a listener, so the direction is rather formed by the shape of the listener’s head and ears. Early reflection sound arrives after being reflected from the wall, so it is affected by the physical room. To apply the early reflection effect, either virtual position of reflected sound from the wall should be tracked or sound particle’s propagation path between the sound source position and the listener should be tracked. Typical ways to track these are image source method and ray-based method. Late reverberation is also determined by the physical characteristics of the room, but is more of a diffused sound without directivity. Late reverberation rather determines the richness of a sound. It is important in making sense of what one hears and ensuring the sound is not monotonous, but placed in a certain space. It boosts the presence and immersion. All in all, reverberation eliminates one from thinking they are experiencing a fake scene.
How is reverberation used in VR gaming?
Indeed, applying RIR or Binaural Room Impulse Response (to be more effective) for VR is challenging. Reverberation by nature already contains certain room information. Even the reverberation measured in the same room could differ based on where sound sources are located or where the microphone is capturing the sound. This means that even a small, single space can have a huge number of scenarios, resulting in a large reverberation database. Furthermore, large memory or computation power is required for the renderer. As an alternative, artificial reverb can be used against measured reverberation, but both have pros and cons, shown in the table below.
A huge drawback of measured reverb is that it’s not applicable in fully-interactive VR content, such as 6DOF (Degrees of Freedom). Just as the popular audio format for cinematic VR and 360 video, Ambisonics signal cannot support a walk-around scenario. Ambisonics is a way of recording and reproducing 3D sound as a snapshot from one static place in the scene, meaning it is a spherical audio signal from a single point of view. The fact that Ambisonics is not applicable in 6DOF hinders the development of its various reverb effects.
As of now, reverberation for 6DOF is being studied from various angles, both in terms of a measured one and artificial one to create the realistic reverb (but at the same time efficient with low complexity). So far, noticeable reverberation solutions have come from graphic processor companies. Nvidia and AMD released spatial audio solutions, VRworks Audio and TrueAudio Next, respectively, which provide artificial reverberation in real time. However, since these solutions actually calculate the sound propagation, they require heavy computing because they can only run on a high-capacity desktop and GPU-based devices. Considering how VR gaming is still highly driven by mobile devices, G’Audio Craft or Oculus spatializer provide alternative options that do not rely on desktop computers. Finding a way to carry reverberation information with light GPU performance will open up more growth opportunities for VR.
What lies ahead for reverberation
There is still a long way to go when it comes to bringing reverberation technology to VR gaming. Even for the game audio engineer, their work entails more than just choosing between measured and artificial reverbs. They may use several different reverb plugins for different types of audio signals, such as mono signal or Ambisonics signal. If the desirable final output is a binaural signal, they need to consider transforming their signals into binaural ones first.
What has ultimately come into use may not be the most realistic solution. The most efficient reverberation solution might just do. Another interesting point and food for thought is how do we go about the scene that we have never experienced in real life? As imaginary places are not uncommon for VR gaming, we ask what kinds of reverberation should be applied?