When you are in a room full of people and you hear the Facebook notification, how many people look at their phone, or is it you?

Sound is often an overlooked aspect of the user experience journey, particularly if the team is without experience in audio design or if they are working on a small budget and have limited resources. but it plays a crucial role in creating a sense of presence and supporting desired outcomes.

Sound can be used to provide cues about the location and movement of virtual objects, as well as to create a sense of atmosphere and realism. Additionally, sound can be used to enhance the overall emotional impact of the experience and to help users orient themselves within the virtual environment. Sound is a key to the human experience.

When working on an immersive experiences, sound should not be treated as an afterthought. In this article, I will be discussing how sound can be used to create a sense of presence and realism in virtual environments, and highlighting some key considerations for you to discuss and incorporate sound into your experience.

Types of Sound

Before discussing ways to create immersion and realism with sounds. I wanted to highlight l types of sounds that can be used in immersive experiences:

Ambient Sounds

These are background sounds that create a sense of atmosphere and realism, such as the sound of wind or water in a virtual environment.

Sound Effects

Sound effects can be used to create a sense of realism and immersion, such as footsteps, heavy machinery or electrical hazards. These can be captured from the environment or the specific asset you are digitising.

Folly

Folly and sound effect are used interchangeably, however I have separated folly as it can be used to accent sounds that improve immersion. Not tangible to physical sounds, however help the user like button presses, menu clicks and User Experience bleep bloops.

Music

Music can be used to create a sense of mood and atmosphere, and to enhance the emotional impact of the experience.

Voiceovers

Voiceovers can be used to provide information, instructions, or narration in the experience.

How to make sound immersive

Sound by itself is just a file that plays and we hear it, however an immersive environment is interactive and reacts to your user. There are a few ways we can play the audio to enhance the immersion.

Spatial/ non-spatial audio

Sound is placed in 3D space to create a sense of distance and direction, allowing users to locate virtual objects and navigate the virtual environment. If the sound is to your right, it will come from the right speaker. Some surround sound systems allow for finer detail of the audio position and be able to hear the sound from above and below. It is commonly used in virtual and augmented reality to create a sense of presence by simulating the sound coming from different sources in the virtual space as if they were in the real world. Alternatively Non-spatial audio has no direction and can appear flat as it's in the centre of the stereo field. However this still has use for context and information

Digetic/ non-diegetic audio

Diegetic audio refers to sound that builds the world. This type of audio is used to create a sense of realism and immersion by simulating sounds that would be present in the real world. For example you see a metal bar fall, and you hear the metal impact. Contrasted with non-diegetic audio which is sound that would not fit in the environment, music and effects that the user wouldn’t experience in that environment.

Dynamic/ non-dynamic audio

Dynamic audio refers to the use of sound that changes in response to the actions or choices to the situation. Dynamic audio can also be used to provide cues or feedback, helping the player to understand what they should do next. Non-Dynamic audio does not change from the experience and can be used to provide a consistent background to underline the experience.

Adaptive Audio

Adjusts the audio based on the actions or choices of the user. It allows the audio to change. Adaptive audio can be used to create a sense of progression used to provide cues or feedback, helping the player to understand what they should do next.

Interactive Audio

The audio is triggered by the user's interactions. It allows the player to actively engage with the sound and it's an important aspect of creating a sense of presence and realism. Interactive audio can be used to provide feedback when the player interacts with virtual objects, such as a button press or a door opening. It also can be used to provide cues about the location and movement of virtual objects, allowing the player to locate and interact with them more easily.

Haptic Audio

This technique uses vibrations of the controller or bass shakers so the user feels the sound rather than just hearing it, providing an additional layer of immersion

Implementing Immersive Sounds

So far we have identified the types of sound and how they can be used, however multiple methods can be mixed together to create a more immersive experience. For example, non-diegetic audio such as background music can be made adaptive, where the music changes depending on the user’s actions or choices. Another example is using spatial audio to make non-diegetic sounds like background music, to simulate as if it’s coming from a specific location or cue the user to complete an action.

When planning the sound for an immersive experience, follow these steps:

Define the goals and objectives of the immersive experience: Consider what the main objective of the experience is, what emotions you want to evoke in the user, and what kind of atmosphere you want to create
Break down the objective into steps: what is the process you need your user to follow and ask yourself what sounds would be useful to reinforce the task at hand. Research other immersive experiences and gather reference material that is relevant to your project. This can include audio recordings, sound effects, and music
Identify the key sound elements: Identify the key sound elements that are necessary for the immersive experience as listed in the types of sounds
Determine the sound sources: Determine how the sound will be used as we discussed in the how we can make sound immersive
Create a sound design plan: this document outlines the different sound elements, how they will be used, and when they will be used for the project

In the end you will have a list of sounds that you will need for your experience to go into production. Now we can discuss how we can use these sounds to support our experience.

Behavioural Feedback Loops

We can use sound to teach and support our users through experiences based on their actions or choices. It helps them understand what they did correctly and what needs to be improved. It can be used to create a sense of progression and to help the user understand how their actions are impacting the virtual environment and meeting the set out completion criteria.

An example of Behavioral Feedback using audio in an OHS (Occupational Health and Safety) safety simulator in virtual reality (VR) could be as follows:

The player is in a virtual factory and they need to identify and fix a gas leak.
As they approach the gas leak, they hear a loud hissing sound (adaptive spatial audio), indicating that the gas is leaking.
The player uses the VR controllers to turn valves and fix the leak. The valves have a metal squeak (interactive dynamic audio) when the player turns the hand wheel.
Once the player successfully fixes the leak, they hear a sound of a valve closing and a beep sound (non-diegetic folly) to confirm that the task is complete.
If the player fails to fix the leak, they hear an alarm sound (diegetic adaptive audio) indicating that the gas levels are dangerous, and they are prompted to try again.
Throughout the simulation, the user also hears environment sounds (ambient diegetic adaptive audio) such as the sound of machines running, people talking and footsteps, to create a sense of realism and immersion. These environment sounds change through progression for example the people talking and footsteps disappear when the alarm turns on to highlight that the area is not safe.

The use of audio in this scenario provides the player with clear feedback and cues about their actions through every step of the scenario and the state of the virtual environment. This helps the player understand what they did correctly and what needs to be improved, and it creates a sense of progression and immersion.

This approach can help to educate and train the user, and as immersive experiences can be repeated. Users gain a framework understanding and reinforce the desired behaviour from the experience.

Instruction

Audio can be used to provide clear instruction and cues in an immersive experience to help the user to understand what they should do next and how to navigate the virtual environment.

Instructional audio can be used to provide spoken instructions for completing tasks, helping the user to understand what they need to do to progress through the experience. This leaves the other senses focused on the task.

Also, audio cues can be used to provide hints and guidance on where to go or what to interact with, such as a sound coming from a specific direction to get the user’s attention and indicate where they should go next.

An example of using audio instruction and cues in an immersive experience to teach mechanical fitting of an engine could be as follows:

The player is in a virtual reality simulation of a mechanical workshop (ambient non-dynamic), and they are tasked with assembling an engine.
As they begin the task, they hear an instructor’s voice providing an overview of the task and the tools they will need to complete it (non-dynamic voice over).
As the player picks up each part of the engine, they hear a sound effect indicating that it has been successfully selected (non-diegetic folly).
As the player begins to fit the parts together, they hear a series of instructions from the instructor on how to properly align and secure each component (dynamic voice over).
If the player makes a mistake, they hear a warning sound indicating that they need to correct the issue before proceeding (dynamic voice over and non-diegetic folly).
Once the engine has been successfully assembled, they hear a sound of the engine running to indicate that the task is complete (dynamic sound effect).

By providing clear audio instruction and cues, the player is guided through the task and can understand how to properly fit the engine. The system can track progress and provide extra explanation to ensure the user understands the task and can complete it. This approach can be used to educate within the immersive environment.

Auditory Navigation

Virtual reality can be disorienting for some users because it creates a sense of presence in a virtual environment that can conflict with the user’s perception of reality. Also the user can find themself lost in a new environment. Sound can help the user to understand their location and provide cues to where they need to goto next in the virtual environment.

Spatial audio is useful here as positional sound helps the player to orientate themselves in the environment the same way visual landmarks can. For example, the user might hear the sound of birds and leaves rustling as they move through a forest, however their instructions are to “find the road”. They cannot see the road however a traffic sound off in the distance can provide an auditory landmark for them to orient themselves. As designers, we have limited senses that we can leverage in these immersive experiences, however capitalising on what we do have can make our experience less disorienting for our user.

An example of using auditory navigation in a fire safety simulator to help the user find the fire could be as follows:

The player is in a virtual reality simulation of a building and they need to locate and put out a fire (ambient diegetic audio).
As they start the simulation, they hear an instructor’s voice providing an overview of the task and the tools they will need to complete it (non-dynamic voice over).
The player is in an area of multiple buildings with his equipment, distance alarm bell is ringing (diegetic spatial audio)
As the player moves through the building, they hear ambient sounds such as the sound of people evacuating, footsteps and machines running to create a sense of realism and immersion (ambient diegetic audio).
As the player approaches the location of the fire, they begin to hear the sound of crackling flames and smoke alarms, indicating that they are getting closer to the fire. (diegetic spatial audio)
The sound of the flames and smoke alarms becomes louder and more intense as the player gets closer to the fire, providing a clear cue about the fire’s location. (dynamic spatial audio)
As the player uses the fire extinguisher to put out the fire, they hear a sound of the fire being put out (diegetic adaptive audio), and an indicator confirms that the task has been completed (non-diegetic folly).

By using auditory navigation, the player can locate the fire by sound alone quickly and safely. Using multi-sensory inputs in immersive experiences, the user has more information and a deeper sense of presence in the virtual environment.

Prepare them for the real world

Audio is a powerful tool that can be used to set the tone of a situation in immersive experiences. It can be used to create a sense of atmosphere, emotion, and realism, helping to immerse the user in the virtual environment.

Now I am not saying slap some soft minor chords on a piano and call it auto sad. However, using the environment itself to explain the situation to the user. For example if you want to build tension to train someone under pressure, using diegetic audio like alarms or flustered people in your immersive experience to try and distract them can cause anxiety, especially paired with a countdown timer.

An example of using audio to set the tone for nurse training to de escalate an aggressive patient could be as follows:

The user is in a virtual reality of a hospital room (ambient Non-dynamic audio) and they are approached by an aggressive patient who is yelling and becoming violent (voice over dynamic audio).
The player is prompted to use de-escalation techniques (non-digetic voice over), such as using a calm and soothing tone of voice using voice recognition. As they do so, they hear their own voice change to a calmer tone, and the patient’s voice becomes less aggressive(voice over dynamic audio).
The patient’s breathing and heart rate can be monitored and used in the experience as the patient could get more aggressive (voice over dynamic audio) if the nurse is stressed, the user can hear the sound of the heart rate monitor beeping (diegetic audio), providing feedback about the patient’s state.
As the patient calms down, the player hears the sound of the heart rate monitor slowing down and the patient’s voice becoming quieter, indicating that the situation has been deescalated (voice over dynamic audio).
Once the situation is deescalated, the player receives feedback and guidance on how to handle similar situations in the future (voice over audio or instructor discussion).

By incorporating audio into the simulation, the player is able to understand the state of the patient and their own actions, as well their actions had an impact in the virtual environment. By providing people with simulations of real-world scenarios, they can gain a deeper understanding of the situation without the risks of failure. This can allow them to practise and prepare for the real-world, equipping them with the knowledge and skills needed to succeed.

What now

When creating an immersive experience, it’s important to consider the use of multiple sensory inputs, including audio. Each element of the experience should work together to create a cohesive and immersive experience.

By breaking down the objective into tasks and considering the different types of audio that can be used to enhance the experience. you can create a comprehensive sound design plan for your immersive experience. This plan will provide a clear understanding of the audio elements that will be used, how they will be used, and when they will be used. It’s also important to note that some sounds may overlap and can be reused in different areas of the experience, this can help to focus on the minimal viable product and ensure that the sound design is efficient and effective.

Sound is an important aspect of immersive experiences and is a broad topic to explore. I hope this article got your creative juices flowing on how sound can be used to enhance the user’s experience journey in immersive technologies.

Thank you for reading, and if you found a part of this useful. Share so it can help others.

Also go come check out my channel on YouTube

See you over on YouTube

Sound is key to immersive experiences