The Impact of Immersive Audio on our Digital Future

Immersive Audio has come into prominence in recent years, with audio data bandwidth and storage space becoming largely irrelevant blockers to a format which promises to encapsulate the listener through a variety of 3-D sonic experiences. At the NAMM show in LA this year, we gathered some of the most prominent minds in audio together to discuss Immersive Audio, its impact on film and music, and its potential for advancements within the tech industry.

https://www.youtube.com/watch?v=ux66Ef-rCZs

Each of the guests on the panel have a deep rooted connection to and passion for Immersive Audio - Jacqueline Bosnjak from Mach 1 and Q Department, Will Eggleston from speaker manufacturer Genelec, Nuno Fonseca from Software developer Sound Particles, Christian Sander from Dear Reality - a leading company in the field of Immersive Audio controllers, and finally, Jean-Marc Jot who has worked for a host of audio firms including iZotope.

"We couldn't look away, our life has changed forever" - Getting Started in Immersive Audio

We start the discussion with Christian Sander outlining what inspired the development of the Dear Reality software: "We were creating virtual or spatial audio radio play, and we needed a specialised tool, a binaural rendering tool, we needed a different acoustic scenario. So we had this project to do and we already have the binaural rendering technology and said, "Okay, what features do we need?" So the creativity drove the feature set, and then we just worked on it, used it for the first productions and improved it over time. And I think that's always the way to go".

Jaqui from Mach 1 explains how her company got started in Immersive Audio: "We're reluctant software developers - we got involved really early on and got locked, kind of like a lot of people did. And then realising that you were at this precipice was so exciting, because spatial audio suspended disbelief in virtual reality, and it needed to be extremely high quality". She goes on to explain what inspired her personal journey into Virtual Reality: "There's a piece called 'Wolves in the Walls', and you put on the Oculus headset, and a little girl comes up to you, but your headset is all smudged and you can't really see and she walks up to you and she rubs it clean! And it's a squeaky squeaky spatial audio. And at that point, she acknowledges that you're in her world and you feel present with her. And when that happened as a studio, we realised that we had left experience economy and moved into memory making, because when you leave these kinds of immersive experiences that are 50% or more sound, you really feel like you were somewhere that you weren't, and you have a memory of being there". She sums up: "We couldn't look away, our life has changed forever".

Nuno Fonseca from Software developer Sound Particles explains: "There's one thing that I also like, is the fact that sometimes this becomes almost like a treasure hunt for the user. There's a lot of people recently using Apple Music, but they listen to spatial audio that they don't even know about, especially because it's on by default".

"It's a Thing!" - Adoption and Accessibility of Immersive Audio

Big companies are now taking Immersive Audio seriously, as Jacqui explains: "I feel like Apple and Dolby have definitely made spatial audio something that's gonna work, it's gonna happen. We used to have a lot of meetings or calls where people were like, but it isn't a thing. And I'd always be like, well, is hearing naturally a thing? It's a thing!". Referencing the current stage in the journey of Immersive Audio, she continues "I think that it's up to people that are here in the room and on this panel to push technology companies to allow artists to do all the things that they want to do. It's not just one way, I think we need to not have boundaries for creativity. And that's kind of why we're excited to be working with GPU Audio - we don't even know what we're going to make in the future. But we tooling for it with GPU Audio and Mach1. I think it's exciting time, but to put pressure on technology companies to open the gates and allow creativity to flow"

Jean-Marc Jot also celebrates the strides Immersive Audio has taken: "I'm grateful to Dolby, Apple, and Sony 360 and others because today, when you create a piece of music, you no longer have to assume before you start how many channels you're creating. There used to be a time when you had to say it's a piece for two, or stereo or 5.1". He adds "You can imagine that you're creating something where a person is going to be in the middle of an immersive environment of sound, you know, instruments, whatever. And you don't have to pre-assume what is the format in which they have been consuming." He goes on to say "And this technology is evolving in such a way that even when you monitor on headphones, you can hope to get a close rendition of what would be experienced when the loudspeaker systems aren't this huge. Two or three years ago, you could not be very realistic unless you were a researcher. And, and now the tools are becoming democratised to enable this, I find that super exciting - I find that a big step."

On the theme of accessible Immersive Audio, Christian adds: "I think spatial audio has become pretty much accessible on the playback side, like Apple Music, for example really pushing it, making it default available for everyone but also on the production side. You can virtually monitor, you don't need a big loudspeaker setup, of course, it helps a lot and you should check your mix in the end. On the third, it really helps you to already start production from wherever you are, I think that helps pretty much for these workflows".

"An amazing layer of information" - The Immersive Audio Landscape for Musicians

Outlining the positive opportunities for musicians using spatial audio, Christian explains "The moment musicians really use the tools creatively and aesthetically, and just because it makes sense from the perception and the emotions that want to be transferred to music, that's a pretty positive development, I see".

Jean-Marc Jot agrees: "If you're a music composer today, you might think that you're composing for a sweet spot, and maybe multiple people are going to be able to enjoy it, if they're sitting close together. But if you think of it as if you're a game developer, and you want to make your game a musical experience, you would probably not think of producing it the same way as a person surrounded by a ball or bubble of sound that walks around".

He outlines the benefits of immersive mixing too; "You can remix in real time, you know, the game could involve changing the balance between multiple instruments, etc. Things you could do in a game but the way we approach music today, once it's delivered - it's baked. The balance and the spatial relative sweet position of the instruments is baked. And I think there's a future I long for where we merge the game and VR and augmented reality technology with the tools that we're using today to create music. And I think it'll require cloud based processing - and GPU Audio has a future there."

Christian responds by illustrating some practical applications of Immersive Audio: "You're not only able to rotate your head, but also walk around. And I think this is super interesting, especially for music creation. So we have a workflow where you can mix your production in the DAW, and then transfer that mix with all the positioning and whatever enhancement into the Unity game engine. And then you can walk through it might have been VR or WASD with the keyboard or whatever, you can walk through it". He continues "Then you really have to talk about distance attenuation, what happens if you're closer to that instrument, you hear the other little lower level, and these things, it's super complex, like mix you do for a gain, which is something they have been doing already for 20-30 years. But merging this part like music, experience in and walking around with this is super interesting."

Jacqui adds her thoughts: "As reality and digital start blending, I feel like there's also the rise of the audio works. And it's sort of the end of the visual industrial complex, and the beginning of the audio works. Because you can walk around the real world and have spatial information for the hands down heads up experience, whereas if it's visual, you have to stop doing whatever you're doing in reality, that's what spatial audio can give us - an amazing layer of information around you."

"It was riveting.. it felt like I literally went to a theatre show" - Producing Bespoke Immersive Experiences

Immersive Audio has the possibility to change how we consume audio, given its ability to adapt depending on our surroundings - and even change our surroundings. Jacqui explains how our geographical location could play a part in immersive experiences: "What we've integrated into our technologies is one of the biggest mapping companies that work for cars, using safety guidance data with spatial audio, but then also you could take that same car and you're driving through a place and we've seen artists put forward apps like this one that I thought was really coo." She explains: "As you're driving along and you're listening to songs, they can tell you the context of that song, so maybe listening to Thom Yorke's Harahill Road, and you driving in that area, and they're like, "Okay, this is the area where David Kelly committed suicide" (Well, that's what the song is about). So the sort of layer of context here, the sort of geo-located context with spatial audio and content, it's kind of interesting, and definitely a different way of making money other than spatial music".

Nuno outlines some of the health benefits which Immersive Audio may bring "There are many other areas for instance, we have some of our users who are using this to create things for Yoga people using Spatial Audio for health. For instance, a year ago I started meeting people using spatial audio for health and helping people in hospitals and other places".

He adds: "Why not use spatial audio for instance, in retail? Why not use spatial audio and sales like people being in the middle of the woods and giving experience to the users wherever they are shopping? If you think outside the box with spatial audio, a perfect fit pretty much every time that you want people to feel immersed. Spatial audio is there as the right tool to help with that illusion". He adds: "In theatre, in parks - only two days ago, I was talking to someone who was saying that a restaurant uses spatial audio because they want to give different experiences to their audiences."

Will from Genelec offers his point of view on immersive experiences: "We've seen our products go into these types of places - but we want to get close to the customer and understand what it is they're doing. And when you actually go visit, and see and hear, feel the whole perception of what it is that they're attempting to do with their clients". He goes on to provide an example of this inspiring work "We've seen this with one particular customer - at some point, he sent me some pictures of a shipping container. And he had modified it, and he turned it into an art exhibit with some sort of software, moving his hands and sending light pulses out, and all the sounds would be heard. And he had around 20 loudspeakers inside of a 40 foot shipping container."

Jacqui follows up with an example of another great immersive experience: "I remember during the pandemic, we actually did this one experience. It's by a British directing company called Dark Field, they have an app where they have a countdown to the club where you get your drink and sit down. And then they said, "Sit at a kitchen table with your partner opposite you blindfold yourself with a glass of water", he had all these props that you had to put around you. And then as the theatre shows started, the first thing they said is like your partner sitting opposite you has been placed, and then the sort of spatial audio experience unfolds, where they actually use the plots of water. So you hear the glass breaking and water splashing. And so they've really been really smart about sort of placing all the objects and then moving them just for sound. But, you know, you are blindfolded during the whole experience, and it was riveting. It was during the pandemic, and it felt like I literally went to the theatre show. And I was like, "Holy shit, directors only using audio?!"".

And on the subject of those with limited vision - Christian explains "We got a lot of feedback, especially from blind people, because for them, this medium is super interesting, and especially if it's binaural. If that's even more spiritual or not more, it has finally spatial information for them. I'm very excited for this medium".

"Fun and Ruthless Perfection" - Constructing Immersive Audio Tools

Nuno explains: "For me, all the tools that we make, must have a certain component of fun. And that's even from the beginning of Sound Particles, I started spending two years of my life creating the company, even before knowing if someone in Portugal, doing something like that could eventually be used in Hollywood". He adds "It's important to have this component of fun and ruthless perfection, especially when you're doing something in this area of specialty, you are pretty much a pioneer creating tools for a market that doesn't exist. Eventually, if you are lucky enough, the future might be an interesting market and you can do so many things to grow your team and do more things. Then our users can have that fun of music".

Will describes his point of view, from a hardware perspective: "We have so many different products that can be used in so many different ways. Everybody has a unique way of doing things. Being able to be closer to the customer is really really important. He adds: "The other part of that is, when you're selling loudspeakers or when you're helping people buy loudspeakers, there's so much of an education that comes back to us. Definitely. And that finds its way back to engineering eventually. And that's really important. The key piece of the puzzle is trying to understand exactly what that customer is doing, and how they're doing what they're doing. What's their future?"

Christian explains how Dear Reality approaches development "So our approach is to offer high quality spatialisation on the spatialised side and on the virtualization side. Also on a price point that is affordable, but also scalable, so we have to offer tools that have a lot of feature sets, as well as smaller features to update things so that users can try out a trial or free version. So I think it's important for every level, you start out as a bedroom producer, professional, semi professional, that you can use trial tools and have the option to use high quality".

Jacqui explains how Mach1 archives this: "As a company we're really behind the idea that spatial audio is a medium, not a feature or remodelling effect. So we focus a lot on building a SDK that's a framework, because that framework allows best practices and standards and interoperability to exist. And that's what we're building with GPU Audio trying to democratise access to that because we do have products out in the marketplace that cost money - although we do offer a really good free trial, if anybody wants to try out our products".

"Shipping Alien to 38 different platforms" - Overcoming Challenges and Final Thoughts

Jacqui explains how the implications of the current form of Immersive Audio can be a challenge: "So what we're trying to do is offer the stereo of spatial audio so that everybody knows what they're mixing to. And everything can sound the same on every playback. And every developer building an app or building it into a platform knows what it should sound like. And as we move around the world, interoperability is becoming so important. And we actually started building the transcoder, when we had to ship Alien for 20th Century Fox, and they asked us to distribute it to 38 different platforms. And we didn't have that line item on our budget. And I went back to them and I said, "Ah, we got to remix this 38 times. That's like, you know, you owe me three times the money". And they were like, "Well, you already signed the contract. We don't care, deliver it". And I remember my team wanting to kill me, we stayed up, like for five nights, 24 hours, just re-mixing things for these different apps. And we're like, "There's no way the industry can move forward like this. It's absolutely insane". But at the same time, we've sort of watched an industry realise this".

She concludes: "You can see the problem - people are hitting those same roadblocks that we hit early on. So I hope that our framework can offer the creative choice of which format you want next - and you shouldn't be on rails to use a format. Yeah, it should be whatever the artist wants to do, whatever sounds the best and so we're trying to be the stereo of spatial and unify a broken pipeline so that everybody can mix and understand what it is, and really build this new medium."

Jean-Marc Jot rounds off the conversation with a final thought: "One thing that I think is going to help us overall is if we democratise the tools to free speech audio. So I think we should get to a point where it's as easy to create Immersive Audio as it is to create stereo, there's a very established practice of reading stereo music. And I think the tools are not yet where they need to be to make it as easy to apprehend creating Immersive Audio". He adds: "But I don't believe that there's an open source solution for that yet. It's more the kind of thing where I work on my own time that I think is one of the gaps that we can help fill. Yeah, one of the issues that is also not open source is that two day binaural rendering on the headphones does not sound as natural as listening on speakers. So I think that's another area that is worth investing research in. "

There we have it - from the inception of Immersive Audio to forging the tools of the trade for future generations. A lot of food for thought, given some of the incredible ways this medium can impact and improve our lives, in practical and emotive ways.

Thanks to our panel for sharing their valued opinions and experiences, and to everyone who joined the innovation lounge at NAMM 2023. You can watch the discussion in full over on the GPU Audio Youtube Channel:

https://www.youtube.com/watch?v=ux66Ef-rCZs