DAW based VR audio content production?

Nuendo
#1

With Nuendo 7 we saw an attempt to tap into game audio, but what about VR?

Ambisonics has been adopted by the big players and is fast becoming the “defacto standard” for VR audio production: Google (ambiX Youtube content), Facebook (recently bought Two Big Ears and Occulus provides its own ambisonic based audio tool-set), Adobe also supports ambisonic audio, and there are many others. But when is Steinberg going to provide Nuendo with appropriate functionality to capitalise here?

Moves to support Nokia’s OZO audio (a €40,000 AV solution) doesn’t help smaller audio content producers who don’t/will not have access to this equipment and Nuendo’s rigid bus architecture continues to be a problem.

In terms of 3D audio, most higher order ambisonic (HOA) VST tools (there are now more available than ever before) are unusable in Nuendo. Higher channel counts per bus are needed, even the recently added 22.2 bus is sub-par at this point.

Is there anything in the pipeline for 7.5? Why is Steinberg behind the curve on all of this? I use Nuendo, and love it, but I’m really feeling felt behind here, if I want to get into DAW based VR audio content production I see no other option than to adopt Reaper (up to 64 channels on a bus) for this type of work, which is a shame.

Anybody got any views on this?

#2

Hello,

may I asked why you feel we are “behind all this”?

First of all, our Nokia OZO VR project should not be mixed up with other types of playgrounds of VR-related formats
and technology. There are only a few, limited tools on the market that allow for “something” like VR production. The entire VR industry is at its beginnings and be sure that we are in contact with ALL the players you listed and a few more most people don’t have on their radar. Integrating Ambisonics will not be the biggest deal and you can expect that this will happen in a future Nuendo version.

The challenge is to establish a real workflow, not just supporting a format. All current technologies are far from being convenient and straightforward. That is why we are not releasing one VST plug-in after the other.

Be sure we do our best to develop technology for VR, besides our focus on game audio, TV and film post-production and ADR workflows.

Thanks,
Timo

#3

Great to hear Timo. This stuff, IMO, is going to be huge for those of us who will be pursuing 360 VR Video as part of our business.

#4

From what I’ve seen at gamescom I can only testify that VR is in its early infancy. There are some great concepts out there, but until this gets to the masses, it will take a while. If only because of the prices for VR gear and the inconvenience of setting this up in the living room. The Vive for example, you need to have multiple cables from the computer, you have to mount these room sensors, you need a super gaming PC at home (which is expensive), and you need to tackle the cables in order not to get entangled in them. This is something only few can buy. I cannot remember the last time something this inconvenient, where you had to rebuild your living or gaming room with cable holders and a peripheral for 800$ won the mass market. So we’re far from mass market. Trade shows, as eye catcher, or multimedia shows in museums or something yes, sure. But again, not wide spread yet.

Nonetheless, everybody wants to get on the bandwagon. On that note, 3D and VR sound is more a thing of the playback device. The audio engine in the game or VR experience has to properly translate audio sources into 3D sound and space, do HRTF and such things so the sound really feels immersive. Currently all the game engines I tried (Unity, Unreal…) do not sound that great when on headphones, not to speak of VR headsets.

But, you’re right, if we are to develop sound for VR experiences, we need means of testing how the sound will feel and export it into a supported format, or read such formats for editing. But just exporting such a file will do not much good, we need testing within Nuendo, how the sound will sound when turning our heads. Not only VR, also better integration in game engines and middleware would be great so we can test sound before we deliver it to the client. Then they have to create a game build, send it to us, we can play it and re-create a game situation to judge the sound. Then we have to go back. It’s cumbersome. And here, workflow is crucial. How could such a workflow look like so it makes sense? More than just linking a Nuendo project to WWise and have roundtrip editing. I guess an exporter is quickly written. But giving you the advantage of testing sound while editing would speed up workflow a lot.

I’m intrigued and looking forward to what Steinberg will come up with.

#5

Thanks for your reply Timo. Well, the VR sector might be new, but as you know, ambisonics is decades old, if integrating it is not a big deal, Steinberg should really have done it already, it’s called foresight, market positioning etc. It doesn’t really look good for you guys when someone goes to this Google page and they are basically told “use Reaper.”

Workflow looks pretty straight forward to me, and this is not about “supporting a format,” it’s about changing Nuendo’s rigid bus architecture so that we are free to do the things we want the way we want (rather than how Dolby etc. wants people to do it) - even Pyramix is getting this right, and you wonder why I’m starting to feel left behind sticking with Nuendo?

#6

Here’ I’m totally with you.

By now, I’ve hit several walls in Nuendo, where I think it’s too rigid and too many manual steps, while software should help us automate as much as possible to get our work done. I guess your issues are sort of related. I’ve filed multiple feature requests and also started a bigger workflow thread for game audio. Most of that can be brought down to this notion of “too rigid” and “too manual”, while it should be “dynamic” and “automatic”.

Maybe I’m just too boisterous.

#7

It could be a generational thing, the open source software movement has always been about keeping things open-ended, it’s antithetical to industrial concerns, but it makes these guys who essentially want a rigid tape+console implementation look like a bunch of dinosaurs, I mean what is it? too many options might fry their heads ; ) what are they going to do when a machine learning based DAW replaces them?

#8

Hi Chris and Tronic,

thank you for your comments and discussion. Chris, your comment is well put.
Tronic, I also agree that Ambisonics is a format that is around for a long time. However, we’re not going to implement technology that has limited use/meaning to our users. Of course we could have integrated the format earlier, but other topics had been more urgent and Ambisonics was not a very relevant format in common post-production workflows before VR became a topic.

However, integrating Ambisonics as a format is still the very beginning of what can be called a workflow.
We will start with that and more things will be coming.

Thanks,
Timo

#9

Ozo is being demonstrated at the Los Angeles AES convention this week !

Looks promising.

#10

The problem with the entire VR thing is that nobody knows if it will ever make it to the mass market.
There are a lot of potential threats to this new technology (price, will a major part of the consumers in the market want to to wear a helmet connected to wires?, the missing killer application for a mass market etc.)

To be quite honest - my bet is that it’s going to be as dead as 3DTV or SACD in the close future because everybody will have tried it once, think it’s “interesting” and then move on.

My 2 cents are that Steinberg is making a huge mistake in wasting development cycles on this. But that might also only be my ignorant bet. I just don’t (want to) see the world connected to a helmet with wires. Maybe I’m getting old.

#11

It was all about VR in IBC this year. And when I wanted to see a better resolution headset there was none. Feature films in VR look weird, pixellated and there are so many technical and artistic issues to to be solved, like depth of field, lighting, action timing to name a few.
I agree we have some time and there is no need for Steinberg to jump in at this point.

Nevertheless I see a lot of potential in VR, but mostly in science, turism and education where you don’t really need fancy VR audio.

People with wires moving their heads definitely look even more isolated than those with phones and facebook.

#12

I do think that VR and 360 movies will become -at least- a hype.
I however don’t think there is business for us in it.

I don’t think that the young people who are now recording their skate/jump or aother adventures with their GoPro’s will go through some kind of audio post process before they post their video on YouTube.
Of course there will be some cool companies that will jump on the format and so some really cool stuff with it.
I don’t think however that there will be (enough) business for us in it.

ANd for the record, I am as sceptical about Dolby Atmos and Auro 3D.
I don’t see the old fashioned 5.1 going away in the near future.
Just for the fact that only one out of 100 movies really benefit from these formats.

I hope I am wrong though …

Fredo

#13

As for VR I’m guessing gaming will be the first field to benefit commercially from it, and it’s also the field in which there’s still a fair amount of revenue to be made and a fair amount of content control (i.e. less copyright infringement compared to music/film/tv). So I think services will still be needed there. If the porn industry follows then that too will possibly push production towards VR.

For more ‘pedestrian’ VR I agree that a lot of people won’t bother with it, but on the other hand I think the bigger issue technically and especially practically is the visual experience. We block out the rest of the world when we put on those VR headsets or glasses, but we don’t actually have to do that with audio. If audio is delivered using regular headsets then the technology is already there. So there’s at least the potential for audio to grow somewhat independently of video, assuming we have the imagination to come up with something useful.

Anyway, I think there’s business in it for engineers, just not for a lot of us, yet. And there’s of course also the question of how much of this will be traditional engineering and how much will be simply creating assets which are then automatically placed/mixed using algorithms.

As for Atmos/Auro3D I slightly disagree though. I have to say that the formats are pretty nice when done right. I think the bigger issue is really aesthetic restraints on the directors. Listening to something like Star Trek “Into Darkness” in Atmos + converted-to-3D was pretty meaningless in my opinion. Both visually and aurally. The mix was just so extremely busy that the enveloping sensation was lost. Gravity on the other hand was completely different. With a more dynamic mix I think Star Trek could have benefited from it as well, but who knows, the movie was pretty much wall-to-wall action anyway so perhaps it wouldn’t have fit the film.

At any rate, I think as we become more comfortable with working with it we’ll see more and better use of it. Some people still argue that stereo is sufficient, yet to me the difference between it and 5.1 is huge once you’re A/B-ing, and I also feel 7.1 is superior to 5.1… and if done well Atmos to 7.1 (I’ve unfortunately never heard Auro 3D).

#14

I’m assuming everyone commenting has experienced Oculus Rift in person? I don’t see how that level of immersion is going away. I think it’s gonna be big, long term.

#15

I concur totally. i think it’ll make it big in video games, documentaries…
Then artistic exhibitions, tourism, theme parks will be more in the realm of Augmented reality. But the two will merge at some point in the future IMO.
I am glad to see Steinberg pioneering in this arena.

#16

Unfortunately this is exactly how I see it. :confused:

#17

Me too. I think VR or AR is going to be huge. But it’s always a question of subsets. A huge number of people have a TV or computer by now in their homes. A subset of those has an HD TV (these days many more, but I’m sure there are still people with old CRTs or flat screens but SD resolution in the world, not everybody is first world), and of those, only a subset has 4K HDTV. That’s the same in gaming. Many people have computers, only a small subset has gaming machines or does gaming on a computer, and of those, only a tiny subset buys peripherals to game in VR or watch a video in VR. So the market will be very small for a long time I think. There will be those super high end demo projects, probably Red Bull and the porn industry will jump on it, but it’s nothing compared to just regular mixing work for TV and cinema.

Also, for games, my opinion is that, because games are interactive, it’s the audio engine that has to do the heavy lifting. Sound designers will continue to provide normal mono or stereo sounds as they always did and the sound engine within a game then has to translate that stuff for VR headphones. Not a lot will change.

For movies, where the position is fixed but the view can be rotated with a headset, there’s probably more work needed to mix that. So there will be more work in movies than games. But again. Tiny subsets. That’s at least my opinion with my current knowledge of things.

#18

Meanwhile, Behringer are working on Augmented Reality for their forthcoming synth, the Deepmind 12.
https://youtu.be/-9MTlsA-wi4

Nothing to do with OZO, but interesting nonetheless in regards to this discussion. At some point, we’ll have immersive sound and AR/VR wtih user interaction.

Now, will it give us jobs ?

#19

The “problem” for us is, that VR unlike movies is a machine mixed format where the computer will do it’s post production on the fly as it happens.

#20

Nailed it.