Audio Q&A: MGS on DSP

John Broomhall: Welcome to Develop, Guy – what does your role in MGS involve?

Guy Whitmore: My team is a resource for the entire first party portfolio of MGS games for 360/Windows so we touch dozens of titles each year. Our work can cover creative and technical consulting through to heading up and undertaking audio recording and production – whatever’s needed. We share our resources and assets over several studios and each of our Audio Directors oversees a collection of games titles/franchises.

JB: Presumably your overall aim is to make sure 360/Windows games – particularly first party games – sound awesome?

GW: That’s the bottom line mandate – at first party, the goal is really to have stand-out games that sell more hardware, and also to push the envelope on quality – to be models for other upcoming titles.

JB: So you have this wide-ranging oversight of audio production across many games on platforms that have a lot of raw power for real-time digital signal processing (DSP) at run-time in games… do you think people are using it enough?

GW: No I don’t. With some smart budgeting you can do amazing things with high quality DSPs and I really want to push this approach. DSP is kind of like the shaders and particle effects of game audio – what those things do to visuals – say real-time lighting changes – creates believability. I think the same can be said of audio DSP techniques – sounds become part of one environment rather than disparate. A few games have used DSP for run-time mastering, not only helping control volume levels but also achieving mastering in the true sense where you may sweeten the highs and bringing out a few lows – creating coherence. Environmental reverb has been used to good effect, but there’s a lot of room for exploration. Currently most audio engines have one reverb algorithm – but what if you don’t particularly like it? The future is sound designers choosing from a variety of DSP options whether created in-house or externally.

JB: You mean third party plugins?

GW: Yes definitely. I see a model coming forward where the same DSP manufacturers that create software for the pro-audio market start to offer their products to the game market for real-time use. These two worlds are just dying to meet each other…

JB: Interesting – do we have enough processing power to run them or do we need specially adapted ‘lite’ versions?

GJ: Many of these plugins could probably work right out of the box. However, efficiency is very important even with the power of the today’s consoles – so they need optimizing.

JB: But nevertheless, you’re still saying that commercial plugin makers like Waves should be having a dialogue with the games business about producing run-time versions of their products for games?

GW: Undoubtedly, and that’s my message here – they really don’t have that far to go. It may not be that hard to optimize them and convert into workable formats for games.

JB: So how can real-time DSP techniques really take us forward in game audio – you’ve talked about believability and variation…

GW: I believe it’s the only way we’re going to really stand out next to any movie in terms of sheer sound creativity. In a non-linear medium the only way that audio can really be immersed with game-play and visuals I think is with real-time DSP. Short of that we can just sort of lay sound on top – there’s a potential disconnect.

JB: I was blown away when interviewing Nick Wiswell of Bizarre Creations for develop magazine a while ago – he told me at that point they were using a whole 360 core for audio processing so clearly run-time audio DSP was critical in Project Gotham Racing – it was seen as a next-gen approach…

GW: Yes! That’s a great example and I worked closely with Nick on versions PGR 3 and 4. There were at least half a dozen DSPs on the cars in addition to the normal pitching and filtering plus real-time mastering plugins running – those guys went all out.

JB: What would be an example from a completely different type of game genre?

GW: Well, take Fable 2. The overall mood can change from light to dark over many hours’ game-play. You want to subtly reflect that using the same ambient sound assets so you apply a combination of filters to make it sound darker. It’s opening up a whole new toolbox for sound designers who I think will increasingly work in the context of a game rather than out of context – it’s a very big deal creatively. Currently most work offline on a DAW perfecting their sound but more and more we’ll see really good assets going into a game that are nevertheless somewhat raw. Putting that extra layer of polish will actually occur in-game and real-time DSP is the only way that can happen. And this is about subtlety as well as car engine distortion – changing filter cut-off in real-time in response to polling of game variables may be a sub-conscious effect but a vital artistic contribution to an overall wow gaming experience.

JB: So we should be experimenting more?

GW: Sure – for example last week we were playing around with an aural exciter on sword play – by changing some of the parameters in real time the sword hits had tons more variation – very cool.

JB: What about the business end for these third party plugin companies?

GW: Well licensing has happened in the past – e.g. Bungie and the Waves plugins. It worked well for them in the context of Halo3 but the crucial next step I saw was to license a set of plugins we could easily use across our entire portfolio. We’re currently finalizing a multi-DSP, multi-license deal with Wave Arts, Inc. –www.wavearts.com – who are converting some of their existing pro-audio DSPs and formatting them to work on the XBOX360 and PC in the XAPO format (part of the XAudio2 specification). The great thing is that one deal like this allows a small company to do the initial legwork to make their products game-friendly – and then they’re free to license to other developers and publishers.

JB: Interesting point. So for all of us in cross-platform development, we need to see the audio middleware guys leading the way here?

GW: Exactly right! Think of a future with great third party cross-platform engines where you download the demo version and lo and behold, there are also demo versions of all these cool DSP plugins.

JB: It makes sense to me and would ultimately lead to better game audio. Do you think there’s any hope for a holy grail of unified plugin format eventually?

GW: Boy… like anyone I would love it because it would make life for all of us easier – maybe we can do what the pro-audio DAW world couldn’t… I guess that has to be the ideal. For now though, we’re drawing together all the best DSP from MGS games and looking to make it available as a centralised resource accessible through my group. But of course if someone is developing cross-platform and they’re interested in the plugins from Wave Arts, there’s nothing to stop a direct approach. Imagine if every major studio/publisher inspired a third party company to create DSP for their platform who could then license it to anyone – before you know it we would have a viable ecosystem of middleware DSP. It would make doing audio for games that much more vibrant I think.

JB: Presumably all the parameters within any of these plugins can be changed in real-time?

GW: Yes – depending on how your game and game audio engine are set up of course, but no problem in principle.

JB: Do you think a time will come when rather than the concept of presets we’ll think more in terms of preset behaviours for real-time plugins?

GW: Yes, presets are just snapshots – that’s cool and useful. But the concept of preset behaviours is important – including in interactive mixing.

JB: So do you see a future where an external mixing engineer – say a film guy – could sit down with pro-audio friendly interfaces and software access available not only to alter volume levels but also all the DSP plugins – so for instance he can get under the hood and change a compressor setting?

GW: Absolutely. We need a complete flexibility for DSP in mixing games and we are moving in this direction – every game we take it one step further formalising the mixing process borrowing somewhat from Hollywood but also creating our own rules for our non-linear medium. We do a lot of production in-house but we also work out-of-house with experienced movie guys like Scott Gershin at Soundelux. More and more, I’m scheduling time at the end of games where we sit down for at least a couple of weeks in the mix studio with game and devkit changing settings as we walk the various scenes. That process is crucial – and DSP is a vital part of it – for instance changing reverbs. You’re making thousands of decisions in the course of an hour of mixing because you can iterate so quickly. I think in the future we’ll have a role of ‘Game Mixer’ and I think it will be very important going forward.

JB: As for dynamic mixing – presumably we also want to get funky with fader moves and DSP behaviours (in response to game variables we’re polling) for purely creative, moment-enhancing, subjective-effect reasons?

GW: Yes – I think most dynamic mix decisions would be based on emotional subjective things rather than realism. For me, mixing for games is way too stuck in a literal distance-based approach – everything getting quieter the further away it is, regardless of how important it is. We need a re-think and again, to me, DSP is a very important component, particularly when you’re getting into ‘I want this section to feel dreamy the second time’ or ‘I want the player to suddenly feel the sadness of the character’s feeling’. It’s the same type of emotional choices a sound designer for a movie might make but working in a non-linear context…

JB: Pretty exciting prospect…

GW: It really is.