Post on 01-Jan-2016
description
Microsoft Confidential
Audio for Games
Scott SelfonXbox Advanced Technology Group
Microsoft Confidential
OverviewThe roles of audio in gamesAudio challenges on a game consoleScore, Sound Effects, and Dialog
ChallengesTypical SolutionsContent creation / implementation process
Audio integration and end gameCase studies on XboxLeading edge / future of game audioQ & A
Microsoft Confidential
Game Audio Has Come a Long Way…
FidelityTinny mono speaker 5.1 home theater systems1 channel/4 bits 256 channels, 24 bitsFM oscillators 48 kHz wavetable synthesis
Audio processing powerOff-line effects processing real-time DSP effects4-channel mixing Realtime Dolby Digital encodeRealtime 3D positioning
Production pipelineLow level chip programming Separate sound designer/composer and audio programmer
Microsoft Confidential
The Roles of Audio in GamesAudio is 50% of the experienceSupport (improve?) the visualsProvide information about what we don’t seeProvide emotional support for gameplay
Rewards/encouragement
The best audio implementations are rarely noticed by the gamer
1995 MIT studyGame reviews and audio
Microsoft Confidential
Audio Challenges on a Game Console: Aesthetic
Competing with other ‘living room’ experiencesBut much less linear
Events may be skipped, repeat, play in unpredictable orderPlayers expect same kind of realism as they get from non-interactive media
Microsoft Confidential
Audio Challenges on a Game Console: Technical
Finite ResourcesStorage space (not as much of an issue currently)Memory (actual playing sound data, audio library size, etc.)Processing power (realtime effects, mixing, etc.)
InteractivityAvoiding repetition is perhaps most importantSecondary goal: compelling non-linear audioNeed solutions that don’t require a lot of development time
Microsoft Confidential
Audio Challenges on a Game Console: Organizational
“Post-Production” mentality towards audio – scheduled late
Often appropriate in film, television
Often viewed in budgeting as expense rather than investment
Tradition & expectations for game audio - hasn’t been always been of highest quality (see previous slide)
No dedicated audio programming resourceOften up to composer/sound designer to understand and determine what’s possible and feasibleCross-platform audio solutionsSimplest-to-implement programmatic solutions used
Microsoft Confidential
Audio Role Comparison
Film/TV/RadioSound DesignerComposer/arranger(s)Orchestrator(s)Recording Engineer(s)Mixing/Mastering Engineer(s)Music SupervisorMusic EditorSFX EditorADR/Dialog EditorFoley Artist(s)Casting (for voices)
GamesSound Designer/Composer (maybe 2 separate people)Audio programmer (part time)
Microsoft Confidential
Hiring a Content CreatorGenerally ‘buy-out’ (work for hire)
Composer paid straight fee (per platform?)Developer/publisher generally owns all rights to musicMay or may not include cost of recording, musicians, mixing, etc.
May negotiate for publishingPer copy sold royalty (rare)Paid for TV/radio spotsRoyalty for thematic use in sequels, spinoffsOpportunity for game soundtrack CD
Microsoft Confidential
Budgeting for AudioMusical underscore
Live, synth, or mix? And how much?How does it start and stop? (linear, looping, branching)
Licensing existing musicSound Effects / Ambience
Live recordings or sample libraries?Creation of new modeled sounds
Sub-contractingOften essential for large projects
Microsoft Confidential
Laying out an Attack PlanLook at all ways system can make noiseWork with technical document to determine what’s preferable/do-able
E.g. MIDI or streamed (CD/DVD/HD) musicReal-time effects (reverb, filtering, etc.)What other game resources are being used
Maximize audio capabilities while minimizing impact on rest of game
Often different for different parts of game
Microsoft Confidential
Tools, Tools, Tools…
Evaluating for technical and aesthetic criteriaMake authoring and programming pipelines relatively independentMaintain creative flexibility
Linear audioAuthoring: Same as for other mediaProgramming: Cross-platform and platform specific options
Non-linear audio = generally proprietaryPC: DirectMusic or ‘in-house’ enginesConsoles: Platform-specific or ‘in-house’ solutions
Microsoft Confidential
Underscore ChallengesFilm
~90 minutes, often <60 minutes of scoreLinear, unchanging storyline
Game10+ hours of gameplayGameplay events may be quite variable (in order, duration, etc.)Don’t need “music” everywhere (but at least ambience)Score: Often linear tracks several minutes long (may loop)
Microsoft Confidential
Typical Underscore Solutions for Consoles
Create several long linear tracksMay loopMay be played ‘soundtrack’ style (no interaction with gameplay)
Save purely musical score for key one-time game points
Use ambience instead to set moodAvoids repetition of recognizable thematic material
Advanced: Non-linear audioAlternate versions with looping and branchingVarious intensity levels of music that crossfade based on game state
Microsoft Confidential
Licensed Music
Not appropriate for all gamesOften difficult to make interactive/dynamicDisconnects score from moment-to-moment gameplay
Can be effective in some situationsName recognition/cross-marketing potentialCan set titles up for user soundtrack support
Typical usesExisting brands/licenses (esp. movies and TV)Heavily used by racing games
Microsoft Confidential
Composing a Game ScoreSpotting session
Where can music support action?
Compose cues for linear sections of game (for instance, cutscenes)
Need final cut for accurate timings (hah hah)
Record and mixPre-rendered: Synth, live players, mixtureReal-time rendered: MIDI, individual pads, etc.
Implement playback in gameTriggers to start, stop, transition, etc.
Edit to match ‘final’ animations
Microsoft Confidential
Sound Effects ChallengesNo pre-existing soundtrack to work withSame sound event may be heard hundreds of times
VariationEnvironmental interaction
Tying FX to game physics and AISheer number of sound elements to create
Combinations and permutationsEverything that can make a noise should
Microsoft Confidential
Typical Sound Effects SolutionsVariation
Implement basic pitch randomizationSeveral versions of each waveBreak a sound into components and record versions of each of those
Tying effects to game physics / AIImplementation varies per-titleOften vast majority of dev time spent here
Advanced: Dynamic processed soundsEnvironmental effects (reverb)Filtering (force of impacts, radioizing, etc.)
Microsoft Confidential
Creating Ambience and Sound Effects
Spotting session (if game playable), or look at artwork (environments, objects)
List everything that can make noise, prioritizeObject interaction matrix
Often start with organic soundsSound effects library ($$$, can be generic)Create own samples (free, unique)
Sound processing (pre-rendered or dynamic)
Reverb, filtering, chorus, effects
Microsoft Confidential
Dialog Challenges
Casting – producer may want involvementDirecting / CoachingStitching (first | and | 10 | from the | 25)Dialog repetition is easily recognizedLocalization
Re-recording voice talentEngine integrationGrammatical differencesRecreating pre-rendered processing
Microsoft Confidential
Typical Dialog SolutionsTriggers for dialog (just like music)Variation for common lines (often too few)Stitching – coach the talentLocalization
Solutions vary per-titleVery careful asset managementSometimes source assets can just be swapped
Advanced: Sophisticated dialog engineVariation tracking to avoid repetitionPeople don’t always have to say anything
Microsoft Confidential
Prepping for DialogContent creator works with producer, game designer, and/or writer to:
List and describe charactersWrite each character’s scripts
Cast person for each characterReal voice talent makes a differenceMany actors can do multiple roles
Microsoft Confidential
Dialog ProductionRecord all dialog
Encourage improvisationCoaching for creating convincing stitchable dialog
Cut up dialog into separate filesClean up for stitching – sound designer for sports title can spend majority of time here
Post-recording processingDocument everything for localization
Microsoft Confidential
Squeezing it all into 5 bytes
Best to determine resources up frontMemory and CPU availableDisk space and bandwidth available
Inevitably, not enough roomAsset swapping (programming time)CompressionUse of streaming where possibleCut assets
Microsoft Confidential
And the Handoff…Source assets given to game
Typically high fidelity + compressed versions deliveredFreelance composer/sound designer: often end of work
Programmer integrates audio assets into titleOften tasked with mixing (either with or without content creator feedback)Implement runtime sound processing (reverb, effects, filtering, etc.) unless engine natively supports
Testers confirm sounds are in placeAsset/trigger lists, scripts, and/or manual triggersVariation testing?
Microsoft Confidential
After the Fact: More Challenges
Archive, archive, archiveCreate a recording of typical gameplay for referenceDisaster always strikes at the 11th hourKeep an organized collection of any sounds created (spreadsheet, database)
LocalizationVoice talent recasting, re-recordingRecreate processing (unless real-time used)
Releasing a CD
Microsoft Confidential
Audio Case Study:Project Gotham Racing
Stereo ambience, with some 3D positioned foreground ambient elementsPlayer’s car
Stereo engine ambienceEngine sound varies with RPM, velocity, physics, etc.Tire sound varies with surface (sidewalk, stairs, street, etc.)
3D positioning on other carsEngines (varies with RPM, velocity, physics)Car hornsCornering, tire squeals, collisions, etc. (all with some variation)
Dynamic soundsTire squeals (randomized between several versions and pitches)Collisions (depend on speed, material involved, etc.)
Microsoft Confidential
Project Gotham Racing: Music
Game disc-supplied licensed songs (WMA)User Soundtrack support
Allows for personalized driving music mixes
Music uniquely integrated into gameUse of ‘car radio’ or ‘CD’ playback‘Radio’: Location-specific DJ interstitials
Microsoft Confidential
Audio Case Study: Splinter CellDetailed ambiences, foley, sound effectsAudio as a gameplay element (stealth)
Stealth: Noise tied to game AIFootsteps, interaction with objects, weapons
Audio as an informational toolInteractive musical scoreTells player when enemies alertedUse of musically appropriate boundaries for transitions
Microsoft Confidential
Audio Case Study: Halo
4,311 triggerable events (soundtags) 11,629 source waves (soundfiles)2.5 GB uncompressed audio
22 kHz mono and 44 kHz stereo
80 minutes of source music20+ hours of gameplay
12 voice talents, 7,735 lines of dialog…all in 2-MB memory footprint
Microsoft Confidential
Halo: Audio Tools UsedPre-rendered linear music/FX = traditional standard audio post production tools
Pro Tools, Studio Vision, Peak, Digital Performer
Dynamic audio = custom tool/engine, randomization/permutation for each event
Pick one or more waves to playRandom variation in pitchWeighting for how often a sound will be used
Microsoft Confidential
Halo Audio Engine: DialogSimulation of actual speech patterns
Close connection to game AILarge number of events a character can respond to, multiple phrases per actionCharacters don’t always speak
Tracking speechMultiple voice talents for multiple charactersCharacter only says one thing at a timePositioning (3D versus locked-to-speaker)Repetition avoidance
Microsoft Confidential
Halo Audio Engine: Sound EffectsVariations in all sound effects
Weighting (frequency for multiple versions of sound)Randomized pitch control
Variations in ambienceVarious loops of different lengthsLayered 3D-positioned detail sounds (birds, crickets, machinery hum, etc.)
“Cascading” sound effectsIf many instances of an effect are playing, play a single more complex sound
Response to game environmentSpecific conditions/events/game physicsEnvironment interaction (reverb)3D spatial positioning versus static
Microsoft Confidential
Halo Audio Engine: MusicTypically linear or looping
If looped, alternate versions with varying lengthsDynamic execution (“spotted” with programmer)
Varying transition types based on situationBranchingCrossfadesGame event tags
Appropriate absence of musicSilence is golden - not scored wall to wallTimeouts
Microsoft Confidential
Halo 5.1 UsageSound effects typically 3D-positioned (4 channels)
Some ambience 3D-positioned as well
“Quad” music and ambienceDoubled stereo (fronts 100% / rears 50%)Crossfades based on location
Center channelCortana (“first person” dialog)Radio communication
LFE channelUsed to emphasize more massive sounds
Heartbeat (low health)Explosions
Microsoft Confidential
Halo Environmental Reverb Usage
Reverb settings changed based on environment
Indoors versus outsideConfined spaces versus large spacesCaves versus hallways
Sound interactions with reverbEspecially 3D-positioned sound effectsSome ambient elements also
Microsoft Confidential
Halo Implementation (1)
2-MB in-memory footprint22-kHz mono ADPCM for 3D sounds44-kHz stereo ADPCM for music + ambienceADPCM = ~3.5:1 compression
Use of Xbox hard diskCopy all sound data to drive at start of level (faster to read than DVD)Pull small chunks into memory rather than entire banks (pseudo-”streaming”)Drive persistence = faster load next time
Microsoft Confidential
Halo Implementation (2)
Sound during load screensSmall looped sampleMultiple instances played with pitch shifting and filtering as load progresses“Dynamic” – speed and actual sound vary with read time
Microsoft Confidential
Leading Edge Audio ChallengesVoice integration (esp. for online titles)
Chat and gameplay implicationsManipulation (masking, radioizing, etc.)
Voice recognitionCommand and control
Further multichannel audio usageFuller 5.1 implementationsDynamic audio mixing
Physical modelingStill too expensive (CPU, programming effort) for most games
Microsoft Confidential
Wrap Up:The State of Game Audio
Lots of independent composers/sound designersMany coming from outside game industryG.A.N.G. (www.audiogang.com)
Still typical for developers to reinvent audio engine with each title (newer tools try to solve)Xbox efforts
Xbox Registered Developer and Registered Content Creator ProgramsTools and dedicated support for audio content creators and programmersGame Designer/Producer education
Microsoft Confidential
Q & Ascottsel@xbox.comResources
G.A.N.G. (www.audiogang.com)Game Audio Pro e-mail list (Yahoo)DirectMusic (www.microsoft.com/directx)2003 Game Developer Conference lectures