Ph.D. Defense: Expressive Sound Synthesis for Animation

tugraz

Expressive Sound SynthesisFor Animation

Cécile Picard-Limpens

University of Nice/Sophia-AntipolisÉcole Doctorale STIC

REVES INRIA Sophia-Antipolis, FranceAdvisors: George Drettakis, INRIA Sophia Antipolis (Reves)

François Faure, INRIA Rhône-Alpes (Evasion)Nicolas Tsingos, DOLBY Laboratories, CA, USA

Defense for Ph.D. in Computer Science

C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation1

tugraz

Outline

1 Sound and Virtuality

2 Physics-Based Sound SynthesisContact ModelingResonator Modeling

3 Example-Based SynthesisFlexible Sound Synthesis

4 Perspectives on a Hybrid ModelMotivation and Application

5 Conclusion and DiscussionContributionsExtensions and Applications


Sound andVirtuality

General Background

Motivation

Physics-BasedSynthesis

Example-BasedSynthesis

Perspectives ona Hybrid Model

Conclusion andDiscussion

tugraz

Sound Renderingfor Virtual Reality and Games

Interactive Audio Rendering

(R. Vantielcke - WipeoutHD on Playstation 3)

Traditional ApproachPre-Recordings Triggered

+ : Easy to implement– : Repetitive audio, discrepancies, lack of flexibility


Sound andVirtuality

General Background

Motivation





tugraz

From Playback of Samplesto Synthesis

Digital Sound SynthesisSource modeling ←↩Sound propagation, Sound reception

(ArtiSynth)

TechniquesRigid body simulationFinite Element Method (FEM)

Physical Sound Simulation+ : Physical approach, easy parametrization,

Low memory usage– : Preprocess computation,

Interface between physics and sound system


Sound andVirtuality

General Background

Motivation





tugraz

Controlling the Sound SimulationChallenges

Sound Coherent With Visuals

Unpredictable character of soundsReal-time sound synthesis

Parametrization and Expressiveness

Control and interactivityAuthoring


Sound andVirtuality

General Background

Motivation





tugraz

Our ContributionThree Research Axes

Physics-Based Sound synthesisContact modelingResonator modeling

Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation

Perspectives on a Hybrid Model


Sound andVirtuality


Contact Modeling

Audio TextureSynthesis For ComplexContacts

Resonator Modeling

A Robust andMulti-Scale ModalAnalysis




tugraz

Overview







Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Sound from Contacts

DichotomyImpactsContinuous contacts

Two Schemes for Contact Force Modelling

Additive synthesis

Feed-forward scheme[van den Doel et al . ′01]

Bristle model

Direct computation of contact forces[Avanzini et al . ′02]


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Contact Modeling

What Are The Current Limitationsfor Continuous Contacts?

Rate for physics engine reportNo geometric details when using visual texturesAuthoring and control are challenging

HOW Can We Solve Them?By extracting

Excitation profiles from visual textureswithAdaptive resolution[Picard et al ., VRIPHYS ′08]


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Method for Impact Sounds


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Method for Continuous Contact SoundsExtraction of Excitation Profiles


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Synthesis of Excitation ProfilesFor the Audio Force Modelling

Technique

Extraction from the visual texture imageRe-sampling along the trajectoryof the contact interaction (60Hz vs 44kHz)

Based on the Complexity of the Histogram

Simple texture image:Gradient of the image intensity

Complex texture image:Isocurves of constant brightness (isophotes)


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Complex TexturesCoding the Excitation Profiles

Isophotes = Large amount of dataHow Can We Lighten the Info?

By Coding the Excitation Profiles= Main Features + Noise Part

= +Noise Part: Statistical approximation


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Real-Time Audio ManagementA Flexible Audio Pipeline

Simulations Driven by Ageia’s PhysX (now NVIDIA)


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Audio Texture SynthesisA Solution for Interactive Simulations

A Sound in Coherence with Visuals

Flexible Resolution

Adapted to Procedural Generation


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Overview







Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Vibration ModelsModal Analysis

Generating Sounds Based on Physics SimulationIn computer musics[Iovino et al . ′97, Cook ′02]

In computer graphics[Van Den Doel ′01, O′Brien et al . ′02]

Improvements for Interactive Sound RenderingModal parameter tracking[Maxwell et al . ′07]

Frequency content sparsity[Bonneel et al .′08]


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz


1 Get a Sounding Object and its Geometry

2 Construct the FEM (ex: Tetrahedral Mesh)3 Apply Newton Second Law to DOF

Md + Cd + Kd = f (1)

4 Eigendecomposition ⇒ Modal Parameters

M = LL−T ; L−1KL−T = V ΛV T (2)where V = matrix of eigenvectors

Λ = diagonal matrix of eigenvalues


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz


In Real-time:Modal synthesis

s(t) =∑1

nai sin(wi t)e−di t (3)

Control for vibration models


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz


What AreThe Current Limitations?

Meshing is difficultNo real control on the FEM resolutionNo clear interface between physics and audio

HOW Can We Solve Them?By proposing

A robust and multi-scale modal analysiswhich isCoherent with the physics simulation[Picard et al ., DAFx ′09]


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Our Deformation Model

Inspired from Work by Nesme et al.[Nesme et al .′06]

TechniqueMerged voxels used as Hexahedral Finite Elements

Implementation with the Sofa Framework

Validation of the ModelTests on a metal cube


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Robustness

Robust Even for Non-Manifold Geometries

Material: Aluminium


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Multi-Scale for Efficient Memory Usage

A Squirrel in Pine Wood


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Multi-Scale for Efficient Memory Usage

A Squirrel in Pine Wood: Different FE resolutions

3x3x3 4x4x4 8x8x8 9x9x9

Frequency Content = f (Hexahedral FE Resolution)

Higher resolution models

Frequency centroid shift

Convergence of the frequency content


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

Comparison with Classical Approach

Sounding Bowl - Material: Aluminium

Classical Approach Our Approach(816 modes) (75 modes)


Sound andVirtuality


Contact Modeling


Resonator Modeling





tugraz

A Robust and Multi-Scale Modal AnalysisA Solution for Sound Synthesis

Realistic

Adapted to Non-Manifold Geometries

Resources Flexibility


Sound andVirtuality



Flexible SoundSynthesis

Retargetting ExampleSounds



tugraz

Overview







Sound andVirtuality







tugraz

Implementation of Signal-Based Models

[Dobashi et al.′03

]

Concatenative Synthesis[Roads ′91, Schwarz ′06]

Sound Textures Based on Physics[Cook ′99][Dobashi et al . ′03, Zheng et al . ′09]

[Cook ′99

]Authoring and Interactive Control[Cook ′02]


Sound andVirtuality







tugraz

Implementation of Signal-Based Models

What AreThe Current Limitations?

Processing is not genericParametrizing is difficult

HOW Can We Solve Them?By

Retargetting example soundsTo physics-driven animation[Picard et al ., AES ′09]


Sound andVirtuality







tugraz

Our Approach

SINUSOIDAL

1 DICTIONARY OF AUDIO GRAINS Impulsive / Continuous

2 CORRELATION PATTERNS

R E T A R G E T T I N G T O A N I M A T I O N

AUDIO RENDERER VIDEO RENDERER

PREPROCESSING

INTERACTIVE

OBJECT GEOMETRYVIRTUAL ENVIRONMENT

RIGID-BODYSIMULATION

BUILD COLLISIONSTRUCTURES

DEFINE PROCEDURES

ANIMATION WITH AUDIO

AUDIORECORDING

Our Contributions

TRANSIENT

Amplitude

Time


Sound andVirtuality







tugraz

Preprocess: A Generic Analysis

Impulsive and Continuous ContactsSpectral Modeling Synthesis (SMS) [Serra ′97]

Automatic Extraction of Audio GrainsDictionary: Impulsive/Continuous

Generation of Correlation Patternsbetween original recordings and audio grains


Sound andVirtuality







tugraz

On-Line: Flexible Sound Synthesis

Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude

Interactive Physics-Driven AnimationsPhysics Info for Retargetting

Contact type: impulsive or continuous?Penetration force and relative velocity

Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes

Spectral domain adaptation/modification


Sound andVirtuality







tugraz

Resynthesis of the Original Recordings

94 recordings (14.6Mb)≈ 5000 grains + 94 Correlation Patterns (20% Gain)

Breaking Glass

Shooting Gun

Rolling

Additional Material:http://www-sop.inria.fr/members/Cecile.Picard/"‘Supplemental AES"’


Sound andVirtuality







tugraz

Flexible Audio Shading Approach

Easy Implementation of Time-Scaling

Faster RollingSlower Breaking

Synthesis of An Infinity Similar Audio Eventsby varying the audio content

Rythmic pattern from Breaking StoneNew material content: stone and gun

Rythmic pattern from Breaking GlassNew material content: ceramic


Sound andVirtuality







tugraz

Interactive Physics-Driven Animations

Simulations Driven by Sofa Framework


Sound andVirtuality







tugraz

Retargetting Example SoundsA Solution for Interactive Simulations

Variety

Adapted to Scenarios

Small Memory FootprintReal-Time Rendering

An attractive solution for industrial applications(Eden Games, an ATARI game studio)


Sound andVirtuality




Motivation

A Hybrid Model forFracture Events


tugraz

Overview







Sound andVirtuality




Motivation



tugraz

Sound ModelingWhen Nonlinearity Occurs

Problems of Single ModelsVibration models assume linearityExample-based sounds are hard to parametrize

Previous WorkModeling nonlinearities[O′Brien et al . ′01, Chadwick et al . ′09][Cook ′02]


Sound andVirtuality




Motivation



tugraz

Fracture Events

BackgroundFrequently occur in virtual environments

Visual rendering[O′Brien et al . ′99, ′02][Parker and O′Brien. ′09]

Sound rendering: Little research[Warren et al . ′84] [Rath et al . ′03]

ChallengesEvent depends on the material involvedDifferents phases emerge from fracture event


Sound andVirtuality




Motivation



tugraz

Parametrization of Our Hybrid Model

Selection CriteriaHybrid model applied when nonlinearity occurs

Techniques

FM synthesis

FM synthesisAudio grains

ParametrizationSmooth transition with vibration modelCoherence inside the hybrid model


Sound andVirtuality




Motivation



tugraz

Discussion

Prospective model

Possible problem: report from the physics engine

Simplicity of the tools allows real-time rendering


Sound andVirtuality





Contributions

Extensions andApplications

tugraz

Overview







Sound andVirtuality





Contributions


tugraz

Synthesis of Sounds for AnimationDifficulties

Audio-Visual Coherence

Extremely Dynamic Character

Precision of Synthesis

Large Variety of Objects


Sound andVirtuality





Contributions


tugraz

ContributionsAn Overview

Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms

Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio

Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities


Sound andVirtuality





Contributions


tugraz

ContributionsPerspectives

A Prospective Hybrid Modelfor Complex Physical Phenomena

Focus on NonlinearityCombination of physically basedand example-based methodsApplication Case: Fracture Events


Sound andVirtuality





Contributions


tugraz

Overview







Sound andVirtuality





Contributions


tugraz

Promising Directions for Future Work

Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments

Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene

Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design


Sound andVirtuality





Contributions


tugraz

Promising Directions for Future Work

Hybrid Model for Fracture EventsFracture sound simulation frameworkTracking of relevant physical data


Sound andVirtuality





Contributions


tugraz

Conclusion

New Physically Based Algorithmsfor Sound RenderingFlexibility of Sound ModelingIdeas on an Adequate Hybrid Sound Model

Additional info:http://www-sop.inria.fr/members/Cecile.Picard/


Sound andVirtuality





Contributions


tugraz

Acknowledgements

George Drettakis, François Faure,and Nicolas Tsingos

REVES TeamMarie-Paule Cani and the Evasion TeamPaul G. Kry at the McGill University, Montréal

Eden Games, an ATARI game studio, Lyon


Ph.D. Defense: Expressive Sound Synthesis for Animation

Technology

Transcript of Ph.D. Defense: Expressive Sound Synthesis for Animation