Physically-Based Parametric Sound Synthesis and Control prc/   Physically-Based Parametric

download Physically-Based Parametric Sound Synthesis and Control prc/   Physically-Based Parametric

of 37

  • date post

    12-Jun-2018
  • Category

    Documents

  • view

    222
  • download

    0

Embed Size (px)

Transcript of Physically-Based Parametric Sound Synthesis and Control prc/   Physically-Based Parametric

  • Course #2 Sound - 1

    Perry R. Cook

    Princeton Computer Science(also Music)

    Physically-Based ParametricSound Synthesis and Control

    Course Introduction

    Parametric Synthesis and Control ofReal-World Sounds for

    virtual reality

    games

    production

    auditory display

    interactive art

    interaction design

  • Course #2 Sound - 2

    Schedule0:00 Welcome, Overview0:05 Views of Sound0:15 Spectra, Spectral Models0:30 Subtractive and Modal Models1:00 Physical Models: Waveguides and variants1:20 Particle Models1:40 Friction and Turbulence1:45 Control Demos, Animation Examples1:55 Wrap Up

    Views of Sound

    Sound is a recorded waveform PCM playback is all we need for interactions, movies, games, etc.

    (Not true!!)

    Time Domain x( t ) (from physics)

    Frequency Domain X( f ) (from math)

    Production what caused it

    Perception our image of it

  • Course #2 Sound - 3

    Views of Sound

    Time Domain is most closely related to

    Production

    Frequency Domain is most closely related to

    Perception

    we will see that many hybrids abound

    Views of Sound: Time Domain

    Sound is produced/modeled by physics,described by quantities of

    Force force = mass * acceleration

    Position x(t) actually < x(t), y(t), z(t) >

    Velocity Rate of change of position dx/dt

    Acceleration Rate of change of velocity dv/dt

    Examples: Mass+Spring+Damper Wave Equation

  • Course #2 Sound - 4

    Mass/Spring/Damper

    F = ma = - ky - rv - mg

    F = ma = - ky - rv

    (if gravity negligible)

    022

    =++ ym

    k

    dt

    dy

    m

    r

    dt

    yd

    ( )D Dr m k m2 0+ + =/ /

    2nd Order Linear Diff Eq. Solution

    1) Underdamped:

    y(t) = Y0 e-t/ cos( t )

    exp. * oscillation

    2) Critically damped:

    fast exponential decay

    3) Overdamped:

    slow exponential decay

  • Course #2 Sound - 5

    Wave Equation

    dfy = (T sin) x+dx - (Tsin)x f(x+dx) = f(x) + f/x dx + (Taylors series) sin = (for small ) F = ma = dx d2y/dt2 (for each dx of string)

    The wave equation: (c2 = T / )) 2

    2

    22

    2 1

    dt

    yd

    cdx

    yd =

    Views of Sound: ProductionThroughout most of history, somephysical mechanism was responsiblefor sound production.

    From our experience, certain gesturesproduce certain audible results

    Examples:Hit harder --> louder AND brighterCant move instantaneouslyCant do exactly the same thing twice

  • Course #2 Sound - 6

    Sound Views: Frequency Domain

    Frequency Domain:

    Many physical systems have modes (damped oscillations)

    Wave equation (2nd order) orBar equation (4th order) need 2 or 4

    boundary conditions for solution

    Once boundary conditions are setsolutions are sums of exponentially damped sines

    the sinusoids are Modes

    The (discrete) Fourier Series

    A time waveform is a sum of sinusoids

    (A is complex)x n Aj nm

    Nmn

    N

    ( ) exp( )==

    20

    1

    =

    =

    +=

    +=

    1

    0

    1

    0

    )2

    cos(

    )2

    cos()2

    sin(

    N

    nmm

    N

    nmm

    N

    nmD

    N

    nmC

    N

    nmB

  • Course #2 Sound - 7

    The (discrete) Fourier Transform

    A m X SRATE m N x njnm

    Nn

    N

    ( ) ( * / ) ( ) exp( )= =

    =

    20

    1

    sinusoidal ASpectrum is a decomposition

    of a signal

    This transform is unique and invertible

    (non-parametric representation like sampling)

    Spectra: Magnitude and PhaseOften only magnitude is plotted

    Human perception is most sensitive to magnitude

    Environment corrupts and changes phase

    2 (pseudo-3) dimensional plots easy to view

    Phase is important, however

    Especially for transients (attacks, consonants, etc.)

    If we know instantaneous amplitude and frequency, we can derive phase

  • Course #2 Sound - 8

    Common Types of Spectra

    Harmonic

    sines at integer

    multiple freqs.

    Inharmonic

    sines (modes),

    but not integer

    multiples

    Common Types of Spectra

    Noise

    random

    amplitudes

    and phases

    Mixtures

    (most real-

    world sounds)

  • Course #2 Sound - 9

    Views of Sound: PerceptionHuman sound perception:

    Auditory cortex:further refinetime & frequencyinformation

    Cochlea:convert tofrequencydependentnerve firings

    Ear:receive1-Dwaves

    Brain:Higher levelcognition,objectformation,interpretation

    Perception: Spectral Shape

    Formants(resonances)are peaks inspectrum.

    Human ear issensitive tothese peaks.

  • Course #2 Sound - 10

    Spectral Shape and Timbre

    Quality of asound isdetermined bymany factors

    Spectral shapeis one importantattribute

    Spectra Vary in Time

    Spectrogram (sonogram)amplitude as darkness (color) vs. frequency and time

  • Course #2 Sound - 11

    Spectra in Time (cont.)

    Waterfall Plotpseudo 3-d amplitude as heightvs. freq. and time

    Each horizontal sliceis an amplitude vs.time magnitudespectrum

    Additive Synthesis

    ( ) ( )[ ]=

    =R

    rrr ttAts

    1

    cos)(

    The sinusoidal model:

    R : number of sinewave components,Ar (t) : instantaneous amplitude,r (t) : instantaneous phase

    Control the amplitudeand frequency of aset of oscillators

  • Course #2 Sound - 12

    Sinusoidal Modeling

    Vocoders Dudley 39, Many more since

    Sinusoidal Models Macaulay and Quatieri 86

    SANSY/SMS Sines + Stochastic Serra and Smith 87

    Lemur Fitts and Hakken 92

    FFT-1 Freed, Rodet and Depalle 96

    Transients Verma, Meng 98

    frequency of partials

    magnitude of partials

    Sinusoidal Analysis Tracks

  • Course #2 Sound - 13

    Magnitude-only synthesis

    original sound

    magnitude-onlysynthesis

    mS

    AAAmA

    lll )()(

    11

    += m

    Sm

    lll )()(

    11

    += )()1()( mlm rrr +=

    ( ) ( )[ ]=

    =lR

    r

    lr

    lr

    l tmAms1

    cos)(

    Magnitude and Phase Synthesis

    ( ) rrt rr dt ++= )()( 00

    original sound

    synthesized soundwith phase matching

    x(n)

    s(n)

    r(t) : instantaneous frequencyr(0) : initial phase valuer : fixed phase offset

  • Course #2 Sound - 14

    Deterministic plus StochasticSynthesis (SMS)

    [ ] )()(cos)()(1

    tettAtsR

    rrr +=

    =

    ( ) dt t rr = 0)(when sinusoids are very stable, the instantaneous phasecan be calculated by:

    otherwise:

    model:

    Ar(t), r(t): instantaneous amplitude and phase of rth sinusoid,e(t) : residual component.

    ( ) rrt

    rr dt ++= )()( 00r(t) : instantaneous radian frequencyr(0) : initial phase value,r : fixed phase offset

    Residual (stochastic component)

    Resynthesis (with

    phase) of sine

    components

    allows extraction

    and modeling of

    residual component

  • Course #2 Sound - 15

    Basic SMS parameters Instantaneous frequency and amplitude of partials Instantaneous spectrum of residual

    Instantaneous attributes Fundamental frequency Amplitude and spectral shape of sinusoidal components Amplitude and spectral shape of residual Degree of harmonicity Noisiness Spectral tilt Spectral centroid

    Region Attributes

    SMS High level attributes

    Transients

    Transients are vertical stripes in spectrogram

    Use DCT to transform backto time domain, then dosinusoidal track analysis on that

    Detection is the hard part

  • Course #2 Sound - 16

    Sines + Noise + TransientsStrengths and WeaknessesStrengths: General signal model

    (doesnt care what made it)

    Closed form identity analysis/resynthesis

    Perceptual motivations (somewhat, not all)

    Weaknesses: No gestural parameterization

    No physics without lots of extra work

    No guaranteed compression or understanding

    Subtractive Synthesis: LPC

    =

    =m

    kk knxcnx

    1

    )()( LPC

    =

    =m

    kk knxcnx

    1

    )()()()()( nxnxne =

    =

    =P

    n

    neP

    E0

    2)(1

    Prediction

    Error

    Mean Squared Errorover block length P

  • Course #2 Sound - 17

    LPC continued

    LPC is wellsuited tospeech

    Also wellsuited tosounds withresonances

    LPC filter envelope (smooth line)fit to human vowel sound / i / (eee)

    Subtractive Synthesis: Formants

    Factor LPC into resonators

    Eigenmodemotivations

    Perceptual motivations

    Vocal production motivations

    Excite with pulse(s), noise, or residual

  • Course #2 Sound - 18

    Modal Synthesis

    Systems with resonances (eigenmodes of vibration)

    Bars, plates, tubes, rooms, etc.

    Practical and efficient, if few modes

    Essentially a subtractive model in thatthere is some excitationand some filters to shape it.

    Modal Synthesis: Strings

    Strings are pinned at both ends

    Generally harmonic relationship

    Stiffness can cause minor stretching ofharmonic frequencies

  • Course #2 Sound - 19

    Modal Synthesis: Bars

    Modes of Bars: Free at each end

    These would be harmonic, but stiffness ofrigid bars stretches frequencies.

    Modes: 1.0, 2.765, 5.404, 8.933

    Modal Synthesis: Tubes

    Open or closed at each end, same asstrings and bars, b