SYNTHESIS BY EXAMPLE Creation of sound synthesis algorithms using evolutionary methods By: Ricardo...

SYNTHESIS BY EXAMPLE

Creation of sound synthesis algorithms using evolutionary methodsBy: Ricardo A. García [email protected]

MIT - Media LabMachine Listening Group© 2001

MIT - Media LabMachine Listening Group

Synthesis by ExampleRicardo A. Garcia © 2001

INTRODUCTION

How is that sound coming out of your soundcard being generated?

Is that a real violin? What kind of controls do we have over it?

• pitch, length, other performance parameters • sound attributes: brightness, harmonic content, etc

The questions: How are those sound synthesis algorithms

conceived? Can this design process be automated?



In a nutshell...

User supplies a set of examples (inputs/target output) The system ”reverse engineers" the sound and

returns a sound synthesis technique that is capable of rendering a sound "similar" to the target sound!

?Inputs

Example

Output

TARGET



Talk outline

Introduction Sound Synthesizers Design of Sound Synthesis Techniques (SST) Proposed approach for design Genetic Programming Automatic Generation of Sound Synthesizers

(AGeSS) system Examples



What is a sound synthesizer?

Any method or device used to produce sound Sound Synthesis Technique (SST): Algorithm that

produces sound samples, usually implemented in a digital computer

Inputs: – internal parameters (init time)– control signals (time varying)

Output: – sound samples (Waveform)

Structured audio, symbolic representationModel analysis



Inside a SST

Functional form of the algorithm Hard to do!• Prefabricated formulas and algorithms

Y=A*sin(x+B)+C

Internal parameters “Easy to do”• Parameter estimation• Analysis tools

A=0.4, B=0.1, C=-0.3



Traditional design

Pick a Functional Form Do parameter estimation to fit your target

…

LPC

FM

Additive

ParameterEstimation

TARGETInputs

USE IT

Human selects Functional Form



Proposed approach “Suggest” functional form (suggest “new” SST) Do parameter estimation Compare with target Suggest new SST

ParameterEstimation

TARGETInputs

SuggestSST

functional form + Initial Parameters

Functional ElementsCompare

with Target



Representation of SSTs

Pseudocode

Formula

Topology graph

assign 3 to xsum x and 0.4return x

)(2sin3.0 tkfy



SST space

SST space: all the possible valid combinations of a set of functional elements and their connections! HUGE!

Hypothesis: ”given a set of inputs, target and fitness metrics, it is possible

to find the functional-form and internal parameters of a SST capable of synthesizing a sound “close” to the target”.



Design of SST as a search

Specs: Inputsoutput (TARGET)Fitness metric

Search: find a point (SST, Functional form + internal parameters) that fulfills the specs

EXAMPLE



The Genetic Programming Loop

O p tim ize in te rn a l P a ram e te rs

R u n to p o lo g y& ren d e r so u n d

C o m p a re ren d e re d so u n dan d ta rg e t so u n d

(F itn ess F u n c tio n )

S e le c t R an d o m In d iv id u a lb ased o n fitn e ss

C rea te N ew In d iv id u a lsu s in g G e n e tic O p era tio n s

(co p y, c ro sso v e r, m u ta tio n )

R ep lace o ld p o p u la tio nw ith n ew p o p u la tio n

S TA RTC rea te n ew

ran d o m p o p u la tio n

N ew P o p u la tio n

Write fitn ess va lue fo r each ind iv id u a l

If f itn e ss v a lu e < d esired f itn essth e n F IN IS H (g o a l a c co m p lish ed )

R en d e red

Targ e t

C o n tro l in p u ts

8 7 .2 3

3 3 2 .8

2 3 6 .6

9 11 .0 1

2 8 7 7 .2 2

HUMANVs

MACHINE



Comparing sounds --fitness function--

Analytical distance metrics:MSE of complex spectrogram(magnitude and phase)



Perceptual Fitness Function

A perceptual distance metricsPerceptually relevant components: full errorAbove masking threshold: error up to M.T.Below masking threshold: no error

O (i,j)T (i,j)

TM (i,j)

fu ll e rro r

O (i,j)T (i,j)

TM (i,j)e rro r u p to TM (i,j)

O (i,j)T (i,j)

TM (i,j)

n o e rro r



Developmentmapping from expression tree to topology graph

S te p 0

S te p 1

S te p 2

S te p 3

S te p 4

S O U R C E 0

S O U R C E 1

1R E N D E R 0

S O U R C E 0

S O U R C E 1

1R E N D E R 0M U LT

S O U R C E 0

S O U R C E 1 R E N D E R 03 2

1M U LT

S O U R C E 0


1M U LT

A D D

S O U R C E 0


1M U LT

A D DK O S C IL

E M B RY O



AGeSS system

Automatic Generation of Sound Synthesizers Matlab scripts and C++ programs

U S E RA G eS S

S Y S T E M

In p u ts- R u n p a ram e te rs- E x am p le s

O u tp u tS S T (to p o lo g y g rap h )

In p u ts Ta rg e t



Examples

A synthetic tone (des35comp2.avi)

A piano note (des44comp2.avi)

FM synthesizer (chow727.avi)



FM synthesis example

Target formula

Output (generation 220+)

FS

MIM

FS

CtAts f

ff 2sin22sin)()(

2010021 )()()()()( ktfkosciltfosciltftfkosciltAkout



FM synthesis example (scale)

Training example (A880, B988)

Output (best of generation 850)

Scale

Scale with best of generation 850



For more information…

Ricardo A. GarciaEmail: [email protected]: http://www.media.mit.edu/~rago

SYNTHESIS BY EXAMPLE Creation of sound synthesis algorithms using evolutionary methods By: Ricardo...

Documents

Transcript of SYNTHESIS BY EXAMPLE Creation of sound synthesis algorithms using evolutionary methods By: Ricardo...