Ananya Misra, Perry Cook Princeton University
description
Transcript of Ananya Misra, Perry Cook Princeton University
![Page 1: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/1.jpg)
Toward Synthesized Environments: A Survey of Analysis and Synthesis Methods
for Sound Designers and Composers
Ananya Misra, Perry Cook
Princeton University
![Page 2: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/2.jpg)
Motivation
• Multitude of sounds in the sonic landscape• Multitude of algorithms• Better knowledge of a variety of algorithms
=> empowerment to create rich sound scenes
![Page 3: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/3.jpg)
Some Related Surveys
• Smith. “Viewpoints on the History of Digital Synthesis.” ICMC 1991.
• Pope. “A Taxonomy of Computer Music.” Contemporary Music Review 1996.
• Tolonen et al. “Evaluation of Modern Sound Synthesis Methods.” Tech. Rep. 1998.
• Vercoe et al. “Structured Audio: Creation, Transmission, and Rendering of Parametric Sound Representations.” Proc. IEEE 1998.
• Widmer et al. “Sound and Music Computing: Research Trends and Some Key Issues.” JNMR 2007.
This one: from the perspective of creating complex environmental sound scenes or compositions
![Page 4: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/4.jpg)
Overview
• Abstract synthesis algorithms• Synthesis from “scratch”• Synthesis from existing sounds
– Concatenative techniques– Additive synthesis– Subtractive synthesis and other techniques
• Analysis not for synthesis
Disclaimer: No taxonomy is clean or 1-1 from method to class
![Page 5: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/5.jpg)
Abstract synthesis algorithms
![Page 6: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/6.jpg)
Oscillators
• From analog days
Sine wave Triangle wave Sawtooth wave
Alarm clock
Examples by ChucK
![Page 7: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/7.jpg)
Frequency Modulation
• Modulation of one oscillator’s phase by another’s output
Basic example
Doorbells
Examples by ChucK
![Page 8: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/8.jpg)
More
• Circle maps as nonlinear oscillators (Essl, ICMC 2006)
• Errant sound synthesis: “the potential of any algorithm cast into the audio range” (Collins, ICMC 2008)
• Auditory display and sonification
![Page 9: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/9.jpg)
Synthesis from scratch
Synthesis from physical or perceptual models, without the raw material of
existing audio samples
![Page 10: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/10.jpg)
Physical models
High-level parametric control over synthesized sound
• Plucked string model, waveguides and more• Reed and bowstring models, singing voice
synthesis, percussive sounds• Synthesis ToolKit => PeRColate (Max/MSP),
ChucK, SuperCollider• Real-world contact / motion sounds
Modal synthesis Gait modelingCook, 2002
![Page 11: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/11.jpg)
Perceptual models
Give desired perceptual characteristics• For speech and singing (Cook, CMJ 1996):
– Formant synthesizers– Formant wave functions (FOFs) (Rodet,
CMJ 1984) • General: Feature-based synthesis (Hoffman,
ICMC 2006)
=>
![Page 12: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/12.jpg)
Synthesis from existing sounds
Creation of sound from existing sound, synthesis by analysis
![Page 13: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/13.jpg)
Concatenative techniques
• Rearrangement of samples in the time domain
• Wavetable synthesis• Concatenative synthesis (Schwarz, JNMR
2006):– Source sound segmented into units– Target sound– Set of unit descriptors– Unit selection algorithm
![Page 14: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/14.jpg)
Concatenative techniques
• Applications: singing, instruments, audio mosaicing
• Granular synthesis: concatenating usually short “sound grains” (Truax, 1990)
![Page 15: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/15.jpg)
Concatenative techniques: granular synthesis
• Formant wave functions, when using arbitrary sound samples
• Dictionary-based methods with time-localized waveforms (Sturm, ICMC 2008) => analytical counterpart
• TAPESTREA: granular synthesis by parametrically looping transformed events
![Page 16: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/16.jpg)
Concatenative techniques for sound textures
• Splitting soundscapes into syllable-like segments (Hoskinson, ICMC 2001)
• Soundscape generation from database of annotated sound files (Birchfield, ICMC 2005)
• Fast sound texture synthesis using overlap-add (Fröjd, ICMC 2007)
From Fröjd&Horner, 2007;algorithm offered in TAPESTREA
![Page 17: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/17.jpg)
Additive synthesis
Spectral analysis, addition of resulting signals
• Channel vocoder: bank of bandpass filters• Phase vocoder: FFT to get phase as well as
magnitude of each frequency band• Pitch and time transformations (offered in
TAPESTREA)• Cross-synthesis
![Page 18: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/18.jpg)
Additive synthesis: sinusoidal modeling
• Speech signals can be modeled using a few sinusoids (McAulay 1986, Quatieri 1986)
• Spectral modeling synthesis (Serra, 1989): sines + noise
• Lemur/Loris, CLAM, SMS, SPEAR, AudioSculpt• Used to extract and transform some
environmental sounds in TAPESTREA
Transformed windchimes Baby chorus / cacophony
![Page 19: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/19.jpg)
Sines + Transients + Noise
Decompose into transients (brief, noisy events) as well as sines and noise
• Transient / onset detection techniques:– Time-domain envelope following (in TAPESTREA)– Comparing energy envelopes of original and
residual noise signals (Levine, 1998)– Comparing energy in short and long signal
segments (Verma, 1998) (in TAPESTREA)– Frequency-domain techniques
![Page 20: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/20.jpg)
Subtractive synthesis and linear predictive coding
• Filtering a signal to shape it by subtracting unwanted components
• LPC: Source-filter model where next = linear combination of previous samples (Atal, 1970)
• Musical composition (Lansky, 1989)• Sound texture synthesis (Athineos, 2003;
Zhu, 2004)
Noise-excited LPCTime-frequency LPC;Athineos&Ellis, 2003
![Page 21: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/21.jpg)
Other tools for sound textures
• Wavelet-tree learning (Dubnov, 2002) (offered in TAPESTREA)
• Parametrically modeling and transforming stochastic components (Miner, 2002)
• Inferring statistical distributions of events (Zhu, 2003)
![Page 22: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/22.jpg)
Analysis not for synthesis
Content-based analysis methods not necessarily designed for synthesis. May provide information to guide
synthesis algorithms.
![Page 23: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/23.jpg)
Analyze to…
• Represent an audio signal in structurally or perceptually meaningful ways
• Understand and use a collection of sounds on a global level
![Page 24: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/24.jpg)
Representing a signal
• Source separation: Computational auditory scene analysis (Ellis, 1992; Melih, 2000)– Automatic and manual grouping of partials offered
in TAPESTREA
• Source separation: Multiple fundamental frequency estimation (Klapuri, 2004)
• Blind source separation (Hoffman, 2009)• Music transcription
![Page 25: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/25.jpg)
Understanding a collection
• Comparison actions: content-based classification, search, recommendation, …
• Automatic timbre recognition• Music information retrieval tools, e.g.
MARSYAS
![Page 26: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/26.jpg)
Conclusions
![Page 27: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/27.jpg)
Advantages of parametric synthesis algorithms
• Many algorithms may contribute to one simple piece
• Many available tools: programming languages, specialized libraries, graphical software
• Compression• Ability to re-render over and over with
changes, interactively or in real-time• Rich palette of techniques for composers
![Page 28: Ananya Misra, Perry Cook Princeton University](https://reader036.fdocuments.us/reader036/viewer/2022062315/5681580c550346895dc57c36/html5/thumbnails/28.jpg)
Taxonomy of methods by soundsSound/Goal Methods
Abstract FM, non-linear oscillators, feature-based synthesis, wavetables, granular synthesis
Acoustic instruments Wavetables, waveguides / physical models, granular synthesis, additive synthesis
Contact sounds Physical models
Cross-synthesis LPC, vocoders
Pitch/time transformations
LPC, vocoders, granular synthesis, additive synthesis
Pitched sounds Additive synthesis, granular synthesis, FM, oscillators
Singing voice FM, formant synthesis, FOFs, granular synthesis, additive synthesis
Speech Formant synthesis, FOFs, granular synthesis, vocoders, additive synthesis, LPC
Textures and soundscapes
Granular synthesis, LPC, stochastic and wavelet-based methods
Transients Onset detection, physical models, granular synthesis, sines+transients+noise models