Timbre Recognition by Spectrum Analysis

MAT201A 2008 Spring Project

JungHo Ohn(ohn@umail.ucsb.edu)

Soo Hwan Park(spark2007@cs.ucsb.edu)

mailto:ohn@umail.ucsb.edu

Agenda

Motivation: Music Information RetrievalTimbre RecognitionAnalysis in time domain & freq. domain

Features of Music Signal for Timbre RecognitionChallengesOur Approach Result

Music Information Retrieval (MIR)

uplifting music & violin

Music Information Retrieval (MIR)

Current Audio Search Engine- Semantic Information

: Title/Tag information- MP3 Tag

: Title, Artist, Album, Year,Comment, Genre Human Work

- Query = Beatles & Let it be

Future Audio Search Engine- Automatic Content-based Indexing- Cords, Notes, Timbre, Beat, Tempo, . - Query = Sad Song and Violin or

Up-beat Blues, but Not Too Slow

Timbre/Instrument Recognition

RecognitionA sound object currently being heard and its correspondence to something that has been heard in the pastMeasuring Similarity: melodic similarity, rhythmic and timbre similarities, genre and style similarities

Timbrea quality of sound that distinguishes one music instrument from anotherImportant factor for Classifying musicSo far, no standard parameters

Analysis in time domain

Correlation methodPitch informationSimilarity of a signal

( * )[ ] [ ] [ ]k

f g n f k g n k

=

= +

Test musical sound4A viola3C viola

Analysis in freq. domain

Spectrum AnalysisSpectrum AnalysisMusical sequence

Triangular Window 2048

4G_violin

5A_flute

4D_viola

Spectrum Analysis(cont.)

Musical sequence

4G_violin 5A_flute 4D_viola

Spectrogram

Spectrum Analysis(cont.)

Rect. Window0.5 sec

Spectrum Analysis(cont.)

B1

A1

A2

A3

1 2 31 1 2 3B A A A = + +i i i

1 11 21 31

2 12 1

3 2

3

1N N

B A A AB AB

B A

=

known known Unknown

Least SquareEstimation

Features of Timbre in Frequency Domain (from PARK 2004)

Harmonic Analysisbehavior of harmonics & locations of harmonics

Inharmonicityerror between measured harmonics and their theoretical ideal harmonics

Harmonic Expansion/CompressionHarmonic SlopeShimmer and Jitter

Shimmer and jitter refer to short-time, subtle irregular variations in the amplitude envelope and frequency envelope of spectra respectively.

Harmonic Analysis (from PARK 2004)

Shimmer - short-time, subtle irregular variations in the amplitude envelope

Jitter - short-time, subtle irregular variations in the frequency envelope

Features of Timbre in Frequency Domain (from PARK 2004)

Spectral EnvelopeSynchronicityTristiumulusSpectral CentroidSpectral IrregularitySpectral FluxLog Spectral SpreadRoll-offPhaseSpectral Flatness Measure

Features of Timbre in Time Domain (from PARK 2004)

Amplitude Envelope: Attack, Steady-state, and DecayAttack: Rise Time and Slope Steady-state: the portion of a sound where the amplitude level is stable and a constant pitchDecay: portion of the envelope relates to the dying phase of a sound, ending with ultimate loss of energy

Attack time (rise-time)Amplitude Modulation (Tremolo)

typically ranges from 4~8 Hz and is usually found in the steady-state portion of a sound

Temporal Centroidsignal descriptors for highly transient and percussive sounds

Attack, Decay, Sustain, Release(from Wikipedia)

Features of Timbre in Time Domain (from PARK 2004)

PitchFrequency ModulationZero-Crossing Rate (ZCR)Linear Predictive Coding (LPC)Formants

Challenges

Spectrum Analysis of Sample InstrumentsDefine our Features for Timbre RecognitionMake Decision Algorithm

Correlation & Least Square Estimation

Test with a Simple Synthesized Music

Our Approach

STFT/FFTSTFT/FFT

SampleData

SampleData

Feature Extraction(Harmonic,

..)

Feature Extraction(Harmonic,

..)

1. Training Phase

Make References

Make References

2. Recognition Test

STFT/FFTSTFT/FFT

LoggingExtractedFeatures

LoggingExtractedFeatures

LeastSquare

Estimation

LeastSquare

Estimation RESULT

Result Table

Time4G

violin

2G_Boublebass

4A_viola

4D_viola

5A_flute

4E_violin

5E_flute

5D_flute

2C_cello

5F_flute

0.5 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

1 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

1.5 0.3577 0.019156 2.7685 0.0969 1.1165 0.15592 0.41089 0.5101 0.1165 0.25907

2 0.3577 0.019156 2.7685 0.0969 1.1165 0.15592 0.41089 0.5101 0.1165 0.25907

2.5 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

3 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

3.5 0.4167 0.10644 0.2513 0.1953 0.5918 1.21157 1.63947 0.462 0.02712 0.12804

4 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

4.5 0.95311 0.78844 0.02793 0.038758 0.093583 0.016083 0.27408 0.22503 0.1151 0.12256

5 0.4167 0.10644 0.2513 0.1953 0.5918 1.21157 1.63947 0.462 0.02712 0.12804

5.5 0.4167 0.10644 0.2513 0.1953 0.5918 1.21157 1.63947 0.462 0.02712 0.12804

6 0.34885 0.13649 0.227 1.46321 0.26759 0.22503 0.1534 1.60457 0.27162 0.17351

Result Sample Timbre Recognition

Visualization with Processing(www.processing.org)

Thanks

Any Questions?

Timbre Recognition by Spectrum Analysis AgendaMusic Information Retrieval (MIR)Music Information Retrieval (MIR)Timbre/Instrument RecognitionAnalysis in time domainAnalysis in freq. domainSpectrum AnalysisFeatures of Timbre in Frequency Domain (from PARK 2004)Harmonic Analysis (from PARK 2004) Shimmer - short-time, subtle irregular variations in the amplitude envelopeJitter - short-time, subtle irregular variations in the frequency envelope Features of Timbre in Frequency Domain (from PARK 2004)Features of Timbre in Time Domain (from PARK 2004)Attack, Decay, Sustain, Release (from Wikipedia)Features of Timbre in Time Domain (from PARK 2004)ChallengesOur ApproachResult Table Result Sample Timbre RecognitionThanks