Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.
-
Upload
oswin-miles -
Category
Documents
-
view
226 -
download
2
Transcript of Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.
![Page 1: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/1.jpg)
Realtime Recognition of
Orchestral Instruments
Ichiro Fujinaga
McGill University
![Page 2: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/2.jpg)
Overview
Introduction
Lazy learning (exemplar-based learning)• k-NN classifier• Genetic algorithm• Features
Results
Conclusions
![Page 3: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/3.jpg)
Introduction
Realtime recognition of isolated monophonic orchestral instruments
Spectrum analysis by Miller Puckette’s fiddle~
Adaptive system based on a exemplar-based classifier and a genetic algorithm
![Page 4: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/4.jpg)
Overall Architecture
Data Acquisition&
Data Analysis(fiddle)
Recognition
K-NN Classifier
Output
Instrument Name
Knowledge BaseFeature Vectors
Genetic AlgorithmK-NN Classifier
BestWeight Vector
Live micInput
Sound fileInput
Off-line
![Page 5: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/5.jpg)
Exemplar-based learning• The exemplar-based learning model is based on the
idea that objects are categorized by their similarity to one or more stored examples
• There is much evidence from psychological studies to support exemplar-based categorization by humans
• This model differs both from rule-based or prototype-based (neural nets) models of concept formation in that it assumes no abstraction or generalizations of concepts
• This model can be implemented using k-nearest neighbor classifier and is further enhanced by application of a genetic algorithm
![Page 6: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/6.jpg)
Exemplar-based categorization
Objects are categorized by their similarity to one or more stored examples
No abstraction or generalizations, unlike rule-based or prototype-based models of concept formation
Can be implemented using k-nearest neighbor classifier
Slow and large storage requirements?
![Page 7: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/7.jpg)
Exemplar-based learning• The exemplar-based learning model is based on the
idea that objects are categorized by their similarity to one or more stored examples
• There is much evidence from psychological studies to support exemplar-based categorization by humans
• This model differs both from rule-based or prototype-based (neural nets) models of concept formation in that it assumes no abstraction or generalizations of concepts
• This model can be implemented using k-nearest neighbor classifier and is further enhanced by application of a genetic algorithm
![Page 8: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/8.jpg)
K-nearest-neighbor classifier
Determine the class of a given sample by its feature vector:
• Distances between feature vectors of an unclassified sample and previously classified samples are calculated
• The class represented by the majority of k-nearest neighbors is then assigned to the unclassified sample
![Page 9: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/9.jpg)
Example of k-NN classifier
Classification of atheletes by height and weight(Rikishi sumo wrestlers and NBA basketball players)
170
180
190
200
210
75 100 125 150 175 200Weight (kg)
Sumo
Chicago Bulls
![Page 10: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/10.jpg)
Example of k-NN classifier
Classification of atheletes by height and weight(Rikishi sumo wrestlers and NBA basketball players)
170
180
190
200
210
75 100 125 150 175 200Weight (kg)
Sumo
Chicago Bulls
Michael Jordan
![Page 11: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/11.jpg)
Example of k-NN classifier
Classification of atheletes by height and weight(Rikishi sumo wrestlers and NBA basketball players)
170
180
190
200
210
75 100 125 150 175 200Weight (kg)
SumoChicago BullsMichael JordanDavid Wesley
![Page 12: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/12.jpg)
Example of k-NN classifier
Classification of atheletes by height and weight(Rikishi sumo wrestlers and NBA basketball players)
180
185
190
195
200
0 50 100 150 200 250 300 350 400 450 500 550 600 650 700Weight (kg)
SumoChicago BullsMichael JordanDavid Wesley
![Page 13: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/13.jpg)
Distance measures
The distance in a N-dimensional feature space between two vectors X and Y can be defined as:
A weighted distance can be defined as:
d = xi −yii=0
N−1
∑
d = wii=0
N−1
∑ xi −yi
![Page 14: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/14.jpg)
Genetic algorithms
Optimization based on biological evolution
Maintenance of population using selection, crossover, and mutation
Chromosomes = weight vectors
Fitness function = recognition rate
Leave-one-out cross validation
![Page 15: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/15.jpg)
Features
Static features (per window)• pitch• mass or the integral of the curve (zeroth-order moment)• centroid (first-order moment)• variance (second-order central moment)• skewness (third-order central moment)• amplitudes of the harmonic partials• number of strong harmonic partials• spectral irregularity• tristimulus
Dynamic features• means and velocities of static features over time
![Page 16: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/16.jpg)
Data
Original source: McGill Master Samples
Over 1300 notes from 39 different timbres (23 orchestral instruments)
Spectrum analysis by fiddle (2048 points)
First 46–232ms of attack (1–9 windows)
Each analysis window (46 ms) consists of a list of amplitudes and frequencies of the peaks in the spectra
![Page 17: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/17.jpg)
Results
Recognition rate
81
50
64
70
10098
88
96
40
50
60
70
80
90
100
Exp I Exp II Exp III
3 instr7 instr39 sintr
Experiment I• SHARC data• static features
Experiment II• fiddle• dynamic features
Experiment III• more features• redefinition of
attack point
![Page 18: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/18.jpg)
Conclusions
Realtime timbre recognition system
Analysis by Puckette’s fiddle
Recognition using dynamic features
Adaptive recognizer by k-NN classifier enhanced with genetic algorithm
A successful implementation of exemplar-based classifier in a time-critical environment
![Page 19: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/19.jpg)
Future research
Performer identification
Speaker identification
Tone-quality analysis
Multi-instrument recognition
Expert recognition of timbre
![Page 20: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/20.jpg)
Recognition rate for different lengths of analysis window
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9
3 instr7 instr39 instr
![Page 21: Realtime Recognition of Orchestral Instruments Ichiro Fujinaga McGill University.](https://reader035.fdocuments.us/reader035/viewer/2022062305/5697c02b1a28abf838cd89f6/html5/thumbnails/21.jpg)
Comparison with Human Performance