Post on 27-Jan-2015
description
0.5setgray0
0.5setgray1
Music and Machine LearningUsing Machine Learning for the Classification of
Indian Music: Experiments and Prospects
Paritosh K. Pandya
School of Technology and Computer Science
Tata Institute of Fundamental Research
email: pandya@tifr.res.in
http://www.tcs.tifr.res.in/∼pandya
SNDT 2006 – p.
Outline
Motivation (Example)
Introduction to Machine Learning
Intelligent Music Processing Examples
Indian Music: Some questions
Automatic Raag Recognition System
SNDT 2006 – p.
Computers and Information Processing
Evolution of computers
Scientific calculations: e.g. planetary orbits
Data processing: e.g. inventory
Multimedia Rich Text, Graphics, Pictures, Animations,Video, Sound and Music.Computers can store, edit, process and display all ofthese!!
Internet and World-wide Web:
SNDT 2006 – p.
The Classical Music Archives
SNDT 2006 – p.
Computers and Arts
Computers and networks are increasingly used forstoring, processing (editing, cataloguing, searching),and disseminating artistic content.
Web Portals with artistic, educational andresearch-oriented content are becoming available e.g.complete works of Shakespeare
Computers can be used to analyse artistic content innew and sophisticated manners.
Computer as a tool for research in humanities and arts:Example Discovery channel news (2003): Computers toreveal Shakespeare
SNDT 2006 – p.
Learning Machines
Traditionally, computers are calculating devices. How tocalculate must be fully pre-programmed.
People observe patterns in nature, they discover rulesand they learn.
Can Computers learn? A question addressed byartificial intelligence.
Learning System A system capable of the autonomousacquisition and integration of knowledge.
SNDT 2006 – p.
How do systems learn?
Supervised Learning: Learn from examples.
Training Example Set (annotated)
Feature Selection (Input)
Target function representation
Statistical learning and classification
X
y
y = Number ofoccurrences of Ma after<Ga,MaTiv,Pa>
x = Number of oc-currences of Re after<Ga,MaTive,Pa>
SNDT 2006 – p.
Neural Nets
Learning functions
Given monthly icecream sales and average temperature forlast 10 years, predict icecream sales this summer.
SNDT 2006 – p.
Why use learning system?
The relationship between data elements is notformalized. Only examples are available.
Relationship between data items is buried within largeamount of data.
Data mining: using historical data to discover relationshipsand using this to improve future performance.
SNDT 2006 – p.
Applications of Machine Learning
Speech recognition
Image recognition (Face recognition)
Identifying Genes
Predicting Drug Activity
Cataloguing Faint Objects in Astronomical Data
Detecting Credit Card Frauds
Predicting Medical Outcomes from Historic Data
Detecting Hacking and Intrusion from Network Load
Computational Linguistics
SNDT 2006 – p. 10
Music Performance Visualisation
Performance worm [Widmer, Vienna]
Different players have different ways of building tensionor expression in the music
Measure subtle changes in beat level tempo versusloudness for each note played.
Represent this visually in tempo-loudness space as atrajectory called "performance worm".
SNDT 2006 – p. 11
Our Experiments
Real-time melody tracker (Click)
Time (s)0 15
Pitc
h (H
z)
120
300
sa
sa
re
ga
ma
pa
dha
ni
SNDT 2006 – p. 12
Recognition of Concert Pianists
Characterisation of Personal Expression FeaturesClassification
Classification between 22 piano players
Classification based on performance worm like data
Achieved accuracy comparable with human listeners.
[Saunders et al (2005)]
SNDT 2006 – p. 13
Islands of Music
Intelligent structuring and exploration of digital musiccollections [Pampalk et al (2004)]
Grouping of Music bySimilarity
Genre and Style
Performer
Timbral and rythmic con-tent
Automatic classification of music by Genre: Classical,Country, Disco, HipHop, Jazz and Rock [Pye 2000]About 90% success on 176 songs
SNDT 2006 – p. 14
Music Structure Analysis
Structure in Music Composition
repetition, transposition, call and response, rythmicpatterns and harmoic sequences
shape of a song e.g. AABA
Automatic structure analysis attempts to discover suchstructure [Danenberg,CMU]
Beat tracking and Tempo Detection
Identifying time signatures and tempo
Marking beat positions within music [Simon Dixon]
SNDT 2006 – p. 15
Music representation: Audio
Audio Waveform Santur playing(Click)
Rich set of features: Pitch, Amplitude, Spectra
SNDT 2006 – p. 16
Structured Music Representation
Example: Musical Score Notation
SNDT 2006 – p. 17
Computers and Music Notation
MIDI files: computer representation of musical score.
Can be recorded from keyboards etc.
Synthesizers: MIDI → Sound
Issue Expressive Music Representation(a hot topic for research!)
Music Notation for Indian Music
Bhatkhande or Paluskar systems
Not used by professional musicians
Lacks structures
SNDT 2006 – p. 18
Swarupa: Structured Music
define kaida2 as pat(
8.pat(dha,te,te,dha | te,te,dha,dha |
te,te,dha,ge | ti,na,ke,na),
8.pat(ta,te,te,ta | te,te,ta,ta |
te,te,dha,ge | dhin,na,ge,na) );
define palta1 as pat(
8.pat(dha,te,te,dha,te,te,dha,dha |
dha,te,te,dha,te,te,dha,dha),
8.pat(dha,te,te,dha,te,te,dha,dha |
te,te,dha,ge,dhin,na,ge,na),
8.pat(ta,te,te,ta,te,te,ta,ta |
ta,te,te,ta,te,te,ta,ta),
8.pat(dha,te,te,dha,te,te,dha,dha |
te,te,dha,ge,dhin,na,ge,na) );
SNDT 2006 – p. 19
Swarupa: Structured Music
define kaida2n as[A::[ 2:dhatita::[dha,te,te], dhadha::[dha,dha]
tite::[te, te], dha,ge | tin,na,ke,na ];B::[ A.1{khali} | C::[A.2 | A.3{bhari}] ]];
define palta1 as[ 3:[A.1 |] C; 3:[B.1 |] C; ];
define palta2 as[ A.1 | D::4:5%[te,te] ; B ;
B.1 | D ; B ];
Synthesis Swarupa → Audio (See MuM Webpage)Music transcription: Audio → Score
SNDT 2006 – p. 20
Indian Music Research using AI
Some topics
Classification of MusicRaag RecognitionClassification of Music in Thaats and JatisClassification of Raags into Time Cycle, SeasonalCycles,Classification of music by gharanasPerformer recognitionIdentification of Raag LakshansAssociation of Bhaav with musical performance
[B.Chaitanya Deva, 1981]
Beat tracking and taal recognition
Identification of musical structureSNDT 2006 – p. 21
Associated Applications
Music Visualisation
Musical query processing from large annotated musicaldatabases.
Automatic music composition
Automatic accompaniment
Pursuit: Distance Education of Indian Music
SNDT 2006 – p. 22
Machine Recognition of Raags
Raga performance as sequence of notes.Stop Sa Re Ga Pa Ga Re Sa Stop Dha Sa Re Ga
Sequential pattern classification problem
Data is not unordered set of samples.
Data elements occur in an order: spatial or temporal.
The probability of next data element crucially dependson the order of occurrence of preceeding elements.
Hidden Markov Models (HMM) are widely used.
SNDT 2006 – p. 23
Finite state automaton for Raag
Bhoopali
2Ga
3Re
4Pa
5Sa
6Dha
7Stop System can be in one of
finite number of states
Current state depends onthe past and current inputseen
Current state and next in-put determines the possiblenext states
Experiments: Manual con-struction of raag automatabased on Bhatkhande Books[Sahasrabuddhe]
SNDT 2006 – p. 24
Hidden Markov Model of a Raag
Bhoopali
2Ga 1.00
0.09
3Re 1.00
0.45
4Pa 1.00
0.235
Sa 0.99
0.06
0.51
0.41
0.45
0.076
Dha 1.00
0.41
0.38
0.07
0.32
0.62
0.30
7Stop 1.00
0.46
0.05
0.18
0.25
0.06
Probability ofseeing a note inthe given state.
Probability ofmoving from onestate to another
An HMM model canbe learnt from train-ing data
Analysis Given anHMM and a note se-quence, compute itsprobability of occur-rence.
SNDT 2006 – p. 25
Raag Recognition using HMM
Hidden Markov Model for a raag
Finite state automata
Probability of “seeing a note” in each state.
Probability of transition between states.
HMM model can be learnt from a set of training data
Given a note seqeuence we can compute its probabilitywithin given Raag HMM model.
SNDT 2006 – p. 26
Kansen: A raga recognition system
An Experiment at TIFR using a Toolkit HTK:
Learns HMM for each raag from training data(Baum-Welch Algorithm)
Training data: (Bhatkhande, IITK) collection of midi filesof raags played on keyboard. We use 29 raag database.
Test data: sequence of notes
Output: probability of the sequence being in each raag.
Preliminary Results
About 86 percent success on 29 raag recognition
Confusion between close raags
Insufficiency of dat a significant reason
(Joint work with Bhaumik Choksi and K. Samudravijaya)SNDT 2006 – p. 27
Bhatkhande
MIDI File Database of Indian Raags (IIT, Kanpur)
Adana, AheerBhairav, AlhiyaBilawal, Bageshri,Bahar, Basant, BasantMukhari, Behag, Bhoopali,BhoopaliTodi, ChandraKauns, ChhayaNut, Des,Durga, Gaud, Hamir, JataShwari, JaunaPuri Jogiya,Lalit, Malkauns, MiyankiMalhar, Multani, Pahadi,Peelu Sohini, TilakKamode, Tilang, Todi
Each midi file created by playing from Keyboard (e.g.Des.mid)
Basic database of 29 Raags (above)
Full database of 300+ Raags
SNDT 2006 – p. 28
Demonstration
Input Stop Sa Sa DhaKo Sa GaKo Re Stop Sa Ga Ga MaGaKo Re GaKo Re Sa Ni DhaKo Ni Sa Re Sa Ni SaOutput Log of probability of being in a raagI=1 t=0.02 W=ChandraKauns v=1
I=2 t=0.02 W=ChhayaNut v=1
I=3 t=0.02 W=Hamir v=1
I=4 t=0.02 W=Pahadi v=1
I=5 t=0.02 W=MiyankiMalhar v=1
I=6 t=0.02 W=Adana v=1
I=8 t=0.02 W=Peelu v=1
J=0 S=0 E=1 a=-164.11 l=0.000
J=1 S=0 E=2 a=-163.60 l=0.000
J=2 S=0 E=3 a=-160.92 l=0.000
J=3 S=0 E=4 a=-160.72 l=0.000
J=4 S=0 E=5 a=-158.36 l=0.000
J=5 S=0 E=6 a=-117.88 l=0.000
J=13 S=0 E=8 a=-88.55 l=0.000
Summary Peelu (-88) Adana (-117) MiyankiMalhar (-158)SNDT 2006 – p. 29
Demonstration (cont)
Stop NiKo Dha Ni Sa NiKo Pa Ma Pa GaKo Ma Re SaBahar (-20.5) MiyankiMalhar (-23) Adana (-53)
Stop Ma Pa NiKo Dha Ni Ni Sa Stop Ni Sa Re GaKoGaKo Ma Re SaMiyankiMalhar (-52) Bahar (-66) Adana (-75)
Stop Ma Pa Ni Ni Sa Ni Sa Sa Stop Pa Ni Sa Re NiKoDha Pa Stop Pa Dha Ma Ga Re Stop Ga Re Ni SaDes (-127) Gaud (-139) MiyankiMalhar (-144)
Stop Ga Ma DhaKo DhaKo Pa Stop Ma Pa GaKo MaReKo Sa Stop Ga Ma Pa DhaKo Ni Sa DhaKo PaBasant Mukhari (-105) Peelu (-106) Jogiya (-130)
SNDT 2006 – p. 30
Demonstration (cont)
Stop Ni Sa GaKo ReKo Sa Stop Ni Sa GaKo MaTiv PaStop GaKo MaTiv Pa Ni Sa DhaKo Pa MaTiv GaKoReKo SaMultani (-76) Todi (-105) ChandraKauns (-169)
Stop DhaKo Ni Sa ReKo GaKo Stop ReKo GaKo ReKoSa Stop Sa ReKo GaKo MaTiv ReKo GaKo ReKo SaTodi (-58) Bhoopali Todi (-83) Multani (-124)
SNDT 2006 – p. 31
Conclusions
Computer analysis and machine learning providesinteresting new method of analysing music. It allowsmany intuitive and qualitative observations to be madeobjective, precise and quantitative.
Research with computational techniques lead to directapplications in music technology.
Intelligent music analysis is almost untried for IndianMusic.
Work requires collaboration between musicologists,computer scientists and electrical engineers.
Music researchers must help by building corpuses andannotated datasets for future machine analysis.
SNDT 2006 – p. 32