Making machines that make music
-
Upload
srihari-sriraman -
Category
Technology
-
view
57 -
download
2
Transcript of Making machines that make music
![Page 1: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/1.jpg)
Making Machines that Make Music
Srihari Sriraman nilenso
![Page 2: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/2.jpg)
should we listen to some now?
![Page 3: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/3.jpg)
Why I do thisI Sing, I do computers
Bleeding edge research Dreamy ambitions
Interesting By-products
![Page 4: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/4.jpg)
What this talk is about
Melody Modelling Synthesis
Generation
Melody Modelling
SynthesisGeneration
![Page 5: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/5.jpg)
Melody ˈmɛlədi
noun a sequence of single notes that is musically satisfying; a tune.
![Page 6: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/6.jpg)
Carnatic Music
![Page 7: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/7.jpg)
Carnatic Music
Kalyani, Extempore MS Gopalakrishnan, Violin
![Page 8: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/8.jpg)
Khamas, Thillana Abhishek Raghuram
![Page 9: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/9.jpg)
Carnatic Music
South Indian classical music Ragas, Gamakams
Vocal Tradition Rich in compositions
Extempore / Manodharma
![Page 10: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/10.jpg)
Tanpura
Veena Mridangam
![Page 11: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/11.jpg)
The foundations of musical abstractions
in Carnatic music
![Page 12: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/12.jpg)
ShruthiTonic note
Choice of artist Swarams are relative to this
![Page 13: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/13.jpg)
LayaRhythm concepts
Similar to time signatures A rather mature system
![Page 14: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/14.jpg)
Sa Ri Ga Ma Pa Da SaNi
Swarams
![Page 15: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/15.jpg)
Sa Ri Ga Ma Pa Da SaNi
S R G M P D SN
R1 R2 R3
G1 G2 G3 M1 M2
D1 D2 D3
N1 N2 N3
Notation
Pronunciation
Variations
![Page 16: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/16.jpg)
SwaramsThe 12 semitones
Elements of a raga Simples are sung
Prescriptive notation
![Page 17: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/17.jpg)
Rāgā
Kaapi, Extempore TM Krishna
![Page 18: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/18.jpg)
RagaHas a name
Rule to ascend Rule to descend
Not necessarily symmetric Not necessarily linear Grouped into families
![Page 19: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/19.jpg)
RagaHas a name
Rule to ascend Rule to descend
Not necessarily symmetric Not necessarily linear Grouped into families
![Page 20: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/20.jpg)
Demo
of the fundamental abstractions.
![Page 21: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/21.jpg)
![Page 22: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/22.jpg)
Tools & Libraries
![Page 23: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/23.jpg)
Fuzzy searchFor indic languages
Needs to be fast Primary stitching mechanism Helps with multi-source data
![Page 24: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/24.jpg)
A quick recapPlay the scale of a raga
Fuzzy find a raga Play a phrase
Play a phrase in the context of a raga Play some prescriptive notation
![Page 25: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/25.jpg)
But..
..that doesn’t sound like Carnatic music, does it?
![Page 26: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/26.jpg)
Synthesis
![Page 27: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/27.jpg)
Enter Melographs
Me
Machine
![Page 28: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/28.jpg)
another phrase
Me
Machine
![Page 29: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/29.jpg)
Prescriptive vs Descriptive
![Page 30: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/30.jpg)
Gamakams
![Page 31: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/31.jpg)
Sphuritam
Orikai
Jaaru
Kampitam
Sphuritam Nokku Ravai
Kandippu Ullasitam Etra-jaru
Iraka-jaru Odukkal
Orikai Vali
Kampitam
![Page 32: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/32.jpg)
Gamakams in SSP
![Page 33: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/33.jpg)
Gamakams in SSP
![Page 34: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/34.jpg)
Gaayaka
| S, N D | N S R G |
((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,))
Subramanian, 2009 Database of phrases
Automatic Gamakam feature – guided
![Page 35: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/35.jpg)
Modelling Gamakams
![Page 36: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/36.jpg)
![Page 37: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/37.jpg)
Me
Machine
Back to this…
![Page 38: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/38.jpg)
PASR
Srikumar 2013 Pitch, Attack, Sustain, Release
Vector specifies the PASR vars for each prescriptive note
![Page 39: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/39.jpg)
Me
Machine
Rendering PASR…
![Page 40: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/40.jpg)
Rendering PASR…
![Page 41: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/41.jpg)
Generation
![Page 42: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/42.jpg)
Random | Within a raga
![Page 43: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/43.jpg)
Random | Within a raga
![Page 44: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/44.jpg)
Get data
![Page 45: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/45.jpg)
Kosha An Open Carnatic Music Database
http://github.com/ssrihari/kosha
![Page 46: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/46.jpg)
![Page 47: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/47.jpg)
![Page 48: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/48.jpg)
![Page 49: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/49.jpg)
Study data
![Page 50: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/50.jpg)
![Page 51: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/51.jpg)
Melographs
![Page 52: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/52.jpg)
Melographs
kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram
kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram
![Page 53: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/53.jpg)
Pitch Histograms
![Page 54: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/54.jpg)
Pitch Histograms
kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
![Page 55: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/55.jpg)
Pitch HistogramsKalyani - Vocal Kalyani - Violin
Mohana - MandolinMohana - Vocal
Revati - Vocal Revati - Instrumental
![Page 56: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/56.jpg)
Extract Music Information
![Page 57: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/57.jpg)
Midi Histogram
Normalised Midi Histogram
![Page 58: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/58.jpg)
Tonic note identification
Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.
Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone.
Srihari, S. (2016) * Pick the most frequent note, it mostly just works.
* not really, no
![Page 59: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/59.jpg)
Tonic note identification
![Page 60: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/60.jpg)
Tonic note identification
![Page 61: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/61.jpg)
Swaram HistogramKalyani
S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
![Page 62: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/62.jpg)
Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S
Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S
![Page 63: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/63.jpg)
Generation with weighted probabilities
![Page 64: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/64.jpg)
![Page 65: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/65.jpg)
In comparison with random
Random
Single Swaram
Weighted
![Page 66: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/66.jpg)
Melody insights #1Tonic note is prominent
Sa, and Pa have higher and sharper peaks Other note peaks are blunt
Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same
![Page 67: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/67.jpg)
Two swaram probabilities
![Page 68: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/68.jpg)
Two swaram probabilities
Prominence of adjacency Encoded rules of Arohanam and Avarohanam
![Page 69: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/69.jpg)
Two swaram probabilities
![Page 70: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/70.jpg)
Single swaram vs Two swarams
Two Swaram
Weighted
Single Swaram
Weighted
![Page 71: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/71.jpg)
Melody insights #2Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded
We begin to see gamakams Sometimes, the in-between is worse than either extreme
![Page 72: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/72.jpg)
Three swaram probabilities and more
A simple markov chain
![Page 73: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/73.jpg)
First Order Matrix
https://en.wikipedia.org/wiki/Markov_chain#Music
Markov Chains in Music
https://github.com/rm-hull/markov-chains
Second Order Matrix
![Page 74: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/74.jpg)
Markov Chains in Music
![Page 75: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/75.jpg)
![Page 76: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/76.jpg)
Melody insights #3Generic markov chains don’t really work
LSTMs also don’t work, probably
![Page 77: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/77.jpg)
By-products
![Page 78: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/78.jpg)
Automatic Transcription
![Page 79: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/79.jpg)
Automatic Transcription
(:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3
:..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)
![Page 80: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/80.jpg)
Raga Identification
![Page 81: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/81.jpg)
Goodness of fit test
![Page 82: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/82.jpg)
:base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58)
:base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80)
:base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)
![Page 83: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/83.jpg)
Raga Identification
Revati sample vs
Revati base
Mohana sample vs
Revati base
![Page 84: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/84.jpg)
What nextModel insights as melodic abstractions
Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice
Deep learning (Recurrent variational auto encoders)
![Page 85: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/85.jpg)
Is this music though?
![Page 86: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/86.jpg)
Behag
Dasarapada Abhishek Raghuram
![Page 87: Making machines that make music](https://reader038.fdocuments.us/reader038/viewer/2022110218/58f296001a28ab3b238b4595/html5/thumbnails/87.jpg)
Making Machines that Make Music
Srihari Sriraman nilenso