Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning...
Transcript of Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning...
![Page 1: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/1.jpg)
Drum fills detectionDrum fills detectionand generationand generation
Frédéric Tamagnan & Yi-Hsuan Yang
![Page 2: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/2.jpg)
IntroductionIntroduction
![Page 3: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/3.jpg)
OriginsOriginsWe wanted to generate drum fills as an answer to
regular patterns with Deep Learning
We needed data
We had to detect and extract drum fills
![Page 4: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/4.jpg)
What is a drum fill ?What is a drum fill ?
https://www.youtube.com/embed/u5MIa4wgmU4?start=140&enablejsapi=1
![Page 5: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/5.jpg)
Why to detect andWhy to detect andgenerate drum fills ?generate drum fills ?
1. To segment a music piece2. To make long-term music generation
with dynamic and variations3. To make short-term music generation
for live performances
![Page 6: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/6.jpg)
Why to detect andWhy to detect andgenerate drum fills ?generate drum fills ?
Kink, boiler room Moscow, Live set, 2015
![Page 7: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/7.jpg)
Why to detect andWhy to detect andgenerate drum fills ?generate drum fills ?
Tr-8S, Roland
![Page 8: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/8.jpg)
Why to detect andWhy to detect andgenerate drum fills ?generate drum fills ?
Tr-8S, Roland
![Page 9: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/9.jpg)
ChallengesChallenges
Hard to define what is a drum fillwith a general ruleNo big datasets with drum fillslabels
![Page 10: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/10.jpg)
Problem definitionProblem definition
We focus on detection and generation of 4/4 barscontaining a drum fillWe don't take in account the precise boundaries of thefillsWe use 9 instruments * 16 timesteps tensor to representa drum bar
![Page 11: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/11.jpg)
Empirical observationsEmpirical observationsDrum fills :
1. A greater use of toms, snares orcymbals, than in the regular drumspattern
2. A difference of played notes betweenthe regular pattern and the drum fill
3. An appearance in general at the endof a cycle of 4 or 8 bars
![Page 12: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/12.jpg)
Datasets at our disposalDatasets at our disposal
1. Labelled dataset : Native instruments +Oddgrooves.com midi drums pack : 5,317 regular patterns bars + 1,1412 drum fills bars
2. Unlabelled dataset : Lakh pianoroll dataset : 21,425songs with their related pianorolls
Dong, H.W.,Hsiao, W.Y., Yang, L.C., Yang, Y.H.: MuseGAN: Multi-track sequential generative
adversarial networks for symbolic music generation and accompaniment. In:Thirty-
Second AAAI Conference on Artificial Intelligence (2018)
![Page 13: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/13.jpg)
Drum fills DetectionDrum fills Detection
![Page 14: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/14.jpg)
Drum fills DetectionDrum fills Detection
2 Methods2 MethodsSupervised LearningRule-based Method
![Page 15: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/15.jpg)
Supervised LearningSupervised LearningFeaturesFeatures
Variational Auto-encoderlatent space features
![Page 16: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/16.jpg)
t-SNE Visualization t-SNE Visualization
Drum fills andregular patterns in
the latent space ofa VAE trained onthe LDP dataset
Hard to separate if we consider all the barsat the same time !
Supervised LearningSupervised Learning
![Page 17: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/17.jpg)
Drum fills andregular patterns in
the latent space ofa VAE trained onthe LDP dataset
t-SNE Visualization t-SNE Visualization
Supervised LearningSupervised Learning
Better if we consider only one genre !
![Page 18: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/18.jpg)
Drum fills andregular patterns in
the latent space ofa VAE trained onthe LDP dataset
t-SNE Visualization t-SNE Visualization
Supervised LearningSupervised Learning
Better if we consider only one genre !
![Page 19: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/19.jpg)
Drum fills andregular patterns in
the latent space ofa VAE trained onthe LDP dataset
t-SNE Visualization t-SNE Visualization
Supervised LearningSupervised Learning
...but not always the case
![Page 20: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/20.jpg)
Supervised learningSupervised learningfeaturesfeatures
VAE latent space features + Handcrafted Features :Instruments usedMax, std, mean of velocity = Dimension of input vector : 59
![Page 21: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/21.jpg)
Supervised LearningSupervised LearningModelModel
Logistic RegressionStandardizationL2 Regularization
![Page 22: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/22.jpg)
Supervised LearningSupervised LearningValidationValidation
NB : Handcrafted features : Velocity features + use ofinstruments
![Page 23: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/23.jpg)
Supervised LearningSupervised LearningValidationValidation
Most correlated Hand-crafted features :
1. max velocity of high tom,2. Std of velocity of mid
tom3. max velocity of low tom
![Page 24: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/24.jpg)
Rule-based MethodRule-based MethodDifference of notes between two bars
![Page 25: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/25.jpg)
Labelling andLabelling andextractionextraction
![Page 26: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/26.jpg)
Data cleaningData cleaningRemoving duplicated rowsRemoving all the couples where the regularpattern or the drum fill have fewer than 7 notesRemoving all the couple where the drum fill has atoo high density of snare notes, above 8
#ML dataset #RB datasetRaw 13,476 97,023
After rule 1 6,324 45,723
After rule 2 5,271 39,108
After rule 3 3,283 32,130
![Page 27: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/27.jpg)
Extraction EvaluationExtraction Evaluation
Amount of notes by instrument for the MLdataset
![Page 28: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/28.jpg)
Extraction EvaluationExtraction Evaluation
![Page 29: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/29.jpg)
Drum fills GenerationDrum fills Generation
![Page 30: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/30.jpg)
GenerationGenerationRNN Many-to-many
Input : Regular pattern barOutput : Drum fill bar
![Page 31: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/31.jpg)
GenerationGenerationEvaluation
Mean of notes by instrument
![Page 32: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/32.jpg)
GenerationGenerationEvaluation
Standard-deviation by instrument
![Page 33: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/33.jpg)
GenerationGenerationEvaluation
Euclidian distance in the latent spae
Sum of euclideandistance
ML fills 93012
RB fills 93844
Original fills 102135
![Page 34: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/34.jpg)
GénérationGénérationEvaluation
User Study
51 participants50% amateur musicians14% semi-professional musicians2% professional musicians Among musicians :78% DAW users53% drummers
![Page 35: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/35.jpg)
GénérationGénérationEvaluation
User Study
We asked people to compare :
1 ML fill1 RB fill1 Original fill (ground truth)1 Rule composed fill (same layer ofcymbals and toms applied on theregular pattern)
![Page 36: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/36.jpg)
GenerationGenerationEvaluationUser Study
https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/797390628&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&
show_reposts=false&show_teaser=true
![Page 37: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/37.jpg)
GenerationGenerationEvaluation
User Study
![Page 38: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/38.jpg)
GenerationGenerationEvaluation
User Study
Hard to evaluate a fill with no musicalbackground playingSpecific and complex notionOnly five sets of examplesHard to give a rating about a reallyshort event...
Why the results are bad, even for thehuman fills ?
![Page 39: Drum fills de tection and gener ation · Drum fills De tection 2 Methods Supervised Learning Rule-based Method. Supervised Learning Features Variational Auto-encoder latent space](https://reader034.fdocuments.us/reader034/viewer/2022052614/605b72312031e75a90683a54/html5/thumbnails/39.jpg)
Future directionsFuture directionsTrain a classifier with handlabelled dataUse of binary neuronsMore sophisticated generation method