Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82...
Transcript of Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82...
![Page 1: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/1.jpg)
Cold-StartMusicRecommendationUsingMultimodalDeepArchitectures
SYSTEMATICAPPROACHESTODEEPLEARNINGMETHODSFORAUDIOESIWORKSHOPVIENNA,AUSTRIA
SEP15,2017
![Page 2: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/2.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 3: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/3.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 4: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/4.jpg)
![Page 5: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/5.jpg)
Cold-Start ProblemTHELONGTAIL
MostPopularTracks
0.01%
![Page 6: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/6.jpg)
Cold-Start ProblemTHELONGTAIL
MostPopularTracks
1%
![Page 7: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/7.jpg)
Cold-Start ProblemTHELONGTAIL
MostPopularTracks
100%
35%oftracks0spinslastweek
![Page 8: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/8.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 9: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/9.jpg)
PROBLEMOVERVIEWCollaborative Filtering
Users
? ?
? ? ?
?
? ?
? ?[Items (Tracks) [
Explicit
Thumbs(upanddown)
StationCreation
Implicit
TrackCompletion
TrackSkips
![Page 10: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/10.jpg)
LATENTFACTORSCollaborative Filtering
AggressiveCalm
SimpleHarmony
ComplexHarmony
![Page 11: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/11.jpg)
MATRIXFACTORIZATIONCollaborative Filtering
Koren,Y.,Bell,R.,&Volinsky,C.(2009).MatrixFactorizationTechniquesforRecommenderSystems.Computer,42(8),42–49.
? ?
? ? ?
?
? ?
? ?[ [
Users
Item
s (Tr
acks
)
⇡[ [
k
Users
Item
s (Tr
acks
)
k[ [
![Page 12: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/12.jpg)
PROBLEMFORMULATIONCollaborative Filtering
GivenItemiandUseru: ? ?
? ? ?
?
? ?
? ?[ [
UsersIte
ms
(Trac
ks)
⇡[ [
k
Users
Item
s (Tr
acks
)
k
[ [ItemLatentFactor:qi 2 Rk
UserLatentFactor: pu 2 Rk
Rating: riuriu
RatingApproximation: r̂iu = qTi pu
qipu
argminq⇤,p⇤X
u,i2S(rui � qTi pu)
2 +�(||qi||2 + ||pu||2)Koren,Y.,Bell,R.,&Volinsky,C.(2009).MatrixFactorizationTechniquesforRecommenderSystems.Computer,42(8),42–49.
![Page 13: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/13.jpg)
Collaborative Filtering
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 TheBeatles ADayInTheLife
Ranked2 TheBeatles ADayInTheLife(LoveVersion)
Ranked3 TheBeatles AcrossTheUniverse
EXAMPLE
![Page 14: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/14.jpg)
Collaborative Filtering
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked35 GeorgeHarrisonWhileMyGuitarGentlyWeeps
(Live)
Ranked82 GeorgeHarrison MySweetLord(Live)
Ranked91PaulMcCartney&Eric
Clapton Something(Live)
EXAMPLE
Ranked158 LedZeppelin Tangerine
![Page 15: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/15.jpg)
THEGOODANDTHEBADCollaborative Filtering
Richpreference-drivensimilarityspace Latentspaceisgenerallynotinterpretable
Powerfulatmatchingtherightsongwiththerightlistener
Canonlyrecommenditemsthathavealreadybeenrated
![Page 16: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/16.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 17: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/17.jpg)
Estimate Collaborative Factors
? ?
? ? ?
?
? ?
? ?[ [
Users
Item
s (Tr
acks
)
⇡[ [
k
Users
Item
s (Tr
acks
)
k[ [
![Page 18: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/18.jpg)
http://blogs-images.forbes.com/kevinmurnane/files/2016/03/google-deepmind-artificial-intelligence-2-970x0-970x646.jpg
k
Approximate Item Factors using Audio
Oord,A.VanDen,Dieleman,S.,&Schrauwen,B.(2013).DeepContent-basedMusicRecommendation.AdvancesinNeuralInformationProcessingSystems,2643–2651.
![Page 19: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/19.jpg)
WITHDEEPLEARNINGApproximating Factors using Audio
4096
4096
DenseLayers1024
1DConvolutionalLayers
2048 2048 4096 4096
108
N
4096x4
TimeGlobalPooling
mean
max
L2var
1DConvolutionalLayer
k
![Page 20: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/20.jpg)
TRAININGDATAApproximating Item Factors using Audio
• (Small)Dataset:
• 83ktracks
• 3patchesof35secondspertrack(251kpatches=)
• (Patchesonlyfortraining!)
• Splits:
• Train:80%
• Validation:10%
• Test:10%
M
{X,Y}
![Page 21: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/21.jpg)
TRAININGApproximating Item Factors
• Lossfunction:
• CosineDistance
• Optimization:
• Adam(defaultparams)
• 50%DropoutonDenseLayers
• EarlyStopping
• Mini-batchesof64examples
L(✓) = 1� 1
M
X
X2X,y2Y
f(X; ✓)Ty
||f(X; ✓)||2||y||2
Cosin
eDista
nce
MiniBatches
![Page 22: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/22.jpg)
RESULTSApproximating Item Factors using Audio
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches) 0.25 22 ~2h
![Page 23: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/23.jpg)
RESULTSApproximating Item Factors using Audio
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches) 0.25 22 ~2h
Audio(FullTracks) 0.21 - -
![Page 24: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/24.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 25: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/25.jpg)
The Music Genome Project™
>1.5Milliontracksmanuallyanalyzed
~400attributespertrack
AttributeExamples
BreathyVoiceNasalVoiceOddMeterHasBanjoJoyfulLyrics
…
![Page 26: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/26.jpg)
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 IVThieves TheSoundAndTheFury
Ranked2 Journey TooLate
Ranked3 AlbertLee LookOutCleveland
Recommending Music using the MGP™EXAMPLE
![Page 27: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/27.jpg)
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 IVThieves TheSoundAndTheFury
Ranked2 Journey TooLate
Ranked3 AlbertLee LookOutCleveland
Recommending Music using the MGP™EXAMPLE
![Page 28: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/28.jpg)
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 IVThieves TheSoundAndTheFury
Ranked2 Journey TooLate
Ranked3 AlbertLee LookOutCleveland
Recommending Music using the MGP™EXAMPLE
![Page 29: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/29.jpg)
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 IVThieves TheSoundAndTheFury
Ranked2 Journey TooLate
Ranked3 AlbertLee LookOutCleveland
Recommending Music using the MGP™EXAMPLE
![Page 30: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/30.jpg)
http://blogs-images.forbes.com/kevinmurnane/files/2016/03/google-deepmind-artificial-intelligence-2-970x0-970x646.jpg
k
Approximate Factors using the MGP™
![Page 31: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/31.jpg)
DEEPARCHITECTUREApproximate Factors using the MGP
N
4096
4096
DenseLayers
k
![Page 32: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/32.jpg)
TRAININGDATAApproximating Item Factors using the MGP™
• (Small)Dataset:
• 83ktracks()
• Splits:
• Train:80%
• Validation:10%
• Test:10%
M
{X,Y}
![Page 33: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/33.jpg)
TRAININGApproximating Item Factors using the MGP™
• Lossfunction:
• CosineDistance
• Optimization:
• Adam(defaultparams)
• 50%DropoutonDenseLayers
• EarlyStopping
• Mini-batchesof256examplesCosin
eDista
nce
MiniBatches
L(✓) = 1� 1
M
X
x2X,y2Y
f(x; ✓)Ty
||f(x; ✓)||2||y||2
![Page 34: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/34.jpg)
RESULTSApproximating Item Factors
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches) 0.25 22 ~2h
Audio(FullTracks) 0.21 - -
MGP 0.15 37 7s
![Page 35: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/35.jpg)
Beyond the MGP™
APPROXIMATEMGPWITHMACHINELISTENING
MACHINELISTENINGGENES
![Page 36: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/36.jpg)
(Coming soon: MGP™ Estimation with Waveforms!)
APPROXIMATEMGPWITHMACHINELISTENING
![Page 37: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/37.jpg)
DEEPARCHITECTUREApproximate Factors using MLG
N
4096
4096
DenseLayers
k
![Page 38: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/38.jpg)
RESULTSApproximating Item Factors
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches) 0.25 22 ~2h
Audio(FullTracks)
0.21 - -
MGP 0.15 37 7s
MLG 0.22 37 7s
![Page 39: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/39.jpg)
• Motivation:TheCold-StartProblem
• Background:CollaborativeFiltering
• Cold-StartMusicRecommendation:
• EstimateCollaborativeFactorsfromAudio
• TheMusicGenomeProject™
• MultimodalDeepArchitectures
Outline
![Page 40: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/40.jpg)
http://blogs-images.forbes.com/kevinmurnane/files/2016/03/google-deepmind-artificial-intelligence-2-970x0-970x646.jpg
k
Combine Methods to Approximate Factors
![Page 41: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/41.jpg)
k
Combine Methods to Approximate Factors
N 4 4 k4 4
Dens(vanden
11D
2 2 4 41
N 40m
mL
v
1D
k
http://blogs-images.forbes.com/kevinmurnane/files/2016/03/google-deepmind-
![Page 42: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/42.jpg)
LATE-FUSIONDEEPARCHITECTURECombine Methods to Approximate Factors
4096x2
4096
4096
DenseLayers
k
![Page 43: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/43.jpg)
RESULTSApproximating Item Factors
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches)
0.25 22 ~2h
Audio(FullTracks)
0.21 - -
MGP 0.15 37 7s
MLG 0.22 37 7s
Audio+MLG 0.19 37 7s
![Page 44: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/44.jpg)
http://blogs-images.forbes.com/kevinmurnane/files/2016/03/google-deepmind-artificial-intelligence-2-970x0-970x646.jpg
k
Further Multimodality to Approximate Factors
![Page 45: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/45.jpg)
LATE-FUSIONDEEPARCHITECTUREFurther Multimodality to Approximate Factors
4096x2+31
4096
4096
DenseLayers
k
one-hotvectorencoding
![Page 46: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/46.jpg)
RESULTSApproximating Item Factors
Input CosDistance #Epochs Time/Epoch
Audio(35sPatches)
0.25 22 ~2h
Audio(FullTracks)
0.21 - -
MGP 0.15 37 7s
MLG 0.22 37 7s
Audio+MLG 0.19 37 7s
Audio+MLG+genres 0.16 37 7s
![Page 47: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/47.jpg)
ISALRIGHTMore data
• LARGEDataset:
• ~900kmostpopulartracks
• 3patchesof35secondspertrack(~2.7Mpatches=)M
{X,Y}
Input Trainedon TestSet CosDistance
Audio SMALL SMALL 0.21
![Page 48: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/48.jpg)
ISALRIGHTMore data
Input Trainedon TestSet CosDistance
Audio SMALL SMALL 0.21
Audio LARGE LARGE 0.37
• LARGEDataset:
• ~900kmostpopulartracks
• 3patchesof35secondspertrack(~2.7Mpatches=)M
{X,Y}
![Page 49: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/49.jpg)
ISALRIGHTMore data
Input Trainedon TestSet CosDistance
Audio SMALL SMALL 0.21
Audio LARGE LARGE 0.37
Audio SMALL LARGE 0.64
• LARGEDataset:
• ~900kmostpopulartracks
• 3patchesof35secondspertrack(~2.7Mpatches=)M
{X,Y}
![Page 50: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/50.jpg)
ISALRIGHTMore data
Input Trainedon TestSet CosDistance
Audio SMALL SMALL 0.21
Audio LARGE LARGE 0.37
Audio SMALL LARGE 0.64
Audio LARGE SMALL 0.21
• LARGEDataset:
• ~900kmostpopulartracks
• 3patchesof35secondspertrack(~2.7Mpatches=)M
{X,Y}
![Page 51: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/51.jpg)
Recommendation Examples
Ar[st Title
QueryTrack TheBeatles WhileMyGuitarGentlyWeeps
Ranked1 BobDylan Knockin’OnHeavensDoor
Ranked2 NeilYoung HeartOfGold
Ranked3 TheRollingStones Angie
![Page 52: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/52.jpg)
Recommendation Examples
Ar[st Title
QueryTrack Sargon Con[nuarà
Ranked1 Mudvayne Happy?
Ranked2 Mudvayne ForgetToRemember
Ranked3 StoneSour Hell&Consequences
![Page 53: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/53.jpg)
Long Tail Context
MostPopularTracks
100%
35%oftracks0spinslastweek
20spinslastweek
![Page 54: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/54.jpg)
Recommendation Examples
Ar[st Title
QueryTrack Sargon Con[nuarà
Ranked1 Mudvayne Happy?
Ranked2 Mudvayne ForgetToRemember
Ranked3 StoneSour Hell&Consequences
![Page 55: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/55.jpg)
Recommendation Examples
Ar[st Title
QueryTrack Sargon Conqnuarà
Ranked1 Mudvayne Happy?
Ranked2 Mudvayne ForgetToRemember
Ranked3 StoneSour Hell&Consequences
![Page 56: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/56.jpg)
Recommendation Examples
Ar[st Title
QueryTrack Sargon Conqnuarà
Ranked1 Mudvayne Happy?
Ranked2 Mudvayne ForgetToRemember
Ranked3 StoneSour Hell&Consequences
![Page 57: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/57.jpg)
Recommendation Examples
Ar[st Title
QueryTrack Sargon Conqnuarà
Ranked1 Mudvayne Happy?
Ranked2 Mudvayne ForgetToRemember
Ranked3 StoneSour Hell&Consequences
![Page 58: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/58.jpg)
Recommendation Examples
Ar[st Title
QueryTrack LaBossad’Urina ElTiempo
Ranked1 IlDivo Hallelujah
Ranked2SarahBrightman&TheLondonSymphony
OrchestraTimeToSayGoodbye
Ranked3 AndreaBocelli Amapola
![Page 59: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/59.jpg)
Long Tail Context
MostPopularTracks
100%
35%oftracks0spinslastweek
0spinslastweek
![Page 60: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/60.jpg)
Recommendation Examples
Ar[st Title
QueryTrack LaBossad’Urina ElTiempo
Ranked1 IlDivo Hallelujah
Ranked2SarahBrightman&TheLondonSymphony
OrchestraTimeToSayGoodbye
Ranked3 AndreaBocelli Amapola
![Page 61: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/61.jpg)
Recommendation Examples
Ar[st Title
QueryTrack LaBossad’Urina ElTiempo
Ranked1 IlDivo Hallelujah
Ranked2SarahBrightman&TheLondonSymphony
OrchestraTimeToSayGoodbye
Ranked3 AndreaBocelli Amapola
![Page 62: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/62.jpg)
Recommendation Examples
Ar[st Title
QueryTrack LaBossad’Urina ElTiempo
Ranked1 IlDivo Hallelujah
Ranked2SarahBrightman&TheLondonSymphony
OrchestraTimeToSayGoodbye
Ranked3 AndreaBocelli Amapola
![Page 63: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/63.jpg)
Recommendation Examples
Ar[st Title
QueryTrack LaBossad’Urina ElTiempo
Ranked1 IlDivo Hallelujah
Ranked2SarahBrightman&TheLondonSymphony
OrchestraTimeToSayGoodbye
Ranked3 AndreaBocelli Amapola
![Page 64: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/64.jpg)
ENSEMBLE OF RECOMMENDERS MAY PRODUCE OPTIMAL RECOMMENDATIONS
![Page 65: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/65.jpg)
MAN vs MACHINE?
![Page 66: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/66.jpg)
MAN + MACHINE
![Page 67: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/67.jpg)
MAN + MACHINE“Mix,of,Art,and,Science”
![Page 68: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/68.jpg)
MAN + MACHINE“Mix,of,Art,and,Science”
Oramas, S., Nieto, O., Sordo, M., Serra, X., A Deep Multimodal Approach for Cold-start Music Recommendation. Deep Learning for Recommender Systems Workshop, RecSys, Como, Italy 2017
Oramas, S., Nieto, O., Barbieri, F., Serra, X., Multi-label Music Genre Classification From Audio, Text, and Images Using Deep Features. Proc. of the 18th International Society for Music Information Retrieval Conference (ISMIR). Suzhou, China, 2017
![Page 69: Cold-Start Music Recommendation Using …urinieto/MARL/publications/...2017/09/15 · Ranked 82 George Harrison My Sweet Lord (Live) Ranked 91 Paul McCartney & Eric Clapton Something](https://reader036.fdocuments.us/reader036/viewer/2022071105/5fdf7ce8e1a9c16f261d8823/html5/thumbnails/69.jpg)
MAN + MACHINE“Mix,of,Art,and,Science”
THANKS! [email protected]
Oramas, S., Nieto, O., Sordo, M., Serra, X., A Deep Multimodal Approach for Cold-start Music Recommendation. Deep Learning for Recommender Systems Workshop, RecSys, Como, Italy 2017
Oramas, S., Nieto, O., Barbieri, F., Serra, X., Multi-label Music Genre Classification From Audio, Text, and Images Using Deep Features. Proc. of the 18th International Society for Music Information Retrieval Conference (ISMIR). Suzhou, China, 2017