Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns...
Transcript of Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns...
![Page 1: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/1.jpg)
1
New School Production
Professional Studio Personal Studio
Old School
![Page 2: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/2.jpg)
2
Distribution Old School New School
Large Scale Small Scale
Physical Virtual
Cumbersome Portable
![Page 3: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/3.jpg)
3
Promotion Old School New School
Critics
Radio DJs
Social Networks
Personalized Internet Radio
![Page 4: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/4.jpg)
4
Age of Music Proliferation
5M Artists
150M Songs
27K Record Labels
50M Customers
31% Americans
Producers Consumers
Semantic Music
Discovery Engine
140M iPods
![Page 5: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/5.jpg)
5
Talk Outline
• Age of Music Proliferation - Sec. 1.1 • Music Search & Discovery - Sec. 1.2 • Semantic Music Discovery Engine - Sec. 1.3 • Collecting Music Information - Ch. 3, 4 • Autotagging System - Ch. 2 • CAL Music Discovery Engine - Sec. 1.4 • Concluding Remarks - Ch. 5
![Page 6: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/6.jpg)
6
Music Search
Search - retrieving specific audio content
Common Paradigms: 1. Query-by-Metadata 2. Query-by-Performance
3. Query-by-Fingerprint
![Page 7: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/7.jpg)
7
Music Discovery
Discovery - finding new music or relationships
Common Paradigms: 1. Recommendation-by-Popularity 2. Browse-by-Genre 3. Query-by-Similarity
• Acoustic • Social
• Semantic
4. Query-by-Description
![Page 8: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/8.jpg)
8
Semantic Music Discovery Engine
Index music with tags so that it can be retrieved using a semantic description
Tag - a short text-based token • mellow, classic rock, acoustic slide guitar • real-valued weight ∝ strength of association
Semantic - use meaningful words to describe music • “mellow classic rock that sounds like the Beatles and
features an acoustic slide guitar” • akin to Internet Search Engines
![Page 9: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/9.jpg)
9
Semantic Music Discovery Engine
Data Sources
Human Annotation
Artists & Record Labels
Audio Tracks Metadata Web-documents
Music Information
Index
Discovery Engine
Search Engine
Internet Radio
Social Network Dis
cove
ry
Extr
actio
n C
olle
ctio
n
Text-mining System
Analytic Systems Audio Characteristics
Automatic Annotation
Music Processing
System
Tags
Surveys Annotation Games
Internet Music Sites
Autotags
Autotagging System
![Page 10: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/10.jpg)
10
Semantic Music Discovery Engine
Artists & Record Labels
Data Sources
Audio Tracks Metadata
Human Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
![Page 11: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/11.jpg)
11
Popu
larit
y
Songs
Music Last.fm - 150M songs by 16M artists
CAL500 - 500 songs by 500 artist
Long Tail Economics - Chris Anderson (2004)
Cold Start Problem - Songs in the long tail are not annotated and thus can not be discovered.
Long Tail - Obscure
Short Tail - Popular
![Page 12: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/12.jpg)
12
Metadata
Factual information about music • song, album, artist, record label • year, biographical, charts
• heterogeneous data • strings, numbers, images, graphs
![Page 13: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/13.jpg)
13
Metadata
http://www.allmusic.com/cg/amg.dll?p=amg&sql=11:difrxqr5ldje
![Page 14: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/14.jpg)
14
Semantic Music Discovery Engine
Artists & Record Labels
Data Sources
Audio Tracks Metadata
Human Annotation
Music Processing
System Analytic Systems
Audio Characteristics
Automatic Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
![Page 15: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/15.jpg)
15
Music Processing Systems
Information extracted from audio signal • Acoustic - noise, roughness
• Rhythmic - tempo, patterns • Harmonic - key, major/minor • Structural - chorus locations
![Page 16: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/16.jpg)
16
Semantic Music Discovery Engine
Artists & Record Labels Surveys Annotation
Games Internet
Music Sites
Data Sources
Audio Tracks Metadata Tags
Human Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
Music Processing
System
Audio Characteristics
![Page 17: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/17.jpg)
17
Surveys Pandora Music Genome Project • 400 “Objective” Genes • 50 trained music experts • 750,000 songs annotated
![Page 18: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/18.jpg)
18
CAL500 Survey • 174-tag vocab - genre, emotion, … • Paid 55 undergrads to annotate music for 120 hours • 500 songs annotated by 3+ people
Surveys
![Page 19: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/19.jpg)
19
Human Annotations
Conducting a survey Reliable, Precise, Tailored to Application
X Expensive, Laborious, Not Scalable
![Page 20: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/20.jpg)
20
Annotation Games
‘Human-Computation’ • Web-based, multi-player game with real-time interaction
• Player contribute useful annotations through game play
• ESPGame for images [Von Ahn] • Listen Game for songs
![Page 21: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/21.jpg)
21
Listen Game
![Page 22: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/22.jpg)
22
Human Annotation
Survey Reliable, Precise, Tailored to Application
X Expensive, Laborious, Not Scalable
Annotation Game Cheap, Scalable, Precise, Personalized X Need to create a viral user experience
![Page 23: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/23.jpg)
23
Music Web Sites
1. Social Tagging Site • Users annotate music with tags
• Last.fm - 960K distinct tags
http://www.last.fm/music/Red+Hot+Chili+Peppers/_/Give+It+Away
![Page 24: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/24.jpg)
24
Music Web Sites
2. Collecting Web Documents • Song & Album Reviews
• Artist Biographies • Music Blogs, Discussion Boards • Allmusic, Rolling Stone, Amazon, Mog
![Page 25: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/25.jpg)
25
Web Documents Genres:
Funk (3) Funk-metal Funk-rock Pop Rap
Vocals: Nasal Staccato Enunciation Distinctive vocals
Instruments: Guitar Bass Jew’s-harp
Adjective:
Hard-rocking (2) Noisy Scratchy Sliding Positive vibes
![Page 26: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/26.jpg)
26
Collecting an Annotated Music Corpus
Survey Reliable, Precise, Tailored to Application
X Expensive, Laborious, Not Scalable
Annotation Game Cheap, Scalable, Precise, Personalized
X Need to create a viral user experience
Music Web Sites Cheap, Annotations for short-tail
X Noisy, long-tail is poorly represented
![Page 27: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/27.jpg)
27
Semantic Music Discovery Engine
Artists & Record Labels Surveys Annotation
Games Internet
Music Sites
Data Sources
Audio Tracks Metadata Tags
Human Annotation
Autotagging System
Analytic Systems Autotags
Automatic Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
Music Processing
System
Audio Characteristics
![Page 28: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/28.jpg)
28
Autotagging System
Our goal is to build a system that can 1. Annotate a song with meaningful tags 2. Retrieve songs given a text-based query
Plan: Learn a probabilistic model that captures a relationship between audio content and tags.
Retrieval
‘Jazz’ ‘Male Vocals’
‘Sad’ ‘Slow Tempo’
Annotation Frank Sinatra ‘Fly Me to the Moon’
![Page 29: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/29.jpg)
29
System Overview
Parameter Estimation
T T
Annotation
Training Data
Data
Audio Feature Extraction
Vocabulary
Annotation Vectors
Representation
Parametric Model
Modeling
Evaluation
Evaluation
Inference
Music Review
Novel Song
(annotation)
Text Query (retrieval)
![Page 30: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/30.jpg)
30
Semantic Representation
Choose vocabulary of ‘musically relevant’ tags • Instruments, Genre, Emotion, Vocal, Usages
Annotations are converted to a real-valued vector • Semantic association between a tag and a song
Example: Frank Sinatra’s “Fly Me to the Moon” Vocab = {funk, jazz, guitar, sad, female vocals} y = [0/4 , 3/4, 4/4, 2/4, 0/4]
![Page 31: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/31.jpg)
31
Acoustic Representation
Each song is represented as a bag-of-feature-vectors • Pass a short time window over the audio signal • Extract a feature vector for each short-time audio segment
• Ignore temporal relationships of time series
X = , . . . , xt x3 , x1 , x2
![Page 32: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/32.jpg)
32
Audio Features
We calculate MFCC+Deltas feature vectors • Mel-frequency Cepstral Coefficients (MFCC)
• Low dimensional representation short-term spectrum
• Popular for both representing speech, music, and sound effects
• Instantaneous derivatives (deltas) encode short-time temporal info
• 5,200 39-dimensional vectors per minute
Numerous other audio representations • Spectral features, modulation spectra, chromagrams, …
![Page 33: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/33.jpg)
33
Statistical Model
Supervised Multi-class Labeling model • One Gaussian Mixture Model (GMM) per tag - p(x|t) • Key Idea: GMM trained with songs associated with tag
Notes: • Developed for image annotation [Carneiro & Vasconcelos 05]
• Scalable and Parallelizable • Modified for real-value weights rather than binary labels
• Extended formulation to handle multi-tag queries
![Page 34: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/34.jpg)
34
Modeling a Song
+ + + + + + + + +
+ + + + + +
+ +
+ + +
+ + +
+ + +
+
+
+
+
+ + +
+ + + +
+
+
+
+
+
+ +
+ + + + + +
+ +
+ +
+
+ +
+
+ + + + +
+
+
+
+ + + + +
+ + +
+
+
+
+
+ + + + +
+ + +
+
+
+
+
+
+ + +
+ + +
+
+
+
+
+ + +
+
+ +
+ EM
Bag of MFCC vectors
+
+ +
+ +
+
+ +
+
+ +
+ + +
+ +
+ + +
+ + +
+
+
+
+
+ + + +
+ + + +
+
+
+
+ +
+ + +
+ +
+
+
+
+ + +
+ + +
+
+
+
+
+
+ + + + +
+
+
+
+ + + + +
+ +
+
+
+
+
+ + +
+ + + +
+
+
+
+ +
+ + +
+ + +
+
+
+
+
+ + + + + +
+
+
+
Algorithm 1. Segment audio signals
2. Extract short-time feature vectors
3. Estimate GMM with EM algorithm
![Page 35: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/35.jpg)
35
Modeling a Tag Algorithm: 1. Identify songs associated with tag t 2. Estimate a ‘song GMM’ for each song - p(x|s) 3. Use the Mixture Hierarchies EM algorithm [Vasconcelos01]
• Learn a ‘mixture of mixture components’
Benefits + Computationally efficient for parameter estimation and inference + ‘Smoothed’ song representation → better density estimate
romantic
Tag Model Mixture Hierarchies
EM
p(x|t)
Standard EM
romantic
![Page 36: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/36.jpg)
36
Assuming
1. Uniform tag prior 2. Vectors are conditionally independent given a tag
3. Geometric average of likelihoods 4. Tags are mutually exclusive and exhaustive
Given a novel song X = {x1, …, xT}, calculate
Annotation
Semantic Multinomial: • P(t|X)’s →multinomial distribution over the tag vocabulary
Annotation: peaks of multinomial
![Page 37: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/37.jpg)
37
Annotation
Semantic Multinomial for “Give it Away” by the Red Hot Chili Peppers
P(t|X)
![Page 38: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/38.jpg)
38
Annotation: Automatic Music Reviews
Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy, hip-hop song that is arousing and
exciting. It features drum machine, backing vocals, male vocal, a nice acoustic guitar solo, and rapping, strong vocals. It is a song that is very danceable and with a heavy beat that you might like listen to while at a party.
Frank Sinatra - Fly me to the moon This is a jazzy, singer / songwriter song that is calming and
sad. It features acoustic guitar, piano, saxophone, a nice male vocal solo, and emotional, high-pitched vocals. It is a song with a light beat and a slow tempo that you might like listen to while hanging with friends.
![Page 39: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/39.jpg)
39
Retrieval 1. Annotate each song in corpus with a semantic multinomial p
• p = {P(t1|X), …, P(t|V||X)}
2. Given a text-based query, construct a query multinomial q • qi = 1/|t| , if tag t appears in the query string
• qi = 0, otherwise
3. Rank all songs by the Kullback-Leibler (KL) divergence
![Page 40: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/40.jpg)
40
Retrieval
Query: ‘a tender pop song with female vocals’
‘tender’ ‘pop’ ‘female vocals’
1. Shakira - The One
2. Alicia Keyes - Fallin’
3. Evanescence - My Immortal
Query Multinomial 0.33
0.024
0.024
0.024
![Page 41: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/41.jpg)
41
Retrieval
‘Tender’ Crosby, Stills and Nash - Guinevere Jewel - Enter from the East Art Tatum - Willow Weep for Me
‘Female Vocals’ Alicia Keys - Fallin’ Shakira - The One Junior Murvin - Police and Thieves
‘Tender’ AND
‘Female Vocals’
Jewel - Enter from the East Evanescence - My Immortal Cowboy Junkies - Postcard Blues
Query Retrieved Songs
![Page 42: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/42.jpg)
42
Semantic Music Discovery Engine
Artists & Record Labels Surveys Annotation
Games Internet
Music Sites
Data Sources
Audio Tracks Metadata Tags Web-documents
Human Annotation
Autotagging System
Text-mining System
Analytic Systems Autotags
Automatic Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
Music Processing
System
Audio Characteristics
![Page 43: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/43.jpg)
43
Text-mining System
Relevance Scoring [Knees 08] • site-specific queries
• Amazon, AMG, Billboards, etc.
• weight-based approach
Step 1: Collect Corpus For each song, use a search engine to retrieve web pages:
• site:<website> “<artist>“ music
• site:<website> “<artist>” “<album>” music review • site:<website> “<artist>” “<song>” music review
Maintain Is,d = mapping of songs to documents
![Page 44: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/44.jpg)
44
Text-mining System
Step 2: Autotag songs For each tag t: 1. Query corpus with tag t to find relevant documents
• wt,d → relevance score for document d 2. For each song s, sum relevance scores for documents that are
related to song s
ws,t = Σd Is,d wt,d
![Page 45: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/45.jpg)
45
Semantic Music Discovery Engine
Artists & Record Labels Surveys Annotation
Games Internet
Music Sites
Data Sources
Audio Tracks Metadata Tags Web-documents
Human Annotation
Autotagging System
Text-mining System
Analytic Systems Autotags
Automatic Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
Music Processing
System
Audio Characteristics Music
Information Index
![Page 46: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/46.jpg)
46
Comparing Tags
Groundtruth • CAL500 - binary labeling of song-tag pairs
• Long Tail - subset of 87 obscure songs
Approaches 1. Social Tags - Last.fm 2. Annotation Game - Listen Game
3. Web Autotags - Site-specific relevance scoring 4. Audio Autotags - SML model w/ MFCCs
![Page 47: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/47.jpg)
47
Comparing Tags
For each approach: For each tag:
1. Rank songs 2. Calculate Area under the ROC curve (AROC)
• 0.5 random ranking (Bad)
• 1.0 perfect ranking (Good)
Calculate mean AROC
![Page 48: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/48.jpg)
48
Comparing Tags
Social Tags CAL500 0.62 Long Tail 0.54
Approach Songs AROC
Game CAL500 0.65
Long Tail *
Web Autotags
CAL500 0.66
Long Tail 0.56
Audio Autotags
CAL500 0.69
Long Tail 0.70
![Page 49: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/49.jpg)
49
Combining Tags
Approaches 1. Autotagging - single best approach
2. “Best Rank” Interleaving 3. Isotonic Regression - [Zadrozny 02]
4. RankBoost - [Freund03]
![Page 50: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/50.jpg)
50
Combining Tags
Approach AROC
Audio Autotags
0.69
“Best Rank” Interleaving
0.74
RankBoost 0.75
Isotonic Regression
0.75
![Page 51: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/51.jpg)
51
Semantic Music Discovery Engine
Music Information
Index
Discovery Engine
Search Engine
Internet Radio
Social Network
Artists & Record Labels Surveys Annotation
Games Internet
Music Sites
Data Sources
Audio Tracks Metadata Tags Web-documents
Human Annotation
Music Processing
System
Autotagging System
Text-mining System
Analytic Systems Audio Characteristics Autotags
Automatic Annotation
Dis
cove
ry
Extr
actio
n C
olle
ctio
n
![Page 52: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/52.jpg)
52
CAL Music Discovery Engine
![Page 53: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/53.jpg)
53
CAL Music Discovery Engine
![Page 54: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/54.jpg)
54
What’s on tap…
Research Challenges 1. Explore music similarity with semantics
2. Explore discriminative approaches [Eck 07]
3. Combine heterogeneous data sources • Game Data, Social Networks, Web Documents, Popularity Info
4. Focus on person rather than population • Demographic and Psychographic Groups
• Individuals
• Emotional states of an Individual
![Page 55: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/55.jpg)
55
References Semantic Annotation and Retrieval [IEEE TASLP 08, SIGIR 07, ISMIR08?]
Music Annotation Games [ISMIR 07a]
Related: Query-by-Semantic-Similarity [ICASSP 07, MIREX 07]
Tag Vocabulary Selection with Sparce CCA [ISMIR 07b] Supervised Music Boundary Detection [ISMIR 07c]
Work-in-Progress: 1. Combining Tags from Multiple Sources
• Rank Aggregation, Kernel Combination [ISMIR 08?] 2. Music Similarity with Semantics 3. (More Social) Music Annotation Games
![Page 56: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/56.jpg)
56
Thanks
Gert, Charles, Lawrence, Shlomo, Serge, Sanjoy
• Advice and perspective
Gary Cottrell, Virginia de Sa, IGERT • Enabling creative and interdisciplinary pursuits
Damien O’malley, Aron Tremble, VLC • Thinking beyond the walls of academia
Luke Barrington, Antoni Chan, David Torres • Friends and collaborators
![Page 57: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/57.jpg)
57
“Talking about music is like dancing about architecture …�it’s a really stupid thing to want to do”
- Elvis Costello and others
Douglas Turnbull Computer Audition Laboratory
UC San Diego [email protected] cs.ucsd.edu/~dturnbul
![Page 58: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/58.jpg)
58
Design and Development of a�
Semantic Music Discovery Engine
Douglas Turnbull Ph.D. Thesis Defense
University of California, San Diego
Committee: Gert Lanckriet, Charles Elkan, Lawrence Saul, Shlomo Dubnov,
Serge Belongie, Sanjoy Dasgupta
May 7, 2008
![Page 59: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/59.jpg)
59
The Age of Music Proliferation
Production: • 5M artist pages - • 150M distinct songs -
Distribution • 1.5M simultaneous P2P users (Feb ‘01) - • 27K record labels - • 4B songs to 50M customers -
Consumption • 11M Internet radio users - • 110M iPods sold -
![Page 60: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/60.jpg)
60
AROC = 5/6
Quantifying Retrieval
Rank order test set songs • KL between a query multinomial and semantic multinomials • 1-, 2-, 3-word queries with 5 or more examples
Metric: Area under the ROC Curve (AROC)
Mean AROC is the average AROC over a large number of queries.
Rank Label TP FP
1
2
3
4
5 0
1
1 False Positive Rate
True Positive Rate
R
-
R
-
-
1/2 0
1/2 1/3
1 1/3 1 2/3
1 1
Rank by ‘Romantic’
![Page 61: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/61.jpg)
61
Comparing Tags
Ground Truth CAL500
All 0.15 1.00 Long-Tail 0.15 1.00
Social Tags Last.fm
All 0.23 0.62 Long-Tail 0.03 0.54
Game Listen Game
All 0.37 0.65 Long-Tail * *
Web Autotags
All 0.67 0.66
Long-Tail 0.25 0.56
Approach Songs Density AROC
Audio Autotags
All 1.00 0.69
Long-Tail 1.00 0.70
![Page 62: Production - Swarthmore Collegeturnbull/Papers/Turnbull_Music... · • Rhythmic - tempo, patterns ... Dr. Dre (feat. Snoop Dogg) - Nuthin' but a 'G' thang This is a dance poppy,](https://reader031.fdocuments.us/reader031/viewer/2022022605/5b785acd7f8b9ad2498e9aaf/html5/thumbnails/62.jpg)
62
Music & Technology
Technology is changing how music is produced, distributed, promoted and consumed.