Five Approaches to Collecting Tags for Music Douglas ...turnbull/Papers/...Strengths Weaknesses...
1
Five Approaches to Collecting Tags for Music Douglas Turnbull, Luke Barrington, Gert Lanckriet Computer Audition Laboratory, UC San Diego Survey Social Tags Game Tags Webtags Autotags Hybrid Experts are paid to annotate songs using a stardard form Custom-tailored vocabulary High-quality annotations Strong labeling Small, predetermined vocabulary Human-labor intensive Time consuming approach lacks scalability CAL 500 Data Set [1] Paid 55 undergraduates to annotate 500 songs by 500 artists using a vocabulary of tags. Each song was annoated my a mini- mum of 3 individuals. This data serves as the ground truth. There are 87 “long tail” songs from Mag- natunes. The vocabulary consists of 109 tags that relate to genre, instrumentation, emotion, usage, rhythm, vocals, and other musical characteristics. 1.00 1.00 0.97 Approach Summary Strengths Weaknesses Algorithm All Songs Density AUC-ROC Top 10 Prec 1.00 1.00 0.57 Long Tail Density AUC-ROC Top 10 Prec Example Large community contributes tags using a social network Collective wisdom of crowds Unlimited vocabulary Provides social context Audioscrobbler Attempted to collect a list of tags associ- ated with each CAL500 song and each CAL500 artist from Last.fm’s Audioscrob- bler website. For each song, the song and artist lists were combined. The combined list was matched to the CAL500 vocabulary. We attempted to use synonyms, alternative spellings, and wild- card matching to improve coverage. 0.23 0.62 0.37 0.03 0.54 0.19 aasdf Ad-hoc annotation behavior Produces weak labeling Sparse/missing data in long tail Players produce tags as they play a video game Collective wisdom of the crowds Fast paced for rapid data collection Entertaining incentives produce high-quality tags ListenGame [2] During a two week pilot study of Lis- tenGame, we collected 16,500 tags for 250 of the CAL500 songs from 440 players. Each of the 27,250 song-tag pairs were present 1.8 times on average. 0.37 0.65 0.32 Based on our experimental setup, the long tail results are misleading because the selection of songs in ListenGame is independent of popularity. aasdf “Gaming” the system Difficult to create viral gaming experience Based on short clips, rather than songs Analyze a corpus of music reviews, artist bios, blogs, discussion boards Large corpus of publicly-available documents No direct human involvement Provides social context Relevance Scoring [3] 1) Collect Corpus query google with song, artist and album 2) Query Corpus with Tag find most relevant song given tag Site-Specific use web documents from sites that are known to have high-quality content (Rolling Stones, AMG AllMusic, etc Weighted RS weight pages by tag relevance 0.67 0.66 0.32 aasdf Text-mining introduces noise Produces weak labeling Sparce/missing data in long tail [1] Turnbull, Barrington, Torres, Lanckriet Semantic Annotation and Retrieval Music and Sound Effects. TASLP 2008 [2] Turnbull, Liu Barrington, Lanckriet Using Games to Collect Semantic Information About Music. ISMIR 2007 [3] Knees, Pohle, Schedl, Schnitzer, Seyerlehner A Document-Centered Approach to a Natural Language Music Search Engine. ECIR 2008 [4] Barrington, Yazdani, Turnbull, Lanckriet Combining Feature Kernels for Semantic Music Retrieval. ISMIR 2008 [5] Turnbull Design and Development of a Semantic Music Discovery Engine. Ph.D. Thesis, UC San Diego 2008 0.25 0.56 0.18 Annotate audio content using signal processing and machine learning Not affected by cold-start problem No direct human involvement Produces strong labeling Supervised Multilabel Model [1] MFCC+Delta Feature Space One GMM per tag Mixture Hiearchies EM to train GMMs Produces “Semantic Multinomial” distribution over tag vocabulary for each novel song Top performing system in 2008 MIREX Audio Tag Classification Task 1.00 0.69 0.33 aasdf Computationally intensive Limited by training data Based solely on audio content, no context 1.00 0.70 0.27 Semantic Music Discovery Engine Combination of approaches Combine social context and audio content Use strengths, remove weaknesses Multi-tiered approach to cold-start problem 1.00 0.74 0.38 aasdf Increased system complexity Combining data sources can be tricky 1.00 0.71 0.28 Rank-Based Interleaving Given a tag, rank songs based on their best rank according to other ap- poaches Other hybrid approachs 1) Kernel Combination [4] 2) RankBoost [5] 3) Calibarated Score Averaging [5] Please Contact: Douglas Turnbull <[email protected]> Data, Papers, and additional Information can be found at: http://cosmal.ucsd.edu/cal/
Transcript of Five Approaches to Collecting Tags for Music Douglas ...turnbull/Papers/...Strengths Weaknesses...
Five Approaches to Collecting Tags for Music Douglas Turnbull, Luke Barrington, Gert LanckrietComputer Audition Laboratory, UC San Diego
Survey Social Tags Game Tags Webtags Autotags HybridExperts are paid to annotate songs using a stardard form
Custom-tailored vocabularyHigh-quality annotationsStrong labeling
Small, predetermined vocabularyHuman-labor intensiveTime consuming approach lacks scalability
CAL 500 Data Set [1]
Paid 55 undergraduates to annotate 500 songs by 500 artists using a vocabulary of tags. Each song was annoated my a mini-mum of 3 individuals.
This data serves as the ground truth. There are 87 “long tail” songs from Mag-natunes.
The vocabulary consists of 109 tags that relate to genre, instrumentation, emotion, usage, rhythm, vocals, and other musical characteristics.
1.001.000.97
Approach
Summary
Strengths
Weaknesses
Algorithm
All
So
ngs Density
AUC-ROC
Top 10 Prec
1.001.000.57Lo
ng T
ail Density
AUC-ROC
Top 10 Prec
Example
Large community contributes tags using a social network
Collective wisdom of crowdsUnlimited vocabularyProvides social context
AudioscrobblerAttempted to collect a list of tags associ-ated with each CAL500 song and each CAL500 artist from Last.fm’s Audioscrob-bler website. For each song, the song and artist lists were combined.
The combined list was matched to the CAL500 vocabulary. We attempted to use synonyms, alternative spellings, and wild-card matching to improve coverage.
0.230.620.37
0.030.540.19
aasdfAd-hoc annotation behaviorProduces weak labelingSparse/missing data in long tail
Players produce tags as they play a video game
Collective wisdom of the crowdsFast paced for rapid data collectionEntertaining incentives produce high-quality tags
ListenGame [2]
During a two week pilot study of Lis-tenGame, we collected 16,500 tags for 250 of the CAL500 songs from 440 players. Each of the 27,250 song-tag pairs were present 1.8 times on average.
0.370.650.32
Based on our experimental setup, the long tail results are misleading because the selection of songs in ListenGame is independent of popularity.
aasdf“Gaming” the systemDifficult to create viral gaming experienceBased on short clips, rather than songs
Analyze a corpus of music reviews, artist bios, blogs, discussion boards
Large corpus of publicly-available documentsNo direct human involvementProvides social context
Relevance Scoring [3]
1) Collect Corpus query google with song, artist and album2) Query Corpus with Tag find most relevant song given tag
Site-Specific use web documents from sites that are known to have high-quality content (Rolling Stones, AMG AllMusic, etc
Weighted RSweight pages by tag relevance
0.670.660.32
aasdfText-mining introduces noiseProduces weak labelingSparce/missing data in long tail
[1] Turnbull, Barrington, Torres, Lanckriet Semantic Annotation and Retrieval Music and Sound Effects. TASLP 2008[2] Turnbull, Liu Barrington, Lanckriet Using Games to Collect Semantic Information About Music. ISMIR 2007[3] Knees, Pohle, Schedl, Schnitzer, Seyerlehner A Document-Centered Approach to a Natural Language Music Search Engine. ECIR 2008[4] Barrington, Yazdani, Turnbull, Lanckriet Combining Feature Kernels for Semantic Music Retrieval. ISMIR 2008[5] Turnbull Design and Development of a Semantic Music Discovery Engine. Ph.D. Thesis, UC San Diego 2008
0.250.560.18
Annotate audio content using signal processing and machine learning
Not affected by cold-start problemNo direct human involvementProduces strong labeling
Supervised Multilabel Model [1]
MFCC+Delta Feature SpaceOne GMM per tagMixture Hiearchies EM to train GMMs
Produces “Semantic Multinomial” distribution over tag vocabulary for each novel song
Top performing system in 2008 MIREX Audio Tag Classification Task
1.000.690.33
aasdfComputationally intensiveLimited by training dataBased solely on audio content, no context
1.000.700.27
SemanticMusic
DiscoveryEngine
Combination of approaches
Combine social context and audio contentUse strengths, remove weaknessesMulti-tiered approach to cold-start problem
1.000.740.38
aasdfIncreased system complexityCombining data sources can be tricky
1.000.710.28
Rank-Based InterleavingGiven a tag, rank songs based on their best rank according to other ap-poaches
Other hybrid approachs1) Kernel Combination [4]
2) RankBoost [5]
3) Calibarated Score Averaging [5]
Please Contact: Douglas Turnbull <[email protected]>Data, Papers, and additional Information can be found at: http://cosmal.ucsd.edu/cal/
![Immune cell infiltration as a biomarker for the diagnosis ... · (ROC) curve and time-dependent ROC [25] curve, respec-tively, and quantified by the area under the ROC curve (AUC).](https://static.fdocuments.us/doc/165x107/5dd11baad6be591ccb64428b/immune-cell-infiltration-as-a-biomarker-for-the-diagnosis-roc-curve-and-time-dependent.jpg)
