Download - I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Transcript
Page 1: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC

ARTIST CLASSIFICATION

Hamid Eghbal-zadeh, Bernhard Lehner

Markus Schedl, Gerhard Widmer

ISMIR 2015

Page 2: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Outline • Introduction

o Timbral features

o Song-level features

o Factor analysis for song-level features

• I-vector Frontend o Related work

o Overview

o I-vector factor analysis

o Feature spaces

o Total variability

o Extraction procedure

• Experiments o Setup

o Evaluation

o Results

• Conclusion

Page 3: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Timbral features

Page 4: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Timbral features

Timbral features (e.g. MFCCs) are very important in content-based MIR tasks • Music similarity [Sereylehner2010,Pohle2007] • Genre classification [Mirex,Sereylehner2010,Pohle2007] • Artist classification [Ellis2007,Su2013,Kuksa2014] • etc

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 5: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Timbral features

Difference in level of features: • Frame-level

Song 1 features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 6: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Timbral features

Difference in level of features: • Frame-level

• Block-level

Song 1 features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 7: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Timbral features

Difference in level of features: • Frame-level

• Block-level

• Song-level processing

Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

Song 1 features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 8: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Timbral features

Difference in level of features: • Frame-level

• Block-level

• Song-level processing

Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

Song 1 features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 9: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Song-level features

Page 10: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007]

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 11: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 12: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song b) Adapt a pre-trained GMM on a song

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 13: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song b) Adapt a pre-trained GMM on a song c) Use a pre-trained GMM as a prior

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 14: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song b) Adapt a pre-trained GMM on a song c) Use a pre-trained GMM as a prior

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 15: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song b) Adapt a pre-trained GMM on a song c) Use a pre-trained GMM as a prior

3. Using GMMs+Factor Analysis [Kruspe2014, Eghbal-zadeh2015]

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 16: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Song-level Timbral features

Difference in level of features:

• Song-level

1. Using single Gaussians [Sereylehner2010,Pohle2007] 2. Using GMMs [Charbuillet2011, Kruspe2014]

a) Train a GMM on a song b) Adapt a pre-trained GMM on a song c) Use a pre-trained GMM as a prior

3. Using GMMs+Factor Analysis [Kruspe2014, Eghbal-zadeh2015]

processing Song 1

Song 2

Song 3

vector 2

vector 1 features

features

features

vector 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 17: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Why Factor Analysis?

Page 18: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Advantages of GMM+FA

• Having a probabilistic representation for a song:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 19: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

• Having a probabilistic representation for a song: regarding a specific variability

Introduction – Advantages of GMM+FA

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 20: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Advantages of GMM+FA

• Having a probabilistic representation for a song: regarding a specific variability

• Lower dimensionality: With a 1024 components GMM from 20 dim MFCCs, we will have 1024 x 20 dimensions

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 21: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Introduction – Advantages of GMM+FA

• Having a probabilistic representation for a song: regarding a specific variability

• Lower dimensionality: With a 1024 components GMM from 20 dim MFCCs, we will have 1024 x 20 dimensions • Better discrimination:

GMM GMM+FA

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

• Samples are taken form GTZAN dataset, projected using PCA

Page 22: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Related work

Page 23: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Related work

• The FA model we use with GMMs is called I-vector FA

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 24: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Related work

• The FA model we use with GMMs is called I-vector FA • Introduced in speaker verification [Dehak2010]

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 25: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Related work

• The FA model we use with GMMs is called I-vector FA • Introduced in speaker verification [Dehak2010] • Speech Emotion Recognition [Chen2011] • Speech Language Recognition [Martinez2011] • Speech Accent Recognition [Bahari2013] • Speech Audio Scene Detection [Elizalde2013] • Singing Language Recognition [Kruspe2014] • Music Artist Recognition [Eghbal-zadeh2015]

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 26: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

How it works?

Page 27: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Overview

Song 1

Song 2

Song 3

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 28: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Overview

Song 1

Song 2

Song 3

features

features

features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 29: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Overview

I-vector extractor Song 1

Song 2

Song 3

features

features

features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 30: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Overview

i-vector 3

I-vector extractor Song 1

Song 2

Song 3

i-vector 2

i-vector 1

features

features

features

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 31: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

I-vector Factor Analysis

Page 32: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

M = m + T y

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 33: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

M = m + T y

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 34: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

class- and song-independent supervector (comes from a Universal Background Model - UBM)

M = m + T y

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 35: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

class- and song-independent supervector (comes from a Universal Background Model - UBM)

low rank matrix, learned from training

M = m + T y

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 36: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

class- and song-independent supervector (comes from a Universal Background Model - UBM)

low rank matrix, learned from training

i-vector M = m + T y

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 37: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

class- and song-independent supervector (comes from a Universal Background Model - UBM)

low rank matrix, learned from training

i-vector M = m + T y

• I-vectors are assumed to be normally distributed

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 38: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Factor Analysis

Song GMM supervector: assumed to be normally distributed

with mean vector m and

covariance matrix 𝑻𝑻𝒕

class- and song-independent supervector (comes from a Universal Background Model - UBM)

low rank matrix, learned from training

i-vector M = m + T y

• I-vectors are assumed to be normally distributed • Unlike GMM supervectors, i-vectors are from a single-Gaussian space

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 39: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Different feature spaces

Page 40: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 41: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space GMM feature space

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 42: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space GMM feature space I-vector feature space

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 43: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space GMM feature space I-vector feature space [Total Variability Space (TVS)]

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 44: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space GMM feature space I-vector feature space [Total Variability Space (TVS)]

20 dim 1024 x 20 dim 400 dim

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 45: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Feature space

Frame-level feature space GMM feature space I-vector feature space [Total Variability Space (TVS)]

20 dim 1024 x 20 dim 400 dim [number of total factors]

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 46: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

What is total variability?

Page 47: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Total variability

In genre classification:

• Genre variability : the variability appears between different

genres.

• Song variability : the variability appears within songs of a genre.

• Total variability : genre + song variability

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 48: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Total variability

In genre classification:

• Genre variability : the variability appears between different

genres.

• Song variability : the variability appears within songs of a genre.

• Total variability : genre + song variability

In artist recognition:

• Artist variability : the variability appears between different artists.

• Song variability : the variability appears within songs of an artist.

• Total variability : artist + song variability

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 49: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

How to extract i-vectors?

1, 2, 3 …

Page 50: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

Frame-level features, 20 dim MFCCs

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 51: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2

Using a universal background model (UBM)

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 52: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2

Using a universal background model (UBM)

UBM is a GMM trained on frame-level features extracted from a set of songs

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 53: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2

Using a universal background model (UBM)

𝑁𝑐 = 𝛾𝑡(𝑐)𝑡∈𝑠

𝐹𝑐 = 𝛾𝑡 𝑐 𝑡∈𝑠 Y𝑡

𝛾𝑡:The posterior probability of component c for observation t, given UBM Y𝑡:Acoustic feature vectors N and F are referred to as GMM supervectors.

UBM is a GMM trained on frame-level features extracted from a set of songs

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 54: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2 step3

Assuming GMM supervector 𝑀

𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇𝑡) Where 𝑚 is artist and

song independent (from UBM)

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 55: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2 step3

Assuming GMM supervector 𝑀

𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇𝑡) Where 𝑚 is artist and

song independent (from UBM)

I-vector 𝑦 is estimated via: 𝑦 = (𝐼 + 𝑇𝑡 Σ−1𝑁 𝑇)−1. 𝑇−1Σ−1𝐹

T: is learned from training data 𝛴: is a diagonal covariance matrix learned in Factor Analysis procedure 𝐹 and 𝑁: GMM supervector

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 56: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

I-vectors – Procedure

step1

i-vector GMM supervector

Factor Analysis

Statistics calculation

Feature extraction

Audio Frame-level features (MFCCs)

step2 step3

Assuming GMM supervector 𝑀

𝑀 ~Normal(𝑚, 𝑇 ∗ 𝑇𝑡) Where 𝑚 is artist and

song independent (from UBM)

I-vector 𝑦 is estimated via: 𝑦 = (𝐼 + 𝑇𝑡 Σ−1𝑁 𝑇)−1. 𝑇−1Σ−1𝐹

T: is learned from training data 𝛴: is a diagonal covariance matrix learned in Factor Analysis procedure 𝐹 and 𝑁: GMM supervector UNSUPERVISED

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 57: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Tasks:

1.Music artist classification

2.Music similarity

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 58: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Tasks:

1.Music artist classification

2.Music similarity

Page 59: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Evaluation

In music artist classification:

• Artist20 dataset, 1,413 songs

• 20 artist, mostly rock and pop, 6 albums from each artist

• 6-fold cross validation

• 5 albums for train, 1 album for test

• UBM and T are trained on training set of each fold

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 60: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Setup

In music artist classification:

• Frame-level features:

a) 20 dim MFCCs

b) Spectrum Derivatives

• 400 total factors

• LDA for dimensionality reduction

• 3 back-ends: PLDA, KNN, DA {I-vector extraction}

{PLDA,…} {MFCC+SD}

Extract features

Extract GMM

supervectors

Front end

Compensation/ Normalization

{LDA/Length norm}

Fram

es

Backend

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 61: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

In music artist classification:

Baselines

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 62: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

In music artist classification:

Baselines

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

MFCCs

Page 63: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

In music artist classification:

Baselines

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

MFCCs

MFCCs+SD

Page 64: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Tasks INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Tasks:

1.Music artist classification

2.Music similarity

Page 65: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Evaluation

In music similarity:

• 1517Artists dataset

used for training T and UBM

• GTZAN dataset

used for music similarity estimation experiments

• KNN [k=1:20]

to evaluate similarity matrix with genre labels

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 66: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Evaluation INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Blues, Classic, Country, Disco, Hip-Hop, Jazz, Metal, Pop, Reggae, Rock GTZAN:

1,000 songs x 30 sec excerpts

10 genres

Alternative & Punk, Blues, Children, Classical, Comedy & Spoken Word, Country, Easy Listening & Vocals, Electronic & Dance, Folk, Hip-Hop, Jazz, Latin, New Age, R&B & Soul, Reggae, Religious, Rock & Pop, Soundtracks & More, World

1517Artists:

3,180 songs x full songs

19 genres

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 67: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Setup

In music similarity:

• Frame-level features:

a) 25 dim MFCCs + delta + delta2

b) Spectrum Derivatives

• 400 total factors

• Cosine distance for pairwise distance matrix

• Distance normalization, distance combination {I-vector extraction}

{Similarity matrix} {MFCC+SD}

Extract features

Extract GMM

supervectors

Front end

Cosine

{Distance}

Fram

es

Backend

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 68: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

Baselines

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

In music similarity:

Page 69: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

MFCC-IVEC

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

In music similarity:

Page 70: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

[MFCC+SD] IVEC

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

In music similarity:

Page 71: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

[MFCC+ SD] IVEC + RTBOF

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

In music similarity:

Page 72: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Results

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

[MFCC+ SD] IVEC + CMB

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

In music similarity:

Page 73: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Tasks INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Tasks:

1.Music artist classification

2.Music similarity

• Extended results

Page 74: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 75: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 76: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 77: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 78: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 79: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

BLS

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 80: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

RTBOF

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 81: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

MFCC-IVEC

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 82: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

MFCC-IVEC+RTBOF

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 83: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Experiments – Extended results

In music similarity:

• 1517Artists

• Not in the paper

• Only MFCCs

• More results:

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

Baselines

MFCC-IVEC+BLS

http://bird.cp.jku.at/ivectors/ I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 84: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Conclusion

Page 85: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Conclusion – Advantages

• Song-level timbral features

• Can capture specific variability

• Low dimensional features

• Can be used for supervised and unsupervised tasks

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 86: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Conclusion – Advantages

• Song-level timbral features

• Can capture specific variability

• Low dimensional features

• Can be used for supervised and unsupervised tasks

• Very good timbral features, please use it!

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

• Contact: [email protected] www.cp.jku.at/people/eghbal-zadeh/

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 87: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Conclusion – Drawbacks

• Can be used with features can be modeled via GMMs

• Relies on a UBM

• Unreliable for very short segments

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 88: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Thank you!

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain

Page 89: I-VECTORS FOR TIMBRE-BASED MUSIC SIMILARITY AND MUSIC ARTIST CLASSIFICATION

Acknowledgement

• We would like to acknowledge the tremendous help by Dan Ellis from Columbia

University, who shared the details of his work, which enabled us to reproduce his

experiment results.

• Thank to Pavel Kuksa from University of Pennsylvania for sharing the details of

his work with us.

• Also thank to Jan Schlüter from OFAI for his help with music similarity baselines.

• And at the end, we appreciate helpful suggestions of Rainer Kelz and Filip

Korzeniowski from Johannes Kepler University of Linz to this work.

• This work was supported by the EU-FP7 project no.601166 (PHENICX), and by the

Austrian Science Fund (FWF) under grants TRP307-N23 and Z159.

INTRODUCTION I-VECTORS EXPERIMENTS CONCLUSION

I-vectors for timbre-based music similarity and music artist classification ISMIR 2015 Malaga, Spain