Boundaries and novelty: the correspondence between points...
Transcript of Boundaries and novelty: the correspondence between points...
![Page 1: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/1.jpg)
Boundaries and novelty:the correspondence between
points of change andperceived boundaries
Jordan B. L. Smith, Ching-Hua Chuan and Elaine ChewDMRN+7 18 December 2012
![Page 2: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/2.jpg)
Outline
I. What the research is about and why it is very interesting
II. How the data were assembled and analyzed
III. What the results of the analysis are
![Page 3: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/3.jpg)
Music is continuous,but we hear it in chunks
![Page 4: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/4.jpg)
Music is continuous,but we hear it in chunks
fig: Cross 1998
![Page 5: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/5.jpg)
I’m going to talk about large-scale structure
![Page 6: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/6.jpg)
I’m going to talk about large-scale structure
What causes a listener to believe there is a boundary here?
![Page 7: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/7.jpg)
What causes a listener to hear a boundary?
change in harmonic progressionchange in melodychange in tempochange in rhythmchange in timbrechange in loudness / dynamicsbreaksglobal structurerepetitions
Clarke and Krumhansl 1990
Bruderer 2008
![Page 8: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/8.jpg)
Aviezer, Trope and Todorov 2012
![Page 9: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/9.jpg)
Aviezer, Trope and Todorov 2012
![Page 10: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/10.jpg)
We can use large-scale MIR studies to learn about perception of structure
novelty-based algorithm ground truth boundariesX
![Page 11: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/11.jpg)
We can use large-scale MIR studies to learn about perception of structure
novelty-based algorithm ground truth boundariesX
naive baseline algorithmY
X – Y = the extent to which a novelty-based algorithm explains the ground truth better
than a naive algorithm
![Page 12: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/12.jpg)
We can use large-scale MIR studies to learn about perception of structure
novelty-based algorithm ground truth boundariesX
Yrandom set of non-boundaries
X – Y = the extent to which novelty explains the boundaries better than it
explains the non-boundaries
![Page 13: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/13.jpg)
II. How the data were assembled and analyzed
![Page 14: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/14.jpg)
SALAMI database: Structural Analysis of Large Amounts of Music Information
![Page 15: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/15.jpg)
SALAMI by genre
LMA382
World217 Popular
322
Jazz237
Classical225
![Page 16: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/16.jpg)
AfricanAmericasArabicAsianBalkanCalypsoCelticChansonCubanEuropeanFlamenco
FusionGypsyIndianKlezmerLatin AmericanMixed TraditionalTangoU.S. Traditional
Alternative Pop / RockAlternative Metal / PunkAlternative FolkClassic RockCountryDance PopElectronica
Hip Hop & RapHumourInstrumental PopMetalReggaeRoots RockSinger/Songwriter Folk
Renaissance / MedievalBaroqueClassicalRomantic20th Century
Acid JazzAvant-GardeBebopCool JazzContemporary Blues
Country BluesDixielandHard BopLatin JazzPost-BopSoul JazzSwingUrban Blues
?LMA382
World217 Popular
322
Jazz237
Classical225
![Page 17: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/17.jpg)
Genre Number of recordings annotated once
Number of recordings annotated twice
Popular 51 101
Jazz 10 112
Classical 44 65
World 30 78
Live Music Archive (LMA) 113 142
Total: 146 498
1142Total number of annotations:
Nutrition Facts
![Page 18: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/18.jpg)
Example SALAMI annotations
![Page 19: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/19.jpg)
Example SALAMI annotations
![Page 20: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/20.jpg)
Carte de audio features
timbre:Mel-frequency cepstral coefficients (MFCCs)
pitch:chromagram
key:center of effect (CE)
rhythm:rhythmogram / fluctuation patterns (FPs)
tempo:periodicity histogram (PH)
![Page 21: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/21.jpg)
From features to novelty functions
“Across theUniverse” byThe Beatles
![Page 22: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/22.jpg)
From features to novelty functions
“Across theUniverse” byThe Beatles
![Page 23: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/23.jpg)
Euclidean distance
“Across theUniverse” byThe Beatles
![Page 24: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/24.jpg)
![Page 25: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/25.jpg)
![Page 26: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/26.jpg)
![Page 27: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/27.jpg)
black = point of greatest change
![Page 28: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/28.jpg)
black = point of greatest changegreen = perceived as a boundary
red = random point
![Page 29: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/29.jpg)
black = point of greatest changegreen = perceived as a boundary
red = random point
2 / 10 guesses were true boundaries: precision = 0.22 / 6 true boundaries were found: recall = 0.33
f-measure = 0.25
![Page 30: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/30.jpg)
black = point of greatest changegreen = perceived as a boundary
red = random point
2 / 10 guesses were true boundaries: precision = 0.22 / 6 true boundaries were found: recall = 0.33
f-measure = 0.25
0 / 10 guesses matched redf-measure = 0
f-measure contrast = 0.25
![Page 31: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/31.jpg)
5 different features 7 different timescales
C.E.
P.H.
FP
Chr.
MFCC
302520151050
.
.
.
.
.
.
.
.
.
.
.
.
![Page 32: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/32.jpg)
5 different features 7 different timescales
C.E.
P.H.
FP
Chr.
MFCC
302520151050
.
.
.
.
.
.
.
.
.
.
.
.
![Page 33: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/33.jpg)
5 different features 7 different timescales
C.E.
P.H.
FP
Chr.
MFCC
302520151050
.
.
.
.
.
.
.
.
.
.
.
.
CENTRAL QUESTION:
Do the points of greatest change predict the boundaries?
<Do the black marks more closely match the green lines than the red lines?>
![Page 34: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/34.jpg)
III. What the results of the analysis were.
![Page 35: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/35.jpg)
0.0
0.2
0.4
0.6
0.8
Fm
easu
re
Boundaries3.0 seconds
Non boundaries3.0 seconds
f-measure for boundaries and non-boundaries
![Page 36: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/36.jpg)
Number of difference functions with a matching peak
Den
sity
0 5 10 15 20 25 30
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
How many changes does each boundary match?
![Page 37: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/37.jpg)
Number of novelty functions with a matching peak
Frac
tion
of a
ll bo
unda
ries
0 5 10 15 20 25 30 35
0.2
0.1
00.
1
BoundariesNon boundaries
How many changes does each non-boundary match?
![Page 38: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/38.jpg)
f-measure contrast for different ____________
1 2 3 4 5 6 7 8 9
0.2
0.1
0.0
0.1
0.2
0.3
0.4
Annotator
Diff
eren
ce in
fm
easu
reannotators
![Page 39: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/39.jpg)
f-measure contrast for different ____________
Popular Jazz Classical World LMA
0.2
0.1
0.0
0.1
0.2
0.3
0.4
Diff
eren
ce in
fm
easu
regenres
![Page 40: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/40.jpg)
f-measure contrast for different ____________
0 5 10 15 20 25 30
0.5
0.0
0.5
1.0
Feature window size (seconds)
Diff
eren
ce in
fm
easu
retimescales
![Page 41: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/41.jpg)
f-measure contrast for different ____________
Timbre Harmony Rhythm Tempo Key
0.4
0.2
0.0
0.2
0.4
0.6
Diff
eren
ce in
fm
easu
refeatures
![Page 42: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/42.jpg)
ConclusionsLarge changes in acoustic features are an indicator of boundaries.
Changes indicate boundaries about twice as strongly as non-boundaries—but only twice.
The more types of change occurring, the greater the odds of being a boundary.
Being a moment of change seems to be a necessary but not sufficient condition for being a boundary.
![Page 43: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/43.jpg)
Wrap-up
We explicitly studied the ground truth by comparing it to a randomized version of itself.
Similar studies examining the role of repetitions and breaks in boundary placement are planned.
![Page 44: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/44.jpg)
Thanks!
This research was supported by the Social Sciences and Humanities Research Council, and by Queen Mary University of London.
![Page 45: Boundaries and novelty: the correspondence between points ...jblsmith.github.io/documents/smith2012-dmrn-boundaries_and_novelty-slides.pdf · Boundaries and novelty: the correspondence](https://reader033.fdocuments.us/reader033/viewer/2022042102/5e7ee9debc140f3b9414d70d/html5/thumbnails/45.jpg)
ReferencesH. Aviezer, Y. Trope, and A. Todorov. “Body cues, not facial expressions, discrimintate between intensive positive and negative emotions.” Science, 30, 2012, pp. 1225–1229.
M. Bruderer. Perception and modeling of segment boundaries in popular music. Ph.D. dissertation, Technische Universiteit Eindhoven. 2008.
E. F. Clarke, and C. L. Krumhansl, “Perceiving musical time,” Music Perception, 7 (3), 1990, pp. 213–251.
I. Cross, “Music analysis and music perception,” Music Analysis, 17 (10), 1998. [image credit]
J. B. L. Smith, J. A. Burgoyne, I. Fujinaga, D. De Roure, and S. J. Downie, “Design and creation of a large-scale database of structural annotations,” in Proc. ISMIR, Miami, FL, 2011, pp. 555–560.
More references for this research not explicitly involved in this presentation can be found in J. B. L. Smith, C.-H. Chuan, E. Chew. “Audio properties of perceived boundaries in music,” submitted to IEEE Trans. Multimedia, which you can get a copy of if you email me or something.