MUSIC BOUNDARY DETECTIONUSING FULLY CONVOLUTIONAL NEURAL NETWORKS

Jeongmin LiuSchool of Electrical Engineeringjeongmin96@kaist.ac.kr

Kyeongseok JeongSchool of Electrical Engineeringchungks603@kaist.ac.kr

Hyunjun LeeSchool of Electrical Engineeringhjlee9605@kaist.ac.kr

ABSTRACT

Music boundary detection is a basic level task of the mu-sic structure analysis in the field of music informationretrieval. Deep neural network-based approach is widelyused for the task in recent. We propose a method usingfully convolutional neural networks (FCNs) that don’t re-quire fixed-size input and output. The FCN takes the mel-scaled log magnitude spectrogram of a full song in orderto consider long temporal information, and estimates theprobability that boundaries occur for every time frames.From the estimated boundary score, peak-picking algo-rithm predicts which time frames include boundaries. TheFCN we proposed is trained and validated by a subset ofSALAMI v2.0 dataset. We evaluate the proposed methodwith our own test set consists of 23 songs, which are Ko-rean pop songs mostly. Test result is mainly given with F1scores and analyzed according to genres. Not superior thanthe baseline, but satisfactory performance is achieved.

1. INTRODUCTION

Music structure analysis is one of the major tasks in themusic information retrieval. A song can be segmented intofunctional sections, such as verse and chorus, or smallersections than that. To accomplish those tasks, musicalboundary detection should be preceded. With an advent ofdeep neural networks (DNNs), algorithms have achievedhigh performance.

In [1], convolutional neural networks (CNNs) contain-ing fully-connected layers are employed. Its input featuresare mel-scale log magnitude spectrogram (MLS) and self-similarity lag matrices, and its output is the probabilitythat the middle frame is a boundary. Because of the fully-connected layers, the input features are only from fixed-size context frames, not from a full song.

Unlike [1], our proposed method takes the MLS of a fullsong as an input. It makes the model can detect musicalboundaries considering long temporal information.

The structure of the paper is as followings: our over-all proposed method is illustrated in Section 2. Section 3

c© Jeongmin Liu, Kyeongseok Jeong, Hyunjun Lee. Li-censed under a Creative Commons Attribution 4.0 International License(CC BY 4.0). Attribution: Jeongmin Liu, Kyeongseok Jeong, HyunjunLee. “Music Boundary Detection using Fully Convolutional Neural Net-works”,

presents the experiment setup and result analysis. Finally,the entire content of the paper is wrapped up in Section 4.

2. PROPOSED METHOD

2.1 Feature Extraction And Data Augmentation

To calculate an MLS from the audio signal, the Hann win-dow of 46 ms length and 2048-points FFT are used. Thehop size is 1024, and 128 mel-scaled filterbanks are used.

All possible combinations of three augmentation typesare applied to original features. First, +1 and -1 step pitch-shifting is applied. Second, -24dBFS of the white noise isadded. Third, SpecAugment [3] is applied: up to 15 ran-domly chosen mel bins are filled with zero.

2.2 Fully Convolutional Neural Network (FCN)

We utilize a fully convolutional neural network (FCN)based on U-Net [5] as shown in Figure 1. There are threedifferences between the original U-Net and our model.First, because our desired output is 1D, our model hasglobal average pooling with respect to the mel axis af-ter the last convolutional layer. Second, skipped featuresare summed to decoded features in our model becausethat shows better results than they are concatenated. Third,our model uses convolutional kernels with long time-axislengths. 3×11 kernels and 1×5 kernels are used to capturelong temporal features.

2.3 Learning Target and Loss Function

The DNN is optimized to estimate boundary scores fromgiven MLS. At every training epoch, one annotation israndomly selected from two annotation data in SALAMI.From the annotation, the frame-wise labels are created: thelabel is 1 if boundary occurs in the frame, and 0 if boundarydoes not occur. Boundary scores are the frame-wise labelssmeared by the 31-length Gaussian kernel [8] as shown inFigure 2.

The loss function is weighted binary cross entropy. Be-cause zero values occur more frequently than non-zero val-ues in target boundary score y’s, the following weight func-tion is needed:

W (a) =

{1, a > 0

K0

K−K0, a = 0

, (1)

where K is the total number of all time frames in the train-ing set and K0 is the total number of all frame index i’s that

Figure 1: The FCN for the proposed method based on U-Net [5].

Figure 2: An example of boundary score.

satisfy y(i) = 0 in all y’s of the training set. For predictedboundary score y, the loss function is defined by

L(y, y) = − 1

T

T−1∑i=0

W (y(i)) [(1− y(i)) ln(1− y(i))

+y(i) ln y(i)] ,

(2)

where T is the number of time frames in y.

2.4 Peak-Picking Algorithm

Peak-picking from estimated boundary score is performedto decide which frame is a boundary. First, the set ofboundary candidate indices B consists of every outputvalue that is a local maximum within a±6 seconds [8]. Forthe sample rate fs, the hop size L, and the round operatorb·e, the number of time frames in 6 seconds is expressed asN = b6fs/Le − 1. Hence, B is defined by

B =

{i

∣∣∣∣i = argmaxi−N≤k≤i+N

y(k)

}. (3)

Second, to predict which candidates are boundary, we pro-pose that the threshold y is defined by the mean of candi-dates’ boundary scores to peak less predicted boundaries.It can be expressed as y =

∑i∈B y(i)/|B|. Finally, the set

of predicted boundaries B consists of the candidates whoseboundary scores are greater than the threshold y:

B ={i∣∣i ∈ B, y(i) ≥ y

}. (4)

3. EXPERIMENTS AND RESULTS

3.1 Dataset

We use a part of SALAMI v2.0 dataset [6] whose audiodata is publicly opened. 373 songs are used for trainingand 66 songs are used for validation. For the test set, weannotate 23 songs from 8 different genres following theannotator’s guide provided by SALAMI. 44.1 kHz samplerate is used for all the audio data.

3.2 Evaluation Metrics

The selected evaluation measure is hit-rate with time toler-ances of ±0.5 s [7] and ±3.0 s [2]. It checks whether pre-dicted boundaries are matched to the target within a timetolerance. Matched predictions are considered true posi-tives. Unmatched predictions and the truth are false posi-tives and negatives each. Precision, recall, and F1 scoresare calculated with these above. The evaluation is pro-ceeded with the Python package mir eval [4].

(a)

(b)

Figure 3: The mel spectrogram (a) and the estimated boundary score (b) of “Take The Dive by종현”, which is one of thetest set. Both (a) and (b) show its target boundary and predicted boundary.

3.3 Results

Figures 3a and 3b illustrates the mel spectrogram and theestimated boundary score of a song in the test set. Both fig-ures show the target boundary and the predicted boundary.The DNN predicts only two boundaries incorrectly.

The F1 scores with time tolerances ±0.5 s and ±3.0 saccording to genres are shown in Figures 4a and 4b respec-tively. The circle marker means the average, the upper errormarker means the maximum, and the lower error markermeans the minimum. The total mean of F1 scores withtime tolerance±0.5 s is 0.432, and that with time tolerance±3.0 s is 0.609. Our test set is different from [1], however,roughly comparing our results to [1] can be done. The testresults of the proposed method is close to that of [1] eventhough our training set is small. The table for precision,recall, and F1 score are presented in Appendix.

There are two problems that make it difficult for theDNN to detect boundaries. One is the incomplete bar prob-lem. It makes the minimum F1 scores of Blues and Elec-tronica in “Gravity by John Mayer” and “아모르 파티 by김연자” respectively. As shown in Figures 5a and 5c, weannotate the first beat of the bar right after an incompletebar as a boundary, but the DNN predicts the beginningof an incomplete bar as a boundary. Because that differ-ence doesn’t exceed 3.0 s usually, the time tolerance ±3.0s makes those cases the correct predictions. Therefore,F1 scores with the time tolerance ±3.0 s for Blues andElectronica show less deviations than F1 scores with thetime tolerance ±0.5 s. The other is about smooth changesof musical ideas. The DNN tends not to be able to de-tect boundaries where smooth changes occur. An exampleis a part of “狂犬時代 by 자우림” shown in Figures 5band 5d. To relieve this problem, the annotators should an-notate boundaries only at the first beat of bars and the DNNshould be trained with the musical tempo information.

(a)

(b)

Figure 4: F1 scores of test set according to genres. Toler-ance is 0.5 s [7] for (a) and 3.0 s [2] for (b).

(a) (b)

(c) (d)

Figure 5: Unmatched predicted boundary and target plot-ted on MLS and estimated boundary score, each incolumn-wise. (a) and (c) are for “아모르파티 by김연자.”(b) and (d) are for “狂犬時代 by자우림.”

4. CONCLUSION

We use an FCN based on U-Net for music boundary detec-tion. MLS of a full song is used for the input, the frame-wise boundary scores with Gaussian kernel of a full song isused for the desired output. Training and validation is pro-cessed with small dataset: only publicly opened audio andannotation data in SALAMI. The proposed method showsslightly lower F1 score than the method in [1], but it isnot a rigorous comparison because we used our own testset. Incomplete bars in music and smooth change of musi-cal ideas make the DNN confused to detect boundaries. Tosolve those problems, boundary detection synchronized tothe musical tempo might be needed.

5. AUTHOR CONTRIBUTION

Jeongmin Liu proposed to use FCNs and implementedoverall algorithms. He also read one paper for the literaturesurvey, annotated 10 songs for the test set, and analyzedtheir test results. For the introduction presentation, the fi-nal poster presentation, and the final report, he wrote theparts about the proposed DNN architecture, training tech-niques, results, and conclusion. He presented the parts hewrote for the introduction presentation.

Kyeongseok Jeong implemented algorithms to evalu-ate and analyze test results. He also read 3 papers and theSALAMI annotator’s guide for the literature survey, anno-tated 13 songs for the test set, and analyzed their test re-sults. For the introduction presentation, he wrote and pre-sented the introduction part. He wrote the introduction andthe results for the final poster presentation, and wrote the

abstract and the introduction for the final report.Hyunjun Lee proposed the project topic and peak-

picking algorithm. He also implemented the peak-pickingalgorithm. For the literature survey, he read 3 papers. Forthe introduction presentation, he wrote and presented pre-vious works part. He wrote peak-picking algorithm partfor the final poster presentation, and presented the entireposter. For the final report, he wrote the parts about featureextraction, data augmentation, the peak-picking algorithmpart, and conclusion.

6. REFERENCES

[1] Thomas Grill and Jan Schluter. Music boundary detec-tion using neural networks on spectrograms and self-similarity lag matrices. In the 23rd European SignalProcessing Conference, pages 1296–1300, 2015.

[2] M. Levy and M. Sandler. Structural segmentation ofmusical audio by constrained clustering. IEEE Trans-actions on Audio, Speech, and Language Processing,16(2):318–326, Feb 2008.

[3] Daniel S Park, William Chan, Yu Zhang, Chung-ChengChiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.Specaugment: A simple data augmentation methodfor automatic speech recognition. arXiv preprintarXiv:1904.08779, 2019.

[4] Colin Raffel, Brian McFee, Eric J Humphrey, JustinSalamon, Oriol Nieto, Dawen Liang, Daniel PW El-lis, and C Colin Raffel. mir eval: A transparent imple-mentation of common mir metrics. In Proc. of the 15thInternational Society for Music Information RetrievalConference. Citeseer, 2014.

[5] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.U-Net: Convolutional Networks for Biomedical Im-age Segmentation. In Nassir Navab, Joachim Horneg-ger, William M. Wells, and Alejandro F. Frangi,editors, Medical Image Computing and Computer-Assisted Intervention 2015, volume 9351, pages 234–241. Springer International Publishing, 2015.

[6] Jordan Bennett Louis Smith, John Ashley Burgoyne,Ichiro Fujinaga, David De Roure, and J StephenDownie. Design and creation of a large-scale databaseof structural annotations. In Proc. of the 12th Interna-tional Society for Music Information Retrieval Confer-ence, volume 11, pages 555–560. Miami, FL, 2011.

[7] Douglas Turnbull, Gert RG Lanckriet, Elias Pampalk,and Masataka Goto. A supervised approach for detect-ing boundaries in music using difference features andboosting. In Proc. of the 8th International Society forMusic Information Retrieval Conference, pages 51–54,2007.

[8] Karen Ullrich, Jan Schluter, and Thomas Grill. Bound-ary Detection in Music Structure Analysis Using Con-volutional Neural Networks. Number Ismir, pages417–422, 2014.

7. APPENDIX

GENRE Precision Recall F1Hip-hop 0.380 0.421 0.394

Dance pop 0.558 0.551 0.554Blues 0.300 0.300 0.300R&B 0.694 0.614 0.652

Electronica 0.322 0.379 0.348Rock 0.470 0.507 0.487Folk 0.229 0.319 0.264

Ballad 0.461 0.542 0.497TOTAL 0.420 0.451 0.432

(a)

GENRE Precision Recall F1Hip-hop 0.490 0.546 0.510

Dance pop 0.791 0.775 0.781Blues 0.592 0.592 0.592R&B 0.792 0.700 0.743

Electronica 0.547 0.622 0.581Rock 0.606 0.660 0.631Folk 0.371 0.537 0.433

Ballad 0.539 0.633 0.582TOTAL 0.591 0.638 0.609

(b)

Table 1: Mean value of precision, recall, and F1 scoresaccording to genres and in total. Tolerance is 0.5 s [7] for(a) and 3.0 s [2] for (b).