Calibration based on duration quality measures function in...
Transcript of Calibration based on duration quality measures function in...
![Page 1: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/1.jpg)
Calibration based on duration quality measures function in noise robust speaker recognition for NIST SRE’12
Miranti Indar Mandasari, Rahim Saeidi and David van Leeuwen.
Biometric Technologies in Forensic ScienceBTFS Conference, 14 October 2013
![Page 2: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/2.jpg)
Outline
● Introduction,
● Speaker recognition system,
● Corpora,
● Experiment setup,
● Calibration techniques,
– Conventional linear, and
– Quality measure function (QMF).
● Performance measures,
● Results, and
● Conclusion.
![Page 3: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/3.jpg)
Introduction
● The importance of likelihood ratio calibration in speaker recognition:
– Likelihood ratio as a preferable form of score for forensic purposes,
– Acknowledged by the speaker recognition community through speaker recognition evaluation (SRE) by NIST, and
– Often, scores produced by the system are not in likelihood ratio form.
● Classic challenges in speaker recognition:
– Short duration, and
– Noisy speech.
![Page 4: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/4.jpg)
Speaker recognition system
● Speech enhancement and feature extraction stage:
– Dynamic noise suppression rule and Wiener filter,
– 60 dimensional MFCCs feature, and
– Speech activity detection and feature warping.
● Modeling stage:
– Gender-dependent and 2048 components universal background model (UBM),
– 400 dimensional i-vectors,
– 200 dimensional linear discriminant analysis (LDA),
– Pre-PLDA modeling: i-vector centering, within class covariance normalization (WCCN), and i-vector length-normalization, and
– Probabilistic linear discriminant analysis (PLDA) scoring.
![Page 5: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/5.jpg)
Corpora
● NIST SRE'12 database:
– Duration variability, and
– Noise conditions (crowd & HVAC):
● Clean / no-alteration, ● 15 dB noisy, and ● 6 dB noisy.
● Three datasets in the experiments:
– Development set from I4U (Dev-I4U),
– Evaluation set from I4U (Eval-I4U), and
– NIST SRE 2012 protocols (Eval-SRE'12).
● I4U is a joint effort from 9 research Institutes and Universities across 4 continents in joining the NIST SRE'12 evaluation.
![Page 6: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/6.jpg)
Calibration
● Calibration is:
– The ability to set a threshold optimally if scores are used for decisions, or
– The ability to produce likelihood ratios that lead to minimum Bayes' risk for any cost function.
● Calibration techniques:
– Linear calibration with 2 parameters (conventional), and
– Linear calibration with additional quality measure function (QMF).
● Calibration stages:
– Training calibration parameters: Dev-I4U, and
– Evaluation of calibration: Dev-I4U, Eval-I4U, and Eval-SRE'12.
![Page 7: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/7.jpg)
Linear Calibration
LikelihoodRatio
OffsetParameter
ScalingParameter
RawScore
● This two parameterized linear calibration refer to as conventional calibration,
● A monotonously increasing score-to-likelihood-ratio transformation so the discriminability stays the same, and
● The parameters w0 and w1 are found by minimizing cross-entropy (or Cllr) on a development set, i.e., by logistic regression.
![Page 8: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/8.jpg)
QMF calibration
● QMF stands for quality measure function,
● QMF calibration is a linear calibration approach with quality measures as extra terms, and
● There are 4 proposed duration QMFs.
Quality Measure Function (QMF)
Duration of Model Segment
Duration of Test Segment
Extra OffsetParameters
LikelihoodRatio
OffsetParameter
ScalingParameter
RawScore
![Page 9: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/9.jpg)
Duration-dependent Offset parameters
![Page 10: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/10.jpg)
Quality measure functions
![Page 11: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/11.jpg)
Performance measures(the lower the values, the better the performance)
● Equal error rate, E= or EER.
– Showing discrimination performance.
● Primary cost, Cprimary, of NIST SRE'12.
– Showing discrimination and calibration performances.
● Cost of log likelihood ratio, Cllr.
– Showing discrimination (minimum Cllr) and calibration (Cmc)
performances.
![Page 12: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/12.jpg)
Results
![Page 13: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/13.jpg)
EER on Dev-I4U
Clean 15 dB 6 dB0
0.5
1
1.5
2
2.5
3
3.5
4
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4E
ER
(%
)Performance
Measure(EER & C-primary)
DatasetCalibrationTechnique
Trials Based on Noise Conditions
![Page 14: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/14.jpg)
Cllr on Dev-I4UN
.A. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
Clean 15 dB 6 dB
0
0.05
0.1
0.15
0.2
0.25
Cmcminimum Cllr
Cllr
PerformanceMeasure
(Cllr, min.Cllr and Cmc)
Dataset
PerformanceMeasuresCmc or miscalibration cost.
Cmc = Cllr - min.Cllr
Trials Based on Noise Conditions
![Page 15: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/15.jpg)
Results
Dev-I4U
![Page 16: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/16.jpg)
EER on Dev-I4U
Clean 15 dB 6 dB0
0.5
1
1.5
2
2.5
3
3.5
4
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4E
ER
(%
)
![Page 17: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/17.jpg)
C-primary on Dev-I4U
Clean 15 dB 6 dB0
0.05
0.1
0.15
0.2
0.25
0.3
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4
C-p
rim
ary
![Page 18: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/18.jpg)
Cllr on Dev-I4UN
.A. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
Clean 15 dB 6 dB
0
0.05
0.1
0.15
0.2
0.25
Cmcminimum Cllr
Cllr
![Page 19: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/19.jpg)
Results on
Eval-I4U
![Page 20: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/20.jpg)
EER on Eval-I4U
Clean 15 dB 6 dB0
0.5
1
1.5
2
2.5
3
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4E
ER
(%
)
![Page 21: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/21.jpg)
C-primary on Eval-I4U
Clean 15 dB 6 dB0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4
C-p
rim
ary
![Page 22: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/22.jpg)
Cllr on Eval-I4UN
.A. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
Clean 15 dB 6 dB
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Cmcminimum Cllr
Cllr
![Page 23: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/23.jpg)
Results
Eval-SRE'12
![Page 24: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/24.jpg)
EER on Eval-SRE'12
Clean 15 dB 6 dB0
1
2
3
4
5
6
7
8
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4E
ER
(%
)
![Page 25: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/25.jpg)
Cllr on Eval-SRE12N
.A. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
N.A
. O Q1
Q2
Q3
Q4
Clean 15 dB 6 dB-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Cmcminimum Cllr
Cllr
![Page 26: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/26.jpg)
C-primary on Eval-SRE'12
Clean 15 dB 6 dB0
0.2
0.4
0.6
0.8
1
1.2
No calibrationsConventional calibrationQMF calibration - Q1QMF calibration - Q2QMF calibration - Q3QMF calibration - Q4
C-p
rim
ary
![Page 27: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/27.jpg)
Distribution of active speech duration in I4U and SRE'12 trials.
![Page 28: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/28.jpg)
Conclusion
● The linear calibration with QMF as the additional terms shows a positive gain in the system performance compared to the conventional linear calibration with two terms.
● It is shown that by adding 1–2 extra parameters in the linear calibration through QMF approach, there is a potential to improve the calibration and discrimination performances of a speaker recognition system.
● In applying a QMF, it is important to design a development set that match the variability of duration in the evaluated set.
![Page 29: Calibration based on duration quality measures function in ...cls.ru.nl/staff/dvleeuwen/btfs-2013/mandasari-slides-btfs2013.pdf · Calibration Calibration is: – The ability to set](https://reader035.fdocuments.us/reader035/viewer/2022081404/5f0474997e708231d40e0dd6/html5/thumbnails/29.jpg)
Thank you!
&
Questions?