Subjective Intelligibility Assessment
description
Transcript of Subjective Intelligibility Assessment
Acousteen Herman J.M. Steeneken
Subjective Intelligibility Assessment
Dr. Herman J.M. Steeneken
2Acousteen Herman J.M. Steeneken
Signal-to-Noise ratio !!!
3Acousteen Herman J.M. Steeneken
Research Questions
• Intelligibility versus Quality assessment
• Evaluation of a system or application
• Ranking of the performance of a number of systems
• Diagnostic assessment
• Prediction of system performance during design
4Acousteen Herman J.M. Steeneken
Assessment Methods
• Subjective assessment with subjects (speakers
and listeners): representative, limited
reproduction, non diagnostic, laborious
• Objective assessment based on physical
properties (measurements): reproducible,
diagnostic, fast
• Prediction of system performance: design tool
5Acousteen Herman J.M. Steeneken
SUBJECTIVE INTELLIGIBILITY
6Acousteen Herman J.M. Steeneken
Subjective Intelligibility methods
• Phoneme level (nonsense words, rhyme words,
consonants, vowels)
• Word level (meaningful words, nonsense words,
phonetically balanced PB, equally balanced Eqb)
• Sentence level (Mean Opinion Score MOS, Speech
Reception Threshold SRT)
7Acousteen Herman J.M. Steeneken
Methodology I
Response categories:
• Open response (e.g., nonsense words)
• Closed response (Rhyme tests, e.g., MRT, DRT)
• Scaling (MOS, five point scale: excellent - bad)
• Ranking (e.g., pair-wise comparison)
8Acousteen Herman J.M. Steeneken
Methodology II
Test design:
• Words embedded in carrier phrase
• Reference conditions (e.g. MNRU, …)
• Speakers (gender, number, non-native, …)
• Listeners ( number of speaker-listener pairs)
• Learning effects
9Acousteen Herman J.M. Steeneken
Listening test with four subjects
10Acousteen Herman J.M. Steeneken
11Acousteen Herman J.M. Steeneken
Embedded CVC words:
versta des overen nu fijs uithet woord zek eindenoteer lal punt
“Semi random”combinationof:
17 initial consonants15 vowels11 final consonants
12Acousteen Herman J.M. Steeneken
Methodology III
Scoring, data analysis:
• Phone-word scores
• Confusion matrices
• Effective gain (e.g. effective SNR)
• Statistics (Anova, scaling, multiple regression, ...)
13Acousteen Herman J.M. Steeneken
Relation Consonants-Vowels
initial-consonant score (%)
vow
el s
core
(%
)
0
20
40
60
80
100
0 20 40 60 80 100
14Acousteen Herman J.M. Steeneken
How to calculate average word scores
Subject responsesmay require to usethe median
15Acousteen Herman J.M. Steeneken
Example relation MOS-CVC
12345
0 10 20 30 40 50 60 70 80 90 100
mean CVC word score (%)
MOS
16Acousteen Herman J.M. Steeneken
Relation between methods and Qualification
111
Inte
lligi
bilit
y sc
ore
(%
)
0
20
40
60
80
100
0.0 0.2 0.4 0.6 0.8 1.0
PB-words
CVC EQB
STI rbad goodfairpoor excellent
sentences(non-optimized SRT)
17Acousteen Herman J.M. Steeneken
Test-retest variability
Cronbach αbased on split ofspeaker- listenerpairs
18Acousteen Herman J.M. Steeneken
Common Intelligibility scale (IEC60849)
After Barnett and Knight 1994
CIS not linear with SNR
= STI
= 100 - ALcons
x = AI
= PB words (256 words)
= Short Sentences
= PB words (1000 words)
= 1000 syllables
Barnett and Knight (1995)
19Acousteen Herman J.M. Steeneken
CVC scores (%) of realistic conditions
male female
Wide band 90.3 89.3
Telephone band 89.5 85.3
White noise SNR 0 dB 58.0 44.1
Speech noise SNR +3 dB 71.3 60.7
Speech noise SNR -3 dB 43.0 40.6
20Acousteen Herman J.M. Steeneken
Example of consonant confusions
p b f v m n R w
p 1068 62 12 4 4 0 0 2
b 112 1002 0 0 11 7 0 50
f 44 1 915 193 0 0 0 0
v 6 4 337 739 0 0 2 43
m 1 5 0 0 1068 113 1 6
n 0 0 0 0 111 1081 0 2
R 1 2 0 2 0 2 1161 3
w 6 3 1 13 30 7 25 1065
21Acousteen Herman J.M. Steeneken
Two dimensional display of confusions
22Acousteen Herman J.M. Steeneken
Introduction of phoneme specific frequency weighting
Four groups of phonemes (SAMPA notation:
• Fricatives (f, s, v, z)
• Plosives (b, d, x, p, t, k)
• Vowel-like consonants (m, n, l, R, j, w, …)
• Vowels (aa, a, ee, e, o, oo, u, uu, au, …)
23Acousteen Herman J.M. Steeneken
Phoneme group specific spectra
octave-band centre frequency (Hz)
rela
tive
octa
ve-b
and
leve
l (dB
)
0
10
20
30
40
50
125 250 500 1000 2000 4000 A
fricativesplosivesvow-like consvowelsmean (PB)
male speech
24Acousteen Herman J.M. Steeneken
Phoneme group specific spectra
octave-band centre frequency (Hz)
rela
tive
oct
ave
-ba
nd
leve
l (d
B)
0
10
20
30
40
50
125 250 500 1000 2000 4000 A
fricativesplosivesvow-like consvowelsmean (PB)
female speech
25Acousteen Herman J.M. Steeneken
Frequency weighting (fricatives)
octave-band centre frequency (Hz)
freq
uenc
y-w
eigh
ting
fact
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
fricatives male speech female speech
26Acousteen Herman J.M. Steeneken
Frequency weighting (plosives)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g fa
cto
r
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
plosives male speech female speech
27Acousteen Herman J.M. Steeneken
Frequency weighting (vowel-like cons)
octave-band centre frequency (Hz)
freq
uenc
y-w
eigh
ting
fact
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
vowel-like consonants male speechfemale speech
28Acousteen Herman J.M. Steeneken
Frequency weightings (vowels)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g f
act
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
vowels male speech female speech
29Acousteen Herman J.M. Steeneken
Frequency weightings (CVC words)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g fa
cto
r
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
CVC words male speech female speech
30Acousteen Herman J.M. Steeneken
Prediction of (CVC) word score by aweighted combination of phoneme group probabilities (DUTCH)
Ci = 0.294 fric + 0.294 plo + 0.412 Cvo
V = V (no weighting)
Cf = 0.273 fric + 0.273 plo + 0.454 Cvo
CVC score = Ci * V * Cf * 100 %
31Acousteen Herman J.M. Steeneken
CVC-word prediction (male)
CVC-word score predicted (%)
CV
C-w
ord
scor
e su
bjec
tive
(%)
0
20
40
60
80
100
0 20 40 60 80 100
male speechS.d.= 4.11%
Male speech
32Acousteen Herman J.M. Steeneken
CVC-word prediction (female)
CVC-word score predicted (%)
CV
C-w
ord
sco
re s
ub
ject
ive
(%
)
0
20
40
60
80
100
0 20 40 60 80 100
female speechS.d. = 3.63%
Female speech
33Acousteen Herman J.M. Steeneken
ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)
Application Minimum intelligibilityrating
Maximum vocal effort
Alert and warning situations (correctunderstanding of simple sentences)
Poor Loud see 5.2
Alert and warning situations (correctunderstanding of critical words)
Fair Loud see 5.2
Person-to-person communications(critical)
Fair Loud see 5.3
Person-to-person communications(prolonged normal communication)
Good Normal see 5.3
Public address in public areas Fair Normal see 5.4
Personal communication systems Fair Normal see 5.5
34Acousteen Herman J.M. Steeneken
Qualification table
Intelligibilityrating
Sentencescore % Meaningful
PB-wordscore %
CVCEQB-non-sense word
Score %
STISIL
DB
SII
Excellent 100 > 98 > 81 > 0,75 21
Good 100 93 - 98 70 - 81 0,60 - 0,75 15 – 21 > 0,75
Fair 100 80 - 93 53 - 70 0,45 - 0,60 10 – 15
Poor 70 - 100 60 - 80 31 - 53 0,30 - 0,45 3 – 10 < 0,45
Bad < 70 < 60 < 31 < 0,30 < 3