Current methods in forensic speaker ID: results of the fake case

Tina Cambier-LangeveldDutch Ministry of Justice

formerly employed by the Netherlands Forensic Institutecurrently employed by the Immigration and Naturalisation

Service

IAFPA24 July 2006

Outline• Main goal• Set-up: preparations, material, time path• The participants• Results summarized per file: Q1-10• Summary of the results• Other findings• Conclusion

Main goal• To document the different methodologies and

strategies that are used in different laboratories / by different experts to come to a final conclusion in a speaker verification task, based on an examination of the same set of materials

• Have as many experts participate as possible (representative of the field, i.e. including all methods that are currently being used)

• Not a proficiency test but a collaborative exercise

Preparations• 20 experts were found willing to participate

• Erik (a former colleague) asked three Canadian friends if we could record some telephone conversations with them for this purpose

• The friend who could also be reached at work was picked as ‘suspect’ or Reference speaker

• Jos Bouten acted as potential customer and speaking partner for the Reference recordings

Material (audio)• The Questioned recordings (Q1-10) were

telephone conversations between friends, very informal, varying in duration between 22 seconds and 11 minutes (file duration, i.e. including ringing)

• The Reference recordings (R1-2) were telephone conversations with a potential customer asking about their services in detail, more formal, quite long: about 5 and 11 minutes (file duration), each containing > 2 minutes of net speech

Material (questionnaire)A questionnaire was added, asking about the experience of the experts (in general and in English casework), the software that was used, the time spent on the fake case, any restrictions that the institute or the law imposes, comments, etc.

Time path• The materials were sent out to 20 potential

participants in August-September 2004• Reports were due back before 1 February 2005• The final (12th) report was received on 29 April

2005, i.e. three months later• Definite results are presented to you now

(as planned)• Paper for the International Journal of Speech,

Language and the Law is currently under review

The participants (1)• 12 participants from 10 different countries,

within and outside of Europe, employed by forensic laboratories, universities, police labs, the military etc. or working as private experts

Methods:• 2 fully automatic• 5 auditory-phonetic, all members of IAFPA• 5 semi-automatic (variable)

pretty representative of the field :-)

The participants (2)• Experience of the participants ranges

between 28 years and one case; average >10 years

• 5 of the 12 participants had never worked on a case in English before

• 5 of the 12 participants had done between 1 and 3 cases in English before

• 2 of the 12 participants had worked on cases in English a lot

Pooling of the resultsThe strength of the conclusion is not reported, only:

• whether a conclusion was drawn• the direction of the conclusion (the same/not

the same speaker as in the Reference recordings)

…since the strength of the conclusion largely depends on the familiarity with the language, and so it would not be a fair comparison.

Exempted from the results• Two of the reports are excluded from this first

section because the experts involved explicitly state that they have assumed that all of the ten questioned recordings came from the same source (1 semi-automatic, 1 auditory-phon.)

• One of the reports is excluded from this section because the expert involved, after first ‘ensuring’ that all the questioned material is from one speaker (?), also examined the ten questioned recordings as a whole (1 auditory-phon.)

Results summarized per file…...for 9 reports (2 x fully automatic, 4 x semi-

automatic, 3 x auditory-phonetic)

Results summarized per file: Q1-Net duration > 2 minutes-not the same speaker

8 x correct conclusion1 x no decision (automatic analysis)

Results summarized per file: Q2-Net duration: approx. 10 seconds-the same speaker

3 x correct conclusion (2 x aud-phon, 1 x semi)6 x too short / not enough information / no decision

Results summarized per file: Q3, 4, 6-Net duration: more than 50 sec. each-the same speaker

9 x correct conclusion

Results summarized per file: Q5-Net duration: approx. 14 seconds-the same speaker

7 x correct conclusion2 x too short / not enough information (2 x semi)

Results summarized per file: Q7-Net duration: approx. 5 seconds-the same speaker

2 x correct conclusion (1 x aud-phon, 1 x semi)7 x too short / not enough information / no decision

Results summarized per file: Q8-Net duration: approx. 15 seconds-not the same speaker

6 x correct conclusion3 x too short / not enough information / no decision (2 x semi, 1 x aut.)

Results summarized per file: Q9-Net duration: approx. 14 seconds-the same speaker

7 x correct conclusion2 x too short / not enough information (2 x semi)

Results summarized per file: Q10-Net duration: approx. 18 seconds-not the same speaker

2 x correct conclusion (1 x aut, 1 x semi)5 x too short / not enough information / no decision2 x wrong conclusion (1 x aud-phon, 1 x semi)

Summary of the results• 3 participants (1 semi-automatic, 2

auditory-phonetic), assuming that all of the Q-files were from one speaker, failed to pick out the three conversations that were from different speakers

• 2 other participants (1 semi-automatic, 1 auditory-phonetic) identify Q1 and Q8 as being from a different speaker, but incorrectly conclude that Q10 is from the Reference speaker.

Summary of the resultsI have not been able to detect any obvious relation between the results and

• the experience of the participants, in general or in English casework

• their proficiency in English• the number of experts involved in the

examination

Some other findings: all 12 reports• F0 measurements are done in 9 reports• Formant measurements are done in 7

reports• Speech rate / articulation rate / speaking

rate measurements are done in 4 reports

…but measurements are often not comparable, because they are implemented differently, done on different selections of the material or reported differently.

Some other findings• In 4 of the 5 auditory-phonetic reports, a

difference in communicative context is noted

• In 2 reports, the accent is correctly identified as Canadian, “possibly Ontario” / “maybe in the area of Toronto”

• Documented feature ‘Canadian raising’ is referred to in 1 report (not a native of English)

• Other accent remarks: RP, general American

Some other findingsGradations in the conclusion:

• 4 x Bayesian format (Likelihood Ratio, or an amount of support for hypothesis H0 vs. H1)

• 6 x in terms of likelihood, possibility etc. (all of these 6 scales are different! The number of possible conclusions ranges from 6 to 11…)

• 1 x categorical (due to the legal system)• 1 x no gradations, just free text (private expert)

Questionnaire results• Time spent on the case ranges from 28

hours to 3 x 40 hours; average 65 hours

• Time spent on an average case ranges from “8 hours for a 1:1 comparison” to “one month”. Average for the other 10 is 48 hours

• Number of researchers involved: one (4x), two (3x), three (3x), four (2x), including technicians

Questionnaire results• Software/speech analysis tools:

Praat (5x), Cool Edit Pro/Adobe Audition (5x), SoundForge (3x), Kay CSL (3x), Wavesurfer (2x), SIVE (2x), Matlab (1x), x-waves Entropic (1x), DCLive Forensics (1x), own systems (5x)

Some are typically suitable for editing and listening (Adobe Audition, SoundForge), others are better analysis tools (Praat, SIVE)

Questionnaire results• “Is your analysis or report in any way

restricted or enforced by e.g. the law, guidelines of your institute, accreditation boards, or previous court decisions?”

- Four experts mention their laboratory / institute in their answer to this question

- Only one participant mentioned the IAFPA Code of Conduct…….

Questionnaire resultsComments include:

• examination limited due to the language (2x)• examination limited for other reasons (3x)• took up too much time (2x)• “This was a very typical case for me”• “well chosen material, not too difficult, not

too easy!”

• words of appreciation (5x) ;-)

Questionnaire resultsSuggestions for the next exercise:

• mismatched recording conditions (2x)• another language / various languages /

multilingual speech (4x)• straight 1:1 comparison• different speaking styles

ConclusionGeneral comments:• Gives great insight into the different methods

that are used• Reports vary widely on almost every aspect

you can think of, and overlap is very limited, also between experts using the same method

• Some of the reports may be made available (selectively), together with the audio material