Establishing a Foundation for Automated Human Credibility ... · Establishing a Foundation for...

Establishing a Foundation for

Automated Human Credibility

Screening

Jay Nunamaker

Aaron Elkins

Douglas Derrick

Nathan Twyman

Intro to the AVATAR

2

Research Goals

• Unobtrusive non-invasive credibility assessment

– No sensors attached to the body

– Real-time, remote analysis

– Scalable and robust for high traffic

– Useful across contexts

– Useful across cultures

3

Research Questions

• What video, audio, and language-based features measure changes in behavior and reliably detect deception, impostership, and malicious intent?

• What psychophysiological measures are reliable indicators of deception?

– Heart rate - Pupil diameter

– Changes in Blood pressure - Gaze behavior

– Respiration - Thermal temperature

– Blink Rate - Galvanic skin response

• Which interviewing tools and techniques are most effective for identifying imposters, smugglers, terrorists, and other criminals?

4

Problem Complexity:

There are Many Ways To Deceive

• Lies

• Fabrications

• Concealments

• Omissions

• Misdirection

• Bluffs

• Fakery

• Mimicry

• Tall tales

• White lies

• Deflections

• Evasions

• Equivocation

• Exaggerations

• Camouflage

• Strategic ambiguity

• Hoaxes

• Charades

• Imposters

5

Problem Complexity

• Rapid Screening

– People usually make rapid decisions about credibility (7-20 seconds)

• Automated Interviews

– Force multiplier

– Interview Control and Consistency

• Detecting which questions are most diagnostic

– Question set changes for every context

6

Problem Complexity:

Overconfidence and Underachievement

• Humans are poor lie-detectors

– ~54% accuracy rate for general population

– Accuracy is a function of the quality of base rates

– Poor performance affects novices and professionals

• Confidence in judgment is not correlated with

accuracy

– Affects attentiveness, verification efforts, and

misallocation of resources

7

Interdisciplinary Research

Automated

Deception

Detection

Computer Science,

Information

Systems

Communication,

Linguistics, Speech

& Hearing

Psychology,

Cognitive Science,

Nueroscience

Engineering

(Electrical &

Systems)

8

BORDERS Collaborators

9

Automated Human Screening: 30+ Years of Collaborative, Multi-disciplinary Research

Bioengineering Biometrics

Communication

Computer Science: Computer Vision,

Artificial Intelligence

Information Systems

Criminology

Linguistics Neuroscience

Oculometrics

Polygraph

Insider Threat

Psychology: Cognitive,

Experimental, Social, Organizational

Signal Processing

Virtual Interactions

A Sample of Collaborators:

Computer Science Collaborator Affiliation Expertise

Dr. Larry Hornak West Virginia University

Computer Science, Biometrics, Bioengineering, Computer Vision

Senya Polikovsky Tsukuba University Computer Vision and Image Media Laboratory

Computer Science, Computer Vision, Bioengineering

Dr. Yoshinari Kameda Tsukuba University Computer Vision and Image Media Laboratory


Dr. Jonathan Gratch USC Department of Computer Science

Computer Science, Virtual Interactions

Enrica Dente Imperial College Computer Vision

Dr. Anil Bharath Imperial College Computer Vision, Bioengineering

Dr. Dmitri Metaxas Rutgers University Center for Biomedical Imaging and Modeling Computer Vision, Data Analytics

Dr. Maja Pantic Imperial College

Computer Vision, Virtual Interactions

Dr. Danilo Mandic Imperial College

Signal Processing, Computer Science


Psychology and Psychiatry

Collaborator Affiliation Expertise

Dr. Frank M. Marchak Veridical Research and Design Oculometrics, Psychology

Dr. John Rohrbaugh Washington University Psychiatry, Bioengineering

Dr. David McNeill Chicago Department of Psychology Center for Gesture and Speech Research Psychology

Dr. Jeff Stone University of Arizona Psychology

Dr. John Allen University of Arizona Psychology

Dr. John Kircher University of Utah Psychology, Bioengineering, Polygraph, Oculometrics

Dr. Gary Bente University of Cologne Department of Psychology Psychology, Communication

Dr. Clifford Nass Stanford University Psychology, Communication, Virtual Interactions


Psychology and Psychiatry (cont.)


Dr. Jeremy Bailenson Stanford University Psychology, Communication, Virtual Interactions

Dr. Mike Woodworth

University of British Colombia Centre for the Advancement of Psychological Science and Law Psychology, Linguistics

Dr. Mary Peterson University of Arizona Psychology, Oculometrics

Dr. Andrew B. Dollins National Center for Credibility Assessment Psychology, Polygraph

Dr. Dean Pollina National Center for Credibility Assessment Psychology, Polygraph

Dr. James J. Blascovich UCSB Department of Psychology Psychology, Virtual Interactions

Dr. Dale Tunnell Forensitec Signal Processing, Criminology, Psychology, Polygraph


Neuroscience and Neurology Collaborator Affiliation Expertise

Bruce M. Coull, M.D. University of Arizona, Department of Neurology Neurology

Dr. John G. Hildebrand University of Arizona Department of Neuroscience Neuroscience

Dr. John Allen University of Arizona Psychology, Neuroscience

Biometrics Collaborator Affiliation Expertise

Dr. Arun Ross West Virginia University Biometrics

Dr. Bojan Cukic West Virginia University Biometrics

Dr. Larry Hornak West Virginia University Computer Science, Biometrics, Bioengineering, Computer Vision


Communications and Linguistics Collaborator Affiliation Expertise

Dr. Jeff Hancock Cornell University Linguistics

Dr. Sandiway Fong University of Arizona Linguistics

Dr. Gary Bente University of Cologne Department of Psychology Psychology, Communication



Dr. Mike Woodworth

University of British Colombia Centre for the Advancement of Psychological Science and Law Psychology, Linguistics


Dr. Frank M. Marchak Veridical Research and Design Oculometrics, Psychology


Dr. Mary Peterson University of Arizona Psychology, Oculometrics

Oculometrics


Signal Processing


Dr. Danilo Mandic Imperial College Signal Processing, Computer Science



Polygraph Collaborator Affiliation Expertise

Jennifer Gordon Department of Defense Polygraph

Dan Hiltz Department of Defense Polygraph

Terrie Ritchie Department of Defense Polygraph

Dan Hartless Department of Defense Polygraph

Dan Baxter Department of Defense Polygraph

Marty Oelrich Director, American Polygraph Association Polygraph


Dr. Andrew B. Dollins National Center for Credibility Assessment Psychology, Polygraph

Dr. Dean Pollina National Center for Credibility Assessment Psychology, Polygraph



Insider Threat & Criminology Collaborator Affiliation Expertise

Lt Col Mark DiCarlo DoD Physical Security Equipment Action Group (PSEAG) Insider Threat

George Randall Applied Research Associates Insider Threat

Tom Monaco Applied Research Associates Insider Threat

Tom Whittle DoD Physical Security Equipment Action Group (PSEAG) Insider Threat



Virtual Interactions Collaborator Affiliation Expertise

Dr. Jonathan Gratch USC Department of Computer Science Computer Science, Virtual Interactions

Dr. Maja Pantic Imperial College Computer Vision, Virtual Interactions



Dr. James J. Blascovich UCSB Department of Psychology Psychology, Virtual Interactions

Dr. Bjorn Schuller Munich University of Technology Virtual Interactions


Information Systems, AI, & Engineering Collaborator Affiliation Expertise

Dr. Hsinchun Chen University of Arizona Artificial Intelligence, Information Systems

Dr. Larry Hornak West Virginia University

Computer Science, Biometrics, Bioengineering, Computer Vision

Senya Polikovsky Tsukuba University Computer Vision and Image Media Laboratory


Dr. Yoshinari Kameda Tsukuba University Computer Vision and Image Media Laboratory


Dr. Anil Bharath Imperial College Computer Vision, Bioengineering

Dr. Joseph Valacich University of Arizona Information Systems

Dr. Thomas Meservy Brigham Young University Information Systems


Information Systems, AI, & Engineering Collaborator Affiliation Expertise

Dr. Doug Vogel City University of Hong Kong Information Systems

Dr. Matthew Jensen University of Oklahoma Information Systems

Dr. John Rohrbaugh Medical School, Washington University Psychiatry, Bioengineering


Dr. Salim Harari University of Arizona Computer Engineering, Cloud Computing

John Howie COO of the Cloud Security Alliance (formerly with Microsoft)

Computer Security, Cloud Computing

Research Approach

UA Credibility Assessment Research:

From Then to Now

No Experiment Sample

Size

No Experiment Sample

Size

1. Desert Survival 1 60 19. Agent99 Trainer AFB Pilot, Main

Study, Replication (AFIT/Keesler)

345

2. Desert Survival 2 52 20. Agent99 Trainer AFB Lab

Experiment (FSU)

71

3. Deceptive Interviews 61 21. U.S. Customs & Border

Protection (CBP)

33

4. Mock Theft (Pilot, Main,

Observer) (MSU)

291 22. CBP – Visa Interviews 50

5. Bunker Buster & Scud Hunt 110 23. CBP – Pedestrian Crossing 600

6. StrikeCom (UA/FSU) 582 24. Air Force ROTC StrikeCom Air

Operations

73

7. Security Police Statements

(UA/OSU)

383 25. Cheating (MSU) 250

8. Behavioral Analysis Interviews

(MSU)

25 26. Enhanced Mock Crime 228

9. Fraudulent Financial Statements 330 27. Cultural Benchmarks 220

10. ARI Experiment 220 28. Bomb Study 1 40

11. Resume Studies (FSU) 316 29. Bomb Study 2 38

12. Multiple Receiver Study (FSU) 234 30. Bomb Study 3 40

13. Criminal Interviews 50 31. Morphing (1 & 2) 150

14. Divorce Mediation 20 32. Affirmation 135

15. Diary Study of Media Use &

Deceit (FSU)

49 33. Security Policy 111

16. Participant-Observer Experiment

(FSU)

261 34. Trusted Traveler Pilot 1 75

17. Survey of Media Use (FSU) 532 35. Trusted Traveler Pilot 2 (Starting

July 15th)

1000

18. Agent99 Trainer Pilot (1 and 2) 120 Total Subjects: 7080

Collaboration Research

Measuring

Deception

31

Five Classes of Indicators

Nonstrategic Strategic Thoughtful, premeditated, planned, rehearsed, and/or monitored behaviors

• Behavioral control

– Efforts to hide or control telltale signs

• Communication strategies and tactics

– Deliberate efforts to manage what is said

– Demeanor/self-presentation

Nonrational, uncontrollable, and/or

uncontrolled behaviors

• Arousal-based indicators – Higher psychophysiological

activation with deception

• Emotion-based indicators – Nonverbal cues of guilt or

fear and use of emotional language

• Memory-based processes – Recollections of imagined

vs. real events

32

Detecting Deception and Intent

Language, vocalics, gestures, and

psychophysiological measures

tell the story

40

The Guiding Premise

Cues to deception can be detected no

matter how hard a person tries to

conceal them

• Goal is to get 12-15 reliable cues – Even the best liars cannot control everything

– The others “leak out”

41

Cues and Sensors for Deception

Detection

Recommendation

42

LDV THERMAL BLINK CAMERA

PUPILLOMETRY EYE TRACKING VOCALIC ANALYSIS

COMPUTER VISION LINGUISTICS FORCE PLATFORM

Non-Invasive Tools for Rapid Screening

43

Vocalics

• Higher vocal pitch, intensity, and tempo occur when excited

• Our muscles (about larynx) tense during stress – Higher Pitch

• The process is complex – Emotions affect the normal

operation

– Deception takes away cognitive resources away and is stressful

• More mistakes, lower quality, increased average and variation in pitch

http://splice.cmi.arizona.edu/

Our Internally Developed Tool and Online Solution

45




Kinesics

• Body Movement

– Hands

– Legs

– Head

• Facial Expressions

Our Approach

Sensor Research

First Screening Kiosk

• Lab was transformed by

packaging sensors into a

kiosk

AVATAR

53

Sample Avatar Interviewers

54

• Travelers packed a bag and were interviewed

by an automated screening kiosk

• Half of participants also constructed and

packed an explosive device:

55

Example Experiments: Bomb Studies

Sample Results: Gaze Analysis

•Control

•Average = 12.43%

•SD = 6.46%

•Bomb

•Average = 28.52%

•SD = 13.67%

66

Eye Gaze: Guilty

67

Eye Gaze: Innocent

68

Pupil Dilation Results

•Control

•Average = .1139 mm

•SD = .1938mm

•Bomb

•Average = .2146 mm

•SD = .2119mm

69

The Voice of a Bomb Maker

“Has anyone given you a prohibited substance

to transport through this checkpoint?”

BOMB MAKER INNOCENT

Bomb Makers had 25.34% greater variation in

their vocal pitch when answer this question:

71

Explaining the Variation in Pitch

•Intonation

•Pitch contours reveal a steep

pitch rise over time

•Increased pitch at the end

of an utterance reflect

uncertainty

72

Bomb Maker Innocent

Pit

ch (

Hz)

Time (s)

Field Experiment: Nogales Port of Entry

SENTRI Screening

Rapid Prototyping • A customizable platform for

automated screening – Height adjustable interface

• Standing or sitting

– Various mounting apparati

• Currently equipped with: – EyeTech Eye Tracker

– Dual Force Platforms

– HD camera

– 3D camera

– Stereoscopic microphone

– Touchscreen

– Adjustable Lighting

74

Future Directions

• Examine and improve resilience to countermeasures – Mental and Physical countermeasure detection

• Non-contact measurement of Electrodermal Activity – Partnering with Night Vision Lab

• Expanded SENTRI role and additional applications – Lowering Language Level

– Reciprocal Avatar Behavior / Interactive Dialog

– Separating Avatar and Human voice

Trusted Traveler Pilot Test DeConcini Port of Entry, Nogales AZ

• Pilot Phase 1 (Dec 2011 – Jan 2012) – Conducted 175 interviews

(English only)

– Demos: Comm. Bersin and Aguilar, Pres. Luc Portelance (CBSA)

• Lessons learned: – Rewrite English script

– Translate Spanish

– Recognize when a person speaks over AVATAR

– Establish a training program

• Pilot Phase 2: July 2012 – Incorporate researcher and

CBP feedback

– 1000 Interviews

Future AVATAR Direction Potential Applications

• All Trusted Traveler Programs

• I-94 visas

• TSA domestic travel

• Apprehension interviews

• Periodic investigations

• New hires

– Customize kiosks for these applications

Path to Transition

• Patent process started for AVATAR

technologies

• Vocalics, Linguistics, Kinesics, Ocular, etc.

• Responding to SBIRs and seeking venture

capital

• Started new company: Discern Science Corp.

• The science of the ability to judge well

Questions

Establishing a Foundation for Automated Human Credibility ... · Establishing a Foundation for...

Documents

Transcript of Establishing a Foundation for Automated Human Credibility ... · Establishing a Foundation for...