HUMAN-BIOMETRIC SENSOR INTERACTION AUTOMATION

USING THE KINECTZACH MOORE

•Can the Kinect 2 be used to determine Human-Biometric Sensor Interaction errors automatically in real-time?

RESEARCH QUESTION

METHODOLOGY

•Phase 1: Programming

•Phase 2: Construction

•Phase 3: Pilot Study

•Phase 4: Data Collection

PHASES

KINECT BODY TRACKING

• All face values are a built in feature of the Kinect.• These track the eyes, nose,

and mouth corners.

• 17 upper body points tracked not including the face.

KINECT MEASUREMENT CHECKS

CORRECT PRESENTATION

INCORRECT PRESENTATION

CLASSIFYING ERRORS

•Subject chooses the type of luggage that closely represents what they usually carry in an airport• They can bring their own, or choose from a

selection

•Given mock passport and immigration form

PROTOCOL

• They walk up to the booth and give the forms to the agent (test admin)

• The test admin asks them to provide their 10-print samples

• Once that’s done, they start he iris capture process.• This is where the Kinect is determining any errors

• They provide one sample, gather their belongings, and walk away from the booth

PROTOCOL

Pilot Ground Truth

Scenario

PROCESS MAPResearch Question: Can the Kinect 2 be used to determine Human-Biometric Sensor Interaction errors automatically in real-time?

Booth and usability study. Proved the Kinect was reliable.

My thesis. Will determine if the Kinect can be used to classify errors automatically.

Future work. Provide real-time feedback to users to test if Kinect affects throughput.

•Reviewed the video footage of all 100 subjects• Used to determine if presentation was correct or

incorrect

•Exported the AOptix logs• Used to determine the HBSI metric

•All done after the data collection had concluded

GROUND TRUTH CLASSIFICATION

•Used the body points from the Kinect sensor• This data was used to determine if the presentation

was correct or incorrect

•Monitored the AOptix state change over the network• Used to determine HBSI Metric

•All done in real-time

KINECT CLASSIFICATION

AOPTIX STATES[1] [2] [3] [4] [5]

[6] [8] [11] [13] [15]

[21] [22] [23] [25]

CLASSIFICATION PROCESS

RESULTS

GENDER REPORT

FemaleMale

Category

47.0%

53.0%

Gender

Gender CountMale 47

Female 53

Total 100

AGE BREAKDOWN

49+41-4833-4026-3218-25

40

30

20

10

0

Age Group

Cou

nt

1110

7

31

41

Age Breakdown

ETHNICITY

IndianAra

bMixe

d

Hispani

c

Asian

or Pa

cific I

sland

erOth

er

Africa

n Ameri

canAsian

Cauca

sian

70

60

50

40

30

20

10

0

Ethnicity

Coun

t

1122248

23

57

Subject Ethnicity

CLASSIFICATION RESULTS

GROUND TRUTH CLASSIFICATIONS

SPSFT PFT DFIDICI

120

100

80

60

40

20

0

120

100

80

60

40

20

0

Metric

Cou

nt 67

10

120

21

37

4

Ground Truth HBSI Metric Classifications

EXAMPLE INTERACTION

Subject ID Ground Truth Classification Kinect Classification Correct Classification

066 FTD FTD Y

066 FTD FTD Y

066 FTD FTD Y

066 FTD FTD Y

066 SPS SPS Y

GROUND TRUTH COMPARED TO KINECT

SPSFTPFTDFIDICI

120

100

80

60

40

20

0

120

100

80

60

40

20

0

Metric

Cou

nt 67

10

120

21

37

4

Ground Truth HBSI Metric Classifications

SPSNONEFTPFTDFIDICI

120

100

80

60

40

20

0

120

100

80

60

40

20

0

Metric

Cou

nt

58

70

9

67

2725

3

Kinect HBSI Metric Classifications

•Cause:• The AOptix device switched states so quickly, that

the Kinect did not detect the change

• The Kinect has a fixed frame refresh rate (30fps)

•From the Kinect’s point of view, no error occurred, so it did not classify the presentation

“NONE” CLASSIFICATION

“NONE” CLASSIFCATION

Refresh FrameRefresh Frame

Kinect

AOptix

Subject ID Ground Truth Classification Kinect Classification Correct Classification

028 FTD FTD Y

028 FTD FTD Y

028 FTD NONE N

028 FTD NONE N

028 FTD NONE N

028 SPS SPS Y

“NONE” EXAMPLE

HBSI METRICS CLASSIFIED AS “NONE”

CIDIFTDFTPSPS

Category

31

52

13

1

HBSI Metrics Classified as "NONE" by Kinect• 70 instances of “NONE” classification total

• Of these 70, the ground truth equivalent metric classification is shown

PRESENTATION ACCURACY

Correct PresentationIncorrect Presentation

Category

48.3%51.7%

Kinect Presentation Classifications

Correct PresentationIncorrect Presentation

Category

23.9%

76.1%

Ground Truth Presentation Classifications

ACCURACY OF KINECT CLASSIFICATIONS

Different ClassificationSame Classification

Category

62.9%

37.1%

Kinect Classifications Compared to Ground Truth

ACCURACY BY METRICCI DI FI

FTD FTP SPS

Different ClassificationSame Classification

Category

50.0%

50.0%

51.4%

48.6%

81.0%

19.0%

52.5%47.5%

80.0%

20.0%

80.6%

19.4%

Kinect Classifications Compared to Ground Truth by Metric

•How accurate was the Kinect at determining these errors when it did notice the state change?

•By removing the observations that include “NONE”, does the accuracy improve?

FURTHER QUESTIONS RAISED

REMOVING “NONE’ CLASSIFICATIONS

Subject ID Ground Truth Classification Kinect Classification Correct Classification

028 FTD FTD Y

028 FTD FTD Y

028 FTD NONE N

028 FTD NONE N

028 FTD NONE N

028 SPS SPS Y

Subject ID Ground Truth Classification Kinect Classification Correct Classification

028 FTD FTD Y

028 FTD FTD Y

028 SPS SPS Y

PRESENTATION ACCURACY – WITHOUT “NONE”

Correct PresentationIncorrect Presentation

Category

25.4%

74.6%

Ground Truth Presentation Classifications

Correct PresentationIncorrect Presentation

Category

29.1%

70.9%

Kinect Presentation Classifications

ACCURACY OF KINECT CLASSIFICATIONS – WITHOUT “NONE”

Different ClassificationSame Classification

Category

85.7%

14.3%

Kinect Classifications Compared to Ground Truth

ACCURACY BY METRIC – WITHOUT “NONE”

CI DI FI

FTD FTP SPS

Different ClassificationSame Classification

Category

66.7%

33.3%

79.2%

20.8%

81.0%

19.0%

91.2%

8.8%

88.9%

11.1%

84.4%

15.6%

Kinect Classifications Compared to Ground Truth by Metric

CONCLUSIONS AND FUTURE WORK

•The Kinect can be used to determine HBSI errors in real-time• The accuracy of which depends on the thresholds the

Kinect operates under

•The refresh rate of the Kinect was not high enough to detect all state changes from the AOptix device

•This research provides a foundation for future work

CONCLUSIONS

• Increasing Kinect refresh rate or using different sensor

• Developing real-time feedback to both subject and test administrator• Test change in throughput and performance

• Adjusting Kinect thresholds for correct/incorrect presentation classifications

• Use Kinect gesture recognition to use for different modalities (fingerprint)

• Implement in operational testing

FUTURE WORK

QUESTIONS?