© Michael Stahl, 2015, All Rights Reserved

Testing Computer-Vision applications

Michael Stahl, SW Validation Architect

Intel

Nov 2015

Is what you seeWhat you get?

1

© Michael Stahl, 2015, All Rights Reserved 2

Agenda

Intel RealSense® technology

Demos

Testing challenges

Some answers

Summary

Intel® RealSense™ TechnologyHuman Senses. Natural, Intuitive and Immersive

Intel® RealSense™ Devices

intel.com/RealSense/devices

5

Intel® RealSense™ Cameras - 2015

devkitsavailable today

products

User-facing (F200) World-facing (R200)

© Michael Stahl, 2015, All Rights Reserved 6

Front-facing 3D camera (F200)

© Michael Stahl, 2015, All Rights Reserved 7

Source: Intel Developers Forum 2015 presentation

Front-facing 3D camera (F200)

© Michael Stahl, 2015, All Rights Reserved 8

Rear-facing 3D camera (R200)

© Michael Stahl, 2015, All Rights Reserved 9

Source: Intel Developers Forum 2015 presentation

Rear-facing 3D camera (R200)

Demos

© Michael Stahl, 2015, All Rights Reserved

Face

11

© Michael Stahl, 2015, All Rights Reserved

Face

12

© Michael Stahl, 2015, All Rights Reserved

Face

13

© Michael Stahl, 2015, All Rights Reserved

Enhanced Photography

14

© Michael Stahl, 2015, All Rights Reserved

Enhanced Photography

15

© Michael Stahl, 2015, All Rights Reserved

Enhanced Photography

16

© Michael Stahl, 2015, All Rights Reserved

Enhanced Photography

17

© Michael Stahl, 2015, All Rights Reserved

Background Removal

18

© Michael Stahl, 2015, All Rights Reserved

Gesture

19

© Michael Stahl, 2015, All Rights Reserved

Augmented Reality

20

Testing Challenges

© Michael Stahl, 2015, All Rights Reserved

Face Detection

22

https://software.intel.com/en-us/articles/intel-realsense-depth-camera-r200-code-sample-face-tracking

Pass?

Fail?

Good Enough?

© Michael Stahl, 2015, All Rights Reserved

HW FW

23

First test

SW (driver, MW, apps)

“Ground Truth” (G.T.)

ComparePass / Fail

© Michael Stahl, 2015, All Rights Reserved

The Ground Truth challenge

24

30 – 60 frames per second

Quis custodiet ipsos custodes?

© Michael Stahl, 2015, All Rights Reserved

The Ground Truth challenge

25

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

26

The Ground Truth challenge: First test

SW (driver, MW, apps)

Record

G.T.

Compare Pass / Fail

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

27

The Ground Truth challenge: Regression

SW (driver, MW, apps)

Record

G.T.

Compare Pass / Fail

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

28

The Ground Truth challenge: HW, FW change

SW (driver, MW, apps)

Record

G.T.

Compare Pass / Fail

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

29

The Ground Truth challenge: HW, FW change

SW (driver, MW, apps)

Record

G.T.

Compare Pass / Fail

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

30

The Ground Truth challenge: HW, FW change

SW (driver, MW, apps)

Record

G.T.

Compare Pass / Fail

© Michael Stahl, 2015, All Rights Reserved

RSSDK

HW FW

31

The Ground Truth challenge: HW, FW change

SW (driver, MW, apps)

Record

G.T.

Compare Grade!

© Michael Stahl, 2015, All Rights Reserved

RSSDK

32

The Test Data challenge

1 Min = 1Gb

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

33

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

34

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

35

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge – What’s better?

36

© Michael Stahl, 2015, All Rights Reserved

The test technique challenge

37

The end of EC testing?

Valid EC Invalid EC

© Michael Stahl, 2015, All Rights Reserved

The test technique challenge

38

What’s the Boundary Value?...

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

39

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

40

© Michael Stahl, 2015, All Rights Reserved

The PC challenge

41

https://www.youtube.com/watch?v=t4DT3tQqgRM

© Michael Stahl, 2015, All Rights Reserved

The PC challenge

42

https://www.youtube.com/watch?v=t4DT3tQqgRM

© Michael Stahl, 2015, All Rights Reserved

The PC challenge

43

https://www.youtube.com/watch?v=MsR7GPnpzMY

© Michael Stahl, 2015, All Rights Reserved 44

The PC challenge

Some answers

© Michael Stahl, 2015, All Rights Reserved

The Ground Truth challenge

© Michael Stahl, 2015, All Rights Reserved

Ground Truth from competing products

47

© Michael Stahl, 2015, All Rights Reserved

Ground Truth from CAD models

48

© Michael Stahl, 2015, All Rights Reserved

Ground Truth by other algorithms

49

Other algorithm(e.g. slow)

Algorithm under Test

© Michael Stahl, 2015, All Rights Reserved

Ground Truth by other algorithms

50

Other algorithm

Algorithm under Test

Edit

© Michael Stahl, 2015, All Rights Reserved

Ground Truth by Invariants

51

© Michael Stahl, 2015, All Rights Reserved

Ground Truth by Databases

52

Ground Truth by Databases

https://www.cs.cmu.edu/~cil/v-images.html

© Michael Stahl, 2015, All Rights Reserved

Ground Truth by Databases

53

Ground Truth by Databases

© Michael Stahl, 2015, All Rights Reserved

Manual creation of Ground Truth

54

• Crowd Sourcing• Outsourcing• Just Do it...

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

56

Ground Truth(G)

Detected(D)

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

57

G D = 56

G D =

8 + 8 + 2*9 + 7 + 56 = 97

© Michael Stahl, 2015, All Rights Reserved

Overlap metrics

58

What part of D area is correct?

Precision =𝐺∩𝐷

𝐷=

56

97= 0.58

What part of G area was detected?

Recall =𝐺∩𝐷

𝐺=

56

72= 0.78

© Michael Stahl, 2015, All Rights Reserved

Extreme cases

59

Precision =𝐺∩𝐷

𝐷=

72

418= 0.17

Recall =𝐺∩𝐷

𝐺=

72

72= 1.0

© Michael Stahl, 2015, All Rights Reserved

Extreme cases

60

Precision =𝐺∩𝐷

𝐷=

1

1= 1.0

Recall =𝐺∩𝐷

𝐺=

1

72= 0.014

© Michael Stahl, 2015, All Rights Reserved

Combined metrics

61

Harmonic Mean

= 2 ∗Precision ∗ Recall

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙0.66 0.29 0.03

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦

=(𝐺 ∩ 𝐷) + (𝐺 ∩ 𝐷)

𝑇𝑜𝑡𝑎𝑙 𝑖𝑚𝑎𝑔𝑒 𝑠𝑖𝑧𝑒

0.860.17

0.83

𝐺 ∩ 𝐷

𝐺 𝑈 𝐷0.58 0.17 0.013

Etc.

© Michael Stahl, 2015, All Rights Reserved

Choosing metric

66

8678387 8678387 8678387

© Michael Stahl, 2015, All Rights Reserved

Choosing metric

67

© Michael Stahl, 2015, All Rights Reserved

Choosing metric

68

© Michael Stahl, 2015, All Rights Reserved 69

No False

Positives

No False

Negatives

No False

Positives

No False

Negatives

Choosing metric

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

70

30 – 60 frames per second

multiple people, multiple conditions

© Michael Stahl, 2015, All Rights Reserved

The metrics challenge

multiple people, multiple conditions

Final Score

?

© Michael Stahl, 2015, All Rights Reserved

The test technique challenge

© Michael Stahl, 2015, All Rights Reserved

The test technique challenge

73

The end of EC testing?

Valid EC Invalid EC

© Michael Stahl, 2015, All Rights Reserved 74

Age

ROI size

Image quality

Color quality

Facial hairSkin tone

Ethnic groups

Accessories

Etc. etc.

“make up” Equivalence Classes

© Michael Stahl, 2015, All Rights Reserved

Boundary Values

75

© Michael Stahl, 2015, All Rights Reserved

Boundary Value by Facts of Life

76

Trial & error, common sense, known technology limitations, etc.

מנפקל "מה שצהזה

© Michael Stahl, 2015, All Rights Reserved

The test technique challenge

77

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

© Michael Stahl, 2015, All Rights Reserved

79

The combinatorial coverage challenge

© Michael Stahl, 2015, All Rights Reserved

80

The combinatorial coverage challenge

One Way coverage

© Michael Stahl, 2015, All Rights Reserved

81

96 tests

The combinatorial coverage challenge

All-pairs

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

Base Choice

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

Base Choice

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

Base Choice

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

85

© Michael Stahl, 2015, All Rights Reserved

The combinatorial coverage challenge

86

© Michael Stahl, 2015, All Rights Reserved

Summary

87

Testing CV applications is a new frontier for testers

G.T. is a huge problem

Metrics, test techniques:

Lots of academic work

Little to No industry information

We figure it out as we go...

© Michael Stahl, 2015, All Rights Reserved 88

Thank You!