Is what you see What you get? -...

© Michael Stahl, 2015, All Rights Reserved

Testing Computer-Vision applications

Michael Stahl, SW Validation Architect

Intel

Nov 2015

Is what you seeWhat you get?

1

© Michael Stahl, 2015, All Rights Reserved 2

Agenda

Intel RealSense® technology

Demos

Testing challenges

Some answers

Summary

Intel® RealSense™ TechnologyHuman Senses. Natural, Intuitive and Immersive

Intel® RealSense™ Devices

intel.com/RealSense/devices

5

Intel® RealSense™ Cameras - 2015

devkitsavailable today

products

User-facing (F200) World-facing (R200)


Front-facing 3D camera (F200)


Source: Intel Developers Forum 2015 presentation

Front-facing 3D camera (F200)


Rear-facing 3D camera (R200)


Source: Intel Developers Forum 2015 presentation

Rear-facing 3D camera (R200)


Face

11


Face

12


Face

13


Enhanced Photography

14



15



16



17


Background Removal

18


Gesture

19


Augmented Reality

20

Testing Challenges


Face Detection

22

https://software.intel.com/en-us/articles/intel-realsense-depth-camera-r200-code-sample-face-tracking

Pass?

Fail?

Good Enough?


HW FW

23

First test

SW (driver, MW, apps)

“Ground Truth” (G.T.)

ComparePass / Fail


The Ground Truth challenge

24

30 – 60 frames per second

Quis custodiet ipsos custodes?



25


RSSDK

HW FW

26

The Ground Truth challenge: First test


Record

G.T.

Compare Pass / Fail


RSSDK

HW FW

27

The Ground Truth challenge: Regression


Record

G.T.

Compare Pass / Fail


RSSDK

HW FW

28

The Ground Truth challenge: HW, FW change


Record

G.T.

Compare Pass / Fail


RSSDK

HW FW

29



Record

G.T.

Compare Pass / Fail


RSSDK

HW FW

30



Record

G.T.

Compare Pass / Fail


RSSDK

HW FW

31



Record

G.T.

Compare Grade!


RSSDK

32

The Test Data challenge

1 Min = 1Gb


The metrics challenge

33



34



35


The metrics challenge – What’s better?

36


The test technique challenge

37

The end of EC testing?

Valid EC Invalid EC



38

What’s the Boundary Value?...


The combinatorial coverage challenge

39



40


The PC challenge

41

https://www.youtube.com/watch?v=t4DT3tQqgRM


The PC challenge

42

https://www.youtube.com/watch?v=t4DT3tQqgRM


The PC challenge

43

https://www.youtube.com/watch?v=MsR7GPnpzMY


The PC challenge

Some answers


Ground Truth from competing products

47


Ground Truth from CAD models

48


Ground Truth by other algorithms

49

Other algorithm(e.g. slow)

Algorithm under Test


Ground Truth by other algorithms

50

Other algorithm

Algorithm under Test

Edit


Ground Truth by Invariants

51


Ground Truth by Databases

52


https://www.cs.cmu.edu/~cil/v-images.html



53



Manual creation of Ground Truth

54

• Crowd Sourcing• Outsourcing• Just Do it...



56

Ground Truth(G)

Detected(D)



57

G D = 56

G D =

8 + 8 + 2*9 + 7 + 56 = 97


Overlap metrics

58

What part of D area is correct?

Precision =𝐺∩𝐷

𝐷=

56

97= 0.58

What part of G area was detected?

Recall =𝐺∩𝐷

𝐺=

56

72= 0.78


Extreme cases

59


𝐷=

72

418= 0.17

Recall =𝐺∩𝐷

𝐺=

72

72= 1.0


Extreme cases

60


𝐷=

1

1= 1.0

Recall =𝐺∩𝐷

𝐺=

1

72= 0.014


Combined metrics

61

Harmonic Mean

= 2 ∗Precision ∗ Recall

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙0.66 0.29 0.03

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦

=(𝐺 ∩ 𝐷) + (𝐺 ∩ 𝐷)

𝑇𝑜𝑡𝑎𝑙 𝑖𝑚𝑎𝑔𝑒 𝑠𝑖𝑧𝑒

0.860.17

0.83

𝐺 ∩ 𝐷

𝐺 𝑈 𝐷0.58 0.17 0.013

Etc.


Choosing metric

66

8678387 8678387 8678387


Choosing metric

67


Choosing metric

68


No False

Positives

No False

Negatives

No False

Positives

No False

Negatives

Choosing metric



70

30 – 60 frames per second

multiple people, multiple conditions



multiple people, multiple conditions

Final Score

?



73

The end of EC testing?

Valid EC Invalid EC


Age

ROI size

Image quality

Color quality

Facial hairSkin tone

Ethnic groups

Accessories

Etc. etc.

“make up” Equivalence Classes


Boundary Values

75


Boundary Value by Facts of Life

76

Trial & error, common sense, known technology limitations, etc.

מנפקל "מה שצהזה



77


79



80


One Way coverage


81

96 tests


All-pairs



Base Choice



85



86


Summary

87

Testing CV applications is a new frontier for testers

G.T. is a huge problem

Metrics, test techniques:

Lots of academic work

Little to No industry information

We figure it out as we go...

Thank You!

Is what you see What you get? -...

Documents

Transcript of Is what you see What you get? -...