InSight: Recognizing Humans without Face Recognition

He Wang, Xuan Bao, Romit Roy Choudhury, Srihari Nelakuditi

2

Motivation – Application Scenarios

2

share a ride

to airpor

t

Bob

ElleBret

3

Overview – Self-Fingerprints

Cloud

Bret Bob John Elle

Self-Fingerprints

4

Overview – Recognition Cloud

Glass

5

Overview – Recognition Cloud

Glass

6

Overview – Recognition Cloud

Glass

7

Challenges

• Perspectives are different

• Clothes have wrinkles

• Lighting conditions change

8

Extracting Fingerprints

Colors

Patterns

Spatiograms

Wavelets

Extracting Fingerprints – Colors

9

RGB HSV

Spatiograms

Color Conversion

Extracting Fingerprints – Colors

RGB HSV

10

color histogram

spatial distribution

Spatiograms

11

Extracting Fingerprints – Patterns

Wavelets

Wavelet sub-bands: vertical, horizontal and diagonal dimensions.

12

Fingerprint Matching Cloud

Glass

Matching Spatiograms

S = {n’, µ’, σ’}

S = {n, µ, σ}cloud

glass

Similarity =

color histograms spatial distributions

13

Fingerprint Matching Cloud

Glass

Matching Wavelets

Bagged Decision Tree (BDT)

W = {f1, f2, f3,…}

W = {f1’, f2’, f3’,…}

cloud

glass

14

Evaluation Setting

• PivotHead glass captured users from the front.

• 15 users was dressed in their regular attires.

• Users actively used their smartphones.

• Phone opportunistically took “profile” pictures of the user.

Evaluation - Matching Color Spatiograms

front

Evaluation - Matching Wavelets of Patterns

front

17

Evaluation – Combining Colors and PatternsEvaluation –

front

18

Evaluation – Performance with Self-Fingerprints

• Matching front view

front

19

Self-fingerprints may not be Sufficient

• Clothes’ difference is not captured when clothes are similar?

• Clothes have different colors/patterns at the back?

Bret

PaulDan

Bret

Refining the Self-Fingerprint (Similar Clothes)

21

Cloud

Glass

DanBob John

refining self-fingerprint

Refining the Self-Fingerprint (Similar Clothes)

22

Dan Paul

Cloud

Glass

Can recognize after refinement!

Refining the Self-Fingerprint

23

Dan

Dan = {F1}

Dan = {F1, F2}

Dan = {F1, F2, F3}

F1 F2

F2 F3

24

Refining the Self-Fingerprint (Back View)

Glass

Cloud

refining back view fingerprint

Bret

25

Evaluation – Performance with Self-Fingerprints

• Matching back view

back

26

Evaluation – Performance with Refined Fingerprints

• Matching back view after refining fingerprints

back

27

Discussion

• Privacy of opportunistic pictures

o User consent before uploading

• Overlapping users in view

o Fingerprint refinement helps

• Incremental deployment with some non-participants

o More time and mobility help

• Cloud vs p2p

o Different trade-offs

28

Conclusion

• Colors and patterns on clothes help fingerprint humans.

• Preliminary evaluation with 15 people provides promising results.

• Other type of fingerprints exists such as motion.

29

29

Looking for interns

Anyone for beer after this talk?

New Primitive for Broadcasting to Visible Vicinity.

Questions, Comments?Thank You