Slide 1

Human-Debugging of Person DetectionDevi Parikh and Larry Zitnick

Computer Vision: Visual Recognition2

STREETCARWINDOWSScene recognition

Object recognition

Object detection

Computer Vision: Visual Recognition3

STREETCARWINDOWSScene recognition

Object recognition

Object detection

Segmentation

State of Machine Visual RecognitionObject RecognitionScene RecognitionObject DetectionSegmentation 4AccuracyMachineHuman5AccuracyMachineHuman Complex systems: lot of progress

Where do we head next?State of Machine Visual RecognitionHuman-Debugging5OutlineWeakest Link in Person Detection

Challenges

ConcludePerson DetectionUsefulChallengingWeve come a long way

Fischler and Elschlager, 1973

Dollar et al., BMVC 2009Still a long way to go

Dollar et al., BMVC 2009

Dollar et al., BMVC 2009

Dollar et al., BMVC 2009Person Detection SystemFeature selection

Felzenszwalb et al., 2005

Hoiem et al., 2006

ColorIntensityEdgesPart detectionSpatial modelNMS / contextHuman-DebuggingFeature selectionPart detectionSpatial modelNMS / contextFeature selectionSpatial modelNMS / context

Feature selectionPart detectionNMS / context

Feature selectionPart detection

Human-DebuggingSMMachineHumanFPNMS

Amazon Mechanical TurkHuman-Debugging15

SMFPNMS

Human-Debugging16

SMFPNMSHuman-Debugging17

SMFPNMSHuman-Debugging18

SMFPNMSHuman-Debugging19

SMFPNMSHuman-Debugging20SMFPNMS

Human-Debugging21SMFPNMS

Human-Debugging22SMFPNMS

Human-Debugging23

SMFPNMSHuman-Debugging24SMFPNMS

PersonNot a personHuman-Debugging25

SMFPNMS

Part Patch Dataset

Part Patch Dataset27

Part Patch DatasetHuman face detection

Part Patch DatasetHuman torso detection

Part Patch DatasetHuman arm detection

Part Patch DatasetHuman hand detection

Part Patch DatasetHuman leg detection

Part Patch DatasetHuman foot detection

Part Patch Dataset

0.3 million patches

10 subjects

6 Parts and RootRegular, Gray-scale, Normalized-gradientHigh and Low resolution

Available online!Parts-based Object Detectors35SMMachineHumanFPNMSResults (P)36

Part Visualizations37

Results (SM)38

Results (NMS)39

Results40

Classify 24x24 patches into one of 6 part categories (+background)Effect of Features (F)

Effect of Context

Sources of Error

Even when subjects are shown the entire image, highly occluded people in bad lighting are missed.43Sources of Error

When subjects classify windows in isolation from the rest of the image as containing a person or not, lack of context leads to false positives when the windows locally appear to have parts of a person44Sources of Error

A machine spatial model applied to near-perfect human part-detections fails because of symmetric part detections. Subjects were asked to classify patches as containing arms, legs, etc. and were not asked to distinguish between left/right arms, legs, etc.45Human-DebuggingImportance of each componentPotential of pipeline as a whole

46Challenges47ChallengesAccessing isolated human models

Visualizing high-dimensional data

Invoking natural visual pathways

48[Chernoff, 1973]AccuracyMachineHumanHumans are a working system! Interactive Figure our brains out Label training data Design algorithms Debug our systems!Conclusion4949Thank you!