Active Pedestrian Safety: from Research to Reality

Dariu M. GavrilaEnvironment Perception

Research and Development

Oxford University, 04-10-2013

2

We originally thought Machine Intelligence would look like

1956 "Forbidden Planet"

Robby the Robot (Flickr)

3

Then more recently, some suggested it would be more like

2004 iRobot 1983-2009 Terminator

when in fact, Machine Intelligence is already with us, and has a familiar embodiement …

The new

Mercedes-Benz S Class (2013)

4

360°CAMERA

4 m range

Sensors and Coverage (Mercedes-Benz E- and S-Class)

5

6

Driver Assistance at Mercedes Benz (E- and S-Class, 2013)

Adaptive High Beam

Pre-Crash Braking (longitudinal & lateral traffic)

(Active) Lane Keeping

Nightview AttentionTraffic Signs

Parking Adapt. Cruise Control w. Steering Assist

(Active) Body ControlS-Class only

?

7

Challenge: Active Pedestrian Safety

8

Worldwide Traffic Fatalities 2010: Share of Vulnerable Road Users

Illustration: Bosch

9

Why is it difficult?

Large variation in pedestrian appearance (viewpoint, pose, clothes).

Dynamic and cluttered backgrounds.

Pedestrians can exhibit highly irregular motion.

Real-time processing required.

Stringent performance requirements (especially for emergency maneuvres).

10

Pedestrian Recognition Today

Pre-crashSurroundview Nightview

Various requirements with respect to sensor coverage, vehicle speed range,

recognition performance

Different HMI and vehicle control strategies

11

EU WATCH-OVER (2008)

85%50 km/h

10

Pedestrian Recognition Performance Last Decade (Downtown)

Correctly recognized

pedestrians

Number of falsely recognized pedestrian trajectories per hour

100%

50%

10000 100

EU SAVE-U (2005)

65%40 km/h

EU PROTECTOR (2003)

40%

600

30 km/h

N.B. # False alarms per hour << # Falsely recognized trajectories per hour

DE AKTIV-SFR (2010)

90%

1212

13

Better Algorithms: System Architecture

Pedestrian

Classification

Pedestrian

ClassificationTracking / Fusion

Driver Warning /

Vehicle Control

Driver Warning /

Vehicle Control

Path Prediction &

Risk Assessment

Path Prediction &

Risk Assessment

Regions of Interest(Stereo, Motion, Geometry)

D. M. Gavrila and S. Munder, „Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle”. Int. J. Computer Vision, vol. 73, 2007

S. Munder, C. Schnörr and D. M. Gavrila. „Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models.”

IEEE Transactions on Intelligent Transportation Systems, vol. 9, nr. 2, 2008

Better Algorithms: Classifier Fusion

M. Enzweiler and D. M. Gavrila.

„A Multi-Level Mixture-of-Experts Framework for Pedestrian

Classification'‘. IEEE Trans. on Image Processing, vol. 20,

nr. 10, 2011

Performance boost over HOG/linSVM: Factor 42 less false positives

14

15

Dense Stereo: Better ROIs, Classification and Localization

C. Keller, M. Enzweiler, M. Rohrbach,

D.-F. Llorca, C. Schnörr, and D.M.

Gavrila. „The Benefits of Dense

Stereo for Pedestrian Detection.“

IEEE Trans. on Intelligent

Transportation Systems, 2011.

1616

17

Better Processors and Sensors

Processing Power over Time

time

GFLOPS/MIPS

1990 2000 2010

100

200

300

400

500

ASIC

FPGA (Xilinx)

GPU (NVidia)

CPU (Intel)

Transputer/x86 P4

Core2Duo

G92

G80

G70

NV40

Virtex 5

Spartan3

Virtex 4

Tyzx

IQ2

Standford engine

3DIP

Processing Power over Time

time

GFLOPS/MIPS

1990 2000 2010

100

200

300

400

500

ASIC

FPGA (Xilinx)

GPU (NVidia)

CPU (Intel)

Transputer/x86 P4

Core2Duo

G92

G80

G70

NV40

Virtex 5

Spartan3

Virtex 4

Tyzx

IQ2

Standford engine

3DIP

Intel

Core i7

100 W

Xilinx

Spartan

5 W

Processing on embedded hardware

(e.g. FPGA) for real-time performance

Shape-based Object DetectionStefan Hezel, A Kugel, R Manner, DM Gavrila

FPGA-based template matching using distance transforms

Proc. IEEE Symp. on Field-Prog. Custom Comp. Machines, 2002

Dense Stereo ComputationS. Gehrig, F. Eberli and T. Mayer.

A Real-Time Low-Power Stereo Vision Engine using Semi-Global

Matching, Computer Vision Systems, 2009

Pedestrian Classification

1818

19

Training Data Size Matters (a Lot) …

ROC performance

improves with enlarged

training set

Labeling pedestrians in images is time-consuming and tedious.

Intelligent tools for the acquisition and enrichment of large databases.

20

Lots of Real Data …

Ulm

Stuttgart

München

Aachen

Amsterdam

Parma

Brussels

Paris

Barcelona

San Francisco

Tokyo

…

O(107) images

O(106) labels

21

… and Some Artificial Data Too („Virtual“ Pedestrians)

Shape

variation

Texture

variation

M. Enzweiler and D. M. Gavrila. „A Mixed Generative-Discriminative Framework for Pedestrian Classification”. CVPR 2008.

22

1. Mono Pedestrian ClassificationS. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification.

IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006.

2. Multi-Modal / Occluded Pedestrian ClassificationM. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with

Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.

3. Mono/Stereo Pedestrian DetectionM. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments.

IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.2179-2195, 2009.

C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection.

Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany, 2011.

23

Daimler Pedestrian Benchmark Data Sets

Training: 14400 peds. / 15000 non-peds.

Test: 9600 peds. / 10000 non-peds. All 18x36 pixel.

Available for download

(Search web)

Training: 15660 peds. / 6744 non-ped images

Test: 21790 images with 259 ped. trajectories

>130.000 samples (intensity, dense stereo,

dense flow), 48x96 pixel

1. 2. 3.

Other benchmarks by CalTech (2009), KITTI (2012), TUD (2009) and UAB (2010)

24

… use a “Dummy” Car!

On-board sensors

Safety cables

How to perform realistic pre-crash tests?

25

Sensor View

2626

27

Driver Assistance at Mercedes Benz (E- and S-Class, 2013)

Adaptive High Beam

Pre-Crash Braking (longitudinal & lateral traffic)

(Active) Lane Keeping

Nightview AttentionTraffic Signs

Parking Adapt. Cruise Control w. Steering Assist

(Active) Body ControlS-Class only

?with Pedestrian Recognition

28

2929

30

Quo Vadis, Pedestrian?

Pedestrian path prediction and

action classification for short

prediction horizons (< 1s)

Probabilistic matching of learned

trajectories

Use of dense stereo / flow

in addition to position

C. G. Keller, C. Hermes and D. M. Gavrila.

„Will the pedestrian cross? Probabilistic

Path Prediction based on Learned Motion

Features”. German Pattern Recognition

Conference (DAGM) 2011 Prize

31

Action Classification (Crossing or not)

1 Frame ≈ 45ms

C. G. Keller and D. M. Gavrila. „Will the pedestrian cross? A Study on Pedestrian Path Prediction”.

IEEE Transactions on Intelligent Transportation Systems, 2013, DOI 10.1109/TITS.2013.2280766

Initiating emergency braking

180 ms earlier reduces

chances for hospital stay by

15% (TTC = 0.7s, vehicle

speed = 50 km/h)

Predicting the correct pedestrian’s action with accuracy 80% is

reached • 570 ms (human)

• 230 ms (Prob. Traj. Matching, Gaussian Process Dynamical Models)

• 90 ms (Interacting Multiple Models KF)

before a possible standstill. Motion features help.

PedCut: An EM-like framework for pedestrian segmentation

combining global shape models and multiple data cues

32

F. Flohr and D. M. Gavrila. „PedCut: an iterative framework for pedestrian segmentation

combining shape models and multiple data cues”. BMVC 2013.

Initial contour Final contour

New public dataset with multi-cue

(color, dense stereo) pedestrian

cut-outs captured from a vehicle

33

Automatic Braking vs. Automatic Evasion

C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D. M. Gavrila. „Active Pedestrian Safety by Automatic

Braking and Evasive Steering”. IEEE Trans. on Intelligent Transportation Systems, vol. 12, nr. 2, 2011

300 ms from first sight

of pedestrian to initiation

of vehicle maneuver

(braking or evasion)

Final Remarks

Dramatic progress on vision-based pedestrian sensing, mirroring the success of

computer vision technology in driver assistance over the past decade.

Perseverance pays off.

Challenges still remain for pedestrian sensing.

Additional focus: data fusion, situation understanding, new actuation concepts.

Accident analysis at Mercedes-Benz indicates that the currently deployed

pedestrian recognition technology could avoid 6 percent of pedestrian accidents

and reduce the severity of a further 41 percent.34

1999 2013

35

Better Algorithms + Market-Ready Hardware + More Data + Extensive Testing =

More Lives Saved.

36

Thank you

Credits

Markus Enzweiler, Christoph Keller, Stefan Munder, Jan Giebel,

Fabian Flohr and others …