Real-time On-Road Vehicle Real-time On-Road Vehicle Detection With Optical Flows Detection With Optical Flows

and and Haar-like feature detectorHaar-like feature detector

Department of Computer ScienceDepartment of Computer Science,,University of Illinois Urbana-University of Illinois Urbana-

ChampaignChampaign

December 05 2006December 05 2006Jaesik Choi (choi31@uiuc.edu)Jaesik Choi (choi31@uiuc.edu)

I. On-Road Vehicle Detection I. On-Road Vehicle Detection ProblemProblem

Given Given The road images from a moving The road images from a moving

vehiclevehicle (monocular camera) (monocular camera) ProvideProvide

Road boundaryRoad boundary Location of vehicles in 3D spaceLocation of vehicles in 3D space

AssumptionAssumption Paved, flat (no hill) and straight Paved, flat (no hill) and straight

(no curve) road(no curve) road How can we solve it in Real-How can we solve it in Real-

time?time? Coming traffic: Optical Flow Coming traffic: Optical Flow

[Lucas and Kanade 81][Lucas and Kanade 81] Same lane traffic: [Viola & Jones Same lane traffic: [Viola & Jones

03]03] Scene context: [Hoiem 06]Scene context: [Hoiem 06] [Ferryman et al. 2000]

ContentsContents

Problem definitionProblem definitionss On Road Vehicle Detection ProblemOn Road Vehicle Detection Problem

Algorithms & Related worksAlgorithms & Related works Overall DiagramOverall Diagram Detection of cars traveling in the Detection of cars traveling in the opposite directionopposite direction : :

Optical FlowsOptical Flows Detection of cars traveling in the Detection of cars traveling in the same directionsame direction : (Viola : (Viola

& Jones 03)& Jones 03) Estimate 3D geometry (D. Hoiem 06)Estimate 3D geometry (D. Hoiem 06)

Technical detailsTechnical details TimesTimes 3D Project3D Project Filtering with Dynamic Bayes NetFiltering with Dynamic Bayes Net

Demo VideoDemo Video

II. Algorithms – Overall II. Algorithms – Overall DiagramDiagram

Input ImageInput Image

Cars traveling in the same direction

Verify a car with rectangle filters that is learned with

‘AdaBoost’(Viola, etc al 2003)

Cars traveling in the same direction

Verify a car with rectangle filters that is learned with

‘AdaBoost’(Viola, etc al 2003)

Kalman Filter (DBN) & Projection into 3D image

Filter invalid rectangles in the consecutive images

Project each rectangle into 3D image

Kalman Filter (DBN) & Projection into 3D image

Filter invalid rectangles in the consecutive images

Project each rectangle into 3D image

Cars traveling in the opposite direction

Corner detector (Shi & Tomasi )Find optical flows (Lucas Kanade

Optical Flow)

Cars traveling in the opposite direction

Corner detector (Shi & Tomasi )Find optical flows (Lucas Kanade

Optical Flow)

Viewpoint filteringEstimate 3D geometry using vanishing point (D.

Hoiem 06)

Viewpoint filteringEstimate 3D geometry using vanishing point (D.

Hoiem 06)

II. Algorithms – Optical FlowsII. Algorithms – Optical Flows Find corners & correspondence between Find corners & correspondence between

consecutive framesconsecutive frames Cluster the small groups Cluster the small groups

if dist( (locationif dist( (location11, vector, vector11), (location), (location22, vector, vector22 ) ) < ) ) < thresholdthreshold

1 2 3

4 5 6

II. Algorithms - II. Algorithms - Viola & Jones Haar-Viola & Jones Haar-like detectorlike detector

Learn the set of important ‘N’ features from training imagesLearn the set of important ‘N’ features from training images

334 positive car rear images 334 positive car rear images 126 images from ‘Cars 1999 (Rear)2 dataset, CALTECH’ 194 images from ‘CBCL CAR DATABASE, MIT’ other 14 images

500 negative images500 negative images

vs

II. Algorithms - II. Algorithms - Viola & Jones Haar-Viola & Jones Haar-like detectorlike detector

Set of rectangle featuresSet of rectangle features Given an image (Hypothesis):Given an image (Hypothesis):

Verify the image (Hypothesis) with rectangle Verify the image (Hypothesis) with rectangle features that is trained with AdaBoost from features that is trained with AdaBoost from training imagestraining images

Feature 1:Feature 1:

Feature 2:Feature 2:

……

-

- > 2

> 1

II. Algorithms - II. Algorithms - Viola & Jones Haar-Viola & Jones Haar-like detectorlike detector

PerformancePerformance Training setTraining set

Number of detect image / all car images = 88.63%Number of detect image / all car images = 88.63% Number of missed image / all car images = 11.37%Number of missed image / all car images = 11.37%

Test set Test set (516 images Cars 2001 (Rear), Caltech)) Number of detect image / all car images = 90.50%Number of detect image / all car images = 90.50% Number of detect image / all car images = 9.50%Number of detect image / all car images = 9.50%

III. Algorithms – Estimate 3D III. Algorithms – Estimate 3D GeometryGeometry

Estimate 3D Geometry Estimate 3D Geometry (D. Hoiem, 06)(D. Hoiem, 06)

P( WindowP( Windowwidthwidth| Dist| Distyy ) ? ) ? WindowWindowwidthwidth: The width of a window of a car: The width of a window of a car DistDistY Y : The vertical distance from the vanishing point: The vertical distance from the vanishing point

IV. Technical Details - TimesIV. Technical Details - Times

Real-time issues – execution timesReal-time issues – execution times

3GHz 2CPU, 1GB Desktop -> 11.36 frames/sec

IV. Technical Details – 3D IV. Technical Details – 3D projectionprojection

Project into 3D Image based on the Probabilistic Project into 3D Image based on the Probabilistic estimate of 3D Geometryestimate of 3D Geometry

h

dist = h * tan( + n *∆ )

IV. Technical Details IV. Technical Details ((Filter outliers using Dynamic Bayes Net)Filter outliers using Dynamic Bayes Net)

Estimate the correct locations of cars using Estimate the correct locations of cars using Dynamic Bayes NetDynamic Bayes Net

P( | )

P( | )

vs

P(Modelt+1|Modelt)= P(Colort+1|Colort) * P(Sizet+1|Sizet) * P(Locationt+1|Locationt) * P(Sizet+1|Locationt+1)

V. Demo Movies V. Demo Movies

ReferenceReference [Betke et al. 2000] M. Betke, E. Haritaglu and L. Davis, “Real-time

multiple vehicle detection and tracking from a moving vehicle,” Machine Vision and Applications, vol. 12, no. 2, 2000.

[Sun et al. 2002] Z. Sun, R. Miller, G. Bebis, and D. DiMeo, “A real-time precrash vehicle detection system,” IEEE International Workshop on Application of Computer Vision, Dec., 2002.

[Wedel et al. 2006] A. Wedel, U. Franke, J. Klappstein, T. Brox, and D. Cremers, “Realtime Depth Estimation and Obstacle Detection from Monocular Video”, DAGM 2006, LNCS 4174, pp. 475-484, 2006.

[Ferryman 1998] J. M. Ferryman, S. J. Maybank, and A. D. Worrall, "Visual surveillance for moving vehicles," Int. J. Comput. Vis., vol. 37, no. 2, pp. 187--197, June 2000.

[Shi and Tomasi 94] Jianbo Shi and Carlo Tomasi. “Good Features to Track,” IEEE Conference on Computer Vision and Pattern Recognition, pages 593-600, 1994.

[Nishigaki 00] [Nishigaki 00] M. Nishigaki, M. Saka, T. Aoki, H. Yuhara, and M. M. Nishigaki, M. Saka, T. Aoki, H. Yuhara, and M. Kawai, Kawai, “Fail“Fail Ouput Algorithm of Vision Sensing,”Ouput Algorithm of Vision Sensing,” Proc. IEEE Proc. IEEE Intelligent VehicleIntelligent Vehicle Symp., pp. 581-584, 2000.Symp., pp. 581-584, 2000.

[Sun et al. 2006] [Sun et al. 2006] Z. Sun, G. Bebis, and R. Miller, ““On-Road Vehicle On-Road Vehicle Detection: A ReviewDetection: A Review”,”, IEEE Trans. Pattern AnalIEEE Trans. Pattern Analysis andysis and Mach. Mach. IntellIntelligence, vol. 28, no. 5, 2006.igence, vol. 28, no. 5, 2006.