REAL-TIME OBJECT DETECTION AND TRACKING

By:Vanya V. ValindriaHammad Naeem

Rui Hua

Outline• Introduction•Hardware in RT Object Detetion & Tracking •Methods

•Result and Conclusion

Traditional Methods:

Absolute Differences Census Method Feature Based

Method

Modern Methods:

KLT Meanshift

Introduction

Definition:Object detection detect a particular

object in an image

Object tracking to track an object (or multiple objects) over a sequence of images

Application: Traffic Information

http://www.youtube.com/watch?v=vA35sXbn7zs

Application: Surveillance

http://www.youtube.com/watch?v=o25fClk9cdg

Application: Mobile Robot

http://www.youtube.com/watch?v=Q4zycRGJFFs

Problems??

•Temporal variation/dynamic environment

•Abrupt object or camera motion

•Multi-camera? Multi-objects?

•Computational expensive

Hardware in Real-time Tracking

•MEMORYImportant Tracking system encountering limited memory

problems.

•FRAME RATE

~30 FPS

•PROCESSORS - DSP• Allow saturated arithmetic operation• Powerful operation ability• Can do several memory accesses in a single instruction

METHODOLOGIES

Object Detection and Tracking

• In a video sequence an object is said to be in motion, if it is changing its location with respect to its background

•The motion tracking is actually the process of keeping tracks of that moving object in video sequence i.e. position of moving object at certain time etc.

Flow ChartIdle

Imageacquisition

ObjectDetection

Imageacquisition

Objecttracking

ObjectLost? No

Yes

Method 1: Absolute Differences = Image subtraction D(t)=I(ti) – I(tj)

Gives an image frame with changed and unchanged regions

Ideal Case for no motion: I(ti) = I(tj), D(t)=0

Moving objects are detected

Results:

Frame1 Frame10

Difference of Two Frames

Absolute DifferenceMethods for Motion Detection

Frame Differencing Background Subtraction

Draw Backs:

involves a lot of computations

Not feasible for DSP implementation

Method 2: Census Transforms

Signature Vector

Extraction

1

124 74 32

124 64 18

157 116 84

1 1 0

1 x 0

1 1 1

1 1 0

1 x 0

1 1 1

If (Center pixel < Neighbor pixel)

Neighbor pixel = 1

Signature Vector11011101

Signature Vector Generation

List Generation2

Signature vector matching 3

128

26 125

243

87

96 76 43 236

125

128

129

235

229

209

228

251

229

221

234

227

221

35 58 98

Image

Signatur vector generation for all pixels

Signature Vectors1 0 1 1 0 1 0 10 0 1 0 1 0 1 1 . . .1 0 1 1 1 0 1 0

List population

1 0 1 1 0 1 0 1

0 0 1 0 1 0 1 1...1 0 1 1 1 0 1 0Generated List

Census Transform:Advantages:

Compare only two values 0 or 1. Similar Illumination Variation for pixel and

neighbouring pixels

Draw Backs:

As we only deal with only 0`s and 1`s, this method is sensitive to noise.

Calculate, store and match process computationally Expensive

Background estimation

Frame differencing

Object Registration

Method 3: Morphology Based Object Tracking

Morphology Based Object Tracking

Background Estimation

•Image Differencing

•Thresholding

Object Registration

•Contours are registered

•Width, height and histogram are recorded for each contour

Feature Vector

•Each object represented by a feature vector (the length, width, area and histogram of the object)

Tracked object

Segmented object

Morphology Based TechniquesAdvantages:

Can Track Multiple objects Objects are registered based on their anatomy

Helpful for Object Merging

Draw Backs:

Object registration complex and slow process For multiple object registration per frame more

complex

Method 4: Lucas-Kanade Technique• Visual motion pattern of objects and surface in a

scene by Optical Flow

Frame 1 Frame 2

Method 5: Mean shift

• An algorithm that iteratively shifts a data point to the average of data points in its neighborhood

Choose a search window

size in the initial location

Compute the MEAN location in

thesearch window

Center the search window

at the mean

Repeat untilconvergence

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

Intuitive Description

Distribution of identical balls

Region ofinterest

Center ofmass

Objective : Find the densest region

Process

CAMSHIFT --Continously Adaptive Meanshift

Modified to adapt dynamically to the colour probability distributions

More real time

For each frame-> MEAN-SHIFT is applied with several iteration

Store the location of the mean and calculate new window

size for next frame

New development

•Combine with different features. SIFT features, colour feature & texture information

•Camshift algorithm combined with the Kalman filter.

Result

Algorithm

Arithmetic and

Logic operations

Time taken by

Algorithm

AbsoluteDifferencing 4230100 16

Census Transform 2416000 5. 4

MorphologicalTracking 352210 14.2

Kanade Lucas 500825 0.486

Comparison

Absolute Differences

Easy to implement

Allows continuous tracking

Computationally expensive

Slow and low accuracy

Census Transform

Immune to noise and

Illumination changes

Complex if Multiple objects

per frame

Computationally expensive

Feature Based Can track multiple objects well

Large Memory consumption

Slow

Comparison

KLT

High accuracy

Less execution time

Large memory

MeanShift & CAMShift

Ineffective ifthere is

heavyocclusion

Robust to noise and dynamic scene

Computationally less expensive

Conclusion

•KLT algorithm has the best performance with higher accuracy and less computation time

• It requires combination of methods to achieve the appropriate object detection and tracking according to the proposed scenario

References• S. Shah, T. Khattak, M. Farooq, Y. Khawaja, A. Bais, A. Anees, and M. Khan, “Real Time

Object Tracking in a Video Sequence Using a Fixed Point DSP,” Advances in Visual Computing, pp. 879–888.

• K. Huang, L. Wang, T. Tan, and S. Maybank, “A real-time object detecting and tracking system for outdoor night surveillance,” Pattern Recognition, vol. 41, no. 1, pp. 432–444, 2008.

• J. Li, F. Li, and M. Zhang, “A Real-time Detecting and Tracking Method for Moving Objects Based on Color Video,” in 2009 Sixth International Conference on Computer Graphics, Imaging and Visualization. IEEE, 2009, pp. 317–322.

• W. Junqiu and Y. Yagi, “Integrating color and shapetexture features for adaptive real-time object tracking,” IEEE Trans on Image Processing, vol. 17, no. 2, pp. 235–240, 2008.

• Q. Wang and Z. Gao, “Study on a Real-Time Image Object Tracking System,” in Computer Science and Computational Technology, 2008. ISCSCT’08. International Symposium on, vol. 2, 2008.

• Y. Meng, “Agent-based reconfigurable architecture for real-time object tracking,” Journal of Real-Time Image Processing, vol. 4, no. 4, pp. 339–351, 2009.

• [Y. Yao, C. Chen, A. Koschan, and M. Abidi, “Adaptive online camera coordination for multi-camera multi-target surveillance,” Computer Vision and Image Understanding, 2010.