Presentation of Visual Tracking

Presentation of Personal Work

Yosen Chen

Yosen Chen, Copyright Reserved

Human Machine Interaction

Basic Visual Tracker

Multi-Cue Particle Filter Tracker

Consistent Labeling in Visual Surveillance

System

Multi-Human Tracker

Multi-Camera Multi-Human Tracker

Consistent Labeling with People Re-Entering

Problem

Outline

2

Implementation Tool

Microsoft Visual C++ with OpenCV Library

IPC (Interprocess Communication) Library

open source @ Carnegie Mellon University

Yosen Chen, Copyright Reserved 3






System

Multi-Human Tracker



Problem

Outline

4

Intelligent Interactive Engine

Image processing

Webcam

Image data

Show message

& animationDisplay Central

Server

Webcam

Our goal is to construct an interactive engine which utilizes target motion as instruction input to the server.







System

Multi-Human Tracker



Problem

Outline

6

Basic Visual Tracking Framework

(Color)


Camera Real-time

image

Feature Extraction

RGB

to

HSV

Hue

Saturation

Value

256-to-16

color Qtz

Similarity Evaluation

Reference

Image

Reference

16-color Histogram

Hue

Saturation

Value

Reference Color

Backprojection Probability

distribution

figure

Probability Peak

Search/Climbing

Target Information

Estimation

(Position, size,...)

If iterative

Reference

16-color Histogram

Reference Color

Backprojection

Input Hue

Image

Probability

distribution

figure

Color

similarity

score

Iterative Search

(Motion Prediction)

Previous

Current

Next

Camera View

Region of Interest

7

Demo Video: Ball Tracking

追球後段


Peak climbing: MeanShift

Related Works

Varieties of Feature Extraction/Evaluation

Feature evaluation: template matching

Background subtraction

Reference color model: YCrCb, RGB

Varieties of Peak Search/Climbing

Gravity center

Max-Likelihood

MeanShift


Cb

Cr

9

Summary

Advantages:

Simple & Fast low-cost

Each pixel (RGBHSV, backprojection) is processed

independently good for parallel processing

Drawbacks:

Hard to combine with more object features

EX: edge information (object shape, contour)

All object features 2D probability distribution figure

Hard to cope with higher-dimension target state

EX: object orientation (rotation angle)


??suitable

10






System

Multi-Human Tracker



Problem

Outline

11

SIR+SIS Particle Filter Tracking


State candidates

(Particle set) Color

Possible region

sampling

SIR

Real-time

State Candidate

Prediction

Frame

Multi-Cue Candidate

Evaluation

1

m m

t tp X X

1 1, , ,

1,...,

mm

t tX x y r

m N

m

t tp O X State Candidate

Resampling

,

equal weight

m m

t t t

k

t

X p O X

X

tO

SIS

Shape

Motion

weighting

Particles from last candidate set

Particles generated

from current image

Tracking result

on Frame tO

m

tX

Best State Candidate

Sequential Importance Sampling (SIS) Particle Filter

Sampling Importance Resampling (SIR) Particle Filter

Color backproject

+ filtering

+ best contour finding

Initial State Estimation State Prediction State Evaluation

Find Best Candidate

Resampling Last Candidates

based on weights

State Prediction State Evaluation

Find Best Candidate

Last candidate set {state candidate, weight}

12

Multi-Cue Likelihood Evaluation


For each state

candidate

Inner color extractionColor

Similarity

Correlation-based

comparison

Color weight

Reference color modelColor

Multi-Cue Candidate

Evaluation

m

t tp O XShape

Motion

weighting

m

tX

tO

inner

outer

High

continuityLow

continuity

id

maxd

ig

, , ,mm

t tX x y r

Shape

Similarity

Motion

continuity

(logic check)

outer color extraction

Shape distance

State difference with

last tracking result

id

Ih i

Oh i

REFh i

Edge strength ig

Max matched size 1 null

MAX

Nr

r N

Symmetry 1i N id d

, , ,x y r

Shape weight

Motion weight

Overall

candidate score m

t tp O X

13

Demo Video: 4D Bottle Tracking

(x, y, r, θ)


Evaluate new features

14

Candidate group behavior

Bottle tracking video

Blue: SIS (from current color projection) Green: SIR (from last candidate set) Red: Best candidate as tracking result


Interaction

System Interaction

Triggering

Object

Checking

Tracking

& Interaction

Losing

Handling

Start

Motion

detection by

Optical flow

Image

Initialize/Reset

all parameters

Motion

detected?

Show Checking

Box

Record target

information and

features

Target Info is

robust?

Tracking

algorithm

Quality index

> threshold ?

Wait until user

puts target in

“Losing Box”

Is target put

back?

Visual reality

interacting

program

image

Yes

No Yes

Yes

No

No

Yes

No

Image

Demo Video:

Interactive Game – Brick Breaker


Bottle tracking 資策會計畫兩段影片

Interaction system: 4 status switching 4 degrees of freedom: location x & y, size, rotation angle

16

Motion Interaction Function Interactive Game Example

Summary

High flexibility

State dimension

Motion model

Easy to add new features & tune weights in tracker

for different applications and scenarios

Trade-off:

Feature description vs. Computation time

Impoverishment(intensive) vs. Degeneracy(sparse)

Many particles concentrate on high-probability region

Many particles are evaluated at low-probability region

Number of particles ～ Complexity







System

Multi-Human Tracker



Problem

Outline

18

Related applications:

Visual surveillance

Remote monitoring

Intelligent security system

Introduction

19 Yosen Chen, Copyright Reserved






System

Multi-Human Tracker



Problem

Outline

20

Multi-Human Tracking Flow


Prediction Evaluation Resampling



…

State joint evaluation

SIR Head tracking

Multi-human

tracking result

Real-time

captured image

Skin color

blob finding

SIS Face detection

Shape &

location filter

ROI location

Sampling

Face detector

(OpenCV)

Losing Track

Detector

(score,

counter)

Delete tracker

Create new tracker

1. SIS face detector SIR human trackers 2. Multi-human state joint evaluation 3. Bad evaluated tracker losing track

detector face detector delete tracker

21

Multi-Cue Likelihood Evaluation

for Head Tracking


High color likelihood Low color likelihood

outer

inner

Skin color mask

Hair color mask

outer

inner

Skin color mask

Hair color mask

1

, ,t

x y r

, ,t

x y r

1

, ,t

x y r

, ,t

x y r

High continuity Low continuity

1n2n

6n

9n

13n

Shape distance1n

2n

6n

9n

13n

Num of Matched point

ig

Edge strength 1n2n

6n

9n

13n

symmetry

Color •Masked skin color comparison •Masked hair color comparison

Texture •Texture SAD similarity

Edge •Shape distance •Number of matched point •Edge strength •Contour symmetry fitting

Motion •State continuity

22

Face Detection Implementation Details


SIS Face Detection Flow =======================Pseudo code======================== cvCalcBackProject; cvEqualizeHist + cvThreshold; cvResize + cvErode + cvDilate; cvFindContours(LinkedList detect_contour); for( ; detect_contour != 0 ; detect_contour = detect_contour->next ) { if (Area < threshold) continue; Use 8 points average to approximate center (xA, yA); if (distance between all target T and (xA, yA) > threshold) { calculate geometric center (xC, yC), deviation σX, σY of detect_contour; if ((σX > 0.7*σY) && (σY > 0.8*σX) && (σX /sqrt(Area) < 0.3) && (σY /sqrt(Area) < 0.4)) { Draw k random samples (x, y) ~ N((xC, yC), (σX, σY)); where k = (Area/MIN_AREA). if (frontalFaceTest(x, y)->detect() || profileFaceTest(x, y)->detect()) return location (x, y) as start point of new tracker; } } } return NOT_DETECTED;

Skin color

blob finding

SIS Face detection

Shape &

location filter

ROI location

Sampling

Face detector

(OpenCV)

Losing Track

Detector

Create tracker

Delete tracker

23

Face Detection Demo


Skin color backproject

Skin color

blob finding

SIS Face detection

Shape &

location filter

ROI location

Sampling

Face detector

(OpenCV)

Losing Track

Detector

Create tracker

Delete tracker

24 Skin color backproject (down-size & morphology)

Last detecting region

Blue region: Last detected face

Red region: color is matched with skin color. Tracking System GUI panel

People Overlapping Problem


When people get close to each other…

State candidates are attracted by other people because trackers don’t know each other!!

PredictionEvaluation

(individual)Resampling

Original Head Trackers

25

Solutions of People Overlapping


Prediction

Evaluation

(Individual

+ joint)

ResamplingJoint state

check

Depth order

check

Body occlusion

handling

Target information

storage

Modified Head Trackers for People Overlapping

For people re-entering

problem

1. Use joint evaluation to avoid candidates merging for people overlapping 2. Solution#1 (left): give higher joint score for candidates further from other people

reduce risk of candidates merging but also lose tracking accuracy 3. Solution#2 (right): kill candidates too close to other people better tracking accuracy

26

版本，

resVideo_FurtherIsGood2.mpg resVideo_CloseThenLowWeight2.mpg

for body information extraction (Details will

Implementation


Prediction

Evaluation

(Individual

+ joint)


check

Depth order

check


Image distance check “if △x2 + △y2 < threshold, …” è group together

target1 target2 target3 target4 target5

Grouping (2nd-order tree)

root leaf leafleaf

Target A

candidate 1

(far from B)

Target B

last best

candidate

Target A

candidate 2

(near B)

Pointer to

target1

Candidate Evaluation (individual + joint)

Individual: (as previous mentioned)

Color, shape, motion

Joint: (not be attracted by other close targets)

Distances to all other targets in the same group

Pointer to

target2………

Depth order: deform 2nd-tree into array, then quick sort by depth features

Sort by depth

features

Depth features:For best candidates of each target

Texture changing rate

Contour matching score

Distance from camera to target ground points

Image source

1:Front

2:Rear

Candidate A2 has high penalty

(since Candidate A2 are too close to B !!)

27

Depth Order Evaluation with Heights

Depth order estimation with height knowledge is accurate Even when people are not overlapping.


Summary of Multi-Human Tracker

We extend the concept of single-object tracking to multi-object tracker

We propose a entire tracking system Face detection multi-human trackers detect

people leaving

Address the solution of people overlapping

Robust multi-human tracker is indeed the basis of correspondence process within multi-camera tracking system







System

Multi-Human Tracker



Problem

Outline

30

Multi-Camera Tracking System


Captured image

Local camera #A

multi-head tracker

1 2,A B

t tz z2tH

1

A

tz 2

A

tz

2

B

tz1

B

tzLocal camera #B

multi-head tracker

Captured image

Central Server

Multi-camera multi-observation

correspondence

Person#1 Person#2

1 2 1,A B

t tz ztH

Identify the relationship of observations

in different cameras

Local trackers: multi-human tracking and occlusion handling

Central server: observation correspondence across cameras

31

: image observation

: 3D line from image head center

j

rt

j

rt

O

L

“Are these cameras tracking the same people? ”

Let be the rth observation tracked by camera at time

Problem of Observation Correspondence

across Cameras

,j j j

rt rt rtO Lz j t

How many

people exist in

surveillance

area?

Ground plane

Camera jCamera k

1

j

tO2

j

tO

3

j

tO1

k

tO2

k

tO

3

k

tO

1

j

tL

2

j

tL

3

j

tL1

k

tL

2

k

tL

3

k

tL


Determine observation correspondence using Multi-Feature Similarity

Multi-Feature Observation Similarity contains:

Head line intersection verification

Body color matching by blocks

Confidence level

Label history consistency

How to Determine Observation

Correspondence across Cameras

Correspondence Flow

Calculate 3D

geometric intersection

Confidence

level

Are they from the

same person?

Camera#A Camera#B

Body Color

comparison by block

Label history

consistency

2D Body image

block allocation

No, if no reasonable

intersectionNo, if color unmatched

Yes, 2 observations

are from the same

person


2D Position on image plane 3D head line in real space

3D ground point in real space 2D position on image plane

observation r

Optic center Image plane of

camera j

Ground plane

Optic center

observation q

Image plane of

camera k

3D Geometric Correspondence across

Cameras

Calibrated camera system Image 2D↔3D coordinate transfer is constructed off-line

Head Point

height

Estimated Ground Point


How to Implement 3D-2D Mapping


Class: Plane3D Member functions:

isPointOnPlane getDistFromPoint getProjPointFromPoint

Class: line3D Member functions:

getPositionFromX,Y,Z isPointOnLine getDistFromPoint getProjPointFromPoint getLinePointOnPlane getMinDistPointFromLine

Class: CameraLine3D Member functions:

setByImageLocation setBy3DPosition

How to verify the mapping accuracy?

3D Optic point

(≈ Camera location)Class: CameraLine3D

2D image

plane

Reference plane

in 3D space

3D Normal vector

Reference 3D

plane center

(0,0)

derived

35

CAM2 CAM1

Locate 3D points/lines on 2D image planes

Human Body Model Based on Head Tracking

Matching body color

Human Body is separated into

4 blocks

ground-point

head

ground-point

head

Image

boundary

Observed ground-point Unobserved ground-point

j

rtzk

qtzCamera kCamera j

qtrt

3D Geometric-based Body Image Block

Allocation

Yosen Chen, Copyright Reserved 36 For some observations, not all the body blocks are unobservable. Here we take the number of observable blocks as confidence level of correspondence result

Observation Correspondence across Cameras

Object Conf = 3

Object Conf = 2

Matching body color (Region-based body color comparison)

Depth order: Front

Depth order: Rear Body Occlusion Handling (based on depth order and 3D-2D block mapping )


Body appearance is available for

correspondence process Body appearance is not available (Color turns darker)

Body Occlusion Handling


Prediction

Evaluation

(Individual

+ joint)


check

Depth order

check

Body occlusion

handling

Target information

storage


Labeling Consistency with Confidence Level


Observation correspondence process should follow rules stated below Rule1. Claiming “match” for one pair will also imply claiming “collision” for all

corresponding residual pairs, with the same confidence level Rule2. Only higher-confidence decisions can enforce changes of lower-confidence ones

√ √ √

A B C

√ 1 0 0 -4

√ 2 -4 -4 4

√ 3 0 0 -4

√ √ X

A B C

√ 1 0 0 0

√ 2 -4 -4 0

√ 3 0 0 0

√ √

A B

√ 1 -3 0

√ 2 -4 -4

√ 3 3 -3

Camera#1

Camera#2

Conf = 4Conf = -4

1 23

AB

C

Camera#1

Camera#2

Conf = -4

1 23

AB

Camera#1

Camera#2

Conf = 3

Conf = -4

Conf = -3

1 23

AB 39






System

Multi-Human Tracker



Problem

Outline

40



Captured image

Local camera #A

multi-head tracker

1 2,A B

t tz z2tH

1

A

tz 2

A

tz

2

B

tz1

B

tzLocal camera #B

multi-head tracker

Captured image

Central Server

Multi-camera multi-observation

correspondence

Person#1 Person#2

1 2 1,A B

t tz ztH

Identify the relationship of observations

in different cameras

3tX2tX1tX

associate with target databases

41

Association Processes with Matching Confidence Orders

Hierarchical Association with target databases

Objects is associated to target databases

1

m

ht hH

1

n

it iX

Consistent Labeling Using Target Database

3tX

2tX

1tX

4tX

1tH

i

TARL

High

Low

1

OBJL1

TARL

2

TARL

3

TARL

4

TARL

compare with

descendent order

1 2 3

TAR TAR TARL L L

: target confidence level (in database)

: object confidence level (from observation)

i

TAR

i

OBJ

L

L

if association success update related database else create new database PS: database association process is only for objects with conf. level ≧2


Association Process

Association process:

To determine if observed object and database are from the same

person

Features are similar to observation correspondence process

People height

Body color comparison by blocks

Association history consistency


Observation-Database Association Flow:

Check height

difference

Are they from the same person?

Body Color

comparison by block

Association history

consistency

No, if diff > threshold No, if color unmatched

Yes, observations &

this database set are

from the same person

2tX

1tHFrom Camera

observations

People information

already stored in

databases 43

How to Model & Update Database

Adaptive Gaussian Mixture Model

3 color candidates (histogram & weight) to describe 1 body block

color

Candidate’s histograms & weights are updated based on current

color of MATCHED observations

Color comparison with observed color (Define MATCHED or not)

Bhattacharyya distance from 3 candidates

3 weighted Gaussian kernels


Color histogram (1-D for demo)

Candidate

weight

Candidate color

Adaptive Gaussian Mixture Model

Candidate#1 Candidate#3 Candidate#2

Candidate color & weight will be updated based on observations

Candidate

variance

Inspired by

[Stauffer., CVPR 2000]

Database Operation Demonstration


Histogram candidates Camera#2 histogram Camera#1 histogram

Matched with camera#2

Matched with camera#1



Database Demonstration (cont’)

46 Yosen Chen, Copyright Reserved

Multi-People Correspondence


1. Multi-people depth order 2. body occlusion handling 3. Multi-camera multi-people correspondence is correct

Multi-People Re-entering


System Operation Flow: 1. Observation correspondence & Body occlusion handling 2. Associate with current databases (Create then) update new information 3. Return people IDs (show by color) to each tracker of each camera

Future Works…

Extend to the 3rd Camera Challenge of communication load, algorithm complexity, and

tracking/labeling stability

Performance Evaluation of Target Database

Comparison with related works

More challenges:

Outdoors Surveillance (human body detection)

Uncalibrated multi-camera system (Training)

Motive camera (rotation 3D-2D mapping)


Thank You

BACKUP SLIDES


Motion Detection : Optical Flow

(Modified from OpenCV Sample Code)

Input gray-scale

image

Optical point

extractionOptical point

mapping & filtering

Motion

detection criteria

Reset criteria

Interaction

trigger

Optical flow

Reset trig

ger

Previous gray-scale

image

Store

Optical Flow: 1. A technique to create edge points then map them between adjacent images.

(edge tracking) 2. Better to analyze group behavior rather than individual (not accurate) 3. Can be utilized as motion detection tool

1. Not enough effective optical points on screen

2. Time out then refresh

1. Average velocity of moving points > threshold 2. Number of moving points > threshold Moving points: points with velocity > 3*average velocity


BACKUP SLIDES END


Presentation of Visual Tracking

Technology

Transcript of Presentation of Visual Tracking