Object Recognition & Model Based Tracking © Danica Kragic Tracking system.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Tracking systemTracking system

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

MotivationMotivation

Manipulating objects in domestic environments

Localization / NavigationObject RecognitionServoing TrackingGrasping Pose estimation

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

StepsSteps

Recognition (2D)Tracking (2D)Pose estimation (3D) :

Initial pose estimation

Where in the image … ?

Where in the world … ?

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Initial Pose EstimationInitial Pose Estimation

Recognition/Tracking Pose estimation (x,y) (X,Y,Z, , , )

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Example ObjectsExample Objects

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

CharacteristicsCharacteristics

Simple geometry (polyhedra, cones,

cylinders)Specular surfacesBackgroundIlluminationSlippery objects

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

CharacteristicsCharacteristics

Simple geometry wireframe modelsSpecular surfaces - ll – Illumination - ll – Background - ll – Highly texture appearanceSlippery objects power grasps

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Model Based Techniques

Model Based Techniques

Appearance based methods Geometry based methods

3D wireframe modelsComplete pose estimationTechniques from computer graphics used for rendering

FUSION!

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Object RecognitionObject Recognition

Removes background, preserves object.Necessary to raise the signal to noise ratio, for the pose estimatior.Solved using color cooccurrence histograms.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Pose EstimationPose Estimation

An apperance based method is used to recognize the object, and estimate an initial pose.A geometric model based method is used to obtain an accurate pose.Algorithm combines the robustness of appearance based methods with the accuracy of feature based methods.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Color Cooccurrence Histograms

Color Cooccurrence Histograms

Apperance based method.Based on color cues only.Superior to standard color histograms.Invariant to translation and rotation.Robust towards scale changes.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Building Color Cooccurrence Histograms


All pairs of pixels within a certain radius contribute to the histogram.Example: 4x4 image with 3 colors, and a maximum radius of 3 pixels.

Histogram:

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic



When all pairs have been counted, the histogram is normalized. Each bin is divided with the total number of pixel pairs.

Histogram:

50 %

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Color Cooccurrence Histograms - Matching

Color Cooccurrence Histograms - Matching

A common histogram matching method is used.

N

i

ihihMatch1

21 ])[],[min(

Reduces the effect of background noise, as unexpected colors will not penalize the match value.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Color QuantizationColor Quantization

Before the histogram can be built, the colors in the image need to be quantized. This is done using k-means clustering.

Red

Green

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Color QuantizationColor Quantization

Images are normalized prior to quantization, in order to decrease the effect of varying lighting conditions.Only the red and green components are preserved.

Performance equal to RGB

and HSV.

Red

Green

bgr

rrnorm

bgr

ggnorm

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Color Constancy Problem

Color Constancy Problem

If lighting conditions change, colors may ”fall out of” their original cluster, or even worse, into another one.

Red

Green

green light

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Object Segmentation - Training

Object Segmentation - Training

The system was trained using both front and back sides of the objects.The background of the training images was manually removed before training.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Object SegmentationObject Segmentation

A search window scans through the image, comparing the cooccurrence histogram with the stored histogram from the training images. The result is a vote matrix.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Object SegmentationObject Segmentation

From the vote matrix, segmentation windows are contructed.Starting from the global maximum, adjacent rows and columns are added as long as the vote values give sufficient support.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Object Segmentation - Results

Object Segmentation - Results

Out of 50 test images, 49 objects were successfully segmented. Average segmentation time was 1.7 s on a 500 MHz Sun station.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic


The geometric model based pose estimator requires an initial pose to converge. The initial pose is estimated using color cooccurrence histograms.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Pose Estimation - Training

Pose Estimation - Training

70 training images were used.The pose of the object varied over the training images.The correct pose of the object in the training image was stored, together with the cooccurrence histogram.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic


The object with the unknown pose is compared to each of the training examples. The result is a match value graph.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic


The match value graph is filtered using a Gaussian kernel.Superior method compared to a nearest-neighbor approach.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Initial Pose EstimationInitial Pose Estimation

Appearance based

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Principle Component Analisys

Principle Component Analisys

Learning stage – compressing image set using eigenspace representation PCA PCAPose recognition stage – closest point search on appearance manifold PCAFitting stage – closest line search for pose refinement

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

PCA – i(q)PCA – i(q)

Pose Appearance

Eigenstructure decomposition problem

PCA

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

PCAPCA

Implicit covariance matrix (conjugate gradient method)

PCA

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

PCAPCA

Pose determination

PCA

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Initialization by PCAInitialization by PCA

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Geometric Model Based Pose Estimation


Finally, the algorithm was integrated with the model based pose estimator.

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic



Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Local refinement by tracking

Local refinement by tracking

H = (14 0 60 15 6 5) [mm, deg]

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Pose estimationPose estimation

DeMenthon and Davis 1995Orthographic projectionIterative methodNo initial guess needed

This step is followed by an extension of Lowe’s nonlinear approach

(Canceroni, Araujo and Brown et al.)

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

TrackingTracking

Lie algebra approach Rigid body motion SE(3) (6D Lie group)

0000000000010010

0000000100000100

0000001001000000

zyxGGG

0000000000010010

0000000100000100

0000001001000000

zyxGGG

0000100000000000

0000000010000000

0000000000001000

zyx ttt GGG

0000100000000000

0000000010000000

0000000000001000

zyx ttt GGG

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Image motionImage motion

with:

L - observed motion in an image point

wvwu

yxandZ

YX

AEwvu

//

1

wvwu

yxandZ

YX

AEwvu

//

1

2

2

1w

wv

w

vw

wu

w

u

yxLandZ

YX

AEGwvu

ii

2

2

1w

wv

w

vw

wu

w

u

yxLandZ

YX

AEGwvu

ii

i

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

3D pose update3D pose update

The change in pose is estimated using least square approach:

where represents the quantities of Euclidian motion

iiji

p

ppj

ppiij

p

ppi

pi

OC

nLnLC

nLdO

1

))((

)(

iiji

p

ppj

ppiij

p

ppi

pi

OC

nLnLC

nLdO

1

))((

)(

i

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

3D pose update3D pose update

iiGIM iiGIM

)(),(),( 1ii

tt GItRHtRH )(),(),( 1ii

tt GItRHtRH

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Task 1 – Align and Track

Task 1 – Align and Track

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Task 2 – Object Positioning

Task 2 – Object Positioning

Ob

ject

Reco

gnit

ion

& M

od

el B

ase

d T

rack

ing

© Danica Kragic

Insertion taskInsertion task

How much a-priori info can we used?

Object Recognition & Model Based Tracking © Danica Kragic Tracking system.

Documents

Transcript of Object Recognition & Model Based Tracking © Danica Kragic Tracking system.