Course14 Dynamic Vision

Course14 Dynamic Vision

Biological vision can cope with changing world.

----- Moving and changing objects----- Change illumination----- Change View-point----- Other changes

Computer vision does in the similar ways. But we have only focus on:

----- Stationary camera, stationary objects----- Stationary camera, moving objects----- Moving camera, stationary objects----- Moving camera, moving objects

1.Change Detection

----- Stationary camera, moving objects in scene.

----- Detect moving object from stationary scene.

----- Two or more frames of image.

----- Images are well aligned.

 

(1)  Difference Pictures:DPjk(x, y)=1 , if |fj(x,y) – fk(x,y)| > TDPjk(x, y)=0 , otherwise

PDPjk(x, y)=1 , if fj(x,y) – fk(x,y) > TPDPjk(x, y)=0 , otherwise

NDPjk(x, y)=1 , if fj(x,y) – fk(x,y) < TNDPjk(x, y)=0 , otherwise

Size filter should used to remove small connected components that are due to noisy pixels.Small motion cannot be detected when use size filter.

(2)  Accumulated Difference Pictures

----- a sequence of images taken by a stationary camera at different time.

----- to detect small/slow motion objects by enhancing the difference of pixel values.

ADP0(x,y) = 0

ADPk(x,y) = ADPk-1(x,y) + DP1k(x,y)

ADP0(x,y) = 0

ADPk(x,y) = ADPk-1(x,y) + DP1k(x,y)

Where

(3) Time-varying Edge Detection:

----- Detect the edges of moving objects.

----- Intensity changes in space and in time make up each other.

dt

tyxdf

ds

tyxdfzyxEt

),,(),,(),,(

otherwise1

1),(if),(),,( yxfyxf

ds

tyxdf

otherwise1

1),,(if),,(),,( tyxdftyxdf

dt

tyxdf

Since spatial and temporal changes of image

complement each other. Either Good intensity

change/Slow object motion or Poor intensity

change/Proper object motion can yield good result

of edge detection of moving objects.

(4) Segment moving object with moving camera

----- Moving objects in stationary scene.

----- Translational moving camera.

When camera is performing translational motion in a stationary scene. The image of the scene looks as if caming out from FOE of the image.

dz

dydz

dx

Y

XFOE

f

f

Do Ego-motion-polar transformation for every frame of images:

In the new images of EMP space, stationary points are displayed only along the polar axis between the frames of image sequence, moving points are not so. Thus, moving points and stationary points can be

separated.

),,(),,( tyxftrE

22 )()( ff YYXXr

)(tan 1

f

f

xX

yY

2. 3D structure from motion

3D structure from images means to find the depth of

scene of the corresponding feature points in image.

(1) 3D structure from motion of point correspondences:

(1)

TpRp3

since perspective projection

We have

(2)

Apply to both sides of equ (2)

So,

TPzRPz

pz

P 1 pz

P 1

P

0 TPPRPz

RPP

TPz

Once knowing point correspondences and motion

parameters, 3D depth of image point can be

computed.

Note: the 3D structure will be up to a scale

factor if the absolute value of translation is

uncertain, such as being computed from two

frames of images.

(2) 3D structure from motion of Line Correspondences.

and

are line correspondence over two frames of images.

C

B

A

N

C

B

A

N

Let the corresponding 3D line has the form:

parameter,: nQpL

NRN

NRNn

1

1

(

)(

f)by yoursel derive()(

)()sgn(

1

1

nN

nN

NRN

NRTznNQ

(3) 3D structure from ego-motion camera. ----- Stationary scene.

----- Translationally moving camera with known speed.

----- FOE of image has been computed.

----- Determine the depth of 3D scene.

Ego-motion Complex Log Mapping:

dz

dydz

dx

b

aFOE

jrebYjaX )()(

Where

Define

i.e., u = lnr v =

Now, we want to find the relation of u, v and camera movement in z-direction. Remember that:

22 )()( bYaXr

)(tan 1

aX

bY

),(),(lnln rjvrujrw

z

xX

z

yY (assume f = 1)

dz

dr

rr

dz

d

dz

du 1ln

dz

bzy

azx

d

bzy

azx

22

22

1

z

1

du

dzz So,

dz

azx

bzy

d

dz

d

dz

dv

1tan

dz

az

x

bz

y

d

az

x

bz

y

2

1

1

02

a

zx

z

zy

bazx

zy

bazx

We can conclude that:

----- in ego-motion complex log mapping (EMCLM) space,

is only related the depth changes of scene.

----- Since dz can be measured from the speed of camera movement and du can be computed from image, the depth of scene can be found by

----- The movement of camera in z-direction does not affect the EMCLM velocity. i.e.

dz

du

du

dzz

0dt

dv

(4) Match Correspondences

-----Find the corresponding image features (Points or Lines) from two images frames that correspond to the same features in 3D scene.

1)  Point Matching (relaxation labeling)

Given: a set of feature points in image frame 1 and another set of feature point in image 2.

Find: a unique correspondences of points between the two sets of points.

(a) Defines object set O={o1, o2, …, om} from image points of

frame 1, each element is a node. Define Label set L={l1,

l2, …, ln} from points of frame 2.

(b) Establish relationship set among the nodes of object

nodes, such as neighboring points.

(c) Establish an initial match set

M(0)={(<o1,l1>,<o1,l2>,…<o1,ln>),

……

(<oi,l1>,<oi,l2>,…<oi,ln>)

(<om,l1>,<om,l2>,…<om,ln>)};

i.e. each object node aligned to all labels.

(d) Establish consistency measurement of each node and its related nodes with respect to aligned pairs.

e.g. <oi,li> consistent with <oj,lh>, <ow,lv>, …

Consistency measurement may be based on:

----- geometric relation among node in image.

----- gray level or gradient in the original image of the node.

(e) Compute similarity (or disparity) of each node with respect to matched pair.

e.g. where di, l----- disparity,

such as displacement vector.

Probability of match between oi and li :

lili d

w,

, 1

1

kki

lii w

wilP

,

,)|(

(f) Update match set M(k) iteratively:

If the similarity of <oi,li> is high, encourage the match of

its consistent nodes, otherwise, discourage the match of

its consistent nodes.

e.g.

and, normalize the updated probability; where and are

constant.

i

ii

o,lo

kj

kil jhpq

of nodes related of

yconsistencby near

)()( )|(

))(|()|( )()()1( kil

ki

ki qilpilp

(g)  Remove the match pair of small similarity (small match

probability ) from match set M(k)

(h) Repeat step (f) and (g) until each node has no more

than one label in M(k).

)|()( ilp ki

(2) Match Line correspondences

Given: Two set of Lines in image A and image B resp

ectively.

Find: Unique correspondences of lines between imag

e A and B.

(a)  Matching function:

----- Position disparity.

Relative position in an image:

where is edge direction.

Position disparity between two sub-sets of image

lines from images A and B.

oiojioiojij ppuppd )]sgn([

iu

1

1 121 ),,,;,,2,1(

n

i

n

ijkBkAijn ji

ddkkknd

----- Orientation disparity:

1

1 121 ),,,;,,2,1(

n

i

n

ijkBkAijn ji

kkkn

----- Other disparity:• Length of Line• Intensity of original image•  Contrast•  Steepness•  Straightness (residues of Least squares)

(b) Kernel match:

Match a small sub-sets from image Lines of frames A and B, for robustness consideration of the kernel,----- Number of lines should be no less than 3.----- Lines should be long (stable).----- Lines should not be paralleled.----- Lines should be separated as much as possible.

Minimize the match function over selected subjects between two image frames.

----- attribute, such as position ,orientation.

----- weight

(c) Match expansion:

Once kernel matching is completed, the line correspondences obtained will serve a reference for the match of remaining lines.

)21),,,;,,2,1( 2121 nxxn ,..,k,k,n;k,,(kkkn

,1,0 x thx

Choose a longest line from unmatched line of image

A. Add it into the subset of matched kernel of image

A, calculate match functions for every unmatched

line in image B. The line of image B with minimum

match function is considered a matched line. Add this

matched pair of lines into matching kernel and repeat

the process until no further line needs to match.

(3) Tracking

Given: m objects moving in scene, a sequence of n image frames is taken from the scene.

Find: the trajectories of each object in the image sequence.

i.e.

Ti = < Pi1, Pi

2,…, Pin>; i=1,2, … m

If we use dik to be trajectory deviation of frame k.

dik=( Pi

k-1 Pik, Pi

k Pik+1)

We want: for each object.

1

2

minn

k

kii dD

(a) Path Coherence Function

Assume in consecutive image frames that:

----- change of object location is small. ----- change of scalar velocity is small. ----- change of moving direction is small.

Path coherence function can be defined as

where w1, w2 are weight.

)21()cos1(),,(21

21

21

11

dd

ddwwPPP k

i

k

i

k

i

----- turning angel from Pik-1 Pi

k to Pik Pi

k+1

d1, d2 ----- distances between Pik-1 , Pi

k and Pik, Pi

k+1

respectively.

(b) Occlusion problem

When occlusions occur in scene from viewpoint. Some target points may lose from some image frames. This causes problem in tracking.

To solve this problem, we introduce the concept of phantom point:

----- Virtual object point in frames. ----- No specified coordinates. ----- Has maximum distance dmax from the consecutive

frames. ----- Has a maximum value of path coherence function .

point phantom ispoint any if

features trueare points both if)distance()(Disp

max

11

d

, PP, PP

ki

kik

ik

i

otherwise

points trueare points threeif),(

point phantom is if0

),Dev(

max

11

1

11

k

ik

ik

i

ki

ki

ki

ki , PPP

P

, PPP

Thus:

Tracking algorithm a) Path Initialization

Make initial trajectories for each feature point by linking the nearest point in consecutive frames from frame 1 to frame n. If feature points are missing in some frames, introduce phantom points.

b)  Exchange assignments:

From frame 2 to frame n–1 , for each frame:

i) Perfrom forward exchange:

For each feature point i of frame k, find all the combinations wi

th possible feature j of k+1 frame within a window of dmax.

Calculate:

Gijk = [ Dev ( Pi

k-1 , Pik, Pi

k+1) + Dev( Pjk-1 , Pj

k, Pjk+1)]

–[ Dev( Pik-1 , Pi

k, Pjk+1) + Dev ( Pj

k-1 , Pjk, Pi

k+1)]

Find the i-j pair with maximum value of Gmax=( Gijk).

If Gmax > 0 , exchange the trajectory assignment of i

and j in k+1 frame.

(ii) Perform backward exchange:

For each point i of frame k, find all the combinations with poss

ible point j of frame k-1 within a window of dmax.

Calculate:Gij

k = [ Dev( Pik-1 , Pi

k, Pik+1) + Dev ( Pj

k-1 , Pjk, Pj

k+1)]

–[ Dev( Pjk-1 , Pi

k, Pik+1) + Dev ( Pi

k-1 , Pjk, Pj

k+1)]

Find the ij pair with Gmax=( Gijk).

If Gmax > 0 ,exchange trajectory assignments of i and j in frame k–1.

Note: Final results of trajectories may involve phantom points.

The lecture is overThe lecture is over

And And

Thank you for your Thank you for your attendanceattendance

Hope you have leant Hope you have leant

some Basic Concept some Basic Concept

ofof

Computer VisionComputer Vision

in this lecture, which in this lecture, which

is helpful in you is helpful in you

research.research.