Multiple Frame Motion Inference Using Belief

Propagation

Jiang Gao Jianbo Shi

Presented By: Gilad Kapelushnik

Visual Recognition, Spring 2005, Technion IIT.

Abstract

• Find “best fit” upper body joint configuration.

• Input is a 2D video

• Each joint is described by its location on a 2D grid.

S1(X,Y)

S2(X,Y)

S4(X,Y)

S6(X,Y)

S5(X,Y)

S3(X,Y)

Let J be a joint configuration – {S1,S2,S3,S4,S5,S6}

We would like to find:

argmax ( | )J

J P J observations

• Step 1: Subtract two sequential frames.

• Step 2: Apply threshold.

Motion Energy Image

From #NrgPixels To Probability• Sum the Energy Pixels in the Patch.

• Calculate probability using the following:

S5(10,60)

S6(40,30)

1 exp( # )

NormConstP

NrgPixels

0.12

0.19

0.84

0.68

0.02

• Find configuration J with the highest probability.

• Computing all possible probabilities is inefficient.

• a-Priori data give better and faster results.

• removing impossible configurations reduce inference time.

Main Idea

( )JX Y

a-Priori Data• A probability table for Each P(Sx,Sy).

• Compute probability at grid crossing.

• Use nearest neighbor for the rest of the image.

Example:

For right arm - P(S2,S3)

Red – Low probability

Green – High probability

P(S2,S3)12…Ns^2

100…0

200.1…0

…………0

Ns^20000

• Face is detected using face detection algorithm.

• Initial assumption of Shoulders from face and pose.

• Even using BP there are too many possible states to go through.

• Candidates for elbows from shoulders & Energy Map.

• Candidates for Wrists from skin color model.

Detect Candidate states (1)

Detect Candidate states (2)• Many states can be discarded.

• Remove close candidate states.

Pros: Much faster inference.

Cons: Less accurate.

• Note: This is only an option.

Fits skin color and wrist location

Pink for right wristRed for left wrist

Blue for elbow

The Markov Model• Empty Circles - States - 2D positions of joints

• Full Circles - Observations - Computed from energy map.

• Each state correspond to an observation.

Belief Propagation (1)• Solve inference problem using an algorithm with Linear complexity.

• Each joint has a vector with probabilities for each candidate.

Shoulder

Elbow

Wrist

Belief Propagation (2)

1

2

3

m23

m32

m21

m12m14

m41For each iteration:

• Each node sends a message to its neighbor nodes containing the “wanted” probability (for each state).

• Messages are computed according to:

( ) /

( )

( , ) ( | ) ( )

( ) ( | ) ( )

i

ij ij i j i i i ki is k N i j

i i i i i ki ik N i

m P s s P x s m s

b s P x s m s

Sum over all candidates

A-priori Data for each state.

Normalize variable.

Observation (# of Energy pixels in patch) for each state

converted to a probability.

Message from k to i (all messages from the

neighbors). This is actually a vector with a probability for

each state.

Message from i to j.

Belief Propagation (3) - Example

21

Message from 1 to 2

4states

2states

Belief Propagation (4)

• BP converge after 2-4 iterations (giving the right a-Priori data).

• For every joint there is a probability vector for each candidate state.

Multiple Frame Probability

• Multiple frame (8) is proposed for smoother transition between configurations.

• Prevents joints changing their state to a different which is “far away” (Euclidian distance).

• Though BP was designed to work with loopy-free models, the author stated that it worked fine.

21

, 1 12

1( , ) exp

2

t ti it t t t

i i i

s sP s s

And for those who really want to know:

2D to 3D

• 2D -> 3D by Taylor (2000).

• Assuming (u1,v1) and (u2,v2) are projections then depth can be retrieved using the following:

1 2 1 2

1 2 1 2

2 22 1 2 1 2

1 2 2

( ) ( )

( ) ( )

( ) ( ){ _ }

co

u u const X X

v v const Y Y

u u v vZ Z arm length

nst

Results(1)

Results(2)

Results(3)

Errors accrue when 2 joints intersect each other.

On some occasions, even when limbs intersect, it was possible to infer correctly.

Q?