Visual Tracking

CMPUT 615Nilanjan Ray

What is Visual Tracking

• Following objects through image sequences or videos

• Sometimes we need to track a single object, sometimes a number of them

• Sometimes we just track the object centroid, sometimes entire object boundary (shape)

Theoretical Foundation

• Visual tracking is a “state” estimation problem• Bayesian inference is at the heart of visual

tracking; it is called sequential Bayesian estimation

• We form the posterior probability of the state, given all evidence or measurements up to the current time point

• Inference is performed from the posterior density

Setting The Stage

Some notations:

Xt: unknown state we want to estimate at time point t; e.g., object centroid

Zt: Measurement/observation made at time point t; e.g., image intensities

The sequential estimation model assumes that we know three probabilitydensities:

p(X0): The initial state density

p(Xt|Xt-1): State transition density or motion model

p(Zt|Xt): Measurement/observation/likelihood density

Sequential Bayesian Estimation(AKA Sequential Bayesian Filtering)

• We want to recursively estimate the state Xt given the observations Z1:t = {Z1, Z2, …, Zt}

Sequential Bayesian Estimation…

Filter:

Bayes’ Rule:

Current posterior

Previous posteriorPrediction

Likelihood/observation density

Bayes’ Rule Derivation

Conditional probability ruleMarginal density rule

Also, because measurement Zt is conditionally independent on the current state Xt:

So, we have the sequential Bayes’ rule:

Filter Derivation

Rule of marginal density Rule of conditional probability

Also, note that Xt is conditionally independent on Xt-1 (Markovianity), so:

Thus we have the filter rule:

Important Assumptions

• Observation is conditionally independent on the current state

• Current state is conditionally independent on the immediate previous state

Computation

• Theory is all good, however we need to show people that it works in practice…

• We will study Particle filter, the framework that can compute the recursive state estimation, i.e., sequential Bayesian estimation

• We will also study Kalman filter, a popular sequential state estimation technique with some more assumptions

What is a Particle Filter?Let the particles represent the previous density

So, the filter step is now:

And the Bayes’ rule is now:

We need to generate the current particle set from p(Xt|Zt):

Particle filter

Factored Sampling

Let h(x) = f(x)g(x) is a product of two functions, where say, g(x) is a density andf(x) is another non-negative function

Factored sampling says that to represent h(x) non-parametrically by a set of particles, generate samples from g(x) and assign weights by f(x)

i.e., {(s1 , w1), …, (sn, wn)}, where si are generated from g(x) and wi = f(si).

This is closely related to another sampling method called importance sampling.

Conditional Density Propagation (CONDENSATION)

This a product of two functions: (1)

and (2)

Following the principle of factored sampling, CONDENSATION generates samples from (1)And assigns weights using (2)

Samples From a Mixture Density

Notice that is a mixture density

To generate samples from the mixture density these two steps are followed:

CONDENSATION Algorithm

How to estimate the state?

• OK, we generated samples, what do we do with them: estimate the current state:

h is any function of the state, for example when h(x) = x, we are performing state estimation

Other PFs

• To date lots of particle filters have been proposed:– Sequential importance re-sampling (SIR)– Auxiliary particle filter (APF)– Likelihood particle filter– Rao-Blackwellized particle filter– A ton others

• A leading researcher in PF : Arnoud Doucet

Some Points to Ponder about PF

• The good point about PF is that it can handle very general likelihood and motion models

• PF inherits a serious shortcoming from non-parametric density representation – curse of dimensionality – when the state space x is large, for example large multiple number of objects etc.