MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Heterogeneous Sensor Webs for Automated Target...

MURI Annual Review, Vanderbilt, Sep 8th, 2009

Heterogeneous Sensor Webs for Automated Target Recognition and Tracking in Urban Terrain

(W911NF-06-1-0076)

Motion Pattern Analysis with(out) Trajectories

John Fisher

MIT CSAIL


People interacting with people

What was he thinking?

Hard questions, most 9-year olds wouldn’t know the answer anyway.

What was he reacting to?

Perhaps a little easier.

Can we analyze the dependency between the player interactions?


People interacting with people

Two camera angles provide noisy position estimates.

Can derive noisy estimates of kinematic state (i.e. position, velocity, etc.)

Could use other features as well (e.g. body position, etc.)

This gives multiple time series.

One for each player plus the ball.

Interaction can be cast as inference over the structure of influence between time-series.


People interacting with their environments

Object tracking is challenging in these scenarios.

Can we derive aggregate models of behavior?

Assume there is a persistent or slowly varying motion pattern.

Can we estimate aggregate motion patterns without tracking each individual?


Motion Pattern Analysis

Inference over aggregate properties of a dynamic scene Scenarios in which tracking each object is intractable

– Challenging when the properties live in a curved space Integration of two mathematical formalisms

– Lie-algebraic representations of motion/deformation– Variational inference in graphical models

Inference over the structure of interactions between multiple time-series

Suppose we are only interested in the graph describing interactions

– Complexity of inference over structure is super-exponential O(NN) in the number of objects

– If the structure of interaction varies dynamically, complexity is exponential in the duration O((NN)T)


Lie Groups and Lie Algebras

Connected by matrix exponentiation and logarithm

Other properties– Identity transform corresponds to zero vector– Inverse transforms corresponds to negation – Commutable multiplication corresponds to addition

I0

G )(GLieexp

log

1 !

)exp(n

n

n

XIXT

1

1

)()1(

)log(n

nn

ITn

TX

0IXT 1


Acting on geometric points

Txx

Acting on images

)())((: 1xTIxITITI

Lie Group Action on Images

Lie group action

Two roles of the transform T

Twofold role is key for estimating T directly from images

),()( , s.t.

),( ::),(

1212 pTTpTTppI

pTpTMMGpT


0t

Action of the infinitesimal generator

T: transform, X: infinitesimal generator

)exp(0

tXdt

dX

t

))(exp(0

ItXdt

dIX

t

1t 2t3t

4t

0tIX


The key relation

We derive the following relation between the motion space and the space of image changes

n

kkk

n

kkk IEIE

11

)(

the decomposition of motion

the decomposition of image changes

The motion can be inferred by decomposing the image changes


Computation

Assume that the motion is characterized by

K

kkkEX

1

K

ktkk

t IEIXt

I

1

)(

)(1

tttt II

tt

I

)()(

)()())((

xExI

xVxIxIE

kT

t

kT

ttk

approximate byfinite difference

negated pointwise inner product of the gradient and the induced velocity


Extension to Triangular Meshes

Extend the formalism with triangle mesh Consistency at boundary

Consistent subspace (m triangles, n vertices) 1A

2A3A

4A5A

6Ax

xAxAxA 621

m6

nmmn

ii 26)1(2

1

n2

joint dim =

-) constraints =

subs. dim = The key relation still applieswith this extension.


Efficient Inference Over Deformations

-

1 K

tI

ttI )(

1ttt II

tt

I

tI1E KE

tIE 1 tK IE

finite difference

p.w. dot product

……

……

gradient

estimate byregression


Multi-flow probabilistic model

ig

iy

G ic

ivM

iz

M models

flow indicator

jg

jy

jc

jv jz

flow indicator

MRFMRF


People interacting with their environments (Lie group)


People interacting with their environments (opt flow)


Object Interaction Analysis

N tracked objects (cars, people, genes, consumers, etc..)

Observe T noisy samples of state (e.g. position/velocity)

Can we infer properties of the interaction?

Who is interacting with whom? Who is the probable leader? Is the nature of interaction

changing over time?


Outline

Challenges of Dependence Analysis– Static Dependence – Dynamic Dependence

Factorization Model Temporal Interaction Model Conclusion

18


Static Dependence Analysis

Given – Observations:

– Generative Model:

– Prior:

Find

19


Challenges of Static Analysis

Structural specification Unknown parameters Number of structures

– e.g. We explore > 109 structures

20


Temporal Interaction Model

x1

x2

x3

1

2

3


Conjugate Prior on Parameters

Given the structure, parameters are independent:

– and modular:

is the same for all such that

22

11

22

33

11

22

33


Conjugate Prior on Structures

The prior on structure factorizes as a product of weights on parent sets:

23

Uniform Dense Sparse


Posterior

Posterior on structure is a simple update to the parent set weights

where:

24

Matrix-T

Matrix Normal-Inverse-WishartMatrix Normal


Computing the Partition Function

How many structures are there?– Each time-series has possible parents

– time-series

25

Super-exponential in N

parent sets

structures


Computing the Partition Function

26

All Directed Structures

Super-exponential to Exponential


Bounded Parent Sets

Assume

,– structures (still super-exponential)

27

Polynomial-time computation

MURI Annual Review, Vanderbilt, Sep 8th, 2009"Motion Pattern Analysis", Fisher 28

Switching Vector Autoregressive Tree

SVART(1)


Bayesian Reasoning over Interaction Structures

MURI Annual Review, Vanderbilt, Sep 8th, 2009"Motion Pattern Analysis", Fisher 30

Basketball


Basketball Results

Using a STIM with 10 states with

Team A onOffense

Team B onOffense

Team B trans.to Offense

Team A trans.to Offense


Basketball Event Probabilities

Team A onOffense

Team B onOffense



Team A onOffense

Team B onOffense




Expected Number of Children

Top 4– Ball (1.8)– Point Guard A (1.7)– Forward 1 A (1.1)– Forward 1 B (1.0)


Influence of Point Guard A

0.0 1.00.5

MURI Annual Review, Vanderbilt, Sep 8th, 200917Apr08

Prior and Posterior Expectations

Can tractably compute expectations of: Multiplicative functions on structure Additive functions on structure

Allows one to calculate: Expected number of children

– How influential? Expected number of parents

– How impressionable?

MURI Annual Review, Vanderbilt, Sep 8th, 200917Apr08

Comments

Focus was on vision sensors, but both methods have wider application

– Seismic data (petroleum exploration)– Audio-visual association (multi-media annotation)

Thank you

MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Heterogeneous Sensor Webs for Automated Target...

Documents

Transcript of MURI Annual Review, Vanderbilt, Sep 8 th, 2009 Heterogeneous Sensor Webs for Automated Target...