MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 1

Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation

Graphical Models for Resource-Constrained Hypothesis Testing and Multi-Modal Data Fusion

MURI Annual Review Meeting

John Fisher, Al Hero, Alan Willsky

November 3, 2008

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 2

Topics

Design of Discriminative Sensor Models Under Resource Constraints Discriminative tree models

Distributed PCA Multi-modal Data Fusion

LIDAR/EO registration

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 3

Common Thread

These problems are formulated as inference in graphical models in which:

Assumptions or partial specifications of the structures and associated parameters of the latent variables lead to approximate inference methods

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 4

Max-weight Discriminative Forests

Discriminative Generalization of Chow-Liu Method for Learning Discriminative Trees Kruskal’s Algorithm Use of J-Divergence

Interpretation & Performance bounds

Algorithm, Proof Sketch Empirical Results

Small Sparse & Non-Sparse Models

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 5

Trees admit a simple factorization

Tree models described by marginal properties on the vertex set pair-wise relationships on the edge set

Entropy has a similarly simple decomposition

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 6

The closest tree model in a K-L divergence sense to any distribution can be found by solving a max-weight spanning-tree (MWST) problem over pair-wise mutual information terms, (Chow-Liu ‘68).

Information Projection Result

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 7

Many algorithms for solving the max-weight spanning (Prim, Kruskal, reverse-delete, Chazelle, etc.)

Kruskal’s is of particular interest Greedy Yields a sequence of optimal k-edge forests

Solving the MWST Problem

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 8

Proof Sketch of Kruskal’s Algorithm

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 9

J-Divergence

This is a well studied information measure (Jeffreys ’46)

Difference between the expected value of the log-likelihood ratio under each hypotheses for binary hypothesis

Upper and lower bound on probability of error (Hoeffding & Wolfowitz ’58, Kailath ’67)

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 10

J-Divergence

We’ll consider an alternative measure

where

If we restrict , to be trees then it turns there is an efficient MWST algorithm for jointly learning tree models for p and q which optimizes

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 11

Multi-valued edge weights

If pA and qB are constrained to be trees:

where

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 12

Multi-valued edge weights

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 13

On the use Kruskal’s Algorithm

The sequence of forests are optimal. At each iteration, the number of edges in the

respective models may be different Early termination possible (i.e. pA and/or qB may

not be spanning trees upon termination) The proof optimality is similar to the generative

case, the primary difference is that some edges may be precluded in one tree, but not the other.

The ws,t become single-valued The max is always reduced

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 14

Empirical Results

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 15

Empirical Results

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 16

A somewhat contrived example

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 17

A somewhat contrived example

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 18

Comments

We have presented a discriminative version of the Chow-Liu method for learning trees using an approximation to J-divergence

This was enabled by defining a multi-valued weight function where elements map to edge choices in one or both tree models.

The method results in structures that differ from the generative models.

Early termination is possible resulting in a forest or non-spanning tree.

Empirical Results Marginal improvements for small graphs which are already sparse. Illustrative example demonstrated that improvement over

generatively learned trees can be significant. Open Questions/Directions

Can one develop a Hoeffding-Wolfowitz type bound using the approximate measure?

What is the behavior on sparse graphs as the size of the graph grows?

Evaluate performance with empirical distributions.

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 19

Outline

Progress in front-end processing and fusion New: graphical models for distributed

decomposable PCA New: graphical models for

hyperspectral image unmixing

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 20

Part III: High level fusion

Application of hierarchical graphical models Distributed decomposable PCA Hyperspectral imaging and unmixing

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 21

Decomposable PCA

Principle components analysis (PCA) is a model-free dimensionality reduction technique used for high level data fusion (variable importance, regression, variable selection)

Deficiencies: PCA does not naturally incorporate priors on

Dependency structure (graphical model) Matrix patterning (decomposability)

Scalability problem: complexity is O(N^3) Unreliable/unimplementable for high dimensional data Ill-suited for distributed implementation, e.g., in

sensor networks

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 22

Networked PCA

Network model: measure sensor outputs Xa, Xb, Xc

Two cliques {a,c} and {b,c}

Separator {c} Decomposable model:

covariance matrix R unknown but conditional independence structure is known.

PCA of covariance matrix R finds linear combinations y=UTX that have maximum or minimum variance

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 23

DPCA formulation

Precision matrix K=R-1 For decomposable model K has

structure

General Representation

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 24

1-dimensional DPCA

PCA for minimum eigenvector/eigenvalue solves

Key observation: This constraint is equivalent to

where

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 25

Extension to k-dimensional DPCA

k-dimensional PCA solves sequence of eigenvalue problems

Dual optimization

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 26

k-dimensional DPCA (ctd)

Dual maximization splits into local minimization with message passing

Message passing

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 27

Tracking illustration of DPCA

Scenario: Network with 305 nodes representing three fully connected networks with only 5 coupling nodes

C1 = {1, · · · , 100, 301, · · · , 305}, C2 = {101, · · · , 200, 301, · · · , 305}, and C3 = {201, · · · , 300, 301, · · · , 305}.

Local MLEs computed over sliding time windows of length n = 500 400 samples overlap.

Centralized PCA computation: EVD O(305)^3 flops DPCA computation: EVD O(105)^3 flops + message

passing of a 5x5 matrix M

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 28

DPCA min-eigenvalue tracker

Iteration 1

Iteration 2

Iteration 3

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 29

DPCA network anomaly detection

SNVA

STTL

LOSA KSCY

HSTN

DNVRCHIN

IPLS

ATLA

WASH

NYCM

Multiple measurement sites (Abilene)

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 30

DPCA anomaly detection

PCA (centralized)

DPCA (E-W decomp)

DPCA (E-W-S decomp)

DPCA (Random decomp)

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 31

Discussion

Take home message: Combination of model-free dimensionality reduction and model-based graphical model can significantly reduce computational complexity of PCA-based high-level fusion

Complexity scales polynomially in clique size not in overall size of problem. Example: 100,000 variables with 500 cliques each of size 200

Centralized PCA: complexity is of order 1015 DPCA: complexity is of order 106

If one can impose similar decomposability constraints on graph Laplacian matrix, be extended to non-linear dimensionality reduction: ISOMAP, Laplacian eigenmaps, dwMDS.

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 32

Outline

Laser Radar 3D Model Generation

Delaunay mesh formation Model Demo

Camera Model Statistical Registration Methods Results

Probing Experiments Registration Demo

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 33

Laser Radar

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 34

Laser Radar

3D point cloud Not gridded Typical lateral resolution

of 0.5 – 1.0 meters

Linear mode sensors: Slower collects Probability of detection

(pdet) values Geiger mode sensors:

Faster collects Geometry only

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 35

3D Model Generation

Ladar point cloud

Inferred surface

Registered optical image

Projective texture mapping

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 36

3D Model Generation

Point cloud Mesh 3D Model

Optical image

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 37

Mesh Generation: Delaunay Triangulation

Maximizes the minimum angle of all the angles of the triangles in the triangulation

Circle circumscribed on any triangle does not contain any other points

Dual graph of Voronoi Tessellation

Provides rough estimate of underlying structure

http://en.wikipedia.org/wiki/Delaunay_triangulation

x

z

y

x

z

y

x

z

y

z

x

y

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 38

Automatic Registration Challenges

GPS/INS has low precision Matching colors with geometry (inherently low

mutual information) Occlusion reasoning with point cloud projection 3D rendering Resolution differences Previous work

Line correspondences (Frueh, Sammon, Zakhor 2004)

Vanishing points (Ding, Lyngbaek, Zakhor 2008)

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 39

Homogeneous Coordinate System

w

v

u

X

wv

wu

v

u~

~

t

z

y

x

X

tz

ty

tx

z

y

x

~

~

~

2D (optical)

3D (ladar)

Homogeneous Euclidean

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 40

Projective Camera

t

z

y

x

pppp

pppp

pppp

w

v

u

T

34333231

24232221

14131211

z

y

x

333231

232221

131211

100

010

001

100

00

00

|

C

C

C

rrr

rrr

rrr

f

f

CIKRT

view)of field s(determinelength Focal :

position projection ofcenter Camera :

angles)Euler 3by ized(parametermatrix Rotation :

matrixn calibratio Camera :

matrix projection Camera :

z

y

x

f

C

C

C

R

K

T

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 41

Projective Camera

x

z

y

Field of view

(determined by focal length)

Image plane

View direction

(characterized by 3 Euler angles)

Center of projection

z

y

x

C

C

C

f

Parameters to estimate:

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 42

Initial Registration

Manual calibration is not tractable User selected correspondence points Minimization of sum of algebraic error

T is parameterized by f, Cx, Cy, Cz, α, β, γ Levenberg-Marquardt iterative

optimization (derivative free)

2alg ),(min i

ii

TTd XX

22algalg )ˆˆ()ˆˆ(ˆ,(),( vwwvwuuwdTd )XXXX

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 43

Statistical Registration Methods

Commonly used for registration of multi-modal medical imagery

Information theoretic similarity measure with optimization algorithm

Ladar Image

Optical Image

Feature detection / classification

Feature detection / classification

Registration

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 44

Example: Joint Entropy

Pdet ValuesLuminance

Optical imageiTi

iT

TJE vupvupT ]),([log)],([minarg

T: camera projection matrixu: optical image features/labelsv: ladar image features/labelsp(·): probability mass function

JE Registration

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 45

Probing Experiments

-100 -80 -60 -40 -20 0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x displacement (meters)

Nor

mal

ized

join

t en

trop

y

-100 -80 -60 -40 -20 0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

y displacement (meters)

Nor

mal

ized

join

t en

trop

y

-100 -80 -60 -40 -20 0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

z displacement (meters)

Nor

mal

ized

join

t en

trop

y

-3 -2 -1 0 1 2 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Field of view displacement (degrees)

Nor

mal

ized

join

t en

trop

y

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 46

Probing Experiments

-3 -2 -1 0 1 2 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

a displacement (degrees)

Nor

mal

ized

join

t en

trop

y

-3 -2 -1 0 1 2 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

b displacement (degrees)

Nor

mal

ized

join

t en

trop

y

-20 -15 -10 -5 0 5 10 15 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

g displacement (degrees)

Nor

mal

ized

join

t en

trop

y

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 47

Automatic Registration Demo

Minimum joint entropy with optical luminance and ladar elevation

Downhill simplex optimization (derivative free)

Ladar rendered using OpenGL Entire ladar data set in graphics

memory Approximate initial registration

MURI: Integrated Fusion, Performance Prediction, and Sensor Management for Automatic Target Exploitation 48

Unsupervised LIDAR/Optical Registration