5 cramer-rao lower bound

1

Cramér – Rao Lower Bound

SOLO HERMELIN

25.09.09http://www.solohermelin.com

http://www.solohermelin.com/

2

Cramér-Rao Lower Bound (CRLB)SOLO

Table of Content

The Cramér-Rao Lower Bound on the Variance of the Estimator

The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case

The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case

GradientMatrix Inversion Relations

Helpfully Relations

Nonrandom Parameters

Random Parameters

Nonrandom and Random Parameters Cramér – Rao Bounds

Discrete Time Nonlinear Estimation

Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound

Discrete Time Nonlinear Estimation –Special Cases

Probability Density Function of is Gaussian0x

Additive Gaussian Noises

References

Linear System with Zero System Noise

Linear/Gaussian Systems

3

Cramér-Rao Lower Bound (CRLB)

v

( )vxh ,z

x

Estimatorx

SOLO

The Cramér-Rao Lower Bound (CRLB) on the Variance of the Estimator

{ }xE - estimated mean vector

[ ]( ) [ ]( ){ } { } { } { } TTT

x xExExxExExxExE

−=−−=2σ - estimated variance matrix

For a good estimator we want

{ } xxE =- unbiased estimator vector

{ } { } { } TT

x xExExxE

−=2σ - minimum estimation covariance

( ) ( ){ }Tk kzzZ 1::1 = - the observation matrix after k observations

( ) ( ) ( ){ }xkzzLxZL k ,,,1,:1 = - the Likelihood or the joint density function of Z1:k

We have:

( )Tpzzzz ,,, 21 = ( ) Tnxxxx ,,, 21 = ( )Tpvvvv ,,, 21 =

The estimation of , using the measurements of a system corrupted by noise is a random variable with

x x zv

( ) ( ) ( ) ( )∫== dvvpxvZpxZpxZL vkvzkxzk ;||, :1|:1|:1

( ) ( )[ ]{ } ( ) ( )[ ] ( ) ( )[ ] ( ) ( )[ ] [ ] ( )xbxZdxZLZx

kzdzdxkzzLkzzxkzzxE

kkk +==

=

∫∫

:1:1:1 ,ˆ

1,,,1,,1ˆ,,1ˆ

- estimator bias( )xb

therefore:

4


v

( )vxh ,z

x

Estimatorx

SOLO

The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case

[ ]{ } [ ] [ ] ( )xbxZdxZLZxZxE kkkk +== ∫ :1:1:1:1 ,ˆˆ

We have:

[ ]{ } [ ] [ ] ( )x

xbZd

x

xZLZx

x

ZxEk

kk

k

∂∂+=

∂∂=

∂∂

∫ 1,

ˆˆ

:1:1

:1:1

Since L [Z1:k,x] is a joint density function, we have:

[ ] 1, :1:1 =∫ kk ZdxZL

[ ] [ ] [ ] [ ]0,,0

,:1

:1:1

:1:1

:1 =∂

∂=∂

∂→=∂

∂∫∫∫ k

kk

kk

k Zdx

xZLxZd

x

xZLxZd

x

xZL

[ ]( ) [ ] ( )x

xbZd

x

xZLxZx k

kk ∂

∂+=∂

∂−∫ 1,

ˆ :1:1

:1

Using the fact that: [ ] [ ] [ ]x

xZLxZL

x

xZL kk

k

∂∂=

∂∂ ,ln

,, :1

:1:1

[ ]( ) [ ] [ ] ( )x

xbZd

x

xZLxZLxZx k

kkk ∂

∂+=∂

∂−∫ 1,ln

,ˆ :1:1

:1:1

5


The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 1)

[ ]( ) [ ] [ ] ( )x

xbZd

x

xZLxZLxZx k

kkk ∂

∂+=∂

∂−∫ 1,ln

,ˆ :1:1

:1:1

Hermann Amandus Schwarz

1843 - 1921

Let use Schwarz Inequality:

( ) ( ) ( ) ( )∫∫∫ ≤ dttgdttfdttgtf22

2

The equality occurs if and only if f (t) = k g (t)

[ ]( ) [ ] [ ] [ ]xZLx

xZLgxZLxZxf k

kkk ,

,ln:&,ˆ: :1

:1:1:1 ∂

∂=−=choose:

[ ]( ) [ ] [ ]

( ) [ ]( ) [ ]( ) [ ] [ ]

∂

∂−≤

∂

∂+=

∂

∂−

∫∫

∫

kk

kkkk

kk

kk

Zdx

xZLxZLZdxZLxZx

x

xb

Zdx

xZLxZLxZx

:1

2

:1:1:1:1

2:1

2

2

:1:1

:1:1

,ln,,ˆ1

,ln,ˆ

[ ]( ) [ ]( )

[ ] [ ]∫

∫

∂

∂

∂

∂+≥−

kk

k

kkk

Zdx

xZLxZL

xxb

ZdxZLxZx

:1

2

:1:1

2

:1:12

:1,ln

,

1

,ˆ

6



[ ]( ) [ ]( )

[ ] [ ]∫

∫

∂

∂

∂

∂+≥−

kk

k

kkk

Zdx

xZLxZL

xxb

ZdxZLxZx

:1

2

:1:1

2

:1:12

:1,ln

,

1

,ˆ

This is the Cramér-Rao bound for a biased estimator

Harald Cramér1893 – 1985

Cayampudi RadhakrishnaRao

1920 -

[ ]{ } ( ) [ ] 1,&ˆ :1:1:1 =+= ∫ kkk ZdxZLxbxZxE

[ ]( ) [ ] [ ] [ ]{ } ( )( ) [ ][ ] [ ]{ }( ) [ ] ( ) [ ] [ ]{ }( ) [ ]

( ) [ ]

1

:1:12

0

:1:1:1:1:1:12

:1:1

:1:12

:1:1:12

:1

,

,ˆˆ2,ˆˆ

,ˆˆ,ˆ

∫

∫∫∫∫

+

−+−=

+−=−

kk

kkkkkkkk

kkkkk

kk

ZdxZLxb

ZdxZLZxEZxxbZdxZLZxEZx

ZdxZLxbZxEZxZdxZLxZx

[ ] [ ]{ }( ) [ ]( )

[ ] [ ]( )xb

Zdx

xZLxZL

xxb

ZdxZLZxEZx

kk

k

kkkkx2

:1

2

:1:1

2

:1:12

:1:12ˆ

,ln,

1

,ˆˆ −

∂

∂

∂

∂+≥−=

∫∫σ

7



[ ] [ ]{ }( ) [ ]( )

[ ] [ ]( )xb

Zdx

xZLxZL

xxb

ZdxZLZxEZx

kk

k

kkkkx2

:1

2

:1:1

2

:1:12

:1:12ˆ

,ln,

1

,ˆˆ −

∂

∂

∂

∂+≥−=

∫∫σ

[ ] [ ][ ]

[ ]

[ ] [ ] [ ] 0,,ln

0,

1, :1:1:1

,

,

,ln

:1:1

:1:1

:1

:1

:1

=∂

∂→=∂

∂→= ∫∫∫∂

∂

=∂

∂

kkk

xZL

x

xZL

x

xZL

kk

kk ZdxZLx

xZLZd

x

xZLZdxZL

k

k

k

[ ] [ ] [ ] [ ] [ ][ ]

0,,ln,ln

,,ln

:1

,

:1:1:1

:1:12:1

2

:1

=∂

∂∂

∂+∂

∂→ ∫∫

∂∂

∂∂

k

x

xZL

kkk

kkk

x

ZdxZLx

xZL

x

xZLZdxZL

x

xZL

k

[ ] [ ]0

,ln,ln2

:12:1

2

=

∂

∂+

∂∂→

∂∂

x

xZLE

x

xZLE kk

x

( )

[ ]( )

( )

[ ] ( )xb

xxZL

E

xxb

xb

xxZL

E

xxb

kk

x2

2:1

2

2

2

2

:1

2

2

,ln

1

,ln

1

−

∂∂

∂

∂+−=−

∂

∂

∂

∂+≥σ

8http://www.york.ac.uk/depts/maths/histstat/people/cramer.gif


[ ]( ) [ ]( )

[ ]

( )

[ ]

∂∂

∂

∂+−=

∂

∂

∂

∂+≥−∫

2:1

2

2

2

:1

2

:1:12

:1,ln

1

,ln

1

,

x

xZLE

xxb

x

xZLE

xxb

ZdxZLxZxkk

kkk

SOLO


( )

[ ]( )

( )

[ ] ( )xb

xxZL

E

xxb

xb

xxZL

E

xxb

kk

x2

2:1

2

2

2

2

:1

2

2

,ln

1

,ln

1

−

∂∂

∂

∂+−=−

∂

∂

∂

∂+≥σ

For an unbiased estimator (b (x) = 0), we have:

[ ] [ ]

∂∂

−=

∂

∂≥

2:1

22

:1

2

,ln

1

,ln

1

x

xZLE

x

xZLE

kk

xσ

Return to Table of Content

9


Gradient

Gradient of a Scalar ( ) ( ) nTnxxxxxL RR ∈=∈ ,,, 21

1

( ) n

nn

x

x

L

x

L

x

L

L

x

x

x

xL R∈

∂∂

∂∂∂∂

=

∂∂

∂∂

∂∂

=∇

2

1

2

1

( ) nxn

nnn

n

n

n

n

Txx

x

L

xx

L

xx

L

xx

L

x

L

xx

L

xx

L

xx

L

x

L

Lxxx

x

x

x

xL R∈

∂∂

∂∂∂

∂∂∂

∂∂∂

∂∂

∂∂∂

∂∂∂

∂∂∂

∂∂

=

∂

∂∂

∂∂∂

∂∂

∂∂

∂∂

=∇∇

2

2

2

2

1

2

2

2

22

2

12

2

1

2

21

2

21

2

21

2

1


10


Gradient

Gradient of a Vector ( ) ( ) ( ) nTn

pTp

p xxxxaaaaxa RRR ∈=∈=∈ ,,,,,, 2121

( ) [ ] nxp

n

p

nn

p

p

p

n

Tx

x

a

x

a

x

a

x

a

x

a

x

a

x

a

x

a

x

a

aaa

x

x

x

xa R∈

∂∂

∂∂

∂∂

∂∂

∂∂

∂∂

∂∂

∂∂

∂∂

=

∂∂

∂∂

∂∂

=∇

21

22

2

2

1

11

2

1

1

212

1

nTx

Tx

nxnTx vvxxvIx RR ∈=∇=∇∈=∇ &1

( ) ( )[ ] ( )[ ] ( ) ( )[ ] ( ) nTx

Tx

abbaT

x xaxbxbxaxbxaTT

R∈∇+∇=∇=

2

( ) ( ) ( ) ( ) xMMxxMxMxxMx T

Mx

Tx

Tx

Tx

TT

+=∇+∇=∇

3



11


Matrix Inversion Relations

∆∆∆−∆+

=

−−−

−−−−−−−

111

1111111

AD

BAADBAA

CD

BA1

2

AofcomplementSchurBADC 1: −−=∆

( ) ( ) 1111111 −−−−−−− +−=+ ADBADCBAADBCA



12


Helpfully Relations

( ) ( ) ( ) ( ) ( )zxfzxfzxfzxf

zxf Txx

Txx

Txx ,ln,ln,

,

1,ln ∇∇−∇∇=∇∇

Proof:

Start with: ( ) ( ) ( )zxfzxf

zxf xx ,,

1,ln ∇=∇

( ) ( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )zxfzxfzxfzxf

zxfzxfzxf

zxfzxf

zxfzxfzxf

zxfzxf

zxfzxf

zxfzxf

zxfzxf

zxf

Txx

Txx

Txx

Txx

Txx

Txx

Txx

Txx

Txx

Txx

,ln,ln,,

1

,ln,,

1,

,

1,ln,

,

1,

,

1

,,

1,

,

1,

,

1,ln

∇∇−∇∇=

∇∇−∇∇=∇

∇+∇∇=

∇

∇+∇∇=

∇∇=∇∇

( ) ++ →= RR pnzxf :,Lemma 1: Given a function the following relations holds:


13


Helpfully Relations

( ) ( ) ( ) ( ) ( )zxfzxfzxfzxf

zxf Txx

Txx

Txx ,ln,ln,

,

1,ln ∇∇−∇∇=∇∇

Proof:

( ) ++ →= RR pnzxf :,Lemma 1: Given a function the following relations holds:

pz R∈Lemma 2: Let be a random vector with density p (y|x) parameterized by the nonrandom vector , then:

nx R∈( ) ( ){ } ( ){ }xzpExzpxzpE T

xxzT

xxz |ln|ln|ln ∇∇−=∇∇

( ){ } ( ) ( ) ( ) ( ){ }xzpxzpExzpxzp

ExzpE Txxz

Txxz

Txxz |ln|ln|

|

1|ln

0

∇∇−

∇∇=∇∇

( ) ( ) ( ) ( ) ( ) ( ) 0||||

1|

|

1

1

=∇∇=

∇∇=

∇∇ ∫∫

pp

zdxzpzdxzpxzpxzp

xzpxzp

E Txx

Txx

Txxz

RR

Proof:

( ) ( ){ } ( ){ }zxpEzxpzxpE Txxzx

Txxzx ,ln,ln,ln ,, ∇∇−=∇∇

( ){ } ( ) ( ) ( ) ( ){ }zxpzxpEzxpzxp

EzxpE Txxzx

Txxzx

Txxzx ,ln,ln,

,

1,ln ,

0

,, ∇∇−

∇∇=∇∇

Lemma 3: Let be random vectors with joint density p (x,y), then: pn zx RR ∈∈ ,

( ) ( ) ( ) ( ) ( ) ( ) 0,,,,

1,

,

1

1

, =∇∇=

∇∇=

∇∇ ∫∫++

pnpn

zdxdzxpzdxdzxpzxpzxp

zxpzxp

E Txx

Txx

Txxzx

RR


14



nx R∈pz R∈

The Score of the estimation is defined by the logarithm of the likelihood ( )xzpx |ln∇ In Maximum Likelihood Estimation (MLE), this function returns a vector valued Score given by the observations and a candidate parameter vector .Score close to zero are good scores since they indicate that is close to a local optimum of , since

pz R∈ nx R∈x

( )xzp |

( ) ( ) ( )xzpxzp

xzp xx ||

1|ln ∇=∇

Since the measurement vector is stochastic the Expected Value of the Score is given by:

pz R∈

( ){ } ( ) ( )

( ) ( ) ( ) ( ) ( ) 0|||||

1

||ln|ln

1

=∇=∇=∇=

∇=∇

∫∫∫

∫

ppp

p

zdxzpzdxzpzdxzpxzpxzp

zdxzpxzpxzpE

xxx

xxz

RRR

R

v

( )vxh ,z

x

Estimatorx

The parameters are regarded as unknown but fixed. The measurements are

nx R∈ pz R∈

15


( ) ( ) ( ){ } ( ){ }xzpExzpxzpExJ Txxz

Txxz |ln|ln|ln: ∇∇−=∇∇=

SOLO

The Fisher Information Matrix (FIM)

Fisher, Sir Ronald Aylmer 1890 - 1962

The Fisher Information Matrix (FIM) was defined by Ronald AylmerFisher as the Covariance Matrix of the Score

( ){ } ( ) ( ) 0||ln|ln =∇=∇ ∫p

zdxzpxpxzpE xxz

R

The Expected Value of the Score is given by:

The Covariance of the Score is given by:

( ) ( ){ } ( ) ( ) ( )∫ ∇∇=∇∇p

zdxzpxzpxpxzpxzpE Txx

Txxz

R

||ln|ln|ln|ln

Nonrandom ParametersThe Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case

16

Fisher, Sir Ronald Aylmer (1890-1962)

The Fisher information is the amount of information that an observable random variable z carries about an unknown parameter x upon which the likelihood of z, L(x) = f (Z; x), depends. The likelihood function is the joint probability of the data, the Zs, conditional on the value of x, as a function of x. Since the expectation of the score is zero, the variance is simply the second moment of the score, the derivative of the lan of the likelihood function with respect to x. Hence the Fisher information can be written

( ) [ ]( ) [ ]( ){ } [ ]( ){ }x

k

xxx

Tk

x

k

x xZLExZLxZLEx ,ln,ln,ln: ∇∇−=∇∇=J



17


( ){ } ( ) rxn

yy

Ty

Trxr

yy

Tyyz ytMyzpEJ RR ∈∇=∈∇∇−=

== **:&|ln:


The Likelihood p (z|x) may be over-parameterized so that some of x or combination ofelements of x do not affect p (z|x). In such a case the FIM for the parameters x becomessingular. This leads to problems of computing the Cramér – Rao bounds. Let(r ≤ n) be an alternative parameterization of the Likelihood such that p (z|y) is a welldefined density function for z given and the corresponding FIM is non-singular.We define a possible non-invertible coordinate transformation .

ry R∈

ry R∈

( )ytx =

Theorem 1: Nonrandom Parametric Cramér – Rao Bound Assume that the observation has a well defined probability density function p (z|y)for all , and let denote the parameter that yields the true distribution of .Moreover, let be an Unbiased Estimator of , and let . The estimation error covariance of is bounded for below by

pz R∈ry R∈ *y y

( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx

( ) ( ){ } TTz MJMxxxxE 1*ˆ*ˆ −≥−−

where

are matrices that depend on the true unknown parameter vector .*y

18


( ){ } ( )**

:&|ln:yy

Ty

T

yy

Tyyz ytMyzpEJ

==∇=∇∇−=



pz R∈ry R∈ *y y

( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx ( ) ( ){ } TT

z MJMxxxxE 1*ˆ*ˆ −≥−−

where


Proof:

( ) ( )[ ] ( ){ } 0|ˆ =−∇∫p

zdyzpytzx Ty

R

Tacking the gradient w.r.t. on both sides of this relation we obtain:y

( ){ } ( ) ( )[ ] ( ){ } ( ) 0|ˆ| =∇−−∇ ∫∫pp

zdyzpytzdytzxyzp Ty

Ty

RR

( ){ } ( ) ( )[ ] ( ) ( ) ( )

1

||ˆ|ln ∫∫ ∇=−∇pp

zdyzpytzdyzpytzxyzp Ty

Ty

RR

( ){ } ( ) ( )[ ] ( ) ( )ytzdyzpytzxyzp Ty

Ty

p

∇=−∇∫R

|ˆ|ln

Consider the Random Vector:

( )

∇

−yzp

xx

y |ln

ˆwhere: ( )

{ }( ){ }

=

∇

−=

∇

−

0

0

|ln

ˆ

|ln

ˆ

yzpE

xxE

yzp

xxE

yz

z

yz

( )[ ] ( ) ( ) ( )[ ] ( )( )

0|ˆ|ˆˆ

sUnbiasenes

zxof

TT

pp

zdyzpytzxzdxzpxzx =−=− ∫∫RR

Using the Unbiasedness of Estimator:

19


( ){ } ( )**

:&|ln:yy

Ty

T

yy

Tyyz ytMyzpEJ

==∇=∇∇−=



pz R∈ry R∈ *y y

( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx ( ) ( ){ } TT

z MJMxxxxE 1*ˆ*ˆ −≥−−

where


Proof (continue – 1):

Consider the Random Vector: ( )

∇

−yzp

xx

y |ln

ˆ

The Covariance Matrix is Positive Semi-definite by construction:

( ){ }

( ){ }

=

∇

−=

∇

−

0

0

|ln

ˆ

|ln

ˆ

yzpE

xxE

yzp

xxE

yz

z

yz

( ) ( ) 00

0

0

0|ln

ˆ

|ln

ˆ1

11 definiteSemiPositive

T

T

T

T

yyz

IMJ

I

J

MJMC

I

JMI

JM

MC

yzp

xx

yzp

xxE

−

−

−−

≥

−

=

=

∇

−

∇

−

( ) ( ){ }Tz xxxxEC −−= ˆˆ: ( ) ( ){ } ( ){ }yzpEyzpyzpEJ T

yyzT

yyz |ln|ln|ln: ∇∇−=∇∇=

( )( ) ( ){ } ( )ytxxyzpEM Ty

Tyz

T ∇=−∇= ˆ|ln:

( ) ( ){ } TTz

NotationsEquivalent

definiteSemiPositive

T MJMxxxxECMJMC 11 ˆˆ:0 −−

− ≥−−=⇔≥−

( ){ } ( ) ( )[ ] ( ) ( )ytzdyzpytzxyzp Ty

Ty

p

∇=−∇∫R

|ˆ|lnWe found:

q.e.d.

where:

20


( ){ } ( ) nxn

yy

Ty

Tnxn

yy

Tyyz ybIMyzpEJ RR ∈∇+=∈∇∇−=

== **:&|ln:


Corollary 1: Nonrandom Parametric Cramér – Rao Bound (Baiased Estimator) Consider an estimaton problem defined by the likelihood p (y|z), and the fixed unknownparameter . Any estimator with unknown bias has a mean square errorbounded from below by

*y ( )zy ( )yb

( ) ( ){ } ( ) ( )***ˆ*ˆ 1 ybybMJMyyyyE TTTz +≥−− −

where


Proof:

Theorem 1 yields that:

Introduce the quantity , the estimator is an unbiased estimator of .( )ybyx +=: ( ) ( )zyzx ˆˆ = x

( ) ( ){ } ( )[ ] ( ){ }( ) ( )[ ]ybIyzpEybIxxxxE Ty

Tyyz

TTy

Tz ∇+∇∇−∇+≥−−

−1|lnˆˆ

Using , we obtain:( )ybyx +=:

( ) ( ){ } ( )[ ] ( ){ }( ) ( )[ ] ( ) ( )ybybybIyzpEybIyyyyE TTy

Tyyz

TTy

Tz +∇+∇∇−∇+≥−−

−1|lnˆˆ

after suitably inserting the true parameter .*y

21


[ ]( ) [ ]( ) [ ] [ ]( ) [ ]( ){ }( ) [ ] [ ] ( ) ( ) ( )

( ) [ ] ( ) ( ) ( )xbxbx

xbI

x

xZLE

x

xbI

xbxbx

xbI

x

xZL

x

xZLE

x

xbI

xZxxZxEZdxZLxZxxZx

T

x

kT

T

x

TkkT

x

TkkkkTkk

+

∂

∂+

∂∂

∂

∂+−=

+

∂

∂+

∂

∂

∂

∂

∂

∂+≥

−−=−−

−

−

∫

1

2

2

1

,ln

,ln,ln

,

SOLO

The Cramér-Rao Lower Bound on the Variance of the Estimator

The multivariable form of the Cramér-Rao Lower Bound is:

[ ]( )[ ]

[ ]

−

−=−

n

k

n

k

k

xZx

xZx

xZx

11

[ ]( ) [ ][ ]

[ ]

∂∂

∂∂

=

∂

∂=∇

n

k

k

kk

x

x

xZL

x

xZL

x

xZLxZL

,ln

,ln

,ln,ln

1

Fisher Information Matrix

[ ] [ ] [ ]

∂∂−=

∂

∂

∂

∂=x

k

x

Tkk

x

xZLE

x

xZL

x

xZLE

2

2 ,ln,ln,ln:J



22


Random Parameters

Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)

( ) ( )[ ] ( )∫ −=p

zdyzpytxybR

|ˆ

( ){ } ( ){ } rxnTyz

TrxrTyyyz ytEMyzpEJ RR ∈∇=∈∇∇−= :&,ln: ,

where

then the Mean Square of the Estimate is Bounded from Below

ynrt RR →: x

For Random Parameters there is no true parameter value. Instead, the prior assumptionon the parameter distribution determines the probability of different parameter vectors.Like in the nonrandom parametric case, we assume a possible non-invertible mapping between a parameter vector and the sought parameter . The vectoris assumed to have been chosen such that the joint probability density p (y,z) is a welldefined density.

y

Let be two random vectors with a well defined joint densityp (y,z), and let be an estimate of . If the estimator bias

pr zandy RR ∈∈( ) nzx R∈ˆ ( )ytx =

satisfies ( ) ( ) njandriallforypyb jzi

,,1,,10lim ===±∞→

( ) ( ){ } TTyz MJMxxxxE 1

, ˆˆ −≥−− ( ) ( ){ } 0ˆˆ 1,


TTyz MJMxxxxE

−− ≥−−−

EquivalentNotations

23


Random Parameters



, ˆˆ −≥−− ( ){ } ( ){ }ytEMyzpEJ Tyz

TTyyyz ∇=∇∇−= :&,ln: ,


Proof:



( ) ( )[ ] ( )∫ −=p

zdyzpytxybR

|ˆ and ( ) ( ) njandriallforypyb jzi

,,1,,10lim ===±∞→

Compute

( ) ( )[ ] ( ) ( )[ ] ( ) ( )( )

( ) ( )

( )

( )[ ] ( ) ( )[ ]∫∫∫ −∇+−∇=−∇=∇ppp

zdytzxyzpzdyzpytzdypyzpytzxypyb Ty

yp

Ty

yzp

Ty

Ty

RRR

ˆ,,|ˆ,

Integrating both sides w.r.t. over its complete range R r yieldsy( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( )[ ]∫∫∫

+

−∇+∇−=∇rprr

ydzdytzxyzpydypytydypyb Ty

Ty

Ty

RRR

ˆ,

The (i,j) element of the left hand side matrix is:( ) ( ) ( ) ( ) ( ) ( ) riiiiyjyj

i

j ydydydydydydypybypybydy

ypyb

r iir

111

00

0 +−−−−∞=+∞===

−=

∂∂

∫∫RR

24


Random Parameters



, ˆˆ −≥−− ( ){ } ( ){ }ytEMyzpEJ Tyz

TTyyyz ∇=∇∇−= :&,ln: ,




( ) ( )[ ] ( )∫ −=p

zdyzpytxybR

|ˆ and ( ) ( ) njandriallforypyb jzi

,,1,,10lim ===±∞→

Proof (continue – 1): We found ( )[ ] ( ) ( )[ ] ( ) ( )∫∫ ∇=−∇+ rrp

ydypytydzdytzxyzp Ty

Ty

RR

ˆ,

( )[ ] ( ) ( )[ ] ( ) ( ) ( ) ( ){ }ytEydypytydzdyzpytzxyzp Tyz

Ty

Ty

rrp

∇=∇=−∇ ∫∫+ RR

,ˆ,ln

Consider the Random Vector: ( )

∇

−yzp

xx

y ,ln

ˆ

The Covariance Matrix is Positive Semi-definite by construction:

( ) ( ) 00

0

0

0,ln

ˆ

,ln

ˆ1

11

,


T

T

T

T

yyyz

IMJ

I

J

MJMC

I

JMI

JM

MC

yzp

xx

yzp

xxE

−

−

−−

≥

−

=

=

∇

−

∇

−

( )( ) ( ){ } ( ){ }ytExxyzpEM Tyz

Tyyz

T ∇=−∇= ˆ,ln: ,

( ) ( ){ } TTz

NotationsEquivalent


T MJMxxxxECMJMC 11 ˆˆ:0 −−

− ≥−−=⇔≥− q.e.d.

( ) ( ){ }Tyz xxxxEC −−= ˆˆ: , ( ) ( ){ } ( ){ }yzpEyzpyzpEJ T

yyyzT

yyz ,ln,ln|ln: , ∇∇−=∇∇=where:

( ){ }

( ){ }

=

∇

−=

∇

−

0

0

,ln

ˆ

,ln

ˆ

,

,

, yzpE

xxE

yzp

xxE

yyz

yz

yyz


25


Nonrandom and Random Parameters Cramér – Rao Bounds

For the Nonrandom Parameters the Cramér – Rao Bound depends on the true unknownparameter vector y , and on the model of the problem defined by p (z|y) and the mapping x = t (y). Hence the bound can only be computed by using simulations, when the true valueof the sought parameter vector y is known.

For the Random Parameters the Cramér – Rao Bound can be computed even in realapplications. Since the parameters are random there is no unknown true parameter value.

Instead, in the posterior Cramér – Rao Bound the matrices J and M are computed bymathematical expectation performed with respect to the prior distribution of the parameters.


26



( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,

, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−

( )0xpIn addition the P.D.F. of the initial state , is also given.

We found that the Cramér – Rao Lower Bound for the Random Parameters is given by:

( ) ( ){ } ( ) ( ){ } ( ){ } 1

:1:1,

1

:1:1:1:1,:1:1|:1:1:1|:1, ,ln,ln,ln:1:1:1:1

−−∇∇−=∇∇≥−− kk

TXXZXkk

TXkkXZX

T

kkkkkkZX XZpEXZpXZpEXXXXEkkkk

( )1−= kk xfxIf we have a deterministic state model, i.e. then we can use the Nonrandom

Parametric Cramér – Rao Lower Bound ( ) ( ){ } ( ) ( ){ } ( ){ } 1

:1:1

1

:1:1:1:1:1:1|:1:1:1|:1 |ln|ln|ln:1:1:1:1

−−∇∇−=∇∇≥−− kk

TXXZkk

TXkkXZ

T

kkkkkkZ XZpEXZpXZpEXXXXEkkkk

After k cycles we have k measurements and k random parameters estimated by an Unbiased Estimator as .

[ ]Tkk zzzZ ,,,: 21:1 =[ ]Tkk xxxxX ,,,,: 210:0 = [ ]Tkkkk xxxX |2|21|1:1|:1 ˆ,,ˆ,ˆ:ˆ =

The CRLB provides a lower bound for second-order (mean-squared) error only. Posteriordensities, which result from Nonlinear Filtering, are in general non-Gaussian. A full statistical characterization of a non-Gaussian density requires higher order moments, inaddition to mean and covariance. Therefore, the CRLB for Nonlinear Filtering does notfully characterize the accuracy of Filtering Algorithms.

27



Theorem 3: The Cramér – Rao Lower Bound for the Random Parameters is given by:

Let perform the partitioning [ ] 11:1:1 ,: xnkT

kkk xXX R∈= − [ ] 1|1:1|1:1:1|:1 ˆ,ˆ:ˆ xnkT

kkkkkk xXX R∈= −−

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,





( ){ } ( ) ( )

( ){ } ( )

( ){ } nxnkk

TxxZXk

nxknkk

TxXZXk

knxknkk

TXXZXk

XZpEC

XZpEB

XZpEA

kk

kk

kk

R

R

R

∈∇∇−=

∈∇∇−=

∈∇∇−=−

−−

−

−−

:1:1,

1:1:1,

11:1:1,

,ln:

,ln:

,ln:

1:1

1:11:1

( ) ( ){ } ( ) nxnkk

Tkkk

TkkkkkkZX BABCJxxxxE R∈−=≥−− −−− 111

||, :ˆˆ( ) ( ){ } ( ) 0ˆˆ11

||,


kkTkk

TkkkkkkZX BABCxxxxE

−−− ≥−−−−

EquivalentNotations

28



The Cramér – Rao Bound for the Random Parameters is given by:

( ) ( ){ } 0,ln,lnˆˆ

1

:1:1,:1:1,,

|

1:11:1|1:1

|

1:11:1|1:1, 1:11:1


kkT

xXkkxXZX

T

kkk

kkk

kkk

kkkZX XZpXZpE

xx

XX

xx

XXE

kkkk

−−−−−−−− ≥∇∇−

−

−

−

−−−

Proof Theorem 3: Let perform the partitioning [ ] 11:1:1 ,: xnkT

kkk xXX R∈= − [ ] 1|1:1|1:1:1|:1 ˆ,ˆ:ˆ xnkT

kkkkkk xXX R∈= −−

( ){ } ( ){ } ( ){ }( ){ } ( ){ }

1

:1:1,:1:1,

:1:1,:1:1,1

:1:1,,,,ln,ln

,ln,ln,ln

1:1

1:11:11:1

1:11:1

−

−

∇∇−∇∇−

∇∇−∇∇−=∇∇−

−

−−−

−−

kkT

xxZXkkT

XxZX

kkT

xXZXkkT

XXZX

kkT

xXxXZXXZpEXZpE

XZpEXZpEXZpE

kkkk

kkkk

kkkk

nkxnkkk

kkTkk

k

kTkk

Tk

kk

I

BAI

BABC

A

IAB

I

CB

BAR∈

−

=

=

−−

−−

− 11

11

1

00

00:

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,





29



( ) ( ){ } ( ) ( ){ }( ) ( ){ } ( ) ( ){ }

( ) 00

0

0

0

ˆˆˆ

ˆ

1

111

111

||,1:11:1|1:1|,

|1:11:1|1:1,1:11:1|1:11:11:1|1:1,


kTkkk

Tkk

kkk

TkkkkkkZX

T

kkkkkkZX

TkkkkkkZX

T

kkkkkkZX

IAB

I

BABC

A

I

BAI

xxxxEXXxxE

xxXXEXXXXE

−−

−−−

−−−

−−−

−−−−−−−−−

≥

−

−

−−−−

−−−−

Proof Theorem 3 (continue – 1): We found

( ) ( ){ } ( ) ( ){ }( ) ( ){ } ( ) ( ){ }

( ) 00

0

0

ˆˆˆ

ˆ

0

11

1

1

||,1:11:1|1:1|,

|1:11:1|1:1,1:11:1|1:11:11:1|1:1,1


kkTkk

k

kTk

TkkkkkkZX

T

kkkkkkZX

TkkkkkkZX

T

kkkkkkZXkk

BABC

A

IAB

I

xxxxEXXxxE

xxXXEXXXXE

I

BAI

−

−−

−

−

−−−

−−−−−−−−−−

≥

−−

−−−−

−−−−

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



30


( ) ( ){ } ( ) 111||, :ˆˆ

−−− −=≥−− kkTkkk

TkkkkkkZX BABCJxxxxE

SOLO


Prof Theorem 3 (continue – 2): We found

( ) ( ){ } ( ) 00

0

ˆˆ*

**11

1

||,


kkTkk

k

TkkkkkkZX BABC

A

xxxxE

−

−−

−

≥

−−

−−

( ) ( ){ } ( ) 0ˆˆ11

||,


kkTkk

TkkkkkkZX BABCxxxxE

−−− ≥−−−−

EquivalentNotations

( ){ } ( ) ( )

( ){ } ( )

( ){ } nxnkk

TxxZXk

nxknkk

TxXZXk

knxknkk

TXXZXk

XZpEC

XZpEB

XZpEA

kk

kk

kk

R

R

R

∈∇∇−=

∈∇∇−=

∈∇∇−=−

−−

−

−−

:1:1,

1:1:1,

11:1:1,

,ln:

,ln:

,ln:

1:1

1:11:1

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



q.e.d.

31


( ){ } ( ) ( )

( ){ } ( )

( ){ } nxnkk

TxxZXk

nxknkk

TxXZXk

knxknkk

TXXZXk

XZpEC

XZpEB

XZpEA

kk

kk

kk

R

R

R

∈∇∇−=

∈∇∇−=

∈∇∇−=−

−−

−

−−

:1:1,

1:1:1,

11:1:1,

,ln:

,ln:

,ln:

1:1

1:11:1

SOLODiscrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound

We found

We want to compute Jk recursively, without the need for inverting large matrices as Ak.

( ) ( ){ } ( ) 111||, :ˆˆ

−−− −=≥−− kkTkkk


( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



Theorem 4:The Recursive Cramér–Rao Lower Bound for the Random Parameters is given by:

( ) ( ){ } [ ]( ) nxnkkkkkk

TkkkkkkZX DDJDDJxxxxE R∈+−=≥−−

−−−+−−+−−+

11211121221

111|111|1, :ˆˆ

( ) ( ){ } ( ) nxnkk

Tkkk

TkkkkkkZX BABCJxxxxE R∈−=≥−−

−−− 111||, :ˆˆ

( ){ }( )[ ]{ } [ ]

( ){ } ( ){ } nxnkk

Tkxxzkk

Tkxxxk

nxnT

kkkTkxxxk

nxnkk

Tkxxxk

xzpExxpED

DxxpED

xxpED

kkkkkk

kkk

kkk

R

R

R

∈∇∇−∇∇−=

∈=∇∇−=

∈∇∇−=

+++++

++

+

+++++

+

+

111|11|22

2111|

12

1|11

|ln|ln:

|ln:

|ln:

11111

1

1

( ) ( ){ }000 lnln000

xpxpEJ Txxx ∇∇=The recursions start with the initial

information matrix J0,

32


( ){ }( ){ }

( ){ }kkT

xxZXk

kkT

xXZXk

kkT

XXZXk

XZpEC

XZpEB

XZpEA

kk

kk

kk

:1:1,

:1:1,

:1:1,

,ln:

,ln:

,ln:

1:1

1:11:1

∇∇−=

∇∇−=

∇∇−=

−

−−

We found

We want to compute Jk recursively, without the need for inverting large matrices as Ak.

( ) ( ){ } ( ) 111||, :ˆˆ

−−− −=≥−− kkTkkk


Start with:

( ) ( ) ( ) ( )kkkkkkkkkkkkk XxZpXxZzpXxZzpXZp :11:1:11:11:11:111:11:1 ,,,,|,,,, +++++++ ==( )

( )

( )( )

( )kk

xxpMarkov

kkk

xzpMarkov

kkkk XZpXZxpXxZzp

kkkk

:1:1

|

:1:11

|

:11:11 ,,|,,|

111

+++ →

+

→

++=

( ) ( ) ( )1:11:11 ,|| −−−= kkkkkk XZpxxpxzp

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



Proof of Theorem 4:


33


( ) 11

1

111

111

111

1

1:11:1,

11|1

|

1:11:1|1:1

11|1

|

1:11:1|1:1

, :,ln

ˆ

ˆ

ˆ

ˆ

1111:11

11:1

11:11:11:11:1

−+

−

+++

+++

+++

−

++

+++

−−−

+++

−−−

=

=

∇∇∇∇∇∇

∇∇∇∇∇∇

∇∇∇∇∇∇

−≥

−

−

−

−

−

−

+++−+

+−

+−−−−

k

kTk

Tk

kkTk

kkk

kk

Txx

Txx

TXx

Txx

Txx

TXx

TxX

TxX

TXX

ZX

T

kkk

kkk

kkk

kkk

kkk

kkk

ZX I

FEL

ECB

LBA

XZpE

xx

xx

XX

xx

xx

XX

E

kkkkkk

kkkkkk

kkkkkk

SOLO

Proof of Theorem 4 (continue – 1):

Compute:

( ) ( ) ( ) ( )kkkkkkkk XZpxxpxzpXZp :1:11111:11:1 ,||, +++++ =

( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } kkk

TXXZX

kkkkkkT

XXZXkkT

XXZXk

AXZpE

XZpxxpxzpEXZpEA

kk

kkkk

=∇∇−+=

++∇∇−=∇∇−=

−−

−−−− ++++++

:1:1,

:1:1111,1:11:1,1

,ln00

,ln|ln|ln,ln:

1:11:1

1:11:11:11:1

( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } kkk

TxXZX

kkkkkkT

xXZXkkT

xXZXk

BXZpE

XZpxxpxzpEXZpEB

kk

kkkk

=∇∇−+=

++∇∇−=∇∇−=

−

−− ++++++

:1:1,

:1:1111,1:11:1,1

,ln00

,ln|ln|ln,ln:

1:1

1:11:1

( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } ( )[ ]{ } 11

:1:1,1|

:1:1111,1:11:1,1

,ln|ln0

,ln|ln|ln,ln:

11

1 kk

C

kkT

xxZX

D

kkT

xxxx

kkkkkkT

xxZXkkT

xxZXk

DCXZpExxpE

XZpxxpxzpEXZpEC

k

kk

k

kkkk

kkkk

+=∇∇−∇∇−=

++∇∇−=∇∇−=

+

++++++

+

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,




34


( ) 11

1

111

111

111

1

1:11:1,

11|1

|

1:11:1|1:1

11|1

|

1:11:1|1:1

, :,ln

ˆ

ˆ

ˆ

ˆ

1111:11

11:1

11:11:11:11:1

−+

−

+++

+++

+++

−

++

+++

−−−

+++

−−−

=

=

∇∇∇∇∇∇

∇∇∇∇∇∇

∇∇∇∇∇∇

−≥

−

−

−

−

−

−

+++−+

+−

+−−−−

k

kTk

Tk

kkTk

kkk

kk

Txx

Txx

TXx

Txx

Txx

TXx

TxX

TxX

TXX

ZX

T

kkk

kkk

kkk

kkk

kkk

kkk

ZX I

FEL

ECB

LBA

XZpE

xx

xx

XX

xx

xx

XX

E

kkkkkk

kkkkkk

kkkkkk

SOLO


Compute:

( ) ( ) ( ) ( )kkkkkkkk XZpxxpxzpXZp :1:11111:11:1 ,||, +++++ =

( ){ } ( ) ( ) ( )[ ]{ }

( )[ ] ( )[ ] ( )[ ] 0,ln|ln|ln

,ln|ln|ln,ln:

0

:1:1,

0

1,

0

11,

:1:1111,1:11:1,1

11:111:111:1

11:111:1

=

∇∇−

∇∇−

∇∇−=

++∇∇−=∇∇−=

+−+−+−

+−+−

+++

++++++

kkT

xXZXkkT

xXZXkkT

xXZX

kkkkkkT

xXZXkkT

xXZXk

XZpExxpExzpE

XZpxxpxzpEXZpEL

kkkkkk

kkkk

( ){ } ( ) ( ) ( )[ ]{ }

( )[ ] ( )[ ]{ } ( )[ ] ( )[ ]{ } 121|

0

:1:1,1,

0

11,

:1:1111,1:11:1,1

:|ln,ln|ln|ln

,ln|ln|ln,ln:

11111

11

kkkT

xxxxkkT

xxZXkkT

xxZXkkT

xxZX

kkkkkkT

xxZXkkT

xxZXk

DxxpEXZpExxpExzpE

XZpxxpxzpEXZpEE

kkkkkkkkkk

kkkk

=∇∇−=

∇∇−∇∇−

∇∇−=

++∇∇−=∇∇−=

++++

++++++

+++++

++

( ){ } ( ) ( ) ( )[ ]{ }

( )[ ]{ } ( )[ ]{ } ( )[ ] 22

0

:1:1,1|11|

:1:1111,1:11:1,1

,ln|ln|ln

,ln|ln|ln,ln:

111111111

1111

kkkT

xxZXkkT

xxxxkkT

xxxz

kkkkkkT

xxZXkkT

xxZXk

DXZpExxpExzpE

XZpxxpxzpEXZpEF

kkkkkkkkkk

kkkk

=

∇∇−∇∇−∇∇−=

++∇∇−=∇∇−=

+++++++++

++++

+++

++++++

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,




35


1

2221

1211

1

111

111

111

11

11|1

|

1:11:1|1:1

11|1

|

1:11:1|1:1

,

0

0

:

ˆ

ˆ

ˆ

ˆ

−−

+++

+++

+++

−+

+++

−−−

+++

−−−

+=

=≥

−

−

−

−

−

−

kk

kkkTk

kk

kTk

Tk

kkTk

kkk

k

T

kkk

kkk

kkk

kkk

kkk

kkk

ZX

DD

DDCB

BA

FEL

ECB

LBA

I

xx

xx

XX

xx

xx

XX

E

SOLO


We found:

[ ] [ ]

+

+

−

+

+

=

−

−−

+

I

DDCB

BAI

DDCB

BADD

DCB

BA

IDCB

BAD

I

Ikkk

Tk

kk

kkkTk

kk

kk

kkTk

kk

kkTk

kk

k

k

0

0

000

0

0

0

12

1

11

12

1

11

2122

11

1

11

211

Therefore: ( ) ( ){ }[ ] [ ] 1211112122

12

1

11

21221

1111|111|1,

00:

ˆˆ

kkkTkkkkk

kkkTk

kk

kkk

kT

kkkkkkZX

DBABDCDDDDCB

BADDJ

JxxxxE

−−

−

+

−+++++++

−+−=

+

−=

≥−−

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,




36


The recursions start with the initial information matrix J0, which can be computed from the initial density p (x0) as follows:

( ) ( ){ } [ ]( ) 11211121221

111|111|1, :ˆˆ−−−

+−−+−−+ +−=≥−− kkkkkkT

kkkkkkZX DDJDDJxxxxE

( ){ }( ){ }

( ){ }kkT

xxZXk

kkT

xXZXk

kkT

XXZXk

XZpEC

XZpEB

XZpEA

kk

kk

kk

:1:1,

:1:1,

:1:1,

,ln:

,ln:

,ln:

1:1

1:11:1

∇∇−=

∇∇−=

∇∇−=

−

−−


( ) ( ){ } ( ) 111||, :ˆˆ

−−− −=≥−− kkTkkk


( ){ }( )[ ]{ } [ ]

( ){ } ( ){ }11|1|22

211|

12

1|11

|ln|ln:

|ln:

|ln:

1111111

11

1

+++

+

+

+++++++

++

+

∇∇−∇∇−=

=∇∇−=

∇∇−=

kkTxxxzkk

Txxxxk

T

kkkTxxxxk

kkTxxxxk

xzpExxpED

DxxpED

xxpED

kkkkkkkk

kkkk

kkkk

( ) ( ){ }000 lnln000

xpxpEJ Txxx ∇∇=

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,




37


( ) ( ){ } [ ]( ) 11211121221

111|111|1, :ˆˆ−−−

+−−+−−+ +−=≥−− kkkkkkT

kkkkkkZX DDJDDJxxxxE


( ){ }( )[ ]{ } [ ]

( ) ( )( ) ( ){ } ( ) ( ){ } nxn

kkTxxxzk

nxnkk

Txxxxk

kkk

nxnT

kkkTxxxxk

nxnkk

Txxxxk

xzpEDxxpED

DDD

DxxpED

xxpED

kkkkkkkk

kkkk

kkkk

RR

R

R

∈∇∇−=∈∇∇−=

+=

∈=∇∇−=

∈∇∇−=

+++

+

+

+++++++

++

+

11|22

1|22

222222

211|

12

1|11

|ln:2|ln:1

21:

|ln:

|ln:

1111111

11

1

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



q.e.d.

( ) [ ] ( )

tMeasuremenUpdated

22

ModelProcessUsingPrediction

1211121221 21: kkkkkkk DDDJDDJ ++−= −

+


38



( ) ( ) ( ) ( )

−−−== −

001

000

0

0000 ˆˆ2

1exp

2

1,ˆ;

0xxPxx

PPxxxp T

xπ

N

( )( ) p

kkk

nkkk

vxhz

wxfx

R

R

∈=

∈= −−

,



Probability Density Function of is Gaussian0x

( ) ( ) ( ) ( )001

0001

0000 ˆˆˆ2

1ln

000xxPxxPxxcxp T

xxx −−=

−−−∇=∇ −−

( ) ( ){ } ( ) ( ) [ ]{ }( ) ( ){ } 1

01

001

01

000001

0

100000

10000

ˆˆ

ˆˆlnln

0

000000

−−−−−

−−

==−−=

−−=∇∇=

PPPPPxxxxEP

PxxxxPExpxpEJT

x

TTxx

Txxxx


39



( ) ( ) ( ) ( )( ) ( )( )

−−−=== +

−++ kkkk

Tkkk

k

kkkwkk xfxQxfxQ

Qwwpxxp 11

11 2

1exp

2

1,0;|

πN

( )( ) p

kkkk

nkkkk

vxhz

wxfx

R

R

∈+=

∈+=

++++

+

1111

1

1& +kk vw are system and measurement Gaussian white-noise sequences, independent of past and current states and on each other with covariances Qk and Rk+1, respectively



( ) ( )( ) ( )( ) ( )[ ] ( )( )kkkkkT

kxkkkkT

kkkxkkx xfxQxfxfxQxfxcxxpkkk

−∇=

−−−∇=∇ +

−+

−++ 1

11

111 2

1|ln

( ) ( ) ( ) ( )( ) ( )( )

−−−=== +++

−++++

++++++ 111

11111

1

11111 2

1exp

2

1,0;| kkkk

Tkkk

k

kkkvkk xhzRxhzR

Rvvpxzpπ

N

( ) ( )( ) ( )( ) ( )[ ] ( )( )1111111111

11111211 111 2

1|ln +++

−++++++

−++++++ −∇=

−−−∇=∇

+++ kkkkkTkxkkkk

Tkkkxkkx xhzRxhxhzRxhzcxzp

kkk

( ) ( )( ) ( )[ ]{ } ( )[ ] 1111 11

|ln −−++ ∇=∇−∇=∇∇

++ kkT

kx

T

kT

kxkT

kkkxkkTxx QxfxfQxfxxxp

kkkkk

( )[ ] ( )[ ]TkTkxk

T

kT

kxk xhHxfFkk 111

:~

&:~

+++∇=∇=

40


SOLO


( )( ) p

kkkk

nkkkk

vxhz

wxfx

R

R

∈+=

∈+=

++++

+

1111

1




( )[ ] ( )( ) ( )( ) ( )[ ]{ } { }1111|111111111

1111|

~~111111 +

−++++

−+++++++

−+++ ++++++

=∇−−∇= kkTkxz

Tk

Tkx

Tk

Tkkkkkkkk

Tkxxz HRHExhRxhzxhzRxhE

kkkkkk

( ){ } ( ){ } { } 1|

1|1|

12 ~|ln

1111

−−+ ++++

−=∇−=∇∇−= kT

kxxkkkTxxxkk

Txxxxk QFEQxfExxpED

kkkkkkkkk

( )[ ] ( )[ ]TkTkxk

T

kT

kxk xhHxfFkk 111

:~

&:~

+++∇=∇=

( ){ } ( ) ( ){ }( )[ ] ( )( ) ( )( ) ( )[ ]{ }

{ }kkT

kxx

T

kT

kxT

kT

kkkkkkkkT

kxxx

kkTxkkxxxkk

Txxxxk

FQFE

xfQxfxxfxQxfE

xxpxxpExxpED

kk

kkkk

kkkkkkkk

~~

|ln|ln|ln:

1|

?

111

|

11|1|11

1

1

11

−

−++

−

+++

+

+

++

=

∇−−∇=

∇∇=∇∇−=

( ) ( ){ } ( ) ( ){ }1111|11|22 |ln|ln|ln:2

11111111 ++++++ ++++++++∇∇=∇∇−= kk

Txkkxxzkk

Txxxzk xzpxzpExzpED

kkkkkkkk

The Jacobians of computed at , respectively.

( ) ( )11& ++ kkkk xhxf

1& +kk xx

( ) ( ){ } ( )( )[ ]{ } 11

1|1|

22

11111|ln:1 −

+−

+ =−∇=∇∇−=+++++ kkkkk

Txxxkk

Txxxxk QxfxQExxpED

kkkkkkk

41


{ }{ }

( )( ) { }1

111|

22

122

1|

12

1|

11

~~2

1

~

~~

11

1

1

+−++

−

−

−

++

+

+

=

=

−=

=

kkTkxzk

kk

kT

kxxk

kkT

kxxk

HRHED

QD

QFED

FQFED

kk

kk

kk

SOLO


( )( ) p

kkkk

nkkkk

vxhz

wxfx

R

R

∈+=

∈+=

++++

+

1111

1




( )[ ] ( )[ ]TkTkxk

T

kT

kxk xhHxfFkk 111

:~

&:~

+++∇=∇= The Jacobians of

computed at , respectively.( ) ( )11& ++ kkkk xhxf

1& +kk xx

( ) [ ] ( )

tMeasuremenUpdated

22


1211121221 21: kkkkkkk DDDJDDJ ++−= −

+

We can calculate the expectations using a Monte CarloSimulation. Using we draw ( ) ( ) ( )01 &, xpvpwp kk +( )

( ) ( ) Nivpvwpw

xpx

kikk

ik ,,2,1~&~

~

11

00

=++

We Simulate System States and Measurements( )

( ) Nivxhz

wxfxik

ikk

ik

ik

ikk

ik ,,2,1

1111

1=

+=

+=

++++

+

We then average over realizations to get J0.We average over realization to get next terms and so forth.

0x1x


42


( ) ( ) 1111

22122112111 2&1&& +−

++−−− ==−== kk

Tkkkkk

Tkkkk

Tkk HRHDQDQFDFQFD

SOLO


pkkkk

nkkkk

vxHz

wxFx

R

R

∈+=

∈+=

++++

+

1111

1



Linear/ Gaussian System

( ) ( ) 1111

11

tsMeasuremenUpdated

1111


111111 +

−++

−−+

−++

−−−−−+ ++=++−= kk

Tk

Tkkkk

LemmaInverseMatrix

kkTkk

Tkkk

Tkkkkkk HRHFJFQHRHQFFQFJFQQJ

Define ( )Tkkkkkkkkkkkkk FPFQPPJPJ |

1|1

1|

11|11 :&:&: +=== −

+−−

+++

( ) 1111

1|11

111

1

|1

1|1 +−++

−++

−++

−−++ +=++= kk

Tkkkkk

Tk

Tkkkkkkk HRHPHRHFPFQP

The conclusion is that CRLB for the Linear Gaussian Filtering Problem is Equivalent to the Covariance Matrix of the Kalman Filter. Return to Table of Content

43



pkkkk

nkkk

vxHz

xFx

R

R

∈+=

∈=

++++

+

1111

11+kv are measurement Gaussian white-noise sequence,

independent of past and current states with covariance Rk+1.Qk = 0.


Linear System with Zero System Noise

Define ( ) 1

|

01

|11

|1

1|11 :&:&:−=

−+

−−+++ === T

kkkk

Q

kkkkkkkk FPFPPJPJk

( ) 1111

1|11

111

1

|1

1|1 +−++

−++

−++

−−++ +=+= kk

Tkkkkk

Tk

Tkkkkkk HRHPHRHFPFP


44


References

http://en.wikipedia.org/wiki/Cramer_Rao_bound

Bergman, N., “Recursive Bayesian Estimation - Navigation and Tracking Applications”, PhD Thesis, Linköping University, 1999, Dissertation No. 579, Ch. 4

Van Trees, H., L., “Detection, Estimation and Modulation Theory”, Wiley, New York, 1968, 2001, pp. 146, 66, 72, 79,84,

Tichavský, P., Muravchik, C, Nehorai. A., “Posterior Cramér – Rao bounds for Discrete-Time Nonlinear Filtering”, IEEE Transactions on Signal Processing, 46(5), 1998, pp. 1386 - 1396

Ristic, B., Arulampalam, S., N., Gordon, N., “Beyond the Kalman Filter – Particle Filters for Tracking Applications”, Artech House, 2004, Ch. 4: “Cramér – Rao Bounds for Nonlinear Filtering”

Ristic, B., “Cramér – Rao Bounds for Target Tracking”, Int. Conf. Intelligent Sensors, Sensor Networks and Information Processing, 6 Dec., 2005, http://www.issnip.org/2005/branko_05.pdf

Van Trees, H., L., “Bayesian Bounds”, Keynote Speech, 2005 Adaptive Sensor and Array Processing Workshop, 7 June 2005, http://www.ll.mit.edu/asap/asap_05/pdf/Presentations/01_vantrees.pdf


January 11, 2015 45

SOLO

TechnionIsraeli Institute of Technology

1964 – 1968 BSc EE1968 – 1971 MSc EE

Israeli Air Force1970 – 1974

RAFAELIsraeli Armament Development Authority

1974 – 2013

Stanford University1983 – 1986 PhD AA

46

Harry L. Van Trees

http://teal.gmu.edu/faculty_info/van.html

Harald Cramér1893 – 1985

Cayampudi RadhakrishnaRao

1920 -


Branko Ristic

Niclas Bergman

Arye Nehorai

Carlos H. Muravchik

Petr Tichavsky

5 cramer-rao lower bound

Science

Transcript of 5 cramer-rao lower bound