CWIT 2005 1

Robust Entropy Rate for Uncertain Sources: Applications to Communication and Control Systems

Charalambos D. CharalambousDept. of Electrical and Computer Engineering University of Cyprus, Cyprus.Also with the School of Information Technology and EngineeringUniversity of Ottawa, Canada.and

Alireza FarhadiSchool of Information Technology and EngineeringUniversity of Ottawa, Canada.

2CWIT 2005

Overview

Robust Entropy

Solution and Relations to Renyi Entropy

Examples Uncertainty in Distribution Uncertainty in Frequency Response

Stabilization of Communication-Control Systems

CWIT 2005 3

Applications

4CWIT 2005

Applications: Robust Lossless Coding Theorem [1]

Source words of block length produced by a discrete uncertain memory less source with Shannon entropy can be encoded into code words of block length from a coding alphabet of size with arbitrary small probability of error if

n

SUDf

)( fHSr D

Dn

rfHS

Df SU

log)(sup

Uncertain Source

nXn n rX̂

Encoder Decoder Destination)( fHS

Uniform Source Encoder

5CWIT 2005

Applications: Observability and Stabilizability of Networked Control Systems [2]

Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encoder and decoder such that

where is the joint density of is the source density uncertainty set. and are fixed and .

1

02 ,)||

~Pr(||

1suplim

t

kkk

DftYY

tSU

SUDf .),...,( 10tr

tYY SUD0 10

21

2 )(|||| yyy tr

6CWIT 2005

Applications: Observability and Stabilizability of Networked Control Systems [2]

Definition. The uncertain controlled source is uniform asymptotically stabilizable in probability if there exists encoder, decoder and controller such that

For and small enough, a necessary condition for uniform observability and stabilizability in probability is

1

02 .)||Pr(||

1suplim

t

kk

DftY

tSU

).()(sup1

lim

robustSU

Dftrobust fH

tC

SU

CWIT 2005 7

Problem Formulation, Solution and Relations

8CWIT 2005

Problem Formulation

Let be the space of density functions defined on , be the true unknown source density, belonging to the uncertainty set . Let also be the fixed nominal source density.

Definition. The robust entropy of observed process having density function is defined by

where is the Shannon entropy and

D d)(yf

DDSU )(yg

SUDyf )(Y

),(sup)( * fHf SDf

robustSU

(.)SH ).(suparg* fHf SDf SU

9CWIT 2005

Problem Formulation

Definition. In Previous definition, if represent a sequence of R.V.’s with length of source symbols produced by the uncertain source with joint density , the robust entropy rate is defined by

provided the limit exits.

trTYYY ),...,( 10

T

)()( TdSU yDyf

),(1

lim)( *fHT robust

Trobust

10CWIT 2005

Solution to the Robust Entropy

Let the uncertain set be defined bywhere is the relative entropy and is fixed.

Lemma. Given a fixed nominal density and the uncertainty defined above, the solution to the robust entropy is given by

where the minimizing in above is the unique solution of .

},)|(;{ cSU RgfHDfD

.)|(.H ),0[ cR

)(yg SUD

,

)(

)()(

),(])(ln)1([min)(

1

1*,

*,10

*, **

dyyg

ygyf

fHdyygssRfH

s

s

s

s

s

sRrobust

s

s

cs

srobust

c

0* s

cs RgfH )|(

**,

11CWIT 2005

Solution to the Robust Entropy

Corollary [1]. Suppose , is uniform Probability Mass Function (PMF), that is

When and consequently Correspond to PMF’s, that is

the previous solution is reduced to

)|( ghHRc )(yh

.1

)(,)()()(1 M

yhyyhyh i

M

iii

)(yg )(yf

M

iii yygyg

1

)()()(

M

iii yyfyf

1

)()()(

.1,

)(

)()(

],)(ln)1([min)(

1

1*,

1

10

*, *

Mi

yg

ygyf

ygssRfH

i

s

s

i

s

s

ii

s

M

i

s

s

ics

sRrobust

c

12CWIT 2005

Solution to the Robust Entropy Rate

Corollary. Let be a sequence with length of source symbols with uncertain joint density function and let we exchange with . Then, the robust entropy rate is given by

where the minimizing in above is the unique solution of

.

trTYYY ),...,( 10 T Td

SU yDyf ,)(

cR cTR

,

)(

)()(

],)(ln)1([min)(

),(1

lim)(

1

1*,

10

*,

*,

*

*

dyyg

ygyf

dyygssTRfH

fHT

s

s

s

s

s

s

s

cs

sTRrobust

sTRrobust

Trobust

c

c

0* s

cs TRgfH )|(

**,

13CWIT 2005

Robust Entropy Rate of Uncertain Sources with Nominal Gaussian Source Density

Example. From Previous result follows that if the nominal source density is -dimensional Gaussian density function with mean and covariance

where is the unique solution of the following nonlinear equation

)(yg Tdm

),,0[, cY R

,detln2

1)2ln(

2)

1ln(

2)(

1 **,Y

sTRrobust T

ed

s

sdfH

Tc

0s

.2

)1

ln(2 s

d

s

sdRc

14CWIT 2005

Relation among Renyi, Tsallis and Shannon Entropies

Definition. For and , the Renyi entropy [3] is defined by

Moreover, the Tsallis entropy [4] is defined by

We have the following relation among Shannon, Renyi and Tsallis entropies.

1,0 1Lf

.)(ln1

1)( dyyffHR

)1)((1

1)(

dyyffHT

).(lim)(lim)(

],1[1

1)(

),(lim)(

11

)()1(

1

fHfHfH

efH

fHfH

RTS

fHT

RS

R

15CWIT 2005

Relation between Robust Entropy and Renyi Entropy

The robust entropy found previously is related to the Renyi entropy as follows.

Let ; then

Moreover,

s

s

1

)](1

[min)()1,0[

*, *

gHRfH RcsR

robustc

).1,0[),(1

)()(min**,

)1,0[

gHRfHgH Rc

sRrobustR

c

CWIT 2005 16

Examples

17CWIT 2005

Examples: Partially Observed Gauss Markov Nominal Source

Assumptions. Let and are Probability Density Functions (PDF’s) corresponding to a sequence with length of symbols produced by uncertain and nominal sources respectively. Let also the uncertain source and nominal source are related by

)(yf )(yg

}.)|(1

;{ cSU RgfHT

DfD

T

18CWIT 2005

Example: Partially Observed Gauss Markov Nominal Source

Nominal Source. The nominal density is induced by a partially observed Gauss Markov nominal source defined by

Assumptions. unobserved process, observed process, is iid , is iid , are mutually independent is detectable is stabilizable and

,...}.1,0{,

,, 01

tDVCXY

XBWAXX

ttt

ttt

ntX d

tY t

lt

mt WVW ,, ),0(~ mmIN tV

),0(~ llIN },,{),,(~ 0000 tt WVXVxNX),(, ACt ))(,( 2

1trBBA

.0D

19CWIT 2005

Partially Observed Gauss Markov Nominal Source: Lemmas

Lemma. Let be a stationary Gaussian process with power spectral density . Let

and assume exists. Then

dY :

)( jwY eS

)(},,...,{],|[ 1011

ttttt

ttt ZCovYYYYYEYZ

tt

lim

].),...,,[(

,detln1

lim)(detln2

1detln

110tr

TY

YT

jwY

YYYCov

TdweS

20CWIT 2005

Partially Observed Gauss Markov Nominal Source: Lemmas

Lemma. For the partially observed Gauss Markov nominal source

where is the unique positive semi-definite solution of the following Algebraic Ricatti equation

,trtr DDCC

V

.][ 1 trtrtrtrtrtr BBACVDDCCVCAVAAVV

21CWIT 2005

Partially Observed Gauss Markov Nominal Source: Robust Entropy Rate

Proposition. The robust entropy rate of an uncertain source with corresponding partially observed Gauss Markov nominal source is

is the Shannon entropy rate of the nominal source, is given in previously mentioned Lemma and is the unique solution of

.detln2

1)2ln(

2)(

),()1

ln(2

)(

ed

s

sd

S

Srobust

)(S

0s

.2

)1

ln(2 s

d

s

sdRc

22CWIT 2005

Partially Observed Gauss Markov Nominal Source: Remarks

Remark. For the scalar case with , after solving , we obtain

Remark. The case corresponds to . Letting , we have

|}.|ln,0max{)2ln(2

1)

1ln(

2

1)( 2 AeD

s

srobust

0cR ss

).()( Srobust

0B V

23CWIT 2005

Example: Partially Observed Controlled Gauss Markov Nominal Source

The nominal source is defined via a partially observed controlled Gauss Markov system given by

Assumptions. The uncertain source and nominal source are related by is stabilizing matrix, is unobserved process, is observed process,

is iid are mutually independent and

tDVCXY

KYUBUAXX

ttt

ttttt

,

,,1

},)|(1

;{ cSU RgfHT

DfD Kn

tX tY ttt VVU ,,

},{),,(~),1,0(~ 0000 tVXVxNXNt .0D

24CWIT 2005

Partially Observed Controlled Gauss Markov Nominal Source: Robust Entropy Rate

Proposition. Using Body integral formula [5] (e.g., relation between the sensivity transfer function and the unstable eigenvalues of system), the robust entropy rate of family of uncertain sources with corresponding partially observed controlled Gauss Markov nominal source is

are eigenvalues of the system matrix and

is the unique solution of

}1|)(|;{

2 .|)(|ln)2ln(2

1)(

),()1

ln(2

1)(

AiiS

Srobust

i

AeD

s

s

)(Ai A 0s

s

d

s

sdRc 2

)1

ln(2

25CWIT 2005

Example: Uncertain Sources in Frequency Domain

Defining Spaces. Let and be the space of scalar bounded, analytic function of . This space is equipped with the norm defined by

Assumptions. Suppose the uncertain source is obtained by passing a stationary Gaussian random process , whit known power spectral density , through an uncertain linear filter where belongs to the following additive uncertainty model

}1||,;{)1( zCzz H)1(z

||.||

).(|,)(|sup||)(||

jjj ezeHeH

:X)( j

X eS

)(~zH)(

~zH

}.1||||

)(,)(),(,)(

),(),(),(~

);()()()(~

;~

{~

andunkowniszfixedarezWzHHzW

zzHzHzWzzHzHHHDH ad

26CWIT 2005

Uncertain Sources in Frequency Domain: Robust Entropy Rate

Results. The observed process is Gaussian random process, and the Shannon entropy rate of observed process is

Subsequently, the robust entropy rate is defined via

and the solution is given by [6]

Y)(|)(

~|)( 2 j

Xjj

Y eSeHeS

deSe jYS )(ln

4

1)2ln(

2

1)(

,)(lnsup4

1)2ln(

2

1)(

~deSe j

YDH

robustad

deSeWeHe jX

jjrobust )(|))(||)(ln(|

4

1)2ln(

2

1)( 2

CWIT 2005 27

Applications in Stabilizability of Networked Control Systems

28CWIT 2005

Observability and Stabilizability of Networked Control Systems

Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encode and decoder such that

Moreover, a controlled uncertain source is uniform asymptotically stabilizable in probability if there exists an encoder, decoder and controller such that

1

02 .)||

~Pr(||

1suplim

t

kkk

DftYY

tSU

1

02 .)||Pr(||

1suplim

t

kk

DftY

tSU

29CWIT 2005

Necessary Condition for Observability and Stabilizability of Networked Control Systems [2]

Proposition. Necessary condition for uniform asymptotically observability and stabilizability in probability is

is the covariance matrix of Gaussian distribution

that satisfies

.det)2ln(2

1)( g

drobustrobust eC

),0(~)( gNyg

)( dy

.)(

2||||

y

dyyg

g

30CWIT 2005

References

[1] C. D. Charalambous and Farzad Rezaei, Robust Coding for Uncertain Sources, Submitted to IEEE Trans. On Information Theory, 2004.

[2] C. D. Charalambous and Alireza Farhadi, Mathematical Theory for Robust Control and Communication Subject to Uncertainty, Preprint, 2005.

[3] A. Renyi, “On Measures of Entropy and Information”, in Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, vol. I, Berkeley, CA, pp. 547-561, 1961.

31CWIT 2005

References

[4] C. Tsallis, Possible Generalization of Boltzmann-Gibbs Statistics, Journal of Statistics Physics, vol. 52, pp. 479, 1998.

[5] B. F. Wu, and E. A. Jonckheere, A Simplified Approach to Bode’s Theorem for Continuous-Time and Discrete-Time Systems, IEEE Trans on AC, vol. 37, No. 11, pp. 1797-1802, Nov 1992.

[6] C. D. Charalambous and Alireza Farhadi, Robust Entropy and Robust Entropy Rate for Uncertain Sources: Applications in Networked Control Systems, Preprint, 2005.