Information Theory Ying Nian Wu UCLA Department of Statistics July 9, 2007 IPAM Summer School

Information Theory

Ying Nian WuUCLA Department of Statistics

July 9, 2007IPAM Summer School

Goal: A gentle introduction to the basic concepts in information theory

Emphasis: understanding and interpretations of these concepts

Reference: Elements of Information Theory by Cover and Thomas

Topics•Entropy and relative entropy•Asymptotic equipartition property•Data compression•Large deviation•Kolmogorov complexity•Entropy rate of process

EntropyRandomness or uncertainty of a probability distribution

kk paX )Pr( )Pr()( xXxp

8/18/14/12/1Pr

Example

Definition

k xpxpppXH )(log)(log)(

Entropy

8/18/14/12/1Pr

Example

1)( XH

Definition for both discrete and continuous

)](log[)( XppH E

xpxfXf )()()]([ERecall

Entropy

Example

Entropy

3321log

8/18/14/12/1Pr

1)( XH

)](log[)( XppH E

Interpretation 1: cardinality

Uniform distribution ][~ AX Unif

There are elements inAll these choices are equally likely

||log|)]|/1log([)](log[)( AAXpXH EE

Entropy can be interpreted as log of volume or size dimensional cube has vertices can also be interpreted as dimensionality

What if the distribution is not uniform?

Asymptotic equipartition property

Any distribution is essentially a uniform distribution in long run repetition

)(~,...,, 21 xpXXX n independently

)()...()(),...,,( 2121 nn XpXpXpXXXp

||/1)( AXp Recall if then a constant]Unif[AX ~

Random?

But in some sense, it is essentially a constant!

Law of large number

ii xpxfXfXf

n)()()]([)(

Long run average converges to expectation

)()...()(),...,,( 2121 nn XpXpXpXXXp

)()]([log)(log1

),...,(log1

11 pHXpXp

Intuitively, in the long run,)(

1 2),...,( pnHnXXp

)(1 2),...,( pnH

||/1)( AXp Recall if

then a constant]Unif[ AX ~

Therefore, as if ]Unif[AXX n ~),...,( 1

)(2|| pnHA ,with

So the dimensionality per observation is )( pH

We can make it more rigorous

1)|)(1

0nn for

ii xpxfXfXf

n)()()]([)(

Weak law of large number

Typical set

)(1 2),...,( pnH

]Unif[AXX n ~),...,( 1)(2|| pnHA ,with

Typical set nnA ,

: The set of sequencesn

nxxx ),...,,( 21,nA))((

21))(( 2),...,,(2 pHn

npHn xxxp

1)( ,nAp

for sufficiently large

))(( 2||2)1(

Typical set)(

1 2),...,( pnHnXXp

Interpretation 2: coin flipping

Flip a fair coin {Head, Tail}Flip a fair coin twice independently {HH, HT, TH, TT}……Flip a fair coin times independently equally likely sequences

We may interpret entropy as the number of flips

Example

TTTHHTHH

4/14/14/14/1Pr

The above uniform distribution amounts to 2 coin flips

)(1 2),...,( pnH

),...,( 1 nXX amounts to )( pnH flips

)( pH flips)(~ xpX amounts to

THTTHHTTH

8/18/14/12/1Pr

3321log

8/18/14/12/1Pr

flips4

1)( XH

THTTHHTTH

8/18/14/12/1Pr

Interpretation 3: coding

Example

00011011

4/14/14/14/1Pr

)4/1log(4log2 length

)(1 2),...,( pnH

How many bits to code elements in A

)( pnH

Can be made more formal using typical set

Interpretation 3: coding

010011001

8/18/14/12/1Pr

100101100010abacbd

Prefix code

bitsE4

11)()()]([

xpxlXl

100101100010abacbd

010011001

8/18/14/12/1Pr

Sequence of coin flippingA completely random sequenceCannot be further compressed

)(log)( xpxl

)()]([ pHXl E

Optimal code

e.g., two words I, probability

Optimal code

Kraft inequality for prefix code

kk plXlL )]([EMinimize

Optimal length kk pl log*

)(* pHL

Wrong model

k ppplXlL log)]([ *** EOptimal code

kkkk qpplXlL log)]([EWrong code

Redundancyk

ppLL log*

Box: All models are wrong, but some are useful

Relative entropy

)(log)(]

)([log)|( E

Kullback-Leibler divergence

Relative entropy

)([log(]

)([log]

)([log)|(

XpqpD ppp EEE

Jensen inequality

)(||log||log)]([log)|( pHXpUpD p E

)(~,...,, 21 xqXXX n independently

kn number of times ki aX

k normalized frequency

),...,(),...,( 11 KK qqqfff

Law of large number

Refinement

kk qnfnf

,...,,...,)Pr(

in xpxxp )(),...,( 1

),...,(),...,( 11 KK qqqfff

Law of large number

Refinement

)|()Pr(log1

)|(2~)Pr( qfnDf

Large deviation

Kolmogorov complexity

Example: a string 011011011011…011

Program: for (i =1 to n/3) write(011) endCan be translated to binary machine code

Kolmogorov complexity = length of shortest machine code that reproduce the string no probability distribution involved

If a long sequence is not compressible, then it has all the statistical properties of a sequence of coin flipping

string = f(coin flippings)

Joint and conditional entropy

2222212

1112111

kkjkkk

kjjk pbYaX )&Pr(

)&Pr(),( yYxXyxp

kk paX )Pr(

jj pbY )Pr(

)Pr()( xXxpX )Pr()( yYypY

Joint distribution

Marginal distribution

e.g., eye color & hair color

jkjjk ppbYaX /)|Pr(

kkjkj ppaXbY /)|Pr(

)|Pr()|(| yYxXyxp YX

)|Pr()|(| xXyYxyp XY

)|()()|()(),( || yxpypxypxpyxp YXYXYX

2222212

1112111

kkjkkk

Conditional distribution

Chain rule

2222212

1112111

kkjkkk

kj yxkjkj YXpyxpyxpppYXH

)],(log[),(log),(log),( E

xypxypxXYH )|(log)|()|(

xypxypxpxXYHxpXYH )|(log)|()()|()()|(

)]|(log[)|(log),()|(,

XYpxypyxpXYHyx

)|()()|()(),( || yxpypxypxpyxp YXYXYX

)|()(),( XYHXHYXH

Chain rule

),...,|(),...,(1

iiin XXXHXXH

),([log))()(|),(();(

YXpypxpyxpDYXI E

yx ypxp

yxpyxpYXI

, )()(

),(log),();(

Mutual information

)|()();( YXHXHYXI

),()()();( YXHYHXHYXI

Entropy rate

Stochastic process ,...),...,,( 21 nXXX not independent

Markov chain:

)|Pr(),...,|Pr( 111111 nnnnnnnn xXxXxXxXxX

),...,(1

lim 1 nn XXHn

H Entropy rate:

Stationary process: ),...,|(lim 11 XXXHH nnn

Stationary Markov chain: )|( 1 nn XXHH

(compression)

Shannon, 1948

1. Zero-order approximationXFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD. 2. First-order approximation OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL. 3. Second-order approximation (digram structure as in English). ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE. 4. Third-order approximation (trigram structure as in English). IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE. 5. First-order word approximation. REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE. 6. Second-order word approximation. THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.

Summary

Entropy of a distribution measures randomness or uncertainty log of the number of equally likely choices average number of coin flips average length of prefix code (Kolmogorov: shortest machine code randomness)

Relative entropy from one distribution to the other measure the departure from the first to the second coding redundancy large deviation

Conditional entropy, mutual information, entropy rate

Information Theory Ying Nian Wu UCLA Department of Statistics July 9, 2007 IPAM Summer School

Documents

Transcript of Information Theory Ying Nian Wu UCLA Department of Statistics July 9, 2007 IPAM Summer School

A Note on Machine Learning Methodsywu/MLnote2020.pdf · A Note on Machine Learning Methods Ying Nian Wu, UCLA Statistics, Based on M231B lectures Updated March 2020 Contents 1 Ptolemy’s

Windows Server 2019 IPAM … · Windows Server 2019 IPAM Erstellt von Jörn Walter 10.04.2020 IP Address Management (IPAM) ist ein Feature für das Management von verschiedenen Netzwerkadressbereichen.

4. IPAM Editing

Causal Model Ying Nian Wu UCLA Department of Statistics July 13, 2007 IPAM Summer School

IPAM WorkshopJan 2006

Discussion Ying Nian Wu UCLA Department of Statistics JSM 2011

Nian Ying CHAN, Malaysia - eduhk.hk for... · Nian Ying CHAN, Malaysia Bachelor of Education (Hons) ... (Secondary) EdUHK had always been my dream university and I couldn’t have

IPAM in University Environments

Building World-Class Universities Building World-Class · Building World-Class Universities Qi Wang, Ying Cheng and Nian Cai Liu (Eds.) Building World-Class Universities Different

Injury Prevention and Management (IPaM) program · Title: IPaM Infographic 2016 Author: Injury Prevention and Management program Subject: A flyer highlighting IPaM infographics Keywords:

IPAM - Windowstechprowindowstechpro.com/wp-content/uploads/2014/09/IPAM.pdf · What Is IPAM? IPAM (IP Address Management) is the New Feature in Windows Server 2012. Enables to Plan,

Statistical Modeling and Learning in Vision --- cortex-like generative models Ying Nian Wu

Ruixun Zhang Peking University Mentor: Prof. Ying Nian Wu Direct supervisor: Zhangzhang Si Department of Statistics.

Primal Sketch: Integrating Texture and Structuresczhu/papers/primal_sketch.pdfPrimal Sketch: Integrating Texture and Structure Cheng-en Guo, Song-Chun Zhu, and Ying Nian Wu Departments

IPAM Products and Services - Calleva Networks Ltd. · IPAM Products and Services from BT Diamond IP BT Diamond IP IPAM products and services automate these three key elements of IPAM:

Representation and Modeling of Natural Scenes Ying Nian Wu UCLA Department of Statistics

Documents.tips Jurnal Bedah Nian

Interpreting CNN Knowledge Via An Explanatory Graph · 2017-11-23 · Interpreting CNN Knowledge Via An Explanatory Graph Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun

Unsupervised Learning of Compositional Sparse Code for Natural Image Representation Ying Nian Wu UCLA Department of Statistics October 5, 2012, MURI Meeting.

Wp ipam infoblox