Information Theory Detour
Transcript of Information Theory Detour
![Page 1: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/1.jpg)
Things we will cover todayEntropy
Information Theory Detour
Hayder RadhaPresented by: Kiran Misra
Department of Electrical and Computer EngineeringMichigan State University
October 8, 2008
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 2: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/2.jpg)
Things we will cover todayEntropy
No Professor Radha!!! My name is “Kiran”
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 3: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/3.jpg)
Things we will cover todayEntropy
Why learn about information theory?
Basic elements of information theory are necessary for manyaspects of Multimedia Coding, Communication and Networking
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 4: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/4.jpg)
Things we will cover todayEntropy
Why learn about information theory?
Basic elements of information theory are necessary for manyaspects of Multimedia Coding, Communication and Networking
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 5: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/5.jpg)
Things we will cover todayEntropy
List of things we cover
Entropy Definition, ExampleJoint Entropy DefinitionConditional Entropy Motivation, Definition, Graphical representationMutual Information Definition, Graphical representation, Inequalities
Some of you may have already seen this in ECE867.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 6: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/6.jpg)
Things we will cover todayEntropy
List of things we cover
Entropy Definition, ExampleJoint Entropy DefinitionConditional Entropy Motivation, Definition, Graphical representationMutual Information Definition, Graphical representation, Inequalities
Some of you may have already seen this in ECE867.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 7: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/7.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 8: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/8.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 9: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/9.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain
→ Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 10: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/10.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 11: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/11.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain
→ Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 12: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/12.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 13: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/13.jpg)
Things we will cover todayEntropy
Entropy
Its a measure of uncertainty associated with a random variable.
Example: Assume you live in a desert where it rains once a year.Random variable of interest: Weather Report
Today’s weather forecast: No Rain → Little Uncertainty → LittleInformation.
Today’s weather forecast: Rain → Lot of Uncertainty → Lot ofInformation.
A more precise definition was formulated by Shannon in 1948.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 14: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/14.jpg)
Things we will cover todayEntropy
Entropy
Alternatively,X → Sender →
msg Receiver
Entropy of X is the “minimum number of bits” needed (“onaverage”) for coding the outcomes of X .
The most likely outcome will require the least number of bits.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 15: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/15.jpg)
Things we will cover todayEntropy
Entropy
Alternatively,X → Sender →
msg Receiver
Entropy of X is the “minimum number of bits” needed (“onaverage”) for coding the outcomes of X .
The most likely outcome will require the least number of bits.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 16: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/16.jpg)
Things we will cover todayEntropy
Entropy
Alternatively,X → Sender →
msg Receiver
Entropy of X is the “minimum number of bits” needed (“onaverage”) for coding the outcomes of X .
The most likely outcome will require the least number of bits.
Hayder Radha Presented by: Kiran Misra ECE 802-606: Information Theory Detour
![Page 17: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/17.jpg)
1/13 Copyright © 2005-2007 – Hayder Radha
![Page 18: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/18.jpg)
2/13 Copyright © 2005-2007 – Hayder Radha
![Page 19: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/19.jpg)
3/13 Copyright © 2005-2007 – Hayder Radha
![Page 20: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/20.jpg)
4/13 Copyright © 2005-2007 – Hayder Radha
![Page 21: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/21.jpg)
5/13 Copyright © 2005-2007 – Hayder Radha
![Page 22: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/22.jpg)
6/13 Copyright © 2005-2007 – Hayder Radha
![Page 23: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/23.jpg)
7/13 Copyright © 2005-2007 – Hayder Radha
![Page 24: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/24.jpg)
8/13 Copyright © 2005-2007 – Hayder Radha
![Page 25: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/25.jpg)
9/13 Copyright © 2005-2007 – Hayder Radha
![Page 26: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/26.jpg)
10/13 Copyright © 2005-2007 – Hayder Radha
![Page 27: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/27.jpg)
11/13 Copyright © 2005-2007 – Hayder Radha
![Page 28: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/28.jpg)
12/13 Copyright © 2005-2007 – Hayder Radha
![Page 29: Information Theory Detour](https://reader033.fdocuments.us/reader033/viewer/2022051919/62851d2090816e0c9d1b9723/html5/thumbnails/29.jpg)
13/13 Copyright © 2005-2007 – Hayder Radha