8/6/2019 6 Information in the Mathematical Theory of Communication

1/12

silan tharakan

INFORMATION IN THEMATHEMATICAL

THEORY OF

COMMUNICATION

8/6/2019 6 Information in the Mathematical Theory of Communication

2/12

8/6/2019 6 Information in the Mathematical Theory of Communication

3/12

silan tharakan

Historical Development developed by Norbert Weiner

According to Weiner, any organization is held

together by the possession of means foracquisition, use, retention and transmissionof information

information theory has developed primarily as

a mathematical theory of communication

8/6/2019 6 Information in the Mathematical Theory of Communication

4/12

silan tharakan

Model of a Communication

System purpose of a communication system is to

reproduce at the destination a message

selected at the source

8/6/2019 6 Information in the Mathematical Theory of Communication

5/12

silan tharakan

8/6/2019 6 Information in the Mathematical Theory of Communication

6/12

silan tharakan

Each sound, inflection, pitch, etc., of thehuman voice is a message in the telephone

system Considerably more messages are required to

relay the voice than the coded letters andnumbers

8/6/2019 6 Information in the Mathematical Theory of Communication

7/12

silan tharakan

Mathematical Definition of

Information It is the average number of binary digits which must

be transmitted to identify a given message from the

set of all possible messages to which it belongs

The message to be transmitted is encoded, thecodes are sent over the channel, and the decoderidentifies the message intended by the codes

size of the code is dependent on the coding scheme

and the number of possible messages

coding scheme for information theory is assumed to

be binary

8/6/2019 6 Information in the Mathematical Theory of Communication

8/12

silan tharakan

communication system is required to respond toonly two responses, "yes" or "no,"

then the system needs to transmit one of only twopossible signals1 and 0

E.g. a system used to transmit birthday greetings

senders may not write their own greetings-they mayonly select from several standard messages

Assume there are eight such messages

8/6/2019 6 Information in the Mathematical Theory of Communication

9/12

silan tharakan

The information content(or code size in bits)

may be generalized as:

where n = the totalnumber of possiblemessages

If n = 8, I = 3 bits

2logI n

8/6/2019 6 Information in the Mathematical Theory of Communication

10/12

silan tharakan

Reduction of Uncertainty Information reduces uncertainty

Partial information will reduce the uncertainty but noteliminate it

in the case of the eight birthday messages, three

bits are required to eliminate uncertainty completelyby identifying the exact message

transmitting a single bit selects one of the twogroups and reduces the possibilities from eight tofour as shown in figure following

8/6/2019 6 Information in the Mathematical Theory of Communication

11/12

silan tharakan

8/6/2019 6 Information in the Mathematical Theory of Communication

12/12

silan tharakan

Redundancy Redundancy reduces the efficiency of a particular

transmission

because more codes are transmitted than are strictly

required to encode the message The transmission of redundant data allows the

receiver to check whether the received message iscorrect and may allow the original message to bereconstructed

redundancy means that the listener need not readand decode every letter in order to understand themessage