1_coding and Decoding

7/31/2019 1_coding and Decoding

1/6


2/6

M. A. Lokhandwala Page 2 of 6

Message source : A generator which can generate source symbols continuously.

Example-3 : A decimal source can generate 100 symbols { 1, 2, , 100 } withcorresponding codes {00, 01, , 99}. Here, the code alphabet employs 10 symbols 0, 1,, 9 and it

e.g. message generated may be { 1 2 6 100 }= {00010599}={ K( 1) K( 2) K( 6) K( 100 )}.

Coding of source messages : For each coding K of source symbols, we define the coding of source messages as the coding rule K*, which assigns to each word m1 m2 m3 mk in the sourcealphabet the word K*( m1 m2 m3 mk ) = K( m1) K( m2) . K( mk ).

e.g. Two digit decimal coding of message word m1m2m3 m4 = a 1 2 6 a 100 = = 00010599.

Uniquely decodable coding : Coding K is said to be uniquely decodable provided arbitrary twodistinct source messages (symbols) have distinct code words. In other words, provided that K* isone-to-one.

Two digit decimal code above is uniquely decodable as all code words are distinct. Message m1

m2 m3 m4 = a 1

2

6 a 100 = 00010599 is uniquely decodable.In contrast, the following coding is not uniquely decodable,

a 00 b 10 c 101 d 110 e 1001 Because for example, 10110 can be decoded as either bd or cb. Here symbols b, c & e are notuniquely decodable.

Two important types of codes :Block codes :

Def : A code formed using pairwise distinct code words of a certain length n is called ablock code.

Block codes are very convenient to decode as we know in advance, which code wordcorresponds to which source symbol.

Block codes are efficient when all source symbols appear with equal probability. Examples : Octal code. Hexa decimal code, ASCII code, Gray code, etc.

Instantaneous Codes : Codes of variable word lengths which are decodable symbols per symbol. Def : A coding is said to be instantaneous provided that no code word is prefix of another

code word; i.e. if a source symbol has a code b 1b2.bn then no other source symbol has acode b1b2.bnbn+1 .bm.

Preferable when probabilities of various source symbols vary substantially. Examples : Morse code : code alphabet = { , , space }. Space is used only at the end

of each code word. Decoding is simple: Look for the space to decide that the code isterminated and then decode.


3/6


Construction of Instantaneous Code :

Let, Source alphabet = { 1, 2, , n},Length of Corresponding code words = d 1, d2, . . . ., d n.

Assumption : d1 d 2 , . . . ., d n.

Choose arbitrarily,1. K( 1) = binary code word of length d 1.

2. K( 2) = binary code word of length d

2,But, avoid all those having prefix K(

1).

The no. of all possible binary word of length d 2 = 2 d2.

The no. of possible assignment as K( 2) with those having K( 1) as prefix = 2 d2-d1 ( because, we have d 2-d 1 digits remaining after the prefix K( 1) ).

For Instantaneous binary word of length d 2 to exist, at least one binary word of length d 2 must exist which doent contain a code word K( 1) of length d 1 (i.e. we should have atleast one choice of K( 2)).

i.e. (2 d2 - 2 d2-d1 ) 1.

2d2 2 d2-d1 + 1 is always satisfied.

3. K( 3) = binary code word of length d 3, But, avoid all those having prefix K( 1) and K( 2).The no. possible word with prefix K( 1) = 2

d3-d1 ,The no. possible word with prefix K( 2) = 2 d3-d2 K( 3), avoiding all those having prefix K( 1) and K( 2), can be selected if ( and only if ),

2d3 2 d3-d1 + 2 d3-d2 +1.1 2 -d1 + 2 -d2 + 2- d3.

4. for K( n) = binary code word of length dn,

1 2 -d1 + 2 -d2 + . + 2 -dn .

The above inequality is both necessary and sufficient condition for a construction of instantaneous codes.

Procedure :1. Select length ds of code words which satisfies above inequality,2. Assign code words starting with that having smallest length keeping in mind that the

code word under consideration doesnt have any prefix which is a code word of lengthless than its length.

Example 4 : Form the binary instantaneous code for the source alphabet {0,1,2,3}.Let, d 1 = 1, d 2 = d 3 = d 4 = 2, then, 2 -1+2 -2+2 -2+2 -2 = 1.25 > 1. Code not possible.

d1 = 1, d 2 = d 3 = 2, d 4 = 3, then, 2 -1+2 -2+2 -2+2 -3 = 1.125 > 1. Code not possible.d1 = 1, d 2 = 2, d 3 = d 4 = 3, then, 2 -1+2 -2+2 -3+2 -3 = 1.0 Code is possible.

The instantaneous code is,0 0 1 10 2 110 3 111.

Note: In above instantaneous coding, we have started with d min = d 1 = 1.


4/6


Example 4 : Form the binary instantaneous code for the source alphabet {0,1,2,3,4,5,6,7}.

n = d max = total no. of source alphabets.Starting with d min = d 1 = 1,

d1 = 1, d 2 = 2, d 3 = 3, d 4 = 4, d 5 = 5, d 6 = 6, d 7 = 7, d 8 = 7then, 2^-1+2^-2+2^-3+2^-4+2^-5+2^-6+2^-7+2^-7 = 1.0.Code is possible.

But, the total no. of bits required = .35728)18()1(18

1

1

1 i

n

i

ini

Starting with d min = d 1 = 2,d1 = 2, d 2 = 2, d 3 = 2, d 4 = 3, d 5 = 4, d 6 = 5, d 7 = 6, d 8 = 6

then, 2^-2+2^-2+2^-2+2^-3+2^-4+2^-5+2^-6+2^-6 = 1.0.Code is possible.

The total no of bits required = .306186][)12( min1

min

min

min

min d nid d n

d i

d

Starting with d min = 3,d1 = 3, d 2 = 3, d 3 = 3, d 4 = 3, d 5 = 3, d 6 = 3, d 7 = 3, d 7 = 3

then, 2^-3+2^-3+2^-3+2^-3+2^-3+2^-3+2^-3+2^-3 = 1.0.Code is possible.

The total no of bits required = (3)(8) = 24. But, this will become a block code.

Important Note : Any block code of length d 2, satisfies above inequality, and hence can bethought of as instantaneous code.

e.g. d 1 = d 2 = d 3 = d 4 = 2,then, 2 -2+2 -2+2 -2+2 -2 = 1.0.

d1 = d 2 = d 3 = d 4 = d 5 = d 6 = d 7 = d 8 = 3,then, 2 -3+2 -3+2 -3+2 -3 +2 -3+2 -3+2 -3+2 -3 = 1.0.

Observations :1. Block codes required the minimum no. of bits for encoding.

2.

then, what is the use of instantaneous code ?3. In above example, minimum bit requirement comes from assumption of equalprobability of occurrence of each source symbol , which is not the case in practice.

4. In practice, source symbols have variable probability of occurrence. One with moreprobability of occurrence can be assigned less bits and hence coding redundancy can beremoved. Here, instantaneous coding is employed.


5/6


Krafts Inequality

Theorem : Given a source alphabet of n symbols and a code alphabet of r symbols , then aninstantaneous code with given lengths d 1, d 2, , d n of code words exists, whenever the followingKrafts inequality

r -d1+ r -d2++ r -dn 1 is satisfies.

Proof :

Let, source alphabet = { a 1 , a 2 ,, a n },Then, expected code words = {K( a 1), K( a 2),, K( an)}

with corresponding lengths d 1 d2 dn.

We can define the instantaneous coding K as follows:

The code word K( a s) for sth symbol must ,

1. have length d s. 2. not have K( a 1), K( a 2),, K( a s-1) as prefix.

Thus, possible combinations of code words which can be assigned to s th symbol are,1

1

s

i

d d d iss r r

There must be at least one choice, so that code word can be formed.

i.e. 11

1

s

i

d d d iss r r

multiplying both sides bysd

r ,si d

s

i

d r r 1

1

1

i.e.

s

i

d s

i

d d iis r r r 1

1

1

1

11

s

i

d ir

This is Krafts inequality.

This must hold for all symbols. The symbol with highest length will have largest summation.Hence the inequality to be tested is

11

n

i

d ir


6/6


McMillan Theorem Every uniquely decodable coding satisfies Krafts inequality.Proof :Let, d i = length of code word K( a i) for symbol a i, i= 1,2,,n.Let, no. of symbols in code alphabet = r.Let, source message = a i1a i2a im , which consists of n source symbols is emitted randomly.

Unique decidability means every source message must be uniquely decodable which is truewhen no. of source messages of length j must be less than or equal to r j

.

Therefore, the no. of source messages r j andlength of this message = j = d i1+d i2++d im.

Therefore, the no. of sums of the form j = d i1+d i2++d im is smaller than or equal to r j.

For the message to be uniquely decodable, we need to prove that the numberm

i

d ir c1

is less than or equal to 1.

and for this we will verify that the numbersmc m are bounded for m = 1, 2,

and this is true for c

1_coding and Decoding

Documents

Transcript of 1_coding and Decoding