Information Theory

Linear Block Codes

Jalal Al Roumy

2

Hamming distanceHamming distanceThe intuitive concept of “closeness'' of two words is formalized through Hamming distance d (x, y) of words x, y.For two words (or vectors) x, y; d (x, y) = the number of symbols x and y differ.Example: d (10101, 01100) = 3, d (first, second, fifth) = 3Properties of Hamming distanceProperties of Hamming distance(1) d (x, y) = 0; iff x = y(2) d (x, y) = d (y, x)(3) d (x, z) ≤ d (x, y) + d (y, z) triangle inequality

An important parameter of codes C is their minimal distance.d (C) = min {d (x, y) | x, y ε C, x ≠ y}, because it gives the smallest number of errors needed to change one codeword into another.Theorem Basic error correcting theorem(1) A code C can detect up to s errors if d (C) ≥ s + 1.(2) A code C can correct up to t errors if d (C) ≥ 2t + 1.Note – for binary linear codes d (C) = smallest weight W (C) of non-zero codeword,

3

Some notationSome notation

Notation:Notation: An (n,M,d) - code C is a code such that• n - is the length of codewords.• M - is the number of codewords.• d - is the minimum distance in C.

ExExamplamplee::

C1 = {00, 01, 10, 11} is a (2,4,1)-code.

C2 = {000, 011, 101, 110} is a (3,4,2)-code.

C3 = {00000, 01101, 10110, 11011} is a (5,4,3)-code.

Comment: A good (n,M,d) code has small n and large M and d.

4

Code RateCode Rate

For q-nary (n,M,d)-code we define code rate, or information rate, R, by

The code rate represents the ratio of the number of input data symbols to the number of transmitted code symbols.

For a Hadamard code eg, this is an important parameter for real implementations, because it shows what fraction of the bandwidth is being used to transmit actual data.

.lg

n

MR q

32/632

)64(log2 R

log2(n) = ln(n)/ln(2)Recall that

5

Equivalence of codesEquivalence of codesDefinition Two q -ary codes are called equivalent if one can be obtained from the other by a combination of operations of the following type:

(a) a permutation of the positions of the code.

(b) a permutation of symbols appearing in a fixed position.

Let a code be displayed as an M ´ n matrix. To what correspond operations (a) and (b)?

Distances between codewords are unchanged by operations (a), (b). Consequently, equivalent codes have the same parameters (n,M,d) (and correct the same number of errors).

Examples of equivalent codesExamples of equivalent codes

Lemma Any q -ary (n,M,d) -code over an alphabet {0,1,…,q -1} is equivalent to an (n,M,d) -code which contains the all-zero codeword 00…0.

102

021

210

222

111

000

2

01011

11101

10110

00000

00011

11111

11000

00100

1

6

The main coding theory The main coding theory problemproblem

A good (n,M,d) -code has small n, large M and large d.

The main coding theory problem is to optimize one of the parameters n, M, d for given values of the other two.

Notation:Notation: Aq (n,d) is the largest M such that there is an q -nary (n,M,d) -code.

7

Introduction to linear codes

codewords.

zero-non theof weights theofsmallest the toequal is codelinear a of distance minimum The

x.of entries zero-non ofnumber thebe todefined is q) V(n,in vector a of x) weight w(The

code.linear a tobelongslly automatica 0 vector zero-all the

code.-d] k, [n,an is code-d) ,q (n,

everynot course ofbut code-d) ,q (n,ary -q a is code-d] k, [n,ary -qA

added. is distance theif code-d] k, [n, andor code-k] [n,

an called is codelinear the thek) (n, V of subspace ldimentiona-k a is C If

codeword. a is codewords any two of sum theifflinear is codebinary A

(q) GF a C, u allfor C a.u (2)

Cin vandu allfor C vu (1)

iff q) V(n, of subspace a is C

n. of valuepositive somefor q)V(n, space vector the

ofsubset a ispower prime a is q whereField] [Galois GF(q)over codelinear A

k

k

8

Linear Block Codes• Information is divided into blocks of length k• r parity bits or check bits are added to each block

(total length n = k + r),.• Code rate R = k/n• Decoder looks for codeword closest to received

vector (code vector + error vector)• Tradeoffs between

• Efficiency• Reliability• Encoding/Decoding complexity

9

Linear Block Codes

e HT = e HT e ) * HT =c HT

e be the received message; c is the correct code and e is the error Let x = c

Compute S = x * HT =( c

If S is 0 then message is correct else there are errors in it, from common known error patterns the correct message can be decoded.

The parity check matrix H is used to detect errors in the received code by using the fact that c * HT = 0 ( null vector)

Generator matrix

G

Code Vector

C

Message vector

m

Parity check matrix

HT

Code Vector

C

Null vector

0

Operations of the generator matrix and the parity check matrix

10

Linear Block Codes• Linear Block Code

The block length C of the Linear Block Code is

C = m G

where m is the information codeword block length, G is the generator matrix.

G = [Ik | P] k × n,

I is unit matrix.• The parity check matrix

H = [PT | In-k ], where PT is the transpose of the matrix p.

11

Forming the generator matrixThe generator matrix is formed from the list of codewords by ignoring the all zero vector and the linear combinations; eg

C

0 0 0 0 0 0 0

1 1 1 1 1 1 1

1 0 0 0 1 0 1

1 1 0 0 0 1 0

0 1 1 0 0 0 1

1 0 1 1 0 0 0

0 1 0 1 1 0 0

0 0 1 0 1 1 0

0 0 0 1 0 1 1

0 1 1 1 0 1 0

0 0 1 1 1 0 1

1 0 0 1 1 1 0

0 1 0 0 1 1 1

1 0 1 0 0 1 1

1 1 0 1 0 0 1

1 1 1 0 1 0 0

giving G

1 1 1 1 1 1 1

1 0 0 0 1 0 1

1 1 0 0 0 1 0

0 1 1 0 0 0 1

C [7,4,3] code

12

Equivalent linear [n,k]-codesTwo k x n matrices generate equivalent linear codes over GF(q) if one matrix can be obtained from the other by a sequence of operations of the following types:

(R1) permutation of rows

(R2) multiplication of a row by a non-zero scaler

(R3) Addition of a scaler multiple of one row to another

(C1) Permutation of columns

(C2) Multiplication of any column by a non-zero scaler

The row operations (R) preserve the linear independence of the rows of the generator matrix and simply replace one basis by another of the same code. The column operations (C) convert the generator matrix to one for an equivalent code.

13

Transforming the generator matrixTransforming to the form G = [Ik | P]

1 1 1 1 1 1 1

1 0 0 0 1 0 1

1 1 0 0 0 1 0

0 1 1 0 0 0 1

1 1 1 1 1 1 1

0 1 1 1 0 1 0

0 0 1 1 1 0 1

0 1 1 0 0 0 1

1 0 0 0 1 0 1

0 1 1 1 0 1 0

0 0 1 1 1 0 1

0 1 1 0 0 0 1

1 0 0 0 1 0 1

0 1 0 0 1 1 1

0 0 1 1 1 0 1

0 0 0 1 0 1 1

1 0 0 0 1 0 1

0 1 0 0 1 1 1

0 0 1 0 1 1 0

0 0 0 1 0 1 1

Therefore G [Ik | A]

1 0 0 0 1 0 1

0 1 0 0 1 1 1

0 0 1 0 1 1 0

0 0 0 1 0 1 1

14

Encoding with the generatorCodewords = message vector u x G

For example, where

G

1 0 0 0 1 0 1

0 1 0 0 1 1 1

0 0 1 0 1 1 0

0 0 0 1 0 1 1

then

0 0 0 0 is encoded  as 0 0 0 0 0 0 0

1 0 0 0 is encoded as 1 0 0 0 1 0 1

1 1 1 0 is encoded  as 1 1 1 0 1 0 0

Note that 1110 is found by adding R1,R2 and R3 of G

15

Parity-check matrixA parity check matrix H for an [n, k]-code C is and

(n - k) x n matrix such that x. HT = 0 iff x C. A parity-check matrix for C is a generator matrix for the duel code C . If G = [Ik | A] is the standard form generator matrix for an [n, k]-code C, then the parity-check matrix for C is H = [-AT | In-k ]. A parity check matrix of the form [B | In-k ] is said to be in standard form.

G

1 0 a11 . . a1,n k

. . .

. . .

0 1 ak1 . . ak ,n k

H

a11 . . ak1 1 0

. .

. .

a1,n k . . ak ,n k 0 1

16

Decoding using Slepian matrixAn elegant nearest-neighbour decoding scheme was devised by Slepian in 1960.• every vector in V(n, q) in in some coset of C• every coset contains exactly qk vectors• two cosets are either disjoint or coincide

Let  G 1 0 1 1

0 1 0 1

  giving C {0000, 1011, 0101, 1110}

codewords 0000 1011 0101 1110

1000 0011 1101 0110

0100 1111 0001 1010

0010 1001 0111 1100

coset

leaders

When y is received  (eg 1111) its  position is  found. The decoder

decides that  the error  is the coset  leader  0100.  x y e 1011.

17

Syndrome decodingSuppose C is a q-ary [n, k]-code with the parity-check matrix H. For any vector y = V(n, q), the row vector

S(y) = y HT is called the syndrome of y. Two vectors have the same syndromes iff they lie in the same coset.

G 1 0 1 1

0 1 0 1

       H

1 0 1 0

1 1 0 1

        

The sydromes of  the coset  leaders  from our  example

S(0000) 00

S(1000) 11

S(0100) 01

S(0010) 10

      this will  then give a syndrome lookup table

Syndromex coset  leader   f (x)

00 0000

11 1000

01 0100

10 0010

18

Decoding procedureThe rules:

Step 1   For a received vector y calculate S(y) yH T

Step 2 Let x S(y), and locate z in the first column of the look-up table

Step 3 Decode y as y - f (x).

For  example, if  y 1111, then S(y) 01 and  we decode as 1111 0100 1011

19

ExampleG

1 0 0 1

0 1 1 1

        codewords 22 4        vectors 24 16

Slepian matrix is 

0000 1001 0111 1110

1000 0001 1111 0110

0100 1101 0011 1010

0010 1011 0101 1100

0011 is decoded  as 0111

1110 1110

Parity check  matrix 

H 0 1 1 0

1 1 0 1

       S(y) yH T     S(y) y

0 1

1 1

1 0

0 1

Syndrome lookup table         

00 0000

01 1000

11 0100

10 0010