ECS 452: Digital Communication Systems 2015/2 HW 1 | Due ... 2015 PreMidterm … · ECS 452:...

ECS 452: Digital Communication Systems 2015/2

HW 1 — Due: Feb 5

Lecturer: Asst. Prof. Dr. Prapun Suksompong

Instructions

(a) Must solve all non-optional problems. (5 pt)

(i) Write your first name and the last three digit of your student ID on the upper-rightcorner of every submitted page.

(ii) For each part, write your explanation/derivation and answer in the space provided.

(b) ONE part of a question will be graded (5 pt). Of course, you do not know which partwill be selected; so you should work on all of them.

(c) Late submission will be rejected.

(d) Write down all the steps that you have done to obtain your answers. You maynot get full credit even when your answer is correct without showing how you get youranswer.

Problem 1 (HW1-2015-2). Consider the code {0, 01}

(a) Is it nonsingular?

(b) Is it uniquely decodable?

(c) Is it prefix-free?

1-1

ECS 452 HW 1 — Due: Feb 5 2015/2

Problem 2 (HW1-2015-2). Consider the random variable X whose support SX containsseven values:

SX = {x1, x2, . . . , x7} .

Their corresponding probabilities are given by

x x1 x2 x3 x4 x5 x6 x7

pX(x) 0.49 0.26 0.12 0.04 0.04 0.03 0.02

(a) Find the entropy H(X).

(b) Find a binary Huffman code for X.

(c) Find the expected codelength for the encoding in part (b).

1-2

ECS 452 HW 1 — Due: Feb 5 2015/2

Problem 3 (HW1-2015-2). Find the entropy and the binary Huffman code for the randomvariable X with pmf

pX(x) =

{x21, x = 1, 2, . . . , 6,

0, otherwise.

Also calculate E [`(X)] when Huffman code is used.

Problem 4 (HW1-2015-2). These codes cannot be Huffman codes. Why?

(a) {00, 01, 10, 110}

(b) {01, 10}

(c) {0, 01}

1-3

ECS 452 HW 1 — Due: Feb 5 2015/2

Problem 5 (HW1-2015-2). A memoryless source emits two possible message Y(es) and N(o)with probability 0.9 and 0.1, respectively.

(a) Determine the entropy (per source symbol) of this source.

(b) Find the expected codeword length per symbol of the Huffman binary code for thethird-order extensions of this source.

(c) Use MATLAB to find the expected codeword length per (source) symbol of the Huff-man binary code for the fourth-order extensions of this source.

(i) Put your answer here.

(ii) Don’t forget to attach the printout of your MALTAB script (highlighting themodified parts if you start from the provided class example) and the expression-s/results displayed in the command window.

(d) Use MATLAB to plot the expected codeword length per (source) symbol of the Huff-man binary code for the nth-order extensions of this source for n = 1, 2, . . . , 8. Attachthe printout of your plot.

1-4

ECS 452 HW 1 — Due: Feb 5 2015/2

Problem 6 (HW1-2015-2). (Optional) The following claim is sometimes found in the liter-ature:

“It can be shown that the length `(x) of the Huffman code of a symbol x withprobability pX(x) is always less than or equal to d− log2 pX(x)e”.

Even though it is correct in many cases, this claim is not true in general.Find an example where the length `(x) of the Huffman code of a symbol x is greater than

d− log2 pX(x)e.Hint: Consider a pmf that has the following four probability values {0.01, 0.30, 0.34, 0.35}.

Problem 7 (HW1-2015-2). (Optional) Construct a random variable X (by specifying itspmf) whose corresponding Huffman code is {0, 10, 11}.

1-5


HW 2 — Due: Feb 19


Instructions

(a) Must solve all non-optional problems. (5 pt)






Problem 1. In this question, each output string from a DMS is encoded by the followingsource code:

x Codeword c(x)‘a’ 1‘d’ 01‘e’ 0000‘i’ 001‘o’ 00010‘u’ 00011

(a) Is the code prefix-free?

(b) Is the code uniquely decodable?

2-1

ECS 452 HW 2 — Due: Feb 19 2015/2

(c) Suppose the DMS produces the string ‘audio’. Find the output of the source encoder.

(d) Suppose the output of the source encoder is0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0

Find the corresponding source string produced by the DMS. Use “/” to indicate thelocations where the sting above is split into codewords.

Problem 2. A DMC has X = {0, 1} and Y = {1, 2, 3}. The following decoding table isused to decode the channel output.

y x̂(y)1 02 13 0

Suppose the channel output string is 212213221133122122132.

(a) Find the corresponding decoded string.

(b) Suppose the channel input string is produced from an ASCII source encoder by thecommand dec2bin(SourceString,7) in MATLAB. Assume that there is no channeldecoding error. Find the corresponding source string.

Problem 3 (HW2-2015-2). Consider a BSC whose crossover probability for each bit isp = 0.35. Suppose P [X = 0] = 0.45.

(a) Draw the channel diagram.

2-2

ECS 452 HW 2 — Due: Feb 19 2015/2

(b) Find the channel matrix Q.

(c) Find the joint pmf matrix P.

(d) Find the row vector q which contains the pmf of the channel output Y .

(e) We now analyze the performance of all four reasonable detectors for this binary channel.Complete the table below:

x̂(y) P[X̂ = 0 |X = 0

]P[X̂ = 1 |X = 1

]P (C) P (E)

y1− y

10

(f) Find the MAP detector and its error probability.

(g) Find the ML detector and its error probability.

2-3

ECS 452 HW 2 — Due: Feb 19 2015/2

Problem 4 (HW2-2015-2). Consider a DMC whose X = {1, 2, 3}, Y = {1, 2, 3}, and Q = 0.5 0.2 0.30.3 0.4 0.30.2 0.2 0.6

. Suppose the input probability vector is p = [0.2, 0.4, 0.4].

(a) Find the joint pmf matrix P.

(b) Find the row vector q which contains the pmf of the channel output Y .

(c) Find the error probability of the naive decoder.

(d) Find the error probability of the (DIY) decoder x̂(y) = 4− y.

2-4

ECS 452 HW 2 — Due: Feb 19 2015/2

(e) Find the MAP detector and its error probability.

(f) Find the ML detector and its error probability.

Problem 5 (HW2-2015-2). Consider a BAC whose Q(1|0) = 0.35 and Q(0|1) = 0.55.Suppose P [X = 0] = 0.4.

(a) Draw the channel diagram.

(b) Find the joint pmf matrix P.

2-5

ECS 452 HW 2 — Due: Feb 19 2015/2

(c) Find the row vector q which contains the pmf of the channel output Y .

(d) We now analyze the performance of all four reasonable detectors for this binary channel.Complete the table below:

x̂(y) P[X̂ = 0 |X = 0

]P[X̂ = 1 |X = 1

]P (C) P (E)

y1− y

10

2-6

ECS 452 HW 2 — Due: Feb 19 2015/2

(e) Find the MAP detector and its error probability.

(f) Find the ML detector and its error probability.

Problem 6 (HW2-2015-2). (Optional) Consider a DMC whose samples of input X andoutput Y are recorded as row vectors x and y in the file HW DMC Channel Data.mat. WriteMATLAB script which uses the recorded information to estimate the quantities below. Notethat most of these can be solved by appropriate parts of the m-files posted on the courseweb site.

(a) The support X of X.

(b) The support Y of Y .

(c) The row vector p which contains the pmf of X.

(d) The Q matrix.

(e) The row vector q which contains the pmf of Y . Do this using two methods:

(i) Count directly from the observed values of Y .

(ii) Use the estimated values of p and Q.

2-7

ECS 452 HW 2 — Due: Feb 19 2015/2

(f) The error probability when the naive decoder is used. Do this using two methods:

(i) Directly construct x̂ from y. Then, compare x̂ and x.

(ii) Use the estimated values of p and Q.

(g) The error probability when the MAP decoder is used. Do this using two methods:

(i) First find the MAP decoder table using the estimated values of p and Q. Then,construct x̂ from y according to the decoder table. Finally, compare x̂ and x.

(ii) Use the estimated values of p and Q to directly calculate the error probability.

(h) The error probability when the ML decoder is used. Do this using two methods:

(i) First find the ML decoder table using the estimated value of Q. Then, constructx̂ from y according to the decoder table. Finally, compare x̂ and x.

(ii) Use the estimated values of p and Q to directly calculate the error probability.

2-8

% MATLAB Script for Q6 of HW2 for ECS 452

% By Asst. Prof. Dr. Prapun Suksompong.

close all; clear all;

tic

load HW_DMC_Channel_Data

% Part

(a) ----------------------------------------------------------------

S_X = unique(x)

% Part

(b) ----------------------------------------------------------------

S_Y = unique(y)

%% Statistical Analysis

n = length(x);

% Part

(c) ----------------------------------------------------------------

% The probability values for the channel inputs

p_X_sim = hist(x,S_X)/n % Relative frequencies from the simulation

% Part

(d) ----------------------------------------------------------------

% The channel transition probabilities from the simulation

Q_sim = [];

for k = 1:length(S_X)

I = find(x==S_X(k)); LI = length(I);

rel_freq_Xk = LI/n;

yc = y(I);

cond_rel_freq = hist(yc,S_Y)/LI; Q_sim = [Q_sim; cond_rel_freq];

end

Q_sim % Relative frequencies from the simulation

% Part

(e) ----------------------------------------------------------------

% Part

(e.i) --------------------------------------------------------------

p_Y_sim = hist(y,S_Y)/n % Relative frequencies from the simulation

% Part

(e.ii) -------------------------------------------------------------

p_Y_sim2 = p_X_sim*Q_sim

%%

p_X = p_X_sim;

Q = Q_sim;

% Part

(f) ----------------------------------------------------------------

%% Naive Decoder

x_hat = y;

% Part

Q6 MATLAB Simulation of the system in Q4Saturday, September 13, 2014 11:44 PM

ECS452_HW_2015_2_Sol Page 1

% Part

(f.i) --------------------------------------------------------------

% Error Probability

PE_sim_Naive = 1-sum(x==x_hat)/n % Error probability from the simulation

% Part

(f.ii) -------------------------------------------------------------

% Calculation of the theoretical error probability

PC = 0;


t = S_X(k);

i = find(S_Y == t);

if length(i) == 1

PC = PC+ p_X(k)*Q(k,i);

end

end

PE_theoretical_Naive = 1-PC

% Part

(g) ----------------------------------------------------------------

%% MAP Decoder

P = diag(p_X)*Q; % Weight the channel transition probability by the

% corresponding prior probability.

[V I] = max(P); % For I, the default MATLAB behavior is

%that when there are multiple max, the

% index of the first one is returned.

Decoder_Table_MAP = S_X(I) % The decoded values corresponding to the

% received Y

% Part

(g.i) --------------------------------------------------------------

% Decode according to the decoder table

x_hat = y; % preallocation

for k = 1:length(S_Y)

I = (y==S_Y(k));

x_hat(I) = Decoder_Table_MAP(k);

end

PE_sim_MAP = 1-sum(x==x_hat)/n % Error probability from the simulation

% Part

(g.ii) -------------------------------------------------------------


Decoder_Table = Decoder_Table_MAP;

PC = 0;


I = (Decoder_Table == S_X(k));

Q_row = Q(k,:);

PC = PC+ p_X(k)*sum(Q_row(I));

end

PE_theoretical_MAP = 1-PC

% Part

(h) ----------------------------------------------------------------

%% ML Decoder

[V I] = max(Q); % For I, the default MATLAB behavior is

%that when there are multiple max, the

% index of the first one is returned.

Decoder_Table_ML = S_X(I) % The decoded values corresponding to the

% received Y


% received Y

% Decode according to the decoder table

x_hat = y; % preallocation

for k = 1:length(S_Y)

I = (y==S_Y(k));

x_hat(I) = Decoder_Table_ML(k);

end

% Part

(h.i) --------------------------------------------------------------

PE_sim_ML = 1-sum(x==x_hat)/n % Error probability from the simulation

% Part

(h.ii) -------------------------------------------------------------


Decoder_Table = Decoder_Table_ML;

PC = 0;


I = (Decoder_Table == S_X(k));

Q_row = Q(k,:);

PC = PC+ p_X(k)*sum(Q_row(I));

end

PE_theoretical_ML = 1-PC

toc

>> HW_DMC_Channel_Estimation_2S_X = 1 2 3S_Y = 1 2 3p_X_sim = 0.2004 0.4002 0.3994Q_sim = 0.4988 0.2012 0.3000 0.3002 0.4000 0.2998 0.2000 0.2003 0.5997p_Y_sim = 0.3000 0.2804 0.4196p_Y_sim2 = 0.3000 0.2804 0.4196PE_sim_Naive = 0.5004PE_theoretical_Naive = 0.5004Decoder_Table_MAP = 2 2 3PE_sim_MAP = 0.4802PE_theoretical_MAP = 0.4802

Results in the command window:


0.4802PE_theoretical_MAP = 0.4802Decoder_Table_ML = 1 2 3PE_sim_ML = 0.5004PE_theoretical_ML = 0.5004Elapsed time is 0.410464 seconds.



HW 3 — Due: Feb 26


Instructions

(a) Solve all non-optional problems. (5 pt)






Problem 1 (HW3-2015-2). Consider a repetition code with a code rate of 1/5. Assumethat the code is used with a BSC with a crossover probability p = 0.4.

(a) Find the ML detector and its error probability.

3-1

ECS 452 HW 3 — Due: Feb 26 2015/2

(b) Suppose the info-bit S is generated with

P [S = 0] = 1 − P [S = 1] = 0.4.

Find the MAP detector and its error probability.

(c) Assume the info-bit S is generated with

P [S = 0] = 1 − P [S = 1] = 0.45.

Suppose the receiver observes 01001.

(i) What is the probability that 0 was transmitted? (Do not forget that this isa conditional probability. The answer is not 0.45 because we have some extrainformation from the observed bits at the receiver.)

3-2

ECS 452 HW 3 — Due: Feb 26 2015/2

(ii) What is the probability that 1 was transmitted?

(iii) Given the observed 01001, which event is more likely, S = 1 was transmitted orS = 0 was transmitted? Does your answer agree with the majority voting rulefor decoding?

(d) Assume that the source produces source bit S with

P [S = 0] = 1 − P [S = 1] = p0.

Suppose the receiver observes 01001.

(i) What is the probability that 0 was transmitted?

(ii) What is the probability that 1 was transmitted?

3-3

ECS 452 HW 3 — Due: Feb 26 2015/2

(iii) Given the observed 01001, which event is more likely, S = 1 was transmitted orS = 0 was transmitted? Your answer may depend on the value of p0. Does youranswer agree with the majority voting rule for decoding?

Problem 2 (HW3-2015-2). A channel encoder map blocks of two bits to five-bit (channel)codewords. The four possible codewords are 00000, 01000, 10001, and 11111. A codewordis transmitted over the BSC with crossover probability p = 0.1.

(a) What is the minimum (Hamming) distance dmin among the codewords?

(b) Suppose the codeword x = 10001 was transmitted. What is the probability that thereceiver observes y = 01001 at the output of the BSC.

3-4

Din

Pencil

Din

Pencil

Din

Pencil

Din

Pencil

ECS 452 HW 3 — Due: Feb 26 2015/2

(c) Suppose the receiver observes 01001 at the output of the BSC.

(i) Assume that all four codewords are equally likely to be transmitted. Given theobserved 01001 at the receiver, what is the most likely codeword that was trans-mitted?

(ii) Assume that the four codewords are not equally likely. Suppose 11111 is transmit-ted more frequently with probability 0.7. The other three codewords are trans-mitted with probability 0.1 each.

Given the observed 01001 at the receiver, what is the most likely codeword thatwas transmitted?

3-5

ECS 452 HW 3 — Due: Feb 26 2015/2

Problem 3 (HW3-2015-2, Optional). Optimal code lengths that require one bit aboveentropy : The source coding theorem says that the Huffman code for a random variable Xhas an expected length strictly less than H(X) + 1. Give an example of a random variablefor which the expected length of the Huffman code (without any source extension) is veryclose to H(X) + 1.

3-6


HW 4 — Due: Not Due


Problem 1 (HW4-2015-2, Free). In each row of the table below, compare the entropy H(X)of the random variable X in the first column with the entropy H(Y ) of the random variableY in the third column by writing “>”, “=”, or “<” in the second column. Watch out forapproximation and round-off error.

H(X) when p = [0.3, 0.7]. H(Y ) when q = [0.8, 0.2].

H(X) when p = [0.3, 0.3, 0.4]. H(Y ) when q = [0.4, 0.3, 0.3].

H(X) when p (x) =

0.3, x ∈ {1, 2} ,0.2, x ∈ {3, 4} ,0, otherwise.

H(Y ) when q = [0.4, 0.3, 0.3].

Problem 2 (HW4-2015-2, Free). Consider random variables X and Y whose joint pmf isgiven by

pX,Y (x, y) =

{c (x+ y) , x ∈ {1, 3} and y ∈ {2, 4} ,0, otherwise.

Evaluate the following quantities.

(a) c

(b) H(X, Y )

(c) H(X)

(d) H(Y )

(e) H(X|Y )

(f) H(Y |X)

(g) I(X;Y )

4-1

ECS 452 HW 4 — Due: Not Due 2015/2

Problem 3 (HW4-2015-2, Free). Consider a pair of random variables X and Y whose jointpmf is given by

pX,Y (x, y) =

1/15, x = 3, y = 1,2/15, x = 4, y = 1,4/15, x = 3, y = 3,β, x = 4, y = 3,0, otherwise.

(a) Find the value of the constant β.

(b) Are X and Y independent?

(c) Evaluate the following quantities.

(i) H(X)

(ii) H(Y )

(iii) H(X, Y )

(iv) H(X|Y )

(v) H(Y |X)

(vi) I(X;Y )

4-2

ECS 452: Digital Communication Systems 2015/2 HW 1 | Due ... 2015 PreMidterm … · ECS 452:...

Documents

Transcript of ECS 452: Digital Communication Systems 2015/2 HW 1 | Due ... 2015 PreMidterm … · ECS 452:...