Shannon_Coding_Extensions.pdf

7/30/2019 Shannon_Coding_Extensions.pdf

1/139

1-1

Shannon Coding for the Discrete NoiselessChannel and Related Problems

Sept 16, 2009

Man DU Mordecai GOLIN Qin ZHANG

Barcelona

HKUST


2/139

2-1

This talks punchline: Shannon Coding can be

algorithmically useful

Shannon Coding was introduced by Shan-

non as a proof technique in his noiseless

coding theorem

Shannon-Fano coding is whats primarily

used for algorithm design

Overview


3/139

3-1

Outline

Huffman Coding and Generalizations

A Counterexample

New Work

Previous Work & Background

Open Problems


4/139

4-1

Prefix-free coding

Let = {1, 2, . . . , r} be an encoding alphabet.Word w is a prefix of word w if w = wuwhere u is a non-empty word. A Code over is a collection of words C = {w1, . . . , wn}.


5/139

4-2

Prefix-free coding


Code C is prefix-free if for all i = j wi is not a prefix of wj .

{0, 10, 11} is prefix-free. {0, 00, 11} isnt.


6/139

4-3

Prefix-free coding


Code C is prefix-free if for all i = j wi is not a prefix of wj .

{0, 10, 11} is prefix-free. {0, 00, 11} isnt.

0 1

0 0

0 0

1

11

1

w1

w2 w3

w4

w5 w6

w1 = 00

w2 = 010

w3 = 011

w4 = 10

w5 = 110

w6 = 111

A prefix-free code can be modelled as (leaves of) a tree


7/1395-1

The prefix coding problem

Define cost(C) = ni=1 cost(wi)pi

Let cost(w) be the length or number of charactersin w. Let P = {p1, p2, . . . , pn} be a fixed discreteprobability distribution (P.D.).


8/1395-2




The prefix coding problem, sometimes known as theHuffman encodingproblem is to find a prefix-free codeover of minimum cost.


9/1395-3





0 10 0

0 0

1

11

1

1

4

1

81

8

1

4

1

81

8

Equivalent to finding tree withminimum external path-length


10/1395-4





0 10 0

0 0

1

11

1

1

4

1

81

8

1

4

1

81

8

Equivalent to finding tree withminimum external path-length

2 14

+ 14

+ 3

18

+ 18

+ 18

+ 18


11/1396-1


Useful for Data transmission/storage.

Modelling search problems

Very well studied


12/1397-1

Whats known

Sub-optimal codes

Shannon coding: (from noiseless coding theorem)There exists a prefix-free code with word lengthsi = logr pi, i = 1, 2, . . . , n .

Shannon-Fano coding: probability splittingTry to put 1

rof the probability in each node.


13/1397-2

Whats known

Sub-optimal codes




Both methods have cost within 1 of optimal


14/1397-3

Whats known

Huffman 1952: a well-know O(rn log n)-time greedy-algorithm (O(rn)-time if the pi are sorted in non-decreasing order)

Optimal codes

Sub-optimal codes




Both methods have cost within 1 of optimal


15/1398-1

Whats not as well known

The fact that the greedy Huffman algorithm worksis quite amazing

Almost any possible modification or generalization tothe original problem causes greedy to fail

For some simple modifications, we dont even havepolynomial time algorithms.


16/1399-1

Generalizations: Min cost prefix coding

Unequal-cost coding

Allow letters to have different costs, say, c(j) = cj .

Discrete Noiseless Channels (in Shannons original paper)This can be viewed as a strongly connected aperiodic directedgraph with k vertices (states).

1. Each edge leaving a vertex is labelled by an encoding letter

, with at most one -edge leaving each vertex.

2. An edge labelled by leaving vertex i has cost ci,.

Language restrictions

Require all codewords to be contained in some given Language L


17/13910-1

Generalizations: Prefix-free coding ....

With Unequal-cost letters

a,1

a,1a,1

b,2

b,2b,2

2/6, aaa, 31/6, aab, 4

1/6, ab, 3

2/6, b, 2c1 = 1; c2 = 2. pi, wi, c(wi)


18/13910-2



a,1

a,1a,1

b,2

b,2b,2

2/6, aaa, 31/6, aab, 4

1/6, ab, 3

2/6, b, 2c1 = 1; c2 = 2.

Corresponds to different letter transmission/storage costs, e.g.,

the Telegraph Channel.Also, to different costs for evaluating test outcomes in, e.g.,group testing.

pi, wi, c(wi)


19/13910-3



a,1

a,1a,1

b,2

b,2b,2

2/6, aaa, 31/6, aab, 4

1/6, ab, 3

2/6, b, 2c1 = 1; c2 = 2.

Corresponds to different letter transmission/storage costs, e.g.,

the Telegraph Channel.Also, to different costs for evaluating test outcomes in, e.g.,group testing.

Size of encoding alphabet, , could be countably infinite!

pi, wi, c(wi)


20/13911-1


In a Discrete Noiseless Channel

S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

1/6, aaa, 4

2/6, b, 3

1/6, abb, 51/6, aab, 5

1/6, aba, 4


21/13911-2



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

1/6, aaa, 4

2/6, b, 3

1/6, abb, 51/6, aab, 5

1/6, aba, 4

Cost of letter depends upon current state.

In Shannons original paper, k = # states and || are both finite


22/139

11-3



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

1/6, aaa, 4

2/6, b, 3

1/6, abb, 51/6, aab, 5

1/6, aba, 4



A codeword has both start and end states. In coded message,new codeword must start from final state of preceeding one.


23/139

11-4



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

1/6, aaa, 4

2/6, b, 3

1/6, abb, 51/6, aab, 5

1/6, aba, 4



A codeword has both start and end states. In coded message,new codeword must start from final state of preceeding one.

Need k code trees; each one rooted with different state


24/139

12-1


With Language Restrictions


25/139

12-2



Find min-cost prefix code in which all words belong to givenlanguage L.


26/139

12-3




Example: L = 01, all binary words ending in 1.Used in constructing self-synchronizing codes.


27/139

12-4




Example: L = 01, all binary words ending in 1.Used in constructing self-synchronizing codes.

One of the problems that motivated this research.Let L be the set of all binary words that do not contain a givenpattern, e.g., 010.

No previous good way of finding min cost prefix code with suchrestrictions.


28/139

13-1


With Regular Language Restrictions


29/139

13-2



In this case, there is a DFA M accepting Language L.


30/139

13-3




S1 S4

1

1

1

1

0 000

S2 S3

L = ((0 + 1)000)C

= binary strings not ending in 000


31/139

13-4




Erasing the nonaccepting states, M can be drawn with a finite #

of states but a countably infinite encoding alphabet.

S1 S4

1

1

1

1

0 000

S2 S3

L = ((0 + 1)000)C


S1

1 1

1

0 0

01

01

S2 S3


32/139

13-5




Erasing the nonaccepting states, M can be drawn with a finite #

of states but a countably infinite encoding alphabet.

S1 S4

1

1

1

1

0 000

S2 S3

L = ((0 + 1)000)C


S1

1 1

1

0 0

01

01

S2 S3Note: graph doesnt need to

strongly connected. It might even

have sinks!


33/139

14-1




34/139

14-2



S1

1 1

1

0 0

0101

S2 S3


35/139

14-3



S1

1 1

1

0 0

0101

S2 S3

Can still be rewritten as amin-cost tree problem

S1

S2 S1

0 1

S1

1

S3

0

S1

1

S2

0

S1

1

S1

001

S1

01

S1

. . .


36/139

15-1

Outline


A Counterexample

New Work


Open Problems


37/139

16-1

Previous Work: Unequal Cost Coding

Letters in have different costs c1 c2 c3 cr.Models different transmission/storage costs


38/139

16-2


Letters in have different costs c1 c2 c3 cr.Models different transmission/storage costs

Karp (1961) Integer Linear Programming Solution

Blachman (1954), Marcus (1957), Gilbert (1995) Heuristics

G., Rote (1998) O(ncr+2) DP solution

Bradford, et. al. (2002), Dumitrescu(2006) O(ncr)

G., Kenyon, Young (2002) A PTAS


39/139

16-3


Letters in have different costs c1 c2 c3 cr.

Still dont know if its NP-Hard, in P or something between.

Models different transmission/storage costs






Big Open Question


40/139

16-4


Letters in have different costs c1 c2 c3 cr.

Still dont know if its NP-Hard, in P or something between.

Models different transmission/storage costs






Big Open Question

Most Practical Solutions are arithmetic error approximations


41/139

17-1



42/139

17-2


Efficient algorithms (O(n log n) or O(n)) that createcodes which are within an additive error of optimal.

COST OP T + K


43/139

17-3



COST OP T + K

Krause (1962)

Csiszar (1969)

Cott (1977)

Altenkamp and Mehlhorn (1980)

Mehlhorn (1980)

G. and Li (2007)


44/139

17-4



COST OP T + K

Krause (1962)

Csiszar (1969)

Cott (1977)


Mehlhorn (1980)

G. and Li (2007)

K is a function of letter costs c1, c2, c3, . . .

K(c1, c2, c3, . . .) are incomparable between different algorithms

K is often function of longest letter length cr, problem when r = .


45/139

17-5



COST OP T + K

Krause (1962)

Csiszar (1969)

Cott (1977)


Mehlhorn (1980)

G. and Li (2007)

K is a function of letter costs c1, c2, c3, . . .

K(c1, c2, c3, . . .) are incomparable between different algorithms

All algorithms above are Shannon-Fano type codes; differ in how they

define approximate split


46/139

18-1

Previous Work:

The Discrete Noiseless Channel: Only previous result seems tobe Csiszar (1969) who gives additive approximation to optimalcode, again using a generalization of Shannon-Fano splitting.


47/139

18-2

Previous Work:


Language Constraints

1-ended codes:

Capocelli, et.al., (1994) Berger, Yeung(1990) Exponential Search

Chan, G. (2000) O(n3) DP algorithm

Sound of Silence Binary Codes with at most k zeros

Dolev, et. al. (1999) nO(k) DP algorithm

General Regular Language Constraint

Folk theorem: If a DFA with m states accepting L, optimal codecan be built in nO(m) time. (O(m) 3m.)


48/139

18-3

Previous Work:


Language Constraints

1-ended codes:

Capocelli, et.al., (1994) Berger, Yeung(1990) Exponential Search

Chan, G. (2000) O(n3) DP algorithm

Sound of Silence Binary Codes with at most k zeros

Dolev, et. al. (1999) nO(k) DP algorithm

General Regular Language Constraint

Folk theorem: If a DFA with m states accepting L, optimal codecan be built in nO(m) time. (O(m) 3m.)

No good efficient algorithm known


49/139

19-1

Previous Work:

Pre-Huffman there were two Sub-optimal construc-tions for basic case


50/139

19-2

Previous Work:

Pre-Huffman there were two Sub-optimal construc-tions for basic case





51/139

20-1

l2

Shannon Coding vs. Shannon-Fano Coding

Shannon Coding

l1 li = logr pi


52/139

20-2

l2


Shannon Coding

l1 li = logr pi

Given depths li, can build tree via

top-down linear scan. Whenmoving down a level, expand allnon-used leaves to be parents.


53/139

20-3

l2


Shannon Coding

l1 li = logr pi

Shannon-Fano Coding

p1, p2, . . . , pn

p1, p2, . . . , pi1 pi1+1, . . . , pi2 pir1+1, . . . , pn

1/r 1/r 1/r

Given depths li, can build tree via

top-down linear scan. Whenmoving down a level, expand allnon-used leaves to be parents.


54/139

21-1


Example: p1 = p2 =13

, p3 = p4 = p5 = p6 =112


55/139

21-2



, p3 = p4 = p5 = p6 =112

Shannon coding

1

3

1

3

1

12

1

12

1

12

1

12

l1 = l2 = 2 = log213

l3 = l4 = l5 = l6 = 4 = log2

112

Has empty slots

can be improved


56/139

21-3



, p3 = p4 = p5 = p6 =112

Shannon coding

1

3

1

3

1

12

1

12

1

12

1

12

Shannon-Fano coding

13

1

3

1

12

1

12

1

12

1

12


57/139

22-1

Shannon Coding vs. Shannon-Fano CodingExample: p1 = p2 =

13

, p3 = p4 = p5 = p6 =112

Shannon-Fano: First, sort items and insert at root.While a node contains more than 1 item, split its items weightsas evenly as possible. At most 1/2 nodes weight in left child.


58/139

22-2


13

, p3 = p4 = p5 = p6 =112

1

3

1

3, 112

, 112

, 112

, 112



59/139

22-3


13

, p3 = p4 = p5 = p6 =112

1

3

1

3, 112

, 112

, 112

, 112

1

3

112

, 112

, 112

, 112

1

3



60/139

22-4


13

, p3 = p4 = p5 = p6 =112

1

3

1

3, 112

, 112

, 112

, 112

1

3

112

, 112

, 112

, 112

1

3

1

3

112

, 112

1

3

112

, 112



61/139

22-5


13

, p3 = p4 = p5 = p6 =112

1

3

1

3, 112

, 112

, 112

, 112

1

3

112

, 112

, 112

, 112

1

3

1

3

112

, 112

1

3

112

, 112

1

3

1

3

112

112

112

112



62/139

23-1

Shannon Fano coding for unequal cost codes

p1, p2, . . . , pn

p1, p2, . . . , pi1

pi1+1, . . . , pi2

pik1+1, . . . , pn

c1

c2 ck

Previous Work. Unequal Cost Codes

c1

c2 ck

: unique positiveroot of

ci = 1


63/139

23-2


p1, p2, . . . , pn

p1, p2, . . . , pi1

pi1+1, . . . , pi2

pik1+1, . . . , pn

c1

c2 ck


c1

c2 ck

Split probabilities so approximately ci of the probability in anode is put into its ith child.


ci = 1


64/139

23-3


p1, p2, . . . , pn

p1, p2, . . . , pi1

pi1+1, . . . , pi2

pik1+1, . . . , pn

c1

c2 ck


c1

c2 ck



ci = 1

Note: This can work for infinite alphabets, as long as exists.


65/139

23-4


p1, p2, . . . , pn

p1, p2, . . . , pi1

pi1+1, . . . , pi2

pik1+1, . . . , pn

c1

c2 ck


c1

c2 ck



ci = 1

All previous algorithms were Shannon-Fano like.

They differed in how they implemented approximate split.


66/139

24-1

Shannon-Fano coding for unequal cost codes

: unique positive root of

ci

= 1Split probabilities so approximately ci of the probability in anode is put into its ith child.


67/139

24-2


: unique positive root ofci = 1Split probabilities so approximately ci of the probability in anode is put into its ith child.

Example: Telegraph Channel: c1 = 1, c2 = 2

1 =512

W

W/

W/2Put

1

of the roots weightin the left subtree and 2 ofthe weight in the right


68/139

25-1




69/139

25-2



Example: 1-ended coding. i > 0, ci = i.

ci = 1 gives 1 = 1

2

Put 2i of a nodes weight

into its ith subtree

ith encoding letter denotes string 0i11.

0001

0001

001

001

01

01

1

1


70/139

26-1

Given coding letter lengths C = {c1

, c2

, c3

, . . .}, gcd(ci) = 1,

let be the unique positive root of g(z) = 1

j cj

Previous Work. Well Known Lower Bound


71/139

26-2

Given coding letter lengths C = {c1, c2, c3, . . .}, gcd(ci) = 1,

let be the unique positive root of g(z) = 1

j cj

Note: sometimes called the capacity



72/139

26-3

Given coding letter lengths C = {c1, c2, c3, . . .}, gcd(ci) = 1,let be the unique positive root of g(z) = 1

j

cj


For given P.D. set H = pi log

pi.



73/139

26-4


j

cj



pi.

Note: If c1 = c2 = 1 then = 2 and H is standard entropy



74/139


75/139

26-6


j

cj



pi.

Note: If c1 = c2 = 1 then = 2 and H is standard entropy

Theorem:Let OP T be cost of min-cost code for given P.D. and letter

costs. Then

H OPT

Note: If c1 = c2 = 1 then = 2 and this is classicShannon Information Theoretic Lower Bound



76/139

27-1

Outline


A Counterexample

New Work


Open Problems


77/139

28-1

Shannon coding only seems to have been used in the proof ofthe noiseless coding theorem. It never seems to have actuallybeen used as an algorithmic tool.


78/139

28-2


All of the (additive-error) approximation algoritms for unequalcost coding and Csiszars (1969) approximation algorithm forcoding in a Discrete Noiseless Channel, were variations of

Shannon-Fano coding


79/139

28-3

The main idea behind our new results is thatShannon-Fano splitting is not necessary;Shannon-coding suffices



Shannon-Fano coding


80/139

28-4

The main idea behind our new results is thatShannon-Fano splitting is not necessary;Shannon-coding suffices



Shannon-Fano coding

Yields efficient additive-error approximation algorithms for un-equal cost coding and the Discrete Noiseless Channel, as well asfor regular language constraints.


81/139

29-1

New Results for Unequal Cost Coding


82/139

29-2


Given coding letter lengths C, let be capacity.Then K > 0, depending only upon C, such that if

1. P = {p1, p2, . . . , pn} is any P.D., and

2. 1, 2, . . . , n any set of integers such thati, i K+ logpi,

then there exists a prefix free code for which the i arethe word lengths.


83/139

29-3






i

pii K+ 1 + H(P) OP T + K+ 1


84/139

29-4






i

pii K+ 1 + H(P) OP T + K+ 1

This gives an additive approximation of same type as Shannon-Fano splitting without the splitting (same time complexity but

many fewer operations on reals).


85/139

29-5






i

pii K+ 1 + H(P) OP T + K+ 1

Same result holds for DNC and regular language restrictions. is a function of the DNC or L-accepting automaton graph


86/139

30-1

Proof of the Theorem

We first prove the following lemma.

Given C and corresponding then > 0 depending only upon C such that if

n

i=1

i ,

then there exists a prefix-free code with word lengths1, 2, . . . , n.


87/139

30-2

Proof of the Theorem

We first prove the following lemma.

Given C and corresponding then > 0 depending only upon C such that if

n

i=1

i ,

then there exists a prefix-free code with word lengths1, 2, . . . , n.

Note: if c1 = c2 = 1 then = 2. Let = 1 and conditionbecomes

2i 1.

Lemma then becomes one direction of Kraft Inequality.


88/139

31-1

Proof of the Lemma

Let L(n) be the number of nodes on level n of theinfinite tree corresponding to C

Can show t1, t2 s.t., t1n L(n) t2n.


89/139

31-2

Proof of the Lemma



l1


90/139

31-3

Proof of the Lemma



l1

l2


91/139

31-4

Proof of the Lemma



l1

l2

Grey regions are parts

of infinite tree that areerased when node k on kbecomes leaf.


92/139

31-5

Proof of the Lemma



l1

l2



Node on k has L(i k)

descendents on i

li


93/139

31-6

Proof of the Lemma



l1

l2



Node on i can becomeleaf iff grey regions do notcover all nodes on level i


descendents on i

li


94/139

31-7

Proof of the Lemma



l1

l2



Node on i can becomeleaf iff grey regions do notcover all nodes on level i


descendents on i

i1k=1 L( k) < L(i)

li


95/139

32-1

Just need to show that 0 < L(i)

i1k=1 L( k).

Proof of the Lemma


96/139

32-2


i1k=1 L( k).

Proof of the Lemma

L(i) i1

k=1

L( k) t1 t2

i1

k=1

k

t1 t2

i1k=1

k

(t1 t2)


97/139

32-3


i1k=1 L( k).

Proof of the Lemma

L(i) i1

k=1

L( k) t1 t2

i1

k=1

k

t1 t2

i1k=1

k

(t1 t2)

Choose < t1t2


98/139

32-4


i1k=1 L( k).

Proof of the Lemma

L(i) i1

k=1

L( k) t1 t2

i1

k=1

k

t1 t2

i1k=1

k

(t1 t2)

Choose < t1t2

> 0


99/139

33-1

Proof of the Main Theorem

Set K = log . (Recall li K + logpi)Then

n

i=1i

n

i=1K log pi

n

i=1

log pi =

ni=1

pi =


100/139

33-2

Proof of the Main Theorem

Set K = log . (Recall li K + logpi)Then

n

i=1i

n

i=1K log pi

n

i=1

log pi =

ni=1

pi =

From previous lemma, a prefix free code with those wordlengths 1, 2, . . . , n exists, and we are done

E l 1 2


101/139

34-1

Example: c1 = 1, c2 = 2

E l 1 2


102/139

34-2

Example: c1 = 1, c2 = 2

= 5+12 , K = 1

E l 1 2


103/139

34-3

Example: c1 = 1, c2 = 2

= 5+12 , K = 1

Consider p1 = p2 = p3 = p4 =14

E l 1 2


104/139

34-4

Example: c1 = 1, c2 = 2

= 5+12 , K = 1


Note that logpi = 3.

E l 1 2


105/139

34-5

Example: c1 = 1, c2 = 2

= 5+12 , K = 1


Note that logpi = 3.No tree with li = 3 exists.But, a tree with li = logpi + 1 = 4 does!

14

14

14

14

Th Al i h


106/139

35-1

The Algorithm

A valid K could be found by working through theproof of Theorem. Technically, O(1) but, practically,this would require some complicated operations onreals.

Th Al i h


107/139

35-2

The Algorithm


Alternatively, perform doubling search for K,the smallest K for which theorem is valid.

Set K = 1, 2, 22, 23 . . ..Test if i = K+ log pi has valid code (can be done efficiently)

until K is good but K/2 is not.

Th Al ith


108/139

35-3

The Algorithm





Note that K/2 < K K

Th Al ith


109/139

35-4

The Algorithm





Now set a = K/2, b = K, and binary search for K in [a,b].Note that K/2 < K K

Th Al ith


110/139

35-5

The Algorithm






Subtle point: Search will find K K for which code exists.

Th Al ith


111/139

35-6

The Algorithm


Time complexity O(n log K).





Subtle point: Search will find K K for which code exists.

The Al o ith fo i fi ite e codi al habets


112/139

36-1

The Algorithm for infinite encoding alphabets

(i) Root of

ci = 1 existsProof assumed two things.

(ii) t1, t2 s.t., t1n L(n) t2n

L(n) is number of nodes on level n of infinite tree



113/139

36-2


(i) Root of




This is always true for finite encoding alphabet



114/139

36-3


(i) Root of





Not necessarily true for infinite encoding alphabets

Will see simple example in next section



115/139

36-4


(i) Root of







But, if (i) and (ii) are true for an infinite alphabet

Theorem/algorithm hold



116/139

36-5


(i) Root of







But, if (i) and (ii) are true for an infinite alphabet

Example: 1-Ended codes. ci = i.

= 12

and (ii) is true Theorem/algorithm hold

Theorem/algorithm hold

Extensions to DNC and Regular Language Restrictions


117/139

37-1


Discrete Noiseless Channels

S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3



118/139

37-2



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

, , t1, t2 s.t., t1n L(n) t2

n

Let L(n) be number of nodes on level n of infinite tree

Fact that graph is biconnected and aperiodic implies that



119/139

37-3



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

, , t1, t2 s.t., t1n L(n) t2

n



Algorithm will still work for i K+ log pi,



120/139

37-4



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

, , t1, t2 s.t., t1n L(n) t2

n




Note: Algorithm must construct k different coding trees. One for each

state (tree root).



121/139

37-5



S2

S1S3

a, 1b, 2

b, 3

b, 3

a, 2

a, 1

start

S1

S2

S3

S3

S1

S2

S3

S2

S3

a, 1

a, 1

a, 1

b, 2a, 2

b, 3 b, 3

b, 3

, , t1, t2 s.t., t1n L(n) t2

n




Subtle point is that anynode on level li can be chosen for pi, independentof its state! Algorithm still works.



122/139

38-1


Regular Language RestrictionsAssumption: Language is aperiodic, i.e., N, such thatn > Nthere is at least one word of length n

S1 S4

11

1

1

0 000

S2 S3

S1

S2 S1

0 1

S1

1S3

0S1

1S2

0

S11

S1

001

S1

01

S1

. . .


, , t1, t2 s.t., t1n L(n) t2

nFact that language is aperiodic implies that

Algorithm will still work for i K + log pi,

is largest dominant eigenvalue of a conn component of the DFA.

Again, any node at level li can be labelled with pi, independent of state

Outline


123/139

39-1

Outline


A Counterexample

New Work


Conclusion and Open Problems

A Counterexample


124/139

40-1

A Counterexample

Let C be the countably infinite set defined by

|{j | cj = i}| = 2Ci1

where Ci =1

i+12ii is the i-th Catalan number.

Constructing prefix-free codes with these C can be shownto be equivalent to constructing balanced binary prefix-freecodes in which, for every word, the number of0s equals thenumber of 1s.

A Counterexample


125/139

40-2

A Counterexample


|{j | cj = i}| = 2Ci1

where Ci =1



No efficient additive-error approximation known.

A Counterexample


126/139

40-3

A Counterexample


|{j | cj = i}| = 2Ci1

where Ci =1



For this problem, the length of a balanced word = # of 0s in word.

e.g., |10| = 1, |001110| = 3.

No efficient additive-error approximation known.

A Counterexample


127/139

41-1

A Counterexample

Let L be the set of all balanced binary words.Set Q = {01, 10, 0011, 1100, 000111, . . .},the language of all balanced binary words without a balanced prefix.

Then L = Q and every word in L can be uniquely decomposedinto concatenation of words in Q.

A Counterexample


128/139

41-2

A Counterexample



# words of length i in Q is 2Ci1.

A Counterexample


129/139

41-3

A Counterexample



# words of length i in Q is 2Ci1.

Prefix coding in L is equivalent to prefix coding with infinite alphabet Q.

01

10 0011

1100

0110

0011 1100

A Counterexample


130/139

42-1

A Counterexample

Note: the characteristic equation is

g(z) = 1j

cj = 1i

2Ci1i =

1 4/

for which root does not exist ( = 4 is an algebraicsingularity).

Can prove that for , K, we can always find

p1, p2, . . . , pn s.t. there is no prefix code with length

li = K+ log pi

A Counterexample


131/139

43-1

A Counterexample

= 4 is algebraic singularity of characteristic equation

A Counterexample


132/139

43-2

A Counterexample

= 4 is algebraic singularity of characteristic equationCan prove that for 4, K, we can always findp1, p2, . . . , pn s.t. there is no prefix code with length

li = K+ log pi

A Counterexample


133/139

43-3

C p


li = K+ log pi

Can also prove that for < 4, K, , we can alwaysfind p1, p2, . . . , pn s.t. if prefix code with lengthsli K+ log pi exists, then

i lipi OP T > .

A Counterexample


134/139

43-4

p


li = K+ log pi

Can also prove that for < 4, K, , we can alwaysfind p1, p2, . . . , pn s.t. if prefix code with lengthsli K+ log pi exists, then

i lipi OP T > .

No Shannon-Coding type algorithm can guarantee anadditive-error approximation for a balanced prefix code.

Outline


135/139

44-1


A Counterexample

New Work





136/139

45-1

We saw how to use Shannon Coding to develop efficientapproximation algorithms for prefix-coding variants, e.g.,unequal cost cost coding, coding in the Discrete NoiselessChannel and coding with regular language constraints.



137/139

45-2


Old Open Question: is unequal-cost coding NP-complete?



138/139

45-3


Old Open Question: is unequal-cost coding NP-complete?

New Open Question: is there an additive-error approximationalgorithm for prefix coding using balanced strings?

We just saw that Shannon Coding doesnt work.G. & Li (2007) proved that (variant of) Shannon-Fano doesnt work.Perhaps no such algorithm exists.

The End


139/139

T HAN K YOU Q and A

Shannon_Coding_Extensions.pdf

Documents

Transcript of Shannon_Coding_Extensions.pdf