15-211Fundamental Structuresof Computer Science

Feb. 24, 2005Ananda Guna

Lempel-Ziv Compression

Recap

Huffman Trees

• Huffman Trees can be used to construct an optimal prefix code.

• What does optimal mean?

• Greedy algorithm to assemble a Huffman tree.• locally optimal steps to global optimization

•Requires symbol frequencies. • read the file twice – counting and encoding

Huffman Encoding Process

Adaptive Huffman or Dynamic Huffman

• Clearly, having to read the data twice (first for frequency count, then for actual compression) is a bit cumbersome. • Perhaps data is available in blocks (streaming data)

• Can build an adaptive Huffman tree that adjusts itself as more frequency data become available.

Adaptive Huffman ctd.. Mapping from source messages to

code words based upon a running estimate of the source message probabilities

Change the tree to remain optimal for the current estimates

adaptive Huffman codes respond to locality

Requires only a single pass of the data

Beating Huffman

How about beating the compression achieved by Huffman?

Impossible! It produces an optimal prefix code.

Right. But who says we have to use a prefix code?

Dictionary-BasedCompression

Dictionary-based methods Here is a simple idea:

Keep track of “words” that we have seen, and replace them with a code number when we see them again.

We can maintain dictionary entries(word, code)

and make additions to the dictionary as we read the input file.

Lempel & Ziv (1977/78)

Fred Hacker’s algorithm… Fred now knows what to do…

( <the-whole-file>, 1 )

Transmit 1, done.

Right?

Fred’s algorithm provides excellent compression, but…

…the receiver does not know what is in the dictionary!And sending the dictionary is the same

as sending the entire uncompressed file

Thus, we can’t decompress the “1”.

Hence…

…we need to build our dictionary in such a way that the receiver can rebuild the dictionary easily.

LZW Compression:The Byte Version

Byte method LZW We start with a trie that contains a

root and n childrenone child for each possible charactereach child labeled 0…n

When we compress as before, by walking down the triebut, after emitting a code and growing

the trie, we must start from the root’s child labeled c, where c is the character that caused us to grow the trie

LZW: Byte method example Suppose our entire character set

consists only of the four letters:{a, b, c, d}

Let’s consider the compression of the stringbaddad

Byte LZW: Compress example

baddadInput:^a bDictionary:

Output:

10 32

c d

Byte LZW: Compress example

baddadInput:^

a bDictionary:

Output:

10 32

c d

1

4

a

Byte LZW: Compress example

baddadInput:^

a bDictionary:

Output:

10 32

c d

10

4

a

5

d

Byte LZW: Compress example

baddadInput:^

a bDictionary:

Output:

10 32

c d

103

4

a

5

d

6

d

Byte LZW: Compress example

baddadInput:^

a bDictionary:

Output:

10 32

c d

1033

4

a

5

d

6

d

7

a

Byte LZW: Compress example

baddadInput:^

a bDictionary:

Output:

10 32

c d

10335

4

a

5

d

6

d

7

a

Byte LZW output So, the input

baddad compresses to

10335 which again can be given in bit form,

just like in the binary method… …or compressed again using

Huffman

Byte LZW: Uncompress example

The uncompress step for byte LZW is the most complicated part of the entire process, but is largely similar to the binary method

Byte LZW: Uncompress example

10335Input:^a bDictionary:

Output:

10 32

c d

Byte LZW: Uncompress example

10335Input:^

a bDictionary:

Output:

10 32

c d

b

Byte LZW: Uncompress example

10335Input:^

a bDictionary:

Output:

10 32

c d

ba

4

a

Byte LZW: Uncompress example

10335Input:^

a bDictionary:

Output:

10 32

c d

bad

4

a

5

d

Byte LZW: Uncompress example

10335Input:^

a bDictionary:

Output:

10 32

c d

badd

4

a

5

d

6

d

Byte LZW: Uncompress example

10335Input:^

a bDictionary:

Output:

10 32

c d

baddad

4

a

5

d

6

d

7

a

LZW Byte method:An alternative presentation

Getting off the ground

Suppose we want to compress a file containing only letters a, b, c and d.It seems reasonable to start with a dictionary

a:0 b:1 c:2 d:3

At least we can then deal with the first letter. And the receiver knows how to start.

Growing pains

Now suppose the file starts like so:a b b a b b …

We scan the a, look it up and output a 0.

After scanning the b, we have seen the word ab. So, we add it to the dictionary

a:0 b:1 c:2 d:3 ab:4

Growing pains

We output a 1 for the b. Then we get another b.

a b b a b b …

output 1, and add bb it to the dictionary

a:0 b:1 c:2 d:3 ab:4 bb:5

So?

Right, so far zero compression. But now we get a followed by b, and ab is in the dictionary

a b b a b b …

so we output 4, and put bab into the dictionary

… d:3 ab:4 bb:5 ba:6 bab:7

And so on

Suppose the input continuesa b b a b b b b a …

We output 5, and put bbb into the dictionary

… ab:4 bb:5 ba:6 bab:7 bbb:8

More Hits

As our dictionary grows, we are able to replace longer and longer blocks by short code numbers.

a b b a b b b b a …0 1 1 4 5 6

And we increase the dictionary at each step by adding another word.

More importantly Since we extend our dictionary in

such a simple way, it can be easily reconstructed on the other end.

Start with the same initialization, thenRead one code number after the other,

look up the each one in the dictionary, and extend the dictionary as you go along.

Again: Extending

where each prefix is in the dictionary.

We stop when we fall out of the dictionary:

a1 a2 a3 …. ak b

We scan a sequence of symbolsa1 a2 a3 …. ak

Again: Extending

We output the code for a1 a2 a3 …. ak andput a1 a2 a3 …. ak b into the dictionary.

Then we set a1 = b

And start all over.

Sort of

Let's take a closer look at an example.Assume alphabet {a,b,c}.

The code for aabbaabb is 0 0 1 1 3 5.

The decoding starts with dictionarya:0, b:1, c:2

Moving along

The first 4 code words are already in D. 0 0 1 1 3 5

and produce output a a b b. As we go along, we extend D:

a:0, b:1, c:2, aa:3, ab:4, bb:5For the rest we get

a a b b a a b b

Done

We have also added to D:ba:6, aab:7

But these entries are never used.

Everything is easy, since there is already an entry in D for each code number when we encounter it.

Is this it?

Unfortunately, no.

It may happen that we run into a code word without having an appropriate entry in D.

But, it can only happen in very special circumstances, and we can manufacture the missing entry.

A Bad Run

Consider input

a a b b b a a ==> 0 0 1 5 3

After reading 0 0 1, D looks like this:

a:0, b:1, c:2, aa:3, ab:4

Disaster

The next code is 5, but it’s not in D.

a:0, b:1, c:2, aa:3, ab:4

How could this have happened?Can we recover?

… narrowly averted

This problem only arises when • the input contains a substring …s s … • s was just added to the dictionary.

Here s is a single symbol, but a (possibly empty) word.

… narrowly averted

But then the fix is to outputx + first(x)

where x is the last decompressed word, and first(x) the first symbol of x.

And, we also update the dictionary to contain this new entry.

Example

In our example we had• s = b• w = empty

The output and new dictionary word is bb.

Another Example aabbbaabbaaabaababb ==> 0 0 1 5 3 6 7 9 5

Decoding (dictionary size: initial 3, final 11)a 0a + 0 aab + 1 abbb - 5 bbaa + 3 bbabba + 6 aabaab + 7 bbaaaaba - 9 aababb + 5 aabab

The problem cases

code position in Da 0a + 0 aa 3b + 1 ab 4bb - 5 bb 5aa + 3 bba 6bba + 6 aab 7aab + 7 bbaa 8aaba - 9 aaba 9bb + 5 aabab 10

Old vs. new

Ordinarily, we use an old dictionary word for the next code word.

But sometimes we immediately use what was last added to the dictionary.

But then it must be of the form s s and we can still decompress.

Pseudo Code: Compression

Initialize dictionary D to all words of length 1.

Read all input characters:output code words from D, extend D whenever a new word appears.

New code words: just an integer counter.

Less Pseudo initialize D; c = nextchar; // next input character W = c; // a string while( c = nextchar ) {

if( W+c is in D ) // dictionaryW = W + c;

elseoutput code(W); add W+c to D; W = c;

}output code(W)

Pseudo Code: Decompression

Initialize dictionary D with all words of length 1.Read all code words and

- output corresponding words from D,- extend D at each step.

This time the dictionary is of the form ( integer, word )

Keys are integers, values words.

Less Pseudo initialize D; pc = nextcode; // first code word pw = word(pc); // corresponding word output pw;

First code word is easy: codes only a single symbol.

Remember as pc (previous code) and pw (previous word).

More Less Pseudowhile( c = nextcode ) {

if( c is in D ) {cw = word(c); pw = word(pc);ww = pw + first(cw);insert ww in D;output cw;

}else {

The hard caseelse {

pw = word(pc);cw = pw + first(pw);insert cw in D;output cw;

}pc = c;

}

Implementation - TriesTries are the best way to implement LZWin the LZW situation, we can add the new word to the trie dictionary in O(1) steps after discovering that the string is no longer a prefix of a dictionary word.

Just add a new leaf to the last node touched.

LZW details

In reality, one usually restricts the code words to be 12 or 16 bit integers.

Hence, one may have to flush the dictionary ever so often.

But we won’t bother with this.

LZW details

Lastly, LZW generates as output a stream of integers.

It makes perfect sense to try to compress these further, e.g., by Huffman.

Summary of LZW

LZW is an adaptive, dictionary based compression method.

Encoding is easy in LZW, but uses a special data structure (trie).

Decoding is slightly complicated, requires no special data structures.