1

Huffman Codes

Drozdek Chapter 11

2

Objectives

You will be able to Construct an optimal variable bit length

code for an alphabet with known probability for each letter occuring in a message.

Huffman Code

Construct a tree for decoding messages encoded in a Huffman code.

Construct a tree for encoding messages encoded in a Huffman code.

3

Huffman Codes

Common character codes such as ASCII and EBCDIC use same size data structure for all characters. Eight bits per character.

Contrast Morse code Uses variable-length sequences.

Variable length codes can produce shorter messages than fixed length codes on average when applied to many messages

with given character probabilities.

4

Variable-Length Codes

Each character in such a codehas a weight (probability) and a length

The expected message length per character is the sum of the products of the code lengths and the probabilties for all the characters

(0.2*2) + (0.1*4) + (0.1*4) + (0.15*3) + (0.45*1) = 2.1

5

Immediate Decodability

When no sequence of bits that represents a character is a prefix of a longer sequence for another character Can be decoded without waiting for

remaining bits. Note how previous scheme is not

immediately decodable. And this one is

6

Immediate Decodability

Codes that are immediatly decodable are called prefix codes.

No valid code symbol is a prefix of another valid code symbol.

Perhaps better called prefix free codes.

7

Optimal Codes

We seek codes that are Immediately decodable. Average message length for a large

number of messages is minimal.

For a set of n characters { C1 .. Cn } with weights { w1 .. wn } We need an algorithm which generates

variable length bit strings representing the characters.

8

Huffman Codes

An optimal code scheme developed by David A. Huffman while a PhD student at MIT.

“A Method for the Construction of Minimum-Redundancy Codes” Proceedings of the I.R.E., Sept. 1952

http://en.wikipedia.org/wiki/David_A._Huffman

http://www.huffmancoding.com/david-huffman/scientific-american

9

Huffman's Algorithm

How to determine an optimal code for a set of N characters given their relative frequencies (or weights).

10

Huffman's Algorithm

Initialize a list of one-node binary trees One node for each character containing

the character and its weight.

While there is more than one tree in the list: Find two trees in the list having minimal weights. Remove those trees from the list and make them

the left and right subtrees of a new node having the sum of their weights as its weight.

Label the arc to the left subtree with 0. Label the arc to the right subtree with 1. Add the new tree to the list.

11

Huffman's Algorithm

The code for character Ci is the bit string along the path from the root to Ci in the final binary tree.

12

Example

Given characters and probabilities:

The end result isCharacter Huffman

  Code

A 011

B 000

C 001

D 010

E 1

Note arbitrary choice for sibling of D.

13

Alternate Result

Average message length is the same.

14

Huffman Decoding Algorithm

Given a message as a string of 0's and 1's:Initialize pointer p to the root of Huffman tree.

While end of message string not reached:Let x be the next bit of the message string.

If x is 0 move p to the left childelse move p to the right child

If p points to a leaf Display the character at that leaf. Reset p to the root of the Huffman tree.

15

Huffman Decoding Algorithm

For message string 0001011010 Using Huffman Tree and decoding

algorithm

Click for answerClick for answer

000 1 011 010

B E A D

16

Implementing a Huffman Code Program

Let’s implement a program to build a Huffman code tree. Encode and decode text messages

using the resulting Huffman code.

Limit input to letters and spaces. Convert to letters to lower case.

17

Implementing a Huffman Code Program

In order to create a Huffman code for English text, we need weighting factors for the letters. Frequency tables are readily available.

To simplify testing and debugging, start with the a small example: Just the letters A, B, C, D, and E

18

Getting Started

Create a new empty C++ project in Visual Studio, Huffman_Code or a directory in Unix.

Add a C++ code file main.cpp

19

main.cpp

#include <iostream>

using namespace std;

int main(void)

{

cout << "This is the Huffman Code program" << endl;

cin.get();

cin.get();

return 0;

}

Build and test

20

Program Running

21

Class char_freq

We need a class to hold the elements of a Huffman tree. Data

Character Frequency (Probability of occurance)

Pointers Left child Right child

Add class Char_Freq

22

Char_Freq.h

#pragma once

#include <iostream>

using std::ostream;

class Char_Freq

{

private:

char ch;

double freq;

Char_Freq* left;

Char_Freq* right;

public:

Char_Freq(void);

Char_Freq(char c, double f);

Char_Freq(char c, double f, Char_Freq* Left, Char_Freq* Right);

char Ch() const { return ch;};

double Freq() const { return freq;};

bool operator<(const Char_Freq& rhs) const;

friend ostream& operator<< (ostream& os, const Char_Freq& cf);

};

23

Char_Freq.cpp

#include "Char_Freq.h"

Char_Freq::Char_Freq(void)

{}

Char_Freq::Char_Freq(char c, double f) :

ch(c), freq(f), left(0), right(0)

{}

Char_Freq::Char_Freq(char c, double f, Char_Freq* Left, Char_Freq* Right) :

ch(c), freq(f), left(Left), right(Right)

{}

bool Char_Freq::operator<(const Char_Freq& rhs) const

{

return this->freq < rhs.freq;

}

ostream& operator<< (ostream& os, const Char_Freq& cf)

{

os << cf.ch << " " << cf.freq;

return os;

}

24

The Huffman Tree

Add class Huffman_Tree

Will hold code to build and access the Huffman code for a specific set of characters and frequencies.

25

Starting the Huffman Tree

We will build multiple trees of Char_Freq elements.

Keep the roots in a list. Use Standard Template Library list class.

Initially one tree per character to be coded. Each tree consists of root only.

Method Add() will be used to add char-freq pairs to the list

26

Huffman_Tree.h

#pragma once

#include <list>

#include "Char_Freq.h"

class Huffman_Tree

{

public:

Huffman_Tree(void);

~Huffman_Tree(void) {};

// Add a single node tree to the list.

void Add(char c, double frequency);

void Display_List(void);

private:

std::list<Char_Freq> node_list;

};

27

Huffman_Tree.cpp

#include <iostream>

#include <string>

#include "Huffman_Tree.h"

using namespace std;

Huffman_Tree::Huffman_Tree(void)

{}

void Huffman_Tree::Add(char c, double frequency)

{

Char_Freq cf(c, frequency);

node_list.push_back(cf);

}

28

Huffman_Tree.cpp

void Huffman_Tree::Display_List(void)

{

cout << "Character frequency list:" << endl;

list<Char_Freq>::iterator itr;

for (itr=node_list.begin(); itr!=node_list.end(); ++itr)

{

cout << *itr << endl;

}

}

29

main.cpp#include <iostream>

#include <string>

#include "Huffman_Tree.h"

using namespace std;

Huffman_Tree huffman_tree;

int main(void)

{

cout << "This is the Huffman code program.\n\n";

huffman_tree.Add('a', 0.2 );

huffman_tree.Add('b', 0.1 );

huffman_tree.Add('c', 0.1 );

huffman_tree.Add('d', 0.15);

huffman_tree.Add('e', 0.45);

huffman_tree.Display_List();

cin.get();

cin.get();

return 0;

}

30

Program in Action

31

Implementing Huffman’s Algorithm

Huffman’s algorithm requires us to identify two trees with minimal total frequency.

To do this we can sort the list. The < operator for the char_freq class

compares the frequency values. So the sort method of the list template

class will sort the trees into increasing order by frequency.

32

Implementing Huffman’s Algorithm

Add function Make_Decode_Tree to class Huffman_Tree.

Repeatedly Sort the list of trees by frequency Remove the first two trees Create a new node with these trees as subtrees.

Frequency is sum of their frequencies Add the new node to the list.

Continue until there is only one node on the list.

33

Huffman_Tree.h

Add new public method: void Make_Decode_Tree(void);

34

Huffman_Tree.cpp

Start by sorting the list. Display the sorted list.

void Huffman_Tree::Make_Decode_Tree(void)

{

node_list.sort();

cout << "\nSorted list:\n";

Display_List();

}

35

main.cpp

Add call to make_decode_tree.

int main(void)

{

cout << "This is the Huffman code program.\n";

huffman_tree.Add('a', 0.2 );

huffman_tree.Add('b', 0.1 );

huffman_tree.Add('c', 0.1 );

huffman_tree.Add('d', 0.15);

huffman_tree.Add('e', 0.45);

huffman_tree.Display_List();

huffman_tree.Make_Decode_Tree();

cin.get();

cin.get();

return 0;

}

36

Program in Action

37

Huffman_Tree.cpp

Add to function Make_Decode_Tree()

while (node_list.size() > 1)

{

Char_Freq* cf1 = new Char_Freq(node_list.front());

node_list.pop_front();

Char_Freq* cf2 = new Char_Freq(node_list.front());

node_list.pop_front();

Char_Freq cf3(0, cf1->Freq()+cf2->Freq(), cf1, cf2);

node_list.push_back(cf3);

node_list.sort();

}

This is the essence of Huffman’s algorithm!

38

Huffman_Tree.h

Add a new private member variable to class Huffman_Tree to hold the root of the tree.

private:

std::list<Char_Freq> node_list;

Char_Freq decode_tree_root;

};

39

Huffman_Tree.cpp

In order to check our results we need to be able to display the tree. Also show the code as a list.

Add public functions to Huffman_Tree.h:

void Display_Decode_Tree(Char_Freq* cf, int indent) const;

void Display_Code(Char_Freq* cf, std::string prefix) const;

Add at top of Huffman_Tree.cpp:#include <iomanip>

40

Display_Decode_Tree()

void Huffman_Tree::Display_Decode_Tree(Char_Freq* cf,

int indent) const

{

if (cf->left != 0)

{

Display_Decode_Tree(cf->left, indent + 8);

}

cout << setw(indent) << " " << *cf << endl;

if (cf->right != 0)

{

Display_Decode_Tree(cf->right, indent + 8);

}

}

Note access of private members of cf. Make class Huffman_Tree a friend of class Char_Freq.

41

Char_Freq.h

Add at the end of Char_Freq.h:

bool operator<(const Char_Freq& rhs) const;

friend ostream& operator<< (ostream& os, const Char_Freq& cf);

friend class Huffman_Tree;

};

42

char_freq.cpp

Update << to handle merged nodes ch will be 0

ostream& operator<< (ostream& os, const Char_Freq& cf)

{

if (cf.ch > 0)

{

os << cf.ch << " " << cf.freq;

}

else

{

os << '*' << " " << cf.freq;

}

return os;

}

43

Huffman_Tree.cpp

Add at the end of function Make_Decode_Tree()

decode_tree_root = node_list.front();

cout << endl << "The Huffman Tree" << endl;

Display_Decode_Tree(&decode_tree_root, 0);

44

Program in Action