7/29/2019 Coding Techs
1/45
1
Chapter 5
VARIABLE-LENGTH CODING
----INFORMATION THEORY RESULTS (II)
5.1 Some Fundamental Results
5.1.1Coding an Information Source
Consider an information source, representedby a source alphabet S.
},,,{21 m
sssS L= , (5.1)
where is s are source symbols.
Terms source symbol and informationmessage.
Used interchangeably in the literature.
7/29/2019 Coding Techs
2/45
2 CHAPTER 5 VARIABLE-LENGTH CODING
We like to consider: an informationmessage can be a source symbol, or acombination of source symbols.
We denote code alphabetbyA and},,,{
21 raaaA L= ,
where ja s are code symbols.
A message code is a sequence of codesymbols that represents a given informationmessage.
In the simplest case, a message consists ofonly a source symbol.Encoding is then a procedure to assign a
codewordto the source symbol. Namely,
),,,(21 ikiiii
aaaAs L= ,
where the codeword iA is a string of k codesymbols assigned to the source symbol is .
The term message ensemble is defined asthe entire set of messages.
7/29/2019 Coding Techs
3/45
5.1 Coding An information Source 3
A code, also known as an ensemble code, isdefined as a mapping of all the possiblesequences of symbols of S (message
ensemble) into the sequences of symbols inA.
In binary coding, the number of codesymbols r is equal to 2, since there are only
two code symbols available: the binary
digits 0 and 1.
Example 5.1
Consider an English article and ASCII code.
In this context, the source alphabet consists
of all the English letters in both lower andupper cases andall the punctuation marks.
The code alphabet consists of the binary 1and 0.
There are a total of 128 7-bit binary
codewords.From Table 5.1, we see that codeword
assigned to capital letter A (a source symbol)is 1000001 (the codeword).
7/29/2019 Coding Techs
4/45
7/29/2019 Coding Techs
5/45
5.1 Coding An information Source 5
Example 5.2
Table 5.2 lists what is known as the (5,2)
code.It is a linear blockcode.
In this example, the source alphabet consists
of the four )2( 2 source symbols listed in the
left column of the table: 00, 01, 10 and 11.
The code alphabet consists of the binary 1and 0.
There are four codewords listed in the rightcolumnof the table.
From the table, we see that the code assigns
a 5-bit codeword to eachsource symbol.
Table 5. 2 A (5,2) linear block code
Source Symbol Codeword
S1 ( 0 0 )
S2 ( 0 1 )S3 ( 1 0 )
S4 ( 1 1 )
0 0 0 0 0
1 0 1 0 00 1 1 1 1
1 1 0 1 1
7/29/2019 Coding Techs
6/45
6 CHAPTER 5 VARIABLE-LENGTH CODING
5.1.2Some Desired Characteristics
5.1.2.1Block Code
A code is said to be a block code if it mapseach source symbol in S into a fixedcodeword in A.
Hence, the codes listed in the above two examples are block codes.
5.1.2.2Uniquely Decodable Code
A code is uniquely decodable if it can beunambiguously decoded.
Obviously, a code has to be uniquelydecodableif it is to be in use.
Example 5.3
7/29/2019 Coding Techs
7/45
5.1 Coding An information Source 7
Table 5. 3 A not uniquely decodable code
Source Symbol Codeword
S1
S2
S3
S4
0 0
1 0
0 0
1 1
Nonsingular Code
A block code is nonsingular if all thecodewordsare distinct.
Table 5. 4 A nonsingular code
Source Symbol Codeword
S1
S2S3
S4
1
1 10 0
0 1
7/29/2019 Coding Techs
8/45
8 CHAPTER 5 VARIABLE-LENGTH CODING
Example 5.4
Table 5.4 gives a nonsingular code since all
four codewordsare distinct.If a code is not a nonsingular code, i.e., at
least two codewords are identical, then the
code is not uniquelydecodable.
Notice that, however, a nonsingular code
does not guarantee unique decodability.
The code shown in Table 5.4 is such anexample in that it is nonsingular while it is
not uniquely decodable (because once the
binary string 11 is received, we do notknow if the source symbols transmitted are
1s followed by 1s orsimply 2s .)
The nth Extension of a Block Code
The nth extension of a block code, whichmaps the source symbol is into thecodeword
i, is a block code that maps the
sequences of source symbols iniisss L
21
7/29/2019 Coding Techs
9/45
7/29/2019 Coding Techs
10/45
7/29/2019 Coding Techs
11/45
5.1 Coding An information Source 11
In decoding we can immediately tell
which source symbols are transmittedsince each codeword has the same length.
q In the second code, code symbol 1functions like a comma. Whenever we see a
1, we know it is the end of the codeword.
q The third code is different from the previoustwo codes in that if we see a 10 string we
are not sure if it corresponds to 2s until wesee a succeeding 1. Specifically, if thenext code symbol is 0, we still cannot tell
if it is 3s since the next one may be 0
(hence 4s ) or 1 (hence 3s ). In this
example, the next 1 belongs to thesucceeding codeword. Therefore we see thatcode 3 is uniquely decodable. It is not
instantaneous,however.
Definition of the jth Prefix
Assume a codeword ikiiiaaaA L
21= . Then
the sequences of code symbols ijii aaa L21
7/29/2019 Coding Techs
12/45
12 CHAPTER 5 VARIABLE-LENGTH CODING
with kj 1 is the jth order prefix of the
codeword iA .
Example 5.7
q If a codeword is 11001, it has the followingfive prefixes: 11001, 1100, 110,11, 1.
q The first order prefix is 1, while the fifth
order prefixis 11001.
A Necessary and Sufficient Condition of
Being Instantaneous Codes
v A code is instantaneous if and only if nocodeword is a prefix of some othercodeword.
v This condition is called the prefixcondition. Hence, the instantaneous codeis also called the prefix condition code or
sometimes simply the prefix code.
v In many applications, we need a blockcode that is nonsingular, uniquely
decodable, and instantaneous.
7/29/2019 Coding Techs
13/45
5.1 Coding An information Source 13
5.1.2.4Compact Code
A uniquely decodable code is said to becompact if its average length is the
minimum among all other uniquelydecodable codes based on the same
source alphabet Sand code alphabetA.
A compact code is also referred to as aminimum-redundancy code, or anoptimum code.
5.1.3Discrete Memoryless Sources
The simplest model of an information source.
In this model, the symbols generated by thesource are independent of each other. That
is, the source is memoryless or it has a zero-memory.
7/29/2019 Coding Techs
14/45
7/29/2019 Coding Techs
15/45
7/29/2019 Coding Techs
16/45
16 CHAPTER 5 VARIABLE-LENGTH CODING
)()( SHnnSH = . (5. 4)
5.1.4.3Noiseless Source Coding Theorem
v For a discrete zero-memory informationsource S, Shannons noiseless coding
theorem can be expressedas
1)()( +
7/29/2019 Coding Techs
17/45
5.1 Coding An information Source 17
v Since )()( nHnH = andavgavg nL
nL = , we have
nSHavgLSH
1)()( +
7/29/2019 Coding Techs
18/45
18 CHAPTER 5 VARIABLE-LENGTH CODING
5.2 Huffman Codes
In many cases, we need a direct encodingmethod that is optimum and instantaneous(hence uniquely decodable) for an
information source with finite source
symbols in source alphabet S.
Huffman code is the first such optimumcode [huffman 1952].
The most frequently used at present.It can be used for r-ary encoding as r>2. For
the notational brevity, however, we discuss
only the Huffman coding used in the binary
casepresented here.
5.2.1Required Rules for Optimum
Instantaneous Codes
Consider an information source:),,,( 21 msss
L= . (5. 9)
WLOG, assume the occurrence probabilitiesof the source symbols are as follows:
7/29/2019 Coding Techs
19/45
Index 19
)()()()(121 mm
spspspsp
L
Since we are seeking the optimum code forS, the lengths of codewords assigned to the
source symbolsshould be
mmllll
121L . (5. 10)
Based on the requirements of the optimumand instantaneous code, Huffman derived
the following rules(restrictions):
1. mm llll = 121 L . (5. 11)2.The codewords of the two least
probable source symbols should be thesame except for theirlast bits.
3.Each possible sequence of length1ml bits must be used either as a
codeword or must have one of its
prefixes used as a codeword.
7/29/2019 Coding Techs
20/45
7/29/2019 Coding Techs
21/45
Index 21
q The same procedure can be applied to thisnewly createdsource alphabet.
qThe second auxiliary source alphabet willagain have one source symbol less than thefirst auxiliary source alphabet.
q The procedure continues. In some step, theresultant source alphabet will have only twosource symbols. At this time, we combine
them to form a single source symbol with aprobability of 1. The coding is thencomplete.
Example 5.9
7/29/2019 Coding Techs
22/45
7/29/2019 Coding Techs
23/45
Index 23
5.2.2.1Applications
q Recall that it has been used in differentialcoding and transformcoding.
q In TC, the magnitude of the quantizednonzero transform coefficients and the run-
length of zeros in the zigzag scan are
encoded by using the Huffmancode.
5.2.3 Modified Huffman Codes
when the occurrence probabilities are skewed,
and the number of less probable source
symbols is large, the required codebookmemory will be large.
The modified Huffman code can reduce the
memory requirement while keeping almost the
same optimality.
7/29/2019 Coding Techs
24/45
24 CHAPTER 5 VARIABLE-LENGTH CODING
5.3 Arithmetic Codes
Arithmetic coding:
quite different from Huffmancoding
gaining increasingpopularity
5.3.1Limitations of Huffman Coding
Huffman coding is optimum for block-encoding a source alphabet, with eachsource symbol having an occurrence probability.
The average codeword length achieved byHuffman coding satisfies the followinginequality [gallagher 1978].
086.0)()( max ++
7/29/2019 Coding Techs
25/45
Index 25
In the case where the probability distributionamong source symbols is skewed (someprobabilities are small, while some are quitelarge), the upper bound may be large,
implying that the coding redundancy may
not be small.
An extreme situation. There are only twosource symbols. One has a very smallprobability, while the other has a very large
probability (very close to1).
Entropy of source alphabet is close to0 since the uncertainty is verysmall.
Using Huffman coding, however, weneed two bits: onefor each.
average codeword length is1
redundancy 1
This agrees with Equation5.22.
This inefficiency is due to the
fact that Huffman coding alwaysencodes a source symbol with an
integer numberof bits.
7/29/2019 Coding Techs
26/45
7/29/2019 Coding Techs
27/45
7/29/2019 Coding Techs
28/45
7/29/2019 Coding Techs
29/45
7/29/2019 Coding Techs
30/45
30 CHAPTER 5 VARIABLE-LENGTH CODING
Figure 5. 2 Arithmetic coding working on the samesource alphabet as that in Example 5.9. The encodedsymbol string is S1 S2 S3 S4 S5 S6.
0.3 0.4 0.6 0.65 0.75 1.00
[ 0, 0.3) [ 0.3, 0.4) [ 0.4, 0.6)
[ 0.6, 0.65) [ 0.65, 0.75)
[ 0.75, 1.0)S1
0.3
0.09 0.12 0.18 0.195 0.225
0
S2
[ 0.09, 0.12)
0.12
0.099 0.102 0.108 0.1095 0.1125
0.09
S3
[ 0.102, 0.108)
0.108
0.1083 0.1044 0.1056 0.1059 0.1065
0.102 S4
[ 0.1056, 0.1059)
0.1059
0.10569 0.10572 0.10578 0.105795 0.105825
0.1056 S5
[ 0.105795, 0.105825)
0.1058250
0.105804 0.105807 0.105813 0.1058145 0.1058175
0.105795 S6
[ 0.1058175, 0.1058250)
(a)
(b)
(c)
(d)
(e)
(f)
7/29/2019 Coding Techs
31/45
Index 31
5.3.2.2Encoding
Encoding the First Source Symbol
Refer to Part (a) of Figure 5.3. Since the first
symbol is 1s , we pick up its subinterval [0,
0.3). Picking up the subinterval [0, 0.3) meansthat any real number in the subinterval, i.e.,any real number equal to or greater than 0 and
smaller than 0.3, can be a pointer to thesubinterval, thus representing the source
symbol 1s . This can be justified by considering
that all the six subintervals are disjoined.
Encoding the Second Source Symbol
Refer to Part (b) of Figure 5.3. We use the
same procedure as used in Part (a) to divide
the interval [0, 0.3) into six subintervals.
Since the second symbol to be encoded is 2s ,we pick up its subinterval [0.09, 0.12).
Notice that the subintervals are recursively
generated from Part (a) to Part (b). It isknown that an interval may be completelyspecified by its lower end point and width.
Hence, the subinterval recursion in the
7/29/2019 Coding Techs
32/45
32 CHAPTER 5 VARIABLE-LENGTH CODING
arithmetic coding procedure is equivalent to
the following two recursions: end pointrecursion and width recursion.
The lower end pointrecursion:
newCPcurrentWcurrentLnewL +=
: the lower end points
W: the width
new: thenewrecursion
current: thecurrent recursion
The width recursionis
)(i
scurrent
WnewW = , (5. 14)
Encoding the Third Source Symbol
Encoding the Fourth, Fifth and Sixth Source Symbols
The resulting subinterval [0.1058175,0.1058250) can represent the source symbol
string654321
ssssss .
7/29/2019 Coding Techs
33/45
7/29/2019 Coding Techs
34/45
34 CHAPTER 5 VARIABLE-LENGTH CODING
That is, the lower end is contained in the
subinterval corresponding tothe symbol 2s .
As a result, 2s is the second decodedsymbol.
The procedure repeats itself until all six
symbols are decoded.
Note that a terminal symbol is necessary toinform the decoder to stop decoding.
q The above procedure gives us an idea ofhowdecoding works.
q The decoding process, however, does notneed to construct Parts (b), (c), (d), (e) and
(f) of Figure 5.3.
q Instead, the decoder only needs theinformation contained in Part (a) of Figure 5.3.
q Decoding can be split into the followingthree steps: comparison, readjustment(subtraction), and scaling [langdon 1984].
7/29/2019 Coding Techs
35/45
Index 35
q In summary, considering the way in whichParts (b), (c), (d), (e) and (f) of Figure 5.3are constructed, we see that the three steps
discussed in the decoding process:comparison, readjustment and scalingexactly undo what the encoding procedure
has done.
q Back to Example:
oAfter symbol 1s is firstdecoded,othe lower end of the corresponding
interval, 0, is subtracted from 0.1058175,
resulting in 0.1058175.
oThis value is then divided by p1=0.3,resulting in 0.352725. Compare this valuewith Part (a), we decode 2
s since0.3
7/29/2019 Coding Techs
36/45
7/29/2019 Coding Techs
37/45
Index 37
when the length of the source symbol string
becomes larger and larger. This causes whatis known as the precisionproblem.
It is this problem that prohibited arithmeticcoding from practical usage for quite a long
period oftime.
Only after this problem was solved in thelate 1970s, did arithmetic coding become an
increasingly important codingtechnique.
It is necessary to have a termination symbolat the end of an input source symbol string.In this way, an arithmetic coding system is
able to know when to terminatedecoding.
To encode the same source symbol string,Huffman coding can be implemented in two
different ways.
One way is shown in Example 5.9. We
construct a fixed codeword for each sourcesymbol. Since Huffman coding is
instantaneous, we can cascade the
corresponding codewords to form the
output, a 17-bit code string00.101.11.1001.1000.01, where the five
7/29/2019 Coding Techs
38/45
38 CHAPTER 5 VARIABLE-LENGTH CODING
periods are used to indicate different
codewords for easy reading.
As we see that for the same source symbolstring, the final subinterval obtained byusing arithmetic coding is [0.1058175,
0.1058250). It is noted that the 15-bit binary
decimal, 0.000110111111111, is equal to the
decimal 0.1058211962, which falls into the
final subinterval representing the string654321
ssssss .
This indicates that the arithmetic coding ismore efficient than the Huffamn coding in
thisexample.
Another way is to form a 6th extension of thesource alphabet as discussed in Section5.1.4: treat each group of six source symbols
as a new source symbol; calculate its
occurrence probability by multiplying therelated six probabilities; then apply the
Huffman coding algorithm to the 6th
extension of the discrete memoryless source.
This is called the 6th extension of Huffman
block code. In other words, in order to
7/29/2019 Coding Techs
39/45
Index 39
encode the source string 654321ssssss , (the
6th extension of) Huffman coding encodes
all of the 4665666 = codewords in the 6th
extension of the source alphabet. This
implies a high complexity in implementationand a large codebook, hence notefficient.
Similar to the case of Huffman coding,arithmetic coding is also applicable to r-ary
encodingwith r>2.
5.3.3Implementation Issues
q The growing precisionproblem.
q This problem has been resolved and thefinite precision arithmetic is now used in
arithmeticcoding.
q This advance is due to the incrementalimplementation of arithmeticcoding.
5.3.3.1 Incremental Implementation
7/29/2019 Coding Techs
40/45
7/29/2019 Coding Techs
41/45
7/29/2019 Coding Techs
42/45
42 CHAPTER 5 VARIABLE-LENGTH CODING
fraction was introduced in Shannons
celebrated paper [shannon1948].
The recursive implementation of arithmeticcoding was devised by Elias (anothermember in Fanos first information theory
class at MIT).
This unpublished result was first introducedby Abramson as a note in his book on
information theory and coding [abramson 1963].
The result was further developed by Jelinekin his book on information theory [jelinek
1968].
The growing precision problem preventedarithmetic coding from practical usage,
however. The proposal of using finiteprecision arithmetic was made
independently by Pasco [pasco 1976] and
Rissanen [rissanen1976].
Practical arithmetic coding was developedby several independent groups [rissanen
1979, rubin 1979,guazzo 1980].
7/29/2019 Coding Techs
43/45
Index 43
A well-known tutorial paper on arithmeticcoding appeared in[langdon 1984].
The tremendous efforts made in IBM lead toa new form of adaptive binary arithmeticcoding known as the Q-coder [pennebaker1988].
Based on the Q-coder, the activities ofJPEG and JBIG combined the best features
of the various existing arithmetic coders anddeveloped the binary arithmetic coding
procedure known as the QM-coder[pennebaker1992].
5.3.5Applications
Arithmetic coding is becomingpopular.
Note that in text and bilevel image
applications there are only two source
symbols (black and while), and theoccurrence probability is skewed. Thereforebinary arithmetic coding achieves high
codingefficiency.
7/29/2019 Coding Techs
44/45
44 CHAPTER 5 VARIABLE-LENGTH CODING
It has been successfully applied to bilevel
image coding [langdon 1981] and adoptedby the international standards for bilevel
image compression JBIG.
It has also been adopted by the JPEG.
5.4 References
[abramson 1963] N. Abramson, Information Theory and Coding, New York: McGraw-
Hill, 1963.
[bell 1990] T. C. Bell, J. G. Cleary and I. H. Witten, Text Compression, Englewood, NJ:Prentice Hall, 1990.
[blahut 1986] R. E. Blahut,Principles and Practice of Information Theory, Reading MA:Addison-Wesley, 1986.
[fano 1949] R. M. Fano, The transmission of information, Technical Report 65,Research Laboratory of Electronics, MIT, Cambridge, MA, 1949.
[gallagher 1978] R. G. Gallagher, Variations on a theme by Huffman, IEEE
Transactions on Information Theory, vol. IT-24, no. 6, pp. 668-674, November1978.
[guazzo 1980] M. Guazzo, A general minimum-redundacy source-coding algorithm,IEEE Transactions on Information Theory, vol. IT-26, no. 1, pp. 15-25, January 1980.
[hankamer 1979] M. Hankamer, A modified Huffman procedure with reducedmemory requirement, IEEE Transactions on Communications, vol. COM-27, no.6, pp. 930-932, June 1979.
[huffman 1952] D. A. Huffman, A method for the construction of minimum-
redundancy codes, Proceedings of The IRE, vol. 40, pp. 1098-1101, September1952.
7/29/2019 Coding Techs
45/45
Index 45
[jelinek 1968] F. Jelinek, Probabilistic Information Theory, New York: McGraw-Hill, 1968.
[langdon 1981] G. G. Langdon, Jr. and J. Rissanen, Compression of black-white
images with arithmetic coding, IEEE Transactions on Communications, vol.COM-29, no. 6, pp. 858-867, June 1981.
[langdon 1982] G. G. Langdon, Jr. and J. Rissanen, A simple general binary
source code,IEEE Transactions on Information Theory, IT-28, 800 (1982).
[langdon 1984] G. G. Langdon, Jr., An introduction to arithmetic coding, IBMJournal of Research and Development, vol. 28, no. 2, pp. 135-149, March 1984.
[nelson 1996] M. Nelson and J. Gailly, The Data Compression Book, second edition,New York: M&T Books, 1996.
[pasco 1976] R. Pasco, Source Coding Algorithms for Fast Data Compression, Ph.D.
dissertation, Stanford University, 1976.
[pennebaker 1988] W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr. and R. B. Arps,
An overview of the basic principles of the Q-coder adaptive binary arithmetic Coder,IBM Journal of Research and Development, vol. 32, no. 6, pp. 717-726, November 1988.
[pennebaker 1992] W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image DataCompression Standard, New York: Van Nostrand Reinhold, 1992.
[rissanen 1976] J. J. Rissanen, Generalized Kraft inequality and arithmetic coding,IBMJournal of Research and Development, vol. 20, pp. 198-203, May 1976.
[rissanen 1979] J.J. Rissanen and G. G. Landon, Arithmetic coding, IBM Journal ofResearch and Development, vol. 23, no. 2, pp. 149-162, March 1979.
[rubin 1979] F. Rubin, Arithmetic stream coding using fixed precision registers,IEEETransactions on Information Theory, vol. IT-25, no. 6, pp. 672-675, November 1979.
[sayood 1996] K. Sayood, Introduction to Data Compression, San Francisco, CA:Morgan Kaufmann Publishers, 1996.
[shannon 1948] C. E. Shannon, A mathematical theory of communication, BellSystem Technical Journal, vol. 27, pp. 379-423 (Part I), July 1948, pp. 623-656
(Part II), October 1948. decodability
Top Related