1 Chapter 3: Compression (Part 1). 2 Storage size for media types (uncompressed)
-
Upload
hector-mcgee -
Category
Documents
-
view
217 -
download
1
Transcript of 1 Chapter 3: Compression (Part 1). 2 Storage size for media types (uncompressed)
1
Chapter 3: Compression(Part 1)
2
Text Image Audio Animation Video
Object Type
-ASCII -EBCDIC -BIG5 -GB
-Bitmapped graphics -Still photos -Fax
-stream of digitized audio or voice
-Synched image and music stream at 15-19 frames/s
-Digital video at 25-30 frames/s
Size and Bandwidth
2KB per page
-Simple (640x480): 1MB per image
Voice/Phone 8kHz/8 bits (mono) 8 KB/s Audio CD 44.1 kHz/16 bits (stereo) 176 KB/s
6.5 MB/s for 320x640x16 pixels/frame (16 bit color) 16 frames/s
29.7 MB/s for 720x480 pixels per frame 24-bit color 30 frame/s
Storage size for media types(uncompressed)
3
A typical encyclopedia
50,000 pages of text (2KB/page) 0.1 GB 3,000 color pictures
(on average 64048024 bits = 1MB/picture) 3 GB 500 maps/drawings/illustrations
(on average 64048016 bits = 0.6 MB/map) 0.3 GB 60 minutes of stereo sound (176 KB/s) 0.6 GB 30 animations, on average 2 minutes in duration
(64032016 pixels/frame = 6.5 MB/s) 23.4 GB 50 digitized movies, on average 1 min. in duration
87 GB
TOTAL: 114 GB(or 180 CD’s, or about 2 feet high of CD’s)
4
Compressing an encyclopedia
Text 1:2
Color Images 1:15
Maps 1:10
Stereo Sound 1:6
Animation 1:50
Motion Video 1:50
0.01
0.1
1
10
100
Tex
t
Co
lor
Imag
es
Map
s
Ste
reo
So
un
d
An
imat
ion
Vid
eo
Uncompressed
Compressed
114 GB -> 2.5 GB
GB
5
Compression and encoding
We can view compression as an encoding method which transforms a sequence of bits/bytes into an artificial format such that the output (in general) takes less space than the input.
CompressorEncoder
CompressorEncoderFat dataFat data
Fat(redundancy)
Slim data
6
Redundancy
Compression is made possible by exploiting redundancy in source data
For example, redundancy found in: Character distribution Character repetition High-usage pattern Positional redundancy
(these categories overlap to some extent)
7
Symbols and Encoding
We assume a piece of source data that is subject to compression (such as text, image, etc.) is represented by a sequence of symbols. Each symbol is encoded in a computer by a code (or codeword, value), which is a bit string.
Example: English text: abc (symbols), ASCII (coding) Chinese text: 多媒體 (symbols), BIG5 (coding) Image: color (symbols), RGB (coding)
8
Character distribution
Some symbols are used more frequently than others.
In English text, ‘e’ and space occur most often. Fixed-length encoding: use the same number of bits
to represent each symbol. With fixed-length encoding, to represent n symbols,
we need log2n bits for each code.
9
Character distribution
Fixed-length encoding may not be the most space-efficient method to digitally represent an object.
Example: “abacabacabacabacabacabacabac” With fixed-length encoding, we need 2 bits for each of
the 3 symbols. Alternatively, we could use ‘1’ to represent ‘a’ and ‘00’
and ‘01’ to represent ‘b’ and ‘c’, respectively. How much saving did we achieve?
10
Character distribution
Technique: whenever symbols occur with different frequencies, we use variable-length encoding.
For your interest: In 1838, Samuel Morse and Alfred Vail designed the
“Morse Code” for telegraph. Each letter of the alphabet is represented by dots (electric
current of short duration) and dashes (3 dots). e.g., E “.” Z “--..” Morse & Vail determined the code by counting the
letters in each bin of a printer’s type box.
11
Morse code table
SOS = — — —
12
Character distribution
Principle: use fewer bits to represent frequently occurring symbols, use more bits to represent those rarely occurring ones.
13
Character repetition
when a string of repetitions of a single symbol occurs, e.g., “aaaaaaaaaaaaaabc”
Example: in graphics, it is common to have a line of pixels using the same color.
Technique: run-length encoding Instead of specifying the exact value of each repeating
symbol, we specify the value once and specify the number of times the value repeats.
14
High usage patterns
Certain patterns (i.e., sequences of symbols) occur very often. Example: “qu”, “th”, ……, in English text. Example: “begin”, “end”, …, in Pascal source
code. Technique: Use a single special symbol to
represent a common sequence of symbols.e.g., = ‘th’ i.e., instead of using 2 codes for ‘t’ and ‘h’, use only 1
code for ‘th’
15
Positional redundancy
Sometimes, given the context, symbols can be predicted.
Technique: differential encoding Predict a value and code only the prediction error. example: differential pulse code modulation (DPCM)
198, 199, 200, 200, 201, 200, 200, 198
198, +1, +1, 0, +1, -1, 0, -2
Small numbers require fewer bits to represent
16
Categories of compression techniques
Entropy encoding regard data stream as a simple digital sequence, its
semantics is ignored lossless compression
Source encoding take into account the semantics of the data and the
characteristics of the specific medium to achieve a higher compression ratio
lossy compression
17
Categories of compression techniques
Hybrid encoding A combination of the above two.
18
A categorization
Entropy Encoding
Huffman Code, Shannon-Fano Code
Run-Length Encoding
LZW
Source Coding
Prediction DPCM
DM
Transformation DCT
Vector Quantization
Hybrid Coding
JPEG
MPEG
H.261
19
Information theory
Entropy is a quantitative measure of uncertainty or randomness. Entropy order
According to Shannon, the entropy of an information source S is defined as:
where pi is the probability that symbol Si in S occurs. For notational convenience, we use a set of probabilities to
denote S. E.g., S = {0.2, 0.3, 0.5} for a set S of 3 symbols.
H S ppi
ii
( ) log 2
1
20
Entropy
For any set S of n symbols 0 = H({1, 0, …, 0}) H(S) H(1/n, …, 1/n) =
log2n. That is, the entropy of S is bounded below and
above by 0 and log2n, respectively. The lower bound is obtained if one symbol
occurs with a probability of 1, while those of others are 0; The upper bound is obtained if all symbols occur with equal probability.
21
Self-information
I(Si) = log2(1/pi) is called the self-information associated with the symbol Si.
It indicates the amount of information (surprise or unexpectedness) contained in Si or the number of bits needed to encode it.
Note that Entropy can be interpreted as the average self-information of a symbol of a source.
22
Entropy and Compression
In relation to compression and coding, one can consider an object (such as an image, a piece of text, etc.) as an information source, and the elements in the object (such as pixels, alpha-numeric characters, etc.) as the symbols.
You may also think of the occurrence frequency of an element (e.g., a pixel) as the element’s occurrence probability.
23
Example
Abracadabra!
Symbol Freq. Prob.
A 1 1/12
b 2 2/12
r 2 2/12
a 4 4/12
c 1 1/12
d 1 1/12
! 1 1/12
A piece of text
Correspondingstatistics
24
Example
Symbol Freq. Prob.
27 0.21
44 0.34
28 0.22
25 0.20
4 0.03
An image
Correspondingstatistics
25
Information theory
The entropy is the lower bound for lossless compression: if the probability of each symbol is fixed, then on average, a symbol requires H(S) bits to encode.
A guideline: the code lengths for different symbols should vary according to the information carried by the symbol. Coding techniques based on that are called entropy encoding.
26
Information theory
For example, consider an image in which half of the pixels are white, another half are black p(white) = 0.5, p(black) = 0.5,
Information theory says that each pixel requires 1 bit to encode on average.
15.0
1log5.0)(
2
12
i
SH
27
Information theory
Q: How about an image such that white occurs ¼ of the time, black ¼ of the time, and gray ½ of the time? Ans: Entropy = 1.5. Information theory says that it takes at least 1.5 bits on
average to encode a pixel How: white (00), black (01), gray (1)
Entropy encoding is about wisely using just enough number of bits to represent the occurrence of certain patterns.
28
Variable-length encoding
A problem with variable-length encoding is ambiguity in the decoding process. e.g., 4 symbols: a (‘0’), b (‘1’), c (‘01’), d (‘10’) the bit stream: ‘0110’ can be interpreted as
abba, orabd, orcba, orcd
29
Prefix code
We wish to encode each symbol into a sequence of 0’s and 1’s so that no code for a symbol is the prefix of the code of any other symbol – the prefix property.
With the prefix property, we are able to decode a sequence of 0’s and 1’s into a sequence of symbols unambiguously.
30
Average code length
Given: a symbol set S = {S1, …, Sn}
with probability of Si = pi
a coding scheme C such that the codeword of Si is ci
and the length (in bits) of ci being li
The average code length (ACL) is given by
n
iiilpACL
1
31
Average code length
Example: S = {a, b, c, d, e, f, g, h} with probabilities {0.1, 0.1, 0.3, 0.35, 0.05, 0.03, 0.02, 0.05} Coding scheme:
a b c d e f g h
100 101 00 01 110 11100 11101 1111
5.2
405.0502.0503.0305.0235.023.031.031.0
ACL
32
Optimal code
An optimal code is one that has the smallest ACL for a given probability distribution of the symbols.
33
Theorem
For any prefix binary coding scheme C for a symbol set S, we have
Entropy is thus a lower bound for lossless compression.
)(SHlpACL ii
34
Shannon-Fano algorithm
Example: 5 characters are used in a text, statistics:
Algorithm (top-down approach):1. sort symbols according to their probabilities.2. divide symbols into 2 half-sets, each with roughly equal
counts.3. all symbols in the first group are given ‘0’ as the first bit of
their codes; ‘1’ for the second group.4. recursively perform steps 2-3 for each group with the
additional bits appended.
Symbol A B C D ECount 15 7 6 6 5
A
B
35
Shannon-Fano algorithm
Example: G1: {A,B}; G2: {C,D,E} subdivide G1:
G11:{A}; G12{B} subdivide G2:
G21:{C}; G22:{D,E} subdivide G22:
G221:{D}; G222:{E}
Symbol A B C D E Code 0.. 0.. 1.. 1.. 1..
Symbol A B C D E
Code 00 01 1.. 1.. 1..
Symbol A B C D E Code 00 01 10 11. 11.
Symbol A B C D E
Code 00 01 10 110 111
36
Shannon-Fano algorithm
Result:
Symbol Count log2(1/p) Code Bits used
A 15 1.38 00 30 B 7 2.48 01 14 C 6 2.70 10 12 D 6 2.70 110 18 E 5 2.96 111 15
39 89
ACL = 89/39 = 2.28 bits per symbolH(S) = 2.19
37
Shannon-Fano algorithm
The coding scheme in a tree representation:
Total # of bits used for encoding is 89. A fixed-length coding scheme will need 3 bits for each
symbol a total of 117 bits.
15 7 6
6 5
11
39
22 17
A B C
D E
0 1
0 1 0 1
10
38
Shannon-Fano algorithm
It can be shown that the ACL of a code generated by the Shannon-Fano algorithm is between H(S) and H(S) + 1. Hence it is fairly close to the theoretical optimal value.
39
Huffman code
used in many compression programs, e.g., zip Algorithm (bottom-up approach):
Given n symbols We start with a forest of n trees. Each tree has only 1 node
corresponding to a symbol. Let’s define the weight of a tree = sum of the probabilities of
all the symbols in the tree. Repeat the following steps until the forest has one tree left:
pick 2 trees of the lightest weights, let’s say, T1, T2. replace T1 and T2 by a new tree T, such that T1 and T2 are T’s
children.
40
Huffman code
a(15/39) b(7/39) e(5/39)c(6/39) d(6/39)
a(15/39) b(7/39) c(6/39)e(5/39)d(6/39)
11/39
a(15/39) e(5/39)d(6/39)
11/39
c(6/39)b(7/39)
13/39
41
Huffman code
a(15/39) e(5/39)d(6/39)
11/39
c(6/39)b(7/39)
13/39
24/39
42
Huffman code
result from the previous example:
43
Huffman code
The code of a symbol is obtained by traversing the tree from the root to the symbol, taking a ‘0’ when a left branch is followed; and a ‘1’ when a right branch is followed.
44
Huffman code
symbol code bits/symbol # occ. #bits
used
A 0 1 15 15
B 100 3 7 21
C 101 3 6 18
D 110 3 6 18
E 111 3 5 15
total: 87 bits
“ABAAAE” “0100000111”
45
Huffman code
symbol code bits/symbol # occ. #bits
used
A 0 1 15 15
B 100 3 7 21
C 101 3 6 18
D 110 3 6 18
E 111 3 5 15
total: 87 bits
“ABAAAE” “0100000111”
Codingtable
46
Huffman code (advantages)
Decoding is trivial as long as the coding table is stored before the (coded) data. Overhead (i.e., the coding table) is relatively negligible if the data file is big.
Prefix code.
47
Huffman code (advantages)
“Optimal code size” (among any coding scheme that uses a fixed code for each symbol.) Entropy is
Average bits/symbol needed for Huffman coding is
( * . * . * . * . * . ) /
. /
.
15 138 7 2 48 6 2 7 6 2 7 5 2 96 39
85 26 39
2 19
87 39
2 23
/
.
48
Huffman code (disadvantages)
Need to know the symbol frequency distribution. Might be ok for English text, but not data files Might need two passes over the data.
Frequency distribution may change over the content of the source.
49
Run-length encoding (RLE)
Image, video and audio data streams often contain sequences of the same bytes or symbols, e.g., background in images, and silence in sound files.
Substantial compression can be achieved by replacing repeated symbol sequences with the number of occurrences of the symbol.
50
Run-length encoding
For example, if a symbol occurs at least 4 consecutive times, the sequence can be encoded with the symbol followed by a special flag (i.e., a special symbol) and the number of its occurrences, e.g., Uncompressed: ABCCCCCCCCDEFGGGHI Compressed: ABC!8DEFGGGHI
If 4Cs (CCCC) are encoded as C!0, and 5Cs as C!1, and so on, then it allows compression of 4 to 259 symbols into 2 symbols + a count.
51
Run-length encoding
Problems: Ineffective with text
except, perhaps, “Ahhhhhhhhhhhhhhhhh!”, “Noooooooooo!”. Need an escape byte (‘!’)
ok with text not ok with data solution?
used in ITU-T Group 3 fax compression a combination of RLE and Huffman compression ratio: 10:1 or better
54
Lempel-Ziv-Welch algorithm
Method due to Ziv, Lempel in 1977, 1978. Improvement by Welch in 1984. Hence, named LZW compression.
used in UNIX compress, GIF, and V.42bis compression modems.
55
Lempel-Ziv-Welch algorithm
Motivation: Suppose we want to encode a piece of English text. We
first encode the Webster’s English dictionary which contains about 159,000 entries. Here, we surely can use 18 bits for a word. Then why don’t we just represent each word in the text as an 18 bit number, and so obtain a compressed version of the text?
56
LZW
A: 34Simple: 278Example: 1022
Dictionary
A_simple_example
34 278 1022
A: 34Simple: 278Example: 1022
Dictionary
Using ASCII: 16 symbols, 7 bits each 112 bits
Using the dictionary approach: 3 symbols, 18 bits each 54 bits
57
LZW
Problems: Need to preload a copy of the dictionary before encoding
and decoding. Dictionary is huge, 159,000 entries. Most of the dictionary entries are not used. Many words cannot be found in the dictionary.
Solution: Build the dictionary on-line while encoding and decoding. Build the dictionary adaptively. Words not used in the text
are not incorporated in the dictionary.
58
LZW
Example: encode “^WED^WE^WEE^WEB^WET” using LZW.
Assume each symbol (character) is encoded by an 8-bit code (i.e., the codes for the characters run from 0 to 255). We assume that the dictionary is loaded with these 256 codes initially (the base dictionary).
While scanning the string, sub-string patterns are encoded into the dictionary. Patterns that appear after the first time would then be substituted by the index into the dictionary.
59
Example 1 (compression)
^WED^WE^WEE^WEB^WET Dict Index Output
^ ^W ^W 256 16 2c WE WE 257 23 2c ED ED 258 40 2c D^ D^ 259 77 2c ^W seen at 256 ^WE ^WE 260 256 3c from 256 E^ E^ 261 40 2c ^W seen at 256 ^WE seen at 260 ^WEE ^WEE 262 260 4c from 260 E^ seen at 261 E^W E^W 263 261 3c from 261 WE seen at 257 WEB WEB 264 257 3c from 257 B^ B^ 265 254 2c ^W seen at 256 ^WE seen at 260 ^WET ^WET 266 260 4c from 260 T 95
buffer
^ 16
W 23
D 77
E 40
B 254
T 95
Base dictionary(partial)
60
LZW (encoding rules)
Rule 1: whenever the pattern in the buffer is in the dictionary, we try to extend it.
Rule 2: whenever a new pattern is formed in the buffer, we add the pattern to the dictionary.
Rule 3: whenever a pattern is added to the dictionary, we output the codeword of the pattern before the last extension.
61
Example 1
Original: 19 symbols, 8-bit code each 19 8 = 152 bits
LZW’ed: 12 codes, 9 bits each 12 9 = 108 bits Some compression achieved. Compression
effectiveness is more substantial for long messages. The dictionary is not explicitly stored and sent to
the decoder. In a sense, the compressor is encoding the dictionary in the output stream implicitly.
62
LZW decompression
Input: “{16}{23}{40}{77}{256}{40}{260}{261}{257}{254}{260}
{95}”. Assume the decoder also knows about the first 256
entries of the dictionary (the base dictionary). While scanning the code stream for decoding, the
dictionary is rebuilt. When an index is read, the corresponding dictionary entry is retrieved to splice in position.
The decoder deduces what the buffer of the encoder has gone through and rebuild the dictionary.
^ 16
W 23
D 77
E 40
B 254
T 95
63
Example 1(decompression)
16.23.40.77.256.40.260.261.257.254.260.95 Dict Index Output
16 ^ 16.23 ^W 256 W 23.40 WE 257 E 40.77 ED 258 D 77.256 D^ 259 ^W 256.40 ^WE 260 E 40.260 E^ 261 ^WE 260.261 ^WEE 262 E^ 261.257 E^W 263 WE 257.254 WEB 264 B 254.260 B^ 265 ^WE 260.95 ^WET 266 T 95
^ 16
W 23
D 77
E 40
B 254
T 95
64
LZW (decoding rule)
Whenever there are 2 codes C1C2 in the buffer, add a new entry to the dictionary: symbols of C1 + first symbol of C2
w = NIL; while ( read a symbol k ) { if ( wk exists in dictionary ) w = wk; // proceed to next else { output code for w; add wk to dictionary; w = k; } }
Compression code
Decompression coderead a code k; output symbol represented by k; w = k; while ( read a code k ) { ent = dictionary entry for k; output symbol(s) in entry ent; add w + ent[0] to dictionary w = ent; }
66
Example 2
ababcbababaaaaaaa
67
Example 2 (compression)
ababcbababaaaaaaa$ Dict Index Output
a 1 b 2 c 3 a ab ab 4 1 ba ba 5 2 abc abc 6 4 cb cb 7 3 bab bab 8 5 baba baba 9 8 aa aa 10 1 aaa aaa 11 10 aaaa aaaa 12 11 a$ 1
68
LZW: Pros and Cons
LZW (advantages) effective at exploiting:
character repetition redundancyhigh-usage pattern redundancy
an adaptive algorithm (no need to learn about the statistical properties of the symbols beforehand)
one-pass procedure
69
LZW: Pros and Cons
LZW (disadvantage) Messages must be long for effective compression
It takes time for the dictionary to build up.
70
LZW (Implementation)
decompression is generally faster than compression. Why?
usually uses 12-bit codes 4096 table entries.
need a hash table for compression.
71
LZW and GIF
11 colorscreate a color map of11 entries:
color code[r0,g0,b0] 0[r1,g1,b1] 1[r2,g2,b2] 2… …[r10,g10,b10] 10
convert the image intoa sequence of pixel colors,apply LZW to thecode stream.
320 x 200
GIF, 5280 bytes(0.66 bpp)
JPEG, 16327 bytes(2.04 bpp)
bit per pixel
73
GIF image
74
JPEG image