Image/Video Compression September 28, 1999 Lawrence A. Rowe University of California, Berkeley URL:...

Image/Video CompressionSeptember 28, 1999

Lawrence A. RoweUniversity of California, Berkeley

URL: http://www.BMRC.Berkeley.EDU/~larry

Copyright @1999, L.A. Rowe

Multimedia Systems and Applications 2

Outline

• Background• Block Transform Coding• Other Coding Algorithms• Software/Hardware CODEC’s• Pragmatic Issues


Video Data Size

1920x1080 1280x720 640x480 320x240 160x1201 sec 0.19 0.08 0.03 0.01 0.001 min 11.20 4.98 1.66 0.41 0.101 hour 671.85 298.60 99.53 24.88 6.22

1000 hours 671,846.40 298,598.40 99,532.80 24,883.20 6,220.80

size of uncompressed video in gigabytes

image size of video

1280x720 (1.77) 640x480 (1.33) 320x240 160x120


Video Bit Rate Calculation

width pixels (160, 320, 640, 720, 1280, 1920, …)

height pixels (120, 240, 480, 485, 720, 1080, …)

depth bits (1, 4, 8, 15, 16, 24, …)

fps frames per second (5, 15, 20, 24, 30, …)

compression factor (1, 6, 24, …)

width * height * depth * fps

compression factor= bits/sec


Effects of Compression

storage for 1 hour of compressed video in megabytes

1920x1080 1280x720 640x480 320x240 160x1201:1 671,846 298,598 99,533 24,883 6,2213:1 223,949 99,533 33,178 8,294 2,0746:1 111,974 49,766 16,589 4,147 1,037

25:1 26,874 11,944 3,981 995 249100:1 6,718 2,986 995 249 62

3 bytes/pixel, 30 frames/sec


Be Careful...

mpeg 200:1, jpeg 24:1

analogsource

digitalrepresentation

compressedrepresentation

vs


Another View

Data Rate Size/Hour128 Kbs 60 MB384 Kbs 170 MB1.5 Mbs 680 MB3.0 Mbs 1.4 GB6.0 Mbs 2.7 GB25 Mbs 11.0 GB


Perceptual Coding

• Encode source signal using lossy compressionLossless algorithms typically reduce signal by 3:1Must use lossy algorithm to get adequate compression

• Hide errors where humans will not see or hear itStudy hearing and vision system to understand how we see/hearMasking refers to one signal overwhelming/hiding another (e.g., loud siren or bright flash)Audio perception is 20-20 kHz but most sounds in low frequencies (e.g., 2 kHz to 4 kHz)Visual perception influenced by edges and low frequencies


What is…

• JPEG - Joint Photographic Experts GroupStill image compression, intraframe picture technologyMJPEG is sequence of images coded with JPEG

• MPEG - Moving Picture Experts GroupMany standards MPEG1, MPEG2, and MPEG4Very sophisticated technology involving intra- and interframe picture coding and many other optimizations => high quality and cost in time/computation

• H.261/H.263/H.263+ - Video ConferencingLow to medium bit rate, quality, and computational cost Used in H.320 and H.323 video conferencing standards


Coding Overview

• DigitizeSubsample to reduce data

• Intraframe compressionRemove redundancy within frame (spatial compression)

• Interframe compressionRemove redundancy between frames (temporal compression)

• Symbol codingEfficient coding of sequence of symbols


Digitizing

• Modify color space24 bit RGB => 15 or 16 bit RGB24 bit RGB => YUV (8 bit Y, 4 bit U, 4 bit V)24 bit RGB => 8 bit color map

• Encode only 1 field• Reduce frame rate

Film is 24 fps so why encode 30 frames of videoIs 20 fps good enough? 18? 15? 12? 8? 4? ...Variable frame rate?


Block Transform EncodingDCT

Zig-zag Quantize

Run-length Code

Huffman Code

011010001011101...


Block Encoding

139 144 149 153144 151 153 156150 155 160 163159 161 162 160

original image

DCT1260 -1 -12 -5-23 -17 -6 -3-11 -9 -2 2-7 -2 0 1

DC component

AC components

Quantize

79 0 -1 0-2 -1 0 0-1 -1 0 00 0 0 0

zigzag79 0 -2 -1 -1 -1 0 0 -1 0 0 0 0 0 0 0

run-lengthcode

0 791 -20 -10 -10 -12 -10 0

Huffmancode 10011011100011...

coded bitstream < 10 bits (0.55 bits/pixel)


Result of Coding/Decoding

139 144 149 153144 151 153 156150 155 160 163159 161 162 160

144 146 149 152156 150 152 154155 156 157 158160 161 161 162

original block reconstructed block

-5 -2 0 1-4 1 1 2-5 -1 3 5-1 0 1 -2

errors


Examples

Uncompressed(262 KB)

Compressed (50)(22 KB, 12:1)

Compressed (1)(6 KB, 43:1)


Discrete Cosine Transform

F[u,v] = 4C(u)C(v)

n2n-1 n-1

j=0 k=0

f(j,k) cos(2j+1)u

2n

(2k+1)v

2ncos

where

C(w) =

1

21

for w=0

for w=1,2,…,n-1

• Inverse is very similar• DCT better at reducing redundancy than Discrete Fourier

Transform but computationally expensive


DCT vs DFT

original signal

recovered from DCT

recovered from DFT


Inter-frame Compression

• Pixel difference with previous frameIf previous pixel very similar, skip it

• Send sequence of blocks rather than framesIf previous block similar, skip it or send difference

• Motion compensationSearch around block in previous frame for a better matching block and encode position and error differenceSearch in future frameAverage block in previous and future frame


Background

• JPEG still image (1 bpp)Symmetric codec

• MPEGInterleaved audio and videoLow cost decoder at expense of high cost encoder (asymmetric)

• H.26xVideo conferencing standard (QCIF, CIF, 4CIF, and 16CIF)Variable bit rates and coding flexibility


MPEG Standards

• MPEG1 - vhs quality (1992)CIF images, 4:2:0 sampling, 1.5 MbsFrame encoding

• MPEG2 - broadcast quality (1994)CCIR 601 images, 4:2:2 sampling, 15 MbsInterlaced and progressive scanningFrame and field encoding


MPEG Technology

• Picture coding types• Motion compensation• Bit rate control• Picture order


Do MPEG slides HERE

• Frame typesI, P, B, and Bi frames

• JargonGOP, macroblock, slice

• Motion vectorsPredicted and interpolated vectors


Compression Formats

• Cinepack (2 bpp)Inter-frame, RGB15, software playback

• Motion JPEG (1-4 bpp)Intra-frame, DCT+Quantization, good for editing

• MPEG (0.5-2 bpp)Inter-frame, DCT+Quantization+Motion Compensation, excellent for playback

• H.261/H.263Inter-frame, DCT+Quantization+Motion Compensation, block stream, excellent for conferencing


Quantitative Measures of Quality

• Mean Squared Error n-1 n-1

MSE = f(i,j) - f(i,j)

i=0 i=0

where f(i,j) is original image and f’(i,j) is image

after compression and decompression

• Perceptual MeasuresPerceptual Distortion Measure (Heeger)Picture Quality Scale (Miyahara, et.al.)Spatial/Temporal Measure (Webster, et.al.)

1

n2


Discussion

• Where are the lossy steps?Quantization and subsampling before coding

• How do you choose quantization matrix?Standard proscribes matrix based on psychophysicsVary quantization by scaling Q matrix (i.e. MQUANT)Custom designed Q matrix

• Can improve compressioin by using motion compensation (MC)

• Comparison of standards…JPEG - still image uses fixed Q matrixH.261 - video conferencing uses MC and variable

quantizationMPEG - video playback uses MC and variable

quantization


Discussion

• How to do very low bit rate (28.8 Kbs)?

Small image, low quality/bit rate, and better codingMPEG4 will address this problem

• Gazillions of other proposals…

Fine, but who needs themScalable proposals will be useful

• Must be prepared to deliver several formats


H.261 Coding Example

1 2

543

6 7

1 2

54

3

6

1 2 3 4 5 6 7

1 2 3 4 5 6


Other Techniques

• Vector QuantizationUse small codebook of valuesSlow encode, fast decode

• Fractal CodingFit curves to signalSlow encode, fast decode?

• Wavelet CodingBetter transform than DCT – incorporates both spatial and frequency in transformUsed in most research work and scalable codecs


Scalable Algorithms• Some applications want to control bandwidth and

performanceSend low quality, low bit rate image on high priority channel and quality improvements on low priority channel(s)May be dictated by network bandwidth (wireless), end station (PDA), or application need (video gallery)

• Scalability parametersImage sizeFrame rateReconstructed image quality


Pyramid Codes

CodedImage-0

CodedImage-1

Low Quality


MPEG4 Standards Work

• Original goal was very low-bit rate codingCan do remarkably well at rates as low as 10 bits/sec but coding/decoding time is largeHierarchical VQ is popular because of low cost to decode, but trade-off is less coding efficiency

• Recent activity has moved towards programmable decoders

Idea is to allow code to be downloaded to the decoding chip

• Difficult trade-off between power consumption, quality and coding algorithm


Understanding CODEC Performance

• Problem is motion vector search in MPEG• Low bit-rate coding (1.5 Mbs) requires

excellent coding

CODEC Compress Decompress

JPEG 314 inst/pixel 283 inst/pixel

MPEG1 1120 inst/pixel 60-100 inst/pixel


MPEG Coding Performance• Decoding is easy

MPEG1 decoding in software on most platformsHardware decoders are widely available ($150/board)Windows graphics accelerators with MPEG decoding now entering market (e.g., Matrox, Diamond, …)

• Encoding is expensiveSequential software encoders are 20:1 real-timeReal-time encoders use parallel processingReal-time hardware encoders are expensive (e.g., $12K-$50K for MPEG1 and $100K-$500K for MPEG2)Hardware-assisted off-line MPEG1 encoders (3:1) used for multimedia authoring at reasonable cost ($5k)


Put decoding/encoding performance slides here


Other Boards

• MJPEGDEC J300 (Turbochannel, CL560) costs $3KParallax (Sun, HP, IBM R6K, CL560) costs $4K-$7.5KSGI Cosmo (CL560) cost (?)Radius VVS (Mac, CL560) costs $3K (?)Radius Truevision (PC, (?)) costs (?)PC boards (ASL, RealMagic MPEG Editor, IIT, …)

• Proprietary AlgorithmsIntel Smart Video Recorder Pro ($3K, I860)

• Programmable BoardsSun VideoPix (CL4K) supports MJPEG, MPEG, CELLB costs $1.5KChromatics MPACT chip board (mid ‘96)

Image/Video Compression September 28, 1999 Lawrence A. Rowe University of California, Berkeley URL:...

Documents

Transcript of Image/Video Compression September 28, 1999 Lawrence A. Rowe University of California, Berkeley URL:...