Low Complexity H.264 to VC-1 Transcoder Vidhya Vijayakumar Electrical Engineering Graduate Student...

Low Complexity H.264 to VC-1 Transcoder

Vidhya Vijayakumar

Electrical Engineering Graduate Student

The University of Texas at Arlington

Advisor

Dr. K. R. Rao, EE Dept, UTA

Co-Advisor

Dr. I. Ahmad, CSE Dept, UTA

July 14, 2010 Low Complexity H.264 to VC-1 Transcoder

2

Agenda Introduction to the field of research Motivation for the research Overview of H.264 Overview of VC-1 H.264, VC-1 Comparison Overview of Transcoding Proposed H.264 to VC-1 Transcoder Results Conclusions Future work References


3

Introduction to the field of research


4

Introduction Importance of video Need for compression

– High bandwidth requirements– Remove inherent redundancy

Need for standardization– Ensures interoperability

Year

Coding

Efficiency

Network

awareness

Complexity20052005

20102010

19991999

19941994

MPEG4MPEG4

H.264H.264

19921992MPEG1MPEG1

Video Conferencing

H.26H.2633

20032003

Mobile Phone

Hand PC

Mobile TV

SVCHDTV

MPEG2MPEG2

H.265/HECH.265/HEC/ NGVC/ NGVC

VC-1


5

Motivation for the research


6

Motivation for a H.264 to VC-1 transcoder Choice of codecs

– Prevalence of H.264 and VC-1 Need for transcoding

– Co-existence in broadcast, streaming, mobile communication, storage etc. – VC-1 is comparable to H.264 on subjective quality – VC-1 being less complex than H.264 – Favorable for limited resource

devices

Broadcast

Streaming

Content Server

Internet

Link

Mobile

Storage

H.264

ISO media file format

http://www.lgmyshop.com/product_image/314/832960.jpg

http://images.google.co.kr/imgres?imgurl=www.eitimes.co.kr/neweitimes/02-10-15/image/tec-report/tec-report2.jpg&imgrefurl=http://www.eitimes.co.kr/neweitimes/02-10-15/teck-report.htm&h=166&w=200&prev=/images%3Fq%3D%25EB%25A9%2594%25EB%25AA%25A8%25EB%25A6%25AC%2B%25EC%258A%25A4%25ED%258B%25B1%26svnum%3D10%26hl%3Dko%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26newwindow%3D1


7

An application scenario

– Network uses H.264 and all end device have complex decoders

– With transcoder, complexity shifted to a single point

– Devices can use less complex VC-1 codec with transcoder

H.264 to VC-1

Transcoder

Motivation for a H.264 to VC-1 transcoder


8

Overview of H.264/AVC


9

H.264: Overview Latest block-oriented motion-compensation-based codec. Good video quality at substantially lower bit rates. Better rate-distortion performance and compression

efficiency than MPEG-2. Simple syntax specifications, very flexible. Network friendly. Layered structure - consists of two layers: Network

Abstraction Layer (NAL) and Video Coding Layer (VCL). Wide variety of applications such as video broadcasting,

video streaming, video conferencing, D-Cinema, HDTV.


10

H.264 - Encoder


11

Design Features Highlights Features for enhancement of prediction

– Directional spatial prediction for intra coding • 9 intra 4x4 modes + 4 intra 16x16 modes + 9 intra 8x8 modes

– Variable block-size motion compensation with small block size • 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4

– Quarter-sample-accurate motion compensation– Multiple reference picture motion compensation– In-the-loop deblocking filtering to remove blocky artifacts

Features for improved coding efficiency– Small block-size transform – 4x4 and 8x8 integer DCT– Exact-match inverse transform – Short word-length transform– Hierarchical block transform– Arithmetic entropy coding– Context-adaptive entropy coding


12

Overview of VC-1


13

VC-1 : Overview Informal name of the SMPTE 421M video codec Standard initially developed by Microsoft – WMV 9 Supported standard for blu-ray discs and windows media video Alternative to H.264/AVC Better visual quality when compared with H.264* and MPEG-2

demonstrated in independent tests. Prevalent codec in Microsoft’s ASF files, Silver light framework, X-

Box 360 and Play station 3 Delivers HD content at bit rates as low as 6-8 Mbps Requires less computational power Block-oriented motion-compensation, DCT based codec Coding tools for interlaced and progressive encoding* - results compared before the FRExts


14

VC-1Codec


15

Design Feature Highlights Adaptive variable transform size (16 bit transforms)

– 8x8, 8x4, 4x8, 4x4 DC/AC intra prediction

– no spatial prediction, always uses 8x8 transform size Simple motion estimation

– block sizes of 16x16 and 8x8 only, ¼ pixel accuracy Deblocking filter

– Overlap transform– In loop filtering

Multiple scanning patterns Quantization with dead zone (uniform and non uniform quantization) Bit plane coding Huffman coding Intensity compensation Range reduction


16

Comparison of H.264 and VC-1


17

H.264 Baseline Vs VC-1 SimpleFeature H.264 Baseline VC-1 Simple

Picture type I, P I, P

Transform Size 4x4 4x4, 4x8, 8x4, 8x8

Transform Integer DCT Integer DCT

Intra Prediction 4x4, 16x16 spatialFrequency domain DC and AC

Prediction

Motion CompensationBlock Size

16x16, 16x8, 8x16,8x8, 8x4, 4x8, 4x4

16x16, 8x8

Total MB Modes 7 inter + (9 + 4) intra 2 inter + 1 intra

Motion Vectorresolution

¼ pixel ¼ pixel

In loop filter Deblocking Deblocking, Overlap transform

Reference Frames Single Single

Entropy coding CAVLC Adaptive VLC


18

Performance Comparison

Codec Ranking WMV-9 H.264/AVC MPEG-2

Sequence 1 – Dick Tracey 1 1 (0) 4 (-0.4)

Sequence 2 – Titan 1 2 (-0.3) 3 (-0.4)

Sequence 3 – Harry Potter 1 2 (-0.2) 2 (-0.2)

Sequence 4 – Stuart Little 2 1 3 (-0.4) 2 (-0.1)

Sequence 5 – Seven 1 3 (-0.1) 1 (0)

Sequence 6 – Monsters Inc 1 2 (-0.1) 3 (-0.6)


19

Overview of Transcoding


20

Transcoding Conversion of video from one

format to another− Bit rate conversion

− Spatial resolution change− Temporal conversion − Format change

Architectures– Cascaded decoder & encoder

– Spatial domain

– Frequency domain

– Hybrid domain


21

Proposed H.264 to VC-1 Transcoder


22

Choice of Profiles The profiles chosen for H.264 is Baseline profile. The

reasons for the choice of this profile are– Used in mobile applications and video conferencing– Less complex, no B pictures– Single reference frame, less memory requirements

The corresponding profile in VC-1 is simple profile which matches most of the features of the baseline profile in H.264


23

Choice of Transcoding Architecture Differences

– ME/MC block sizes• H.264 supports 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 macroblock

partitions

• VC-1 supports 16x16 and 8x8

– Transform size and type• H.264 supports 8x8 and 4x4 block sizes

• VC-1 supports 8x8, 8x4, 4x8 and 4x4 block sizes

Hence heterogeneous transcoding in pixel domain is chosen


24

Reference & Proposed Architecture

H.264 Encoder

H.264 Decoder

VC-1 Encoder

VC-1 Decoder

Original YUVReconstructed

YUV

H.264 bitstream

VC-1 bitstream

H.264 Encoder

H.264 Decoder

Reduced complexity

VC-1 Encoder

VC-1 Decoder

Original YUV

Reconstructed YUV

Reusable information – MB mode, Motion vector

H.264 bitstream

VC-1 bitstream

Reference

Proposed


25

Methods used to reduce complexity To achieve low complexity

– Re-use of mode decisions– Re-use of macroblock partition sizes for macroblock partitioning (16x16 or

8x8)– Transform size selection (8x8, 8x4, 4x8, 4x4) from macroblock partitions– Re-use of motion vectors and thereby eliminating the motion estimation block

Value of eliminating the mode decision in VC-1– Complex manipulations relating to RD cost of each mode is not done

Value of eliminating the motion estimation in VC-1– Motion estimation is the most complex block in block based codecs– Up to 70% of the encoding time is taken up by motion estimation– This translates into lesser hardware costs, lesser cycles to process data and

lesser power consumption in the device Value of eliminating the transform size selection in VC-1

– Deciding the transform size involves complex calculations for each 8x8 block


26

Extracted information from incoming H.264 bitstream

H.264 bitstream contains information about Macroblock type

– P16x16 – P block type 16x16– P16x8 – P block type 16x8– P8x16 – P block type 8x16– P8x8 – P block type 8x8– I4MB – I block type 16x16– I16MB – I block type 16x16

Macroblock sub block type– SMB8x8 – sub macroblock type 8x8 – SMB8x4 – sub macroblock type 8x4– SMB4x8 – sub macroblock type 4x8– SMB4x4 – sub macroblock type 4x4

Reference picture index Motion vector x, y

I frame P Frame


27

Mode decision and MV re-useIntra MB Mapping Inter MB Mapping

Skip as Skip Same reference picture

H.264 Intra MB VC-1 Intra MB

Intra 16x16 (Any mode) Intra MB 8x8

Intra 4x4 (Any mode) Intra MB 8x8

H.264 Inter MB VC-1 Inter MB Transform size in VC-1

Inter 16x16 Inter 16x16 8x8







H.264 Inter MB VC-1 Inter MB Motion Vector Re-use

Inter 16x16 Inter 16x16 Same motion vectors for 16x16 block

Inter 16x8 Inter 8x8 Median of motion vectors for each 8x8 block


Inter 8x8 Inter 8x8 Same motion vectors for 8x8 block





28

Motion vector selection

med - median


29

Motion vector selection 16x8 to 8x8 / 8x16 to 8x8 mode

mapping– Since the euclidean distance is

the same between 2 MV, the average is chosen

8x8 to 8x8 mode mapping – Choose MVi with minimum di


30

Proposed Algorithm

N

Y

Update for VC1 Inter8x8

N

End

MB == 8x8

Start

First

frame

Update for VC1 SKIP




MB == 16x16

MB == 16x8

MB == 8x16

Y

N

N N N

YY Y Y

Perform VC1 Intra

MB == Skip


31

Implementation and Results


32

Implementation and testing H.264 decoder - JM 16.1

– The decoder was modified to output the needed information (C implementation) VC-1 Reference software from SMPTE

– The encoder was modified to re-use the information (C implementation) The simulation was carried out using 5 sequences

– The sequences were chosen with a variety of motion in them– Resolutions commonly used in mobile devices were chosen

Sequence Size Motion

Akiyo QCIF (176x144) Low

Miss America QCIF (176x144) Low

Foreman CIF (352x288) Medium-High

Football CIF (352x288) Medium-High

Mobile Calendar CIF (352x288) Medium


33

Results for AkiyoAkiyo - Comparison of Y MSE

0

5

10

15

20

25

30

35

10 15 20 28 35

QP

Y M

SE

Re-encodingProposed

Akiyo - Comparison of Y PSNR

30.00

32.00

34.00

36.00

38.00

40.00

42.00

44.00

10 15 20 28 35

QP

Y P

SN

R (

dB)

Re-encodingProposed

Akiyo - Comparison of Y SSIM

0.88

0.90

0.92

0.94

0.96

0.98

1.00

10 15 20 28 35

QP

Y S

SIM

Re-encodingProposed

Akiyo - Comparison of encoding times

0

2

4

6

8

10

12

14

10 15 20 28 35

QP

En

cod

ing

tim

e (

s)

Re-encodingProposed


34

Results for Akiyo

Akiyo - % change in bitstream file size

-60.00

-50.00

-40.00

-30.00

-20.00

-10.00

0.00

10 15 20 28 35

QP

% C

ha

ng

e in

bits

tre

am

file

siz

e

(a) Original (b) H.264 decoded

(c) Reference cascade decoded at QP 10

(d) Proposed transcoder decoded at QP 10


35

Results for Akiyo with respect to original videoAkiyo - Comparison of Y MSE

0

10

20

30

40

50

10 15 20 28 35

QP

Y M

SE

Re-encoding

Proposed

H.264

Akiyo - Comparison of Y PSNR

30.00

35.00

40.00

45.00

50.00

55.00

10 15 20 28 35

QP

Y P

SN

R (

dB

)

Re-encoding

Proposed

H.264

Akiyo - Comparison of Y SSIM

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

10 15 20 28 35

QPY

SS

IM

Re-encoding

Proposed

H.264


36

Results for ForemanForeman- Comparison of Y MSE

0

5

10

15

20

25

30

35

10 15 20 28 35

QP

Y M

SE

Re-encodingProposed

Foreman - Comparison of Y PSNR

32.00

34.00

36.00

38.00

40.00

42.00

44.00

10 15 20 28 35

QP

Y P

SN

R (

dB

)

Re-encodingProposed

Foreman- Comparison of Y SSIM

0.88

0.90

0.92

0.94

0.96

0.98

1.00

10 15 20 28 35

QP

Y S

SIM

Re-encodingProposed

Foreman - Comparison of encoding times

0

10

20

30

40

50

60

70

10 15 20 28 35

QP

En

cod

ing

tim

e (

s)

Re-encodingProposed


37

Results for Foreman

Foreman - % change in bitstream file size

-60.00

-50.00

-40.00

-30.00

-20.00

-10.00

0.00

10 15 20 28 35

QP

% C

hang

e in

bits

tre

am fi

le s

ize





38

Results for Foreman with respect to original videoForeman- Comparison of Y MSE

0

10

20

30

40

50

60

10 15 20 28 35

QP

Y M

SE

Re-encoding

Proposed

H.264

Foreman - Comparison of Y PSNR

28.00

33.00

38.00

43.00

48.00

10 15 20 28 35

QP

Y P

SN

R (

dB

)

Re-encoding

Proposed

H.264

Foreman- Comparison of Y SSIM

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

10 15 20 28 35

QPY

SS

IM

Re-encoding

Proposed

H.264


39

Results for FootballFootball - Comparison of Y MSE

0

10

20

30

40

50

60

70

10 15 20 28 35

QP

Y M

SE

Re-encodingProposed

Football - Comparison of Y PSNR

28.00

30.00

32.00

34.00

36.00

38.00

40.00

42.00

44.00

10 15 20 28 35

QP

Y P

SN

R (

dB

)

Re-encodingProposed

Football - Comparison of Y SSIM

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

10 15 20 28 35

QP

Y S

SIM

Re-encodingProposed

Football - Comparison of encoding times

0

10

20

30

40

50

60

70

10 15 20 28 35

QP

En

cod

ing

tim

e (

s)

Re-encodingProposed


40

Results for Football

Football - % change in bitstream file size

-18

-16

-14

-12

-10

-8

-6

-4

-2

0

2

10 15 20 28 35

QP

% C

ha

ng

e in

bits

tre

am

file

siz

e





41

Results for Football with respect to original videoFootball - Comparison of Y MSE

0

20

40

60

80

100

120

10 15 20 28 35

QP

Y M

SE

Re-encoding

Proposed

H.264

Football - Comparison of Y PSNR

24.00

29.00

34.00

39.00

44.00

49.00

10 15 20 28 35

QP

Y P

SN

R (

dB

)

Re-encoding

Proposed

H.264

Football - Comparison of Y SSIM

0.70

0.75

0.80

0.85

0.90

0.95

1.00

10 15 20 28 35

QP

Y S

SIM

Re-encoding

Proposed

H.264


42

Conclusions


43

Conclusions Two dominant video codecs, H.264 and VC-1, were compared and

contrasted. A low complexity H.264 to VC-1 transcoder was developed

– Efficient re-use of mode decisions, block partitions and motion vectors

– Complete by pass of motion estimation in VC-1 re-encoding

– Proposed transcoder achieves comparable quality to reference re-encoding transcoder

– Reduced average encoding time by 80% compared to the re-encoding scheme.

– Performance of VC-1 transcoder is comparable to H.264’s subjective quality at low QP values.

– Performance of the transcoder falls short at high QP values.


44

Future work


45

Future work The current proposal does not consider motion vector

refinement. This can be researched in future for better performance.

The profiles considered for the input H.264/AVC bitstream is baseline profile. This can be extended to other profiles like main and high profiles. – Since main and high profiles of H.264 involve bi-directional

prediction with multiple reference pictures and VC-1 allows only two reference pictures, algorithms to re-scale the motion vectors according to the appropriate reference pictures can be explored.


46

References


47

References1. I. E.G. Richardson, “H.264 and MPEG-4 video compression: video coding for next-generation multimedia”,

Wiley, 2003.2. E. Lallana and M. Uy, “The Information Age,” UNDP-APDIP, 2003.3. Open source article, “Digital Revolution,” Wikipedia Foundation, http://en.wikipedia.org/wiki/Digital_Revolution4. J. Ribas-Corbera, “Windows Media 9 Series — a platform to deliver compressed audio and video for Internet

and broadcast applications”, EBU Technical Review No. 293, Jan. 20035. K. Sayood, "Introduction to Data Compression,” 3rd Edition, Morgan Kaufmann Publisher Inc., 2006. 6. Open source article, “Rec 601,” Wikipedia Foundation, http://en.wikipedia.org/wiki/Rec._6017. R. Schafer and T. Sikora, "Digital video coding standards and their role in video communications," Proceedings

of the IEEE, Vol. 83, pp. 907-923, Jan. 1995.8. Information technology-generic coding of moving pictures and associated audio information: ISO/IEC 13818-2

(MPEG-2) Std.9. Advanced Video Coding for Generic Audiovisual Services, ITU-T Rec. H.264 / ISO / IEC 14496-10, Nov. 2009.10. VC-1 Compressed Video Bitstream Format and Decoding Process (SMPTE 421M-2006), SMPTE Standard,

2006 (http://store.smpte.org/category-s/1.htm).11. A. Vetro, C. Christopoulos and H. Sun, “Video transcoding architectures and techniques: an overview,” IEEE

Signal Processing Magazine, pp 18-29, March 2003.12. I. Ahmad et al, “Video transcoding: An overview of various techniques and research issues”, IEEE Trans. on

Multimedia, vol. 7, pp. 793-804, Oct. 2005 13. C. Chen, P-H.Wu and H. Chen, “MPEG-2 to H.264 transcoding,” Picture Coding Symposium, pp. 15-17, Dec.

2004.14. Open source article, “H.264/MPEG-4 AVC,” Wikipedia Foundation, http://en.wikipedia.org/wiki/H.264/MPEG-

4_AVC15. G.A Davidson et al, “ATSC video and audio coding”, Proc. IEEE, vol 94, pp. 60-76, Jan. 2006 (www.atsc.org).16. G.F.-Escribano et al, “ An MPEG-2 to H.264 video transcoder in the baseline profile”, IEEE Trans. CSVT, vol.

20, pp. 763-768, May 2010

http://en.wikipedia.org/wiki/Rec._601

http://store.smpte.org/category-s/1.htm


48

References17. Jae-Beom Lee and H. Kalva, "An efficient algorithm for VC-1 to H.264 video transcoding in progressive

compression," IEEE International Conference on Multimedia and Expo, pp. 53-56, July 2006.18. Open Source Article, “VC-1 Technical Overview”, Microsoft Corporation,

http://www.microsoft.com/windows/windowsmedia/howto/articles/vc1techoverview.aspx 19. DVD Forum, http://www.dvdforum.org/forum.shtml20. DV Magazine, http://www.dv.com/21. Tandberg Television, http://www.tandberg.com/index2.jsp22. CT Magazine, http://www.heise.de/ct/23. European Broadcasting Union, http://www.ebu.ch/24. Blue-ray Disc Association - http://www.blu-raydisc.com/index.htm25. Real Media, http://www.real.com/26. Envivo MPEG-4, http://www.envivio.com/27. Apple MPEG-4, http://www.apple.com/quicktime/technologies/mpeg4/28. M. Jeffrey, “The SMPTE 421M "VC-1" Standardization Project”, ITU-T Workshop Video and Image Coding and

Applications, July 200529. R. Pereira, K.R. Rao and A. Kruafak, “Efficient transcoding of an MPEG bitstream to an H.264 bit stream”,

University Scientific Journal series Telecommunications and Electronics, vol.11, pp. 5-31, Poland, 200830. S. Sharma and K.R. Rao, “Transcoding of H.264 bitstream to MPEG-2 bitstream”, IEEE APCC, pp. 391-396

Bangkok, Thailand, 18-20 Oct. 2007.31. S. Moiron et al, “Video transcoding from H.264/AVC to MPEG-2 with reduced computational complexity”, Signal

Processing: Image Communication, vol 24, pp. 637-650, Sept. 200932. S. Kwon, A. Tamhankar and K.R. Rao, ”Overview of H.264 / MPEG-4 Part 10”, J. Visual Communication and

Image Representation, vol. 17, pp.183-216, April 2006.33. T. Wiegand and G. J. Sullivan, “The H.264 video coding standard”, IEEE Signal Processing Magazine, vol. 24,

pp. 148-153, March 2007.34. A. Puri et al, “Video Coding using the H.264/ MPEG-4 AVC compression standard”, Signal Processing: Image

Communication, vol. 19, pp: 793 – 849, Oct. 2004.


49

References35. D. Marpe, T. Wiegand and S. Gordon, “H.264/MPEG4-AVC Fidelity Range Extensions: Tools, Profiles,

Performance, and Application Areas”, Proc. IEEE International Conference on Image Processing 2005, vol. 1, pp. 593 - 596, Sept. 2005.

36. G. Sullivan, P. Topiwala and A. Luthra, “The H.264/AVC Advanced Video Coding Standard: Overview and Introduction to the Fidelity Range Extensions”, SPIE conference on Applications of Digital Image Processing XXVII, vol. 5558, pp. 53-74, Aug. 2004.

37. K. R. Rao and P. C. Yip, “The transform and data compression handbook”, Boca Raton,FL: CRC press, 2001.38. T. Weigand et al, “Introduction to the Special Issue on Scalable Video Coding—Standardization and Beyond”

IEEE Trans on Circuits and Systems for Video Technology, vol. 17, pp. 1034, Sept. 2007.39. HHI presentation of the Scalable Extension of H.264/AVC, http://ip.hhi.de/imagecom_G1/savce/index.htm40. S. Srinivasan and S. L. Regunathan, "An overview of VC-1," Visual Communications and Image Processing

(VCIP), Proc. SPIE, vol. 5960, pp. 720-728, July 2005.41. Open Source Article, “VC-1”, Wikipedia Foundation, http://en.wikipedia.org/wiki/VC-142. S. Srinivasan et al, "Windows Media Video 9: overview and applications", Signal Processing: Image

Communication, vol. 19, Issue 9, pp. 851-875, Oct. 200443. Y. Huh, K. Panusopone, K.R. Rao, “Variable block size coding of images with hybrid quantization”, IEEE Trans.

Circuits Systems Video Technol. 6, pp 679–685, Dec. 199644. J. Ribas-Corbera and D.L. Neuhoff, “Optimizing Block Size in Motion Compensation”, Journal of Electronic

Imaging, vol. 7, pp.155-165, Jan. 199845. T.D.Tran, J.Liang and C. Tu, “Lapped Transform via Time-Domain Pre- and Post-Filtering”, IEEE Trans on

Signal Processing, vol.51, no.6, pp. 1557-1571, Jun. 2003. 46. W.B.Pennebaker and J.L.Mitchell, JPEG Still Image Data Compression Standard, Van Nostrand Reinhold,

1993. 47. M. Wien, “Variable block size transforms for H.264/AVC”, IEEE Trans. Circuits Systems Video Technology, vol.

13, pp. 604–613, July 2003.48. S. Gordon, “Adaptive Block Transform for Film Grain Reproduction in High Definition Sequences”, ”, Joint Video

Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, doc. JVT-H029, Geneva, Switzerland, 23-27 May, 2003 (available via anonymous ftp from ftp://ftp.imtc-files.org/jvt-experts/)


50

References49. Tektronics Picture Quality Analyzer PQA 300 (http://www.tek.com/site/ps/0,,25-11735-INTRO EN,00.html).50. Microsoft, WMV-9—an Advanced Video Codec for 3GPP, 3GPP SA4 Meeting #18, document S4-030613.

(available from http://www.3gpp.org/ftp/tsg sa/WG4 CODEC/TSGS4 28/Docs/)51. Nokia, Proposal to support MPEG-4 AVC / H.264/ AVC in Rel-6, 3GPP SA4 Meeting #27, document S4-030478.

(available from http://www.3gpp.org/ftp/tsg sa/WG4 CODEC/TSGS4 27/Docs/)52. Z. Volta, Kompressionisten, C’T Mag. 10 (2003) 146-159, (in German, summary at

http://www.heise.de/ct/03/10/146/).53. Windows Media Web site for Consumer Electronic devices,

http://www.microsoft.com/windows/windowsmedia/conselec.asp.54. M. Ravassi, M. Mattavelli and C. Clerc, “JVT/H.26L decoder complexity analysis”, Joint Video Team (JVT) of

ISO/IEC MPEG & ITU-T VCEG, doc. JVT-D153, Klagenfurt, Austria, 22–26 July, 2002 (available via anonymous ftp from ftp://ftp.imtc-files.org/jvt-experts/).

55. H. Sun, X. Chen and T. Chiang, “Digital video transcoding for transmission and storage,” CRC Press, 2005. 56. J. Xin, C. Lin and M. Sun, “Digital Video Transcoding”, Proceedings of the IEEE, Vol 93, pp 84-96, Jan. 200557. B. Girod, “Overview: Video coding standards,” Stanford University coursework,

http://www.stanford.edu/class/ee398b/handouts.htm58. G. Sullivan, “Overview of international video coding standards (preceding H.264/AVC),”ITU-T VICA workshop,

Geneva, July 2005.59. M.-T. Sun, T.-D. Wu and J.-N. Hwang, “Dynamic bit allocation in video combining for multipoint conferencing,”

IEEE Trans. Circuit Syst. II, vol. 45, no. 5, pp. 644-648, May 1998.60. O. Werner, “Re-quantization for transcoding of MPEG-2 intra frames,” IEEE Trans. Image Processing, vol. 8, no.

2, pp. 179-191, Feb. 1999.61. T. Shanabelah and M. Ghanbari, “Heterogeneous video transcoding to low spatial temporal resolutions and

different encoding formats,” IEEE Trans. Multimedia, vol. 2, no. 2, pp. 101-110, Jun. 2000.62. K.-H. Tan and M. Ganbari, “Layered image coding using the DCT pyramid,“ IEEE Trans. Image Processing, vol.

4, no. 4, pp. 512-516, Apr. 1995.


51

References63. J.-N.Hwang and T.-D. Wu, “Motion vector re-estimation and dynamic frame-skipping for video transcoding,” Conf

Rec. 32nd Asilomar Conf. Signals, Systems and Computer vol 2, pp 1606-1610, 1998.64. J. Youn, M.-T. Sun and C.-W. Lin, “Motion vector refinement for high-performance transcoding,” IEEE Trans.

Multimedia, vol. 1, no. 1, pp. 30-40, Mar 1999. 65. M.-J. Chen, M.-C. Chu and C.-W. Pan, “Efficient motion estimation algorithm for reduced frame-rate video

transcoder,” IEEE Trans. Circuits Syst. Video Technol., vol. 12, pp. 269-275, Apr 2002.66. N. Bjork and C. Christopoulos, “Transcoder architecture for video coding,” IEEE Trans. Consumer Electronics., vol.

44, pp. 88–98, Feb. 1998.67. R. Mohan, J. R. Smith, and C. Li, “Adapting multimedia internet content for universal access,” IEEE Trans.

Multimedia, vol. 1, no. 1, pp. 104–114, Mar. 199968. J.-N. Hwang and T.-D. Wu, “Motion vector re-estimation and dynamic frame-skipping for video transcoding,” in

Conf. Rec. 32nd Asilomar Conf. Signals, System & Computer, vol. 2, 1998, pp. 1606–1610.69. I. E. Richardson, “The H.264 Advanced Video Compression Standard”, Second Edition, Wiley, May 2010.70. JM reference software http://iphome.hhi.de/suehring/tml/71. VC-1 SMPTE software http://store.smpte.org/category-s/30.htm72. A. Luthra, G. Sullivan and T. Wiegand, “Introduction to the special issue on the H.264/AVC video coding standard”,

IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, issue 7, pp. 557-559, July 2003. 73. J. Padia, “Complexity reduction for VP6 to H.264 transcoder using motion vector re-use”, M.S. Thesis, EE Dept, UT

Arlington, May 2010. 74. Open source article, “MPEG-2,” Wikipedia Foundation, http://en.wikipedia.org/wiki/MPEG-275. Open source article, “MPEG-1,” Wikipedia Foundation, http://en.wikipedia.org/wiki/MPEG-176. Open source article, “H.261,” Wikipedia Foundation, http://en.wikipedia.org/wiki/H.26177. Open source article, “H.263,” Wikipedia Foundation, http://en.wikipedia.org/wiki/H.26378. Open source article, “MPEG-4 - Part 2” Wikipedia Foundation, http://en.wikipedia.org/wiki/MPEG-4_Part_279. H. Kalva and J.B. Lee, “The VC-1 and H.264 Video Compression Standards for Broadband Video Services”,

Springer, 2008. 80. H. Kalva and J.B. Lee, “The VC-1 Video Coding Standard,” IEEE Multimedia, Vol. 14, No 4, pp. 88-91, Oct.-Dec.

2007.

Thank you


53

H.264 - Profiles


54

Design Features Highlights Features for enhancement of prediction

– Directional spatial prediction for intra coding– Variable block-size motion compensation with small block size– Quarter-sample-accurate motion compensation– Motion vectors over picture boundaries– Multiple reference picture motion compensation– Decoupling of referencing order from display order– Decoupling of picture representation methods from picture

referencing capability– Weighted prediction– Improved “skipped” and “direct” motion inference– In-the-loop deblocking filtering


55

Design Features Highlights Features for improved coding efficiency

– Small block-size transform– Exact-match inverse transform– Short word-length transform– Hierarchical block transform– Arithmetic entropy coding– Context-adaptive entropy coding


56

Design Features Highlights Features for robustness to data errors/losses

– Parameter set structure– NAL unit syntax structure– Flexible slice size– Flexible macroblock ordering (FMO)– Arbitrary slice ordering (ASO)– Redundant pictures– Data Partitioning– SP/SI synchronization/switching pictures


57

Directional spatial prediction for intra coding Intra prediction is to predict the

texture in current block using the pixel samples from neighboring blocks

Intra prediction for 44 (9 modes) and 16 16 blocks (4 modes) are supported in all H.264 profiles.

Intra prediction for 8x8 (9 modes) is supported in the high profiles.

Intra 4x4

Intra 8x8Intra 16x16


58

Variable block-size motion compensation Partitioned in 2 stages In the 1st stage, determine first 4

modes– 1616– 168– 816– 88

If mode 4 (88) is chosen, further partition into smaller blocks for every 88 block– 84– 48– 44

At most 16 motion vectors may be transmitted for a 1616 macroblock

Sub pixel accuracy Large computational complexity to

determine the modes but efficient encoding


59

Multiple reference picture motion compensation P Slice

– More than one prior coded picture can be used as reference for MC prediction

– Reference index parameter is transmitted for each MC 1616, 168, 816 or 88

– For smaller blocks within the 88 use 1 reference index

– P-Skip type is supported B Slice

– Utilize two distinct lists of reference pictures

– Four different types of inter-picture predict• List 0, list 1, bi-predictive, and direct

– Bi-predictive• weighted average of MC list 0 and

list 1– Direct prediction

• Inferred from previously transmitted syntax

• Either list 0 or list 1 prediction or bi-predictive

– Similar macroblock partitioning as P slices is utilized

– B Skip mode is supported

P frame

B frame


60

Hierarchical block transform 4x4 and 8x8 (high profile only) multiplier-free integer DCT

transform Transform coefficients perfectly invertible Hierarchical transform (Integer DCT and Hadamard) For macroblock coded in 1616 Intra mode and

chrominance blocks

– DC coefficients are further grouped and transformed Hadamard transform is used for chrominance block

Integer DCT 4x4 Integer DCT 8x8 Hadamard 4x4

Hadamard 2x2


61

In loop deblocking filter Block based operations are

responsible for blocking artifacts In-loop deblock filter –smoothes

blocky edges; increases rate-distortion performance.

Applied to all 4x4 blocks except at picture boundaries.

Filtering adaptive at Slice level Block level Pixel level

Vertical edges filtered first (left to right)

Followed by horizontal edges (top to bottom)


62

Entropy encoding CAVLC (Context-based Adaptive Variable

Length Coding).

CABAC (Context-based Adaptive Binary

Arithmetic Coding).

CAVLC makes use of run-length encoding.

CABAC utilizes arithmetic coding; codes

both MV and residual transform coefficients.

Typically CABAC provides 10-15 %

reduction in bit rate compared to CAVLC,

for the same PSNR.

All other syntax elements are encoded by

Exp-Golomb codes (Universal Variable

Length Codes (UVLC)).

CAVLC

CABAC


63

Computational Overhead

Entropy encoding Multiple block size Smaller block size Integer transform In-loop deblocking


64

H.264 Extensions Scalable video coding

Application scenario


65

H.264 Extensions Scalable video coding


66

Types of Scalability


67

H.264 Extensions Multi view coding

Applications

3-D Video Stereoscopic TV


68

H.264 Extensions Multi view coding


69

VC1 Decoder : Simple and Main Profile

DECODED FRAME BUFFER

Decoding Process Block Diagram for VC-1 Simple and Main Profiles

BITSTREAM PARSING VLC

DECODE COEFFS

INVERSE QUANT

INVERSE TRANS

VLC DECODE

MVMV PRED

MOTION COMP

½ or ¼ pel interpolation

1 MV /4 MV

RANGE RED/ INTENSITY

COMP

+

VLC DECODE COEFFS

INVERSE QUANT

INVERSE TRANS

INTER

INTRA

AC/DC PRED

OVERLAP SMOOTH

LOOP FILTER

DECODED FRAME

DERING/DEBLOCK

DISPLAY PROCESS

COLOR CONV

RESIZE

Etc.

IMPLEMENTATION SPECIFIC

CONFORMING IMPLEMENTATION


70

VC-1 Decoder : Advanced Profile

DECODED PICTURE BUFFER

Decoding Process Block Diagram for VC-1 Advanced Profile

BITSTREAM PARSING VLC

DECODE COEFFS

INVERSE QUANT

INVERSE TRANS

VLC DECODE

MVMV PRED

MOTION COMP

½ or ¼ pel interpolation

1 MV /4 MV

INTENSITY COMP

+

VLC DECODE COEFFS

INVERSE QUANT

INVERSE TRANS

INTER

INTRA

AC/DC PRED

OVERLAP SMOOTH

LOOP FILTER

RANGEMAP

DERING/DEBLOCK

DISPLAY PROCESS

COLOR CONV

RESIZE

Etc.

IMPLEMENTATION SPECIFIC

CONFORMING IMPLEMENTATION

DECODED FRAME


71

VC1 ProfilesProfile Level Label

SimpleLow SP@LL

Medium SP@ML

Main

Low MP@LL

Medium MP@ML

High MP@HL

Advanced

L0 AP@L0

L1 AP@L1

L2 AP@L2

L3 AP@L3

L4 AP@L4


72

Design Feature Highlights Adaptive variable transform size (16 bit transforms) Multiple scanning patterns Quantization with dead zone Bit plane coding DC/AC intra prediction Simple motion estimation Huffman coding Intensity compensation Range reduction Overlap transform In loop filtering


73

VC1 - Intra prediction No spatial prediction like

H.264/AVC Mandatory DC prediction in the

transform domain Optional AC prediction in the

transform domain Independent luma and chroma

intra prediction Always uses 8x8 transform

size Inter MBs can have up to three

8x8 intra blocks


74

VC-1 Inter prediction Two block sizes

– 16x16, 8x8 Half and quarter pixel accuracy

– Bi-linear and bi-cubic interpolation filters

Four ME methods– Mixed block size (16x16 and

8x8), ¼ pixel, bicubic– 16x16, ¼ pixel, bicubic – 16x16, ½ pixel, bicubic – 16x16, ½ pixel, bilinear

Combined block size, MV resolution and filter representation


75

VC-1 Transform A variation of the discrete

cosine transform Transform sizes of 8x8,

8x4, 4x8,4x4 Bit accurate transform Fast algorithm for inverse

operation


76

Intensity Compensation Reference frame (luma and

chroma) data are scaled before using it for motion estimation

Useful in fade in and fade out scenes

Lesser residual energy by using intensity compensation

Defined only for P pictures (not for B pictures)


77

Overlap Transform Part of de-blocking process Removes blocking effect in high-low

quality discontinuity Switches over the edge data of two

adjacent blocks High quality and low quality blocks

diffuse each other Instead of simple switch, filtering

operation is performed Only on intra 8x8 blocks Not on inter, intra boundaries

(inter contains residual)


78

In loop filtering I, B – All 8x8 blocks P – Depending on

transform size used Horizontal edge followed

by vertical edge Only 2 pixels are filtered Filtering decision per

boundary


79

Innovations in VC-1 Adaptive block size transform

– Four different sizes 8x8, 8x4, 4x8,4x4– Smaller transforms are better in areas with discontinuities – Fewer ringing artifacts

16-bit implementation of the transform– Ease of implementation on chips

Multiple precision modes for motion compensation – 16x16 and 8x8, ¼ pixel accuracy, bi-linear/bi-cubic filters

Uniform and non-uniform quantization – Dead zone and uniform quantization

Loop-filtering– Less complex – lesser pixels filtered, lesser filtering decisions

Overlap smoothing– Reduces artifacts in intra coded blocks

Fading compensation– Better prediction in fade in, fade out scenes


80

H.264 Vs VC-1 H.264

– H.264 is significantly computationally involved • CABAC, loop filtering, smaller ME block sizes

– More memory requirement• Multiple reference picture buffers, max of 16

– Spatial intra prediction• 9 (intra 4x4) + 4 (intra 16x16) = 13 modes

VC-1– VC-1 is comparatively simpler

• Simpler entropy coding, simpler loop filtering, lesser ME block sizes

– Lesser memory requirement• Only 2 reference pictures

– No spatial intra prediction• Only DC/AC prediction in the transform domain


81

Need for Transcoding• Multimedia applications on different devices and platforms• Different bitrates, frame rates, spatial resolution & complexity• Different video standards; communication & inter-operability

Application Bitrate Video standardDigital TV broadcasting

2 to 6 Mbps (10 to 20 Mbps for HD broadcast)

MPEG-2, H.264

DVD Video 6 to 8 Mbps MPEG-2Internet video streaming

20 to 200 kbps Flash – Sorrension spark (based on H.263), VP6 and H.264; Silverlight uses VC-1; and also MPEG-4 Part 2

Video conferencing and video-telephony

20 to 320 kbps H.261, H.263, H.263+

Video over 3G wireless

20 to 100 kbps H.263, MPEG-4. Part 2

High definition – Blu-ray and HD-DVD

36 to 54 Mbps H.264, VC-1 and MPEG-2


82

Cascaded encoder and decoder Simplest but most in-efficient

as it involves re-encoding No degradation in visual

quality Full scale motion re-estimation

is needed.


83

Spatial Domain Transcoding


84

Frequency Domain Transcoding


85

Pseudo code #ifdef H264VC1TRANSCODER

if((mbType == I4MB) || (mbType == I16MB)){

// update block types for each 8x8 block}else if((mbType == P16x16) || (mbType == P16x8) || (mbType == P8x16) || (mbType == PSKIP)){

if(mbType == P16x16) // update all 4 8x8 block types as 1 MV MB

// update the motion vectors from the input fileif(mbType == P16x8)

// update all 4 8x8 block types as 4 MV MB// compute the median MV from input file// update the motion vectors as the median MV


// compute the median MV from input file// update the motion vectors as the median MV


// compute the median MV from input file depending on the sub macroblock type// update the motion vectors as the median MV

if(mbType == PSKIP) // update all 4 8x8 block types as skip

// update MV as the predicted MV}

#endif

Low Complexity H.264 to VC-1 Transcoder Vidhya Vijayakumar Electrical Engineering Graduate Student...

Documents

Transcript of Low Complexity H.264 to VC-1 Transcoder Vidhya Vijayakumar Electrical Engineering Graduate Student...