Slide 1

Video Coding with Spatio-temporal Texture Synthesis and Edge-based inpainting Chunbo Zhu, Xiaoyan Sun, Feng Wu, and Houqiang Li ICME 2008

Slide 2

Outline Introduction Framework Previous work Encoder Decoder Experiment Result Conclusion

Slide 3

Introduction Texture synthesis Image/video inpainting

Slide 4

Introduction Encoder Decoder [1] D. Liu, X. Sun, F. Wu, S. Li, and Y.-Q. Zhang, Image compression with edge-based inpainting, IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2007. Original image Edge map Removed blocks Structure propagation Texture synthesis JPEG

Slide 5

Introduction Texture regions can be well synthesized. Selectively removed during encoding. Restored by texture synthesis and edge-based inpainting in the decoder. Assistant information can be extracted to guide restoration. Region types, edge information, motion parameters. Keep spatio-temporal consistency

Slide 6

Frameworkprevious work [2] C. Zhu, X. Sun, F. Wu, and H. Li, Video coding with spatio-temporal texture synthesis, IEEE International Conference on Multimedia and Expo, 2007. I BPBB

Slide 7

Frameworkencoder I BPBB Local Motion Region Global Motion Region Same as previous work

Slide 8

Frameworkencoder

Slide 9

Structure blocks Texture blocks Block Categorization Edge detection Block categorization Exemplar selection examplars Check connective degree removed blocks do not constitute a large region

Slide 10

Frameworkencoder

Slide 11

Motion Threading [3] L. Luo, F. Wu, S. Li, Z. Xiong, and Z.Q. Zhuang, Advanced motion threading for 3D wavelet video coding, Signal Processing: Image Communication, Vol. 19, Issue 7, pp. 601-616, Aug. 2004. Spatial and temporal consistency Block-based backward motion estimation Motion threads in texture regions are pruned B0 B2B3 B4 B5 B1

Slide 12

Exemplar Selection Choose the threads with higher variation Output a sequence of binary masks indicating which blocks are removed

Slide 13

Frameworkencoder

Slide 14

Sprite Generation GMR pixel motion can be described by a parametric geometrical model Sprite (background mosaic) A sprite for a GOP [4] Y. Lu, F. Wu, S. Li, and Y.-Q. Zhang, Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques, IEEE Transactions on Circuits and Systems for Video Technology, May 2003. (x, y) and (x,y) are coordinates of pixels in original frame and reference image

Slide 15

Frameworkencoder

Slide 16

removable blocks unremovable blocks Region removal in GMR Mapping into original frame inverse warping if one 8x8 block in an original frame contains more than half of the removable pixels, it is selected as a removable block and then skipped when encoding. Otherwise, it is selected as an exemplar and coded with H.264 encoder. Edge Detection Region Classification Mapping Exemplar Selection RSRS RFRF SPT R S : Removable blocks in sprite R F : Removable blocks in original frame

Slide 17

Frameworkdecoder Mask1: region map Mask2: removing map

Slide 18

Decoder Spatio-temporal texture synthesis in LMR Texture synthesis module Arbitrary-shaped regions Perform frame by frame Square patch Confidence map Similarity of candidate patch and target patch Squared difference of known pixels

Slide 19

Frameworkdecoder

Slide 20

Decoder Edge-based inpainting in GMR Warping Blending Edge inpainting Texture Synthesis GMR GMR parameter SPT edges SPT H SPT

Slide 21

Experimental Results Implemented into JM 10.2 of H.264 YUV 4:2:0 sequence format with CIF resolution(352x288) rate distortion optimization (RDO) and CABAC were turned on

Slide 22

Experimental Results

Slide 23

Conclusion In this paper, we present a video coding scheme in which some regions are removed in the encoder and restored in the decoder by spatio-temporal texture synthesis and edge-based image inpainting. To keep temporal consistency, different types of motion have been considered in region removal and restoration for both textural and structural regions.