Methods of Video Object Segmentation in Compressed Domain
Cheng Quan Jia
Presentation Outline
• Features for Segmentation in Compressed Domain
• Using Motion Vectors in Segmentation
• Confidence Measure• Conclusion• Q & A
Features for Segmentation in Compressed Domain
An introduction to Compressed Domain
Compressed Domain: Definition
• Compressed Domain refers to video compression techniques that expliots Spatial and Temporal Redundancy using – DCT & Quantization– Motion Compensation
• Examples include MPEG-1/-2/-4, H.261 and H.263
Compressed Domain: Definition
• Opreations in the Compressed Domain involves processing of
– DCT coefficients (from I-macroblocks)– Motion Vectors (from P-/B-macroblocks)
Compressed Domain: Parsing
• Unlike pixel domain, operations in the compressed domain do not require the input bitstream to be decoded
• Instead, they are Parsed
Compressed Domain: Parsing
Features for Segmentation
• After Parsing, we have– DCT coefficients (from I-macroblocks)– Motion Vectors (from P-/B-macroblocks)
• Which coresspond to – Frequencies of texture change– Motion of the macroblock
Using Motion Vectors in Segmentation
Acquiring Dense Motion Field
• Many video object segmentation methods attempt to acquire a dense smooth motion field in order to create object masks
• For this end spatial interpolation and motion accumulation are employed
Motion Accumulation
Motion Accumulation
• Due to the different magnitude and signs of motion vectors, the obtained MVs are normalized, e.g. MVs in B-macroblocks would have their signs reversed
• Filtering is applied to remove non-uniform MV and smooth the motion field
Motion Accumulation
• Chen and Bajic [chen2009] employs MV Integration block-wise and pixel-wise to enhance the Motion Field
Motion Accumulation
Chen and Bajic [chen2009] Babu et al. [babu2004]
Porikli et al.’s Investigation
• The Compression Domain segmentation system published by Porikli et al. [porikli2010] experimented the effect of DCT coefficients and MV on segmentation performance– The DC parameters(for Y, U, V channels) of the I-
frame– Low vertical and horizontal frequency AC values– A spatial energy term– Aggregated motion flow of the corresponding
macroblock
Porikli et al.’s Investigation
• They create a Frequency-temporal data structure for each macroblock with the features and perform volume segmentation
• Their results show that using DCT terms in FT segmentation and using MV in the hierarchical clustering, on average, gives better results.
Porikli et al.’s Investigation
• The Block Matching Process in encoding stage looks for only the best match for a macroblock rather than object motion
Porikli et al.’s Investigation
Confidence Measure of Motion Vectors
• Coimbra and Davies [coimbras2005] try to approximate Lucas–Kanade optical flow in MPEG-2 Compressed Domain
Approximating Optical Flow
• They argue that AC[1] and AC[8] in an I-macroblock can be used as confidence measure
• The confidence update step will have a 8×8 macroblock referencing a 16×16 image block in the I-frame, and the confidence of the motion vector of the macroblock is the weighted average of confidence in the 16×16 window
Confidence Measure
Confidence Measure
Original image MPEG-2 smooth motion field afterconfidence threshold
Conclusion
• Due to block matching process, motion vectors in P-/B- frames do not necessary relate to object motion
• To ensure a motion vector is correlated to object motion, some sort of confidence measure is required
• [coimbras2005] demonstrated that edge strength can be an effective measure
Conclusion
• Problems not discussed here – Camera motion– Changes in illumination– Occlusions
Conclusion
References1. R. V. Babu, K. R. Ramakrishnan, and S. H. Srinivasan.
Video Object Segmentation: A Compressed Domain Approach. IEEE Transactions on Circuits and Systems for Video Technology, 14(4):462–473, April 2004.
2. Y.-M. Chen and I. V. Bajic. Compressed-Domain Moving Region Segmentation with Pixel Precision using Motion Integration. In IEEE Pacific Rim Conference on Computers and Signal Processing, 2009, pages 442 – 447, August 2009.
3. M. T. Coimbra and M. Davies. Approximating Optical Flow Within the MPEG-2 Compressed Domain. IEEE Transactions on Circuits and Systems for Video Technology, 15(1):103–107, January 2005.
4. F. Porikli, F. Bashir, and H. Sun. Compressed Domain Video Object Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 20(1):2–14, January 2010.
Q & A SECTION
Comments and Suggestions
Top Related