MPEG-4

• Objective• Standardize algorithms for audiovisual coding in

multimedia applications allowing for• Interactivity

• High compression

• Scalability of audio and video content

• Support for natural and synthetic audio and video

• The Idea• An audiovisual scene is a coded representation of

audiovisual objects related in space and time

MPEG-4: Scenario

• A/V object• A video object within a scene

• The background

• An instrument or voice

• Coded independently

• A/V scene• Mixture of natural or synthetic objects

• Individual bitstreams multiplexed and transmitted

• One or more channels

• Each channel may have its own quality of service

MPEG-4: Video Object Plane

• Video frame = sum of segmented regions with arbitrary shape (VOP)

• Shape motion and texture information of VOPs belonging to the same video object is encoded into a video object layer (VOL)

• Encode• VOL identifiers• Composition information

• Overlapping configuration of VOPs

MPEG-4: Coding

• Shape coding• Shape information in alpha planes

• Transparency of shape encoded

• Inter and intra shape coding functions

• After shape coding each VOP in a VO is partitioned into non-overlapping macroblocks

• Motion coding• Shift parameter wrt reference window

• Standard macroblock

• Contour macroblock

MPEG-4: Coding• Texture coding

• Intra-VOPs, residual errors from motion compensation are DCT coded like MPEG-1

• 4 luminance and 2 chrominance blocks in a macroblock

• P-VOPs (prediction error blocks) may not conform to VOP boundary

• Pixels outside the active area are set to a constant value

• Standard compression

• Efficient prediction of DC and AC components from intra and inter coded blocks

• Multiplexing• Shape motion texture coded data

• Motion and DCT coefficients can be jointly (H.263) or individually coded

MPEG-4 Video Object Segmentation-I

• Construct a video object• User selects start frame, outlines polygon designating rough object

boundary• Refine boundary using snake algorithm, if needed

• Compute a k-pixel bounding box around the object• Within bounding box compute

• Edge map: bit plane, after thresholding a convolution kernel• Color map: compute luminance and chrominance, quantize by k-

means clustering, keep quantization table• Motion field: block-based motion vector

• Segment into regions no significant edge, smooth color having smooth motion

• Intersect segments and initial object boundary and determine foreground and background region

• Estimate the motion of regions in the next frame with an affine motion model

MPEG-4 Video Object Segmentation-II

• Track object• Locate estimated position of foreground and background regions

from previous frame. Call this the object mask.

• Generate same three feature maps with the quantization table; Requantize if error is large

• Classify regions into foreground/background and new regions

• Intersection ratio r with object mask

• For foreground regions, if r > 80% OR foreground mask, mark as foreground; label foreground - mask as new

• For new regions, if r < 30% mark as new; if r > 80% mark as foreground; else find nearest-motion-similar neighbor. If it is in the foreground, do previous step, else keep region as new

• Iterate until stable