The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE
description
Transcript of The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE
![Page 1: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/1.jpg)
11
The MPEG-7 Visual Standard for The MPEG-7 Visual Standard for Content Description-An OverviewContent Description-An Overview
Thomas Sikora, Thomas Sikora, Senior Member, IEEESenior Member, IEEE
A presentation by Modupe OmuetiA presentation by Modupe Omueti
ForFor
CMPT 820:Multimedia Systems CMPT 820:Multimedia Systems Spring 2005Spring 2005
![Page 2: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/2.jpg)
22
ContentsContents IntroductionIntroductionScopeScopeMethodologyMethodologyVisual DescriptorsVisual DescriptorsConclusionConclusion
![Page 3: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/3.jpg)
33
IntroductionIntroductionMoving Pictures Expert GroupMoving Pictures Expert Group
MPEG-1 for interactive video (1992)MPEG-1 for interactive video (1992)MPEG-2 for digital television (1994)MPEG-2 for digital television (1994)MPEG-4 for multimedia with emphasis on MPEG-4 for multimedia with emphasis on
visual objects (1998 v1, 1999 v2)visual objects (1998 v1, 1999 v2)MPEG-7 for multimedia content description MPEG-7 for multimedia content description
(2001)(2001)
![Page 4: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/4.jpg)
44
TrendsTrends Initially few sources of audio, image and Initially few sources of audio, image and
videovideo Increase in volume of digitized audio, Increase in volume of digitized audio,
images and videoimages and videoStill images digital videoStill images digital video
![Page 5: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/5.jpg)
55
MPEG-7MPEG-7 Formally named Multimedia Content Description Formally named Multimedia Content Description
InterfaceInterface Supports some degree of interpretation of the Supports some degree of interpretation of the
information’s meaninginformation’s meaning Interpretation can be passed on to or accessed Interpretation can be passed on to or accessed
by a device or computer codeby a device or computer code Not aimed at one application in particularNot aimed at one application in particular
![Page 6: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/6.jpg)
66
ScopeScope GoalsGoals
Standardized descriptionsStandardized descriptions Meaningful descriptionsMeaningful descriptions
ElementsElements Description tools: visual decriptors and description Description tools: visual decriptors and description
schemesschemes Description Definition LanguageDescription Definition Language System toolsSystem tools
Figure 1: Scope of MPEG-7 Figure 1: Scope of MPEG-7 Figure 2: MPEG-7 main elementsFigure 2: MPEG-7 main elements
![Page 7: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/7.jpg)
77
Figure 1: Scope of MPEG-7Figure 1: Scope of MPEG-7
Normative part of MPEG-7 standard
![Page 8: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/8.jpg)
88
Figure 2: MPEG-7 main elementsFigure 2: MPEG-7 main elements
![Page 9: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/9.jpg)
99
ApplicationsApplicationsDigital libraries (image catalogue, film)Digital libraries (image catalogue, film)Broadcast media selection (TV Broadcast media selection (TV
channels)channels) Investigation services (human Investigation services (human
characteristics recognition, forensics)characteristics recognition, forensics)Multimedia editing (personalized Multimedia editing (personalized
electronic news service)electronic news service)Figure 3: Abstract RepresentationFigure 3: Abstract Representation
![Page 10: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/10.jpg)
1010
Figure 3: Abstract representation of Figure 3: Abstract representation of possible applications using MPEG-7possible applications using MPEG-7
![Page 11: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/11.jpg)
1111
MethodologyMethodologyStandard Development
Specification for Technology Requirements
Technology Request
Proposal Evaluation
Experimentation Model Definition
Core Experiments
![Page 12: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/12.jpg)
1212
Visual DescriptorsVisual Descriptors
General visual descriptors General visual descriptors Color, texture, shape, and motion featuresColor, texture, shape, and motion features
Domain specificDomain specific Identification of human faces and face Identification of human faces and face
recognitionrecognition
![Page 13: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/13.jpg)
1313
Visual Color Visual Color DescriptorsDescriptors Color Spaces (HSV, HMMD)Color Spaces (HSV, HMMD)
Supports above for normative purposes Supports above for normative purposes Also supports RGB, YCbCr color spacesAlso supports RGB, YCbCr color spaces
Scalable color descriptorScalable color descriptor Figure 4Figure 4 Global color Distribution of Images in color Global color Distribution of Images in color
histograms histograms HSV space, uniformly quantized into 255 bins HSV space, uniformly quantized into 255 bins Haar Transform used to encode histogram Haar Transform used to encode histogram Histogram bin non-uniformly quantized Histogram bin non-uniformly quantized color coefficients or histogram bin values for color coefficients or histogram bin values for
matchingmatching
![Page 14: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/14.jpg)
1414
Visual Color DescriptorsVisual Color Descriptors Dominant color descriptorDominant color descriptor
Global + local spatial color distributionGlobal + local spatial color distribution Colors clustered into a small no of representative colors Colors clustered into a small no of representative colors
representative color, %age, spatial coherency, variance representative color, %age, spatial coherency, variance Color layout descriptorColor layout descriptor
Spatial distribution of color in an arbitrarily shaped regionSpatial distribution of color in an arbitrarily shaped region Color structure descriptorColor structure descriptor
HMMD, local color feature, sliding windowHMMD, local color feature, sliding window Histogram on color appearance countHistogram on color appearance count
Group of Frames/Group of PicturesGroup of Frames/Group of Pictures SCD for a collection of similar images (frames) or video SCD for a collection of similar images (frames) or video
framesframes Average, median, intersection histograms of GoF or GoP Average, median, intersection histograms of GoF or GoP
![Page 15: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/15.jpg)
1515
Figure 4: Three color images and their MPEG-7 histogram color distribution, depicted using a simplified color histogram. Based on the color distribution, the two left images would be recognized as more similar compared to the one on the right.
![Page 16: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/16.jpg)
1616
Visual Texture DescriptorsVisual Texture DescriptorsTexture FeaturesTexture Features
Visual patterns (homogenous or non-Visual patterns (homogenous or non-homogenous)homogenous)
Multiple colors in imagesMultiple colors in imagesMultiple intensities in imagesMultiple intensities in imagesSurface structural information Surface structural information Figure 5Figure 5
![Page 17: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/17.jpg)
1717
Figure 5: Examples of grayscale images with different textures. Using the MPEG-7 Visual texture descriptors, the two images on the bottom would be rated of similar texture, while less similar in texture compared to the two images on the top.
![Page 18: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/18.jpg)
1818
Visual Texture DescriptorsVisual Texture Descriptors Homogenous texture descriptorHomogenous texture descriptor Figure 6Figure 6
Scale and orientation sensitive filtersScale and orientation sensitive filters Mean and SD of frequency coefficients (RT-FT)Mean and SD of frequency coefficients (RT-FT) Scale and rotation-invariant description and matchingScale and rotation-invariant description and matching 2D Gabor functions for filtering feature channels2D Gabor functions for filtering feature channels
Non homogenous texture descriptorNon homogenous texture descriptor (Edge Histogram) (Edge Histogram) Spatial distribution of edgesSpatial distribution of edges Division of image into 16 non overlapping blocks of equal Division of image into 16 non overlapping blocks of equal
sizesize Five edge categories: vertical, horizontal, 45 , 135 , and non Five edge categories: vertical, horizontal, 45 , 135 , and non
directional edge.directional edge. Rotation-sensitive and rotation-invariantRotation-sensitive and rotation-invariant Non uniform quantization using 3 bits, descriptor size of 240 Non uniform quantization using 3 bits, descriptor size of 240
bits (16x5x3)bits (16x5x3)
![Page 19: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/19.jpg)
1919
Figure 6: Frequency layout for MPEG-7 Homogenous Texture Descriptor frequency extraction. Energy and energy deviation values are extracted from this frequency division into 30 channels.
![Page 20: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/20.jpg)
2020
Visual Shape DescriptorsVisual Shape Descriptors
Provides a powerful visual clueProvides a powerful visual clue Invariant to scaling, rotation, and translationInvariant to scaling, rotation, and translation 2-D or 3-D in nature2-D or 3-D in nature
For 2-D there are two categoriesFor 2-D there are two categories Contour based which uses only boundary Contour based which uses only boundary
information of objectsinformation of objects Region-based which the entire shape regionRegion-based which the entire shape region
![Page 21: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/21.jpg)
2121
Visual Shape DescriptorsVisual Shape Descriptors 3-D Shape Descriptor—Shape Spectrum
Based on a shape spectrum concept Histogram of a shape index Measures local convexity of each local 3-D surface Histograms with 100 bins are used—each quantized
by 12 bits. Region Based Shape Descriptor (Art) Figure 7
Uses all pixels constituting a shape within a frame Region-based moments invariant to transformations Coefficients of ART basis functions quantized
![Page 22: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/22.jpg)
2222
Figure 7: Examples of various shapes that can be indexed using MPEG-7 Region-Based Shape Descriptor. Images contained in either of the sets (a)–(d) would be rated similar and dissimilar to the ones in the remaining sets. For example, images in set (a) would be identified being similar and dissimilar to the ones in set (b), (c), or (d).
![Page 23: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/23.jpg)
2323
Contour based shape descriptor Contour based shape descriptor Figure 9Figure 9Curvature scale-space (CCS)Eccentricity and circularity valuesRobust to non-rigid motion partial occlusion of
the shape and perspective transformations2-D/3-D shape descriptor
Representation of 3-D objects using multiple 2-D snapshots
Types of Visual Shape DescriptorsTypes of Visual Shape Descriptors
![Page 24: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/24.jpg)
2424
Figure 8: Examples of shapes that can be indexed using MPEG-7 Contour-Based Shape Descriptor.
![Page 25: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/25.jpg)
2525
Motion Descriptors for VideoMotion Descriptors for Video Motion Activity DescriptorsMotion Activity Descriptors
Activity level and pace of motion in a sceneActivity level and pace of motion in a scene Motion activity intensity descriptorMotion activity intensity descriptor SD of motion vector magnitudeSD of motion vector magnitude SDs quantized into five activity levelsSDs quantized into five activity levels Optional FeaturesOptional Features
motion directionmotion direction spatial distribution of motion activityspatial distribution of motion activity Temporal distribution of motion activityTemporal distribution of motion activity
Camera Motion Descriptor Camera Motion Descriptor Figure 9Figure 9 Global motion parameters in timeGlobal motion parameters in time
zoom activityzoom activity translatory motiontranslatory motion
Motion similarity matching in particular time periodsMotion similarity matching in particular time periods
![Page 26: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/26.jpg)
2626
Figure 9: Camera model for MPEG-7 Camera Motion Descriptor. Perspective projection to image plane p and camera motion parameters. The (virtual) camera is located in O.
![Page 27: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/27.jpg)
2727
Warping Parameters Parametric motion descriptor Object description using 2-D parametric models
translations, rotations, scaling and combination of them planar perspective models quadratic models
Arbitrary objects, defined as regions (group of pixels) in the image over a specified time interval
Global sprite or mosaic Motion Trajectory Motion Trajectory
Description for independently moving objectsDescription for independently moving objects Object displacement over timeObject displacement over time
Motion Descriptors for VideoMotion Descriptors for Video
![Page 28: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/28.jpg)
2828
Identify, filter and browse images using Identify, filter and browse images using visual contentvisual content
Specification to allow interoperability and Specification to allow interoperability and flexibilityflexibility
Other MPEG-7 standardsOther MPEG-7 standardsStorage, access and transmission of Storage, access and transmission of
descriptors and descriptors schemes in descriptors and descriptors schemes in system specificationsystem specification
ConclusionConclusion
![Page 29: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/29.jpg)
2929
![Page 30: The MPEG-7 Visual Standard for Content Description-An Overview Thomas Sikora, Senior Member, IEEE](https://reader035.fdocuments.us/reader035/viewer/2022062816/56815612550346895dc3d2db/html5/thumbnails/30.jpg)
3030
Thank you