CS257 Modelling Multimedia InformationCS257 Modelling Multimedia Information
LECTURE 5LECTURE 5
IntroductionIntroduction
• Let’s review…– What we’re dealing with (video data)– What kinds of moving images might be stored
in a digital video database– What kinds of queries people might want to
make to a video database Techniques for modelling and querying
video content
Digital Video Data is…Digital Video Data is…
• A sequence of frames, where each frame is an image (typically 25 frames per second)
• May include a soundtrack: Speech (commentary / monologue / dialogue); Music; Sound effects
• May include text, i.e. subtitles / closed captions• Video data has temporal aspects as well as spatial
aspects: the temporal organisation of the moving images conveys meaning in its own right – change the order of scenes change the meaning of a film
• Cinematic techniques (pan, zoom, etc.) and editing effects can also convey information
Moving Images include…Moving Images include…
• Television• Cinema• Meteorological images• Surveillance cameras• Medical images• Biomechanical images• Dance
Diverse range of video database applications, users and information needs
What is Video Content?What is Video Content?
• Generally speaking video content can be said to comprise:– Objects (including people) with propertiesAND– Activities (actions, events) involving 0 or more objects– Organised in space and time
• (This is an attempt at a generic approach, i.e. one that is applicable to all kinds of moving images)
Example queries to film Example queries to film archivesarchives“Transport”
“Meercats”“Portillo”“Ferries departing at night”“Animal rights protesters outside Hillingdon cat
farm, wearing masks”
From Enser and Sandom (2002).
Queries to Video DatabasesQueries to Video Databases
• User may not want to see the whole of a video file (e.g. a whole film), may just want certain parts or intervals (e.g. shots or scenes)
Need to attach metadata to different parts of the video [More on this in Part 1 of this Lecture]
Queries to Video DatabasesQueries to Video Databases
• Users may want to query for a particular event involving particular people, e.g. “find me video with Bill hitting Tom” – why not use a list of keywords [hit, Bill, Tom] for query and to represent film content?
Need more structured descriptions of what’s happening (both for queries and for video metadata), i.e. who is doing what to whom with what and why. [More on this in Lecture 6]
Queries to Video DatabasesQueries to Video Databases
• User may want to specify a temporal sequence of events, e.g. “find me video where this happens then this happens while that happens”
[More on this in Lecture 6]
Overview of LECTURE 5Overview of LECTURE 5
• PART 1: Need to be able to index (attach metadata to) and retrieve parts of video data files: Video Data Models
• PART 2: How developments in the MPEG standards have enabled new ways to access and create digital video
• LAB – Finish previous exercises; bring coursework questions
PART 1:PART 1: Video Data Models Video Data Models
• Must decide how to model (i.e. structure) video data so that metadata can be attached appropriately – must consider potential user information needs
• Video data can be modelled as:– BLOb’s (Binary Large Objects)– Frames– Intervals (discrete, hierarchical, overlapping); – Object-based schemes (cf. MPEG-4)
• NOTE, this does not entail chopping up the video data file
Modelling Video Data as a BLObModelling Video Data as a BLOb
• Metadata may be associated with the video data as a whole
• The kinds of metadata for visual information discussed in Lecture 4 apply equally well to moving images; but note – ideally only metadata that is true for the whole video data file should be associated with a BLOb
Modelling Video Data as a BLObModelling Video Data as a BLOb
Time
Attribute1:
Attribute2:
Attribute3:
AttributeX…
Modelling Video Data as FramesModelling Video Data as Frames
• An exhaustive metadata description of a video data file would include details for each and every frame (remember each frame is a still image)
• However, with 25-30 fps, the cost of this is usually prohibitive and there are few applications where it would be beneficial
Modelling Video Data as FramesModelling Video Data as Frames
Time
Frame: 1
Attribute A:
Attribute B:
Frame: 2
Attribute A:
Attribute B:
Frame: 3
Attribute A:
Attribute B:
Modelling Video Data as IntervalsModelling Video Data as Intervals
• It is more usual to model video data as ‘meaningful’ intervals – where ‘meaningful’ depends on the particular domain and application
• The intervals may be discrete or overlapping• The intervals may be arranged in a hierarchy so
that metadata descriptions can be inherited
Discrete IntervalsDiscrete Intervals
Time
Interval: 1
Start Time / Frame: 1
End Time / Frame: 103
Attribute A:
Attribute B:
Attribute C:
Interval: 2
Start Time / Frame: 104
End Time / Frame: 155
Attribute A:
Attribute B:
Attribute C:
Interval 1 Interval 2 Interval 3 Interval 4
Overlapping IntervalsOverlapping Intervals
Time
Interval 1
Interval 2Interval 3
Interval 4
Interval: 2
Start Time / Frame: 008
End Time / Frame: 035
Attribute A:
Attribute B:
Attribute C:
Interval: 3
Start Time / Frame: 045
End Time / Frame: 086
Attribute A:
Attribute B:
Attribute C:
Hierarchical IntervalsHierarchical Intervals
Time
Interval 1 Interval 2 Interval 3 Interval 4 Interval 5 Int 6Interval 7 Interval 8 Interval 9
Interval 10
Interval: 10
Start Time / Frame: 000
End Time / Frame: 210
Attribute A:
Attribute B:
Attribute C:
Hierarchical IntervalsHierarchical Intervals• One example of video data that can modelled as hierarchical intervals are news broadcasts, made up of separate parts (main news, sports news, weather) and each part has separate stories, and each story is made up of shots and scenes
• Then attach metadata to each shot and scene
From: Corridoni et al (1996), ‘Multi-perspective Navigation of Movies’, Journal of Visual Languages and Computing 7, 445-466. [Available through Library eJournals].
Exercise 5-1Exercise 5-1• Metadata about both audio and video data streams can be
organised by modelling the data as a Binary Large Object (BLOb) or in terms of intervals. Intervals may be modelled as single- or multi-levelled, they may be discrete or overlapping, and they may be organised in a hierarchy. Give an example of a video stream and an example of an audio stream which would be best modelled as:
• (i) A Binary Large Objects (BLOb)• (ii) Overlapping multi-level intervals• (iii) A hierarchy of discrete intervals
• For each of (i), (ii) and (iii) you should discuss the information needs of potential users and give examples of the metadata you would attach to the data streams
Part 2: Developments in MPEG Part 2: Developments in MPEG standards for digital videostandards for digital video
Video Data: In the beginning….• Before the mid-1980s, digitising, storing and displaying still images,
never mind moving ones, required dedicated high power and high cost hardware
• Now video data is commonplace on home PCs; and throughout the TV and film industries
• Moreover, the ways in which video data can be accessed have become increasingly intuitive and flexible. In a large part these developments have been driven by successive MPEG standards
The MPEG standards: some hows and whys….
Moving Pictures Expert GroupMoving Pictures Expert Group• A committee comprising 100s of international
experts• MISSION: “Development of international
standards for compression, decompression, processing and coded representation of moving pictures, audio, and their combination, in order to satisfy a wide variety of applications”
Each standard developed with a specific aim
MPEG-1: why?MPEG-1: why?
Compression: video data at 1.5 Mbits/s• e.g. for storage on a CD-ROM and playback on
a low-spec PC • then for distribution of video data over the
Internet (not necessarily in real time) – progressive rendering
Started 1988; finished 1992(MPEG-1 Audio Layer III MP3)
MPEG-1: how?MPEG-1: how?
Combines visual, audio and timing information into one data stream suitable for digital transmission
Removes redundancy• Inter-frame redundancy (cf. still images)• Intra-frame redundancy – i.e. regions that do not
change (significantly) -> store some frames and interpolate others
Quality issues: fidelity of colour map; pixel resolution; frame resolution
MPEG-2: why?MPEG-2: why?
Television requirements• higher quality• “efficient transmission over error-prone delivery
systems”Set top boxes and DVDsTV and film production
Started 1990; finished 1996
MPEG-2: how?MPEG-2: how?
Digital Storage Media Command and Control: protocols for managing bitstreams
Requirements• Higher bandwidth• Dedicated hardware for coding (far from real
time); professional systems still cost many £K
(?MPEG-3?)
MPEG-4: why?MPEG-4: why?
Next generation mobile communication• Further compression• PLUS… Reuse / Personlisation / Customisation /
Interactivity / Creativity
Requirements– Composition of real and synthetic media– Multiplex and synchronise objects over networks– Interact with audiovisual scene that is generated at the receiver’s
end
(Started 1993; parts finished in 2000/1)
MPEG-4: how?MPEG-4: how?
**FROM FRAMES TO AUDIO-VISUAL OBJECTS**
Coding of audio-visual objects - real or synthetic• Still images (e.g. a fixed background)• Video objects (e.g. talking person); need not be
rectangular frames• Audio objects (e.g. person’s speech)• Text and graphics – inc. VR• Synthetic heads, speech and sound
MPEG-4: how?MPEG-4: how?
Standard scene description (cf. VRML):• Place media objects in coordinate system• Apply transforms to change geometry / acoustics• Form compound media objects• Apply streamed data to objects to modify
attributes• Interactively change user’s viewing and listening
points
MPEG-7MPEG-7
Indexing and retrieval“Multimedia Content Description Interface”“different from previous MPEG standards…
what is represented is not the information itself but information about the information”
i.e. it is a metadata standard.
MPEG-7MPEG-7
Description Definition Language– for low-level visual features; – for low-level audio features;– for ‘semantic content’, e.g. descriptions
of the narrative world depicted by a piece of media
• Started 1997; official standard in 2001• MPEG-7 cameras developed to capture
some features at time of recording
EXERCISE 5-2EXERCISE 5-2• Consider the following people who are each developing a different multimedia
system to handle digital video data. Which of the MPEG standards would be of interest to each person and why?
– Annabel: she has collected a digital library of historical documentaries which she would like people to be able to access via a high-speed intranet by making a range of queries about semantic content
– Brian: he is distributing a video browser with his home-made cartoons on a CD-ROM
– Claire: she is setting up a pay-per-view digital television service– Dave: he is an experimental filmmaker who is developing a system to help him edit
video material in new and interesting ways
• It can be argued that the format in which multimedia data is coded has an impact on the potential functionality of systems that will store, retrieve and manipulate that data. Discuss, with explanations, the impact on potential system functionality that you think could arise from the following choices of coding format:
– Coding a moving image in MPEG-1 or coding it in MPEG-4– Coding a piece of music in ‘.wav’ or coding it in ‘.midi’
LECTURE 5:LECTURE 5:LEARNING OUTCOMESLEARNING OUTCOMES
• After the lecture, you should be able to:– Select an appropriate video data model given
a specific kind of video data and some user information needs, and explain / justify your decision
– Describe and explain how each MPEG standard enabled new ways to access and create digital video
OPTIONAL READINGOPTIONAL READING
• For information about all the MPEG standards, see: www.chiariglione.org/mpeg/index.htm
• For an overview of MPEG-7: www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm
OPTIONAL READINGOPTIONAL READING
• Benitez et al. (2002), “Description of a Single Multimedia Document”, in: Manjunath, Salembier and Sikora (eds.), Introduction to MPEG-7: multimedia content description interface.
There are 4 copies of this article in the Library Article Collection.
• Note, focus on pages 113-4 and 124-138, and particularly on Figs. 8.9, 8.11, 8.14 and 8.16.
OPTIONAL READINGOPTIONAL READINGMultimedia Standards in Television / Media Production• Issues of multimedia standards are discussed by a
number of other professional groups, including those interested in mxf (multimedia exchange format) and aaf (advanced authoring format).
For a short article about mxf and aaf, see: www.snellwilcox.com/knowledgecenter/mxf_aaf.html
Homepages;mxf: www.pro-mpeg.org/aaf: www.aafassociation.org/
OPTIONAL READINGOPTIONAL READING• The following articles are available for download via the University
Library’s e-journals section. Many of the authors are from the MPEG-7 committee but the papers are more concise than the official MPEG technical documents.
Martinez, J.M., Koenen, R., and Pereira, F. (2002), “MPEG-7: the generic multimedia content description standard, part 1.” IEEE Multimedia 9 (2), April-June 2002, 78–87
Martinez, J.M. (2002), “Standards - MPEG-7 overview of MPEG-7 description tools, part 2. ” IEEE Multimedia 9 (3), July-Sept. 2002, 83 -93.
Chang, S.-F., Sikora, T. and Purl (2001), “An Overview of the MPEG-7 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 11 (6), 688-695.
Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 1.” IEEE Multimedia 6 (3), July-Sept. 1999, 65 –77.
Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 2.” IEEE Multimedia 6 (4), Oct.-Dec. 1999 , 64 –73
Today’s LabToday’s Lab
• Continue with image retrieval
Top Related