Video Information Retrieval

Mark RuzomberkaIST 49711/07/02

Joke

OutlineWhat is Video Information Retrieval (VIR) ?Reasons VIR is necessaryTheoreticalWhere we are todayExamplesProblemsFuture Work Conclusion

What is Video Information Retrieval (VIR) ?

Recognition technologies Image Voice Text transcripts

Document retrieval technologies Topic segmentation Topic matching Text summarization

Presentation Technologies Combine Recognition and retrieval technologies

Result is an integrated application

VIR-Need, or Why do I care?

Consider the task of trying to find a five minute video clip of interest in a library of 1000 hour long tapes.

Consider the “go to the part where” problem

What do people want from IR

D-Lib Magazine’s asks:

“What do People want from Information Retrieval?”

# 8 Multimedia

Specificly, Reasons for Video IR

Reading is slow compared to your potential for understanding information

Humans think in pictures not words Reading is particularly slow on a computer screen Example: Daydreaming while some one is talking Reading a page in a book and not remembering what it was about

VIR makes for quicker human understanding. Palm/Grafitti 25 Hand Writing 35-40 Typing 50-70 Speaking 135-175 Reading 200 Listening 400 - 500 Thinking 500+

•Video IR allows for faster access to information

Theoretical: Think of the “Jetsons mail system”

You “talk” to the computer, Computer intelligently “talks” back to you

Where we are today

Two of Video Information Retrieval System are currently available:

Type One- keyword/text basedType Two- Content based

Type One- keyword/text based

•DVR- basic expansion of image IR, •not as interesting

Type Two- Content based

Video Mail Informedia MSR Video Skimmer

Example: Video Mail University of Cambridge

1994-1996

AT&T 1999

2000-project ended

Video Mail: Medusa network

Medusa multimedia environment at Olivetti Research Ltd. In Cambridge

It takes a modular approach unlike that of a pc or workstation Unified by a common interface to ATM network Devices plug directly into network and include:

Cameras Audio devices Networked frame buffers Processor farms Disk drives

Video Mail: Medusa Network

“The network is the computer” metaphor is used Solves storage and network speed problems Complicates expense problem

How it works-Overview

The Integrated Application

“narrow” by sender,date, time

Video Mail: Video Browser

Content is now being viewedKeywords are flagged

Video Mail: Video BrowserIn the latest version

“thumb-nailed” pictures of key frames replace color coded line of the search keyword

Informedia

The Informedia Digital Video Library Project automatically combines speech, image and natural language understanding to create a full-content searchable digital video library.

Informedia

Informedia: human factor issues

Interaction MotivationEffective usage modes

Commercial compressionVHS quality playback. Terabyte (1,000 gigabytes) of storage 1000 hours of video.

Problems

1.Human understanding2.Spoken document retrieval3.Poor video browsers4.Expensive5.Slow access to data6.Large amounts of data

Microsoft Research (MSR) Video Skimmer

Microsoft Research (MSR) Video Skimmer

Enhanced Browser Controls: Time Compression Pause Removal Textual Indices:

TOC, Notes Visual Indices

Shot Boundary FramesTimeline Markers

Jump Control (Back/Next)

Problem: Poor Content Based Video Browsers

Current VCR model allows for poor navigation “go the the part where they say” problem

Problem: Expensive

Hard drive space expensive Video adds to problem

High bandwidth needs are also expensive

Year Drive Size Drive Cost Per MB/Cost

1956 5 megabytes 50,000.00 10,000.00

1980 26 megabytes 5,000.00 193.00

1985 10 megabytes 710.00 71.00

1989 40 megabytes 1,199.00 36.00

1995 1.2 gigabytes 680.00 68.60

2000 30.0 gigabytes 249.99 0.96

•http://www.littletechshoppe.com/ns1625/winchest.html

Problem: Slow Access to Data

Broadband still not available everywhereAvailability doesn’t mean acceptanceEspecially after dot com crash 2000

Problem: Large Amounts of Data

Current Systems use MPEG2Newer compression technologies

MPEG 4-DIVX -DVD QualityVideo consumes orders of magnitude

more storage than textMPEG 7 is on horizon

Future Work ?

Sky the limit ?Sci-Fi the limit ?

Hard Drive Space, Bandwidth are current limitations.

ConclusionNot yet ready for prime timeStorage and Network Costs decreasingSuccess is in day to day usageSlowly Becoming Mainstream E.x.TivoProblems of “real world tests”

Idiot proof ATM and Medusa aren’t mainstream

Papers Video Mail Retrieval Using Voice: Report on Keyword.. - Jones, Foote, Jones.. (1994) What do people want from Information Retrieval?. Croft, Bruce W. D-Lib Magazine. (1995) Video Skimming for Quick Browsing based on Audio and Image.. - Smith, Kanade (1995) The VISION digital video library (context) - Gauch, Li et al. – (1997) Informedia: News-on-Demand Multimedia Information.. - Hauptmann, Witbrock (1997) M.G. Christel and D.J. Martin, "Information Visualization within a Digital Video Library", J.

Intelligent Info. Systems 11(3), (1998), pp. 235-257 Browsing Digital Video. Li, Gupta, Sanocki et. Al.

Questions?

Joke?

"There are 10 types of people in the world...

those who understand binary and those who don't."