Semantic Search Technologies for Video Archives

Harald SackInternet Technologies and Systems (ITS) Future Internet Technologies / Semantic TechnologiesHasso-Plattner-Institute for IT Systems Engineering

Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010Bejing, China

Semantic Search Technologies for Video Archives

Jörg Waitelonis, Harald Sack, WWW2010, Raleigh (NC), April 26th 2010

2

How to Find Something in an Audiovisual Archive ?

• To enable computer supported access and content-based search for digitized multimedia data, descriptive metadata have to be provided as text.

Manual Analysis and AnnotationMetadata

Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China

3

How to Find Something on the Web?


4

The Web is Huge....

To be more precise, the WWW is rather huge...•more than 25 x 109 documents in

Search engine indexes (TNL Blog: Google has 24 billion items index, considers MSN search nearest competitor, September 2005)

•Google Web Crawler found more than 1012 documents(The Official Google Blog: We knew the Web was Big....., Juli 25, 2008)

•New Google Search Index Caffeine comprises 100 Million Gigabytes of datai.e. 1017 Byte (SMX Video: Google’s Matt Cutts On Caffeine Launch, June 9, 2010,http://searchengineland.com/smx-video-googles-matt-cutts-on-caffeine-launch-43933)

•And then, there is also the DeepWeb (Darkweb) ...and it is supposed to be up to 500 time larger than the Surface Web(Bergman, 2001)

http://www.tnl.net/blog/2005/09/27/google-has-24-billion-items-index-considers-msn-search-nearest-competitor/




http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html

http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html


5

The Web is Growing...

Multimedia, Real-Time Data, Sensor Data, ....

in 06/2010: 7 TB/day

in 05/2010: •24 h of video upload / minute•2 billion streamed videos per day

in 06/2010: 7 TB/day


6

The ‘Web of Data‘

Semantic Web Technologies

• Interoperable and machine understandabledata semantics

• Based on formal knowledge representations

• Creating a ‘Web of Data‘


7

• State-of-the-Art Video Analysis

• Semantic Video Search and Applications



8

State-of-the-Art Video Analysis I

Structural Video Analysis

video

scenes

shots

subhots

frames

• Identification of semantically consistent substructures (by content)


9

State-of-the-Art Video Analysis I

Structural Video Analysis

shots

• Identification of semantically consistent substructures (by content)• Example: Shot Boundary Detection

• Identification of• Hard Cuts• Drop Outs• Soft Cuts, as e.g., Dissolve, Fade, Cross-Fade, etc.

• Analytical Shot Boundary Detection• Analysis of Luminance/Chrominance Histograms• Analysis of Motion Vectors

• Machine Learning• Classification of Hard/Soft Cuts based on Image Features• Random Trees • Support Vector Machines

color histogram

histogram differences


10

State-of-the-Art Video Analysis IIvideo frame

audio segment

Intelligent Character Recognition (ICR)

• Preprocessing• Script identification• Script filtering• Adaption of script

geometry (Deskew)• Image quality

enhancement

• Optical Character Recognition (OCR)• with standard software

• Postprocessing• Keyterm spotting• Lexical Analysis and

Filtering


11

State-of-the-Art Video Analysis IIIvideo frame

audio segment

Audio Analysis

• Structural Audio Analysis• Identify semantically

coherent substructures• Speech Detection• Music Detection

• Advanced Audio Analysis• Gender Detection• Speaker Detection• Speaker Identification• Genre Detection

• Automated Speech Recognition (ASR)• Keyterm Spotting


12

State-of-the-Art Video Analysis IVvideo frame

audio segment

Genre Analysis

• Clustering by video content

• Supervised machine learning• Predefined genres, as e.g.,

• Lecture with slides• Lecture without slides• Indoor / Outdoor• Day / Night• News / Feature Film /

Commercial / ... • etc.

• Unsupervised machine learning• No predefined genres• Clustering by similarity


13

State-of-the-Art Video Analysis Vvideo frame

audio segment

Advanced Object Analysis

• Object Detection• Object / Background

separation• Robust real-time Face

detection • Viola-Jones• Haar Cascades

• Object Tracking• Recover already detected

object in same/different video

• Track object in scenes

• Object Identification• Machine Learning


14





15

From Analysis to Metadata

Content Based Video Analysis

• Result: Video segments with annotated metadata

• Metadata consist out of low level / high level feature descriptors• Metadata serve as basis for classical information retrieval methods

• Problem:• Metadata are often ambiguous, inconsistent, or incomplete• Only poor retrieval performance

Metadata Extraction


16

From Metadata to Semantic Metadata

Semantic Video Analysis

time

e.g., person xy

location yz

event abc

e.g., bibliographical data,geographical data,encyclopedic data, ..

Metadata Extraction

Entity Recognition/ Mapping


29

history

search term

related resources with properties

Waitelonis, Sack: Augmenting Video Search with Linked Open Data, in Proc. I-Semantics , Graz 2009.


29

CONTENTUS• Next generation

digital multimedia library

• Semantic Multimedia Search Engine


19




Thank you for your Attention!

Semantic Search Technologies for Video Archives

Technology

Transcript of Semantic Search Technologies for Video Archives