Post on 12-Jul-2015
Harald SackInternet Technologies and Systems (ITS) Future Internet Technologies / Semantic TechnologiesHasso-Plattner-Institute for IT Systems Engineering
Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010Bejing, China
Semantic Search Technologies for Video Archives
Jörg Waitelonis, Harald Sack, WWW2010, Raleigh (NC), April 26th 2010
2
How to Find Something in an Audiovisual Archive ?
• To enable computer supported access and content-based search for digitized multimedia data, descriptive metadata have to be provided as text.
Manual Analysis and AnnotationMetadata
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
3
How to Find Something on the Web?
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
4
The Web is Huge....
To be more precise, the WWW is rather huge...•more than 25 x 109 documents in
Search engine indexes (TNL Blog: Google has 24 billion items index, considers MSN search nearest competitor, September 2005)
•Google Web Crawler found more than 1012 documents(The Official Google Blog: We knew the Web was Big....., Juli 25, 2008)
•New Google Search Index Caffeine comprises 100 Million Gigabytes of datai.e. 1017 Byte (SMX Video: Google’s Matt Cutts On Caffeine Launch, June 9, 2010,http://searchengineland.com/smx-video-googles-matt-cutts-on-caffeine-launch-43933)
•And then, there is also the DeepWeb (Darkweb) ...and it is supposed to be up to 500 time larger than the Surface Web(Bergman, 2001)
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
5
The Web is Growing...
Multimedia, Real-Time Data, Sensor Data, ....
in 06/2010: 7 TB/day
in 05/2010: •24 h of video upload / minute•2 billion streamed videos per day
in 06/2010: 7 TB/day
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
6
The ‘Web of Data‘
Semantic Web Technologies
• Interoperable and machine understandabledata semantics
• Based on formal knowledge representations
• Creating a ‘Web of Data‘
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
7
• State-of-the-Art Video Analysis
• Semantic Video Search and Applications
Semantic Search Technologies for Video Archives
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
8
State-of-the-Art Video Analysis I
Structural Video Analysis
video
scenes
shots
subhots
frames
• Identification of semantically consistent substructures (by content)
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
9
State-of-the-Art Video Analysis I
Structural Video Analysis
shots
• Identification of semantically consistent substructures (by content)• Example: Shot Boundary Detection
• Identification of• Hard Cuts• Drop Outs• Soft Cuts, as e.g., Dissolve, Fade, Cross-Fade, etc.
• Analytical Shot Boundary Detection• Analysis of Luminance/Chrominance Histograms• Analysis of Motion Vectors
• Machine Learning• Classification of Hard/Soft Cuts based on Image Features• Random Trees • Support Vector Machines
color histogram
histogram differences
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
10
State-of-the-Art Video Analysis IIvideo frame
audio segment
Intelligent Character Recognition (ICR)
• Preprocessing• Script identification• Script filtering• Adaption of script
geometry (Deskew)• Image quality
enhancement
• Optical Character Recognition (OCR)• with standard software
• Postprocessing• Keyterm spotting• Lexical Analysis and
Filtering
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
11
State-of-the-Art Video Analysis IIIvideo frame
audio segment
Audio Analysis
• Structural Audio Analysis• Identify semantically
coherent substructures• Speech Detection• Music Detection
• Advanced Audio Analysis• Gender Detection• Speaker Detection• Speaker Identification• Genre Detection
• Automated Speech Recognition (ASR)• Keyterm Spotting
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
12
State-of-the-Art Video Analysis IVvideo frame
audio segment
Genre Analysis
• Clustering by video content
• Supervised machine learning• Predefined genres, as e.g.,
• Lecture with slides• Lecture without slides• Indoor / Outdoor• Day / Night• News / Feature Film /
Commercial / ... • etc.
• Unsupervised machine learning• No predefined genres• Clustering by similarity
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
13
State-of-the-Art Video Analysis Vvideo frame
audio segment
Advanced Object Analysis
• Object Detection• Object / Background
separation• Robust real-time Face
detection • Viola-Jones• Haar Cascades
• Object Tracking• Recover already detected
object in same/different video
• Track object in scenes
• Object Identification• Machine Learning
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
14
• State-of-the-Art Video Analysis
• Semantic Video Search and Applications
Semantic Search Technologies for Video Archives
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
15
From Analysis to Metadata
Content Based Video Analysis
• Result: Video segments with annotated metadata
• Metadata consist out of low level / high level feature descriptors• Metadata serve as basis for classical information retrieval methods
• Problem:• Metadata are often ambiguous, inconsistent, or incomplete• Only poor retrieval performance
Metadata Extraction
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
16
From Metadata to Semantic Metadata
Semantic Video Analysis
time
e.g., person xy
location yz
event abc
e.g., bibliographical data,geographical data,encyclopedic data, ..
Metadata Extraction
Entity Recognition/ Mapping
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
29
history
search term
related resources with properties
Waitelonis, Sack: Augmenting Video Search with Linked Open Data, in Proc. I-Semantics , Graz 2009.
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
29
CONTENTUS• Next generation
digital multimedia library
• Semantic Multimedia Search Engine
Dr. Harald Sack, Sino-German Forum on “Digital University and High Education” in conjunction with “5. tele-TASK Symposium, Sept. 15-19, 2010 Bejing, China
19
• State-of-the-Art Video Analysis
• Semantic Video Search and Applications
Semantic Search Technologies for Video Archives
Thank you for your Attention!