Bertini - Automatic Metadata Extraction in VidiVideo & im3i @EUscreen Mykonos

Automatic Metadata Extraction

Marco BertiniUniversità di Firenze - MICC

www.micc.unifi.it

giovedì 24 giugno 2010

http://www.micc.unifi.it

http://www.micc.unifi.it

The problem

The massive increase in digital audio-visual information poses high demands on advanced storage and search engines for consumers and professional archives.

Video is now a natural form of communication for the Internet and mobile devices.

Video search engines are the product of progress in many technologies: visual and audio analysis, machine learning techniques, as well as visualization and interaction.


Two solutions

www.im3i.euwww.vidivideo.info


http://www.im3i.eu

http://www.im3i.eu

http://www.vidivideo.info

http://www.vidivideo.info

VidiVideo: project overview

The VidiVideo project addressed the challenge of creating a substantially enhanced semantic access to video, implemented in a search engine.

The outcome of the project is an audio-visual search engine, composed of two parts: a automatic annotation part, that runs off-line, where detectors for more than 1000 semantic concepts are collected in a thesaurus to process and automatically annotate the video and an interactive part that provides a video search engine for both technical and non-technical users.


VidiVideo: project resultsThe automatic annotation part of the system performs audio and video segmentation, speech recognition, speaker clustering and semantic concept detection.

The VidiVideo system has achieved the highest performance in the most important object and concept recognition international contests (PASCAL VOC and TRECVID).

The interactive part provides a desktop-based and a web-based search engines. The system permits different query modalities (free text, natural language, graphical composition of concepts using boolean and temporal relations and query by visual example) and visualizations for video retrieval and browsing.


VidiVideo: project partners

Page 1/73

© 2009 EUTV Consortium EUTV Adaptive Channels in Europe

CONFIDENTIAL

EUTV – Adaptive Channels in Europe

Type CP

Call Identifier FP7-SME-2010-1

Submitted 03 December 2009 Name of the co-ordinating person Dr.-Ing. Georgios Ioannidis

E-Mail [email protected]

Fax +49-179-33-2286677 No. Participant Name Type Short Name Country 1 IN2 search interfaces development Ltd SME IN2 UK 2 spring techno GmbH SME SPRING DE 3 VISup Srl SME VISUP IT 4 Hogeschool voor de Kunsten Utrecht RTDP HKU NL 5 University Firenze RTDP UNIFI IT 6 Instituto de Engenharia de Sistemas e

Computadores RTDP INESC-ID PT


IM3I: project overviewIM3I aims to provide the creative media sector with new ways of searching, summarising and visualising large multimedia archives.

IM3I will provide a service-oriented architecture that allow multiple viewpoints upon multimedia data that are available in a repository, and provide better ways to interact and share rich media. This paves the way for a multimedia information management platform which is more flexible, adaptable and customisable than current repository software.

This in turn enables new opportunities for content owners to exploit their digital assets.


IM3I: project resultsDeveloped a set of tools for automatic audio-visual annotation and search

Developed a set of web services to manage, create and orchestrate the indexing services

Developed a set of specialized search and management interfaces

IM3I authoring platform: allows professional users to import and publish repositories of digital media, authoring of web-based environments for the end-users, creation of elaborate workflow patterns and search & retrieval interfaces to allow a diversity of end-user interactions and scenarios


IM3I: project partners


The VidiVideo backend


Video and scene segmentation•Developed a new gradual transition detection algorithm•Uses novel individual criteria that exhibit less sensitivity to local or global motion:

•Color Coherence Change•Macbeth Color Histogram Change•Luminance Center of Gravity Change

•Combines these criteria (and their multi-scale extensions) using a machine learning technique•Advantages:

•Significantly improved performance•Lack of need for any threshold selection

Scene or story unit: collection of temporally consecutive shots which are about the same topic or event•Developed a multimodal scene segmentation based on Scene TransitionGraph

• Significantly improved performance over visual-only STG


Audio analysis in VidiVideo

• Audio segmentation / audio diarization

• Audio events detection (AED)

• Automatic speech recognition (ASR)

• Language identification (LID)


Block diagram of audio processing

61 AE + 10 Sports (testing)

6 Speech

3 Monologue Dialogue

3 Classes (base)

4 New (testing)

1 Telephone Detector -------------- Total 74+10 (testing)

+(-3+4) (change music detectors)

Current Detectors

Feature extraction Feature

Reduction

SVM classification

Video Processing

Audio Data

Audio event detection framework

Audio Segmentatio

n Speech

Music Detector

Telephone detector

Concepts

Non Speech

Reasoning Speech

Narrator, Anchor …

Speaker ID

Music

Low Frequenc

y Audio Processing

Audio + Video

AE


Audio events corpora• Sound effect corpus: 18,700 short files (290 hrs.),

provided by B&G. Intrinsically labelled corpus.

• Selection of subset for training 61 semantic concepts with more examples.

• Extended feature set: MFCCs, ZCR, Brightness / Audio spectrum centroid, Bandwidth / Audio spectrum spread Audio spectrum envelope, Audio spectrum flatness, Pitch, Harmonicity

• Tested on Movies, Documentaries, Broadcast News, and Talk Shows (TS).

• Mean Average Precision=0.459 (6 test concepts)


Machine learning• Learning of many independent binary classification

tasks is computationally expensive

• KDA using Spectral Regression to solve this problem:

• The time complexity scales linearly with respect to number of labels (i.e. concepts)

• Training in just 1.3 hours compared to 30.2 hours using SVM, over 20 times faster! (MAP ~ the same)

• Tested on Pascal VOC 2008 (20 Concepts)

• Best Method in Pascal VOC 2008

• Ranked First in 9 out of 20 concepts


Color Features

Point sampling • Harris-Laplace • Dense sampling

Color Descriptor • SIFT • OpponentSIFT • WSIFT • rgSIFT • Transformed color SIFT

Spatial Pyramid • 1x1 • 2x2 • 1x3


Results

•Good local descriptors: SIFT, OpponentSIFT, rgSIFT/WSIFT, Transformed color SIFT

•Combining these color features gives state-of-the-art performance

•Drawback: computational costs, reduced adopting GPU implementations (codebook creation is 80% of CPU time!) for 17x speed-up

0 20 40 60 80 100 120 140 160 180 200 2200

0.05

0.1

0.15

0.2

0.25

Concept Detection Task Submissions

0 5 10 15 20 250

0.05

0.1

0.15

0.2

0.25

Interactive Search Task Submissions

216 other concept detection methodsMediaMill concept detection method

22 users of other video retrieval systems2 users of MediaMill video search engine

Mea

n (In

ferr

ed) A

vera

ge P

reci

sion

MediaMill Semantic Video Search Engine at TRECVID 2009

Figure 11: Overview of the 2009 TRECVID benchmark tasks in which MediaMill was the best overall performer. Top: concept detectionand bottom: interactive search, all runs ranked according to mean inferred average precision.

4.2 Guiding The User to Results

Our TRECVID Interactive Retrieval experiments focus onhelping users to determine the utility of a given retrievalstrategy, and on guiding them to a correct set of results. Tothis end we investigate the benefit of two strategies withinthe MediaMill Semantic Video Search Engine.

To help users determine the utility of a given retrievalstrategy we introduce Active Zooming. This aids users bothby helping determine that a subset of visible results is notrelevant, and by helping to find a starting point within theselected results. Active Zooming enables the user to quicklyand seamlessly visualize a large set of results from a singlethread at once. This allows users to make blink-of-an-eyedecisions about the contents of a single thread, or, in thecase of many relevant results, to quickly select large batchesof relevant results at once. The user is then able to ei-ther continue browsing the thread, or go back to any otherthread.

To help guide users to correct results we introduce a Rel-evance Feedback strategy based on passive sampling of userbrowsing behavior in order to guide users to more relevantresults. For this, the system continuously monitors user be-havior and uses this information on-demand to generate a

new set of results. It does so by training a support vectormachine model based on positive examples obtained fromthe user, and negative examples obtained by passive moni-toring. By using a pre-computed kernel matrix of inter-shotdistances this can be done interactively. The end result is areranking of the entire collection, which is then available asa thread for visualization.

4.3 Interactive Search Results

We submitted two runs for interactive search. The Sauronrun was performed by a single expert user. The user wasinstructed to use the ForkBrowser with Gabor and Wic-cest [44] similarity threads. The user was allowed to useActive Zooming and Relevance Feedback techniques on de-mand. The Saruman run was performed by another sin-gle expert user. The user was instructed to use the Cross-Browser together with Active Zooming and Relevance Feed-back. The logging data and final obtained results of bothruns are currently being analyzed, and only a limited anal-ysis of results is presented for the moment.

In Figure 10 we show a per-topic overview of interactivevideo retrieval results. The log-analysis indicates that theusers employed a variety of strategies to retrieve results. We

Our results

TRECVid 2009


The IM3I backend


Visual annotation

• Split a video detecting shots and large content changes with very fast algorithm

• Use different annotation strategies and types of detectors:

• low level (color, B/W, motion)

• Haar-based boosted classifiers

• HOG + SVMs

• Bag-of-words

• k-NN + voting

• simple MPEG-7 XML format (full and fragment)


Baseline: typical BoW

Learning

Featureextract.

Hierarch.clustering

visual wordshisto


Fusion schemes

• Early fusion: integrates unimodal features before learning concepts.

• Late fusion: first reduces unim. feat. to separately learned concepts scores, then these scores are integrated to learn concepts.


Early fusion approach

• Hypothesis: MSER isolate semantically relevant information.

• Idea: represent points that have some spatial relation with regions that are inside, outside, just on the border

• Sampling: SIFT-SURF, dense.

Hierarch.clustering


Late fusion approach

• Use SURF/SIFT + MSER

• Use geometric descriptors for MSERs

!1 !2

!"###$%#&'%(!")#*%+,$-#&'-(!")#*%+......$%#&'%(!")#*/+,$-#&'-(!")#*/+#

!"#

Hierarch.clustering

Hierarch.clustering


Test: baseline

• Best: SURF 64 Grid 10 (accuracy, computational cost)

• SURF 64 Grid 5: +7-8% accuracy, +300% time

• the number of points influences accuracy

Method Sampling # points Time Time Avg. accuracy

Max accuracy


Test: early fusion

• Best: EF SURF 64 Grid 10 (accuracy, computational cost)

• EF SURF 64 Borders: many points, accuracy ~ that of Grid 10 but higher computational costs

• EF SURF 64 Grid 10 is worst than SURF 64 Grid 10, but much faster (50% of execution time)

Avg. accuracy

Max accuracy

Method TimeSampling # points Time


Test: late fusion

• weighting 0.6 (best method) and 0.4 (worst method) lead to good results

• best performance: dense sampling + sparse sampling

• best combination: SURF 64 + EF SURF 64 Grid 10 (improved accuracy, modest computational cost increase)

Method 1 Method 2 Accuracy


Conclusions

• Early fusion strategies:

• ~ baseline accuracy

• faster

• Late fusion strategies:

• better accuracy than baseline

• each method corrects some errors made by the other

• fuse keypoints/regions (SURF, fusion of SURF and MSER)

• IM3I users will be able to chose what’s best for them


The users


Video search engine

Our goal is to provide a search engine for videos for both technical and non-technical users.

Provide different interfaces that permit different query modalities: free-text, natural language, graphical composition of concepts using boolean and temporal relations and query by visual example.

In addition, exploit ontologies and their structure to encode semantic relations between concepts permitting, for example, to expand queries to synonyms and concept specializations.


Sirio and Orione

• System features:

• Sirio is a Rich Internet Application (in Adobe Flex) front end.

• Orione is web service search engine

• Support for multiple ontologies and ontology reasoning

• Results are in Media RSS format (queries treated as RSS feeds)

• New search engine able to scale to large number of instances of ontology concepts

• System interface query options:

• ontology exploration using a graph-based view

• compact keyframe-based results presentation / streaming videos

• concept drag&drop facility (to build complex queries)

• natural language query (with Boolean/temporal ops.)

• free text query (for Google-like search)

• Design goals/assumptions:

• semantic content-based retrieval

• efficient web-based interface


Sirio and Orione


Andromeda

• Design goals/assumptions:

• semantic content-based browsing

• efficient web-based interface using RIA


• Query manager as a Rich Internet Application (in Adobe Flex). Connects to web service (search engine)

• Support for multiple ontologies and ontology reasoning

• System interface query options:

• Shows the concepts with more instances in a concept cloud view

• Graph representation of semantic data structure

• Multiple automatic layout algorithms for spatial positioning and manual drag & drop

• Thumbnails view of the instances of each concept

• Access to video metadata and video streaming

• Access to social content related to ontology concepts (Flickr, YouTube, and real time tweets from Twitter)


Andromeda


Pan• Design goals/assumptions:

• complete/correct automatic annotations

• help in training new automatic concept detectors


• Rich Internet Application (in Adobe Flex).

• video streaming using the same system of Sirio and Andromeda

• new backend

• geotagging using Google Maps

• System interface options

• Integrated with web-based search engine and automatic video annotation

• Multiple user profiles: a simple user may change his own annotations, while a super user can import the annotations of other users, e.g. to supervise the annotation process within an organization.


Pan

!!giovedì 24 giugno 2010

Pan


Daphnis• Design goals/assumptions:

• build on image tagging made popular by Flickr and tag clouds

• connect to social web sites

• allow CBIR


• Rich Internet Application (in Adobe Flex).

• Connects to Flickr (and also Facebook, if needed)

• Approximate nearest neighbour search using MPEG-7 descriptors, to scale to large number of images

• System interface options

• users can tag images and retrieve images based on tags, or use tags to filter the results of similarity based retrieval.

• Ongoing work:

• merging with automatic video annotation for automatic tagging

• adoption of mechanisms for tag suggestion, based on recent research work in this field (use content, tags and geolocalization)


Daphnis

!!


Daphnis


Daphnis

!!


Daphnis


IM3I: authoring platformA CMS approach to repository

analysis, authoring and publication


IM3I: authoring platform

Authoring IM3I end-user functionality typically covers 5 distinctive stages:

• Importing an existing repository from RSS and various XML streams

• Extending the associated datamodel

• Editing layout and editing features

• Editing Search and Retrieval interfaces

• Embedding the IM3I end-user interfaces in a (corporate) website


Editing workflow demo

•Step 1: Importing a video-repository

•Step 2: Enhancing the datamodel

•Step 3: Authoring layouts

•Step 4: Publishing the repository


I: Importing a repository

•Importing an existing repository to an internal and flexible datamodel

•Aggregating and harmonizing multiple repositories

•Visualisation of markup and preview of contents

•Flexibly mapping by drag-and-drop


I: Importing a repository

Mapping the contents of video RSS to an IM3I Datamodel


II: Enhancing the Datamodel•Datamodels contain the descriptions of your repository and in this way stipulate what can be shown to- or retrieved by an end-user.

•Datamodels can reference to each other

•Datamodels can be extended overtime by adding elements

•Elements are based on types: media files, URIs, date, string, etc.

•Elements can be shared across datamodels to allow search & retrieval across multiple collections


II: Enhancing the Datamodel

Adding a ‘translation’ element to the datamodel


III: Layout and Functionality

Easy manipulation of layout to a repository by:

•Table metaphor (easy editing of table characteristics)

•Drag and drop graphical elements

•Drag and drop contents of repository in cells

•Easy manipulation of look and feel

•Easy adding editing functionalities to a layout

•Easy preview and markup functionalities



Defining a layout table



Dragging repository contents to layout



Previewing layout


IV: Embedding in website

Easy blend- in of layouts in corporate websites

•By means of plugins for CMSs (e.g. Webmanager, WordPress, Typo3)

•By <embed> </embed>

•Allowing for elaborate workflow patterns in combining multiple layouts


IV: Embedding in website

Original contents Added

Translation Functionality


The super users


Atlante - process manager

• Web application that is used for creation, technical administration and monitoring of IM3I processing pipeline (e.g. automatic annotation process, media transcoding, etc.)

• This web application has multiple user profile:

• managers

• administrators

• Main functions of this application are:

• creation of new type of (distributed) process

• params setting for new type of process

• creation of “Multiprocess” composed by sets of single (distributed) Processes

• starting/pausing/stopping a process

• monitoring running processes


Atlante

!!


Gaia - media manager

• Web application that will be used for a technical administration and monitoring of the database

• Main functions of this application are:

• media management

• configuration of metadata, broadcasters, Annotations types, Concept types and Media types

• media annotations monitoring by technical backend


Gaia

!!giovedì 24 giugno 2010

Gaia

!!


One more thing...


ACM MM 2010 Workshop3rd International Workshop on Automated Information Extraction in Media Production

AIEMPro'10

Organizers:

Dr. Robbie De Sutter

Vlaamse Radio- en Televisieomroep - Medialab

Jean-Pierre Evain

European Broadcasting Union . Union Européenne de Radiotélévision

Dr. Gerald Friedland

ICSI (International Computer Science Institute)

Dr. Alberto Messina

RAI Radiotelevisione Italiana, Centre for Research and Technological Innovation

Dr. Masanori Sano

NHK (Japan Broadcasting Corporation) Science and Technology Research Laboratories


“Sirio” R.I.A. search engine demo


Web-based R.I.A. archive browsing


Bertini - Automatic Metadata Extraction in VidiVideo & im3i @EUscreen Mykonos

Technology

Transcript of Bertini - Automatic Metadata Extraction in VidiVideo & im3i @EUscreen Mykonos