D2 Conceration, Vilamoura, April 16th 2008 1

Video & Image Indexing and Retrieval in the Large Scale

Video & Image Indexing and Retrieval in the Large Scale

V T A L A S

Progress

Nozha Boujemaa, Scientific Coordinator

D2 Conceration, Vilamoura, April 16th 2008 2

Video & Image Indexing and Retrieval in the Large Scale

Consortium Instrument: Integrated Project Start-End Date: 1 January 2007 - 31 December 2009 Project co-ordinator: ERCIM Scientific coordinator: INRIA Consortium: 12 partners from 7 countries ERCIM (FR), EADS (FR), CWI (NL), Fraunhofer (GE), INRIA

(FR), ROBOTIKER (ES), Univ. of Sunderland (UK), CERTH-ITI (GR), Codeworks (UK), INA (FR), Belga (BE), IRT (GE) + AFP

D2 Conceration, Vilamoura, April 16th 2008 3

Video & Image Indexing and Retrieval in the Large ScaleObjectives

VITALAS is a use-case driven project that aims at providing a pre-industrial prototype system dedicated to intelligent access services to multimedia professional archives that would provide the consumer with new technological functionalities.

VITALAS will provide advanced solution for indexing, searching and accessing large scale of non previously (or partly) annotated content.

Cross-media (audio/speech, video, image, text) Content enrichment / indexing Interactive retrieval methods (query refinement with log files,

RFB, context adaption) Vitalas system to be deployed by INA, Belga

and AFP (V1 results)

D2 Conceration, Vilamoura, April 16th 2008 4

Video & Image Indexing and Retrieval in the Large ScaleFirst year achievements

User issues - WP1: Description of users’ requirements,

Definition of VITALAS Use Cases, Elaboration and availability of test corpora

Enabling technologies – WP2: Beyond state-of-the-art mono-media content description

techniques (New local visual features [Rebai2007] [Joly2007]; Well-trained models for word- and syllable-based recognition of German data)

New probabilistic similarity search structure, 5 to 50 times faster than SoA techniques on large datasets.

Integrated modules as web services

+

D2 Conceration, Vilamoura, April 16th 2008 5

Video & Image Indexing and Retrieval in the Large ScaleFirst year achievements

Cross-Media indexing – WP3: Integration (web services) of partners’ know-how cross-

media indexing tool (text and visual features; combination using SVM)

Research activities on cross-media concept vocabularies: Concept dictionaries for cross-media indexing has been derived

Cross-Media Retrieval –WP4: Query expressions to represent VITALAS use-cases Search log analysis of Belga image collection to refine

query expression Coordination of and participation in INEX multimedia track

VITALAS Review, Paris, March 14th 2008 6

Vitalas System - V1

Architecture validation Services interfaces Exchange formats Hosting structure (ESB, Orchestrator)

Evaluate & Validate a development and integration methodology of heterogeneous components Origin (Offline part : IRT, UoS, CERTH-ITI, INRIA, EADS Online part : INRIA, INA, EADS, Robotiker) Form (COTS, Open Source) Programming languages (Java, C++, …)

Produce a first complete and running application Currently running, and being improved

“WebLab” open source platform available on the webhttp://weblab-project.org/

VITALAS Review, Paris, March 14th 2008 7

Demonstration use case

• Cross media search : text + image + automatic metadata generation

• Dataset : 10000 images from Belga• Offline part : Indexation Process

– Image normalization and metadata extraction : IRT

– Text concept extraction : UoS

– Image concept extraction : CERTH-ITI

– Text and Annotation Indexation : EADS

– XML Repository : EADS

– Image Indexation : INRIA

VITALAS Review, Paris, March 14th 2008 8

• Online part :– Search engine GUI : Robotiker

– XML Repository : EADS

– Text or annotation retrieval : EADS

– Image Search : INRIA

– Cluster Map Visualisation : INA

– Orchestration and Cross-medis Search : EADS

Demonstration use case

VITALAS Review, Paris, March 14th 2008 9

Exploration Map XFiles

Subtask technical achievements

Interactive exploration from visual similaritiesTask: exploring images with horses

VITALAS Review, Paris, March 14th 2008 10

10

QUERY:

Unwanted results.... Just because ‘KIA’ is part

of the title/caption

Remember:Belga Use-Case => Public Relations Manager of KIA: “… I want to be alerted whenever the KIA logo is used…”

VITALAS Review, Paris, March 14th 2008 11

11

MISSED RESULT

Traditional ‘Jouranlistic annotation no longer meets user requirements’ But extra-, post-editing of more ‘conceptual’ metadata is

expensive/time-intensive…

Remember:Belga Use-Case => Public Relations Manager of KIA: “… I want to be alerted whenever the KIA logo is used…”

VITALAS Review, Paris, March 14th 2008 12

What’s next?

Vitalas System V1+ & V2

V1+: (M18) (image and text) Large scale indexing and retrieval Full integration of Map Visualization tools

V2: (M24) (audio and Video) Integration of Audio modules (speech to text) Integration of Video Mining and Search tools

VITALAS Review, Paris, March 14th 2008 13

VITALAS Review, Paris, March 14th 2008 14

VITALAS Review, Paris, March 14th 2008 15

Logical architecture

SubmitQuery

ESB

ESB

ImageSearch

INRIA::Maestro<<Service>>

TextAndAnnotationSearch

<<Service>>EADS::Lucene

XMLRepository

<<Service>>EADS::WebLabRepo

Robotiker::SearchGUI

<< JSP >>

Tomcat

<<WebServer>>

EADS::FusedSearch

INA:ClusterGUI

<< JNLP >>

VITALAS Review, Paris, March 14th 2008 16

Physical architecture

Text /Annotation

SearchImageSearch

GUI

ESBFused Search

RepositoryImage Server