Search and Hyperlinking Task at MediaEval 2012

Search and Hyperlinking Taskat MediaEval 2012

Maria Eskevich1, Gareth J.F. Jones1, Shu Chen1

Robin Aly2, Roeland Ordelman2Martha Larson3

1 Dublin City University, Dublin, Ireland2 University of Twente, The Netherlands

3 Delft University of Technology, Delft, The Netherlands

Background

Brave New Task

Brave New Task and Crowdsourcing

Brave New Task: Search and Hyperlinking

Previous work:I MediaEval 2011: Rich Speech Retrieval TaskI VideoCLEF 2009: Linking Task

What is Brave New about the Search and Hyperlinking?I Unified scenario for 2 tasks:

Search output results as input for LinkingI Use of crowdsourcing for results assessment:

workers define the relevance of the video segments,the HIT can be run for each new submission andenrich overall results

I Search: ME10WWW dataset − > blip10000

Brave New Task: Search and Hyperlinking

Previous work:I MediaEval 2011: Rich Speech Retrieval TaskI VideoCLEF 2009: Linking Task

What is Brave New about the Search and Hyperlinking?I Unified scenario for 2 tasks:

Search output results as input for LinkingI Use of crowdsourcing for results assessment:

workers define the relevance of the video segments,the HIT can be run for each new submission andenrich overall results

I Search: ME10WWW dataset − > blip10000

Sub-tasks

I Search sub-task:I Known-item search:

I Textual queries onlyI Multimodal queries (text + video clues)

I Required runs:1 submission for 1-best output of each ASR transcript

I Linking sub-taskI Ad-hoc search for videos that can potentially be linked to

the anchor videos (ground truth of the Search subtask)I Required runs:

use of ASR 1-best transcripts as representationI *Additional runs:

use team output of Search subtask as video anchors forLinking sub-task.

Data: blip10000

I ASR Transcript:I LIMSI/Vocapia:

I Confusion networksI LIUM:

I 1-bestI LatticesI Confision networks

I Video clues:I Shot boundaries (TU Berlin)

I Concept-based descriptors based on a list of 589 concepts(University of Oxford)

I Face detection results (INRIA)

Evaluation: Search sub-task

I Mean Reciprocal Rank (MRR):

RANKI Mean Generalized Average Precision (mGAP):

GAP =1

RANK. PENALTY

I Mean Reciprocal Rank (MRR):

RANKI Mean Generalized Average Precision (mGAP):

GAP =1

RANK. PENALTY

I Mean Average Segment Precision (MASP):Ranking + Length of (ir)relevant content

Segment Precision (SP[r ]) at rank r :

Average Segment Precision:

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

rel(sr ) = 1, if relevant content is present,otherwise rel(sr ) = 0

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

ASP =1n.

N∑r=1

SP[r ] · rel(sr )

Relevance Evaluation via Crowdsourcing:Amazon MTurk HIT

Evaluation: Linking sub-task

* Brave New Task issue: Late evaluation results :-(

Relevance Evaluation via CrowdsourcingDetails

I Assessment level:I VideosI Provided segments:

Length varies across runs and participants

I Workers explanation of relevance judgement:I Relevant:

I ”Same” videosI Videos are on the same topicI Same program different topics

I Irrelevant:I Same program different topicsI Different topicsI Different video styles: interview vs presentation

Relevance Evaluation via CrowdsourcingDetails

I Assessment level:I VideosI Provided segments:

Length varies across runs and participants

I Workers explanation of relevance judgement:I Relevant:

I ”Same” videosI Videos are on the same topicI Same program different topics

I Irrelevant:I Same program different topicsI Different topicsI Different video styles: interview vs presentation

Participants

Group Search LinkingCharles University in Prague (CUNI) 5 –

Dublin City University (DCU) 4 6Ghent University (MMLab) 4 4 + 4

INRIA/IRISA – 5University of Twente (UTwente) 3 1

Participants

Group Search LinkingCharles University in Prague (CUNI) 5 –

Dublin City University (DCU) 4 6Ghent University (MMLab) 4 4 + 4

INRIA/IRISA – 5University of Twente (UTwente) 3 1

Search sub-task results

Linking sub-task results

Thank you for your attention!

Buon compleanno Robin!

Participants presentations:I Search sub-task:

I Maria Eskevich, Dublin City UniversityI Petra Galuscakova, Charles University in Prague

I Search and Linking sub-task:I Danish Nadeem, University of Twente

I Linking sub-task:I Camille Guinaudeau, INRIA/IRISAI Shu Chen, Dublin City University

I Search and Linking sub-task:I Tom de Nies, University of Chent

Thank you for your attention!Buon compleanno Robin!

Participants presentations:I Search sub-task:

I Maria Eskevich, Dublin City UniversityI Petra Galuscakova, Charles University in Prague

I Search and Linking sub-task:I Danish Nadeem, University of Twente

I Linking sub-task:I Camille Guinaudeau, INRIA/IRISAI Shu Chen, Dublin City University

I Search and Linking sub-task:I Tom de Nies, University of Chent

Search and Hyperlinking Task at MediaEval 2012

Technology

Transcript of Search and Hyperlinking Task at MediaEval 2012