LEMoRe A Lifelogging Engine for Moments Retrieval at NTCIR-12 Lifelog Task

LEMoRe Team

de Oliveira Barra G Cartas Ayala A Bolantildeos M Dimiccoli M Aghaei M Carneacute M Giro-i-Nieto X Radeva P

ContactGabriel de Oliveira Barra gabrieldeoliveiraubedu httpwwwubeducvub

Computer Vision Center

0314

NTCIR-12 Challenge

- 90586 egocentric images obtained from 3 users during 1 month

- 89593 tags

- 6 activity classes- 35 clustered locations

Slide 4

Queriesbull Precision (image-level) ndash ldquoFind the moment(s) when I was getting a key madeldquobull Recall (event-level) ndash ldquoFind the moment(s) in which I was grocery shopping in the supermarketrdquo

0314

Motivation

How to find a needle in a haystackAll-in-one system covering the 3 main blocks of retrieval

1 Parsing

2 Indexing

3 Retrieval

Slide 5

Requirementsbull Fastbull Scalablebull Flexiblebull OS and device independentbull Automated

0314

Motivation Our Proposal

Interactive Retrieval

Slide 4

Visual featuresTemporal browsingTextual information

0314

Methodology Visual features

Slide 7

Visual descriptors

bull Hand-craftedbull HOG ndash Histogram of Oriented Gradientsbull CL ndash Color Layoutbull EH ndash Edge Histogrambull JCD ndash Jaccard Composite Descriptor

bull CNNsbull Layer ldquofc6rdquo extracted from CaffeNet

0314

Methodology Visual indexing

Slide 7

Object detection by CNN

- CaffeNet -gt 1000 object classes

- LSDA -gt 3822 additional tags

- 4308 total (merged) unique tags

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Motivation

How to find a needle in a haystackAll-in-one system covering the 3 main blocks of retrieval

1 Parsing

2 Indexing

3 Retrieval

Slide 5

Requirementsbull Fastbull Scalablebull Flexiblebull OS and device independentbull Automated

0314

Motivation Our Proposal

Interactive Retrieval

Slide 4

Visual featuresTemporal browsingTextual information

0314

Methodology Visual features

Slide 7

Visual descriptors

bull Hand-craftedbull HOG ndash Histogram of Oriented Gradientsbull CL ndash Color Layoutbull EH ndash Edge Histogrambull JCD ndash Jaccard Composite Descriptor

bull CNNsbull Layer ldquofc6rdquo extracted from CaffeNet

0314

Methodology Visual indexing

Slide 7

Object detection by CNN

- CaffeNet -gt 1000 object classes

- LSDA -gt 3822 additional tags

- 4308 total (merged) unique tags

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Motivation Our Proposal

Interactive Retrieval

Slide 4

Visual featuresTemporal browsingTextual information

0314

Methodology Visual features

Slide 7

Visual descriptors

bull Hand-craftedbull HOG ndash Histogram of Oriented Gradientsbull CL ndash Color Layoutbull EH ndash Edge Histogrambull JCD ndash Jaccard Composite Descriptor

bull CNNsbull Layer ldquofc6rdquo extracted from CaffeNet

0314

Methodology Visual indexing

Slide 7

Object detection by CNN

- CaffeNet -gt 1000 object classes

- LSDA -gt 3822 additional tags

- 4308 total (merged) unique tags

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Methodology Visual features

Slide 7

Visual descriptors

bull Hand-craftedbull HOG ndash Histogram of Oriented Gradientsbull CL ndash Color Layoutbull EH ndash Edge Histogrambull JCD ndash Jaccard Composite Descriptor

bull CNNsbull Layer ldquofc6rdquo extracted from CaffeNet

0314

Methodology Visual indexing

Slide 7

Object detection by CNN

- CaffeNet -gt 1000 object classes

- LSDA -gt 3822 additional tags

- 4308 total (merged) unique tags

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Methodology Visual indexing

Slide 7

Object detection by CNN

- CaffeNet -gt 1000 object classes

- LSDA -gt 3822 additional tags

- 4308 total (merged) unique tags

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Methodology Textual features

Slide 7

WordNet is a large lexical database of English where nouns verbs adjectives and adverbs are grouped into sets of cognitive synonyms each expressing a distinct concept

It suggests semantically similar tags from within the 4308 unique tags available in the corpus

ie Domestic cat

Egyptian cat Siamese cat Abyssinian Persian cat Mouser Kitty Tabby Tiger cat Manx

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LEMoRe the query

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LEMoRe textual retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

Visual indexing by a CNN

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LeMoRE Temporal Browsing

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LeMoRE Visual Search

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

LEMoRe the image retrieval

ldquoFind the moments when Irsquom drinking coffee in front of my laptoprdquo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Validation

bull Users (12) ndash 3 runs bull Run1 (experts wo wordnet) bull Run2 (experts with wordnet) bull Run3 (beginners with wordnet)

bull Measuresbull mAPbull rPresicionbull Binary preference etc

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Results (I)

Slide 14

50 100 150 200 250 300Sec o n d s

00

01

02

03

04

05

Mean

NDCG

Run 1Run 2Run 3

00 02 04 06 08 10Rec al l

00

02

04

06

08

10

Inter

polat

ed Pr

ecisi

on

Run 1 Sec 10Run 1 Sec 120Run 1 Sec 300Run 2 Sec 10Run 2 Sec 120Run 2 Sec 300Run 3 Sec 10Run 3 Sec 120Run 3 Sec 300

Mean average precision over time for each run of event-level retrieval

Event-level interpolated precision over recall for all submitted runs on seconds 10120 and 300

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Results (II)

Slide 15

Image-Level and Event-Level results for the total number of images retrieved over all event queries

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Conclusions

Slide 16

bull Egocentric lifelog retrieval tool based on

bull Semantic search

bull Image query-by-sample search

bull Visual relevance-over-time browsing

bull Better performance over event-level vs image-level queries

bull Better semantics improves event retrieval results (difference between first and second runs)

bull Improved semantic and tags (activity places objects) recognition

bull Scalability

bull Privacy issues

Future Work

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Foto del grupo

0314

Live Demo

Slide 13

LEMoRe Live demo

0314

Live Demo

Slide 13

LEMoRe Live demo