Multimedia Information Retrieval on a Very Large Scale · I sensed a scream passing through nature;...

Fabrizio [email protected]

2016-03-04

Multimedia Information Retrievalon a Very Large Scale

Introduction

mailto:[email protected]

2

Overview

IntroductionFabrizio Falchi

1. Visual FeaturesFabrizio Falchi

2. Indexing for Similarity SearchGiuseppe Amato

3. Large Scale CBIR using Standard Text Retrieval EnginesClaudio Gennaro

3

Multimedia Information Retrieval

• The process of

o searching for and finding multimedia documents

• The corresponding research field is concerned with

o building the best possible multimedia search engines.

• The intriguing bit here is that

o the query itself can be a multimedia excerpt.

[“Multimedia Information Retrieval”, Stefan Rüger 2009]

4

Multimedia (adj)

• Of art, education etc.:

using more than one medium of expression or communication

• Of computer applications:

incorporating audio and video, especially interactively

5

Multimedia documents

• Consist of multimedia data

(text, images, audio, video, etc.)

• Are semistructured, i.e., contain

o structured data (e.g., metadata)

o unstructured data (e.g., text, images, audio, video, etc.)

[“Multimedia Information Retrieval”, Stefan Rüger 2009]

6

Multimedia retrieval

• There are basically two options

o Metadata based

• Multimedia documents are described by metadata

• Search is performed on metadata

• Metadata can be generated manually or automatically

o Similarity based (often called content based)

• Mathematical descriptions of media content is generated

• Retrieval is performed by searching for similar mathematical

descriptions

• Automatic medatadata generation

o Obtained leveraging on classification techniques

7

Change of the Search Paradigm

7

• Traditional YES-NO keyword search will not suffice - sortable

domains of data (numbers, strings) are assumed

• New types of data need gradual comparison and/or ranking

based on:

o similarity,

o dissimilarity,

o proximity,

o distance, closeness, etc.

8

Similarity Search

Focus on:

• efficient ways

• to locate user-relevant information in collections of objects,

• the similarity of which is quantified using a

pairwise distance measure

9

Image Similarity Search Problem

9

image database

10

Feature-based Approach

10

image layer

R

B

G

feature layer

11

Library

12

Library Catalogue

13

Catalog card

14

How did/do we search for content?

• Content was in book stored in libraries

• We use(d) card catalog containing metadata

• Metadata can be:

o Structural: data about the containers of data

o Descriptive: about the data content

• Nowadays we usually search in the (text) content

(e.g., web search engines)

15

Google Books

16

Google Books

17

How would you search for the name of this Library?

18

Results

19

Same images, different sizes

20

Guessed text

21

Guessed text

22

What’s that?

23

The Scream, Edvard Munch

• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg

http://upload.wikimedia.org/.../475px-The_Scream.jpg

24



• One of the files of the same picture as


25



• One of the files of the same picture

• Almost the same as


26




• Almost the same

• A picture of the object at National Gallery, Oslo as


27




• Almost the same

• A picture of the object at National Gallery, Oslo

• One of “The Scream” by Edvard Munch as


28




• Almost the same


• One of “The Scream”s by Edvard Munch

• A painting by Edvard Munch as


29




• Almost the same



• A painting by Edvard Munch

• One of “The Scream”s by various artists as


30




• Almost the same




• One of “The Scream”s by various artists

• An expressionist painting as


31




• Almost the same





• An expressionist painting

• A painting as


32




• Almost the same






• A painting

• An hand made object as


33




• Almost the same






• A painting

• An hand made object

• An artificial objectbeing the product of intentional human manufacture


34

Recognition and Semantic

• Represents Valhallveien, above Oslo

35

Painting meaning?

• “I stopped and looked out over the fjord—the sun was setting, and the clouds

turning blood red. I sensed a scream passing through nature; it seemed to me

that I heard the scream. I painted this picture, painted the clouds as actual

blood. The color shrieked. This became The Scream.” (Edvard Munch)

• Reddish sky in the background is the artist's memory of the effects of the

powerful volcaniceruption of Krakatoa

• The imagery of The Scream has been compared to that which an individual

suffering from depersonalization disorder experiences, a feeling of distortion of

the environment and one's self, and also facial pain.

• "Whistler's Mother, Wood's American Gothic, Leonardo da Vinci's Mona

Lisa and Edvard Munch's The Scream have all achieved something that most

paintings—regardless of their art historical importance, beauty, or monetary

value—have not: they communicate a specific meaning almost immediately

to almost every viewer. (Martha Tedeschi)

Wikipedia

36




• Almost the same






• A painting


• An artificial object

being the product of intentional human manufacture

Low-level features

High-level semantic


37




• Almost the same






• A painting


• An artificial object

being the product of intentional human manufacture

Classification

Matching

Recognition


38

Related Research Fields

• Computer Vision

o To understand what is in a visual content the device have to “see”

• Multimedia Information Retrieval

o To retrieve visual content from a huge datasets

(it is not feasible to “see” everything online, even for the computers)

• Data Mining

o How to extract knowledge from the visual content we have

39

Matching

• Dictionary:

o to equal; be equal to

• In some terms, the match have to be exact

• Can often be done using signature

• Examples:

o Copy detection

40

Classification (of the visual content)

Contains:

• Red/orange sky

• 3 Humans

• Road

• Oslo

• 2 coves

• Cathedral

41

Tag

Tags:

• Edvard Munch

• The Scream

• Norway

• Oslo

• Scream

• Art robbery

• Munchmuseet

• Oil

• Tempera

• Pastel

• Cardboard

• Skrik

• …

An index term assigned to a piece of information; a type of meta-

information that captures knowledge about an information resource

42

Flickr Automatic Tagging

https://www.flickr.com/photos/fabriziofalchi/2810872125

https://www.flickr.com/photos/fabriziofalchi/2810872125

44

MIR for Augmented Reality

45

MIR for Tourism

http://www.visitotuscany.it/

To provide tourists with immediate access to information related to

monuments and artworks

http://www.visitotuscany.it/

46

Smart Ticketing

To provide citizens with effective ways to get information related to events

and making the booking process quick and easy

47

Smart shopping/marketing

To provide effective service in supermarkets by rising the efficiency of total

supply chain through quick billing and promotion of products.

Multimedia Information Retrieval on a Very Large Scale · I sensed a scream passing through nature;...

Documents

Transcript of Multimedia Information Retrieval on a Very Large Scale · I sensed a scream passing through nature;...