Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)

1Information and Communication Technologies

Some ideas around PIADA: Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)(Picture Indexing: Affect, Description and Availability)

Diana Santos


PIADA in a nutshell

Context: making sense of images High-level concerns:

Purpose Interaction with text/context Cultural factors Jokes, irony, metaphor, affect, humour

Practical setup: Web archive, image repositories, Wikimedia Expected outcomes

Picture ontologies (as well as integration with other NLP ontologies) Reasoning with images Cross-cultural understanding of image differences


Presentation outlook

A short description of the main ideas of PIADA 5 things not yet solved How to go about it? Data; 3 kinds of reasoning

SINTEF’s LEG (Language Engineering Group) PIADA in a Norwegian context


1. Find images with feelings

Most image descriptions are about “objective” features like who (names), where (places), dates, colours and objects

Asterix sad – no way, and we know there exist these pictures Try Hillary happy and Hillary angry and you get mostly the same

pictures: those of Hillary Clinton together with texts that may report whatever about happiness or sadness of her competitors or fans


2. Find culturally implicit information

Japanese people in the underground (tube): most probably posted by Japanese people, not with captions saying that people are Japanese... Try Google images and give up

Man reading. This can be a good enough caption in an European museological context, but certainly not in an Asian, or African context


3. Find synergy between images and text

why this caption or this illustration? why do they work together? creative use of pictures in text: or: how many of these images of Sócrates are jokes?

from the obvious: a happy baby in a diaper’s advertisement to the provocative or very very subtle humour (was it?)


4. Find the picture’s purpose

Why is this picture here? Reasoning about a multimedia world...

finding the cause for humour and the important details

Vi roser Anne!How my students end up....

http://staff.science.uva.nl/~marx/


5. Help discuss or describe pictures for different purposes

Properties of pictures are often mentioned in some contexts (didactical, antropological, documentary, scientific) to focus on

particular details: see the character behind Jesus, see the tree on the left, note the tool on his hand, look at the back wings, at the dark clouds, at the tumor ...

(police or law courts) to explain why the picture was taken: revolver under the table, near the corpse

(artistic setting): in sunlight, in rainy wather, with a special lins...


In a nutshell, we believe that

there are many aspects of picture description and reasoning that have not received any or enough attention, namely emotions creativity and humour crosscultural differences intertextuality with pictures

a natural language processing angle is the right way to attack them


Real applications

Significantly enhancing image bank providers activity Helping professionals that need images and text

teachers museum staff encyclopedia authors (the Wikipedia community) other multimedia content providers: for games, educational CDs, textbooks advertisers historians and biographers

Common multilingual image search


How to go about it? Data

Image collections to study Image ontologies and folksonomies available “Text and image” collections

Wikipedia Web pages (Internet archive) Special sites and or multimedia products (guided tours)

Image search logs Elicitation collections: sets of stories about image search Game results: eliciting similarities or associations among images


How to go about it? Reasoning 1

If you want to find a picture of a strong healthy man, or of a genius, you probably find an instance of a person that illustrates these qualities (such as Johny Weissmuller or Einstein and look for them)

If you want to find a picture of Asterix angry, you can look for more “objective” descriptions such as Asterix shouting or Asterix beating

If you want to illustrate the property black you probably look for concepts where black is stereotypical, such as coffee...



You need to know the context of the annotation or text to know what should be implicit and what should be probably commented out

searches in .pt for Sócrates are probably about the prime-minister but in .no, after the philosopher, comes the Brazilian football player

the stereotypical image for man or woman is obviously different depending on the gender of the beholder, no matter sexual orientation, and the same for places and cultures

a “typical” restaurant (and its food) varies widely pictures captioned Lisbon (or Dublin, or Helsinki) are most often by

tourists. People who live in Lisbon give the precise names of what they take the picture of, or don’t even bother to specify location



Why are the images chosen? What is the kind of connection? What is their import? Which other associations -- interpictuality -- they bring? Why can they be considered offensive or funny? Do they feel old-fashioned? Do they feel modern?


Language engineering at SINTEF

Question answering Ontologies Geographical reasoning Contrastive studies Information extraction Corpus search

We believe all these pieces will help us to address the image search and indexing issue.


PIADA in a Norwegian context

We want to develop specific knowledge on images in Norwegian The vocabulary of images and image search in Norway Demo collections Study of user behaviour: what do people ask for, what do they want to

see? Picture reasoning ontologies We hope that by cooperating with commercial actors we will do

something useful not only for research purposes


Specific proposal

brukerstyrte innovasjonsprosjekter (BIP)

OR kompetanseprosjekter med brukermedvirkning (KMB) Scanpix Norge as prime contractor SINTEF writes most of the proposal ABM-utvikling is also involved Other Norwegian actors also contacted

Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)

Documents

Transcript of Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)