NLPainter

“Text Analysis for picture/movie generation”

David LeoniEduardo Cárdenas

12/01/2012

2

MOTIVATION FOR CHOOSING THE PROJECT:

The purpose of our project is to transform text in images trying that both express the same mining.

• Adding illustrations to text can be of great help to memorize its contents• But searching images that represent the text is a time consuming task• Drawing entirely new images from scratch takes even longer.

3

HOW THE PROBLEM CAN BE SOLVE?

In order to solve this problem we are going to use different techniques like text mining, natural language processing and semantic web:

We obtained a big Image database.

We have image with tags with the things that are inside of them.

We selected the most representative picture in our database that describes a specific object.

4

HOW THE PROBLEM CAN BE SOLVE?

We used some text mining techniques in order to obtain entities, attributes, etc.

We used the PoS of the phrase that we want to convert to image.

We associated the text with the images.

5

DATABASES

The following databases of images was used for our project:

LabelMe images are annotated with the shapes of the objects contained in the scene. labeling was done by unpaid users More than 70,000 shapes where obtained!

Animal Diversity Web •we fetched nearly 10000 pages.•1545 were information about animals.•3500 picture pages of animals (and for each picture page we extracted ~5 pics links) and 5000 were simply the pages about the hierarchy, needed to arrive to the information at the leaves•we fetched mammals,reptiles ,birds, bony fishes, insects, echinoderms, arthropods

6

LabelMe

DATABASES

7

Animal Diversity

DATABASES

8

General Diagram:

Domain(Ani

mals)

Obtain andproce

ssText

Obtain and proces

spictur

es

Create

Ontology

and data

Query according

to the text

Image representing the text

Semanti

c Web

Image

processin

g

Obtain the text

Parse the text (NLP)

EntitiesAttributes

PathActions

Text ready for be

converted to image

9

Specific Diagram (Text):

9

Pictures in

Internet

Pictures in Databases (LabelMe Matlab)

XML to RDF

Retrieve Image

Image processi

ng

10

Specific Diagram (Images):

10

Specific Diagram (Ontology):

Domain(Animals)

Create the ontology

Extract linked data into asemantic repository

Semantic Web

Application

11

11

Technologies and algorithms(Text)

12

Programming Environment:Netbeans

Packages:Stanford Parser

Additional Packages: Image Generator

12

Technologies and algorithms(Image)

13

Programmation Language:MATLAB, Java

Programming Environment:Netbeans

Packages:LabelMeXOM

13

Technologies and algorithms(Ontology)

14

Editor:Protégé 4.1

RDF engine: OWLim Lite

Upper ontology: Wordnet

14

Technologies and algorithms(General project)

15

Programming Environment:NetBeans

RDF engine:OWLIM lite

Packages:XOM

Web server:Apache Tomcat 7.0 JSP

15

Technologies and algorithms(General project)

16

Documentation: Google Wiki

Versioning:SVN

Project Web Page:http://code.google.com/p/nlpainter/

16

How to run the project?

17

17

18

The Story Picturing Engine

A Text-to-Picture Synthesis System for Augmenting Communication

WordsEye

Comparison with other results:

Our Project working!19

19

20

Some Results:

21

Some Results:

22

Some Results:

23

Some Results:•The car and the sky, and the street.•The bike is at left of the car.•A person walking.•a person in the hotel.•the tree and a person.•a person in the water.•a door is to the right of a bed.•a door is to the left of a bed.

Let see it works!!!

24

Conclusions

25

References:• [LM] Bryan C. Russell and Antonio Torralba and Kevin P. Murphy and

William T. Freeman}, Labelme: A database and web-based tool for image annotation, MIT AI Lab Memo, 2005

• [DBP] Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, Sebastian Hellmann: DBpedia – A Crystallization Point for the We of Data. Journal of Web Semantics: Science, Services and Agents on the World Wide Web, Issue 7, Pages 154–165, 2009.

• [TRA] Mihalcea, R., and Tarau, P. 2004. TextRank: Bringing order into texts. In Proc. Conf. Empirical Methods in Natural Language Processing, 404–411

• [CAPS] Ken Xu and James Stewart and Eugene Fiume , Constraint-Based Automatic Placement for Scene Composition, Proc. Graphics Interface, 2002,May, Calgary, Alberta, pp 25--34[ADW] Myers, P., R. Espinosa, C. S. Parr, T. Jones, G. S. Hammond, and T. A. Dewey. 2006. The Animal Diversity Web (online). Accessed November 01, 2011 at http://animaldiversity.org