WordsEye: From Text To Pictures

The very humongous silver sphere is fifty feet above the ground. The silver castle is in the sphere. The castle is 80 feet wide. The ground is black. The sky is partly cloudy.

Why is it hard to create 3D graphics?

The tools are complex

Too much detail

Involves training, artistic skill, and expense

Pictures from Language

No GUI bottlenecks - Just describe it!• Low entry barrier - no special skill or training required

• Give up detailed direct manipulation for speed and economy of expression

– Language expresses constraints

– Bypass rigid, pre-defined paths of expression (dialogs, menus, etc) as defined by GUI

– Objects vs Polygons – draw upon objects in pre-made 3D and 2D libraries

Enable novel applications in education, gaming, online communication, . . .

• Using language is fun and stimulates imagination

Semantics

• 3D scenes provide an intuitive representation of meaning by making explicit the contextual elements implicit in our mental models.

WordsEye Initial Version (with Richard Sproat)

Developed at AT&T Labs• Graphics: Mirai 3D animation system on Windows NT

• Church Tagger, Collins Parser on Linux

• WordNet (http://wordnet.princeton.edu/)

• Viewpoint 3D model library

• NLP (linux) and depiction/graphics (Linux) communicate via sockets

• WordsEye code in Common Lisp

Siggraph paper (August 2001)

New Version (with Richard Sproat)

Rewrote software from scratch

• Linux and CMUCL

• Custom Parser/Tagger

• OpenGL for 3D preview display

• Radiance Renderer

• ImageMagic, Gimp for 2D post-effects

Different subset of functionality

•No verbs/poses yet

Web interface (www.wordseye.com)

• Webserver and multiple backend text-to-scene servers

• Gallery/Forum/E-Cards/PIctureBooks/2D effects

A tiny grey manatee is in the aquarium. It is facing right. The manatee is six inches below the top of the aquarium. The ground is tile. There is a large brick wall behind the aquarium.

Example

A silver head of time is on the grassy ground. The blossom is next to the head. the blossom is in the ground. the green light is three feet above the blossom. the yellow light is 3 feet above the head. The large wasp is behind the blossom. the wasp is facing the head.

Example

The humongous white shiny bear is on the American mountain range. The mountain range is 100 feet tall. The ground is water. The sky is partly cloudy. The airplane is 90 feet in front of the nose of the bear. The airplane is facing right.

Example

A microphone is in front of a clown. The microphone is three feet above the ground. The microphone is facing the clown. A brick wall is behind the clown. The light is on the ground and in front of the clown.

Example

ExampleUsing user-uploaded image

Mary uses the crossbow. She rides the horse by the store. The store is under the large willow. The small allosaurus is in front of the horse. The dinosaur faces Mary. The gigantic teacup is in front of the store. The gigantic mushroom is in the teacup. The castle is to the right of the store.

Exampleoriginal version of Software

Web Interface – preview mode

Web Interface – rendered (raytraced)

WordsEye Overview

Linguistic Analysis• Parsing• Create dependency-tree representation• Anaphora resolution

Interpretation• Add implicit objects, relations• Resolve semantics and references

Depiction• Database of 3D objects, poses, textures• Depiction rules generate graphical constraints• Apply constraints to create scene

Linguistic Analysis

Tag part-of-speech

Parse

Generate semantic representation

•WordNet-like dictionary for nouns

Anaphora resolution

Example: John said that the cat is on the table.

Parse tree for: John said that the cat was on the table.

Said (Verb)

John (Noun)That (Comp)

On (Prep)Cat (Noun)

Table (Noun)

Was (Verb)

Nouns: Hierarchical Dictionary

Living thing

Animal Plant

Cat Dogcat-vp2842

cat-vp2843

dog-vp23283

dog_standing-vp5041

Physical Object

Inanimate Object

. . .

WordNet problems

Inheritance conflates functional and lexical relations• “Terrace” is a “plateau”

• “Spoon’ is a “container”

• “Crossing Guard” is a “traffic cop”

• “Bellybutton” is a “point”

Lack of multiple inheritance between synsets• “Princess” is an aristocrat, but not a female

• "ceramic-ware" is grouped under "utensil" and has "earthenware", etc under it. But there are no dishes, plates, under it because those are categorized elsewhere under "tableware"

Lacks relations other than ISA. Thesaurus vs dictionary.• Snowball “made-of” snow

• Italian “resident-of” Italy

Cluttered with obscure words and word senses• “Spoon” as a type of golf club

Create our own dictionary to address these problems

Semantic Representation for: John said that the blue cat was on the table.

1. Object: “mr-happy” (John)2. Object: “cat-vp39798” (cat)3. Object: “table-vp6204” (table) 4. Action: “say”

:subject <element 1> :direct-object <elements 2,3,5,6> :tense “PAST”

5. Attribute: “blue” :object <element 2>

6. Spatial-Relation “on” :figure <element 2>

:ground <element 3>

Anaphora resolution: The duck is in the sea. It is upside down. The sea is shiny and transparent. The ground is invisible. The apple is 3 inches below the duck. It is in front of the duck. The yellow illuminator is 3 feet above the apple. The cyan illuminator is 6 inches to the left of it. The magenta illuminator is 6 inches to the right of it. It is partly cloudy.

Indexical Reference: Three dogs are on the table. The first dog is blue. The first dog is 5 feet tall. The second dog is red. The third dog is purple.

Interpretation

Interpret semantic representation• Object selection

• Resolve semantic relations/properties based on object types

• Answer Who? What? When? Where? How?

• Disambiguate/normalize relations and actions

• Identify and resolve references to implicit objects

Object Selection: When object is missing or doesn't exist . . .

Text object: “Foo on table” Substitute image: “Fox on table”

Related object: “Robin on table”

Object attribute interpretation (modify versus selection)

Conventional: “American horse” Unconventional: “Richard Sproat horse”

Substance: “Stone horse” Selection: “Chinese house”

Semantic Interpretation of “Of”

Containment: “bowl of cats”

Part: “head of the cow” Property: “height of horse is..”

Grouping: “stack of cats” Substance: “horse of stone” Abstraction: “head of time”

Implicit objects & references

Mary rode by the store. Her motorcycle was was red.red.

• Verb resolution: Identify implicit vehicle

•Functional properties of objects

• Reference

•Motorcycle matches the vehicle

•Her matches with Mary

Implicit Reference: Mary rode by the store. Her motorcycle was red.

Depiction

3D object and image database

Graphical constraints

•Spatial relations

•Attributes

•Posing

•Shape/Topology changes

Depiction process

3D Object Database

2,000+ 3D polygonal objects

Augmented with:

•Spatial tags (top surface, base, cup, push handle, wall, stem, enclosure)

•Skeletons

•Default size, orientation

•Functional properties (vehicle, weapon . . .)

•Placement/attribute conventions

2000+ 3D Objects

10,000 images and textures

B&W drawings Texture Maps Artwork Photographs

3D Objects and Images tagged with semantic info

Spatial tags for 3D object regionsSpatial tags for 3D object regions

Object type (e.g. WordNet synset)Object type (e.g. WordNet synset)

• Is-a

• represents

Object sizeObject size

Object orientation (front, preferred supporting surface -- Object orientation (front, preferred supporting surface -- wall/top)wall/top)

Compound object consituentsCompound object consituents

Other object properties (style, parts, etc.)Other object properties (style, parts, etc.)

Canopy (under, beneath) Top Surface (on, in)

Spatial Tags

Spatial Tags

Base (under, below, on) Cup (in, on)

Spatial Tags

Push Handle (actions) Wall (on, against)

Spatial Tags

Stem (in) Enclosure (in)

Stem in Cup: The daisy is in the test tube.

Enclosure and top surface: The bird is in the bird cage. The bird cage is on the chair.

Spatial Relations

Relative positions• On, under, in, below, off, onto, over, above . . .• Distance

Sub-region positioning• Left, middle, corner,right, center, top, front, back

Orientation• facing (object, left, right, front, back, east, west . . .)

Time-of-day relations

Vertical vs Horizontal “on”, distances, directions: The couch is against the wood wall. The window is on the wall. The window is next to the couch. the door is 2 feet to the right of the window. the man is next to the couch. The animal wall is to the right of the wood wall. The animal wall is in front of the wood wall. The animal wall is facing left. The walls are on the huge floor. The zebra skin coffee table is two feet in front of the couch. The lamp is on the table. The floor is shiny.

Attributes

Size

•height, width, depth

•Aspect ratio (flat, wide, thin . . .)

Surface attributes

•Texture databaseTexture database

•Color, Texture, Opacity, reflectivity

•Applied to objects or textures themselves

•Brightness (for lights)

Attributes: The orange battleship is on the brick cow. The battleship is 3 feet long.

Time of day & cloudiness

Time of day & lighting

Poses (original version only -- not yet implemented in web

version)

Represent actions

Database of 500+ human poses

• Grips

• Usage (specialized/generic)

• Standalone

Merge poses (upper/lower body, hands)

• Gives wide variety by mix’n’match

Dynamic posing/IK

Grip wine_bottle-bc0014 Use bicycle_10-speed-vp8300

Poses

Throw “round object” Run

Poses

Combined poses: Mary rides the bicycle. She plays the trumpet.

The Broadway Boogie Woogie vase is on the Richard Sproat coffee table. The table is in front of the brick wall. The van Gogh picture is on the wall. The Matisse sofa is next to the table. Mary is sitting on the sofa. She is playing the violin. She is wearing a straw hat.

Combined poses

Mary pushes the lawn mower. The lawnmower is 5 feet tall. The cat is 5 feet behind Mary. The cat is 10 feet tall.

Dynamically defined poses using Inverse Kinematics (IK)

Shape Changes (not implemented in web version)

Deformations Deformations

•Facial expressions

•Happy, angry, sad, confused . . . mixtures

•Combined with poses

Topological changesTopological changes

•Slicing

Facial Expressions

Edward runs. He is happy. Edward is shocked.

The rose is in the vase. The vase is on the half dog.

Depiction Process

Given a semantic representation

•Generate graphical constraints

•Handle implicit and conflicting constraints.

•Generate 3d scene from constraints

•Add environment, lights, camera

•Render scene

Example: Generate constraints for kick

• Case1Case1: : No path or recipient; Direct object is largeNo path or recipient; Direct object is largePose: Actor in kick posePosition: Actor directly behind direct objectOrientation: Actor facing direct object

• Case2Case2: : No path or recipient; Direct object is smallNo path or recipient; Direct object is smallPose: Actor in kick posePosition: Direct object above foot

• Case3: Case3: Path and RecipientPath and RecipientPose+relations . . . (some tentative)

Some varieties of kick

Case3: John kicked the ball to the cat on the skateboard

Case1: John kicked the pickup truck

Case2:John kicked the football

Implicit Constraint. The vase is on the nightstand. The lamp is next to the vase.

Figurative & Metaphorical Depiction

• Textualization

• Conventional Icons and emblems

• Literalization

• Characterization

• Personification

• Functionalization

Textualization: The cat is facing the wall.

Conventional Icons: The blue daisy is not in the army boot.

Literalization: Life is a bowl of cherries.

Characterization: The policeman ran by the parking meter

Functionalization: The hippo flies over the church

Future/Ongoing Work

Build/use scenario-based lexical resource• Word knowledge (dictionary)• Frame knowledge

– For verbs and event nouns

– Finer-grained representation of prepositions and spatial relations

• Contextual knowledge – Default verb arguments

– Default constituents and spatial relations in settings/environments

• Decompose actions into poses and spatial relations

• Learn contextual knowledge from corpora

Graphics/output support• Add dynamic posing of characters to depict actions• Handle more complex, natural text

• Handle object parts

• Add more 2D/3D content (including user uploadable 3D objects)

• Physics, animation, sound, and speech

FrameNet – Digital lexical resource http://framenet.icsi.berkeley.edu/

• 947 hierarchically defined frames947 hierarchically defined frames

• 10,000 lexical entries (Verbs, nouns, adjectives)10,000 lexical entries (Verbs, nouns, adjectives)

• Relations between frame (perspective-on, subframe, Relations between frame (perspective-on, subframe, using, …)using, …)

• Annotated sentences for each lexical unitAnnotated sentences for each lexical unit

Lexical Units in “Revenge” Frame

Frame elements for avenge.v

Frame Element Core Type

Degree Core

Depictive Peripheral

Injured_party Extra_thematic

Injury Core

Instrument Core

Manner Peripheral

Offender Peripheral

Place Core

Punishment Peripheral

Purpose Core

Result Extra_thematic

Time Peripheral

Annotations for “avenge.v”

Relations between frames

Frame element mappings between frames

• Core vs Peripheral

• Inheritance

• Renaming (eg. agent -> helper)

Valence patterns for verb “sell” (commerce_sell frame) and two related frames

<LU-2986 "sell.v" Commerce_sell> patterns:<LU-2986 "sell.v" Commerce_sell> patterns: (33 ((Seller Ext) (Goods Obj))) (33 ((Seller Ext) (Goods Obj))) (11 ((Goods Ext))) (11 ((Goods Ext))) (7 ((Seller Ext) (Goods Obj) (Buyer Dep(to)))) (7 ((Seller Ext) (Goods Obj) (Buyer Dep(to)))) (4 ((Seller Ext))) (4 ((Seller Ext))) (2 ((Goods Ext) (Buyer Dep(to)))) (2 ((Goods Ext) (Buyer Dep(to))))

<frame: Commerce_buy> patterns:<frame: Commerce_buy> patterns: (91 ((Buyer Ext) (Goods Obj))) (91 ((Buyer Ext) (Goods Obj))) (27 ((Buyer Ext) (Goods Obj) (Seller Dep(from)))) (27 ((Buyer Ext) (Goods Obj) (Seller Dep(from)))) (11 ((Buyer Ext))) (11 ((Buyer Ext))) (2 ((Buyer Ext) (Goods Obj) (Seller Dep(at)))) (2 ((Buyer Ext) (Goods Obj) (Seller Dep(at)))) (2 ((Buyer Ext) (Seller Dep(from)))) (2 ((Buyer Ext) (Seller Dep(from)))) (2 ((Goods Obj))) (2 ((Goods Obj)))

<frame: Expensiveness> patterns:<frame: Expensiveness> patterns: (17 ((Goods Ext) (Money Dep(NP)))) (17 ((Goods Ext) (Money Dep(NP)))) (8 ((Goods Ext))) (8 ((Goods Ext))) (4 ((Goods Ext) (Money Dep(between)))) (4 ((Goods Ext) (Money Dep(between)))) (4 ((Goods Ext) (Money Dep(from)))) (4 ((Goods Ext) (Money Dep(from)))) (2 ((Goods Ext) (Money Dep(under)))) (2 ((Goods Ext) (Money Dep(under)))) (1 ((Goods Ext) (Money Dep(just)))) (1 ((Goods Ext) (Money Dep(just)))) (1 ((Goods Ext) (Money Dep(NP)) (Seller Dep(from)))) (1 ((Goods Ext) (Money Dep(NP)) (Seller Dep(from))))

Parsing and generating semantic relations using FrameNetNLP> NLP> (interpret-sentence "the boys on the beach said that the fish swam to island”)(interpret-sentence "the boys on the beach said that the fish swam to island”)

Parse:Parse:(S(S (NP (NP (DT "the") (NN2 (NNS "boys"))) (NP (NP (DT "the") (NN2 (NNS "boys"))) (PREPP* (PREPP (IN "on") (NP (DT "the") (NN2 (NN "beach")))))) (PREPP* (PREPP (IN "on") (NP (DT "the") (NN2 (NN "beach")))))) (VP (VP1 (VERB (VBD "said"))) (COMP "that") (VP (VP1 (VERB (VBD "said"))) (COMP "that") (S (NP (DT "the") (NN2 (NN "fish"))) (S (NP (DT "the") (NN2 (NN "fish"))) (VP (VP1 (VERB (VBD "swam"))) (VP (VP1 (VERB (VBD "swam"))) (PREPP* (PREPP (TO "to") (NP (NN2 (NN "island"))))))))) (PREPP* (PREPP (TO "to") (NP (NN2 (NN "island")))))))))

Word Dependency:Word Dependency:((#<noun: "boy" (Plural) ID=18> (:DEP #<prep: "on" ID=19>))((#<noun: "boy" (Plural) ID=18> (:DEP #<prep: "on" ID=19>)) (#<prep: "on" ID=19> (:DEP #<noun: "beach" ID=21>)) (#<prep: "on" ID=19> (:DEP #<noun: "beach" ID=21>)) (#<verb: "said" ID=22> (:SUBJECT #<noun: "boy" (Plural) ID=18>) (#<verb: "said" ID=22> (:SUBJECT #<noun: "boy" (Plural) ID=18>) (:DIRECT-OBJECT #<verb: "swam" ID=26>)) (:DIRECT-OBJECT #<verb: "swam" ID=26>)) (#<verb: "swam" ID=26> (:SUBJECT #<noun: "fish" ID=25>) (#<verb: "swam" ID=26> (:SUBJECT #<noun: "fish" ID=25>) (:DEP #<prep: "to" ID=27>)) (:DEP #<prep: "to" ID=27>)) (#<prep: "to" ID=27> (:DEP #<noun: "island" ID=28>))) (#<prep: "to" ID=27> (:DEP #<noun: "island" ID=28>)))

Frame Dependency:Frame Dependency:((#<relation: CN-SPATIAL-RELATION-ON ID=19>((#<relation: CN-SPATIAL-RELATION-ON ID=19> (:FIGURE #<noun: "boy" (Plural) ID=18>) (:FIGURE #<noun: "boy" (Plural) ID=18>) (:GROUND #<noun: "beach" ID=21>)) (:GROUND #<noun: "beach" ID=21>)) (#<action: "say.v" ID=22> (#<action: "say.v" ID=22> (#<frame-element: "Text" ID=29> #<action: "swim.v" ID=26>) (#<frame-element: "Text" ID=29> #<action: "swim.v" ID=26>) (#<frame-element: "Author" ID=30> #<noun: "boy" (Plural) ID=18>)) (#<frame-element: "Author" ID=30> #<noun: "boy" (Plural) ID=18>)) (#<action: "swim.v" ID=26> (#<action: "swim.v" ID=26> (#<frame-element: "Self_mover" ID=31> #<noun: "fish" ID=25>) (#<frame-element: "Self_mover" ID=31> #<noun: "fish" ID=25>) (#<frame-element: ("Goal") ID=32> #<prep: "to" ID=27>)) (#<frame-element: ("Goal") ID=32> #<prep: "to" ID=27>)) (#<prep: "to" ID=27> (:DEP #<noun: "island" ID=28>))) (#<prep: "to" ID=27> (:DEP #<noun: "island" ID=28>)))

Acquiring contextual knowledge

Where does “eating breakfast” take place?Where does “eating breakfast” take place?

• Inferring the environment in a text-to-scene conversion system.Inferring the environment in a text-to-scene conversion system. K-CAP 2001 K-CAP 2001 Richard SproatRichard Sproat

Default locations and spatial relations (by Gino Miceli) Default locations and spatial relations (by Gino Miceli)

• Project Gutenberg corpus of online English prose (http://www.gutenberg.org/), Project Gutenberg corpus of online English prose (http://www.gutenberg.org/),

• Use seed-object pairs to extract other pairs with equivalent spatial relations Use seed-object pairs to extract other pairs with equivalent spatial relations (e.g. (e.g. cupscups are (typically) on are (typically) on tablestables, while , while booksbooks are on are on desksdesks). ).

• Leverage verb/preposition semantics as well as simple syntactic structure to Leverage verb/preposition semantics as well as simple syntactic structure to identify identify spatial templatesspatial templates based on verb/{preposition,particle} plus based on verb/{preposition,particle} plus intervening modifiers. intervening modifiers.

Pragmatic Ambiguity: The lamp is next to the vase on the nightstand . . .

Syntactic Ambiguity: Prepositional phrase attachment

John looks at the cat on the skateboard.

John draws the man in the moon.

Potential Applications

• Online communications: Electronic postcards, visual chat/IM, social networks

• Gaming, virtual environments

• Storytelling/comic books/art

• Education (ESL, reading, disabled learning, graphics arts)

• Graphics authoring/prototyping tool

• Visual summarization and/or “translation” of text

• Embedded in toys

Storytelling: The stagecoach is in front of the old west hotel. Mary is next to the stagecoach. She plays the guitar. Edward exercises in front of the stagecoach. The large sunflower is to the left of the stagecoach.

Scenes within scenes . . .

Greeting Cards

1st grade homework: The duck sat on a hen; the hen sat on a pig;...

Conclusion

New approach to scene generation

•Low overhead (skill, training . . .)

• Immediacy

•Usable with minimal hardware: text or speech

input device and display screen.

Work is ongoing

•Available as experimental web service

Related Work

• Adorni, Di Manzo, Giunchiglia, 1984

• Put: Clay and Wilhelms, 1996

• PAR: Badler et al., 2000

• CarSim: Dupuy et al., 2000

• SHRDLU: Winograd, 1972

Bloopers – John said the cat is on the table

Bloopers: Mary says the cat is blue.

Bloopers: John wears the axe. He plays the violin.

Bloopers: Happy John holds the red shark

Bloopers: Jack carried the television

Web Interface - Entry Page (www.wordseye.com)

• Registration

• Login

• Learn more

• Example pictures

Web Interface - Public Gallery

Web Interface - Add Comments to Picture

Web Interface - Link Pictures into Stories & Games

The tall granite mountain range is 300 feet wide.The enormous umbrella is on the mountain range.The gray elephant is under the umbrella.The chicken cube is 6 feet to the right of the gray elephant.The cube is 5 feet tall. The cube is on the mountain range.A clown is on the elephant. The large sewing machine is on the cube.A die is on the clown. It is 3 feet tall.