Voice of the Machine-

Human Neural NetworkGeorgios SpithourakisPhD Candidate, UCL

Part 1Voice of the Machine

Humans Describe What They See

Encoding Decoding

???

Information Processing

Information Processing

Machines Describe What they See

Encoding Decoding

???

Encoding an Image• Convolutional Neural Networks (CNNs)

Encoding an Image• Convolutional Neural Networks (CNNs)

Layer 1 Layer 2

Encoding an Image• Convolutional Neural Networks (CNNs)

Layer 1 Layer 2

Generating Text (decoding) (1)

Nude Descending a Staircase,Duchamp, 1912

Generating Text (decoding) (1)Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind.

We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.

One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,

Duchamp, 1912 X. J. Kennedy (1961)

Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind.

We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.

One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,

Duchamp, 1912 X. J. Kennedy (1961)

Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.

We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.

One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,

Duchamp, 1912 X. J. Kennedy (1961)

Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.

We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __.

One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,

Duchamp, 1912 X. J. Kennedy (1961)

Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.

We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by.

One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,

Duchamp, 1912 X. J. Kennedy (1961)

• Recurrent Neural Networks (RNNs)

Input

Output

Generating Text (decoding) (2)

<START> Toe upon

Toe upon toe

• Recurrent Neural Networks (RNNs)

Input

Output

Generating Text (decoding) (2)

<START> Toe upon

Toe upon toe

• Recurrent Neural Networks (RNNs)

Input

Output

Generating Text (decoding) (2)

<START> Toe upon

Toe upon toe

Aardvark 0.1%

.

.

.

Toe 20%

.

.

.

Zebra 0.2%

Part 2Human Neural Network

Exercise Goals• Humans become a machine• Each group is a neural machine• Each individual is a neuron

• Adaptation• Communicate in words (not numbers)• More flexibility

Encoding an image• Describe what you see• Pieces of an image• Objects in a scene• A story for the scene

Encoding an Image – Pieces• For each piece, choose 3 words that describe its content

BlueUniformEmpty

SeaBlueCalm

SkyCloudSunny

CloudSkyStick

SkyCloudCotton

TightropeManBalancing

CloudsSkyLines

GreyCloudTripod

CityLandscapeRiver

SkyscrapersLandscapeHorizon

CornerBuildingsWindow

PuddleMirrorPavement

RiversideTownPark

BuildingsRoadsBelow

CityBrownishMetal

RiverBrownCity

Encoding an Image – Objects (individual)• Group together pieces to identify up to 5 objects• Describe each with 1 word• A story should start forming

BlueUniformEmpty

SeaBlueCalm

SkyCloudSunny

CloudSkyStick

SkyCloudCotton

TightropeManBalancing

CloudsSkyLines

GreyCloudTripod

CityLandscapeRiver

SkyscrapersLandscapeHorizon

CornerBuildingsWindow

PuddleMirrorPavement

RiversideTownPark

BuildingsRoadsBelow

CityBrownishMetal

RiverBrownCity

Sky

Man

Roof

City

Encoding an Image – Objects (group)• Reach an agreement as a group (up to 5 objects, 1 word each)

Sky

Man Tightrope

Roof

Town

Sky

Someone

Reflection

Town

Sky

Man Tightrope

Roof

City

Encoding an Image – Scene/Story• Decide on story of up to 5 words

Manwalkstightropeabovetown

Sky

Man Tightrope

Roof

Town

Man walks tightrope above town

BlueUniformEmpty

SeaBlueCalm

SkyCloudSunny

CloudSkyStick

SkyCloudCotton

TightropeManBalancing

CloudsSkyLines

GreyCloudTripod

CityLandscapeRiver

SkyscrapersLandscapeHorizon

CornerBuildingsWindow

PuddleMirrorPavement

RiversideTownPark

BuildingsRoadsBelow

CityBrownishMetal

RiverBrownCity

Sky

Man Tightrope

Roof

Town

The Encoded Image

Generating the Poem (Decoding)• React to what we saw• Write a poem word-by-word• Individually propose alternative continuations• Collaboratively select one

Decoding – Choose First Word

I

He

The

High

I 4

He 1

The 0

High 1

• Each person proposes 1 word

• Each person votes (up to 2 votes) for ‘best’ word

• Cannot vote yourself!

• Count votes

• Write highest scoring word to poem

• If tied, repeat voting only between tied words (or flip coin)

POEMI

Decoding – Choose Next Word

really

only

walk

stand

really 3

only 0

walk 0

stand 1

• Each person proposes 1 word

• Each person votes (up to 2 votes) for ‘best’ word

• Cannot vote yourself!

• Count votes

• Write highest scoring word to poem

• If tied, repeat voting only between tied words (or flip coin)

POEMI really

Decoding – Choose Next Word

knowing

walking

trotting

standing

knowing 0

walking 3

trotting 0

standing 1

• Each person proposes 1 word

• Each person votes (up to 2 votes) for ‘best’ word

• Cannot vote yourself!

• Count votes

• Write highest scoring word to poem

• If tied, repeat voting only between tied words (or flip coin)

POEMI really had an easy way of walking

Decoding – Speed it up• One word at a time is too slow for humans…

• Propose whole phrases (as many words as you like)

Decoding – Choose Next Phrase

However tall

I never thought

Looking at a small

Despite

However tall 0

I never thought 1

Looking at a small 3

Despite 1

• Each person proposes 1 phrase

• Each person votes (up to 2 votes) for ‘best’ phrase

• Cannot vote yourself!

• Count votes

• Write highest scoring phrase to poem

• If tied, repeat voting only between tied words (or flip coin)

POEMI really had an easy way of walkingLooking at a small

Part 3Conclusion

Image: Man leads caravan through desert

The camel holds the hand of the poor manHe’ll watch us tread, he’ll watch us fall

The long shadow of a donkey across the cracked sandShabbily clad but standing tall

We each of us must go, all

Surrounded by another expedition, Across an ocean on the sea of sand,

Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.

Image: Man free falls to ground

We gotta hide behind the Beaver lake! I wanna know a better place or where,

Another day a little kiss and take, An angel on the other side of there.

The forest and cliffs standing against the skyA human being in free fall

Sailing through the air like a flyCome to me, the grass, a call.

Image: Crowd watches fishes at aquarium

In a box of Nothingfar from the deep

where waves are thrustingwhere light goes to sleep

Expecting something from an empty zoo, Surrounded by an ocean full of fish,

On the other side of me and you, Beneath the carpet like a jellyfish.

Image: Two men round a campfireA living fire becomes a doubles title.

To stay protected by the sons of men, We stuck together like a semi final,

The one and two and three or four of ten.

Marshmallows at dawnFreshly cut wood burns in the fire

Time slips out a wide yawnThey all sing out in choir

Acknowledgements• Zena Edwards, CV:iD• Daniela Paolucci, Apples and Snakes

• Sebastian Riedel, UCL• Piotr Mirowski, HumanMachine/Deepmind• Mandana Seyfeddinipur, SOAS

• Marjan Ghazvininejad, USC• Generating Topical Poetry, EMNLP 2016

Thank you!