mith College

Computer Science

Dominique Thiébaut dthiebaut@smith.edu

Of Food and ComputersPresentation for

The Kahn InstituteNov 2, 2018

D. Thiebaut, Computer Science, Smith College

A Two-PartDegustation

https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcQeDV-F2E-QDSY2IpPVFG0o0sFqxDz1FYh3fQsSwmucJGpb34AQG1hUHZeObj0o-u1zDBFYLIgZrL3bdamwisMHNMibJerEAQ&usqp=CAYhttp://lh6.ggpht.com/6f4-GGlXjSPVddlQvIqDQNflex3BmSPIxvz3nXF-rfBqlarz76nLPsJITZacrg3_fEww-drM867dQZb6=s480

D. Thiebaut, Computer Science, Smith College

Examples of ML

https://www.youtube.com/watch?v=-RHG5DFAjp8

Google’s CEO, Sundar Pichai, demonstrated the capability on stage at the Shoreline Amphitheater during Google I/O

D. Thiebaut, Computer Science, Smith College

Examples of ML

https://www.youtube.com/watch?v=-RHG5DFAjp8

Google’s CEO, Sundar Pichai, demonstrated the capability on stage at the Shoreline Amphitheater during Google I/O

D. Thiebaut, Computer Science, Smith College

Examples of MLFully driverless cars are on public roads in Texas. Drive.ai is the second company to remove the safety driver from its autonomous vehicles

https://www.theverge.com/2018/5/17/17365188/drive-ai-driverless-self-driving-car-texas

D. Thiebaut, Computer Science, Smith College

Examples of ML

https://www.youtube.com/watch?v=rVlhMGQgDkY

D. Thiebaut, Computer Science, Smith College

Examples of ML

https://qz.com/877721/the-ai-master-bested-the-worlds-top-go-players-and-then-revealed-itself-as-googles-alphago-in-disguise/

In March 2016, DeepMind's AlphaGo beat Lee Sedol in a five-game match, the first time a computer Go program has beaten a 9-dan professional without handicaps.[5] Final score: 4 games to 1 in favor of AlphaGo.

D. Thiebaut, Computer Science, Smith College

Examples of ML

D. Thiebaut, Computer Science, Smith College

Examples of ML

https://www.youtube.com/watch?v=AmUC4m6w1wo

Researchers at the University of Washington have produced a photorealistic former US President Barack Obama. Artificial intelligence was used to precisely model how Mr Obama moves his mouth when he speaks. Their technique allows them to put any words into their synthetic Barack Obama’s mouth.

D. Thiebaut, Computer Science, Smith College

Neural Networks

D. Thiebaut, Computer Science, Smith College

Neural Nets

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

100

60

20

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

100

60

20

Go

Stop

0.6

0.3

r

y

g

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

r

y

g

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

r

y

g

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

r

y

g

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Train

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Train

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Train

D. Thiebaut, Computer Science, Smith College

From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx

Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Test

D. Thiebaut, Computer Science, Smith College

Model Error

D. Thiebaut, Computer Science, Smith College

Data VisualizationDaniel Smilkov and Shan Carter

Daniel Smilkov and Shan Carter (Google) Fernanda Viégas and Martin Wattenberg (IBM)

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

https://ml4a.github.io/ml4a/looking_inside_neural_nets/

http://techundred.com/how-snapchat-filter-work/

28

28

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

https://think-data.github.io/machine%20learning/python/2016/12/03/simple-neural-network-using-tensorflow.html

D. Thiebaut, Computer Science, Smith College

/ …….Training = Testing

From https://docs.google.com/viewer?url=https%3A%2F%2Fcs.brown.edu%2Fcourses%2Fcs143%2Flectures%2F17.ppt

Training Images Testing Images

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

Recognize bird songs Ask Alexa for the current weather Identify sounds in a video

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

the quick red fox jumped over the lazy brown sleeping dog

1: brown 2: dog 3: fox 4: jumped 5: lazy 6: over 7: quick 8: red 9: sleeping 10: the

brown dog: 1 1 0 0 0 0 0 0 0 0 red lazy: 0 0 0 0 1 0 0 1 0 0

quick over: 0 0 0 0 0 1 1 0 0 0 the sleeping fox: 0 0 1 0 0 0 0 0 1 1

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

1: eggs 2: flour 3: water 4: salt 5: parsley 6: cheese 7: milk 8: spinach 9: tomatoes 10: …

Recipe 1: 1 1 1 0 0 0 0 0 0 0… Recipe 2: 1 0 0 1 1 1 0 0 0 0… Recipe 3: 0 0 0 0 0 1 1 0 0 0… Recipe 4: 0 0 0 0 0 0 1 0 0 0…

D. Thiebaut, Computer Science, Smith College

What can be "Learned?"

egg

s fl

our

wat

er

sal

t p

arsle

y c

hees

e m

ilk

spi

nach

t

omat

oes

…

Recipe 1: 1 1 1 0 0 0 0 0 0 0… Recipe 2: 1 0 0 1 1 1 0 0 0 0… Recipe 3: 0 0 0 0 0 1 1 0 0 0… Recipe 4: 0 0 0 0 0 0 1 0 0 0…

D. Thiebaut, Computer Science, Smith College

#1

Dessert Time!

Photo credits: Zeng Mu

D. Thiebaut, Computer Science, Smith College

ML and Food

D. Thiebaut, Computer Science, Smith College

ML and Food

Computer VisionIEEEACM

D. Thiebaut, Computer Science, Smith College

Watson

D. Thiebaut, Computer Science, Smith College

Watson

• 2011, Jeopardy

• Ken Jennings & Brad Rutter

• About Watson: "It is no longer purely a question answering computing system designed from Q&A pairs but one that can now 'see', 'hear', 'read', 'talk', 'taste', 'interpret', 'learn' and 'recommend'."

D. Thiebaut, Computer Science, Smith College

Watson

• Applied to healthcare, investment management, insurance, diamond purchasing, oncology, and

food! • $1M for hardware, $1M for algorithm

• Currently, size of 3 stacked pizza boxes

D. Thiebaut, Computer Science, Smith College

D. Thiebaut, Computer Science, Smith College

Chef Watson!

• Started 2015, released to public 2016, retired 2018

• Based on the recipe archive of Bon Appétit

• Implements hedonic psychophysics & food pairing

• Web-based app

• Cookbook!

D. Thiebaut, Computer Science, Smith College

Cookbook

D. Thiebaut, Computer Science, Smith College

Instituteof CulinaryEducation

D. Thiebaut, Computer Science, Smith College

Roasted Tomato& Mozzarella Tart

D. Thiebaut, Computer Science, Smith College

In-HouseHigh-Flying Cook

D. Thiebaut, Computer Science, Smith College

https://www.slideshare.net/AndersQuitzauIbm/big-data-in-foods-chef-watson-agrofood-park

D. Thiebaut, Computer Science, Smith College

Watson can invent several dozen recipes that prominently feature prunes; it can satisfy a request for banana biscotti in a Creole or a Basque style; and it makes suggestions that no human would ever make, like adding milk chocolate to a clam linguine or mayonnaise to a Bloody Mary.

Alexandra Kleeman Cooking with IBM The New Yorker, Nov. 2016

https://www.newyorker.com/magazine/2016/11/28/cooking-with-chef-watson-ibms-artificial-intelligence-app

D. Thiebaut, Computer Science, Smith College

1 Million Recipes

#2Photo credits: Zeng Mu

D. Thiebaut, Computer Science, Smith College

http://pic2recipe.csail.mit.edu/

D. Thiebaut, Computer Science, Smith College

pairing of recipes

(instructions + ingredients) with

images

D. Thiebaut, Computer Science, Smith College

DemoUpload image, get recipe

http://pic2recipe.csail.mit.edu/demo

D. Thiebaut, Computer Science, Smith College

Stats

D. Thiebaut, Computer Science, Smith College

Stats12 images/recipe

D. Thiebaut, Computer Science, Smith College

Stats

9 ingredients/recipe

D. Thiebaut, Computer Science, Smith College

Stats

9 ingredients/recipe

34 ingredients!

D. Thiebaut, Computer Science, Smith College

Stats

9 instructions/recipe

D. Thiebaut, Computer Science, Smith College

Stats

9 instructions/recipe

1 instruction!

D. Thiebaut, Computer Science, Smith College

Food Arithmetic

-

-

-

+

+

+

=

=

=

D. Thiebaut, Computer Science, Smith College

And now,for the

piece de resistance

…

#3Photo credits: Zeng Mu

D. Thiebaut, Computer Science, Smith College

A Survey of Dish IdentifyiersComparison of Image Recognition APIs on food imagesRohan Kulkarnibytes.grubhub.com

https://bytes.grubhub.com/https-medium-com-rohan-kulkarni-comparison-of-image-recognition-apis-on-food-images-cddc9105fc33

D. Thiebaut, Computer Science, Smith College

Nancy

D. Thiebaut, Computer Science, Smith College

Nancy

dish, food, cuisine, seafood, asian food, european food, paella, spanish cuisine, thai food, portuguese food

D. Thiebaut, Computer Science, Smith College

Noa

D. Thiebaut, Computer Science, Smith College

Noa dish, cuisine, food, leaf vegetable, vegetarian food, italian food, cookware and bakeware, european food, frittata, meal,

D. Thiebaut, Computer Science, Smith College

Tom

D. Thiebaut, Computer Science, Smith College

Tom

dish, cuisine, food, asian food, recipe, monjayaki, Okonomiyaki Monjayaki Recipe

D. Thiebaut, Computer Science, Smith College

Susie

D. Thiebaut, Computer Science, Smith College

Susie dish, curry, cuisine, food, gravy, stew, vegetarian food, indian cuisine, recipe, soup, taco soup

D. Thiebaut, Computer Science, Smith College

Dominique

D. Thiebaut, Computer Science, Smith College

Dominique dish, escargot, food, vegetarian food, vegetable, finger food, recipe, cuisine, Escargot Vegetarian cuisine

D. Thiebaut, Computer Science, Smith College

Amy

D. Thiebaut, Computer Science, Smith College

Amy

plant, flora, leaf, branch, tree, fruit, cherry, Leaf

D. Thiebaut, Computer Science, Smith College

Susan

D. Thiebaut, Computer Science, Smith College

Susan

vegetable, potato and tomato genus, fruit, food, vegetarian food, tomato, natural foods, nightshade family, Tomato Vegetarian cuisine

La Quinta Inns & Suites

D. Thiebaut, Computer Science, Smith College

Suleiman

D. Thiebaut, Computer Science, Smith College

Suleiman

dish, food, cuisine, vegetarian food, fried food, baked goods, side dish, Vegetarian cuisine Food Side dish

D. Thiebaut, Computer Science, Smith College

Pat

D. Thiebaut, Computer Science, Smith College

Pat

vegetable, potato and tomato genus, tomato, fruit, food, nightshade family, Tomato

D. Thiebaut, Computer Science, Smith College

Roberto

D. Thiebaut, Computer Science, Smith College

Roberto

still life photography, still life, gourd, cucurbita, Still life photography Photography Still life Gourd

D. Thiebaut, Computer Science, Smith College

Roberto(Second

Trial)

D. Thiebaut, Computer Science, Smith College

dish, meat, food, venison, animal source foods, steak, lamb and mutton, cuisine, Lamb and mutton Venison

Roberto(Second

Trial)

D. Thiebaut, Computer Science, Smith College

Lois

D. Thiebaut, Computer Science, Smith College

Lois

"A baked goodie - a fruit crisp I made in the summer, I think blueberry and peach"

D. Thiebaut, Computer Science, Smith College

Lois

chocolate brownie, baking, parkin, recipe, capirotada, stuffing, streusel, american food, bread pudding, chocolate brownie

D. Thiebaut, Computer Science, Smith College

Mona

D. Thiebaut, Computer Science, Smith College

Mona dish, food, meal, cuisine, salad, asian food, breakfast, leaf vegetable, southeast asian food, lunch,

D. Thiebaut, Computer Science, Smith College

Sara

D. Thiebaut, Computer Science, Smith College

Sara

dish, food, cuisine, meal, tableware, breakfast, recipe, vegetarian food, Vegetarian cuisine Breakfast

D. Thiebaut, Computer Science, Smith College

Kris

D. Thiebaut, Computer Science, Smith College

Kris

dish, cuisine, food, vegetarian food, rice, arroz con pollo, recipe, asian food, middle eastern food, breakfast, indian cuisine, full breakfast, american food, Ackee and saltfish

D. Thiebaut, Computer Science, Smith College

To Be

Continued…

D. Thiebaut, Computer Science, Smith College

FIN