Google Machine Learning APIs - puppies or muffins?
-
Upload
bret-mcgowen -
Category
Technology
-
view
767 -
download
1
Transcript of Google Machine Learning APIs - puppies or muffins?
![Page 1: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/1.jpg)
Puppies or muffins?Easily leverage machine learning in your apps
Sara Robinson@SRobTweets
Bret McGowen@bretmcg
![Page 2: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/2.jpg)
2@SRobTweets @bretmcg
Who are we?
Developer Advocate, Google Cloud PlatformSara Robinson / @SRobTweets
● New York, NY● Swift fan (Taylor and language)● Harry Potter aficionado
Developer Advocate, Google Cloud PlatformBret McGowen / @bretmcg
● New York, NY● U2 fan (band and plane)● Lord of the Rings aficionado
![Page 3: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/3.jpg)
What we’ll cover
01
02
03
04
05
A (very) brief overview of machine learning
Machine learning at Google
Vision API
Speech API
Natural Language API
![Page 4: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/4.jpg)
01 A (very) brief overview of machine learning
![Page 5: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/5.jpg)
5@SRobTweets @bretmcg
Machine Learning, Then
![Page 6: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/6.jpg)
6@SRobTweets @bretmcg
Machine Learning, Now
![Page 7: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/7.jpg)
7@SRobTweets @bretmcg
![Page 8: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/8.jpg)
02 Machine Learning at Google
![Page 9: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/9.jpg)
![Page 10: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/10.jpg)
![Page 11: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/11.jpg)
![Page 12: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/12.jpg)
![Page 13: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/13.jpg)
Combined vision and translation
![Page 14: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/14.jpg)
![Page 15: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/15.jpg)
15@SRobTweets @bretmcg
Image Search
![Page 16: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/16.jpg)
16@SRobTweets @bretmcg
Google Photos
![Page 17: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/17.jpg)
Google Cloud Platform 17
Doesn't seem THAT hard...
![Page 18: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/18.jpg)
Images: WikimediaSource: https://commons.wikimedia.org/wiki/File:Red_Apple.jpg https://en.wikipedia.org/wiki/Orange_(fruit)#/media/File:Orange-Whole-%26-Split.jpg
![Page 19: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/19.jpg)
Google Cloud Platform 19
What if I'm not a machine learning expert?
![Page 20: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/20.jpg)
20@SRobTweets @bretmcg
The Machine Learning Spectrum
TensorFlow Cloud Machine Learning Machine Learning APIs
BYOML skills
(Friendly Machine Lea
rning)
Pre-packaged ML
![Page 21: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/21.jpg)
02 The Cloud Vision API Complex image detection with a simple REST request
![Page 22: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/22.jpg)
![Page 23: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/23.jpg)
03 Making an API request
![Page 24: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/24.jpg)
Making a request{ "requests":[ { "image": { "content": "base64ImageString"
// Alternatively, you can pass a Google Cloud Storage url here }, "features": [ { "type": "LABEL_DETECTION", "maxResults": 10 }, { "type": "FACE_DETECTION", "maxResults": 10 },
// More feature detection types... ] } ]}
![Page 25: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/25.jpg)
Google Cloud Platform 25
Let’s see some JSON responses
![Page 26: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/26.jpg)
26
{ "labelAnnotations" : [ { "mid" : "\/m\/01wydv", "score" : 0.92442685, "description" : "beignet" }, { "mid" : "\/m\/0270h", "score" : 0.90845567, "description" : "dessert" }, { "mid" : "\/m\/033nb2", "score" : 0.74553984, "description" : "profiterole" }, { "mid" : "\/m\/01dk8s", "score" : 0.71415579, "description" : "powdered sugar" } ] }
Label Detection
26
![Page 27: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/27.jpg)
"landmarkAnnotations": [
{
"mid": "/m/0c7ln",
"description": "Navy Pier",
"score": 36,
"boundingPoly": {
"vertices": [
{
"x": 275,
"y": 102
}, //...
]
},
"locations": [
{
"latLng": {
"latitude": 41.888685,
"longitude": -87.601311
}
}
], //...
},
{
"mid": "/m/01_d4",
"description": "Chicago",
"score": 31,
"boundingPoly": {
"vertices": [
{
"x": 1086,
"y": 346
},
{
"x": 1496,
"y": 346
},
{
"x": 1496,
"y": 832
},
{
"x": 1086,
"y": 832
}
]
},
"locations": [
{
"latLng": {
"latitude": 41.866724,
"longitude": -87.60852
}
}
],
"lat": 41.866724,
"long": -87.60852,
"latText": "41.866724",
"longText": "-87.608520"
},
{
"mid": "/m/06_7k",
"description": "Chicago",
"score": 23,
"boundingPoly": {
"vertices": [
{
"x": 1194,
"y": 344
},
{
"x": 1537,
"y": 344
},
{
"x": 1537,
"y": 899
},
{
"x": 1194,
"y": 899
}
]
},
"locations": [
{
"latLng": {
"latitude": 41.889232,
"longitude": -87.62312299999999
}
}
],
"lat": 41.889232,
"long": -87.62312299999999,
"latText": "41.889232",
"longText": "-87.623123"
}
],
Landmark Detection
27
![Page 28: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/28.jpg)
28
..."itemListElement": [ { "@type": "EntitySearchResult", "result": { "@id": "kg:/m/0c7ln", "name": "Navy Pier", "@type": [ "Thing", "Place", "LandmarksOrHistoricalBuildings", "TouristAttraction" ], ...
"detailedDescription": { "articleBody": "Navy Pier is a 3,300-foot-long pier on the Chicago shoreline of Lake Michigan. It is located in the Streeterville neighborhood of the Near North Side community area.", "url": "http://en.wikipedia.org/wiki/Navy_Pier"
...
Knowledge Graph sidebarGET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}
![Page 29: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/29.jpg)
29
"textAnnotations": [ { "locale": "en", "description": "U.S. COAST GUARD AUXILIARY\n242039\n", "boundingPoly": { "vertices": [ { "x": 429, "y": 307 }, { "x": 1178, "y": 307 }, { "x": 1178, "y": 770 }, { "x": 429, "y": 770 } ] } }, // ... ]
Text Detection
![Page 30: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/30.jpg)
30
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_LIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
"rollAngle" : 2.8030474,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 221.60617,
"y" : 638.263,
"z" : 0.0017568493
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 89,
"y" : 436
},
...
Face Detection
"detectionConfidence" : 0.98838496,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : -1.0822374,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -2.5003448,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"
![Page 31: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/31.jpg)
31
"faceAnnotations" : [
{
"headwearLikelihood" : "VERY_UNLIKELY",
"surpriseLikelihood" : "VERY_UNLIKELY",
rollAngle" : -4.6490049,
"angerLikelihood" : "VERY_UNLIKELY",
"landmarks" : [
{
"type" : "LEFT_EYE",
"position" : {
"x" : 691.97974,
"y" : 373.11096,
"z" : 0.000037421443
}
},
...
],
"boundingPoly" : {
"vertices" : [
{
"x" : 743,
"y" : 449
},
...
Face Detection
"detectionConfidence" : 0.93568963,
"joyLikelihood" : "VERY_LIKELY",
"panAngle" : 4.150538,
"sorrowLikelihood" : "VERY_UNLIKELY",
"tiltAngle" : -19.377356,
"underExposedLikelihood" : "VERY_UNLIKELY",
"blurredLikelihood" : "VERY_UNLIKELY"
![Page 32: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/32.jpg)
32
"labelAnnotations" : [
{
"mid" : "\/m\/01fklc",
"score" : 0.9337945,
"description" : "pink"
},
{
"mid" : "\/m\/09g5pq",
"score" : 0.83878618,
"description" : "people"
},
{
"mid" : "\/m\/017ftj",
"score" : 0.71847415,
"description" : "sunglasses"
},
{
"mid" : "\/m\/019nj4",
"score" : 0.69381392,
"description" : "smile"
}
]
Putting it all together: face + label + landmark "landmarkAnnotations" : [
{
"boundingPoly" : {
"vertices" : [
{
"x" : 153,
"y" : 64
},
...
]
},
"mid" : "\/m\/0c7zy",
"score" : 0.56636304,
"description" : "Petra",
"locations" : [
{
"latLng" : {
"longitude" : 35.449361,
"latitude" : 30.323975
}
}
]
}
]
![Page 33: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/33.jpg)
33
"safeSearchAnnotation" : {
"spoof" : "VERY_UNLIKELY",
"medical" : "VERY_UNLIKELY",
"adult" : "VERY_UNLIKELY",
"violence" : "VERY_UNLIKELY"
}
But wait...is it appropriate?
![Page 34: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/34.jpg)
04 Live Demo
![Page 35: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/35.jpg)
03 The Speech API Speech to text transcription in over 80 languages
![Page 36: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/36.jpg)
36@SRobTweets @bretmcg
What can I do with the Speech API?● Speech to text transcription in over 80 languages
● Supports streaming and non-streaming recognition
● Filters inappropriate content
![Page 37: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/37.jpg)
37@SRobTweets @bretmcg
Translation Response "responses": [{
"results": [{
"alternatives": [
{
"transcript": "how old is the Brooklyn Bridge",
"confidence": 0.987629
}],
"isFinal": true
}
]
}]
![Page 38: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/38.jpg)
38@SRobTweets @bretmcg
Let’s make a recording!1. Make a recording using SoX, a command line utility for audio files2. Base64 encode the recording3. Build our API request in a JSON file4. Send the JSON request to the Speech API
Bash script for this: bit.ly/speech-request-script
![Page 39: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/39.jpg)
04 Cloud Natural Language API Perform sentiment analysis and entity recognition on text
![Page 40: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/40.jpg)
40@SRobTweets @bretmcg
What can I do with the Natural Language API?Three methods:
1. Analyze entities - The Cubs are an MLB team from Chicago
2. Analyze sentiment - I love Chicago
3. Analyze syntax - Michelle Obama is married to Barack Obama
![Page 41: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/41.jpg)
41@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra
![Page 42: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/42.jpg)
42@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra
![Page 43: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/43.jpg)
43@SRobTweets @bretmcg
Analyze Entities
Chicago is
The Wrigley Building, Chicago is
The Union Stockyard, Chicago is
One town that won't let you down
It's my kind of town
-- Frank Sinatra
![Page 44: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/44.jpg)
44@SRobTweets @bretmcg
Analyze Sentiment
Chicago is the best city in the world.
{
“documentSentiment”: {
“polarity”: 1,
“magnitude”: 0.8
}
}
![Page 45: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/45.jpg)
45@SRobTweets @bretmcg
Analyze Syntax
“The Chicago Cubs are an American professional baseball team based in
Chicago, Illinois.”
![Page 46: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/46.jpg)
46@SRobTweets @bretmcg
Analyze Syntax
![Page 47: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/47.jpg)
47@SRobTweets @bretmcg
Let’s see a demo!
Twitter Streaming NL API BigQuery
bit.ly/nl-olympics
![Page 48: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/48.jpg)
48@SRobTweets @bretmcg
APIs we covered● Vision: cloud.google.com/vision
● Speech: cloud.google.com/speech
● Natural Language: cloud.google.com/natural-language
Related APIs:
● Translate: cloud.google.com/translate
● Prediction: cloud.google.com/prediction
● Knowledge Graph API: developers.google.com/knowledge-graph
![Page 49: Google Machine Learning APIs - puppies or muffins?](https://reader033.fdocuments.us/reader033/viewer/2022042520/58f212cc1a28ab1b778b4571/html5/thumbnails/49.jpg)
Thank YouSara Robinson@SRobTweets
Bret McGowen@bretmcg