Google Cloud Platform: Prototype ->Production-> Planet scale
-
Upload
idan-tohami -
Category
Technology
-
view
389 -
download
1
Transcript of Google Cloud Platform: Prototype ->Production-> Planet scale
![Page 1: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/1.jpg)
Nir ChinskyHead of Google Cloud Platform, MEA
![Page 2: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/2.jpg)
Google Cloud Platform 2
1 Billion Users
![Page 3: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/3.jpg)
![Page 4: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/4.jpg)
Data &Analytics
Infrastructure& Operations
Application Development
![Page 5: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/5.jpg)
Confidential & ProprietaryGoogle Cloud Platform 5
Everything You Need To Build And ScaleComputeFrom virtual machines with proven price/performance advantages to a fully managed app development platform.Compute EngineApp EngineContainer EngineContainer RegistryCloud Functions
Storage and DatabasesScalable, resilient, high performance object storage and databases for your applications.Cloud StorageCloud BigtableCloud DatastoreCloud SQL
NetworkingState-of-the-art software-defined networking products on Google’s private fiber network.Cloud Virtual NetworkCloud Load BalancingCloud CDNCloud InterconnectCloud DNS
Management ToolsMonitoring, logging, and diagnostics and more, all a easy to use web management console or mobile app.Stackdriver OverviewMonitoringLoggingError ReportingDebuggerDeployment Manager & More
Big DataFully managed data warehousing, batch and stream processing, data exploration, Hadoop/Spark, and reliable messaging.BigQueryCloud DataflowCloud DataprocCloud DatalabCloud Pub/SubGenomics
Machine LearningFast, scalable, easy to use ML services. Use our pre-trained models or train custom models on your data.Cloud Machine Learning PlatformVision APISpeech APITranslate API
Developer ToolsDevelop and deploy your applications using our command-line interface and other developer tools.Cloud SDKDeployment ManagerCloud Source RepositoriesCloud EndpointsCloud Tools for Android StudioCloud Tools for IntelliJGoogle Plugin for EclipseCloud Test Lab
Identity & SecurityControl access and visibility to resources running on a platform protected by Google’s security model.Cloud IAMCloud Resource ManagerCloud Security ScannerCloud Platform Security Overview
![Page 6: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/6.jpg)
![Page 7: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/7.jpg)
Hitting the limits, early on...
The Anatomy of a Large-Scale Hypertextual Web Search Engine
1996, Sergey Brin and Lawrence PageComputer Science Department, Stanford University, Stanford, CA 94305
![Page 8: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/8.jpg)
Building on Google’s infrastructure
2.8 million devices
activatedEvery Day (1.1 billion
devices)
10 billionhours
watchedEvery Month (100 hours
of new content uploaded every minute)
43 billionpages
crawledEvery Day
![Page 10: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/10.jpg)
Challenges
11010101010010
1
1011101
011110010001101
Legacy Applications
Regulatory Environment
Changing View on Value of
Data
No Cloud Skills, Hard to
Recruit
Data Silos Everywhere
![Page 11: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/11.jpg)
![Page 12: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/12.jpg)
GOOGLE CONFIDENTIAL
We want to change what you can do.Not just where you can do it.
![Page 13: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/13.jpg)
Select
IaaS
OS Setu
p
Containerization
Storage
Container Orchestrati
on
Networking
Monitoring
Logging
Testing
DevUser
![Page 14: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/14.jpg)
Prototype
Production
Planet-scale
![Page 15: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/15.jpg)
![Page 16: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/16.jpg)
Original Launch Target Estimated Worst Case Actual Traffic
Cloud Datastore Transactions Per Second
50XActual Traffic
5XWorst Case
Estimate
1XTarget Traffic
![Page 17: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/17.jpg)
Google Cloud Platform 17
The Journey to a Web-Scale Cloud
2nd Wave1st Wave 3rd Wave
![Page 18: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/18.jpg)
Open Matters
![Page 19: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/19.jpg)
Walking the Walk
![Page 20: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/20.jpg)
Leading Open Source Communities
#1 Highest Engagement on Github #2Highest
Engagement on Github
Kubernetes Tensorflow
Source: Analyzing GitHub issues and comments with BigQuery
![Page 21: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/21.jpg)
KubernetesApps
![Page 22: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/22.jpg)
![Page 23: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/23.jpg)
![Page 24: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/24.jpg)
600+Security
Engineers
Security and Compliance
700+CVEs Discovered
and/or Fixed
160Academic ResearchPapers on Security
![Page 25: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/25.jpg)
Google’s Data Research
2002
2004
2006
2008
2010
2012
2014
2016
GFS
MapReduce
TensorFlow
BigTable
Dremel
Colossus
Flume
Megastore
Spanner
Millwheel
PubSubF1
![Page 26: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/26.jpg)
Google’s Data Products
2002
2004
2006
2008
2010
2012
2014
2016
ML
PubSub
DataFlow
DataStore
DataFlow
Cloud Storage
BigQuery
BigTable
DataProc
Cloud Storage
![Page 27: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/27.jpg)
Data & Analytics
Insight Data
![Page 28: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/28.jpg)
Programming
Resource provisionin
gPerformance tuning
Monitoring
Reliability
Deployment & configuration
Handling growing
scale
Utilization improvemen
ts
Typical Big Data Jobs
![Page 29: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/29.jpg)
Big Data with GoogleFocus on insights.Not infrastructure.
Programming
Understanding
![Page 30: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/30.jpg)
Dataproc: Fully managed Hadoop and Spark w/ industry-leading performance
BigQuery: Fully managed data warehouse for large-scale analytics
Dataflow: Real-time data pipelines, with open source SDK via Apache Beam
Big Data with Google
30
![Page 31: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/31.jpg)
31
No DBA
NBD(no big deal)
![Page 32: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/32.jpg)
“Right at the start of the partnership we were able to reduce time to insight from 96 hours to 30 minutes by using
BigQuery”
Gary SandersHead of Digital Analytics
![Page 33: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/33.jpg)
Music for Everyone
75M+ Users2B+ Playlists30M+ Songs
Data is the center of the Spotify music experience
With GCP, data teams get big data insights in minutes versus hours
![Page 34: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/34.jpg)
“From traditional batch processing to rock-solid event delivery to the nearly magical abilities of BigQuery, building on Google’s data
infrastructure provides us with a significant advantage where it matters the most.”
Nicholas HarteauVP of Engineering and Infrastructure
![Page 35: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/35.jpg)
![Page 36: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/36.jpg)
Analytics to Machine Learning
![Page 37: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/37.jpg)
Cloud Vision API
Cloud Translate API Cloud
Natural Language API
Cloud Speech API
Ready to Use Machine Learning
Models
Cloud Machine Learning
DEVELOP - MODEL - TEST
Use Your Own Data To Train Models
![Page 38: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/38.jpg)
Hypertune
Cloud Machine Learning
NoOps Machine Learning
Training at Scale
Cloud Datalab UI
![Page 39: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/39.jpg)
Rapidly Accelerating Use of Deep Learning at Google
Number of directories containing model description files
2012 2013 2014 2015
1500
1000
500
0
Used across products:
![Page 41: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/41.jpg)
Auto-organizing photo library
We will group your photos together,
recognize the important people, and
group them in collections
for you to view.
We just released this feature worldwide couple weeks ago.
![Page 42: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/42.jpg)
Navigate your photos by place
We can understand the
objects and things in your
photos
![Page 43: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/43.jpg)
PhotoNet by Google
DeepMind
![Page 44: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/44.jpg)
Confidential & ProprietaryGoogle Cloud Platform 44
Cloud Vision APIInsight from images with our
powerful Cloud Vision API
![Page 45: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/45.jpg)
FacesFaces, facial landmarks, emotions
OCRRead and extract text, with support for > 80 languages
Cloud Vision APICall API from anywhere, with support for embeddable images, and Google Cloud storage
LabelDetect entities from furniture to transportation
LogosIdentify product logos
Landmarks & Image PropertiesDetect landmarks & dominant color of image
Safe SearchDetect explicit content - adult, violent, medical and spoof
![Page 46: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/46.jpg)
API Usage: Detect Objects in an Image
Image Detected
ItemsVision API
Upload ImagesCloud/On premise
Process the response
Call the REST API1 2 3
![Page 47: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/47.jpg)
![Page 49: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/49.jpg)
Digital Transformation = APIs
![Page 50: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/50.jpg)
![Page 51: Google Cloud Platform: Prototype ->Production-> Planet scale](https://reader035.fdocuments.us/reader035/viewer/2022062316/5871643f1a28ab58758b505b/html5/thumbnails/51.jpg)