Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes
-
Upload
coreos -
Category
Technology
-
view
77 -
download
2
Transcript of Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes
![Page 1: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/1.jpg)
Mailiao Refinery, Taiwan May 31, 2016
Troy Toman @troytoman
Managing 6PB of Earth Imagery with Kubernetes
![Page 2: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/2.jpg)
Dongtaijinai’er Salt Lake, China MAY 3, 2016
You think you have cloud problems? Let me show you cloud problems
![Page 3: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/3.jpg)
Uluru, Australia, DEC 2, 2015
and some Kubernetes
SPACE!
![Page 4: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/4.jpg)
AGENDA
• Planet’s Mission 1• Where we started• Why Kubernetes• Where we are going
![Page 5: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/5.jpg)
To image the whole world every day, making change visible, accessible and actionable.
![Page 6: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/6.jpg)
CAPE TOWN, SOUTH AFRICAMARCH 9, 2016
MONITORING PORT ACTIVITY
![Page 7: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/7.jpg)
CAPE TOWN, SOUTH AFRICAMARCH 14, 2016
MONITORING PORT ACTIVITY
![Page 8: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/8.jpg)
CAPE TOWN, SOUTH AFRICAMARCH 15, 2016
MONITORING PORT ACTIVITY
![Page 9: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/9.jpg)
SAN GABRIEL MOUNTAINSJUNE 4, 2016
![Page 10: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/10.jpg)
SAN GABRIEL MOUNTAINSJUNE 4, 2016
SAN GABRIEL WILDFIRES
VEGETATION INDEX MAP
VEGETATION
bare sparse densemoderate
![Page 11: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/11.jpg)
SAN GABRIEL WILDFIRES
FIRE RETARDENT LINES ARE IN RED
JUNE 22, 2016SAN GABRIEL MOUNTAINS
![Page 12: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/12.jpg)
JUNE 22, 2016SAN GABRIEL MOUNTAINS
FALSE COLOR COMOPOSITE
REVEALS BURN SCARS
![Page 13: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/13.jpg)
Muir Woods & Mt. Tamalpais, California, USA DEC 23, 2015
OUR APPROACH
![Page 14: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/14.jpg)
HISTORICAL APPROACH • Tasking• Low coverage
• Weeks to gain access
• Expensive
PLANET APPROACH • Monitoring• Global coverage• Daily online delivery
• Affordable
![Page 15: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/15.jpg)
13 Builds in only 3 years
BUILD 1 APR 2012
BUILD 6 APR 2013
BUILD 13 JUN 2015
AGILE AEROSPACE
![Page 16: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/16.jpg)
12+SATELLITES
3 - 5M RESOLUTION
4-BAND IMAGERY
10 GROUND STATIONS
4SITES
10,000IMAGES PER DAY
APIFOR INTERNAL DATA
PIPELINE
100SOF VIRTUAL MACHINES
TBsof STORAGE
The early days …
![Page 17: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/17.jpg)
Early approach
• One service team• 2 basic services
• Storage• Jobs
• Everything packaged as debs• Deployed on VMs w/Ansible• Everything tested/deployed as one
![Page 18: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/18.jpg)
Planet’s Evolving Infrastructure
JobsWorker Spot
Instances
Imagery Storage
Storage Elastic Search
Planet API v0
![Page 19: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/19.jpg)
100+SATELLITES
3 - 5M RESOLUTION
4-BAND IMAGERY
26 GROUND STATIONS
9SITES
10+TBPER DAY
EVERY DAY
APIFOR INTERNAL DATA
PIPELINE AND PLATFORM ACCESS,
CHANGE DETECTION, BASE MAPS,
MOSAICS, and MORE
10,000sOF VIRTUAL MACHINES
PBsof STORAGE
The very near future …
![Page 20: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/20.jpg)
PLATFORMAUTOMATED DATA
PIPELINE
DATA AND FUNCTIONAL EXPLOSION!
MOSAIC & IMAGE
PROCESSING
ANALYTICS CONTAINER
API – RESTFUL & JSON
PlanetScope~150 Mkm2/DAY
RapidEye~6 Mkm2/DAY
Processing Engine
Ingestion API WEB TILES
API
CATALOG
SCENES, MOSAIC, TILE APIANALYTICS ANALYSIS API
ARCHIVE GUIFIND, DOWNLOAD
DATA VIA WEB BROWSER
TOOLSSAMPLE CODE
3RD PARTY APPLICATIONS
More to come …
Landsat 8
![Page 21: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/21.jpg)
Large Scale, Dynamic Environment
![Page 22: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/22.jpg)
We’ve Seen Change
• Growing number of service teams• Dozens of services
• Different scales• Different maturity• Different markets
• Unintentional monolith and/or unmanageable fringe services
• Infrastructure as impediment
![Page 23: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/23.jpg)
How our software world is changing• Containers
• Flexible per service packaging• Faster, simpler, consistent deploys
• Kubernetes for orchestration• Dynamic reallocation of resources• Isolate development teams from infra
changes and each other• Couple service changes
where necessary
![Page 24: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/24.jpg)
Planet’s Evolving Infrastructure
JobsWorker Spot
Instances
Imagery Storage
Storage Elastic Search
Planet API v0
Planet API v1
Infra
Apps
![Page 25: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/25.jpg)
Planet Kubernetes Tooling
• Klondike• Bootstraps k8s on AWS for Planet• https://github.com/planetlabs/klondike• Focused on infrastructure deployment
• Planet-Express• Environments, Services, Deployments, Secrets• Template driven manifests and artifacts
kubectl create -f build/<platform>/<service>/ kubectl replace -f build/<platform>/<service>/
• Focused on services development and deployment
![Page 26: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/26.jpg)
Key Learnings
• Kubernetes increases our flexibility and responsiveness
• Move can be evolutionary• Tooling is leveraged across teams• More good stuff to come• More change on the way - better
prepared to exploit opportunity
![Page 27: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/27.jpg)
London Array Wind Farm, United Kingdom, APR 17, 2016
See Change. Change the World.
(applies to software and infrastructure too!) [email protected]
@troytoman
![Page 28: Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes](https://reader031.fdocuments.us/reader031/viewer/2022030305/5871ae051a28abda6a8b60c7/html5/thumbnails/28.jpg)
London Array Wind Farm, United Kingdom, APR 17, 2016