Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna,...
Transcript of Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna,...
![Page 1: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/1.jpg)
![Page 2: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/2.jpg)
2© 2019 Pinterest. All rights reserved. ok
Adam McKenna, Site Reliability Engineer, Pinterest@deathtocss/in/admckennaPronouns: He, his
Invisible OS and Platform Upgrades
![Page 3: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/3.jpg)
![Page 4: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/4.jpg)
4© 2019 Pinterest. All rights reserved.
Our mission
to create a lifethe inspirationeveryone
they love.
To bring
@deathtocss
![Page 5: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/5.jpg)
5© 2019 Pinterest. All rights reserved. @deathtocss
Pinterest Scale
- 30 Million monthly active users- ~80,000 hosts under management- Dockerized micro-service architecture
![Page 6: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/6.jpg)
6© 2019 Pinterest. All rights reserved.
Core SRE @ Pinterest● Overall Pinterest Uptime● Internal Services
About Me:● Linux user since 1993, Sysadmin
since 1998● DevOps practitioner since early
2000’s
@deathtocss
![Page 7: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/7.jpg)
7© 2019 Pinterest. All rights reserved.
About this talk
Deck elements 1 2 3 4
@deathtocss
![Page 8: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/8.jpg)
8© 2019 Pinterest. All rights reserved. @deathtocss
Target Audience- DevOps/SRE/Sysadmin/Release Engineer- Don’t like having prod issues- Looking for ways to improve reliability of deploys
![Page 9: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/9.jpg)
9© 2019 Pinterest. All rights reserved. @deathtocss
Key Takeaways- What Canary Analysis is and isn’t- What it’s useful for- Practical Considerations- Example System / Components
![Page 10: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/10.jpg)
10© 2019 Pinterest. All rights reserved. @deathtocss
Prerequisites- Your org already uses CI/CD for deploys- Critical service metrics stored in a time series DB- Organizational buy-in
![Page 11: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/11.jpg)
11© 2019 Pinterest. All rights reserved.
DefinitionsDeck elements 1 2 3 4
@deathtocss
![Page 12: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/12.jpg)
12© 2019 Pinterest. All rights reserved.
“Enabling Invisible Infrastructure Upgrades with Automated Canary Analysis”
@deathtocss
![Page 13: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/13.jpg)
13© 2019 Pinterest. All rights reserved.
“Enabling Invisible Infrastructure Upgrades with Automated Canary Analysis”
@deathtocss
![Page 14: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/14.jpg)
14© 2019 Pinterest. All rights reserved. @deathtocss
Invisible
Does not require a significant time investment by the service owner(s)
![Page 15: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/15.jpg)
15© 2019 Pinterest. All rights reserved. @deathtocss
Infrastructure
“the underlying foundation or basic framework (as of a system or
organization)” - Merriam-Webster
![Page 16: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/16.jpg)
16© 2019 Pinterest. All rights reserved. @deathtocss
- Physical Hardware- Cloud provider Instance Type/CPU Generation- Storage Technology- Networking stack
- Operating System- Kernel- User space
- Language - Compilers (C++/Go/etc.)- Runtime (JVM/Python Interpreter/etc.)- Dependencies (versions of vendored libraries/modules/etc)
Infrastructure
![Page 17: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/17.jpg)
17© 2019 Pinterest. All rights reserved.
Canary Analysis
Masters 1 2 3 4
![Page 18: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/18.jpg)
18© 2019 Pinterest. All rights reserved.
Image credits: Michael Sonnabend (flickr)https://pxhere.com/en/photo/1412043
@deathtocss
![Page 19: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/19.jpg)
19© 2019 Pinterest. All rights reserved. @deathtocss
Infrastructure has an expiration date
![Page 20: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/20.jpg)
20© 2019 Pinterest. All rights reserved. @deathtocss
Infrastructure has an expiration date
●OS/Container Runtimes○ Ubuntu
● 12.04, 14.04 EOL○ Docker
● Quarterly Updates○ Windows
● Windows 7 EOL (Pinterest not affected)
●Language Runtimes○ Python
● 2.x to 3.x○ Java
● Oracle to OpenJDK● Java version updates
○ Go○ C++
![Page 21: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/21.jpg)
21© 2019 Pinterest. All rights reserved. @deathtocss
Upgrading is not optional●Meet industry compliance requirements
●Retain support○ Security bug fixes
●Access to new features○ Latest hardware support (CPU, GPU, etc.)○ Performance improvements○ New storage technologies
The business needs to:
@deathtocss
![Page 22: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/22.jpg)
22© 2019 Pinterest. All rights reserved. @deathtocss
Developers want●Access to new packages /
features●Platform Efficiency
@deathtocss
![Page 23: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/23.jpg)
23© 2019 Pinterest. All rights reserved. @deathtocss
But upgrading is hard!●Complexity
○ Hundreds of microservices across tens of thousands of hosts
●Service owners○ Don’t like downtime.○ Migration work is generally not a preferred task
![Page 24: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/24.jpg)
24© 2019 Pinterest. All rights reserved. @deathtocss
Canary AnalysisA tool that helps us automate and normalize the most mundane migration tasks
- Automate- looking at charts, comparing metrics
- Normalize- The same process can be applied to many similar systems.
![Page 25: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/25.jpg)
25© 2019 Pinterest. All rights reserved. @deathtocss
What Canary Analysis isn’t- A replacement for traditional testing patterns:
- Unit and integration tests- Service Health Checks
- Mysterious or Magical- Off the Shelf- Instantaneous
![Page 26: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/26.jpg)
26© 2019 Pinterest. All rights reserved. @deathtocss
Typical migration ‘toil’ pattern1. New Image is released2. Service owner updates Image ID in deploy system3. Operator triggers rolling cluster upgrade4. Humans watch charts and cross fingers5. Did things go OK? Good!6. Is the site down/degraded? Bad! Rollback!
Outage?
![Page 27: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/27.jpg)
27© 2019 Pinterest. All rights reserved. @deathtocss
“Looks good to me!”Can you make a Go / No-Go decision based on these two charts?
![Page 28: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/28.jpg)
28© 2019 Pinterest. All rights reserved. @deathtocss
Sometimes humans disagree
![Page 29: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/29.jpg)
29© 2019 Pinterest. All rights reserved. @deathtocss
How about these?
![Page 30: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/30.jpg)
30© 2019 Pinterest. All rights reserved. @deathtocss
The human factor
Gene● His service is his “baby”● Extremely Risk Averse● Attains 99.999999% uptime,
but runs on Ubuntu 10.04
What might these Service Owners think?
![Page 31: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/31.jpg)
31© 2019 Pinterest. All rights reserved. @deathtocss
Gene’s call“No way. Look at that I/O line! Let’s meet on Monday and figure out what’s going on.”
![Page 32: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/32.jpg)
32© 2019 Pinterest. All rights reserved. @deathtocss
Angus● Last job was at a Bitcoin startup● Likes to “Move fast and break
stuff”● Yells “TONIGHT WE TEST IN
PROD” at least once a day
![Page 33: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/33.jpg)
33© 2019 Pinterest. All rights reserved. @deathtocss
Angus’s call“LGTM. SHIP IT!”
![Page 34: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/34.jpg)
34© 2019 Pinterest. All rights reserved. @deathtocss
Carol● Has 100% test coverage on all
her code● Is researching statistical models
to improve fault detection
![Page 35: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/35.jpg)
35© 2019 Pinterest. All rights reserved. @deathtocss
Carol’s call“I’ll need a couple hours to run some additional tests against the new hosts. Stand by.”
![Page 36: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/36.jpg)
36© 2019 Pinterest. All rights reserved. @deathtocss
<deleted>● The employee who left the
company● All of the knowledge they didn’t
write down about their service left with them!Image credit: svgsilh.com
![Page 37: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/37.jpg)
37© 2019 Pinterest. All rights reserved.
Practical Considerations
Charts and tables 1 2 3 4
@deathtocss
![Page 38: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/38.jpg)
38© 2019 Pinterest. All rights reserved.
Components●CI/CD Pipeline○ Workflow orchestration
● https://www.spinnaker.io/
●Time-Series Metrics Database●Execution environment for custom code●Canary Judge software
@deathtocss
![Page 39: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/39.jpg)
39© 2019 Pinterest. All rights reserved. @deathtocss
![Page 40: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/40.jpg)
40© 2019 Pinterest. All rights reserved. @deathtocss
Canary Best Practices
@deathtocss
● Canary and Control should be similarly sized, in order to have a good basis for comparison.
● Both clusters should be serving a meaningful amount of production traffic.
● A minimum of 50 data points is required for a reliable Canary Analysis score
● You need to have metrics that reflect your application’s health. (Golden Signals: latency, traffic, errors, and saturation.)*
*https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals
![Page 41: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/41.jpg)
41© 2019 Pinterest. All rights reserved.
● About one year of effort for 2-3 FTE● Custom Python CLI application● Kayenta: No native support for OpenTSDB, so we
wrote our own
Overall Effort
@deathtocss
![Page 42: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/42.jpg)
42© 2019 Pinterest. All rights reserved.
● UX is important● Don’t pilot ACA using Gene’s app.● Pilot ACA using Gene’s app.● Have good and bad versions of your app to test with.
Lessons Learned
@deathtocss
![Page 43: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/43.jpg)
43© 2019 Pinterest. All rights reserved.
● Kayenta: https://cloud.google.com/blog/products/gcp/introducing-kayenta-an-open-automated-canary-analysis-tool-from-google-and-netflix
● Spinnaker Canary Analysis doc: https://www.spinnaker.io/setup/canary/● Waze presentation on Canary Analysis:
https://cloud.google.com/blog/products/devops-sre/canary-analysis-lessons-learned-and-best-practices-from-google-and-waze
● Netflix presentation: https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
Additional Resources
@deathtocss
![Page 45: Platform Upgrades Invisible OS and · © 2019 Pinterest. All rights reserved. ok 2 Adam McKenna, Site Reliability Engineer, Pinterest @deathtocss /in/admckenna Pronouns: He, his Invisible](https://reader033.fdocuments.us/reader033/viewer/2022042202/5ea2fbe8a370c73b0321ff3f/html5/thumbnails/45.jpg)
45© 2019 Pinterest. All rights reserved.
Questions?Example slides 1 2 3 4
@deathtocss