Continues Deployment - Tech Talk week
-
Upload
rantav -
Category
Technology
-
view
2.150 -
download
0
description
Transcript of Continues Deployment - Tech Talk week
![Page 1: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/1.jpg)
Continuous Deployment
@rantav@outbrain
![Page 2: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/2.jpg)
How much time does it take your code to meet the users?
![Page 3: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/3.jpg)
How much time does it take your code to meet the users?• More than a year?
![Page 4: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/4.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
![Page 5: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/5.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
![Page 6: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/6.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
![Page 7: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/7.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
![Page 8: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/8.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
• 1 - 24 Hours?
![Page 9: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/9.jpg)
How much time does it take your code to meet the users?• More than a year?
• 6 - 12 Months?
• 1- 6 months?
• 2 - 4 weeks?
• 1 - 14 Days?
• 1 - 24 Hours?
• < 12 Minutes?
![Page 10: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/10.jpg)
~ $ svn ci -m "Implement the super-sharp image scaler. #deloy:ImageServer #to:ny"
![Page 11: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/11.jpg)
Outbrain enables readers to discover the most interesting, relevant and timely links to stories (paid and organic)
![Page 12: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/12.jpg)
![Page 13: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/13.jpg)
Multi Billion Page Views per month
Note: Outbrain is typically installed on *every* article/blog post on each of these sites, immediately under the content.
![Page 14: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/14.jpg)
Who's in?
![Page 15: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/15.jpg)
WHY
![Page 16: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/16.jpg)
WHY
For the Business
For Fun
![Page 17: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/17.jpg)
WHYWhat was so bad before ?
![Page 18: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/18.jpg)
WHY
What is a startup?
• A startup is always on a quest to find Product-Market fit
• => You have to iterate Fast• ... before running out of money
![Page 19: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/19.jpg)
WHY
• Feedback loop speed is important (see REPL in programming)
![Page 20: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/20.jpg)
WHY
What's used to be the case before:
• Inefficient Waits (wait for QA, wait for other features etc)
• Inefficient Context Switch• Feature Delayed - b/c of other features• Big Changes – Big Problems.
![Page 21: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/21.jpg)
HOW
![Page 22: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/22.jpg)
HOW
![Page 23: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/23.jpg)
HOWContinuous Deployment Themes
• Release is a marketing decision.
• Deployment is an engineering decision!
![Page 24: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/24.jpg)
HOWContinuous Deployment Themes
• Small Changes – Reduce Risks
• Kent Beck: You can spill a bucket but you can't spill a Hose.
![Page 25: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/25.jpg)
HOWContinuous Deployment Themes
• Deploy fast, release often
• Fast turnarounds lead to happy customers
• ... and happy developers
![Page 26: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/26.jpg)
HOWCulture
• Everyone have to care about everything!o buildo testso qualityo productiono monitoringo business
![Page 27: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/27.jpg)
HOWCulture
• No Broken Windows!
Broken Windows Theory
![Page 28: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/28.jpg)
HOWCulture
“What's the worse that could happen?”
![Page 29: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/29.jpg)
WORKFLOW
![Page 30: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/30.jpg)
WORKFLOWDev
![Page 31: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/31.jpg)
WORKFLOWDev
![Page 32: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/32.jpg)
The
INGREDIENTS
![Page 33: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/33.jpg)
INGREDIENTSTrunk Stable
•Everyone run tests before commit•No branches, really•Trunk may get released any
moment. If you commit now, users will see it really soon.
=> Test your code, really good•Use feature flags (but only if you
must)• Forward and backward compatibility
![Page 34: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/34.jpg)
INGREDIENTSAutomated Tests
•We have about 2000 test cases•They run in < 4 minutes•We use TeamCity•We regularly test production
Desired state: No manual QA whatsoever.
![Page 35: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/35.jpg)
INGREDIENTSInfrastructure Automation
"Infrastructure as code"
•Using kickstart script to install OS and Chef agent.
•All infrastructure is deployed by Chef.
•All apps are deployed by Glu•All scripts, and chef cookbooks, glu
configs are SVNed.•Very easy to deploy large number of
machines.
![Page 36: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/36.jpg)
INGREDIENTSDeployer
![Page 37: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/37.jpg)
![Page 38: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/38.jpg)
![Page 39: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/39.jpg)
![Page 40: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/40.jpg)
INGREDIENTSServicization
• At outbrain we have ~ 25 services• Each service is deployed at > 1 server• If there's damage, it's contained• It's easy to make small changes• If there's an error it's easy to find it• If rollback is needed it's easier• Everything is either proxies (HAProxy) or queued (AMQ)
• Challenge: It's sometimes hard to find the right balance:o number of services that need maintananceo complexityo performanceo api conformance
![Page 41: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/41.jpg)
INGREDIENTSThe Immune System
• Codingo Reviewo Testing
• Monitoringo We use Nagios for the vitalso Regularly check servers via instrumentationo KeyNoteo Test Production (selenium,APT)
• Instrumentationo Self-Testo Performance
• Monitoring KPIso PVs, o 3 different CTRs,o clicks, revenue, RPMo etc...
![Page 42: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/42.jpg)
INGREDIENTSThe Immune System
• The line of defense will always be broken
=> Multiple lines of defense
![Page 43: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/43.jpg)
INGREDIENTSVisibilty
• Everyone monitor the services • When there's a deployment, you see it on the graph
![Page 44: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/44.jpg)
INGREDIENTSVisibilty
• Yammer
![Page 45: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/45.jpg)
INGREDIENTSVisibilty
• svn changelog• glu changelog• glu audit log
![Page 46: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/46.jpg)
INGREDIENTSLearn Fast, Adapt Fast
• 5 Ys
![Page 47: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/47.jpg)
ARSENALOutbrain's arsenal
Architecture:
• Java in most parts, Spring for the most and Struts2• SOA (dah...) with REST• Message queues (ActiveMQ)• Service multiplicity with HAProxy• Data stored in:
o MySQL, Cassandra, HDFS, NFS• Caching: Memcached
![Page 48: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/48.jpg)
ARSENALOutbrain's arsenal
Testing:
• JUnit• TeamCity• Selenium• Staging environment (for high risk deployments)
![Page 49: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/49.jpg)
ARSENALOutbrain's arsenal
Deployment:
• Kickstart• Chef• Glu• TeamCity• In-house Deployment Manager• RPM + YUM
![Page 50: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/50.jpg)
ARSENALOutbrain's arsenal
Monitoring:
• Nagios• "Testing production" every 10 minutes. • Keynote
![Page 51: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/51.jpg)
ARSENALOutbrain's arsenal
Communications:
• Yammer
![Page 52: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/52.jpg)
Fun Numbers
• 5-50 production changes a day!!!• More then 2000 code tests running in
less then 4 minutes.• More then 600 production services tests
runs every 10 minutes.• It takes ~30 minutes from code
complete to ~100 machines deployed.
![Page 53: Continues Deployment - Tech Talk week](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b520b54a7959cd118b45ef/html5/thumbnails/53.jpg)
References
• Why Continuous Deployment / Eric Ries• Continuous Deployment at outbrain / Ran Tavory • Deployment Infrastructure for Continuous Deployment /
WealthFront• Continuous Deployment presentation / Eishay Smith• Quantum of Deployment / Etsy• Chrome Release Cycle / Anthony Laforge