Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source
24
-
Upload
dianne-marsh -
Category
Software
-
view
481 -
download
3
description
How can we engineer velocity by weaving together tooling and culture with software development to expose and elevate highly effective practices? This talk describes various components of Netflix’s continuous delivery platform -- much of which is available in open source. I’ll show how these pieces fit together and allow us to build scaffolding so that we’re comfortable with software developers making the decision to push the button for prod deployment -- and helps them to recover if necessary. As a result, we can run fast, trusting our tooling and our culture. I’ll also describe how we test our resiliency through simulating failure, unleashing the monkeys (Simian Army) on our production environment. Because if you’re afraid of cute little monkeys, imagine how afraid you’ll be of a production environment that offers those same risks but doesn’t give you an opportunity to test your response to those dangers. Throughout this talk, I hope that you will challenge yourself to consider how your company can "shift the curve" through tooling and to achieve a high velocity environment without negatively impacting reliability.
Transcript of Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source
Continuous Delivery at Netflix: Powered by Open Source
Dianne MarshSan Francisco, May 6, 2014
http://www.linkedin.com/in/diannemarsh
Building Continuous Delivery
Netflix GoalsHigh Availability
But … Move fastTools encourage Best Practices
But … Freedom to do the right thing
Teams Deploy Their Own CodeRun What You Wrote• Rapid Innovation• Rapid Detection• Rapid Response= Freedom + Responsibility
Nebula/Gradle• Netflix Build Language• Gradle plugins• Supports internal and open source
builds
https://github.com/nebula-plugins
BUILD
BUILD Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code
BUILD BAKE
Aminator• Create AMI from Base AMI• Image contains service and
everything needed to run it• Unit of Deployment for Test and
Prod• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
BAKE
BUILD DEPLOYBAKE
DEPLOY
Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push
http://www.infoq.com/presentations/asgard
CANARY ANALYSISTest, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue
Asgard Developer Portal
GLISTEN Extending Asgard’s WorkflowAutomated Red/Black Push
Run canary/analysisScale up new instancesTurn on traffic/Run Tests/Turn off trafficWait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
One Button Deployment?
Regional IsolationLimit Impact of Human Error• Stagger deployments?• Canary testing per region?
Know your service!
Multi-Region ConsistencyBuild Tooling to:• Schedule Deployments• Prefer off peak• Choose next available
region automatically• Provide high visibility
per region
BUILD
DEPLOY
BAKE
Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey (and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
Send in the Conformity MonkeyHave deployments diverged? Balance regional consistency
with regional isolation Provide meaningful thresholds Build best practices into
tooling and reporting
Clean up with the Janitor Monkey• Disassociate unused EIPs• Delete unassociated
Amazon EBS volumes• Delete older Amazon
EBS snapshots• Leverage Amazon S3
Object Expiration
Key Elements for Netflix• Value Self-service • Test Everywhere• Build Awareness of Multiple Regions• Avoid peak times• Roll back quickly and easily• Be Cloud Native
Netflix OSS
More at netflix.github.com
