Deploying software at Scale
-
Upload
kris-buytaert -
Category
Documents
-
view
2.005 -
download
2
description
Transcript of Deploying software at Scale
![Page 1: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/1.jpg)
Deploying Software at Deploying Software at ScaleScale
Kris Buytaert
@krisbuytaert
![Page 2: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/2.jpg)
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops
![Page 3: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/3.jpg)
Todays GoalsTodays Goals● A reproducable way to deploy and upgrade A reproducable way to deploy and upgrade
softwaresoftware
● AutomaticallyAutomatically
● FastFast
● ConsistentConsistent
![Page 4: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/4.jpg)
What's this devops thing What's this devops thing anyhow ? anyhow ?
![Page 5: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/5.jpg)
The community of developers whose work you The community of developers whose work you see on the Web, who probably don’t know what see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy ADO or UML or JPA even stand for, deploy betterbetter systems at systems at less costless cost in in less timeless time at at lower risklower risk than we see in the Enterprise. This is true even than we see in the Enterprise. This is true even when you factor in the greater flexibility and when you factor in the greater flexibility and velocity of startups.velocity of startups.
Tim Bray , on his blog January 2010Tim Bray , on his blog January 2010
What's the problem ?What's the problem ?
![Page 6: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/6.jpg)
The Old DaysThe Old Days● ““Put this Code Live, here's a tarball” NOW! Put this Code Live, here's a tarball” NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ? High Availability ?
● Scalability ?Scalability ?
● My computer can't install this ?My computer can't install this ?
![Page 7: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/7.jpg)
devopsdevops● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
![Page 8: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/8.jpg)
devops (<)> continuous delilvery devops (<)> continuous delilvery
![Page 9: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/9.jpg)
NirvanaNirvanaAn “ecosystem” that supports continuous delivery, from An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to infrastructure, data and configuration management to business.business.
Through automation of the build, deployment, and testing Through automation of the build, deployment, and testing process, and improved collaboration between developers, process, and improved collaboration between developers, testers, and operations, delivery teams can get changes testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code matter what the size of a project or the complexity of its code base.base.
Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
![Page 10: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/10.jpg)
![Page 11: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/11.jpg)
How many times a day ?How many times a day ?● 10 @ Flickr10 @ Flickr
● Deployments used to be painDeployments used to be pain
● Nobody dared to deploy a siteNobody dared to deploy a site
● Practice makes perfectPractice makes perfect
● Knowing you can vs constantly doing it Knowing you can vs constantly doing it
![Page 12: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/12.jpg)
" Our job as engineers (and ops, dev-ops, QA, " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to support, everyone in the company actually) is to enable the business goals. We strongly feel that enable the business goals. We strongly feel that in order to do that you must have in order to do that you must have the ability to the ability to deploy code quickly and safelydeploy code quickly and safely. Even if the . Even if the business goals are to deploy strongly QA’d code business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all once a month at 3am (it’s not for us, we push all the time), having a reliable and easy the time), having a reliable and easy deployment should be deployment should be non-negotiablenon-negotiable." ."
Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
![Page 13: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/13.jpg)
How do we get there ?How do we get there ?
![Page 14: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/14.jpg)
OS BaselineOS Baseline● Automated DeploymentsAutomated Deployments
● Reproducable Reproducable
● Kickstart, FAI, Preseeding, Kickstart, FAI, Preseeding,
● JeOSJeOS
![Page 15: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/15.jpg)
Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code
● Development best practicesDevelopment best practices
• Model your infrastructureModel your infrastructure
• Version your cookbooks / manifestsVersion your cookbooks / manifests
• Test your cookbooks/ manifestsTest your cookbooks/ manifests
• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring )
● Think Puppet, Chef, Cfengine, Ansible , ....Think Puppet, Chef, Cfengine, Ansible , ....
![Page 16: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/16.jpg)
Version ControlVersion Control● Git !Git !
● Version Version ALLALL the things: the things:
• Source code Application Source code Application
• Source code Infrastructure Source code Infrastructure
• BuildsBuilds
• TestsTests
• PipelinesPipelines
• ScriptsScripts
• Documentation Documentation
• Monitoring scriptsMonitoring scripts
•
•
•
![Page 17: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/17.jpg)
Continuous IntegrationContinuous Integration
● Builds Builds
● Nightly Builds Nightly Builds
● Builds with tests Builds with tests
● Nightly Builds with tests Nightly Builds with tests
● Frequent integration Frequent integration
● Continuous Integration Continuous Integration
![Page 18: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/18.jpg)
JenkinsJenkins● Open Source Continuous Integration ServerOpen Source Continuous Integration Server
● A zillion plugins (400)A zillion plugins (400)
● Have developers build stable and deployable Have developers build stable and deployable codecode
● Test Infra code Test Infra code
![Page 19: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/19.jpg)
Jenkins Pipeline Jenkins Pipeline
![Page 20: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/20.jpg)
What's in your Pipeline ?What's in your Pipeline ?
![Page 21: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/21.jpg)
A pipelineA pipeline● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● Package Package
![Page 22: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/22.jpg)
App RequirementsApp Requirements● TestableTestable
● Configuration isolatedConfiguration isolated
● Automated DeploymentsAutomated Deployments
● ““If my computer can't install it , the installer is If my computer can't install it , the installer is borken” borken” Luke Kanies at Fosdem (2007)Luke Kanies at Fosdem (2007)
● Bulk provisioning of data Bulk provisioning of data
● http://www.krisbuytaert.be/blog/how-i-my-javahttp://www.krisbuytaert.be/blog/how-i-my-java
![Page 23: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/23.jpg)
Why ops like to package Why ops like to package ● Packages give you featuresPackages give you features
•Consistency, security, dependenciesConsistency, security, dependencies
● Uniquely identify where files come fromUniquely identify where files come from
•Package or cfg-mgmt Package or cfg-mgmt
● Source repo not always availableSource repo not always available
•Firewall / Cloud etc .. Firewall / Cloud etc ..
● Weird deployment locations , no easy accessWeird deployment locations , no easy access
● Little overhead when you automateLittle overhead when you automate
● CONFIG does not belong in a packageCONFIG does not belong in a package
![Page 24: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/24.jpg)
#packagingsucks#packagingsucks• Really ..Really ..
● Missing upstreamMissing upstream
● Ancient upstreamAncient upstream
● Unneeded dependenciesUnneeded dependencies
● Broken upstreamBroken upstream
● Distro PoliciesDistro Policies
● ““Maximum RPM” ?Maximum RPM” ?
● Packaging is needed Packaging is needed
● Dependencies, tooling, repositoriesDependencies, tooling, repositories
● Anger driven developmentAnger driven development
![Page 25: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/25.jpg)
NotNot all all packagespackages are are equalequal
![Page 26: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/26.jpg)
#packaginlove#packaginlove
![Page 27: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/27.jpg)
fpmfpmfpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit 0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ /usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+ /usr/lib/rpm/brp-strip-static-archive+ /usr/lib/rpm/brp-strip-comment-noteProcessing files: hornetq-2.2.5-1.x86_64Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created /usr/local/hornetq-2.2.5.x86_64.rpm
![Page 28: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/28.jpg)
fpm in actionfpm in action● https://github.com/Inuits/build-gemshttps://github.com/Inuits/build-gems
● Fork, pull Fork, pull
● Jenkins pulls , builds , pushes to repoJenkins pulls , builds , pushes to repo
● (variants for Nagios Plugins / Jenkins Plugins (variants for Nagios Plugins / Jenkins Plugins available) available)
![Page 29: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/29.jpg)
A pipelineA pipeline● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● Package Package
● Upload to RepoUpload to Repo
![Page 30: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/30.jpg)
Pulp Pulp Pulp is a Python application for managing Pulp is a Python application for managing software repositories and their associated software repositories and their associated content, such as packages, errata, and content, such as packages, errata, and distributions. It can replicate software distributions. It can replicate software repositories from a variety of supported sources, repositories from a variety of supported sources, such as http/https, file system, ISO, and RHN, to such as http/https, file system, ISO, and RHN, to a local on-site repository. It provides mechanisms a local on-site repository. It provides mechanisms for systems to gain access to these repositories, for systems to gain access to these repositories, providing centralized software installation.providing centralized software installation.
![Page 31: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/31.jpg)
Pulp Pulp ● Redhat Community Redhat Community
● Redhat Emerging TechnologyRedhat Emerging Technology
● Part of Katello Part of Katello
![Page 32: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/32.jpg)
![Page 33: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/33.jpg)
Pulp Pulp ● ““manages” its own apache instancemanages” its own apache instance
● Symlinks , no copiesSymlinks , no copies
● QueuesQueues
•Syncing in the background Syncing in the background
•No more screens ;)No more screens ;)
● Actions are not instantlyActions are not instantly
•e.g. Add / sync / deletee.g. Add / sync / delete
● Hello mongodb :(Hello mongodb :(
● v1 vs v2v1 vs v2
● Only use repo functionality , cfgmgmt is in charge of packagesOnly use repo functionality , cfgmgmt is in charge of packages
![Page 34: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/34.jpg)
Repository ManagementRepository Management
![Page 35: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/35.jpg)
Version vs LatestVersion vs Latest● Version your repos ?Version your repos ?
ensure => latestsensure => latests
● Latest your environments ?Latest your environments ?
● Strict versioning in config ?Strict versioning in config ?
ensure => '0.98.4'ensure => '0.98.4'
![Page 36: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/36.jpg)
A pipelineA pipeline● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● Package Package
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
![Page 37: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/37.jpg)
The Marionette CollectiveThe Marionette Collective● Distributed ssh ++Distributed ssh ++
● What version of ssh do I have installed on my What version of ssh do I have installed on my servers ? servers ?
● On what servers is XYZ running ? On what servers is XYZ running ?
● Clean all my ssl certs ? Clean all my ssl certs ?
● Restart apache on all servers with fact X Restart apache on all servers with fact X
![Page 38: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/38.jpg)
mc-packagemc-packagemc-package -W /dev/ status jdk * [ ============================================================> ] 33 / 33
servicesdb01.dev.com version = -absent services.dev.google.com version = jdk-1.6.0_13-fcs drbdtest02.dev.google.com version = -absent services3.dev.google.com version = jdk-1.6.0_20-fcs um.dev.google.com version = jdk-1.5.0_19-fcs devtools03.uat.com version = jdk-1.6.0_29-fcs alexandria02.dev.google.com version = -absent weblink01.dev.com version = -absent wikitest.dev.google.com version = jdk-1.6.0_24-fcs payment.dev.google.com version = jdk-1.5.0_17-fcs tiff2pdf01.dev.com version = -absent devdoos.dev.com version = jdk-1.6.0_30-fcs wiki.dev.google.com version = jdk-1.6.0_24-fcs reporting01.dev.com version = -absent devtools01-dev.uat.com version = jdk-1.6.0_23-fcs devtools02.uat.com version = jdk-1.6.0_29-fcs drbdtest01.dev.google.com version = -absent
---- package agent summary ---- Nodes: 33/33 Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0_29-fcs, 1 * 1.6.0_30-fcs, 1 * 1.6.0_31-fcs, 22 * absent Elapsed Time: 1.73 s
![Page 39: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/39.jpg)
What to Trigger ?What to Trigger ?● Update PackageUpdate Package
• Only updates packageOnly updates package
● Trigger Puppet Run Trigger Puppet Run
• Updates config + packageUpdates config + package
![Page 40: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/40.jpg)
A pipelineA pipeline● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● Package Package
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● More TestsMore Tests
● PromotePromote
● Deploy on UATDeploy on UAT
● More TestsMore Tests
● PromotePromote
● Deploy on ProdDeploy on Prod
![Page 41: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/41.jpg)
Done ?Done ?● Close the feedback loop,Close the feedback loop,
● Send metric on deployment Send metric on deployment
echo "deployed.$package_name 1 `date +echo "deployed.$package_name 1 `date +%s`" > /dev/tcp/<%= graphite_host %>/2003 %s`" > /dev/tcp/<%= graphite_host %>/2003
![Page 42: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/42.jpg)
Done ?Done ?
A Software project is not done until your last A Software project is not done until your last enduser is in his grave ! enduser is in his grave !
![Page 43: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/43.jpg)
But remember But remember Everything is a Fscking DNS Problem
No really, Everything is a Fscking DNS ProblemIf it's not a fucking DNS Problem ..
It's an arp problemIf it's not an arp problem...
It's a Full Filesystem ProblemIf your filesystem isn't full
It's a Spanning Tree problemIf it's not a spanning Tree problem...
It's a USB problemIf it's not a USB Problem
It might be an ntp problem If it's not an ntp problemIt's a sharing IRQ Problem
If it's not a sharing IRQ ProblemBut most often .. its a Freaking Dns Problem !
Or someone playing tricks on you
Jan 2006
![Page 44: Deploying software at Scale](https://reader033.fdocuments.us/reader033/viewer/2022052410/5549b3b0b4c905e5048b47b4/html5/thumbnails/44.jpg)
ContactContactKris Buytaert Kris Buytaert [email protected]@inuits.be
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/bhttp://www.krisbuytaert.be/blog/log/http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221