Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and...
-
Upload
puppet-labs -
Category
Documents
-
view
549 -
download
7
description
Transcript of Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and...
![Page 1: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/1.jpg)
Application Deployment Orchestrationwith Puppet and JenkinsAnton Gurov, Chaminda Delpagodage
August 20, 2014
![Page 2: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/2.jpg)
22
About Us
Chaminda DelpagodagePaydiant Technical Operations TeamRelease Engineering, Systems Administration, Automationlinkedin.com/in/chamindad
Anton GurovPaydiant Technical Operations TeamInfrastructure, Systems Administration, Securitylinkedin.com/in/antongurov
![Page 3: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/3.jpg)
33
Cloud-based mobile wallet solution
Open ecosystem for mobile payments, offers and loyalty
Completely white-label
“Bank grade” platform of shared services↘ SaaS
↘ Secure SDKs for iPhone and Android
Top tier investors and well capitalized
![Page 4: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/4.jpg)
44
Paydiant Puppet Use
Puppet Enterprise (PE) users since day one
100% PE coverage of Paydiant platform↘ PE handles everything after instance bootstrap
Multiple environments actively managed by PE↘ 4 Puppet Masters in multiple datacenters and security zones
↘ 8 Environments
Licensed node count doubling every year
Hosts0
100200300400500600700800900
Estimated by Year-End
Nodes under man-agement
2011 20122013 2014 EST
![Page 5: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/5.jpg)
55
Paydiant Puppet Use
‘11-12 – Bi-annual production platform releases↘ Waterfall – major platform change
↘ Big outage – 1-2 days on the weekend
‘13-14 – Transition to daily/weekly non-production and monthly production releases
↘ Agile – smaller platform changes
↘ Zero-downtime deployment
↘ 100% Production release success rate since inception
Heavy usage of Puppet Dashboard, Puppet APIs and Jenkins
![Page 6: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/6.jpg)
66
Puppet Dashboard as data repository
Why Dashboard?↘ Visual, flexible, powerful (if used right)
↘ Allows for business data edits by teams unfamiliar with Puppet
↘ Hiera not available at the time
Decided early on to keep Puppet code and data separate
Came up with our own Dashboard pattern – “Classes, Parameters and Supergroups”
Puppet Module
Code
PuppetDashboard
BusinessData
Puppet Module
Parameters
![Page 7: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/7.jpg)
77
Puppet Dashboard as data repositoryClasses, Parameters and Supergroups pattern overview
class_C
supergroup_type_A
class_Bclass_A
parameters_X parameters_Y parameters_Z…
…
node 1 node 2 node X…node 4node 3
Groups
Nodes
![Page 8: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/8.jpg)
88
Puppet Dashboard as data repositoryClasses, Parameters and Supergroups pattern overview
class_C
supergroup_type_B
class_Bclass_A
parameters_X parameters_Y parameters_Z…
…
node 1 node 2 node X…node 4node 3
Groups
Nodes
![Page 9: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/9.jpg)
99
class_Bdef: default paramsincl:
Puppet Dashboard as data repositoryClass building block
class B
class A class B
class_Adef: default paramsincl: class A
class_Cdef: default paramsincl: class C
class C
…
Classes
Groups
Group name prefixed with class_Contains Puppet class and some default variables/parameters for the class
![Page 10: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/10.jpg)
1010
Puppet Dashboard as data repositoryClass building block - example
![Page 11: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/11.jpg)
1111
Puppet Dashboard as data repositoryParameters building block
Group name prefixed with parameters_
Only contains data and data overrides
Arbitrary hierarchy levels
Allows for inheritance and reuse
parameters_X_1incl:def: params overridesdef: additional params
parameters_Xdef: default params
parameters_X
supergroup_A supergroup_B
parameters_X_2incl:def: params overridesdef: additional params
parameters_X
supergroup_C
![Page 12: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/12.jpg)
1212
Puppet Dashboard as data repositoryParameters building block – inheritance example
![Page 13: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/13.jpg)
1313
Puppet Dashboard as data repositorySupergroup building block == server “role”
Group name prefixed with supergroup_
Contains all the “ingredients” for the node to configure and define itself
Node can belong to only one supergroup (many-to-one)
supergroup_type_Aincl:
def: params overrides (if any)def: additional params (if any)
class_B
class_A parameters_X
parameters_Z
node 1 node 2
Groups
Nodes
class_B
class_A
parameters_X
parameters_Z
![Page 14: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/14.jpg)
1414
Puppet Dashboard as data repositorySupergroup building block - example
2-3 pages condensed
![Page 15: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/15.jpg)
1515
Classes, Parameters and Supergroups pattern Pros
All parameters and classes are visible on the Supergroup page↘ See missing parameters (if inherited “SET ME!” from parent for example)
↘ See parameter clashes (Dashboard will warn if parameter is defined in 2 places)
↘ See exactly where parameter is defined
Allows teams unfamiliar with Puppet to make changes via Dashboard
Arbitrary data hierarchy/inheritance
Data reuse
![Page 16: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/16.jpg)
1616
Classes, Parameters and Supergroups pattern Cons
Version control is difficult↘ Have to resolve to group cloning/export/import (custom RAKE copy/clone command from Puppet support)
↘ Puppet roadmap to fix this
Dashboard UI could use some help↘ Too much data on the screen sometimes
↘ Lack of sorting/grouping
Can’t store complex multi-line variables like text blobs
![Page 17: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/17.jpg)
Zero-Downtime Deployment architecture …
![Page 18: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/18.jpg)
v.1
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.1
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.1
parameters_deployment-staging-FE-BankApaydiant_deployment_bank=STAGING-FRONTEND-Apaydiant_app_operation_mode=LIVEpaydiant_app_version=1
v.1
High-level platformrepresentation
parameters_deployment-staging-BE-BankApaydiant_deployment_bank=STAGING-BACKEND-Apaydiant_app_operation_mode=LIVEpaydiant_app_version=1
parameters_deployment-staging-FE-BankBpaydiant_deployment_bank=STAGING-FRONTEND-Bpaydiant_app_operation_mode=LIVEpaydiant_app_version=1
parameters_deployment-staging-BE-BankBpaydiant_deployment_bank=STAGING-BACKEND-Bpaydiant_app_operation_mode=LIVEpaydiant_app_version=1
![Page 19: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/19.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.1
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.1
Disable B(FE+BE)
v.1v.1
parameters_deployment-staging-FE-BankBpaydiant_deployment_bank=STAGING-FRONTEND-Bpaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=1
parameters_deployment-staging-BE-BankBpaydiant_deployment_bank=STAGING-BACKEND-Bpaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=1
![Page 20: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/20.jpg)
v.2a
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.1
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.1
Run first phase of database changes(i.e. adds new stuff & migrate data)
v.2aDB changes Phase 1
![Page 21: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/21.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Upgrade B (FE+BE)
v.2av.2a
parameters_deployment-staging-FE-BankBpaydiant_deployment_bank=STAGING-FRONTEND-Bpaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=2
parameters_deployment-staging-BE-BankBpaydiant_deployment_bank=STAGING-BACKEND-Bpaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=2
![Page 22: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/22.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Re-enable B (FE+BE)
v.2av.2a
parameters_deployment-staging-FE-BankBpaydiant_deployment_bank=STAGING-FRONTEND-Bpaydiant_app_operation_mode=LIVEpaydiant_app_version=2
parameters_deployment-staging-BE-BankBpaydiant_deployment_bank=STAGING-BACKEND-Bpaydiant_app_operation_mode=LIVEpaydiant_app_version=2
![Page 23: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/23.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Disable A(FE+BE)
v.2av.2a
parameters_deployment-staging-FE-BankApaydiant_deployment_bank=STAGING-FRONTEND-Apaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=1
parameters_deployment-staging-BE-BankApaydiant_deployment_bank=STAGING-BACKEND-Apaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=1
![Page 24: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/24.jpg)
FrontendLoad Balancer
FE-B FE-Av.2
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.2
FE-B BE-Bv.2
Upgrade A (FE+BE)
v.2av.2a
parameters_deployment-staging-FE-BankApaydiant_deployment_bank=STAGING-FRONTEND-Apaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=2
parameters_deployment-staging-BE-BankApaydiant_deployment_bank=STAGING-BACKEND-Apaydiant_app_operation_mode=MAINTENANCEpaydiant_app_version=2
![Page 25: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/25.jpg)
FrontendLoad Balancer
FE-B FE-Av.2
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.2
FE-B BE-Bv.2
Re-enable A (FE+BE)
v.2av.2a
parameters_deployment-staging-FE-BankApaydiant_deployment_bank=STAGING-FRONTEND-Apaydiant_app_operation_mode=LIVEpaydiant_app_version=2
parameters_deployment-staging-BE-BankApaydiant_deployment_bank=STAGING-BACKEND-Apaydiant_app_operation_mode=LIVEpaydiant_app_version=2
![Page 26: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/26.jpg)
v.2
FrontendLoad Balancer
FE-B FE-Av.2
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.2
FE-B BE-Bv.2
Run second phase of database changes(Cleanup old v.1 data)
v.2DB changes Phase 2
![Page 27: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/27.jpg)
Details of the upgrade sequence …
![Page 28: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/28.jpg)
v.1
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.1
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.1
Putting a set of nodes into maintenance mode
![Page 29: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/29.jpg)
2929
Putting nodes into maintenance mode Using LB node health check – http://nodeX:8080/healthcheck.jsp
Puppet ERB template for healthcheck.jsp content
………
Pseudo code:Check if “maintenance mode” throw exception elseIf “module A” present
Check if module A is upIf “module B” present
Check if module B is up…Throw 503 if any exception caught
![Page 30: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/30.jpg)
3030
Putting nodes into maintenance mode cont.
A parameter group controls the maintenance mode
E.g. Parameter group “parameters_deployment-staging-BankB” controls “paydiant_app_operation_mode” for the nodes in set FE-B of the Staging environment
![Page 31: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/31.jpg)
3131
Putting nodes into maintenance mode cont.
Update group parameter using Rake API (as ‘puppet-dashboard’ user)
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile nodegroup:variables [parameters_deployment-stagin-BankB, 'paydiant_app_operation_mode=MAINTENANCE’]
Puppet run-once using MCO (as ‘peadmin’ user)
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
While loop… check the health check page till all nodes return 503 (i.e. in maintenance) status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=\''curl --silent http://localhost:8080/healthcheck/healthcheck.jsp
![Page 32: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/32.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Upgrading applicationson a set of nodes
v.2a
![Page 33: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/33.jpg)
3333
Upgrading Application Version
Disable Puppet agent
mco puppet disable --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Stop Tomcat service
mco service tomcat stop --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Cleanup exploded Tomcat webapps directory (for sanity)
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd='find $TOMCAT_HOME/webapps/ -maxdepth 1 -mindepth 1 -type d -exec rm -rf {} \;’
![Page 34: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/34.jpg)
3434
Upgrading Application Version Cont.
Upgrade the application version
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile nodegroup:variables [parameters_deployment-stagin-BankB, ’paydiant_app_version=2’]
Re-enable Puppet
mco puppet enable --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
Puppet run-once
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
![Page 35: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/35.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Taking a set of nodes out ofmaintenance mode
v.2a
![Page 36: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/36.jpg)
3636
Taking nodes out of maintenance mode
Update parameter using Rake API (as ‘puppet-dashboard’ user)
RACK_ENV=production /opt/puppet/bin/rake -s -X -f /opt/puppet/share/puppet-dashboard/Rakefile nodegroup:variables [parameters_deployment-staging-BankB, 'paydiant_app_operation_mode=LIVE’]
Puppet run-once using MCO (as ‘peadmin’ user)
mco puppet runonce --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B
While loop… check the health check page till all nodes return 200 (i.e. live) status
mco shellcmd --with-fact fact_paydiant_deployment_bank=STAGING-FRONTEND-B --cmd=\''curl --silent http://localhost:8080/healthcheck/healthcheck.jsp
![Page 37: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/37.jpg)
FrontendLoad Balancer
FE-B FE-Av.1
FE-B FE-Bv.2
BackendLoad Balancer
FE-B BE-Av.1
FE-B BE-Bv.2
Switching traffic toupgraded stack
v.2a
![Page 38: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/38.jpg)
Viewing transition in Splunk across multiple datacenters
Version Transition
![Page 39: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/39.jpg)
Jenkins …
![Page 40: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/40.jpg)
4040
What is Jenkins
Tool to schedule and monitor the execution of repeated jobs
![Page 41: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/41.jpg)
4141
Why Jenkins ?
Configurability↘ Different types of input parameters
↘ Invoke shell scripts
↘ Post-build actions (automatic/manual)
![Page 42: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/42.jpg)
4242
Why Jenkins ? cont.
Plugin support↘ More than 600 plugins (https://wiki.jenkins-ci.org/display/JENKINS/Plugins)
↘ Eg. vSphere plugin (stop/start, snapshots, rollbacks…)
↘ Build pipeline plugin
↘ Parameterized remote trigger plugin
![Page 43: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/43.jpg)
4343
Why Jenkins ? cont.
Keeps all your console logs at a single place↘ No need to hunt for 10 log files on 5 different machines
↘ Visual representation of passed/failed/in-progress status, based on downstream shell scripts or other jobs
![Page 44: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/44.jpg)
4444
Why Jenkins ? cont.
And it’s…
![Page 45: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/45.jpg)
MCO
Rake API
DB FE-B FE-* FE-B BE-*
Source code, liquibase
change sets
![Page 46: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/46.jpg)
4646
Jenkins – Puppet Integration
![Page 47: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/47.jpg)
4747
Jenkins – Puppet Integration cont.
![Page 48: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/48.jpg)
4848
Jenkins – Puppet Integration cont.
![Page 49: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/49.jpg)
4949
Jenkins – Puppet Integration cont.
![Page 50: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/50.jpg)
5050
Jenkins – Puppet Integration cont.
Jenkins invoke local bash scripts, which in turn use SSH to call;↘ MCO (as ‘peadmin’ user on Puppet Master)
↘ Rake API (as ‘puppet-dashboard’ user on Puppet Master)
SSH login as ‘peadmin’ and ‘puppet-dashboard’ is password-less, using PKI↘ Generate RSA keypair for the local Jenkins user, using ssh-keygen command
↘ Append the public key to ~/.ssh/authorized_keys file of ‘peadmin’ and ‘puppet-dashboard’ users, on Puppet Master
MCO special purpose sub commands we use;↘ puppet
↘ service
↘ shellcmd* (ask your Puppet Enterprise Support for this custom MCO plugin)
![Page 51: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/51.jpg)
5151
Links
Rake API: https://docs.puppetlabs.com/pe/latest/console_rake_api.html
MCO: https://docs.puppetlabs.com/mcollective/reference/basic/basic_cli_usage.html
Jenkins: http://jenkins-ci.org/
Liquibase: http://www.liquibase.org/documentation/index.html
![Page 52: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/52.jpg)
5252
Recap/Takeaways… Use Puppet Enterprise
↘ Support is awesome (Celia Cottle, Jay Wallace, Ken Johnson, Zachary Stern – you guys rock!)
↘ Gotten help and features from James Turnbull and Nigel Kersten with some early versions of PE
↘ Live management and Mcollective are essential for any self-respecting enterprise
Zero-downtime upgrades↘ To Dashboard or not to Dashboard?
↘ Database update phases
↘ Managing LB health check monitors dynamically using Puppet
Automation baby steps – don’t boil the ocean↘ Understand what you are doing before automating it - develop runbooks
↘ Identify manual steps and script some of them
↘ Add scripts to orchestration tool (Jenkins, ServiceNow, whatever else you use in-house)
![Page 53: Puppet Camp Boston 2014: Orchestrating Infrastructure Change Using Puppet Rake, mcollective, LM and Jenkins (Intermediate)](https://reader034.fdocuments.us/reader034/viewer/2022042814/5554c7e5b4c90559398b537b/html5/thumbnails/53.jpg)
Thank you.