Puppet Release Workflows at Jive Software

Puppet Release Workflows at Jive

Devon Peters

2 © Jive confidential

What is a Release?


Push to production, always, immediately


Releases…

•  Software Release Cycle –  “A software release life cycle is the sum of the stages of development and

maturity for a piece of computer software: ranging from its initial development to its eventual release…” – Wikipedia

•  Release Management –  “the process of managing software releases from development stage to

software release” – Wikipedia

•  Release Engineering –  “Automation of release management throughout all stages of release” – Me


More Dev in our Ops

OBLIGATORY

DEVOPS

IMAGE

7 © Jive confidential 7 © Jive confidential

We have formal release management processes for all of our puppet Domains


Different Releases for Different Domains

•  Core puppet –  All infrastructure –  Many shared services

•  Hosted puppet

–  Hosted Jive installations –  Puppet is tied closely to Administration tooling

•  Continuous Deployment –  New-style services –  Some infrastructure


Releasing Core Puppet


Core Puppet

•  All core infrastructure systems •  Most shared services •  Puppet+hiera does everything on these systems (very few

exceptions) •  Simple deployment environment breakdown

–  Dev, QA, Production (multiple production environments) •  Dynamic puppet environments

–  ‘production’ puppet environment is default –  ad-hoc environments are used for testing/staging

•  Use git flow branching strategy


Git Details

•  Wrappers to simplify and standardize common tasks –  j-tech (j-new, j-hotfix, j-commit, many more) –  Tied into Jira and Crucible –  Feature branches are named after Jira tickets

•  Code Deployments –  Dev/QA: develop is deployed to ‘production’ puppet environment –  Prod: master is deployed to ‘production’ puppet environment


Testing Changes

•  Automated –  Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg

•  Local –  Vagrant VM –  Push branch out to dev puppet master

•  Environment Specific –  Push branch to env-specific puppet master


Merging Changes

•  j-review –  Submit a code review of your branch

•  j-commit –  Merges feature branch to develop –  Merges hotfix branch to master and develop

•  develop branch is deployed to Dev on commit –  jenkins

•  develop branch is deployed to QA Mon-Fri @ 10am –  jenkins

•  If it’s not a hotfix it won’t go to production yet… –  Technically, this because the change isn’t in the ‘master’ branch yet, but

there’s more to it than that


Change Control for Production

•  Bi-weekly CC meetings –  Monday and Thursday

•  Puppet changes go through CC process –  Hotfixes can be promoted outside of CC process –  Weekly change windows for high-impact changes

•  If it’s a puppet change, it’s done as a hotfix

•  Puppet release started every Thursday @ 4pm –  j-release -S: starts a release branch –  Jenkins runs this, and generates a CCR ticket with all commits –  Changes are reviewed in Monday CC meeting

•  Puppet release finished every Tuesday –  j-release -F: finishes a release branch (manual) –  Jenkins code deployment jobs are triggered manually


Core Release Overview


Releasing Hosted Puppet


Hosted Puppet

•  Nodes that run hosted customer installations •  Very homogenous •  Relatively simple puppet code

–  Puppet mostly supplements an in house administration tool (JCA) •  Deployment environment breakdown

–  Dev, QA, Prod •  Uses static puppet environments

–  ENC dictates the environment for a given node –  Jenkins jobs deploy from git to appropriate puppet environment

•  Uses the git flow branching strategy –  All the same j-tech tools


Testing Changes

•  Automated –  Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg

•  Get your own installation setup in Dev

–  Commit changes, and don’t walk away till you know they work •  It’s somewhat acceptable to break dev, but try not to

–  Most OS related changes are just plucked from Core, and were likely tested more thoroughly there


Merging Changes

•  j-review –  Submit a code review of your branch

•  j-commit –  Merges feature branch to develop –  Merges hotfix branch to master and develop

•  develop branch is deployed to Dev on commit –  jenkins –  make sure you don’t break it

•  develop branch is deployed to QA ad-hoc –  QA changes are tied to JCA application release QA cycles –  jenkins


Creating Releases

•  Puppet releases are typically tied to JCA application releases –  The app releases every 2 weeks –  A puppet release could be a part of it

•  Doing a release –  j-release -A: create, and finish a release branch


Staged Production Deployments

•  UAT –  UAT nodes use the same puppet infrastructure as production –  Jenkins deploys master to the ‘hosted_uat’ puppet environment

•  Production –  Jenkins deploys master to the ‘hosted’ puppet environment


Hosted Release Overview


Continuous Deployment


CD Overview

•  Deployable –  Framework for deploying java (and other) services

•  Many java services (over 70) •  Data Infrastructure

–  Kafka, Hadoop, HBase, SenseiDB, Elasticsearch •  Other Infrastructure

–  Puppet, Nginx, Sensu, OpenTSDB •  Gerrit

–  Code reviews are mandatory –  Branch strategy is still git flow style

•  Complex Puppet Code •  Complex Deployment Pipeline


Environment Overview

•  We call them clusters –  virtual, dev, integ, test, release, preprod, prod

•  We also have geo-specific datacenters –  local, intinteg, inttest, intrelease, phxpreprod, phxprod, amsprod

•  Hiera hierarchy includes all of these –  These exist in hiera for other Puppet domains as well

•  Deployable configuration hierarchy includes all of these


Puppet Details


Puppet Overview

•  Dedicated puppet master(s) per cluster •  Puppet agent is run on-demand by the deployment process •  CD pipeline determines if puppet code can be promoted to the next

cluster •  “Special” module and hiera trees

–  We don’t want to duplicate everything in Core –  Developers need the ability to change puppet code or hiera data –  We setup something we call Puppet for Projects

•  Combines Core puppet code with Project puppet code


Puppet for Projects - Layout

•  Core puppet –  Basic layout is:

•  hiera/ •  manifests/ •  modules/

–  manifests/site.pp is basically: hiera_include(‘classes’) –  Every commit triggers an artifact build job (jenkins)

•  Artifacts are uploaded to a Nexus repo, as puppet-0.0.1-<count>-<committish>

•  Project puppet –  A project is basically a repo –  The project repo contains the following directories:

•  puppet/hiera •  puppet/modules

–  Contains maven configuration for Core puppet artifact and Combined puppet artifact


Puppet for Projects – Configuration

•  puppet.conf –  modulepath = /path/modules:/path/project/modules

•  hiera.yaml –  project/%{some-hierarchy} –  %{some-hierarchy}


Puppet for Projects - Artifact

•  Combined artifact –  A commit to the puppet code in the project triggers a combined artifact build –  Artifact contains:

•  hiera/ (from Core artifact) •  hiera/project (from puppet/hiera in the project repo) •  manifests/ (from Core artifact) •  modules/ (from Core artifact) •  project/modules (from puppet/modules in the project repo)

–  Module Collisions •  If a module with the same name exists in both, the project always wins and the

Core module is excluded from the final artifact


Deployable


Deployable Framework

•  Provides standardized… –  Configuration

•  j-config – CLI or service •  Hierarchical JSON

–  Logging •  log-publisher service, writes to Logstash

–  Metrics •  metric-publisher service, writes to OpenTSDB

–  Monitoring •  Autogenerated sensu checks

–  Deployment •  Supports multiple run phases

–  Service Management •  j-status, j-start, j-stop


Puppet as a Deployable

•  puppetmaster-deployable –  Target: puppet master –  puppet tree is packaged into a <release-ver> artifact –  Artifact is deployed to /etc/puppet/environments/jive_<release-ver> on the

puppet master(s) •  puppet-deployable

–  Target: all systems –  j-start executes: puppet agent --environment jive_<release-ver> –  If puppet fails, the deploy fails and stops

•  <release-ver> is always the version of the artifact being built/deployed –  0.0.1-<count>-<committish> –  The deployment process converts the string to a puppet safe string

•  0_0_1_<count>_<committish>


Deployment Run Phases

•  ops-tools –  Deploy j-tech first so everything else will work

•  puppetmaster –  Get out teh codes

•  puppet –  Run puppet –  Includes Hadoop, HBase, Zookeeper, Elasticsearch

•  pre –  Deploy core/base services –  Includes Kafka, and SenseiDB

•  Main –  Everything else


Making a Puppet Change


Making a Puppet Change – Virtual

•  Vagrant VM –  It’s big

•  minimum 4CPU, 8GB – for the VM alone –  Full stack gets deployed and run

•  Hadoop, Kafka, etc, etc, etc – even all 70+ services if you want –  j-vm -r

•  Fetches proper Core puppet artifact •  Builds and deploys a puppet tree for vagrant, based on your local git repo •  Executes vagrant puppet provisioner

–  Once it works on the vm, submit a review to Gerrit


Making a Puppet Change – Integ

•  Once the review is submitted, jenkins will: –  Build Artifacts –  Launch a VM-test job

•  Validating that what you did works on a vagrant VM there –  Launch an integ deployment

•  Validating that things work in a multi-node non-vagrant environment –  Comment on your review with Validated +1, or -1 (depending on results) –  Once other reviewers give you +2 Gerrit will merge your commit –  Once it’s merged…


Making a Puppet Change – Test

•  Once the commit is merged, jenkins will: –  Build artifacts –  Launch a Test deployment –  Run more extensive tests –  If the deployment and all tests pass, it’s ready for the next step


Making a Puppet Change – Release

•  Daily at 8am all commits that have passed Test are merged from develop to master

•  Once this happens, jenkins will: –  Build artifacts –  Trigger a Release deployment –  Rerun all of the tests –  If all tests pass…


Making a Puppet Change – Preprod

•  If all tests pass, jenkins will: –  Promote artifacts to preprod Nexus repo –  Trigger a Preprod deploy –  At this point, everything should be stable


Making a Puppet Change – Prod

•  During the next scheduled production release, someone will: –  Trigger a Prod deploy –  Currently done manually


That’s about it…


Review

•  Core –  Complex puppet code to manage everything –  Releases tied to Change Control –  ~1000 nodes

•  Hosted –  Relatively simple puppet code –  Releases tied to Administration tool’s application releases –  ~14000 nodes

•  Continuous Deployment –  Complex puppet code –  Fully Automated Release and Deployment –  <200 nodes (but growing)


Thank You!


Questions?

Puppet Release Workflows at Jive Software

Technology

Transcript of Puppet Release Workflows at Jive Software