DTO #ChefConf2012

Using Rundeck and Chef to build DevOps

ToolchainsAnthony Shortland - [email protected]

Moses Lei - [email protected]

DTO Solutions

Process Improvement

Design

Business

Release Engineering

Development

QA

Operations

Program Management

L/T Lead timeP/T Process timeH/C Head countS/R Scrap rate

D DefectsEP Extra processesM Motion

PD Partially doneTS Task switchingW Waiting

Service Delivery Process Current state value stream map

DevelopmentEnvironments

ReleaseBuild and deploy

QAEnvironments

PerformanceEnvironment

StagingEnvironment

ProductionEnvironment

Merge project branch

Deploy to Performance

Web Assets, flows and mockups

Features and fixes

Set of service

branches

Documented test plans and cases

Request QA build

Daily Scrum meeting

Daily Scrum meeting

Deployment Meeting Business Systems

Support

Stan Walters

Change Control

Project Management

Eric Switzer

Change request

BusinessAnalysis

Luke Hanson

Full Projects / Expedites

Estimates and

requirements

Technical Specification

Deployment Document

Change Control Forms

Bugzilla issues

Jira issues

Jira issues

Tested packages

Development

Joe Harbaugh

L/T 10 weeksP/TH/CS/R

6 weeks438%

Developweb assets

Bob McNulty


6 weeks158%

DevelopmentBuild and deploy

Joe Harbaugh

L/T 1 daysP/TH/CS/R

2 hours220%

Development Integration testing

Joe Harbaugh

L/T 2 dayP/TH/CS/R

2 day1196%

Performance testing

Deploy toProduction

Build test plan

Raj Lee

L/T 8 daysP/TH/CS/R

4 days230%

Build test cases

Raj Lee


8 days220%

Fernando Gomez

L/T 11 hoursP/TH/CS/R

8 hours25%

Acceptance testing

Raj Lee

L/T 5 daysP/TH/CS/R

8 hours210%

Feature testing

Raj Lee


3.5 weeks520%

Allen Cannata

L/T 1 weekP/TH/CS/R

3 days166%

RT tickets

Cut project branch

Chris Holmes

L/T 20 minutesP/TH/CS/R

8 minutes20%

Deploy to Staging

Frances Middleton

L/T 7 daysP/TH/CS/R

110 minutes470%

Staging testing

Frances Middleton

L/T 4 daysP/TH/CS/R

3.5 days566%

Allen Cannata

L/T 10 hoursP/TH/CS/R

7 hours550%

Frances Middleton

L/T 6 daysP/TH/CS/R

90 minutes366%

Fernando Gomez

L/T 1 hourP/TH/CS/R

30 minutes150%

Program Committee

Eric Switzer

5 weeks

3 weeks

5 weeks

5 weeks

5 hours

2.5 days

3 day 8 hours 6 hours

2 hours 5 days

2 weeks

2.5 weeks

110 minutes

6.75 days

4.5 days

2.5 days

9 hours

2 hour

Process Time = 59 days

Lead Time = 136 days

D

EP

DEPx2

Mx3W

PDD

Dx3Mx3

TSx5 Dx2 Wx2 PDx3

W

DPDx4 Mx7

WDx2

Mx3PD

WW

Estimates and

requirements

TechnicalAnalysis

Maya Highsmith

Full Projects / Expedites

Test Requirements

Luke Hanson

L/T 21 daysP/TH/CS/R

3 days250%

EP

EP

Value Stream

65% +5%

7 Days 3h, -5h

Delivery Cycle Time

Value-Creating Time (VCT)

Metrics: 1 week, 1 week historical

Scott, Erica, Reigner,…

2 versions4 environments8 15m2 Hours

-30m

Provision Lead Time

85%, -5%

Stability

Environment: Production

5% 1/20

5Delivery Support

Deployments

+2 environments +1 versionProvision Scrap Rate

20% 2/10

Delivery Scrap Rate

-3+1

RetailTrader - v133 "Add logic updates " 4 Days Remain 10d total

4/16 Open

Work Tasks

15 Closed

Build Success

OK retailtr-v133.war

80% 4/5-5%

-5%

10% Support

Team Utilization

34m avgLead Time

MilestoneProduct

Features

Process Automation

• End-to-end automation• Supports business goals• Optimize the system, not the parts

Take a toolchain approach

Unix Tool Chain

Measurement Instrumentation

System

Charting, History, SPC

Eventing, Alarm Mgmt

Control

Runbook Automation

Industrial Control Automation

Introduction

E-commerce and software-as-a-service business models have matured quickly, but the

quality of the web operations that support these businesses has lagged behind.

Outages are all too common. High variability and defect rates are bemoaned but have

become an accepted reality. Key engineers spend all day (and sometimes all night)

mired in deployment issues and bottlenecks. And topping it all off, what tooling that

does exist are usually a custom one-offs that are brittle and expensive to maintain.

Today’s business of operating software over the Web as a revenue producing service is a

dramatic departure from the days when software was primarily produced for delivery on

physical mediums and IT Operations was considered a back-of-the-house support

function. Shouldn't we be completely rethinking our tooling and operational capabilities

to match these new innovations?

In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated

approaches, and low expectations -- and into Web Operations 2.0 where tools and

procedures are built from the ground up for highly efficient, reliable, and agile

operations.

There are multiple factors that go into achieving excellence in Web Operations, but the

linchpin that holds it all together is a fully automated provisioning system.

In this paper we will be:

1. Defining what we mean by "fully automated provisioning"

2. Explaining why virtualization and cloud computing efforts fail without fully

automated provisioning capabilities

3. Proposing a reference open source tool chain for fully automated provisioning

4. Describing a live implementation where a leading online retailer is actively rolling

out a fully automated provisioning system using all open source tools

2 Web Ops 2.0: Achieving Fully Automated Provisioning

” Today’s business of

operating software over the

Web as a revenue producing

service is a dramatic

departure from the days

when software was primarily

produced for delivery on

physical mediums...”

+

Service Delivery Platform Elements

Design points1. Everything is code

2. Everything is packaged

3. Separate code and configuration

4. Separate environment-dependent attributes

5. Balance distributed vs local orchestration

6. Resolve directed vs convergent orchestration

7. Integrate application and infrastructure provisioning

8. Design for flow not the organization

Example Service Delivery Platform

Integration Plugins • Jenkins/GitHub - https://wiki.jenkins-ci.org/display/

JENKINS/Github+Plugin

• Jenkins/Rundeck - https://wiki.jenkins-ci.org/display/JENKINS/RunDeck+Plugin - Vincent Behar

• Nexus/Rundeck - https://github.com/vbehar/nexus-rundeck-plugin - Vincent Behar

• Nexus/Yum - http://code.google.com/p/nexus-yum-plugin/

• Chef/Rundeck - https://github.com/opscode/chef-rundeck - Adam Jacob

Example Service Delivery Platform Layout

Development ServerDeploy ServerBuild Server

Source CodeBuild ArtifactRepository ClientServer

Deployment ConsoleBuild Console

Application Server Database

Develop

Build - Projects by artifact

Build - Trace by build number

Build - Template RPM SPEC

[ec2-user@centos-ami bank]$ cat SPECS/dukesbank-ear.spec.template Summary: Duke's Bank ear fileName: dukesbank-earVersion: ${version}Release: ${release}

License: LGPLGroup: Applications/System

Prefix: /opt/jboss-4.0.3SP1BuildArch: noarchRequires: jboss = 4.0.3SP1

.

.

.

Build - Register packages

.

.

. <target name="rpmdeploy" depends="resolve,rpmbuild"> <ivy:makepom ivyfile="${basedir}/ivy.xml" pomfile="RPMS/noarch/dukesbank-ear-${version}-${release}.noarch.pom"/> <ivy:publish resolver="nexus-deploy" pubrevision="${version}-${release}" overwrite="true" publishivy="true"> <artifacts pattern="RPMS/noarch/[artifact]-[revision].noarch(-[classifier]).[ext]"/>  </ivy:publish> </target>...

</project>

Repository - Build view

Repository - Deploy view

Deploy - Package version list[root@centos-ami yum.repos.d]# cat /etc/yum.repos.d/dukesbank.repo [dukesbank]name=DukesBank demo repobaseurl=http://ec2-75-101-135-205.compute-1.amazonaws.com:8081/nexus/content/repositories/releasesenabled=1gpgcheck=0metadata_expire=1s[root@centos-ami yum.repos.d]# repoquery --repoid=dukesbank --search dukesbank-eardukesbank-ear-0:1.0-14.noarchdukesbank-ear-0:1.0-15.noarchdukesbank-ear-0:1.0-16.noarchdukesbank-ear-0:1.0-18.noarchdukesbank-ear-0:1.0-19.noarchdukesbank-ear-0:1.0-20.noarchdukesbank-ear-0:1.0-21.noarchdukesbank-ear-0:1.0-22.noarchdukesbank-ear-0:1.0-23.noarchdukesbank-ear-0:1.0-24.noarchdukesbank-ear-0:1.0-25.noarchdukesbank-ear-0:1.0-26.noarch[root@centos-ami yum.repos.d]# cat ~jenkins/jobs/dukesbank-ear.rpm/workspace/versions.json [ "1.0-14", "1.0-15", "1.0-16", "1.0-18", "1.0-19", "1.0-20", "1.0-21", "1.0-22", "1.0-23", "1.0-24", "1.0-25", "1.0-26", "1.0-27", "1.0-28", "1.0-29", "1.0-30", "1.0-31", "1.0-32", "1.0-33", "1.0-34" ][root@centos-ami yum.repos.d]#

Deploy - Nodes & roles

Deploy - Node provider[ec2-user@ip-10-36-119-128 ~]$ curl http://ec2-75-101-135-190.compute-1.amazonaws.com:9980/<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd"><project><node name="ip-10-195-11-159.ec2.internal" type="Node" description="ip-10-195-11-159.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="_default,base,chef_server" username="ec2-user" hostname="ip-10-195-11-159.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-195-11-159.ec2.internal/edit"/><node name="ip-10-36-119-128.ec2.internal" type="Node" description="ip-10-36-119-128.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="dev,base,dukesbank_database_master,dukesbank" username="ec2-user" hostname="ip-10-36-119-128.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-36-119-128.ec2.internal/edit"/></project>

Deploy - Application attributes[ec2-user@centos-ami chef-repo]$ cat data_bags/apps/dukesbank.json { "id": "dukesbank", "server_roles": [ "dukesbank" ], "type": { "dukesbank": [ "jboss", "jboss_webapp" ] }, "database_master_role": [ "dukesbank_database_master" ], "packages": { "dev": { "dukesbank-ear": "1.0-23" } }, "jboss_state": { "dev": "start" },...

Links

• DTO Solutions - http://dtosolutions.com• DTO Labs - http://dtolabs.com• DTO Labs Github - https://github.com/dtolabs • Rundeck - http://rundeck.org• Devops-Toolchain -

http://groups.google.com/group/devops-toolchain

DTO #ChefConf2012

Business

Transcript of DTO #ChefConf2012