Evolving Infrastructure

Evolving InfrastructureLouis Dunne, Platform Engineering

[email protected]

Workday Introduction• Background / Our Own Cloud

2.0 Architecture• Config Management / Chef Build Pipeline /

Chef Development Tools

3.0 Architecture• Release & Deployment Changes / Image Build Pipeline /

Release Management / Planned vs. Unplanned Changes

Enterprise HCM & Financials software

in the cloud

First releaseNovember 2006

One of the DevOps mantras is

Infrastructure is Code

If it’s code…

→ you need to test it

→ you need a build & test pipeline

Local Development

CodeReview

C.I.Unit Tests

All triggered by developers

pushing code

Bronze Cookbook Artefacts

Local Development

CodeReview

C.I.Unit Tests

Local Development

CodeReview

C.I.Unit Tests

C.I.System Tests

C.I.System Tests

C.I.System Tests

Failures(back to dev)

System TestsTriggered on

the Hour

Bronze Cookbook Artefacts

Silver Cookbook Artefacts

C.I.Integration Tests

Gold Cookbook Artefacts

Silver Cookbook Artefacts

Failures(back to dev)

Integration TestsTriggered Several

Times a Day

reekChefSpec

RSpec

System &Integration

reekChefSpec RSpec

Unit Testing

Where Do We Run Our ServerSpec Tests?• Lab Hardware?• Vagrant?• AWS?

The Lab?• Hardware Based• Can’t set machine state before the test run• Can’t reset machine state after the test run

Run In Vagrant?• Good for simple cases• Harder for integration testing a few dozens

Chef roles• Prefer a hosted platform with longer running

nodes for some services like artefact repos

Unit System Integration

https://www.chef.io/delivery/

https://downloads.chef.io/chef-dk/

Platform Services

Appliances

Workday

Linux Servers

Image Based Deployment

Chef / CobblerBased Deployment

■ Cobbler for the OS■ Chef based deployments

of system / infrastructure changes

■ Custom tooling for applications deployments

■ Cobbler / Chef for bare metal

■ Most services moving to image based deployments

■ Custom deployment tools to manage VM lifecycle

2.0 Deployments 3.0 Deployments

1. Where in the build & test pipeline do the Machine Images get created?

2. What technology & processes are used to create them?

• Early in the pipeline

• Application teams → image artefact

• Image artefact → build & test pipeline

• Lots of tools to choose from:

• Diskimage-builder

• VMBuilder

• Box Grinder

• Packer

• Imagefactory

• We use Oz (https://github.com/clalancette/oz)

KickStartFile

OZTemplate

Base Image

OZCentOS Core

Image Manifest

Unit + SystemTests

System RPMs

OZTemplate

Application Image

OZ

Base Image

Image ManifestApplication

RPMs

Image Build Service

Gold Cookbook Artefacts

Gold Application

Artefacts

Promoteto Staging

UNIT SYSTEM INTEGRATION

UNIT INTEGRATION

I N F R A S T R U C T U R E

A P P L I C A T I O N SSYSTEM

Promote to Production

Image Build Service

SYSTEM TESTS

UNIT INTEGRATION

I N F R A S T R U C T U R E

SYSTEM

Promoteto Staging

A P P L I C A T I O N S

Base Image

Base Image

Application RPMs

Application Image

Application Image

The ability to push code to environments easily and quickly - push button deploys

A stable framework for development, testing, deployment and auditing

Deploy applications and operational environments in the same way each time

Source code, tools and all components that make up releaseIdentifiability

Reproducibility

Consistency

Agility

■ Images■ Startup Properties

■ Cookbooks■ Roles■ Data Bags■ Encrypted Data Bags■ RPMs■ Other artefacts (e.g. ruby

& python packages)

Config Management Image Deployment

■ Destroy and recreate rather than change in place

■ All facets of the OS are captured by the image artefact

■ Server state mutated over time as updates are applied

■ Impractical to manage every last detail of the OS with config management

Config Management Image Deployment

• The ideal is…• Push all changes through the image pipeline

for both planned and unplanned changes• If your pipeline is reliable with a quick

turnaround you can use this for all changes• We also like to have a Break The Glass option

Planned Vs Unplanned Changes

• For low impact changes only• Bash script → RPM• Agent on each VM can deploy RPMs• Trigger remotely via secure channel• Testable, easy to roll out, good for auditability

Break The Glass

MonitoringDeployment

(*)

(*) developed in house

Stats & MetricsLogging

“If it hurts,do more of

it”

http://www.beatcleaver.com/portfolio/https://www.flickr.com/photos/cote/ DC Image

Melissa StolbergMichael CotéPaul McAuley

Evolving Infrastructure

Software

Transcript of Evolving Infrastructure