Virgil Chereches & Robert MirceaMay 28th, 2015

Our Journey toContinuous Delivery

2

“How long would it take your organization to deploy a change that

involved just one single line of code?

Do you do this on a repeatable, reliable basis?”

Mary and Tom Poppendieck, Implementing Lean Software Development

3

“Failure is not an option”

Learning opportunity

Detectable quickly

Restore service quickly

Failure is inevitable

Orange

4

IT Performance and DevOps Practices

“Our data shows that IT performance and well-know DevOps practices, such as those that enable continuous delivery, are predictive of organizational performance.

As IT performance increases, profitability, market share and productivity also increase.

IT is a competitive advantage, not just a utility, and it’s more critical now than ever to invest in IT performance.”

http://bit.ly/1LwY4oB

5

The Need for Continuous Delivery

Agility for our Business

Eliminate wasted time due to hand offs between teams

6

Enable Architectural Decomposition of Systems

7

Do Continuous Delivery to be Ready for…

Microservices

Swarm

8

Share the Vision of CD

Spread

the “WHY”

the “WHAT”

ensure they “GET IT”

Started organic, inspired by a few enthusiasts.

Understand that it is about changing the way we work

9

How We Joined the Army

Dev Ops

Enthusiasts

7 x DevOps Wannabes

One project

One opportunity

R&D style

Managers jumped on

board from the start

DevOps

10

What We Want to Achieve

Push to release

at Orange scale

11

JBoss cluster

standalone

JBoss cluster

standalone

v1.0 v1.1

JBoss cluster

standalone

v1.1

…

VMWare VCenter Infrastructure

JBoss cluster

standalone

v1.0

ProdDev/UAT

v1.1

commit

https://www.orange.ro/myaccount

12

From What We Start

Business

requests in the

hundreds/year

200+ apps in

Java, C#, C, a

few PHP. Lots of

PL/SQL

~150 people –

Dev, QA, BA,

Ops, PM

Mélange of

proprietary and

lots of custom

development apps

13

What is Continuous Delivery?

Small, frequent changes delivered constantly

A method to reduce risk of failure

A way to give to our business a glimpse of the shape of product being built

A set of practices enabled by a toolset

14

Many deployments eventually lead to a Product Launch

What is the difference between

a Deployment and a Product Launch?

15

Changes flowing through deployment pipeline

Push the

button

“Approval”

16

Orange Deployment Pipeline Architecture

(Atlassian Stash Git / SVN)

(Artifactory)

(Jenkins/Teamcity/Sonar) (Selenium)

(Enterprise Tester)

(JMeter)

Rundeck

/Puppet

17

Best Practices for CD

build application only once and use it in all environments

keep application releasable all the time – master is releasable at all times

hide new functionality until it is finished – use feature flags

make small changes incrementally – commit and integrate often

use snapshots (in Maven) with care

18

Techniques for achievingContinuous Deployment

19

Blue-Green Deployments

v1.0-WL

App Server

v1.0-Jboss

www.orange.ro/myaccount-corporate

Weblogic

JBoss

20

Feature flags

Integrating with

production is a test

On/Off

Orange IPs

List of Users

Time interval

Release date

Gradual Rollout (%)

Blacklist/whitelist IPs

Enable/disable features

quickly (no restart)

ORO testers check feature

in live

21

Prefer Non-Breaking Expansion in Database

Database migration tools

FluentMigrator (.NET)

22

Code Based Migration - FluentMigrator

Run migration

.\migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a

"..\Migrations\Migrations.dll" --task migrate:up

Rollback change

.\migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a

"..\Migrations\Migrations.dll" --task migrate:down

23

Automated tests AFTER deploy

Run smoke tests automatically to validate deployment success

Examples:

make a HTTP request to homepage and look for a string

login with a test user

make a recharge with a test PrePay

24

Metrics

Server Metrics

CPU utilization

disk space

Application Metrics

JMS messages sent/received,

memcached hits

Business Metrics

Logins / Registration

Orders

Recharges

Options activations

25

Naming Metrics

Establish consistent rules in naming metrics

EnvApp name

Server

name

Module

name

Message

type

Target

Action

prod.apps.maco.platini.jms.vantive.customers.request.sent

26

Graphite & Grafana - real-time metrics

27

Instrumenting Code for Metrics Collection

Trivial counters with Statsd (https://github.com/etsy/statsd/)

Types of counters:

Counters

Timers

Gauges

Histograms

28

Centralized Logging

Apache Logs

App logs

Logstash

LumberjackLogstash

Log shipping

Logstash

(parse log into fields)

Elasticsearch

(index logs)

Statsd + Graphite

(metrics dashboard)

(log search & analysis)

29

Infrastructure Automation

30

“At an abstract level, a

deployment pipeline is an

automated manifestation of

your process for getting

software from version control

into the hands of your users.”

Deployment Pipeline

31

Provisioning managers

Configuration managers

Workflow managers

32

Configuration Manager Choice

33

Software components

HIERA

34

Coding best practices

Data/code separation

r10k

HIERA

Keep code in version controlreaktor

Unit & acceptance testing

Development workflow feature branch

workflow

puppet-rspec

server-spec

35

Node breeds

Several items give

node breed: puppet role class

Foreman host

group (one-to-one

relationship)

stage

(dev/prod/qa/test)

Foreman compute

profile (compute

provider, CPUs,

RAM, Disk)

36

Tag-based everything

Nodes Workflow

stepsSecurity

37

Node herds

Pre-provisioned nodes

Shorten

deployment

time

Breed

matters

Hostname

does not

matter

38

Don’t be afraid to get dirty

Rundeck API Foreman API PuppetDB API

BigIP iControl Atlassian Stash API

39

Scaling Puppet & Foreman

40

What’s Next

• Sponsor Rundeckdevelopment for SCM integration and sensitive options encryption

• Evaluate network orchestration

• Containers

• Acceptance testing of puppet code with beaker (rspec+serverspec)

multumim