Continuous Deploymentof Architectural Change

Matt GrahamCore Engineer @ EtsyContinuous Deployer

#iasanySeptember 27, 2011

A Brief History of Deployment

The Internet

Web Applications

Agility

Continuous

Database Deployment @ Etsy

Thursday

What it's all about

● Broadly Applicable● Tools Help● Enables the Unfeasible

MTTR vs MTTF

MTTR vs MTTF

● Cheaper for electrons

MTTR vs MTTF

● Cheaper for electrons

● Cheaper for protons & neutrons

MTTR vs MTTF

● Cheaper for electrons

● Cost prohibitive for humans

● Cheaper for protons & neutrons

MTTR vs MTTF

● Cheaper for electrons

● Cost prohibitive for humans

● Necessitates low MTTD

● Cheaper for protons & neutrons

The Case of Electrons

● Consider a project that has 6 bugs● continuous deployment testing misses 4& deploys fixes in 4 hours

● monthly release testing misses only 2& deploys fixes in 24 hours

● 16 hours of broken < 48 hours of broken

Good Excuses

● Infrequent Changes

Good Excuses

● Infrequent Changes● Infrequent Executions

Good Excuses

● Infrequent Changes● Infrequent Executions● Life and Death

Good Excuses

● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment

Deployment Tools @ Etsy

Culture Before Tools

● No planned deploys; only on demand

Culture Before Tools

● No planned deploys; only on demand

● A positive change is ready & tested

Culture Before Tools

● No planned deploys; only on demand

● A positive change is ready & tested

● Software is stable & supported

Jenkins

● Unit Tests

Jenkins

● Unit Tests● Functional Tests

Jenkins

● Unit Tests● Functional Tests● Manual Testing

Nagios & Naglite2

github.com/lozzd/Naglite2

tail -f | grep

github.com/etsy/deployinator

IRC

Key Tools

Communication Tools

Graphs

Feature Flags

Feature Flags

Deploy != Product Launch

Dark Launch

def get_feature_a_link():

return ...

Dark Launch

def get_feature_a_link():

if enabled('NewFeatureA'):

return new_feature_a_link()

else:

return old_feature_a_link()

Dark Launch

application_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on

Ramp Up

application_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on

Ramp Up

application_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on

Ramp Up

application_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on

Ramp Up

application_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on

AB Testing

● Prove success of interface changes

● Prove interest in new features

Graphs!!!

Ganglia

Graphite

Event Overlay

StatsD

github.com/etsy/statsd

if ($success) { StatsD::timing('feature.runtime', $time);} else { StatsD::increment('feature.failure');}

github.com/etsy/logster

github.com/etsy/logster

Community Communication

Forums / Message Boards

etsystatus.com

twitter.com/etsystatus

twitter.com/etsystatus

Deployment is First Class

Deployment is aFirst Class Feature

Engineers are Users Too

Enable Architectural Change

Architecture is Relative

Organic Architecture

Premature Architecture

Premature Architecture

What now?

Passing Time => Change

● Scale● Product● Technology● Engineering Team

Passing Time => Change

● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes

Examples

● Images From Twisted to PHP● From PostgreSQL to MySQL Shards

From Twisted to PHP

● Run Apache/PHP on a new port

From Twisted to PHP

● Run Apache/PHP on a new port● Implement one service in PHP

From Twisted to PHP

● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service

From Twisted to PHP

● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services

From Twisted to PHP

● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version

PostgreSQL to MySQL Shards

● Migrate table by table

PostgreSQL to MySQL Shards

● Migrate table by table● Teed writes to MySQL

PostgreSQL to MySQL Shards

● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL

PostgreSQL to MySQL Shards

● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs

PostgreSQL to MySQL Shards

● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL

PostgreSQL to MySQL Shards

● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables

Continuous Deploy Pattern

● Change in small steps

Continuous Deploy Pattern

● Change in small steps● Dark launch placeholder via config

Continuous Deploy Pattern

● Change in small steps● Dark launch placeholder via config● Deploy to production while dark

Continuous Deploy Pattern

● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel

Continuous Deploy Pattern

● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture

Continuous Deploy Pattern

● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture

Review

● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication

● Enables Architectural Change● It's really fun!!!

Continuous Deploymentof Architectural Change

Matt Grahamhttp://twitter.com/lapsu

http://lapsu.tv

Core Engineer @ EtsyContinuous Deployer

http://codeascraft.etsy.comWe're Hiring!

http://bit.ly/etsywantsawesome