Continuous Deployment of Architectural Change
-
Upload
matt-graham -
Category
Technology
-
view
1.686 -
download
0
description
Transcript of Continuous Deployment of Architectural Change
![Page 1: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/1.jpg)
Continuous Deploymentof Architectural Change
Matt GrahamCore Engineer @ EtsyContinuous Deployer
#iasanySeptember 27, 2011
![Page 2: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/2.jpg)
A Brief History of Deployment
![Page 3: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/3.jpg)
The Internet
![Page 4: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/4.jpg)
Web Applications
![Page 5: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/5.jpg)
Agility
![Page 6: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/6.jpg)
Continuous
![Page 7: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/7.jpg)
Database Deployment @ Etsy
Thursday
![Page 8: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/8.jpg)
What it's all about
● Broadly Applicable● Tools Help● Enables the Unfeasible
![Page 9: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/9.jpg)
MTTR vs MTTF
![Page 10: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/10.jpg)
MTTR vs MTTF
● Cheaper for electrons
![Page 11: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/11.jpg)
MTTR vs MTTF
● Cheaper for electrons
● Cheaper for protons & neutrons
![Page 12: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/12.jpg)
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Cheaper for protons & neutrons
![Page 13: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/13.jpg)
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Necessitates low MTTD
● Cheaper for protons & neutrons
![Page 14: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/14.jpg)
The Case of Electrons
● Consider a project that has 6 bugs● continuous deployment testing misses 4& deploys fixes in 4 hours
● monthly release testing misses only 2& deploys fixes in 24 hours
● 16 hours of broken < 48 hours of broken
![Page 15: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/15.jpg)
Good Excuses
● Infrequent Changes
![Page 16: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/16.jpg)
Good Excuses
● Infrequent Changes● Infrequent Executions
![Page 17: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/17.jpg)
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death
![Page 18: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/18.jpg)
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment
![Page 19: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/19.jpg)
![Page 20: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/20.jpg)
![Page 21: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/21.jpg)
![Page 22: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/22.jpg)
![Page 23: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/23.jpg)
![Page 24: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/24.jpg)
![Page 25: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/25.jpg)
![Page 26: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/26.jpg)
Deployment Tools @ Etsy
![Page 27: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/27.jpg)
Culture Before Tools
● No planned deploys; only on demand
![Page 28: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/28.jpg)
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
![Page 29: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/29.jpg)
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
● Software is stable & supported
![Page 30: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/30.jpg)
Jenkins
● Unit Tests
![Page 31: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/31.jpg)
Jenkins
● Unit Tests● Functional Tests
![Page 32: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/32.jpg)
Jenkins
● Unit Tests● Functional Tests● Manual Testing
![Page 34: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/34.jpg)
tail -f | grep
![Page 36: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/36.jpg)
IRC
![Page 37: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/37.jpg)
Key Tools
Communication Tools
Graphs
Feature Flags
![Page 38: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/38.jpg)
Feature Flags
Deploy != Product Launch
![Page 39: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/39.jpg)
Dark Launch
def get_feature_a_link():
return ...
![Page 40: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/40.jpg)
Dark Launch
def get_feature_a_link():
if enabled('NewFeatureA'):
return new_feature_a_link()
else:
return old_feature_a_link()
![Page 41: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/41.jpg)
Dark Launch
application_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on
![Page 42: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/42.jpg)
Ramp Up
application_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on
![Page 43: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/43.jpg)
Ramp Up
application_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on
![Page 44: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/44.jpg)
Ramp Up
application_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on
![Page 45: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/45.jpg)
Ramp Up
application_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on
![Page 46: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/46.jpg)
AB Testing
● Prove success of interface changes
● Prove interest in new features
![Page 47: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/47.jpg)
Graphs!!!
![Page 48: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/48.jpg)
Ganglia
![Page 49: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/49.jpg)
Graphite
![Page 50: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/50.jpg)
Event Overlay
![Page 51: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/51.jpg)
StatsD
github.com/etsy/statsd
if ($success) { StatsD::timing('feature.runtime', $time);} else { StatsD::increment('feature.failure');}
![Page 54: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/54.jpg)
Community Communication
![Page 55: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/55.jpg)
Forums / Message Boards
![Page 56: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/56.jpg)
etsystatus.com
![Page 57: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/57.jpg)
twitter.com/etsystatus
![Page 58: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/58.jpg)
twitter.com/etsystatus
![Page 59: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/59.jpg)
Deployment is First Class
Deployment is aFirst Class Feature
![Page 60: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/60.jpg)
Engineers are Users Too
![Page 61: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/61.jpg)
Enable Architectural Change
![Page 62: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/62.jpg)
Architecture is Relative
![Page 63: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/63.jpg)
Organic Architecture
![Page 64: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/64.jpg)
Premature Architecture
![Page 65: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/65.jpg)
Premature Architecture
What now?
![Page 66: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/66.jpg)
Passing Time => Change
● Scale● Product● Technology● Engineering Team
![Page 67: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/67.jpg)
Passing Time => Change
● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes
![Page 68: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/68.jpg)
Examples
● Images From Twisted to PHP● From PostgreSQL to MySQL Shards
![Page 69: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/69.jpg)
From Twisted to PHP
● Run Apache/PHP on a new port
![Page 70: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/70.jpg)
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP
![Page 71: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/71.jpg)
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service
![Page 72: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/72.jpg)
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services
![Page 73: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/73.jpg)
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version
![Page 74: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/74.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table
![Page 75: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/75.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL
![Page 76: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/76.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL
![Page 77: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/77.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs
![Page 78: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/78.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL
![Page 79: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/79.jpg)
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables
![Page 80: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/80.jpg)
Continuous Deploy Pattern
● Change in small steps
![Page 81: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/81.jpg)
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config
![Page 82: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/82.jpg)
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark
![Page 83: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/83.jpg)
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel
![Page 84: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/84.jpg)
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture
![Page 85: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/85.jpg)
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture
![Page 86: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/86.jpg)
Review
● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication
● Enables Architectural Change● It's really fun!!!
![Page 87: Continuous Deployment of Architectural Change](https://reader036.fdocuments.us/reader036/viewer/2022081400/554bb435b4c9053a298b4701/html5/thumbnails/87.jpg)
Continuous Deploymentof Architectural Change
Matt Grahamhttp://twitter.com/lapsu
http://lapsu.tv
Core Engineer @ EtsyContinuous Deployer
http://codeascraft.etsy.comWe're Hiring!
http://bit.ly/etsywantsawesome