Post on 03-Apr-2018
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
1/46
Continuous Deployment at
March 9, 2013
Ross Snyder
ross@etsy.com
@beamrider9
:A Tale of Two Approaches
mailto:rsnyder@etsy.commailto:rsnyder@etsy.commailto:rsnyder@etsy.com7/29/2019 Etsy LeanStartup presentation at SXSW 2013
2/46
A quick primer on
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
3/46
is:
The global marketplace we make together.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
4/46
is:
The premier destination for handmade goods,
vintage items, and craft supplies.
simplertimestoys lacklusterco norwesterseaglass
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
5/46
quick facts:
22+ million members
800,000+ active shops
18+ million items currently for sale
20 cents to list item, 3.5% transaction fee
400+ employees (majority in Brooklyn)
(as of March 2013)
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
6/46
$0
$200
$400
$600
$800
$1,000
2005 2006 2007 2008 2009 2010 2011 2012
Gross Merchandise Sales ($MM)
Since opening its doors in June 2005, Etsy
has grown virtually non-stop.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
7/46
A nice problem to have:
Our site is sosuccessful, how can we
move fast enough to
keep up with demand?
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
8/46
CONTINUOUSDEPLOYMENT
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
9/46
:The Early Years
(2005 - 2008)
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
10/46
: The Early Years
1. Spend significant time writing code
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
11/46
: The Early Years
2. Painful source control merge
1. Weeks writing code
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
12/46
: The Early Years
3. Hand off to someone else to deploy
1. Weeks writing code2. Painful merge
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
13/46
: The Early Years
4. Deploy, site goes down
1. Weeks writing code2. Painful merge
3. Hand off to deployers
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
14/46
: The Early Years
5. Roll back deploy
1. Weeks writing code2. Painful merge
3. Hand off to deployers
4. Deploy, site down
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
15/46
: The Early Years
6. Spend hours (days?) fixing bugs
1. Weeks writing code2. Painful merge
3. Hand off to deployers
4. Deploy, site down5. Roll back deploy
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
16/46
: The Early Years
7. Go back to step 2
1. Weeks writing code2. Painful merge
3. Hand off to deployers
4. Deploy, site down5. Roll back deploy
6. Fix bugs
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
17/46
: The Early Years
WATERFALL!
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
18/46
: The Early Years
Pros:
Early Etsy engineers
used this releasecycle to bootstrap the
marketplace from
nothing.
Forever grateful.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
19/46
: The Early Years
Large changesets
Infrequent deploys
Weak confidence in deploy success
Significant time spent deploying Low ability to experiment/iterate/react
Developer stress/unhappiness
Cons:
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
20/46
: The Early Years
By late 2008, Etsy is still a startup, but has thedeploy process of a much bulkier company.
Popularity is on the verge of outpacing capacity.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
21/46
:
Today
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
22/46
: Today
1. Small changesets, deployed frequently
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
23/46
: Today
2. Engineers deploy the site
1. Small changesets
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
24/46
: Today
And not just engineers, but also:
Designers
Product Folks
Upper Management
Board Members
Dogs
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
25/46
: Today
3. Deploys are fast and near-effortless
1. Small changesets2. Engineers deploy
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
26/46
: Today
4. Most changes behind config flags(safer deploys)
1. Small changesets2. Engineers deploy
3. Deploys are fast
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
27/46
: Today
5. Graphs/metrics to assess deploy
1. Small changesets2. Engineers deploy
3. Deploys are fast
4. Changes behind flags
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
28/46
: Today
6. If issues, fix immediately & roll forward
1. Small changesets2. Engineers deploy
3. Deploys are fast
4. Changes behind flags5. Copious graphs/metrics
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
29/46
This isnt license to break stuff, quickly.
Engineer-driven QA and solid unit testing
are integral parts of the process.
: Today
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
30/46
: Today
7. Repeat 25+ times per day, every day
1. Small changesets2. Engineers deploy
3. Deploys are fast
4. Changes behind flags5. Copious graphs/metrics
6. Fix fast & roll forward
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
31/46
Then:
1. Small changesets
2. Engineers deploy
3. Deploys are fast
4. Changes behind flags
5. Copious graphs/metrics
6. Fix fast & roll forward
1. Weeks writing code
2. Painful merge3. Hand off to deployers
4. Deploy, site down
5. Roll back deploy
6. Fix bugs, go to step 2
Now:
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
32/46
Etsy Deploy Stats: 2012
Deployed to production 6,419 times
On average, 535/month, 25/day Additional 3,851 config-only deploys
196 diff
erent people deployed to prod Nov/Dec 2012: deployed 752 times
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
33/46
Why does it work?
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
34/46
N = # of deploys
P = probability of site degradation
S = average severity of degradation
T = time to detect/resolve
= N * P * S * TExpected
Downtime
Continuous Deployment Math
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
35/46
N = 1
P = 0.5
S = 0.7
T = 100
Before:
E.D. = 35
N = 250
P = 0.1
S = 0.05
T = 5
Now:
E.D. = 6.25
N = # of deploys
P = prob. of degradation
S = avg. severity of degradation
T = time to detect/resolve
(all numbers completely arbitrary)
Continuous Deployment Math
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
36/46
Etsy circa 2013 (400+ employees)
acts, in some ways, more like a startup than
Etsy circa 2008 (40+ employees).
Big Takeaway
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
37/46
Continuous Deployment makes possible:
Continuous Experimentation
http://etsy.me/continuous-experimentation
http://github.com/etsy/deployinatorhttp://github.com/etsy/deployinatorhttp://github.com/etsy/deployinator7/29/2019 Etsy LeanStartup presentation at SXSW 2013
38/46
Continuous Experimentation
1. Small changes
2. Run experiment (A/B test)
3. Analyze data
4. Re-examine assumptions
Repeat continuously in pursuit
of larger goals.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
39/46
Heard since 2010:Neat experiment, but
this will never scale.
As of 2013, Etsy has
100+ engineers -still going strong.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
40/46
Some Etsy Customizations
Deploying is a first-class feature. Inability todeploy is a P1 incident (same as site down).
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
41/46
Some Etsy Customizations
Internal admin site
API
Big data
Search Blog
Deployinator itself
We continuously deploy not just the main
Etsy website, but as much as possible:
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
42/46
Some Etsy Customizations
Database schema changes
PCI-DSS environment (credit cards)
In the rare case we cant continuously
deploy, we create alternative tools:
We do continuously deploy as much of our
payment processing as is safe & legal (98%).
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
43/46
Some Etsy Customizations
Keeping deploys fast is paramount and worththe investment in manpower & hardware.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
44/46
Some Etsy Customizations
Continuous deployment is all about moving
forward, sometimes at the expense of the past.
Our solution: engineering-wide bug rotation,
one day a month, every engineer participates.
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
45/46
Continuous Deployment is a fantastic
recruitment tool for attracting engineers
who like to move fast and get stuff done.
Fun Fact:
7/29/2019 Etsy LeanStartup presentation at SXSW 2013
46/46
Learn more:http://codeascraft.etsy.com/
Join the fun:http://www.etsy.com/careers
Etsy open source (Deployinator, StatsD)http://etsy.github.com/
http://etsy.github.com/http://etsy.github.com/http://www.etsy.com/careershttp://www.etsy.com/careershttp://codeascraft.etsy.com/http://codeascraft.etsy.com/