Strategies In Continuous Delivery

Learn the “magic” behind the scenes

DEVOPS Days Tel Aviv 2013

Aviran MordoHead Of Back-End Engineering @ Wix

@aviranmhttp://www.linkedin.com/in/aviran

00:00

About Wix

00:00

Wix in Numbers

• Over 39,000,000 users– Adding over 1,000,000 new users each month

• Static storage is over 150TB of data– Adding over 1TB of files every day

• 3 Data centers + 2 Clouds (Google AE, Amazon)– Around 300 servers

• Over 100,000,000 Server API calls per day• ~450 people work at wix

– ~ 150 people in R&D

00:00

Do You Have The Guts To Deploy 10 Times A Day?

00:00

00:00

From Jan’ 2013 – Jun’ 2013

• 1500• 470• 200

00:00

From Jan’ 2013 – Jun’2013

• 1500 Deployments• 470 A/B Tests• 200 Feature Toggles

Continuous Delivery – Key points

• Abandon the “VERSION” paradigm – move to a feature centric methodology

• Make small and frequent release as soon as possible • Automate everything – TDD/CI/CD• Measure Everything–A/B test every new feature–Monitor real KPIs (business, not CPU)

• Deploy without downtime

00:00

Test Driven Development

• No new code is pushed to Git without being fully tested–We currently have around 10,000 automated tests

• Before fixing a bug first write a test to reproduce the bug

• Cover legacy (untested) systems with Integration tests

00:00

Test Driven Development

• Actual impact on development–We develop products faster–Removes fear of change–Easier to enter some one else’s project–Do we really need QA? (Yes, they code tests)–Writing a feature is 10-30% slower, 45-90%

less bugs–50% faster to reach production.–Considerably less time to fix bugs00:00

Feature Toggles

00:00

Feature Toggles

• Everyone develops on the Trunk

• Every piece of code can get to production at anytime

00:00

Code branch

00:00

New Code Old Code

FT Opene

d

Yes No

Usage example

Simple “if” statement in your code

00:00

DB Schema Changes Without Downtime

• Adding columns– Use another table link by primary key– Use blob field for schema flexibility

• Removing fields– Stop using. Do not make any DB

schema changes

00:00

New DB schema with data migration

• Plan a lazy migration path controlled by feature toggle

1. Write to old / Read from old2. Write to both / Read from old 3. Write to both / Read from new, fallback to old4. Write to new / Read from new, fallback to

old5. Eagerly migrate data in the background6. Write to new / Read from new

00:00

Feature Toggle Strategies (gradual expose users)

• Company employees• Specific users or group of users• Percentage of traffic• By GEO • By Language• By user-agent• User Profile based• By context (site id or some kind of hash on site id)

00:00

Feature Toggle Override

• By specific server– Used to test system load– New database flows/migration– Refactoring that may affect performance and memory usage

• By Url parameter– Enable internal testing– Product acceptance– Faking GEO

• By FT cookie value– Testing– When working with API on a single page application

00:00

Feature Toggles Management

00:00

A/B Tests

00:00

A/B Test

• Every new feature is A/B tested• We open the new feature to a % of

users– Define KPIs to check if the new feature is

better or worse – If it is better, we keep it– If worse, we check why and improve– If we find flaws, the impact is just for %

of our users (kind of Feature Toggle)

00:00

An interesting site effect on product

• How many times did you have the conversion “what is better”?– Put the menu on top / on the side

• Well, how about building both and A/B Testing?

00:00

Marking users with toss value in a cookie

• Anonymous user– Toss is randomly determined– Can not guarantee persistent experience if changing

browser

• Registered User– Toss is determined by the user ID– Guarantee toss persistency across browsers– Allows setting additional tossing criteria (for example

new users only)– Only use this for sections that a user has to be

authenticated

00:00

• Do not mix anonymous and registered tests

• AB test parentage of users with optional filters–New Users Only (Registered users only)–By language –By GEO–By Browser –user-agent –OS–Any other criteria you have on your users

00:00

A/B Test Features

• A/B Test Override– Allows to set a value of a test for validation– Helps support experience what users experiencing

• Override methods– Via URL parameter– Via cookie

• Start/Stop Test• Pause tests• Bots always get “A”00:00

Manage your tests

00:00

Self Test / Post Deployment Test

After each server deployment run a self test before deploying the next server.

• Checking server configuration and topology– Make sure database is accessible (DB connection string)– Is the schema the one I expect– Access required local resources (data files, other config files,

templates, etc’)– Access remote resources– RPC / REST endpoints reachable and operational

• Server will refuse requests unless it passes the self test• Allow a way to skip self test (and continue deployment)00:00

Tools - App-info

00:00

Where are we today?

• We have re-written our flash editor product as an HTML 5 editor– In just 4 months

• Introduced Wix 3rd party applications (developers API)– In just 6 weeks

• We are easily replacing significant parts of our infrastructure

• And we are doing ~10 releases a day!

00:00 1/1/2013 1/14/2013 1/28/2013 2/10/2013 2/24/2013 3/10/2013 3/24/2013 4/14/2013 4/29/2013 5/13/2013 5/28/2013 6/9/20130

5

10

15

20

25

30

35

Total

Total

00:00

Read more: The Road To Continuous Delivery: http://goo.gl/K6zEK

http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery

Aviran Mordo

@aviranmhttp://www.linkedin.com/in/aviranhttp://www.aviransplace.com