BW11 Session 6/5/2013 3:45 PM 

       

"Continuous Delivery at Ancestry.com"

   

Presented by:

Seng Shee Ancestry.com

        

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Seng Lin Shee Ancestry.com

Leading the testing effort for the API team of Ancestry.com, Seng Lin Shee educates the team and defines test strategies and direction of the testing effort. He has introduced the team to behavior-driven design and test-driven design methodologies, and contributed to the design of the continuous delivery pipeline within the company. Seng Lin’s experience ranges from software development and testing, to router/switch manufacturing, to microprocessor architecture research. Seng Lin was formerly a software development engineer in test at Microsoft where he worked on Windows Server 2008 R2, Active Directory Federation Services, and Access Control Services.

 

Continuous Delivery at Ancestry.comSeng Lin Shee

sshee@ancestry.com@ y

Take Away from this Presentation

• Understanding of Continuous Delivery

• Role of Continuous Delivery in web services

I h A• Impact on the teams at Ancestry.com

• Strategies for an effective Continuous Delivery pipeline

• Changes in responsibilities and expectations

2

Agenda

1. What is Continuous Delivery

2. Continuous Delivery Architecture

3 S f Lif l i C i D li3. Software Lifecycle via Continuous Delivery

4. Testing Methodology

5. Roles and Responsibilities (Mindsets)

6. Learnings / Pitfalls

3

The Issue…

• Traditional deployment strategy of web services:– Involvement of entire engineering team– Manual monitoring– Manual test execution– Tests executed within unstable environment

4

Value Stream

• Flow of materials and information required to bring a product or service to a consumer

• Value stream map: value add and wait timep– Product concept– Product discovery– Product planning– Development– Testing

5

– Release

• We need to be more Agile!

Philosophy

• “The essence of my philosophy to software delivery is to build software so that it is always in a state where it could be put into production. We call this Continuous Delivery because we are continuously runningDelivery because we are continuously running a deployment pipeline that tests if this software is in a state to be delivered.” (Martin Fowler)

6

What is Continuous Delivery?

Continuous Deployment

Continuous Delivery

Continuous Integration

7

Cycle Time

• How to optimize cycle time– Teach developers how to write production‐ready code– Running Continuous Integration and tests on production‐like systems

– Implement an Agile development process – Instituting cross‐functional teams– Automating the build, test and release process– Implement Continuous Delivery using a Deployment Pipeline

8

Learning Release Management

• Continuous Delivery is about keeping your software in a state where it could be released into production at any time AND being able to actually release it when the business wants to.business wants to.

9

Release Management Principles and Practices

• Release small changes frequently rather than large changes infrequently

• Automated Deploymentp y

• Automated Configuration Management– OS, services, applications, everything– No more manual configuration

• Automated processes are self documenting

• Automated change tracking to production resources

• Automated deployment smoke tests (test the deploy)

• Automation, automation, automation…10

Tooling

• Provides the following services– Distributed build systemDistributed build system– Continuous integration– Workflow engine (deployment pipeline)– Job control system– Dashboard (web application)

• Hub spoke design multi platform• Hub‐spoke design, multi‐platform– Single Go server– Multiple Go agents (one or more per machine)

11

Continuous Delivery Architecture

Traditional Deployment Environments

DEV Environment STAGE Environment LIVE Environment

13

Version Control System

Towards Continuous Delivery

Tests critical functions

Tests remaining functionalities

Tests all functionalities

Build, Unit Tests & Packaging

14

Acceptance Preproduction LIVEBuild

Then & Now

NOW• All environments

are provided with isolated boxes for testing

• Weekly rolls take no more than 15 minutes

THEN• DEV & STAGE

environments are easily broken

• Weekly rolls take hours to complete testing

15

Maturity of Pipeline

Rolls are Continuous Continuous Automatic roll

MATURITY LEVEL

No

pipe

line

Rolls are performed in a concerted effort every fortnight

Basi

c te

sts

are

in

plac

ed in

pip

elin

e

Continuous Delivery pipeline rolls binaries every fortnight

ke t

ests

and

mor

e ro

bust

tes

ts

Continuous Delivery pipeline rolls with tight supervision

depl

oym

ent

issu

es

afte

r 1

mon

th

Automatic roll during business hours

p

Smok

No

d

16

Software Lifecycle via Continuous Delivery

Kanban Principals

• Very much suited for our release cycles

• Favored by Project Managers

A i DONE h f i DEPLOYED• A story is DONE when a feature is DEPLOYED to a particular ENVIRONMENT

BACKLOG IN DEVELOPMENT DONE

18

Testing Methodology

Smoke Tests during Deployment

Internal smoke tests

Dependency Dependency smoke tests

20

Profile of Tests at each Quality Gate

Tests critical functions

• Acceptance Tests

• Regression Tests

P f T• Performance Tests

• Tests are categorized into– READY– QUARANTINE– REGRESSION

Acceptance

21

Acceptance

GO Pipeline FAQ

• READYThese are tests that are in the Acceptance Test project which are ready, but with stories not completed. These tests will not block the pipeline We should be observing this group of testblock the pipeline. We should be observing this group of test

• QUARANTINEThese are tests which are known to fail, due to a certain bug (which will take time to fix). We do not mark them as ignored because these are still valid tests.

• REGRESSIONThese are normal test that should pass 100% of the time. If they fail, please retry the respective block of test.

22

Tests Parallelism

• Pipelines should be made as wide and as short as possible

• GOAL– Reduce time for testing

• CHALLENGES– Tests should  not interfere with one another– Tests should not depend on sequence of execution– Non‐performance tests should not overload server– Non‐performance tests should not overload server

• SOLUTION– Parallelism via namespaces / test categories / test classes

23

Roles and Responsibilities (Mindset)

Continuous Integration Key Principles and Practices

• Maintain a Single Source Repository

• Everyone Commits To the Mainline Every Day

A FAST lf i b ild ( f h M i li• Automate a FAST, self‐testing build (of the Mainline check‐ins) on an integration machine

• Test in a Clone of the Production Environment

• Make it Easy for Anyone to Get the Latest Executable

E h ' h i• Everyone can see what's happening

25

Continuous Delivery Key Principles

• Single source code branch (e.g. trunk/mainline)

• Single build (no DEV and PROD build definition)

D l h i• Deploy the same way to every environment

• Configuration management

• Infrastructure as code

# of tests

Execution time

26

Pipeline progress

# of tests

Continuous Delivery Values and Practices

• Automated testing and deployment

• Workflow handles check‐in to release (deployment pipeline)p p )– Pipeline execution instances are versioned– Pipeline output becomes an artifact in repository

• Teams own their build and workflow– Use Ant/NAnt script to define build – Use Continuous Integration tool to design workflow– Use Continuous Integration tool to design workflow– Elements: pipeline group, pipeline, stage, job and task

27

Continuous Delivery Anti‐Patterns

• Multiple branches

• Feature branches that are long‐lived (> 1 day)

C i ki l h 10 i• Commit tests taking longer than 10 minutes

• Team culture allows ignoringwhat happens after check‐in

• Multiple builds in a pipeline

28

Developers Own the Releases

• Developers are testers

• Responsible to address checked‐in builds that fail in any pipeline stagesp p g

• File bugs, mark tests are quarantined and move on with development

29

Broken Pipelines happen…stuff happens…

• Run local builds with unit tests first if your check‐ins are constantly causing build breaks (pipeline breaks)

• Breaks are transparent; fix within 10 minutes, or roll backp

• “Always in a state where it could be put into production”

30

Learnings and Pitfalls

Warm up time of services

• A .NET Web Service (IIS hosted services) cannot handle requests immediately  upon deployment.

• Loading and compiling of .NET assemblies on first g p grequest.

• The DEPLOY job has been configured to performed smoke tests which exercises critical areas of code, a cluster at a time.

• Product code has to be engineered for:Product code has to be engineered for:– Quick boot up time– Robust against frequent recycling of application pools

32

Extreme Long Release Latency

• A deployment pipeline has the following goal– Increase throughput via simultaneous execution of stages– Early fault detection

• Longer delay between CheckIn’s and Releases due to– Increase in the number of quality gates– Repetition of the exact tests in differing quality gates– Longer sequential tests

• This impacts the Time To Resolve when a quick fix is• This impacts the Time‐To‐Resolve when a quick fix is needed

33

Backward and Forward Compatibility

• Deploy at anytime

• Release features independently

B ki h f il d CD Obj i• Breaking change ‐> failed CD Objective

• Invoke test suites of dependent components

34

Conclusion

Conclusion

• Continuous Delivery ensures that code is always ready for release when the business wants to.

• Quality is owned by the entire team.y y

• Pipelines should be made as wide and as short as possible

• Quickly detect faults

• Teams confidently, rapidly rolly p y

36

QUESTION TIME