Continuous Delivery for Linux/Windows/Hadoop

Wisely Chen

Wednesday, October 23, 13

Agenda

• Background

• Problem

• Solution

• Demo

• Q & A

Wednesday, October 23, 13

Who I am

• Wisely Chen ( thegiive@gmail.com )

• Release manager of Yahoo! Taiwan shopping and data team

• Enjoys promoting open source tech

• Hadoop Summit 2013 San Jose

• Ruby and Rails (All in Taiwan): Coscup 2006, Ubisunrise 2007, OSDC 2007

• Release Practice (All in Taiwan): Webconf 2013, Coscup 2012,2013, PHPConf 2012 , RubyConf 2012

Wednesday, October 23, 13

Team

Sanching YuWindows

Neal LeeHadoop

Wednesday, October 23, 13

Yahoo! Taiwan Shopping

• Top B2C site in Taiwan

• Monday.com acquired in 2008

• 8 engineering teams

• 15K files changed monthly

• 100K lines of code changed monthly

Wednesday, October 23, 13

PlatformBackend

(Window)Frontend(Linux)

Data(Hadoop)

Catalog

Mobile

Member

Payment

Cash Flow

CMS

Logistic

Supplier

Machine Learning

CRM

Business Intelligence

Data Mart

Wednesday, October 23, 13

Problem

FrontendTeam

BackendTeam

DB

DataTeam

HadoopWebService API

Wednesday, October 23, 13

We need a multi-platform build and release solution

Wednesday, October 23, 13

Solution

• Create B&R process

• Build CI infrastructure

• Automate everything

Wednesday, October 23, 13

Solution

• Create B&R process

• Build CI infrastructure

• Automate everything

Wednesday, October 23, 13

DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Build and Release Process

CommitUATTest

Wednesday, October 23, 13

DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Hadoop Build and Release Process

CommitUATTest

Wednesday, October 23, 13

Original DifferenceFrontend Team Backend Team Data Team

Process Scrum Waterfall Waterfall

OS Linux Windows Linux

Language PHP/JAVA .NET Pig/MapReduce

Issue Tracking Bugzilla Custom System N/A

Repos Subversion VSS N/A

CI N/A N/A N/A

Int Test N/A N/A N/A

Release Management

Routine RE Another RE SDE

Wednesday, October 23, 13

Multiplatform Complexity Reduction

Frontend Team Backend Team Data Team

Process Scrum(not finish yet)Scrum(not finish yet)Scrum(not finish yet)

OS Linux Windows Linux

Language PHP/JAVA .NET Pig/MapReduce

Issue Tracking BugzillaBugzillaBugzilla

Repos Subversion(migrate to git)Subversion(migrate to git)Subversion(migrate to git)

CI JenkinsJenkinsJenkins

Int Test SeleniumSeleniumSelenium

Release Management

Delicated Release ManagerDelicated Release ManagerDelicated Release Manager

Wednesday, October 23, 13

Communication Cost

巨大Huge Large

Wednesday, October 23, 13

Solution

• Create B&R process

• Build CI infrastructure

• Automate everything

Wednesday, October 23, 13

WebSite CI InfrastructureJenkins Master

WindowsJenkins Slave

Farm

SeleniumServer Farm

LinuxJenkins Slave

Farm

Beta Server Farm

WindowsFarm

LinuxFarm

Staging Server Farm

WindowsFarm

LinuxFarm

Wednesday, October 23, 13

Hadoop CI Infrastructure

Jenkins Master

Alpha

Jenkins Slave

Beta Cluster

HadoopJobTracker

JenkinsSlave Hadoop

nodeHadoop

nodeHadoop

nodeHadoop

nodeSlave Node

Prod. ClusterGateway

PigServer

Wednesday, October 23, 13

DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Build and Release Process

CommitUATTest

Wednesday, October 23, 13

PigUnit

• A simple xUnit framework

• No cluster set up is required in local mode

• We change default PigUnit behavior

• All PigUnit tests run on single pig server

• Speed up PigUnit

Wednesday, October 23, 13

DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Build and Release Process

CommitUATTest

Wednesday, October 23, 13

Vaidya

• Rule based performance diagnosis of M/R jobs

• Extendable framework

• You can add your own rules

• Write complex rules using existing rules

Wednesday, October 23, 13

Vaidya Performance Test

Pig Job

Pig Job Conf

Sampling Data

Wednesday, October 23, 13

Vaidya Performance Test

Pig JobPig Job History

Pig Job Conf

Sampling Data

Wednesday, October 23, 13

Vaidya Performance Test

Pig JobPig Job History

Vaidya

VaidyaRulePig Job

Conf

Sampling Data

Wednesday, October 23, 13

Vaidya Performance Test

Pig JobPig Job History

Vaidya

VaidyaRulePig Job

Conf

NotifyUser

Sampling Data

Wednesday, October 23, 13

Vaidya Performance Test

Pig JobPig Job History

Vaidya

VaidyaRulePig Job

Conf

NotifyUser

Performance Result

Next CI Stage

Sampling Data

Wednesday, October 23, 13

<Diagnos)cTest>

<Title><![CDATA[Balanaced Reduce Partitioning]]></Title>

//////<ClassName>//////////<[CDATA[org.apache......BalancedReducePar))oning]]>///////</ClassName>///////<Descrip)on>///////////<![CDATA[This/rule/tests/........]]>///////</Descrip)on>///////<Importance><![CDATA[High]]></Importance>///////<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold>///////<Prescrip)on><![CDATA[advice]]></Prescrip)on></Diagnos)cTest>

See/if/the/reduce/job/is/balanced/or/not/

Vaidya Rule Example

Rule/Importance

Diagnosis/success/threshold

Wednesday, October 23, 13

Solution

• Create B&R process

• Build CI infrastructure

• Automate everything

Wednesday, October 23, 13

DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Build and Release Process

CommitUATTest

Wednesday, October 23, 13

Before Deploy

• Check bug status

• Documention

• Create a git tag

• Auto doc generating on bugzilla

• Deploy to production cluster

• Change bug status

Wednesday, October 23, 13

Auto comment in bugzilla

Repos urlRelease Note

Issue statuschange

Wednesday, October 23, 13

Auto create git tag

Release Note[Bug xxx] log....

Git Tag

Wednesday, October 23, 13

Application Logic Monitor

0

25

50

75

100

2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-22 22-24 24-2

Today Same day in last week Same day in last month

Monitor today’s sales revenue with baseline

Notify user of something wrong

SalesNum

Time

Wednesday, October 23, 13

Result

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

High Bug Rate(72%)

Low Throughput

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q1 : Create B&R Process...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q1 : Create B&R Process...

巨大 bug rate drop

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q1 : Create B&R Process...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q2 : Build CI Infrastructure...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q3 : Build CI Infrastructure...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q4 : Build CI Infrastructure...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q4 : Build CI Infrastructure...

Dramatic throughput increase

Bug rate down to 22%

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

12Q4 : Build CI Infrastructure...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

13Q1 : Automate Everything...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

13Q2 : Automate Everything...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

13Q3: Automate Everything...

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

13Q3: Automate Everything...

Deploy time decrease

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

13Q3: Automate Everything...

Wednesday, October 23, 13

Hadoop CI Demo

• Demo1 : Unit test

• Demo2 : Performance test

• Demo3 : Deploy

Wednesday, October 23, 13

Wednesday, October 23, 13

Wednesday, October 23, 13

Conclusion

• This process can reduce bug rate

• This process can increase productivity

• Jenkins can fit in any platform

Wednesday, October 23, 13

謝謝大家, 祝你有美好的一天

Wednesday, October 23, 13

Appendix

Wednesday, October 23, 13

0

22.5

45

67.5

90

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

ReleaseBugRate ReleaseThroughput DeployCostTime

On Demand Bi-weekly Weekly

Wednesday, October 23, 13

Web Site CI Process

Backend3 Team

(Window)

Cash Flow

Frontend2 Team(Linux)

Catalog

Mobile

Member

Payment

Logistic

Supplier

CMS

Data2 Team

(Hadoop)MachineLearning

BusinessIntelligence

Data Mart

CRM

Wednesday, October 23, 13

Build and release processPHP CodeCommit

PHP Unit Test

DeployLinux Beta

BetaAuto Test

Deploy PHP Staging

StagingAuto Test

StagingUAT

Check Test Status

Release Meeting

Code Freeze

Release

.NET CodeCommit

.NET Unit Test

DeployWin Beta

Deploy .NET Staging

Deploy PHP Prod

Deploy .NET Prod

ProdAuto/UAT

DONE

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Cost Time

Throughput increased dramaticly

Bug rate down to 22%

Build CI infrastructure

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Cost Time

Significant bug rate drop

12Q1 : Create B&R Process

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Cost Time

Throughput increased dramaticly

Bug rate down to 22%

Build CI infrastructure

Wednesday, October 23, 13

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Cost Time

Deploy cost time drop

Automate Everything

Wednesday, October 23, 13

Two B&R ProcessesFrontend

Team

BackendTeam

DataTeam

Website Process

Data Process

Wednesday, October 23, 13

Automate Everything

Check PointScript

Bugzilla

Source ReposLog

DeployScript

Notify UserScript

Bug List Yes

No

Check Bug Status

Auto Manual

JenkinsStyle/Static/Similarity Analysis

Wednesday, October 23, 13