Hadoop Summit 2013 : Continuous Integration on top of hadoop

Continuous Integration on top of hadoop

Wisely Chen & Neal Lee

Tuesday, June 11, 13

Agenda

• Who I am

• Problem

• Solution

• Demo

• Q&A


Who I am

• Wisely Chen ( [email protected] )

• Release manager of Yahoo![Taiwan] shopping and data team

• Love to promote open source tech at Taiwan

• Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007

• Puppet : PHPConf 2012 , RubyConf 2012

• Release Practice : Webconf 2013, Coscup 2012


mailto:[email protected]

mailto:[email protected]

Who I am

• Neal Lee (@neal_lee)

• Data Engineer at Yahoo![Taiwan]

• Aiming to build up an easy use of self-service BI platform connecting to Hadoop.


Story 1


Another Story


Yet Another Story


Solution


One click

• Manual commit code to SCM

• And DONE

• Auto unit testing

• Auto push beta for performance testing

• Auto push to production grid

• Auto trigger code


This feeling is 爽!


Continuous Integration


Continuous Integration

• A software engineering practice • Maintain code repos

• Automate the build

• Make the build self-testing

• Everyone commit to the baseline everyday

• Every commit should be a build

• Test in a clone of production environment

• Make it easy to get the latest deliverables

• Everyone can see the result of latest build

• Automate deployment


We focus on• A software engineering practice

• Maintain code repos

• Automate the build

• Make the build self-testing

• Everyone commit to the baseline everyday

• Every commit should be a build

• Test in a clone of production environment

• Make it easy to get the latest deliverables

• Everyone can see the result of latest build

• Automate deployment


CI flow

4. CI slave exec localUnitTest

7. CI slave exec

Performanc

11. CI exec pig

People

DEV Alpha Beta Grid Prod Grid

2. notify CI

5. deploy 8. deploy

CI Master

1. CommitCode

SCM

3. Call 6. Call

10. Call

9. git tag

12. notify user


CI flow


CI slave exec

PerformancCI exec pig

People


2. notify CI

CI Master

1. CommitCode

SCM

3. Call

5. Notify user


CI flow


7. CI slave exec


People


2. notify CI

5. deploy

CIMaster

1. CommitCode

SCM

3. Call 6. Call

8.Notify user


CI flow


7. CI slave exec


People


2. notify CI

5. deploy 8. deploy

CI Master

1. CommitCode

SCM

3. Call 6. Call

9. Notify user


Unit Test


7. CI slave exec

Performanc

11. CI exec pig

People


2. notify CI

5. deploy 8. deploy

CI Master

1. CommitCode

SCM

3. Call 6. Call

10. Call

9. git tag

12. notify user


PigUnit

• A simple xUnit framework

• No cluster set up is required in local mode

• Unit testing, regression testing, and rapid prototyping on the fly


Using PigUnit

• Coding

• Write PigUnit test case

• Run local PigUnit test

• Push to grid

• Run Pig on grid

• Get right result !


Unit test is live doc

• Unit test is runnable live doc

• Pass test case and meet previous requirement


Performance Test


7. CI slave exec

Performanc

11. CI exec pig

People


2. notify CI

5. deploy 8. deploy

CI Master

1. CommitCode

SCM

3. Call 6. Call

10. Call

9. git tag

12. notify user


Vaidya

• Rule based performance diagnosis of M/R jobs

• Extensible framework

• You can add your own rules

• Write complex rules using existing rules


CI toolset

CI slave exec localUnitTest

CI slave exec


People


notify CI

deploy deploy

CI

CommitCode

SCM

Call

Vaidya

BASH


CI is flexible

• MapReduce can use MapUnit

• Hive can use hive_test

• Pig can use PigUnit


Github trigger CI


CI testing build pipeline


Testing Trend


DEMO


Conclusion

• Auto testing will save your life

• CI will boost your productivity

• This process can feed in any platform


謝謝大家


Hadoop Summit 2013 : Continuous Integration on top of hadoop

Technology

Transcript of Hadoop Summit 2013 : Continuous Integration on top of hadoop