03 - Continuous Integration

CONTINUOUS INTEGRATION

2

TRAINING GOALS

Show that there is no way present-day project could be successful without usage of

continuous integration practice

Establish connection between CMMI product

integration process area and continuous integration

practice

3

TRAINING PLAN

1. What is continuous integration?2. Why do we need continuous integration?3. Prerequisites for continuous integration

process4. General CI workflow5. How does continuous integration affect our

development process?6. Tools and their features7. When CI is not effective?8. We have "true CI". What next?9. CI and CMMI product integration process

area

4

INTRODUCTION TO CONTINUOUS INTEGRATIONBasic ideas

5

WHAT IS CONTINUOUS INTEGRATION?

6


• Integration is when you make everything work together• Continuous is when you make everything work together very

often• You would go mad if you had to do it manually• That’s why it is not about manual actions, it is about

automation

7


Software engineering

practice

Approach helping to

reduce risks

Another step to the

development process

automation

8

WHY DO WE NEED CONTINUOUS INTEGRATION?

Martin Fowler: … team integrate their work frequently,… leading to multiple integrations per day. … this approach leads to significantly reduced integration problems and allows a team to

develop cohesive software more rapidly. CI is the one of the XP practicesAllows to have fresh latest build

9

PREREQUISITES FOR CONTINUOUS INTEGRATION

Not everything

continuously integrate

you can

10

PREREQUISITES FOR CONTINUOUS INTEGRATION

Single source code repo (VCS system)

Build automation

Self-testing app (unit-

tests)

Separate server

(usually)

CI tool

11

GENERAL CI WORKFLOW

Committing latest

changes

Update by scheduler

(on CI server)

Build (compilation,

unit-testing, db integration,

etc)

Application is ready

12

GENERAL CI WORKFLOW

commit

repository deployment server

~

build & deployupdate

changes detected

continuous integration server

http://images.google.com/imgres?imgurl=http://www.mcapodcasts.co.uk/images/buttons/database.jpg&imgrefurl=http://www.mcapodcasts.co.uk/library/14publicsafetyatsea.htm&usg=__LXHR57NdxKk6G7Fgz-m6intTVjs=&h=50&w=50&sz=3&hl=en&start=8&um=1&itbs=1&tbnid=D9gVJTpblxe2sM:&tbnh=50&tbnw=50&prev=/images?q=database&imgsz=i&hl=en&client=firefox-a&rls=org.mozilla:en-GB:official&sa=N&tbo=1&um=1

13

GENERAL CI WORKFLOW

Check modificatio

n

Buildapplication

14

TOOLSInstruments for continuous integration and their features

15

TOOLS CLASSIFICATION

CI tools

• Code coverage analysis• Static analysis (syntax check, code dependencies)• Copy/paste detectors

Build management tools

Unit testing tools

Inspection tools

Documentation generation tools

16

CI TOOLS CruiseControl CruiseControl.NET CruiseControl.rb JetBrains TeamCity Apache Hudson Apache Continuum Atlassian Bamboo FinalBuilder phpUnderControl XInc

17


Most common, flexible, configurable, easy to use, de facto choice for Java projects

18


Most common, flexible, configurable, easy to use, de facto choice for .NET projects

19


Most successful adaptation of CI principle for the interpreted language (Ruby)

20


One of the best CI tools: • pre-tested commit• build agents• multi-platform builds• build dependencies• comprehensive statistics and reporting

21


Tagging after successful build, distributed builds

22


Distributed builds

23


Focused on process of building, CI is just a feature

24


CI tools written in PHP for PHP projects

25


Yet another CI tool written in PHP for PHP projects

26

CI TOOLS Too many tools All are great All have the same principle But each has specific features

Diagrams, metrics, reporting

Integration with SCM

tools

Notification

Interface, usability

27

CI TOOLS. ALM APPROACH

InitiationRequirements

development & management

DesignImplementation

Integration Testing

Deployment

Maintenance & support

Utilization

28




Integration Testing

Deployment


Utilization


Basic continuous integration

29




Integration Testing

Deployment


Utilization


Basic continuous integration

Extended build management

30

BUILD TOOLS

Ant NAnt MSBuild Maven Make Phing Rake …

31

BUILD TOOLS


Most common, flexible, configurable, easy to use, de facto choice for Java projects

32

BUILD TOOLS


Most common, flexible, configurable, easy to use, de facto choice for .NET projects

33

BUILD TOOLS


de facto case for Visual Studio and TFS users

34

BUILD TOOLS


Build management + dependencies management for Java projects. Introduced

POM concept.

35

BUILD TOOLS


Standard de facto for UNIX applications

36

BUILD TOOLS


Implementation of build management tool in PHP for PHP projects

37

BUILD TOOLS


Implementation of build management tool in Ruby for Ruby projects

38

UNIT TESTING TOOLS

Junit NUnit CppUnit PHPUnit SimpleTest JSUnit J3Unit

39

INSPECTIONS TOOLS.TEST COVERAGE

Cobertura Atlassian Clover jTest JCoverage CodeCover EMMA Parasoft Insure+

+ Ncover Xdebug Coverage.py

40

INSPECTIONS TOOLS.TEST COVERAGE

C++C#

PHPPython

Java

Cobertura Atlassian Clover jTest JCoverage CodeCover EMMA Parasoft Insure+

+ Ncover Xdebug Coverage.py

41

INSPECTIONS TOOLS.STATIC ANALYSIS

PMDFindBugsJLint FxCopCheckstyleReSharperPHP_CodeSnifferYascaNDepend

42

INSPECTIONS TOOLS.COPY/PASTE DETECTORS

CPD (Copy paste detector)CheckstyleSimianJplagAtomiqClone diggerPBA

43

DOCUMENTATION GENERATIONTOOLS

Doxygen

JavaDoc

NDoc

CppDoc

phpDocumentor

pyDoc

RDoc

44

OTHER TOOLS

Source code metrics (loc)

Dead code detectors

Profiling …

CI IN EPAM MICROSOFT TECHNOLOGIES DIVISIONProject Project Product Primary

LanguageCCNET Code Analysis

toolsUnit testing

Test coverage tools

Copy & Paste detectors

Automate build delivery

Automate deployment

ACT-INT C# x - x (Unit) - - - -

ACT-LVI C# x x (FxCop, PBA) x - - - -

CON-GEN C#, ASP.NET - - - - - - -

EPM-UTIL C#, ASP.NET x x (FxCop,PBA) x (NUnit) x (NCover) x (Simian) + +

ESS-MSFT - - - - - - -

FSTIDX C#, ASP.NET x x (PBA) x - - - -

ICPDP C# x - - - - - -

LOGIDEX Java, J#, C#, ASP.NET - - - - - - -

LSEEC C# x x (FxCop, PBA) x (NUnit) - x (Simian) - +

MIS-OPC C# x x x - - - -

MIS-SPR C# x x x (Nunit) x - - -

MultAPI VBScript - - - - - - -

MultCons C# x x (FxCop, PBA) - - x (Simian) - -

MultData C#, C++, HTML, VB x - x - - - -

MultEDG C++, C# - x (FxCop) x - - x -

Mult-IPAS C#, C++ x x (FxCop) - - - - -

MultRIN C# - - x - - - x

Mult-Torn C++ - - x - - - -

Project Project Product Primary Language

CCNET Code Analysis tools

Unit testing

Test coverage tools

Copy & Paste detectors

Automate build delivery

Automate deployment

RTRS-AAAM C# - - - - - - -

RTRS-MTST C#, ASP.NET - x (FxCop) - - - - -

RTRS-RKSD C# x x - - x (Simian) - -

RTRS-RKWM C# - x - - - - -

TAK-PBLD C# - x (FxCop) x (Nunit) - - - -

TRR-ODC (O2I) C#, ASP.NET x x (FxCop,PBA) x (Nunit) - x (Simian) - -

TRR-ODC (SapSn) C# x x (FxCop,PBA) - - x (Simian) - -

CI IN EPAM MICROSOFT TECHNOLOGIES DIVISION

46

MOST TYPICAL QUESTIONS

All the above-described info sounds cool. I feel that it might be useful in my project. But I am not sure I can find enough resources to start and support this activity.

EPM-BET project is started long time ago for this purpose. It provides corresponding resources and support

There are too many different tools. They are all different. How could I use them both independently and all at once?

I recommend to use AgileSCM approach. It uses unified versions numbering which could be used by any mentioned tool.

Q

A

Q

A47

48

ADVANCED CONTINUOUS INTEGRATIONMore complex issues related to the continuous integration

49

HOW DOES CI AFFECT DEVELOPMENT PROCESS?

Commit policy (mainlines are on the watch)

Don't break the build rule (make local build)

Everyone commits to the mainline everyday

Keeping the build fast Run fast tests first

We can automate deploymentBuild is primary, deployment is

secondary Continuous design

50

“DON’T BREAK THE BUILD” RULE Test code properly before

checking in Avoid depending on a local

resource that is not under version control or does not exist on the target computer.

Write tests to cover common problems

Make sure local inspections were run successfully

Broken build in most cases means that it is impossible or not recommended to use corresponding codebase until it has been fixed

51

“DON’T BREAK THE BUILD” OBSESSION It is often impossible to

reproduce integration problem without checking in.

You might spend A LOT of time looking for a failure reason in someone’s else module

During that time you will be the one WHO FAILED THE BUILD

If you want to avoid such situations, every checking in becomes a nightmare

Attitude to the “don’t break the build” rule may be a detector of how mature the team is.

52

“DON’T BREAK THE BUILD” OBSESSION. SOLUTION Nightly builds instead of

builds ‘on commit’ Team needs more strict and

less strict variations of the “don’t break the build” rule during different SDLC phases

There are architectural issues to pay attention to

Sometimes several small projects should be started instead of one large

Build policy is to be developed

53

WHEN CI IS NOT EFFECTIVE?

DVCS Experimental development Small projects Interpreted languages without unit-tests When database is being changed too

often Types of projects

Documentation (BA, etc)DBSupport & maintenance

54

WE HAVE "TRUE CI". WHAT'S NEXT?

Building tags Separate builds having different

maturity levels Codebase structure filtering

Triggering build only for changes in specified list of folders

Integration with issue tracking systems Different approach for different SDLC

phases

56

/1.x.x

/1.0.x

1.x.x 2.x.x

1.x.0

1.x.1

1.x.2

1.x.3

1.x.4

2.x.0

2.x.1

1.x.5 2.x.2

/trunk

PA

A

Bbuilds

AR

BR

RC

ST

releases

1.0.0

1.0.1

1.0.2 1.0.3

1.0.4

WE HAVE "TRUE CI". WHAT’S NEXT?

57

builds

integration (dev)

integration (dev)

WE HAVE "TRUE CI". WHAT’S NEXT?

/1.x.x

/1.0.x

1.x.x 2.x.x

1.x.0 1.x.1 1.x.2 1.x.3 1.x.4

2.x.0 2.x.1

1.x.5

2.x.2

/trunk

1.0.0 1.0.1 1.0.2 1.0.3 1.0.4

CI server

1.x.0.0 1.x.0.1 1.x.1.0 1.x.1.1 1.x.2.0 2.x.x.0 2.x.x.1 2.x.x.2 2.x.x.3 2.x.x.4 …

1.x.2.1 … 1.x.3.0 1.x.3.1 1.x.4.0 1.x.4.1 1.x.4.2 1.x.4.3

2.x.0.0 2.x.0.1 2.x.1.0 2.x.1.1

builds

CI server

1.0.x.0 1.0.0.0 1.0.0.1 1.0.1.0 1.0.1.1 1.0.2.0 1.0.2.1 1.0.3.0 1.0.3.1integration (rel)

releases

58

N. ?. ?.L

N .x.x .L

N .x.K.L

N .M. x.L

N .M. K.L

WHAT NEXT? VERSIONS NUMBERING PATTERNS

1.x.2

1.x.3 1.x.4

2.x.0

1.x.2.1 1.x.3.0 1.x.3.1

1.x.1.0 1.x.1.1

1.x.2.0

2.x.x.0 2.x.x.1 2.x.x.2 2.x.x.3 2.x.x.4 …

…

1.0.0 1.0.11.0.x.0 1.0.0.0 1.0.0.1

2.0.02.0.x.0 2.0.x.1

1.1.01.1.x.0

N.x N.Mdev rel

integrationpilot integration

development integration

pre-release integration

release integration

CONCLUSION

There is a lot of possibilities to make your continuous integration better:Several integration streamlines (not only

trunk)Development and release integration

streamlinesIntroduce version number consisting of

four symbolsProper version number inheritanceIntegration of builds and releases…

CONCLUSION

CI is another SCM tool (practice)Middle size project should

definitely have it.It is used mostly in agile worldAgile SCM is the result of attempt

to understand what version should have artifacts which are being built by CI server

62

AFTERWORD

63

RECOMMENDED READING1. Continuous Integration: Improving Software

Quality and Reducing Risk by Paul M. Duvall, Steve Matyas, Andrew Glover

64

RECOMMENDED READING2. Continuous Delivery: Reliable Software Releases

through Build, Test, and Deployment Automation by By Jez Humble, David Farley

65

RECOMMENDED READING3. Release It!: Design and Deploy Production-

Ready Software by Michael T. Nygard

66

RECOMMENDED READING4. Pragmatic Project Automation: How to Build,

Deploy, and Monitor Java Applications by Mark Clark

67

USEFUL LINKS http://

martinfowler.com/articles/continuousIntegration.html - the main article about CI from Martin Fowler

http://www.infoq.com/news/2009/03/Continuous-Deployment - Beyond Continuous Integration: Continuous Deployment

http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html - Continuous Deployment in 5 easy steps

http://www.proudlyserving.com/archives/2007/10/fear_of_a_broke.html - ‘fear of the broken build’ effect

http://lib.custis.ru/Continuous_Integration - wiki about Continuous Integration in Russian

http://habrahabr.ru/blogs/php/91777/ - Providing quality for web applications (rus)

http://www.youbrokethebuild.com/ - great posters created with the purpose of motivation people avoid breaking the build

http://martinfowler.com/articles/continuousIntegration.html

http://martinfowler.com/articles/continuousIntegration.html

http://www.infoq.com/news/2009/03/Continuous-Deployment

http://www.infoq.com/news/2009/03/Continuous-Deployment

http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html



http://www.proudlyserving.com/archives/2007/10/fear_of_a_broke.html



http://lib.custis.ru/Continuous_Integration

http://lib.custis.ru/Continuous_Integration

http://habrahabr.ru/blogs/php/91777/

http://habrahabr.ru/blogs/php/91777/

http://www.youbrokethebuild.com/

03 - Continuous Integration

Technology

Transcript of 03 - Continuous Integration