Continuous Integration

Processes and SCM

To Support Test Automation

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

SIGIST Conference, July 2013

Gal Fatal

Gal.fatal@ATT.com

054-342-3864

AT&T Israel Center of Excellence

AT&T Worldwide

• One of the largest telecom companies in the world

• Over 250,000 employees worldwide

AT&T Israel R&D Center

• Formerly Interwise, Ltd.

• Purchased by AT&T in 2007

• Employment growth of 1,000% in 5 years

Main Lines of Business

• Connect

• Unified Communications

• Consumer Mobility

• IMS (Backend Components)

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.2

Gal Fatal

MBA

Managing Configuration Management teams since 1999

Pre-AT&T: Intel, Siemens, Perigen

AT&T (Interwise) since 1997

Experience:

• Source control services

• Installer development

• Continuous integration processes

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

Moving from Waterfall to Agile (scrum) at AT&T

Continuous Integration processes

Bug lifecycle in scrum

How we provide fast and flexible builds

Sanity tests and automation

How we make multi-platform testing possible

Saving time with automatic environment installations

Demo

Agenda

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.4

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.5

Waterfall Development1994-2011

Waterfall

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

6

Dev group QA group

Waterfall

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

7

Dev group QA group

Waterfall Lifecycle at AT&T Connect®

Main Development

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.8

Dev

Build and Integration tests

QA runs tests and opens bugs

Milestone

3 weeks

1 week

Build took 4 days• Compilation problems• Manual processes• Build machine resources• Environments

Bugs and StabilityVersion Release

1 2 3 4 5 6 7 8

Waterfall Lifecycle at AT&T Connect

Stability and Bugs

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.9

Devresolves bugs

Build and Integration tests

QA verifies bugs

Milestone

3 weeks

1 week

Bugs and stabilityVersion release

1 2 3 4 5 6 7 8

Agile Scrum

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

10

Developers QA

Waterfall to Agile Scrum at AT&T Connect

Agile Scrum Development

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.11

Dev + QA

Sprint

3 weeks Continuous Integration

StabilityVersion Release

91 2 3 4 5 6 7 8 9 11 12 1310

Continuous Integration Processes

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.12

Source control build

Static code analysis

Run automated unit Tests

PackagingPrepare test environment

Deploy to test environment

Run automated sanity tests

Publish report

Continuous Integration

Bug Lifecycle in Scrum

In scrums, bugs have the highest priority and there’s a “Zero-bug-approach!”

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.13

Sometimes, to save time, QA don’t open bug entries

We have to rapidly supply builds as part of Continuous Integration

Complete user story

Build Find bug Fix bug Build Verify bug

How We Provide Fast and Flexible Builds

• We moved from physical-build machines to VMs

• Each product build runs on a separate VM

• Builds are run in parallel

• We use IncrediBuild for a 10-fold increase in compilation time

• Scrum Masters or authorized users can start a build, any time, with a single click

• We added flexible reporting

• We use Jenkins

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.14

Sanity Tests: We Automate Them

Sanity test =

• Product installation• Application load• Basic functionality working as expected• Reporting

We use JSystem (based on Java®) to write the tests

QA team write and maintain scenarios

We run nightly sanity tests, using PowerShell® scripts

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.15

How We Made Multi-platform Testing Possible

The Challenge

We don’t know our client configurations…

• OS: XP®/Windows 7®/Windows 8®, 32/64 bit• Browsers: Chrome™/Internet Explorer® 7- 10• MS Office®: 2003/2007/2010• Admin/None Admin

We can think of 100s of combinations

Manual testing by QA on all platforms is impossible

Early feedback is missing

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.16

How We Made Multi-platform Testing Possible

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.17

Win7 32 Win7 64 XP 32 Win 8 Restart VM

Go back to

snapshot

Install latest

product version

Run sanity tests

Report

Shutdown VM

Running nightly

Chrome FireFox IE9 IE10

Virtual Machines

We Save Time with AutomaticEnvironment Installations

• Our environment includes more than 10 different kinds of servers and DBs

• Environment preparation takes no more than 1 day

• We use VMs

• We prepare snapshots in advance

• CI environments go back to production versions every night and automatically upgrade to latest versions

• We have different environments for QA, Dev, CI, etc.

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.18

Demo

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.19

Report Example

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.20

Summary

Waterfall to Agile

Continuous Integration

Rapid, fast builds

Sanity tests

Multi-platform testing

Environment installations

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.21

Thank You

© 2013 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

SIGIST Conference, July 2013

Gal Fatal

Gal.fatal@ATT.com

054-342-3864