End-to-End Testing in an Enterprise Agile Environment · End-to-End Testing in an Enterprise Agile...

W15 Track 5/6/2009 3:00:00 PM

"End-to-End Testing in an Enterprise Agile Environment"

Presented by:

Billie Bell Intuit, Inc.

Presented at:

330 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 · 904-278-0524 · [email protected] · www.sqe.com

Billie Bell Billie J. Bell, E2E QA Program Manager for Intuit, developed the end-to-end (E2E) QA Program Manager role after recognizing the benefits of engaging QA teams across Intuit's primarily agile project teams in understanding how cross-project integration testing influences achievement of overall business objectives. Using the rollout of specific company-wide initiatives provides a pragmatic context and motivation for development of new processes and promotes an E2E quality mindset beyond QA. Billie’s recent accomplishments include coordinating the integration of 30+ applications with a PCI compliance platform, a consolidated entitlement and software license management platform infrastructure integrating desktop and Web service applications, and a 10-layer subscription and billing SAAS initiative integrating 3rd party and internally-developed applications. Including her eight years at Intuit, Billie brings more than 20 years of experience in enterprise financial software development in QA management, product management, and program management roles in both Fortune 500 and start-up software development companies.

2/16/2009

1

Business Demands

Complex Technology

End-to-End Testing in an Enterprise Agile EnvironmentBillie BellIntuit, Inc.

A il

1

TechnologyAgileDevelopment

This presentation discusses the theory and practice of large scale integration testing

An enterprise working to reach middle

Best practice

Some practical examples of mechanisms Intuit has found to be

An enterprise working to reach middle ground between agile and process-based methodologies

Context

2

practice tip

mechanisms Intuit has found to be effective appear throughout the presentation

2/16/2009

2

Three forces in today’s business environment interact to create challenges for quality leaders

Accelerated Business Demands

Complex Agile

Development

3

Complex Technology

Development

Despite our best efforts, results can sometimes come across as grim headlines


Complex Technology

AgileDevelopment

v

4

2/16/2009

3

We expect headlines that reflect our intentions to deliver a high quality release on time


Complex Technology

AgileDevelopment

v

5

Today’s challenges bring opportunities to take a fresh look at quality leadership

6

2/16/2009

4

Community

Quality leaders are uniquely capable to set the course for success in today’s business setting

Community organizers for quality

Visibility

7

Visibility promoters

QA leadership role Expand quality definition Enlist all stakeholders

As a community organizer for quality you can build the foundation for end-to-end mindset

8


2/16/2009

5

Start with your own vision

9


Develop shared vision

10


2/16/2009

6

Lead process change across the organization

11


A quality leader’s enhanced role should also include promoting visibility

√ Help QA and other teams see their applications in p Q ppthe context of the enterprise

√ Help the organization stay one step ahead by acknowledging and managing risk

12


2/16/2009

7

Promoting visibility into business rationale is an element of the new role QA leaders play

Business context and relative priority result in more effective testing More of the business

t ti i t idAccelerated

Business Demands

Complex Technology

AgileDevelopment

v

transaction is outside a single application

13

Agile best practices emphasize connection with the customer

QA is a business discipline, not just a set of tasks that occur in serial fashion after development

Best practices to promote visibility bring practicality to the quality community

Some examples of ways to promote visibility

Best practice

tip

p y p y√ Weekly 1-hour Spotlight Sessions for overview

presentation by √ business process experts√ Enterprise and system architects

√ 2-day workshops for more complex, new systems√ Cross-project test case reviews

14


2/16/2009

8

Quality leaders have the perspective to lead the organization in seeing the risks ahead

15


Risk assessment and management by a cross-team network promotes accountability

Use risk planning sessions to

1. Identify risks2. Identify indicators3. Plan mitigation4. Identify response

k

16

5. Monitor risks


2/16/2009

9

Best practices used to manage risk include collaboration and analysis

Ways to uncover risks

Best practice

tip

y√ Pre-assessment questionnaire√ Defect analysis from previous projects√ Risk workshops

17


With a collective understanding of where we are, we can examine how we measure quality

18

2/16/2009

10

Beyond a single

Extend definition of quality

Beyond a single application

Across release cycles

19

cycles


Order System High Level Architectural Diagram

blankblankOrder Sources Payment

Processing and

Cal

l Cen

ter

App

Web applications

blankCall center

and Accounting

20

Proc

esso

r

Legacy applications

2/16/2009

11

Order System High Level Architectural Diagram

blankblank

Cal

l Cen

ter

App

Web applications

blankCall center

21

Proc

esso

r

Legacy applications

The Purchase use case is architecturally nine layers deep

Cal

l Cen

ter

App

22

Proc

esso

r

2/16/2009

12

The Update customer information use case spans five layers

Business Use CaseWeb App

APurchase Update

customer Info

Bill

Order Engine

Web Order Processing

3rd Party CC

Processor

Cal

l Cen

ter

AppCall Centers,

Mail and Fax orders

Process 2

Web App B

Web App CProcess

1

Credit Card Authorization

Cancel Bill subscription

23

Engine

Order Source

Proc

esso

r

IVR

C

Settlement

Financial Accounting

Customer Information

The Cancel use case is 5 layers deep

Cal

l Cen

ter

App

24

Proc

esso

r

C

2/16/2009

13

Bill Subscription use case is executed outside the application

Cal

l Cen

ter

App

25

Proc

esso

r

C

Realities of today’s enterprise technology can leave our traditional quality monitors lacking

26

2/16/2009

14

When it comes to code coverage, the whole is not the sum of its parts

Component Coverage

Component 380%

80% x 80%64% x 80%

Component 280%

Component 180%

27

Process Coverage

80% 64% 51%


Payoff for integration analysis from business and technical view is more robust test cases

Integration analysisIdentify systems and system-to-system interfacesIdentify and high level business transactions that are dependent

Best practice

tip

Identify and high-level business transactions that are dependent on these interfaces or trigger their executionAnalyze the use of data across multiple systems to track data flow and reconciliationSimplify and communicate all of the above

Use UML diagrams to illustrate functional touchpointsSummarize and conceptualize the wealth of informationCommunicate the big picture, provide maps to details

28

Reporting test execution results by functional use case


2/16/2009

15

Onboarding applications post release extend accountability for quality End-to-end testing becomes more like no-end-in-sight testing

29


Onboarding a new applications to our model may appear as a new box on the diagram

Process 2

Web App B

New Web AppProcess

1

Web App A

Order Engine


3rd Party CC

Processor

Order Source so

r

Cal

l Cen

ter

AppCall Centers,

Mail and Fax orders

Process 21

Credit Card Authorization

Settlement

30

Source

Pro

ces

IVR Financial Accounting

Customer Information


2/16/2009

16

But in reality, the picture is multi-dimensional

• Backward compatibility

Considerations

New

Web App


New

Web App

Process 2Process 1

Web App A

New

Web App

p y• Persistent test infrastructure• Technical support mechanisms

31



Test automation and easy integration test mechanisms ensure quality post release

Automated regression testsAutomated smoke tests in pre-production environments on a frequent basis

Best practice

tip

frequent basisAutolab to run build verification regression tests

Can be run on demand by development engineers

Test data mechanismsBaseline test data sets for key data repositoriesTools that verify data synchronization Dynamic data reservation systems to avoid data collisions

32

Dynamic data reservation systems to avoid data collisionsExamples: Test credit cards, license numbers


2/16/2009

17

Success in building an enterprise-wide quality community requires collaboration and inclusion

33

Enlist all stakeholders

34

The less obvious the obvious


2/16/2009

18

QA teams can leverage new best practices used by their development partners

Certain agile best practices provide clear benefits to quality for the project at

Other best practices evolve to provide even greater benefit to end-to-end quality for the project at

hand– Test-driven development

can result in better unit tests

– Retrospectives result in more frequent lessons-learned

benefit to end to end testing– Business use cases created

by system and enterprise architects

– Imbedded developers from upstream projects

35


Business process experts are key contributors throughout the process

I lInvolve business process matter experts in all

Where you find them can differ in every situation

Be flexible in your expectations

36


phases situation

2/16/2009

19

In summary, an effective E2E test strategy is based on shared vision and credible results

• Quality leaders as activists who C t b d lit it – Create a broad quality community

– Enable the organization to maintain sight of the big picture

• Quality defined as a continuum– Outward-focused in addition to being inward-focused– Extending beyond the release cycle

37

• Operationally inclusive of all stakeholders and methodologies– Leveraging agile development practices– Finding more ways to gain from business process expertise across

the company

No single tool or methodology guarantees success for an end-to-end test strategy

• It is a strategy based on a combination of people, tools, mechanisms, and best practices

• It works best when it is emergent and organic, but the seed has to be planted and nurtured

• In many ways the best practices are the same as traditional; what sets it apart is the focus on the end-to-end business process

38

It’s about people more than tools

It’s about mindset more than skill set

End-to-End Testing in an Enterprise Agile Environment · End-to-End Testing in an Enterprise Agile...

Documents

Transcript of End-to-End Testing in an Enterprise Agile Environment · End-to-End Testing in an Enterprise Agile...