Performance TestingApps Christine Jackson Regional Manager Software Quality Assurance & Testing...

Performance Testing Apps

Christine JacksonRegional Manager

Software Quality Assurance & Testing Practice

April 9, 2003


2

Today’s Topics

Industry Quality Metrics

What is Different?

Risk Mitigation Approaches

Case Studies

Keys to Successful Tests


3

Is there really a Software Quality Problem?

According to a federal study, buggy software costs the

U.S. economy nearly $60 billion$60 billion a year. (May, 2002)

www.nist.gov/director/prog-ofc/report02-3.pdf

“Agilent Technologies last week joined the list of companies blaming balky software for crimping the bottom line. Agilent's claim followed similar charges from Hershey, W.L. Gore & Associates, W.W. Grainger, Whirlpool, Foxmeyer and Nike--all of which in the past few years have cited botched software installations for disrupting operations to the degree their finances suffered.” CNET News.com August 27, 2002

Industry Metrics


4

Software is complex and errors propagate throughout the development process.

55% of the defects are traceable back to the Requirements & Design phases

Source: Capers Jones, SPR, Inc.

Project Size Small Medium Medium Medium Large LargeFTEs 2 6 10 22 68 113Defects in Projects 400 1,260 2,175 5,000 15,660 26,900Defects into Production 40 132 272 850 2,897 5,515% Defects in Production 10% 10% 13% 17% 18% 21%

By the way, these statistics come from companies with By the way, these statistics come from companies with metrics…metrics…

Industry Metrics


5

If 55% of defects are introduced in the first two stages of a project, there are early discovery opportunities

Start -- Project Life Cycle -- Finish

$ $$$

With early involvement of QA, and the correct processes and procedures, we can drive your discovery bubble earlier in the lifecycle. Thereby reducing your costs to correct defects.

With early involvement of QA, and the correct processes and procedures, we can drive your discovery bubble earlier in the lifecycle. Thereby reducing your costs to correct defects.

For most, the typical defect discovery bubble occurs just prior to implementation, or worse…just after implementation. Both incur the greatest costs to correct the defects.

For most, the typical defect discovery bubble occurs just prior to implementation, or worse…just after implementation. Both incur the greatest costs to correct the defects.

Industry Metrics


6

0

1

2

3

4

5

6

7

8

9

10

Rel

ativ

e Im

pro

vem

ent

Where are the Metrics for Performance & Load Testing?

Source: Capers Jones, SPR, Inc.

Low-volume Beta Test = <10 participants

High-volume Beta Test = >1,000 participants

Start

Finish

$$$$

Industry Metrics


7

We Rarely Tested Performance or Load Before..


8

..So What's Different?

Mainframe Apps– System Programmer– DBA– Mostly Batch– Throw more Horsepower into it– Small number of machines

What’s Different?

I Planet Portal(Sun One Portal)

LDAP ServerCertificate ServerGateway ServerContent Server

Pix FirewallsStealth FirewallsSSL Accelerator

(www.)

UI ServerSun 480R4x900 mhz CPU8 gb Ram


Reference ServerSun 280R2x900 mhz CPU4 gb Ram

Inbound ServerSun 280R2x900 mhz CPU4 gb Ram

Oracle DB ServerHigh Availability MirroringSun 450010x400 mhz CPU10GB Ram

Data

(Tandem)MVSMQ

MessagingPersistence

RoutingSun 4500

10x400 mhz CPU10 gb Ram

User Access via Portal

DB2

Clu

ster

ed

Tibco Bus

MQqueue

(outbound)x50

MQqueue

(inbound)x50

MQqueue

User Access via Lan

COL-ACT-STA-

1 2 3 4 5 6 7 8 9101112HS1 HS2 OK1 OK2 PS

CONSOLE

CiscoArrowpoint


9

Today’s Java World is More Complex

What’s Different?




(www.)






Data

(Tandem)MVSMQ


RoutingSun 4500



DB2

Clu

ster

ed

Tibco Bus

MQqueue

(outbound)x50

MQqueue

(inbound)x50

MQqueue

User Access via Lan

COL-ACT-STA-

1 2 3 4 5 6 7 8 9101112HS1 HS2 OK1 OK2 PS

CONSOLE

CiscoArrowpoint

Enterprise Integration

Java Apps– Developer / Architect

– DBA

– Distributed Configuration

– Minimal Batch

– Garbage Collection, Heap Size

– Multiple Threads, but sequenced

– Multiple Connections

– Multi-phase Commits

– Routers, Hubs, Switches

– Portals, SSL

– Multiple Browser Flavors

– Scaling up Small Dept. Apps

– Access from anywhere, anytime, anyway, & don’t make me wait


10

Risk Mitigation ?

Testing is Really About Risk MitigationKey Risks to MitigateKey Risks to Mitigate– Risk to Brand– Risk to Revenue– Regulatory or Litigation Risk– Risk to Expense

– Risk to Resume

Risk Mitigation


11

Risk Mitigation Approaches via Load & Performance Testing

Workload must be representative– Provide adequate functional coverage– Realistic implementation & usage patterns– Relevant to the goals

Workload must be measurable– Have well-defined metrics– Provide a stable measurement period to gather

statistics– Key Metrics

• Throughput• Response time• Number of simultaneous requests

(Injection rate or # of concurrent users)

Risk Mitigation


12

Risk Mitigation Approaches cont’d.

Workload must be repeatable– If the test adds or modifies data, DB needs to be

restored each time– Test cases should be repeatable, and controlled

Environment must be as close and true to production as possible

– Hardware, Software, & Infrastructure– Data– Functionality Scenarios

Risk Mitigation


13

Methodology

Establish a baseline


CollectData

CollectData

IdentifyHurdles

IdentifyHurdles

IdentifyAlternatives


ApplySolution

ApplySolution

Use stress tests & performance monitoring tools/techniques

Analyze the data collected to identify potential performance impairments

Identify, research, prioritize, & select mitigation alternatives

Make your changes. Be careful to limit the amount of change introduced

Define your performance targets, configuration, initial settings

Risk Mitigation

Repeat cycle until satisfied


14


Performance Requirements are critical

– Throughput(number of operations per unit of time)

– Response time(amount of time that it takes to process individual transactions)

– Number of simultaneous requests (Injection rate or # of concurrent users)

Consider two sets of requirements

– Launch

– Optimal

Risk MitigationEstablish a

baseline


CollectData

CollectData

IdentifyHurdles

IdentifyHurdles



ApplySolution

ApplySolution


15




(www.)






Data

(Tandem)MVSMQ


RoutingSun 4500



DB2

Clu

ster

edTibco Bus

MQqueue

(outbound)x50

MQqueue

(inbound)x50

MQqueue

User Access via Lan

COL-ACT-STA-

1 2 3 4 5 6 7 8 9101112HS1 HS2 OK1 OK2 PS

CONSOLE

CiscoArrowpoint

Collect Data

Generating Controlled Load– Mercury Loadrunner, Winrunner– Compuware QALoad, QARun– Others or home-grown

Collection Points– Servers stats– Weblogic stats– DB Queries– Network Latency– Application Threads

How?

Risk MitigationEstablish a

baseline


CollectData

CollectData

IdentifyHurdles

IdentifyHurdles



ApplySolution

ApplySolution


16

Tiny Sample - ApplicationVantage

Risk Mitigation


17

Next 3 Steps

Partnership with:– Architects– Developers– DBAs– Network Engineers– Business SMEs– ISP– Server Admins– System Programmers– Test Architect– Project Manager

Avoid a large number of changes at once…Think Cause & EffectThink Cause & Effect



CollectData

CollectData

IdentifyHurdles

IdentifyHurdles



ApplySolution

ApplySolution

Analyze the data collected to identify potential performance impairments

Identify, research, prioritize, & select mitigation alternatives

Make your changes. Be careful to limit the amount of change introduced

Risk Mitigation


18

Phased Reengineering (Mainframe to Distributed)

Challenge– Large high-transaction mainframe system reengineered in phases to a

distributed, browser-based system. Maintain integration with mainframe applications until they are reengineered.

Approach– Supplied team of Java Architect, SW Engineer, Test Architect, & Automation

Engineer. Reviewed design, instrumented & profiled the code, identified performance opportunities, plus performance tested the system

Outcome & Lessons Learned– Recommendations made to development included coding techniques,

optimization of a problematic query, plus knowledge transfer of how & where to profile.

– Defined, coordinated, facilitated, and conducted all aspects of the performance test. Worked with development to ensure changes were implemented.

– Significant functional problems were discovered under load that could not be identified during a typical functional test, including sequencing issues.

– One server identified as having borderline capacity during extreme peak loads.

– New systems must be performance tested prior to implementation– Dedicated performance environment is established complete with regression

test cases.

Case Study


19

Slow Portal

Challenge– New application coming online, with access through existing Portal. Significant

user load expected, but Portal will not handle the growth. Sub-second response time inside firewall, 8 seconds to minutes through portal. Current Security policies, SSL inhibited significant tool diagnostics

Approach– Created load with QALoad, & QARun. Tested remotely from SPR to simulate

outside access. Placed additional players just outside firewall. Utilized ApplicationVantage, ServerVantage, & mass-manual testing. Coordination between networking, application development, infrastructure, Sun Services, and QA.

Outcome & Lessons Learned– Portal response time now meets performance requirements– Just throwing larger servers at a problem does not automatically fix the

problem– Sometimes cultures cannot accept what they do not understand. Did not

understand the internals of a tool.– Many hurdles were identified including threads, heap size, garbage collection,

java version, and a defective hub creating significant volume of retries– Even a Portal needs a Performance Test Environment separate from Production

Case Study


20

Pre-Purchase Decision

Challenge– Subscriber System needed replacing. Build or Buy? Package looked

good but would it scale to support the needs of a paper larger than any other?

Approach– Install a vanilla instance out of the box. Defined critical success

business requirements. Utilized Loadrunner & Winrunner to simulate user interaction. Recorded response times, size requirements, other options.

Outcome & Lessons Learned– Package scaled to support the projected subscriber growth for the next

10 years– Performance degraded significantly when free disk space went below

50%– The business case was adjusted to double the disk space avoiding a

potential implementation project cost overrun– Performance baseline was established for acceptance testing and

written into the implementation services contract– A head start on a solid regression test bed

Case Study


21


Begin with the End in Mind– Define your Test

Requirements - What is Important

– Define launch criteria, & optimal criteria

Simulate Production as Close as Practical– Hardware, Software, &

Infrastructure– Data– Functionality Scenarios

Conducting the Test– Have Functioning code– Test under load– Utilize automation – you’re

going to repeat these tests– Benchmark

You are not alone– Consider the culture– Partner with Architect,

Developer, Test Automation, Business SMEs…

– Compromise carefully – remember the risks being mitigated


Performance TestingApps Christine Jackson Regional Manager Software Quality Assurance & Testing...

Documents

Transcript of Performance TestingApps Christine Jackson Regional Manager Software Quality Assurance & Testing...