Agile Open Source Performance Testing Workshop for Business Managers

Agile Open Source Performance Testing

Frank Cohen, +01 (408) 871-0122, [email protected]

January 4, 2011

The Agile Performance Test Methodology and Tools

A Workshop For CIOs, CTOs, Senior Managers

Open Source Test Automation

Slides on Twitter‣http://twitter.com/fcohen

‣Please retweet

‣Please follow me

‣#ost

2


About PushToTest‣Our Mission: Test, To Make A More Reliable World

‣Make It Easier To Move From Manual To Automated Testing

‣Support Advanced Testing, including Agile, Load, Integration

‣Open Source Testing (OST) For Everyone

3


Web 2.0 Success Driving Business‣Rich Internet Applications

• Fast Time-To-Market

• Complex Business Workflows

• Client-less Access

‣SOA Composite Applications

• Faster Time-To-Market

• Increase Revenue

• Cost Savings

‣Service Enabled Applications

• Deliver More Integration Faster

• Less Cost4


When OST Is Your Best Choice

5

‣Modern Agile Software Development Techniques

‣Highly-Usable, Highly-Functional Applications for Web Deployment

‣Complex Data, Sophisticated User Interaction, Application Business Logic (Workflow)

http://www.pushtotest.com/10-ways-to-build-and-test-rich-internet-applications-ria.html


Agile Repurposing‣Agile Delivers More Projects Faster

‣Requires Repurposing Between Teams

‣Treats Tests As Business Asset (Version Control, Sharing, Reuse)

6

DevTest

Biz

IT

Business Requirements to LaunchIntegration Test and SLA Compliance

Functional Tests Repurposed AsLoad and Performance Tests


‣Proprietary Test Tools Offer:

‣QA Labs for 10,000 Users = $35,000 Per Year

‣Does Not Include: Cost of Test Employees, Root Cause Analysis Consultants

‣Does Not Guarantee Success

Testing Costs (TCO Model)

7

Functional Test

Functional Test Platform $19,000

Functional Tests $8,000

Protocols: AJAX, SOAP $12,000

Support $5,000

$40,000

Load Test

Load Test Platform $20,000

Simulated User Licenses $120,000

Recoding The Test $35,000

Re-qualify The Test $12,000

$187,000


What We Have For You

8

‣ Implement effective tests

‣Deploy tests to desktop, grid, and cloud environments

‣A clear tutorial approach to the PushToTest methodology

‣How It All Fits Together


OST Architecture

9

Functional

Test

Production

Monitor

Load Test

Continuous Integration

Repository Bug TrackerTest

ManagementMonitoring

Test Authoring

SoapUI (Pro)

Sahi (Pro)

JUnit Class Framework

Eclipse Java IDE

Fitness

Languages: Java, Python

Wiki Collaboration

Application Under TestWeb, RIA, BPM, SOA

PTTMonitorCPU, Net, Memory

Test Injector

BundlesMySQL, HtmlUnit

ScriptRunnersSahi, Selenium, SoapUI, Java

Data Production Librariescsv files, rdbms queries

Test Operation Console

RepositoryMySQL RDBMS

Results AnalysisBusiness Intelligence Tool

DesignerRecord/Playback

Grid and CloudTest Deployment


PTTMonitor

Load Injectors

An OST Cloud

10

Application Under Test

Central location from which a tester, developer, IT manager, CI service runs and monitors tests

TestMaker

Console

Thousands of virtual users perform real-life scenarios on the Application Under Test

TestNode

Stores results of every transaction, step, and monitored resource

Results DB

400+ charts identify the root cause to performance bottlenecks, functional issues

Analysis


PTTMonitor

Load Injectors

An OST Cloud

11

Application Under Test

Central location from which a tester, developer, IT manager, CI service runs and monitors tests

TestMaker

Console

Thousands of virtual users perform real-life scenarios on the Application Under Test

TestNode

Stores results of every transaction, step, and monitored resource

Results DB

400+ charts identify the root cause to performance bottlenecks, functional issues

Analysis


Component Approach

12

Data Production

Log-In

Component

setUp

runTest

tearDown

ApplicationTestScenario

Test Use Case

Product Search

Order Product

Log-Out

RDBMS DPL

CSV DPL

Custom DPL

Load Test

100 Users 500 Users 1000 Users

Log-In

Component

Component

Development

Database

QA Regression

Database

Load Test

Database Confirm Order

Production

Database


Ready For Load Testing?‣Selenium Core and RC Require A Browser

‣Selenium IDE Record In Firefox Only, Coding Required To Be Useful, No Data Enablement

‣TestMaker Object Designer for recording in IE

‣Selenium Grid for Functional Testing Only

‣No Reports or Results Analysis

‣No Cloud or Grid Deployment

‣No CI or QC Integration

‣RIA Testing using Ajax, Flex, Flash13


Test Authoring and Scale

14

Approach Advantages and Disadvantages Scale

Protocol Level Testing1000+ VUs per TestNodeScript-level codingExtra code for JavaScript

1000+ VUs

HTTP Archive (HAR)1000+ VUs per TestNodeScript-less Test AuthoringAutomatic for JavaScript

1000+ VUs

SeleniumHtmlUnit Record/PlaybackOperates client-side logic 20-200 VUs

Browser Functional TestingSame Test ScriptVisual Test DebuggingSlow Test Operation

2-4 VUs


15

Repurposing


TestMaker Demonstration‣Fun And Easy Test Script Creation

‣Selenium and TestMaker

‣Repurposing A Single Test

‣Functional Test, Load and Performance Test, Production Monitor

‣Results Analysis

‣Chart Engine

16


Calibration Testing

17


Expensive For Time

18


Start In The Middle

19


Work Down or Up

20

Next let’s talk about


PushToTest Options

‣OnDemand

‣QuickStart

‣POC Option

21


Immediate Benefits‣Product Solutions

‣Better, Lower-Cost, More Efficient Test Tool

‣Business Service Monitoring at the Application Tier (SLA Compliance)

‣Methodology Solutions

‣ Improved Test Methodology for my Organization and Team

‣Services

‣Turnkey Load Testing

‣ Integrate TestMaker Into My Build Environment

‣Add A Protocol, Chart, Task to TestMaker To Meet My Needs22


Individuals and Organizations

‣ Functional Testing with TestMaker Object Designer

‣ Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance tests, and business service monitors

‣ Load Testing To 50 Virtual Users

‣ Data-driven tests using CSV files.

‣ 25 Charts and Reports

‣ Community Support and access to all documentation23

‣ Load Testing to Millions of Virtual Users

‣ Cloud and Grid Test Deployment

‣ Collaborative Test Object Repository

‣ Data-driven tests, RDBMS Data Production Libraries

‣ Collaborative Results Analysis Repository

‣ Flex/Flash 3.0, 4.0, AMF

‣ 1000+ Charts and Reports

‣ Support, QuickStart, Training, Consulting

For the individual tester needing a no-cost test tool to test Web and RIA for functionality, load and performance, and monitoring.

For organizations running large scale business-critical Web, RIA, SOA, BPM and needing world-class tools, methodology, training, and support.


Pricing‣Annual Subscription Model Based On:

‣Seats for TestMaker Console Installation

‣Seats for SOAP and REST Test Authoring

‣Virtual User Levels for Load Testing

‣Count of Business Service Monitor

‣Technical support: 1 business day or faster response to emails, phone calls, instant messages

24


25

‣As-Needed, On-Demand Basis

‣Test In Your Environment, Or Our Cloud Test Environment, Or Both

‣ Ideal For Consulting Organizations Where Test Needs Are Unknown

‣AMI Distribution

‣Pay Only For What You Use

‣Annual Cap On Total Costs


The QuickStart Option

26

‣Your First Test, Built and Operated by PushToTest Experts

‣ Ideal For Organizations With Little or No Load Testing Experience

‣PushToTest Experts Build Your First Test

‣PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation

‣PushToTest Trains Your Team To Build Their Own Tests

‣$10,000 USD


The POC Option‣PushToTest Experts Help Your TestMaker Evaluation

‣Ensures TestMaker Works In Your Application/System

‣PushToTest Issues Price Proposal

‣Applies Engineers To Build and Operate Your Test

‣PushToTest Trains Your Team To Use TestMaker

‣Success of POC Confirms Intention To Purchase

‣Cost: $027


PushToTest Global Services‣Put Our Experts To Work For Your Projects

‣Methodology, Solutions, Support, Enthusiasm

‣Offices: USA (Texas and California,) Bangalore, London

‣The Near Shore Option

‣150 Test Experts

‣Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours

‣http://www.pushtotest.com/services

28

“PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our

Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, Tribal DDB Dallas


Avoid Retreats‣Load Test Shows: 3,230 Success, 12 Failures

‣Avoid The Urge For Perfection

‣Even The Best Managers Sometimes Go Back To Functional Testing

‣Even Failed Transactions Generate Load

‣Sometimes the whole point is...

29


30


31

Alex: load test is running fineInga: Coolio!Alex: Uh ohAlex: Success: 3210, Failure: 15Alex: Success: 3210, Failure: 204Alex: Success: 3210, Failure: 483Inga: What’s up?Alex: The whole application failedInga: Yep, avail threads = 0Alex: Back to the drawing board


32

Fear NotWe get paid to break things.

When you break it you understand it.When you understand it you can fix it.


Slides on Twitter‣http://twitter.com/fcohen

‣Please retweet

‣Please follow me

‣#ost

33


Where To Go From Here‣Evaluate Selenium, soapUI, TestMaker

‣http://www.pushtotest.com/products-comparison

‣Watch A Screencast and Tutorial

‣http://www.pushtotest.com/screencastcentral

‣Attend An Open Source Test Workshop

‣http://workshop.pushtotest.com

‣Ask For A Proposal on Licenses, Training, Support, Consulting

‣Call +01 408 871 0122, [email protected]

34

Agile Open Source Performance Testing Workshop for Business Managers

Technology

Transcript of Agile Open Source Performance Testing Workshop for Business Managers