Test Smarter (with Pair-wise Testing)

Test Smarter(with Pair-wise Testing)

Presented by: Bill LewisCEO, Smartware Technologies, Inc.

(469) 951-7548www.smartwaretechnologies.com

Copyright 2010 Smartware Technologies, Inc. 2

Agenda

• Challenges of application development• Automated Testing• What is orthogonal arrays?• Power of pair-wise testing• Transforming theory to practice• Summary• Q&A


“V” Model

Coding

Unit TestingDesign

Integration TestingArchitecture

System TestingFunctional &NF Requirements

UA TestingUser Requirements

Project SuccessBusinessRequirements

Static TestingVerify

Dynamic TestingValidate50 % 50 %


Tractability Matrix• A Trace Matrix can be used to track test cases

during development and execution

– Identification of uniquely “testable” requirements• Eliminate wishes, hopes, dreams, and other un-testable

statements

• Eliminate duplicates

• Assign a unique ID for each requirement and design objective

• Determine its priority if possible

– Verification of implementation of all requirements


Traceability Matrix Example


Issues Facing Software Testing Organizations

• Today’s software is getting more and more complex with thousands if not millions of different combinations that need to be tested.

• Humans can’t think of all the test scenarios• Takes too much time• Too error prone• Which scenarios do you test?

• The length of time to test modern software is increasing.

• Requirements documentation is often ambiguous or inadequate

• Testing process is poorly defined or non-existent


Simple Example: Configuration Testing


Simple Example (Con’t)• Suppose you have an application that needs to be tested

on several hardware and software configurations.

• You are told that the application must be able to run on a Windows NT, Windows 2000 and Windows XP machine.

• It also has to work on a system with a minimum of 128Meg and a max of 512Meg of RAM.

• Finally, the application must also run on a Pentium II, III, and IV processor and work with Oracle, SQL and Access Databases.


Simple Example (Con’t)

Operating System

RAM PC Processor

Database

Windows NT 128 Meg Pentium II Oracle

Windows 2000

256 Meg Pentium III SQL

Windows XP 512 Meg Pentium IV Access

Determine test cases for testing the an application on the following:


Combinations of Test Cases3 x 3 x 3 x 3 = 81 Test Cases


Why Automate?

• Over time, number of functional features changes/ increase

and• Number of people and

amount of testing time for each new release

– Remains flat– May even decline

• Test coverage steadily decreases

– Increases the risk of failure– Potential business losses

Source: Linda Hayes, “Why Certify”


• Latest generation of automated testing tools support variable test data

• Difference between static and variable capture/ replay – With Static the inputs and outputs are fixed– With variable inputs and outputs are variable– Replacing the captured inputs and expected outputs

with variables – Variable capture/ replay reduces the risk of not performing

regression testing on existing/ new features

Today’s Test Automation Variable Capture/ Replay Tools


Test Automation Framework

Source: Ross CollardSource: Ross Collard

What’s the Missing Link?

Answer: scientifically generated test data


Simple Example Revisited

Operating System

RAM PC Processor

Database

Windows NT 128 Meg Pentium II Oracle

Windows 2000

256 Meg Pentium III SQL

Windows XP 512 Meg Pentium IV Access

Determine test cases for testing the an application on the following:

* Constraint: Windows XP cannot run on a Pentium II PC Processor.


Optimal Test Cases(via Pair-Wise Analysis)

Test Case #

Operating System

RAM PC Processor Database

1 Windows NT 128 Meg Pentium II Oracle

2 Windows NT 256 Meg Pentium III SQL

3 Windows NT 512 Meg Pentium IV Access

4 Windows 2000 128 Meg Pentium III Access

5 Windows 2000 256 Meg Pentium IV Oracle

6 Windows 2000 512 Meg Pentium II SQL

7 Windows XP 128 Meg Pentium IV SQL

8 Windows XP 512 Meg Pentium III Oracle


Pairwise Definition

Pairwise testing – an approach to combinatorial testing that executes a pairwise test data set.

Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables.

Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way.


Secret Sauce(Orthogonal Arrays)

Mathematical curiosity discovered by monks in the 1400s

Promoted by Dr. Genichi Taguchi in the 1950s for use in statistical test design in manufacturing, agriculture, medicine, etc.

Orthogonal Array: Two dimensional array of numbers in which any two columns in the array receive an even distribution of all pairwise combinations of values in the array.

• Orthogonal arrays are a standard construction used for statistical experiments.


Example Orthogonal Arrays and Notation1 2 3 4 5 6 7

1 0 0 0 0 0 0 0

2 0 1 2 2 0 1 1

3 0 2 1 2 1 0 2

4 0 1 1 0 2 2 3

5 0 2 0 1 2 1 4

6 0 0 2 1 1 2 5

7 1 1 1 1 1 1 0

8 1 2 0 0 1 2 1

9 1 0 2 0 2 1 2

10 1 2 2 1 0 0 3

11 1 0 1 2 0 2 4

12 1 1 0 2 2 0 5

13 2 2 2 2 2 2 0

14 2 0 1 1 2 0 1

15 2 1 0 1 0 2 2

16 2 0 0 2 1 1 3

17 2 1 2 0 1 0 4

18 2 2 1 0 0 1 5

1 2 3

1 0 0 0

2 0 1 1

3 1 0 1

4 1 1 0

L4 (23)

1 2 3 4

1 0 0 0 0

2 0 1 1 1

3 0 2 2 2

4 1 0 1 2

5 1 1 2 0

6 1 2 0 1

7 2 0 2 1

8 2 1 0 2

9 2 2 1 0

L18(3661)

L9(34)


Why is Pair-wise Testing So Efficient?

• Example: Insurance Application (GUI)

• Suppose a bug exists wherein the application produces a garbled record when the spouse's name is empty, but only when the "married" radio button is selected

• With pair-wise testing we’ll find the bug!

Dependents (checkbox)

Marital Status

(radio buttons)

Spouse First Name (text

field less or equal 20

characters)

No Children Single Mary

Dependent Children

Married Greater than 20 characters

Adult Children

Divorced Empty


Exhaustive TestingNo Children Single Mary

No Children Single Greater than 20 characters

No Children Single Empty

No Children Married Mary

No Children Married Greater than 20 characters

No Children Married Empty

No Children Divorced Mary

No Children Divorced Greater than 20 characters

No Children Divorced Empty

Dependent Children

Single Mary

Dependent Children

Single Greater than 20 characters

Dependent Children

Single Empty

Dependent Children

Married Mary

Dependent Children

Married Greater than 20 characters

… 3 x 3 x 3 = 27 tests


Pair-wise Results

When “business rules” (functional coverage) are also applied to the results, the probability of detecting a defect is

dramatically increased.

(Only 9 test cases are required)


Real Results of Pair-wise Testing

• Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects) Clarification

• All single-mode defect will be found if every option is tested at least once (unit elements)• 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect • 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect

“Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically finds 75% of defects Source: Kuhn

NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing

Source: 27th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.

U.S. Food and Drug Administration – study showed 98% bugs discovered using “pair-wise” defect detection Source: 27th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002.

“The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5

• Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week

• In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability

Managing the Test Combination Explosion

Transforming Theory to Practice


How does Smart Test™ work?


Demonstration

Invoke Smart Test™ Generator


Smart Test™ Generated Test Data

QTP Script Ready to Executed


Agenda

• Challenges of application development

• Evolution of software testing• What is orthogonal arrays?• Power of pair-wise testing• Transforming theory to practice• Summary• Q&A


Questions ?


Contact Information

• If you want to contact me– Bill Lewis, CEO & President Smartware Technologies, Inc.– www.smartwaretechnologies.com– (469) 951-7548

Test Smarter (with Pair-wise Testing)

Documents

Transcript of Test Smarter (with Pair-wise Testing)