Fundamental Test Design Techniques

TC AM Tutorial

10/14/2014 8:30:00 AM

"Fundamental Test Design Techniques"

Presented by:

Lee Copeland

Software Quality Engineering

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Lee Copeland

Software Quality Engineering With more than thirty years of experience as an information systems professional at commercial and nonprofit organizations, Lee Copeland has held technical and managerial positions in applications development, software testing, and software process improvement. At Software Quality Engineering, Lee has developed and taught numerous training courses on software development and testing issues, and is a sought-after speaker at software conferences in the United States and abroad. Lee presents at software conferences in the United States and abroad. He is the author of the popular reference book, A Practitioner’s Guide to Software Test Design.

Mastering Test Design V 3.2 © SQE 2001-2006 1

This is the definition for “test” from IEEE Std 610, which is a dictionary of technical terms related to software. It is both a verb and a noun.


technical terms related to software. It is both a verb and a noun.

This and all other IEEE Software Engineering Standards are available from computer.org (ieee.org is for all IEEE standards, including many related to hardware. It has a link to the Computer Society’s site, which is where the software related standards are found.

The definition of what constitutes a “test case” varies. This is the definition for purposes of this class. It includes everything needed to execute the test in a


purposes of this class. It includes everything needed to execute the test in a reproducible fashion, which includes:

• One or more data values which are to be input

• The trigger event that starts the test

• One or more data values which will be generated as a result of the input(s) being processed (the output)

• The tools and data required to allow the test to execute (the environment)

• The order in which the tests are to be run

Often, the only inputs considered are those entered through the user interface. In reality, there are often many other inputs.


In reality, there are often many other inputs.

Outputs are just as varied as inputs.


Testing every possible data value, every possible navigation path through the code, or every possible combination of input values is almost always an infinite


code, or every possible combination of input values is almost always an infinite task which is impossible to complete. Even if it were possible, it is not a good idea because many of the test cases would be redundant, consume resources to create, delay time to market, and not add anything of value.

Since it is impossible to test everything, the coverage is based on trade-offs, considering factors such as the following:


considering factors such as the following:

• How much time is available to prepare and/or execute tests

• The most cost-effective approach in terms of the return on the investment

• Priorities based on risk (test more of higher risk)

• Use tools when appropriate

• Find as many defects as possible in the time available

This series of sections is the heart of the tutorial.

This chapter includes the “scientific” methods—the ones with definitions and


This chapter includes the “scientific” methods—the ones with definitions and specific ways for selecting test data. It is the largest and most comprehensive chapter in the tutorial. It is followed by white box or structure based tests. The last chapter in this section will address the “creative” (not structured) ways of testing.

Each of these techniques has certain capabilities and limitations.There are some types of defects/problems that each technique can detect and there are


some types of defects/problems that each technique can detect and there are some things that the individual techniques are not designed to detect. By combining several techniques the individual strengths can be used to offset the weaknesses.

The techniques presented in this chapter apply to all levels of test—from a single unit to many interfacing systems.The common factor is the emphasis on


single unit to many interfacing systems.The common factor is the emphasis on input data ranges.

There are four basic rules to defining equivalence classes. These are noted here and on the next slide.


here and on the next slide.

Depending on the complexity of the input it may also be necessary to subdivide a partition into smaller partitions. Although the need to subdivide may be predicated on what the set of values for a partition is.

For example:

The invalid partitions above may also contain alphabetic or special characters and possibly negative numbers. These particular values can be used in either invalid partition, essentially subdividing that partition into four groups: Numeric, negative numeric, alphabetic and special character.

Whether you do this subdivision or not is up to the tester. It may not be necessary as these are actually “field type” tests and could be done as a separate test depending on how the fields are created for the user interface.

If a developers kit is used to define the fields a single test of the kit for these events may be sufficient.

You can also define classes for outputs as well as inputs.


Although the technique focuses on one requirement at a time it is permissible to create test cases looking at all the requirements as a set. This will reduce the


create test cases looking at all the requirements as a set. This will reduce the number of test required to validate the requirements under analysis.

If we were to write test to validate each individual requirement the number of tests could increase substantially, especially if many requirements are involved.

Most will note that these are not the “best” test cases.The key is to define the “minimum” number of tests required to test the requirements “partitions.”


“minimum” number of tests required to test the requirements “partitions.”

By combining requirements together it takes one test to check all the valid partitions and six to test all the invalid partitions. We have to test invalids individually because we do not know how the developer coded the error handling (this is black box) and it is possible that the first error could stop the test. Later, if we discover the developer uses a table (or similar construct) to show all errors, we can combine the error tests as we did the valid tests.

One of the key reasons to do Equivalence Partitioning is to locate the boundaries. If developers are going to make a mistake they will probably do it at


boundaries. If developers are going to make a mistake they will probably do it at a boundary.

As noted sets are different. To ensure the set is complete and defined correctly each member of the set is tested to ensure the set is coded correctly as they


each member of the set is tested to ensure the set is coded correctly as they don’t actually have a boundary. So the third valid test is for the type of property variable. You can choose any valid value for the other partitions.

These values do a better job of testing the partitions and the boundaries of the partitions. You will notice that by focusing on multiple inputs together (four corners) we don’t actually increase the number of tests but make them more effective.

Invalids must still be tested individually for the same reasons previously noted.

The ground rule would be to have every test include (“cross”) at least three functional areas. For example, Test 1 includes Remote Application 1, Mail


functional areas. For example, Test 1 includes Remote Application 1, Mail server 1, and the User interface. Test 3 crosses five functional areas.

Conditions “1” through “X” show the allowed input conditions


Actions “1” through “Z” are the actions that will be taken based on combinations of conditions.

Each Rule defines a unique set of conditions that result in the indicated set of actions taking effect. Actions do not care about the order of the conditions or the order in which they are checked, the key is their combined values.

Conditions that are simply yes or no events, binary decisions, are very easy to map to a decision table.


map to a decision table.

You’ll notice that rules one and two contain a “immaterial” condition. The “Unearned income >$1560” has no affect on the resulting action. Technically we could ignore this value in testing, however, if we did how could we be sure that it did not alter the outcome/action?

If claims and age are valid, then the three actions will be applied accordingly.


If claims or age is invalid, there is no action to take. In fact it is “impossible” to have an invalid claim. If you enter a negative number this would be a “field type” test and in reality you would not file a “negative” claim. All claims above 5 are treated the same, no matter how many.

As noted earlier, you may have a license and be over 85 or under 16 but this application does not accept those values and would generate an error. If we wanted to we could add a fourth action, generate error message.

Here is where we can combine techniques.

Using EP/BV and four corners, we can reduce the number of tests required for


Using EP/BV and four corners, we can reduce the number of tests required for each rule.

• The lower boundary value of all but Rule 1 puts you into another rule.

• The upper boundary value of all but Rule 8 puts you into another rule.

This is a simplified diagram for making an airline reservation.


After each user activity “event,” the system/application is in some “state.” It is this state that will affect what the requestor can or cannot do next. To test this process, you need to know how to perform each activity, what the acceptable pathways are supposed to be, and the business rules that apply, before, during and after each state and event.

This type of diagram helps you plan your functional testing to ensure you not only exercise the functions of the systems but also check the system/design itself.

Trace Matrices can support multiple levels. This is an example of a four level matrix.


matrix.

This is an example of a Traceability Matrix linking requirements to design. It can be used for any predecessor/successor pair of documents. It can also be used


be used for any predecessor/successor pair of documents. It can also be used to document any requirements which are NOT going to be addressed by the tests. Some useful optional additional columns:

• Comments: any questions, assumptions, etc.

• Priority: the importance of the requirement (with the implication that the lower rated ones have less tests which may not even get run when time is short).

This is a summary of the techniques in this section vs. the possible applicable levels of Black Box testing. It is not intended in any sense to be restrictive - use


levels of Black Box testing. It is not intended in any sense to be restrictive - use whatever works in your environment.

This is the last of the Test Design Techniques sections. It includes some very interesting tests - those which are NOT based on a logically structured


interesting tests - those which are NOT based on a logically structured technique!

All of the scientific techniques in the last two chapters are of great value, but there is still more needed. Since the programming field tends to attract people


there is still more needed. Since the programming field tends to attract people who are very logical, it is helpful to have someone—who can do the illogical—test the system, too. After the system is in production, real circumstances will not be constrained to only logical processes, so it is good to test them before the system goes into use.

This is a talent that some people have, and some don’t (the author of this course is a “don’t”). Some people can just walk up to a system, take one look at


course is a “don’t”). Some people can just walk up to a system, take one look at it, and say, “I bet if I do this it will break,” and be right! It is great for a testing team to have people with this kind of talent participating.

One of many excellent James Bach quotes!


This is a formalization of what many testers have been doing for a long time. Even with specifications, testers rarely have all of the information they really


Even with specifications, testers rarely have all of the information they really need. And not everyone gets a complete set—or sometimes any—specifications!

James Bach, the leader in this field, is researching and working on quantifying what a good tester does when confronted with unproven software and not enough information.

These are the basic steps for exploratory testing, which consists of starting with whatever we know and going forward in a constructive manner. This is based on


whatever we know and going forward in a constructive manner. This is based on James Bach’s description of the process.

The last category has no bounds at all. Just do the tests totally illogically!


These slides on Taxonomies are based on information, provided by James Bach and Cem Kaner in their course on “black box testing” and from Lee Copeland’s


and Cem Kaner in their course on “black box testing” and from Lee Copeland’s book, A Practitioner’s Guide to Software Test Design.

These are sources for some example Taxonomies


This is Boris Beizer’s high level taxonomy for where defects can occur.


You may want to build a similar taxonomy based on your applications and systems. You can use one of the samples noted to get started.

Even considering the top two levels, it is quite extensive. All four levels of the taxonomy constitute a fine grained framework on which to categorize defects.


taxonomy constitute a fine grained framework on which to categorize defects.

Your instructor will review which parts of the class helped with the challenges identified in the beginning of the course.


identified in the beginning of the course.

Some suggestions for your evaluations!


Hope this class really helped!


Fundamental Test Design Techniques

Technology

Transcript of Fundamental Test Design Techniques