Software and System

Test Documentation

IEEE 829 - Standard for Software Test Documentation

TEST PLANNING AND CONTROL

Master Test Plan

Level Test Plan

TEST ANALYSIS AND DESIGNLevel Test Design

Level Test Case

TEST IMPLEMENTATION AND EXECUTION Bug Report

Level Test Procedure

EVALUATING EXIT CRITERIA AND REPORTING

Interim Test Status Report

Test Log

Level Test Report

TEST CLOSURE ACTIVITIES Master Test Report

LEVELS OF TEST PLANDevelop Master Test Plan

Develop Detailed Test Plans

Unit Test Plan

Integration Test PlanSystemTest Plan

AcceptanceTest Plan

Level of Test Plan defines what the test plan is being created for.

Test Plan document follows the same structure for each level of test plan. The only difference is the content and detail.

Unit+Integration Test Plans Example

System+Acceptance Test Plans Example

Master Test Plan Example

TEST PLANNING AND CONTROL

MASTER TEST PLAN

The purpose of the MTP is to:

provide the overall framework for all the testing activitiesdefine the scope of the testing

identify whether there is any unnecessary duplication of testing taking placeidentify the departures from the Test Process documentation set

define the approach to each stage of testingspecify the ABC project’s staff’s responsibilities for testing activities for each stage of testing

A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process.

A test plan that typically addresses multiple test levels.

A test plan that typically addresses one test phase.

Test Plan

The format and content of a software test plan vary depending on the processes, standards, and test management tools being implemented. Following format provides a summary of what a test plan can/should contain.

1) Test Plan ID: Unique No or Name2) Introduction: Provide an overview of the test plan, specify the

goals/objectives.3) Test Items: Modules/ Functions/ Services/ Features/ etc.4) Features to be tested: Responsible Modules for Test design5) Features not to be tested: Which feature is not to be tested

and Why? 6) Approach: List of selected testing techniques to be applied on

above specified modules in reference to the TRM (Test Responsible Matrix)

7) Feature pass or fail criteria: When a feature is pass or fail description

8) Suspension criteria9) Test Environment: Required software & Hardware to be tested

on above features10)Test Deliverables: Required testing document to be prepared11)Testing Task: Necessary tasks to do before start every feature

testing12)Staff & Training:   Names of selected Test Engineers & training

requirements to them13)Responsibilities:   Work allocation to every member in the team14)Schedule:   Dates & Times of testing modules15)List & Mitigation:   Possible testing level risks & solution to

overcome them16)Approvals:   Signatures of Test plan authors & Project Manager /

QA

Wha

t to

test

?H

ow to

test

?W

hen

to

test

?

TEST DESIGN LEVELS STRUCTURE BASED ON THE V-MODELOverall Business Requirements

Acceptance Test Design

Software requirements

System Test Design

High level requirements

Integration Test Design

Low level requirements

Component Test Design

Coding Unit Test Design

Unit Test Execution

Component Test Execution

Integration Test Execution

System Test Execution

Acceptance Test Execution

TEST ANALYSIS AND DESIGN

TEST DESIGNTest Design Phase – In software engineering, test design

phase is a process of reviewing and analyzing test basis, selecting test design techniques and creating designed test cases, checklists and scenarios for testing software.

Test Design Specification - it is a document that describes features to be tested and specifies list of all test scenarios or test cases, which should be designed for providing the testing of software. Basically test design is the act of creating and writing test suites for testing a software.

Test design could require all or one of:1) Knowledge of the software, and the business area it operates on2) Knowledge of the functionality being tested 3) Knowledge of testing techniques4) Planning skills to schedule in which order the test cases should be designed, given the effort, time and cost needed or the consequences for the most important and/or risky features

Review and Analyze Test

Basis

Select Test Design

Techniques

Create Test Design

Specification

Create Test Cases

Specification

Test Plan

Requirements Mock-

ups

Test Design

Specification

Test Case Specificati

on

BASIC TEST DESIGN STEPS:

• The internal factors that influence the decision about which technique to use are: – Tester knowledge and experience – Expected defects – Test objectives – Documentation – Life cycle model

• The external factors are: – Risks – Customer requirements – System type– Time and budget

Choosing A Test Design Technique

According to IEEE-829 standard template structure looks in the following way:

1. Test Design Specification Identifier1.1 Purpose1.2 References1.3 Definitions, acronyms and abbreviations

2. Features to be Tested3. Approach Refinements4. Test Identification

4.1 <Test Item 1>4.2 <Test Item …>4.3 <Test Item N>

5. Feature Pass/Fail Criteria

Test Design Specification Structure

Belo

w –

th

e

exp

lan

ati

on

3)Approach Refinements section describes the following:– Specific test techniques to be used

for testing features or combinations of features

– Types of testing which will be provided

– Methods of analyzing test results– Test results reporting– Whether automation of test cases

will be provided or not– Any other information which

describes approach to testing

Test Design Specification Structure

1) Test Design Specification Identifier section covers:– Purpose of the document– Scope of the document– List of references which should

include references on test plan, functional specification, test case specification, etc.

– Definitions, acronyms and abbreviations used in Test Design Specification

2) Features to be Tested identifies test items and describes features and combinations of features that are the object of this design specification. Reference on Functional Specification for each feature or combination of features should be included.4) Feature Pass/Fail Criteria specifies the criteria to be used to determine whether the feature or feature combination has passed or failed.

The following items can be considered as “pass / fail criteria”:

1) Feature works according to stated requirements

2) Feature works correctly on the test platforms

3) Feature works correctly with other modules of application

4) All issues with High and Medium Priority will be verified and closed

5) Test Identification section is separated to sub-section

according to the amount of test items identifying future

documentation which will be created for testing features or

combinations of features that are the object of this design

specification.

Features can be covered by test objectives in different ways

depending on projects needs, approaches for testing etc.

Three examples of such

coverage:

Feature covered by test cases

Feature covered by test

scenarios Feature covered

by checklists

Test Design With Scenarios

Test Design With Test Cases

Test Case Specification

Test Specification – It is a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Revision History - This section contain information like Who created the

test specification? When was it created? When was the last time it was updated?Feature Description – A brief description of what area is being tested.What is tested? – An overview of what scenarios are tested.What is not tested? - Are there any areas that are not being tested. Nightly Test Cases – A list of the test cases and high-level description of what is tested whenever a new build becomes available.Breakout of Major Test Areas - It is the most interesting part of the test specification where testers arrange test cases according to what they are testing.Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.Security tests – Any tests that are related to security.Accessibility Tests – Any tests that are related to accessibility.Performance Tests - This section includes verifying any performance requirements for your feature.Localization / Globalization - tests to ensure you’re meeting your product’s Local and International requirements.NOTE: Test Specification document should prioritize the test case easily like nightly test cases, weekly test cases and full test pass etc:Nightly - Must run whenever a new build is available.Weekly - Other major functionality tests run once every three or four builds.Lower priority - Run once every major coding milestone.

Con

ten

ts o

f a T

est

Sp

ecifi

cati

on

:

Test cases are a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. There must be at least one test case for each requirement for traceability.

Test Cases

Test Case

Test Step – specifies an

action to perform, and the expected response of the test application.

For example: Action : Type the password in the password box,

Expected result: Your password

should be dotted / hidden.

Test Case – a list of test steps. Also

defines the environmental

situation and may link to related

bugs, requirements etc.

Test Scenario – usually comes directly from

business requirements or

user story. Scenario contains a list of test cases

and often their sequence.

Test scenario Test case Test Step

Standard fields of sample test case template:

Test case ID: Unique ID for each test case. Test priority (Low/Medium/High): This is useful while test execution. Test priority for business rules and functional test cases can be medium or higher whereas minor user interface cases can be low priority. Module Name – Mention name of main module or sub module.Test Designed By: Name of testerTest Designed Date: Date when wroteTest Executed By: Name of tester who executed this test. To be filled after test execution.Test Execution Date: Date when test executed.Test Title/Name: Test case title.Test Summary/Description: Describe test objective in brief.Pre-condition: Any prerequisite that must be fulfilled before execution of this test case. List all pre-conditions in order to successfully execute this test case.Dependencies: Mention any dependencies on other test cases or test requirement.Test Steps: List all test execution steps in detail. Write test steps in the order in which these should be executed. Make sure to provide as much details as you can. Test Data: Use of test data as an input for this test case. You can provide different data sets with exact values to be used as an input.Expected Result:  What should be the system output after test execution? Status (Pass/Fail): If actual result is not as per the expected result mark this test as failed. Otherwise passed.Notes/Comments/Questions: To support above fields if there are some special conditions which can’t be described in any of the above fields or there are questions related to expected or actual results mention those here.

Fields of sample test case template, if necessary:

Defect ID/Link: If test status is fail, then include the link to defect log or mention the

defect number.

Test Type/Keywords: This field can be used to classify tests based on test types. E.g. functional, usability, business rules etc.

Requirements: Requirements for which this test case is being written. Preferably the exact

section number of the requirement doc.

Attachments/References: This field is useful for complex test scenarios.

Automation? (Yes/No): Whether this test case is automated or not. Useful to track automation

status when test cases are automated.

The purpose of an LTPr is to specify the steps for executing a set of test cases or, more generally, the steps used to exercise a software product or software-based system item in order to evaluate a set of features.

1. Identification (Each Test Procedure Specification should be assigned a unique identifier.)

2. Purpose (explain the purpose of the test procedure and reference any test cases it executes.)

3. Special Requirements (list any special hardware, software or training requirements for this procedure.)

4. Procedure Steps (actual steps of the procedure are described. IEEE has listed several steps:

Log: explain methods / formats for logging results of test execution.

Set up : actions necessary to prepare for the execution of procedure.

Start : actions necessary to begin execution of the procedure.

Proceed : actions necessary during the execution of the procedure.

Measure : explain ho test measurements need to be made.Shutdown : actions necessary to suspend testing when

some unscheduled event happens.Restart : actions necessary to restart the procedure. Stop : actions necessary to bring execution to systematic

halt.

Level Test Procedure

Test

Pro

ced

ure

S

pecifi

cati

on

sh

ou

ld h

ave t

he fo

llow

ing

ele

men

ts:

Test Procedure

TEST IMPLEMENTATION AND EXECUTION

ID Unique identifier given to the defect.

Project Project name.

Release Version Release version of the product. (e.g. 1.2.3)

Module Specific module of the product where the defect was detected.

Detected Build Version

Build version of the product where the defect was detected (e.g. 1.2.3.5)

Summary Summary of the defect. Keep this clear and concise.

Description Detailed description of the defect. Describe as much as possible but without repeating anything or using complex words. Keep it simple but comprehensive.

Steps to Reproduce Step by step description of the way to reproduce the defect. Number the steps.

Actual Result The actual result you received when you followed the steps.

Expected Results The expected results.

Attachments Attach any additional information like screenshots and logs.

Remarks Any additional comments on the defect.

Defect Severity The seriousness of defect with respect to functionalityHigh(Show Stopper):-Without fixing this defect tester not able to

continue testing.Medium:-Able to continue but mandatory to fix.Low:-Able to continue testing but may or may not to fix. 

Defect Priority The importance of defect fixing with respect to customer interest.

Reported By The name of the person who reported the defect.

Assigned To The name of the person that is assigned to analyze/fix the defect.

Status The status of the defect. (new/reopen)

Fixed Build Version Build version of the product where the defect was fixed (e.g. 1.2.3.9)

DEFECT REPORT

Test Log

Test Log is details of what tests cases were run, who ran the tests, in what order they were run, and whether or not individual tests were passed or failed. The purpose of the Level Test Log is to provide a chronological record of relevant details about the execution of tests. An automated tool may capture all or part of this information.

Test Log

EVALUATING EXIT CRITERIA AND REPORTING

Interim Test

Status Report

The purpose of the ITSR is to summarize

the results of the designated testing

activities and optionally to provide

evaluations and recommendations

based on these results.Interim Test Status Report

Eight Interim

Reports:

Functional Testing Status 

Functions Working Timeline

Expected verses Actual

Defects Uncovered Timeline

Defects Uncovered verses

Corrected Gap

Timeline

Average Age of

Uncorrected

Defects by Type

Defect Distributio

n

Relative Defect

Distribution

Testing Action

Functional Testing Status Report

Fully tested

Tested with open defects

Not tested

This report will show percentages of the

functions which have been:

Functions Working Timeline

This report will show the actual plan to have all functions working

versus the current

status of functions working.

An ideal format could be a line

graph.

Expected versus Actual Defects Uncovered Timeline

This report will provide an analysis between the number of defects being generated against the expected number of defects expected from the planning stage.

This report, ideally in a line graph format, will show the number of defects uncovered versus the

number of defects being corrected and accepted by the testing group.

If the gap grows too large, the project may not be ready when

originally planned.

Defects Detected versus Corrected Gap

This report will show the average days of outstanding defects by type. In the planning stage, it is beneficial to determine the acceptable open days by defect type.

Average Age Uncorrected Defects by Type

This report will show the defect distribution by function or module. It can also show items such as numbers of tests completed.

Defect Distribution

This report will take the previous report (Defect Distribution) and normalize the level of defects.   An example would be one application might be more in depth than

another, and would probably have a higher level of defects. However, when normalized over the number of functions or lines of code, would show a more

accurate level of defects.

Relative Defect Distribution

This report can show many different things, including possible shortfalls in testing.   Examples of data to show might be

number of Sev 1 defects, tests that are behind schedule, and other information that would present an accurate testing

picture.

Testing Action

Level Test

Report (LTR)

The purpose of the Level Test Report

(LTR) is to summarize the results of the

designated testing activities and to

provide evaluations and

recommendations based on these

results

Master Test

Report

The purpose of the MTR is to summarize the results of the

levels of the designated testing activities and to

provide evaluations based on these results. This report may be used by any organization using the MTP. Whenever an

MTP is generated and implemented, there needs to be a corresponding MTR that describes the results of the

MTP implementation

Master Test Report

TEST CLOSURE ACTIVITIES