1

Ian McDonaldImplementing Test Scripting

© 2006, 2007, 2010, 2013 Ian McDonald

2

Contents

PurposeOverviewTest CasesManual Test ScriptsAutomated Test Scripts

3

Purpose These slides were put in place over a period of

time, as I found myself often having to lead teams on fast delivery projects, where testers had no previous experience or training. It is very much a quick guide and assumes that the slides are supported by a skilled practitioner leading the team. The approach assumes basic level tooling. Obviously with specialist tooling the approach will be adapted.

There is also an assumption that other training is also available to support test analysis to create test cases and the use of specific tools.

4

Overview

This presentation is written to help those who are new to test scripting, who may have had little previous experience in this task.

The slides address: The layout and management of Test Cases. Naming requirements for Test Scripts. Limitations imposed by Test Tooling. Creation and content of Test Scripts. Structuring of Automated Test Scripts.

5

Test Cases

A Test Case is described as: “A set of inputs, execution, preconditions, and

expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.” BS7925-1

Tools like Classification Tree Editor (CTE) allow users to generate Test Cases in terms of lists identifying input parameters (or Test Vector sets). NOTE: A separate presentation is available that deals with CTE.

The Test Cases will therefore form a list of what needs to be tested, against a Test Case will be hooked in one or more Test Scripts which provide step by step instructions in how to implement the Test Case.

6

Test Case and Test Script

A test case defines the parameters that are the vector values which are input for a specific set of tests. Defined with the input values are corresponding output values that are expected for the test outcome.

A test script details the step by step actions and observations to implement the input of test vectors and observations.

Often a test case may be implemented in a number of different scripts. This might occur, for example, if there are different input methods.

7

Creation of Test Cases

Test cases are identified through systematic analysis, to identify the test vectors to be used for testing.

Techniques such as boundary analysis, equivalence partitioning, state transition testing, etc can be found in BS7925-2. These techniques can be applied to system as well as unit testing.

Note: Details of BS7925-1 and BS7925-2 can be found at:http://www.testingstandards.co.uk/

An additional technique is classification tree (CT), which is also supported by the classification tree editor (CTE) tool.

Note: The CT approach and the CTE tool are described in a separate presentation.

8

Test Case Work Packages

Following test analysis a series of work packages are identified. These are the individual test cases, or test vector sets.

These test cases then need to be developed into a series of test scripts.

The test scripts outline the starting assumptions and preparation for implementing a test, followed by step by step clear repeatable actions and the observations to make and what is expected to be observed for a successful outcome.

Later a test manager will want to run specific scripts associated with test cases.

9

Structuring Test Cases – A Practical Approach

Test Cases are structured into well organised folders to allow efficient management. During a test programme, the Test manager will need to: Assign test cases for the creation of test scripts. Run scripts for specific test case sets for different software

deliveries and project milestone phases. Assign specific work packages to testers (so a tester will be

given a defined set of tests to run within a period of time). Prioritise testing for a defect fix, software delivery or

confidence test. Consequently Test Cases and Scripts need to be well

organised and the organisation well understood and easily recognised by the team.

10

Early Organisation of Test Cases

Test Cases are usually organised under a series of folders. The organisation may differ from project to project. In structuring Test Cases or Scripts the following will need to be considered as a possible workable approach for organising Tests: Requirements Business Use Cases Functional Grouping Equipment Functionality Data Type Other….

11

Getting Started

When creating Test Cases, it may be that the Test Cases have been created with tooling such as CTE and the test cases have been created as a flat table or series of flat tables. The test manager will want to group these for assigning to testers to create Test Scripts.

It is often easier to initially assign the test cases to high level folders and then as the Test Scripts are created, the Test Cases are moved to a more appropriate folder structure grouping.

12

Setting Out Test Cases

Test Cases can be stored initially at a high level and as sorted then transferred to lower level folders.

Portal

Server Agent

Project ABC

Admin

User Set-Up

System User Functions

Positive Test

Negative Test

13

Considerations for Priority and Regression

Typically during testing a test manager will need to do the following, which can be helped by good Test Case structure: Assign specific sets of Test Cases to specialist testers. Prioritise the creation of Test Cases by Positive testing.

Positive tests check that the normal functional path is working for normal inputs.

In contrast a negative test will provide an illegal entry and will thus test the resilience and error handling capability of the functionality. Often negative tests are more efficient at finding less obvious defects and specifically those defects that may cause a major system failure.

Prioritise Test Cases for regression test sets. Prioritise Test Cases for automation. Choose Test Cases to test a defect fix. Choose a small subset of Test Cases to decide if a build is fit for

accepting into testing – eliminating bad build injection into test. Choose a subset of Test Cases for reuse in Acceptance Testing.

Hence the reason for structure needs to be understood by the team.

14

Dealing with Test Phases

It is often simpler to create a library of Test Scripts, where these are maintained centrally and then copy down Test Scripts to appropriate level folders for the test phase:

Project ABC

Integration

Build1

FAT

UAT

SAT

OAT

COPY

Test Sub-Folders

Specialist test management tooling allows results and scripts to be managed without the need for copying. However some tools are better than others.

15

Limitations on Folder Structure

Be aware that some test tools and older versions of well established tooling, have limitations on path length.

The total number of characters for the Test Case name (path) and Test Script (file) are limited.

If this happens to be the case then appropriate abbreviations will need to be used that are well understood by the team.

16

Agile

If using an Agile approach, then you will need to consider how you add Test Cases and map these to Agile Stories and at the same time allow your administration of Regression Tests.

Ensure that Test Cases for Performance and Security are added early. The content will improve over the life span of the test delivery, however they need to be created early on in the test cycle.

17

Warning

Take great care to maintain a sustainable test script library.

There is always a danger that with a poorly defined system, tests are not maintained and new and even repeated tests are added. This is dangerous for a project because: Older tests will eventually break, causing further delays to test

runs. There is a greater maintenance cost as more tests get added. The time to run a suite of tests increases, adding significant

costs to testing. Make certain the test team understand the need for

maintenance and make sure that tests for specific functions can be quickly identified.

18

Test Script Overview

Test Scripts have 3 key elements.: Step Number – providing the sequence for Action (which includes Observation). Action – A unique step in a sequence of Action or Observation. The Expected Outcome from the result, against which a Pass or Fail is measured.

Note: When using traditional Test Management Tools, further columns are added for Pass/Fail and Comments. However if making do with tools such as Excel, there is a need to add columns for this and have a way of referencing any attachments – such as screen captures. So you will need to add a column for comments, defect ID cross reference and for any hyperlink to screen captures, which will need to be stored in a structured directory.

With an excel solution, there also needs to be an approach for making sure the issue of the test script is maintained and the results of [past test runs are mapped to a specific issue of the script.

Test script maintenance can also be achieved by each Excel workbook having a master blank script, which is copied within the file and used for completion during a specific run. Try to avoid the need to edit multiple scripts, so work maintenance is minimised.

Step Action Expected Result1    

     

19

Setting Out Test Actions #1

Test Script Actions need to be set out in a logical order of events.

Step 1 – will detail any test assumptions. Do consider that often a test run may at times vary. So if a test is for example about logging in with a user name of 3 characters, then state this early. This then avoids the situation of a busy test team using a standard assigned user name with more characters. Or avoiding the automated tester writing a replacement automated test script which misses the point of the test.

20

Setting Out Test Actions #2

In setting out steps, we are ensuring that we can carry out tests in a precise and ordered approach that is repeatable. The developer should be able to repeat the test step and obtain the same result and understand exactly what the failure was.

Test Actions also include the action of Observation. So for specific observations ensure that these are included as single steps. This then guides the automated tester to ensure that specific observations need to be captured.

21

Setting Out Test Actions #3

Finally when all the script test actions are complete, there is a final step which will either:Set the system to a known state ready for

another test ORDirect the user to run a cleanup script.

22

Level of Detail Required

There is a fine balance in ensuring that there is sufficient information that can be used to understand what action needs to be taken and the amount of information that needs to be maintained.

If testing is likely to utilise an army of untrained testers, then more detail will be required within a test script. This has to be considered early on. Depending on the company this could include: undergraduates, new graduates with no or little previous experience, users, administration staff, etc. So if this is likely then the scripting needs to be addressed early on in development of the scripts.

If code is likely to change significantly then you do not want to create a large number of test scripts with lots of fine detail that will take longer to maintain than it takes to change the code. So in this instance test scripts will be general in description and be less prone to the need of continuous maintenance. However the key purpose of the test case and the choice of vectors needs to be stated clearly.

23

Test Script Referencing #1

In Test Management Tooling, Test Scripts are often contained in a flat table with a file reference. These files are then hooked into Test Cases. This approach has a specific advantage that if a test case is copied and repeated several times, they will always reference the same batch of test scripts, that can be maintained centrally. Changing one test script will update all instances of the Test Script.

Do ensure that a test run results are not impacted by changes in the working Test Script. Any change in a Test Script should not impact any recorded running history, where the previous version was used. Not all Test Tools are helpful in this instance. If this causes a problem you will need to ensure that any configuration control deployed compensates for the flaw in the tooling.

The reason for this is that if you needed to re-run an old version of a test to check a result, or if development roll back code to a previous build, you need to ne able to reference the appropriate version of the test script. If you do not have control of this, then you will be heading for problems.

24

Test Script Referencing #2

In creating Test Scripts, the Script name will need to reflect a number of criteria: Identify any specific order that a series of scripts

has to be run in. Identify any priority (High, Medium, Low) that is

required for automation and regression testing. Identify, where necessary any version control

reference. Identify the functionality being tested. This can

include reference to positive or negative testing.

25

When to Automate

Automation creates a level of administration which can at times be unsustainable. So in automating ensure: Code has stabilised and is unlikely to need automated tests to be

updated. Do not rely on record and play, where possible use coding to alter

data inputs. This means that scripts are far more easily adaptable and maintainable.

Take care with bit map comparisons. Different graphics cards have in the past given rise to sufficient differences to cause failure, where a pass is correct. Modern GUI automation tools allow the threshold for bit map comparison to be adjusted to compensate. Also take care not to compare a whole screen as routine. Positions may vary slightly and so if looking for example a logo, it is possible in some tools to highlight the image that is to appear on the screen and will pass the test if the image appears anywhere on the screen, within specific set boundaries.

26

Cutting Duplication in Test Scripting This applies to both automated and manual testing. If there is a set-up

sequence say of 12 steps, then if you have to repeat that for 100 test scripts that is 1,200 steps that you need to maintain. This is very expensive.

Instead consider setting up a preliminary set up script (say called “preliminary_set_up_test24v1”. This would then have those 12 steps. Any test that needed this would then have a step that says “first run preliminary_set_up_test24” (note no version, however the date and issue control will allow anyone to identify the version run). Any automated scripting would then run the preliminary test before the extension. However any Test Script name would need to indicate that a preliminary test needs to be run first. This is then easier to check a regression suite for the inclusion of any preliminary test.

For Manual Testing this is easily implemented by the tester running the repeat step, with a print out. While in the single repeat step will have a failure that will correspond to one of many steps in the preliminary preparation script, the results should have the ability to detail in which step of the preliminary script the failure occurred. However in practice if one had may repeated failures in a preliminary script, one would raise a defect against the preliminary script and abandon testing on any script that relied on that preliminary test being carried out.