Fundamentals Test Process

Fix component test plan and repeat

Fix test design and repeat

Fix test design and repeat

Fix component or test cases/scripts

and repeat

5 Phases of the Fundamental Test Process

Fundamental Test Process

Test Planning

and Control

Test Analysis

and Design

Test Implementatio

n and Execution

Evaluating Exit Criteria and Reporting

Test Closure Activities

Test Planning

• Specifies how the test strategy and project Test Plan

A document describing the scope, approach, resources and schedule of intended test activities

apply to the software under test

• Principally:

– verify the mission,

– define the Test objectives

– Specify the Test Activities required to meet the mission and objectives

Test Planning and Control

Test Planning (continued)

• Major Tasks are :-

– Identify the objectives of testing

– Determine Scope

– Determine the Test Approach

– Determine the required test resources

– Implement the test policy and/or the test strategy

– Schedule test analysis and design tasks

– Schedule test implementation, execution and evaluation

– Determine the Start and Exit Criteria

Jobs for a test manager!!

Test Planning and Control

Test Planning and Control

Test Control

• The ongoing activity of comparing actual progress against the plan

• Reporting status, including deviations from the plan

• Taking actions necessary to meet the mission and objectives of the project

• Test Planning takes into account the feedback from monitoring and control activities.

• Major Tasks are :-

– measure and analyse results

– Monitor and document progress, test coverage and exit criteria

– initiate corrective actions

– make decisions

Unit Testing Entry Criteria Example

• Business Requirements are at least 80% complete and have been approved to-date

• Technical Design has been finalized and approved

• Development environment has been established and is stable

• Code development for the module is complete

Unit Testing Exit Criteria Example

• Code has version control in place

• No known major or critical defects prevents any modules from moving to System Testing

• A testing transition meeting has be held and the developers signed off

• Project Manager approval has been received

Test Analysis and Design

• Reviewing the test basis (such as requirements, architecture, design, interfaces).

• Evaluating testability of the test basis and test objects.

• Identifying and prioritizing test conditions based on analysis of test items, the specification, behaviour and structure.

• Designing and prioritizing test cases.

• Identifying necessary test data to support the test conditions and test cases.

• Designing the test environment set-up and identifying any required infrastructure and tools.

• Each Test Case is specified in terms of :-

– its objective

– the initial state of the system

– the input

– the expected result.

Test Analysis and Design

Fundamental Test Process

• “The act of designing tests is one of the most effective error prevention mechanisms known

... The thought process that must take place to create useful tests can discover and eliminate problems at every stage of development.“

Beizer 1983

Testing Pearl of Wisdom

Test Implementation and Execution

• Test Cases are combined in a particular order to form Test Procedures Specifications (aka Test Scripts) A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script

• Test Data is created

• The test environment is created

• Major tasks are:

Create Test Scripts and Data

– Develop, implement and and prioritise Test Cases

– Create the Test Scripts

– Create test data

– Preparing test harnesses

– Write automated test scripts

– Create test suites from the test cases for efficient test execution

Check the Environment

– Verify that the test environment has been set up correctly

Test Script Example

def sample_test_script (self):

type ("TextA")

click (ImageButtonA)

assertExist (ImageResultA)

• Major tasks (continued):

Execute the Tests

– Execute the Test Cases (manually or through tools) according to the planned sequence.

– Log the outcome of test execution

– Test execution records should uniquely identify the versions of the software under test, test tools and Testware

– It should be possible to establish that all testing has been carried out by reference to the test records.

Test Implementation and Execution

• Major tasks (continued):

Check the Results

– Compare actual results with expected results

– Report discrepancies as Incidents

– Analyse Incidents to establish Root cause

– Repeat, as necessary, test activities as result of action taken for each discrepancy

– The test coverage levels achieved for those measures specified as test completion criteria should be recorded.

Test Implementation and Execution

The Execution Phases

• Pre-test / Smoke test:

– Run a few test cases in order to see if the product is ready for testing.

• Main test:

– Run all planned test cases.

• After test:

– Run extra test cases.

– Repeat the test after repair (regression test).

Fundamental Test Process

• “Thoroughly inspect the results of each test”

Myers - 2004

Testing Pearl of Wisdom

Ref: Myers, The Art of Software Testing, J Wiley and Sons, 1979

Evaluating Exit Criteria and Reporting

• Test execution is assessed against the objectives defined in Test Planning

• This should be done for each Test Level (i.e. test stage)

A group of test activities that are organized and managed together.

• Major tasks are:

– Check test logs against the exit criteria specified in Test Planning

– If the exit criteria has not been met

• Assess if more tests are needed

• Assess which test activities may need to be repeated

– Assess if the exit criteria specified should be changed

– Produce a test summary report for stakeholders review

Test Closure Activities

• Collect data from completed test activities to consolidate experience, Testware, facts and numbers

• Major tasks are:

– Check which planned deliverables have been delivered

– Check that Incident reports status are up-to-date (e.g. Closed)

– Ensure all Incident reports have associated change records

– Record acceptance of the system

– Finalise and archive Testware, the test environment and the test infrastructure for later reuse

– Handover Testware to the maintenance organization

– Analyse lessons learned for future releases and projects, and for process improvement.

Psychology of Testing

• Historically testing was viewed as showing the system meets its requirements

• This has evolved to a stage where testing is performed with the primary aim of finding faults rather than proving correctness. It is perceived as a destructive process

• Seeking to find failures (the right mindset) can be viewed as criticism of the product and/or its author

• But looking for failures is constructive!

– Time can be saved

– Risks reduced

– Costs reduced

– Skills improved

Psychology of Testing

• It is important that the Objectives of testing are clearly understood as humans will moderate their behaviour accordingly (however sub-consciously):-

– “If testing is showing the system meets its requirements then I will just produce tests that show this.”

– “If testing is aimed at finding faults then I will be measured on this so I will put effort into designing tests that are more likely to find faults.”

• The Testing mindset is different from a Developer’s

The Testing Mindset

Goal Conflicts About Testing

• You (as developer) are blind for your own mistakes.

• You are constructive, testing is destructive.

• You want to get finished. Testing delays you.– Someone else should test!

• Learning takes time someone else.– Test yourself!

• A good test cannot save a bad program.

Psychology of Testing

• “Testing is an extremely creative and intellectually challenging

task”

• “Tests must be written for invalid and unexpected, as well as valid and expected, input conditions”

Myers - 1979

Testing Pearl of Wisdom

Psychology of Testing

• A Tester needs:

– good communication skills

– good observation skills

– people handling skills

– Curiosity

– patience

– reliability

– thoroughness

– an inquisitive nature

– attention to detail

– creativity in terms of identifying likely faults

– Experience

• However as with most other disciplines an effective test team will need a mix of skills so it is difficult to generalise

Traits of Good Testers

Psychology of Testing

• The relationship between a Developer and a Tester is not normally an easy one because:-

– testers point out problems with software

– developers like to think their software is perfect

– testers are perceived as delaying the project by finding faults in the system

– when the development slips testers normally have to work long hours to test the product, which in turn can cause resentment.

• It is important that they work together

• It is also important that they have mutual respect for each other.

• Collaboration is the right approach – we work to a common goal!

• Communicate findings objectively, not subjectively

Developer vs Tester Relationship

Developer vs Tester Relationship

Developer vs Tester Relationship

Psychology of Testing

• The right mindset could enable Developers to test the code

• However, passing this responsibility to trained and professional testing resources has many benefits (such as higher defect find rates)

• Authors tend to bring across assumptions they have made when developing the software. They are less likely to write tests to show faults in their own software (human nature)

• With testing performed by independent testers, testing effort is focused and not compromised by development effort and bias

• It is generally believed that objective independent testing is more effective

Independent testing

Psychology of Testing

• There are several levels of Independence (from Low to High)

– Tests designed by the person(s) who wrote the software under test

– Tests designed by another person(s) (e.g. from the development team).

– Tests designed by a person(s) from a different organizational group (e.g. an independent test team).

– Tests designed by a person(s) from a different organization or company (e.g. outsourcing to an in-house or external test specialist organisation)

Independent testing