1/38

2013 REU Program at ECUSoftware Testing - Foundations, Tools,

and Applications

Lecture 1 May 21, 2013

Introduction to Software Testing

Dr. Sergiy Vilkomir

2/38

Dr. Sergiy Vilkomir

Instructor’s office: S & T Building, C-111

Email: vilkomirs@ecu.edu

http://www.cs.ecu.edu/reu/index.html

http://core.ecu.edu/vilkomirshttp://core.ecu.edu/STRG/

3/38

Software life cycle

• concept • requirements • design • implementation (coding)• testing • installation• operation and maintenance

testing

Is testing important ?

4/38

Software Errors

Software errors can be found and eliminated during software testing

“Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST)”.

(NIST News, June 38, 2002 )

5/38

Software Errors

Buggy Software May Have Crashed Mars Polar Lander (March 2000)

• The software problem crashed the Mars Polar Lander into the Red Planet’s frozen ground

• The most likely cause of the lander’s failure, investigators decided, was that a spurious sensor signal associated with the craft’s legs falsely indicated that the craft had touched down when in fact it was some 130-feet (40 meters) above the surface.

• This caused the descent engines to shut down prematurely and the lander to free fall out of the martian sky.

• The error was traced to a single bad line of software code.

Causes of software defects

Software Bugs with Fatal Consequences

• 1985 - 1986 few patients were killed in a hospital in USA because the medicine dose was calculated wrongly (software bugs in Therac-25 treatment machine lead to radiation overdoses, approximately 100 times the intended dose).

6/38

7/38

Causes of software defects

• In1996 the Ariane 5 rocket of the European space agency was destroyed because they used the software from Ariane 4 rocket (was self-destroyed 40 seconds after takeoff; more than $370 million was lost ).

8/38

Causes of software defects

Software Bugs with Fatal Consequences

• In 1962 NASA lost their 80 Million Dollar "Mariner 1" because of a missing "hyphen" in the program code.

9/38

Causes of software defects

Software Bugs with Fatal Consequences

During the Gulf War in February, 1991, a Patriot missile system operating in Dhahran, Saudi Arabia, failed to track and intercept an incoming Scud. The Iraqi missile impacted into an army barracks, killing 38 U.S. soldiers and injuring another 98.

10/38

The cause of the missile system failing to defend against the incoming Scud was traced back to a bug in Patriot’s radar and tracking software.The bug occurs in the calculation of the next location of the incoming target. That modified software reached Dhahran the day after the attack.

Computers and Software Errors

11/38

• On Tuesday, June 3, 1980, at 1:26 a.m., the display system at the command post of the Strategic Air Command (SAC) near Omaha, Nebraska, indicated that two submarine-launched ballistic missiles (SLBMs) were headed toward the United States.

• The SAC duty controller directed all alert crews to move to their B-52 bombers and to start their engines

• Land-based missile crews were put on a higher state of alert, and battle-control aircraft prepared for flight

• Fortunately, there were a number of factors which made those involved in the assessment doubt that an actual attack was underway

• The cause of these incidents was eventually traced to the failure of a single integrated circuit chip in a computer which was part of a communication system

Causes of software defects

Software Bugs with Fatal Consequences

• Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983.

• The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so.

• Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten.

12/38

Software Errors

13/38

Investigation found that the satellite in question had picked up the sun's reflection off the cloud tops and somehow interpreted that as a missile launch.

CNN Money - August 9, 2012

In less than an hour, Knight Capital's computers executed a series of automatic orders that were supposed to be spread out over a period of days. Millions of shares changed hands. The resulting loss, which was nearly four times the company's 2011 profit, crippled the firm

and brought it to the edge of bankruptcy.

14/38

Computer glitch that set fire to $440 million of Knight Capital Group's funds

Top Software Failures Of 2011(from http://www.businesscomputingworld.co.uk)

1. Financial services giant fined $25 million for hiding software glitch that cost investors $217 million

2. Computer system bugs cause Asian banking facilities’ downtime

3. Cash machine bug benefits customers by giving them extra money

4. 22 people wrongly arrested in Australia due to failures in new NZ $54.5 million courts computer system

5. 38,380 cars recalled after airbag-related software glitch

15/38

Top Software Failures Of 2011(from http://www.businesscomputingworld.co.uk)

Conclusions

• Software failures are costing companies and consumers large amounts of money

• The worst software failures have damaged reputations, impacted negatively on financials and caused stress to users.

• Each of the top 2011 software failure examples could easily have been avoided through an effective quality management and proper software testing 16/38

17/38

Software testing

• Software errors can be found during software testing

• Testing can help but it is not for free

• The cost of software testing could be around 40% of the overall costs of software development

18/38

Terminology

Software testingError

FaultFailure

Input domain

Oracle

Regression testingUnit testing

Functional testingVerification Validation

V&V

19/38

Terminology

• Unfortunately, SE terminology in textbooks is often contradictory

• Two different textbooks could provide two different definitions for the same term

• What to do in this situation?

Solution: use terminology from standards

20/38

IEEE (Institute of Electrical and Electronics Engineers) standard

IEEE Std 610.12 1990

IEEE Standard Glossary

of Software Engineering

Terminology

21/38

New standard

• IEEE Std 610.12 1990 has not been updated for many years• Some terms are obsolete

• Instead: ISO/IEC/IEEE 24765:2010 “Systems and software engineering – Vocabulary”

22/38

S. Vilkomir, 2010

From IEEE Std 610.12 1990

testing - the process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component

Contrast with: Static Analysis, Inspection

23/38

S. Vilkomir, 2010

From IEEE Std 610.12 1990

• Static Analysis - the process of evaluating a system or component based on its form, structure, content, or documentation.

• Inspection - a static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems.

24/38

S. Vilkomir, 2010

From IEEE Std 610.12 1990

• Fault - an incorrect step, process, or data definition in a computer program. Note: In common usage, the terms "error" and "bug" are used to express this meaning.

• Failure - the inability of a system or component to perform its required functions within specified performance requirements. The result of the fault.

25/38

S. Vilkomir, 2010

From IEEE Std 610.12 1990

• test case - a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

26/38

S. Vilkomir, 2010

• Input domain – the set of all possible inputs to a program

• test oracle - a source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual, or an individual’s specialized knowledge, but should not be the code

27/38

Y= a X + b – it is very easy to determine expected results

expected results ?

28

Goal of software testing

Test case that did not find an error Test case that discovered a new error

- successful

- successful

Increasing the quality and reliability of the program after removing the error

Increasing our confidence on the quality and reliability of the program

29

Goal of software testing

Dual Goal

• To find undiscovered errors AND

• to get confidence on the quality and reliability of the program

30

Lifecycle phase in which testing takes place

© Aditya P. Mathur 2005

31

From IEEE Std 610.12 1990• black-box testing -Testing that ignores the

internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.

Syn:. functional testingContrast with: structural testing.• white-box testing -Testing that takes into

account the internal mechanism of a system or component. Types include branch testing, path testing, statement testing.

Syn: structural testing, glass-box testing Contrast with: functional testing

Is testing always done in the same way?

Types of software Consequences of defects

Program A

Program B

Program C

Program D Safety-critical software

Personal software

Company’s internal software

Commercial software

internal problems

nothing serious

financial losses, damaged reputation

loss of human life

32/38

• Testing is done differently in different contexts.– Type of software– Recourses (constraints): human recourses,

financial recourses, time– Regulatory requirements

• Conclusion: testing is context dependent

33/38

Causes of software defects

Human mistake (error)

Defect (bug, fault)in software

Failureduring software execution

34/38

How much testing is enough?Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible

Example:• The exam has 30 questions and four possible answers for each question.• What is a number of all possible combinations of answers?• If we can test 1000 combinations per second, how long will we test all combinations? (An hour? A day? A month?)

Total number of combinations: 444 …. 4= 430 1.000.000.000.000.000.000

30 times

1 year 38.000.000 sec. We need 38.000.000 years for testing 35/38

How much testing is enough?

Not possible to say in generalDepends on each specific situation

• Pressures on a project include time and budget• Risk assessment to decide how much testing to do.

Important:• prioritization (do the most important tests first)• setting criteria when to stop testing• Take into account technical and business risk• Take into account project constraints.

36/38

Testing principles

Testing shows presence of defects

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness

37/38

“Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of ?

Testing principles

Testing shows presence of defects

38/38

“Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of Dijkstra

Edsger Dijkstra (1930 – 2002) was a Dutch computer scientist. Winner of the 1972 Turing Award; creator of "structured programming."