Week6 testing-intro
-
Upload
alexandru-gala-popescu -
Category
Technology
-
view
110 -
download
0
Transcript of Week6 testing-intro
C om b ining the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche s te r
COMP23420 Sem 2 week 6Software testing concepts
John Sargeant
REMINDER: PLEASE ENSURE THAT YOUR PHONE IS SWITCHED OFF
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Overview
• Testing strategy
• Practical issues
• Safety critical systems
• Basic testing techniques
• Kinds of testing
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
This software has bugs in…Embury’s law: “This software has bugs in, we just don’t
know what they are yet”.
Applies to any SW system of any size, e.g.
• Eurofighter fuel control system 80K lines
• Modern airliner ~10M lines
• ABC ~150K (our code) + ~300K (Java library code)
Also, unlike e.g. rostering, most real systems are concurrent and reactive not simple functions from inputs to outputs.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Testing to failThe fundamental rule of testing: a successful test is one
which causes the software to fail.
Difficult for programmers to do with their own code – not the same as debugging.
• Traditional solution: have a separate testing team who are as nasty to the software as possible (see Software testing, Ron Patton, Sams 2006 for a guide to this approach).
• Agile solution – write the tests before the code – also helps to clarify requirements.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
How many bugs?Traditional estimate is 3-5 per hundred lines of C code.
That’s 30,000 – 50,000 for a million line program!
• Of those ~90% will probably be found by routine debugging
• And ~90%of the rest by sensible testing
• And ~90% of the rest by really rigorous testing
But that still leaves 30-50, and getting to 99.9% is very expensive.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Factors affecting bug density• Design – good designs lead to fewer bugs
• Type of application: concurrent, reactive systems are much more difficult than sequential transformation systems.
• Programming language: Java < C < Perl
• Programmer competence and experience
• In general expect 1-10 bugs per hundred lines
• (Probably a lot less for pair programming, but at a factor of 2 cost).
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
When you can’t afford 30-50 bugsMajor issue for safety critical systems, e.g. fly-by-wire.
• Conventional aircraft: pilot input directly controls flight surfaces (via hydraulics in large aircraft).
• FBW aircraft: a computer interprets the pilot’s inputs, and relays these electronically to the flight surfaces
• First used in the F16, 1974. Allows military aircraft to be inherently unstable, hence more manoeuvrable (also helps stealth).
• First civilian application Airbus A320, early 1980s. Provides protection and reduces pilot workoad.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Triple redundancy (1)• Airbus claimed that the A320 FBW software was
designed to fail no more than once in 109 flight hours. How could they possible claim that?
• Once in 104.5 (33K) hours might be plausible – but nowhere near enough.
• But add a second computer, with different software written by a different team. Now (in theory) both will fail at the same time once in 109 hours.
• But if there is a discrepancy you don’t know which one’s wrong – so you need a third computer and take a majority vote: triple redundancy.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Triple redundancy (2)Other advantages:
• Provides graceful degradation: don’t have to revert to manual control immediately with one computer out.
• Gives continuous testing for free – each discrepancy reveals a bug! So eventually the system should be extremely reliable – recent safety record of the A320 is outstanding.
• Redundancy is essential to providing reasonable levels of safety in complex safety-critical systems.
• Note: the actual A320 system has a lot more redundancy than described above
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Quiz(1)
Suggest at least three reasons why the theoretical sum 104.5 x 104.5 = 109 may not reflect what happens in practice. Hints:
• Is there still a single point of failure in the system?
• Remember: not all bugs are in the actual software
• When is it true that P(A and B) = P(A) x P(B)?
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Exhaustive testing is impossiblepublic static double divide(double a, double b){
return a / b;
}
A Java double is 64 bits so there are 2128 possibilities – intractable.
Similarly a reactive system such as FBW has a huge number of possible (state, input, time) combinations.
So we have to find a large number of bugs within a huge search space – we have to focus effort on the most “interesting” parts of that space.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Testing as a search problem• Equivalence partitioning: split up the space into areas
where similar tests are likely to lead to similar results
• e.g. if 2/3 works then 3/5 probably works too (but not necessarily 3/3
• Boundary value analysis: concentrate on boundaries between different parts of the space.
• E.g. b == 0, b very close to 0, a and/or b close to MAX_DOUBLE etc.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Black box or white box?Black box: testing the software against its spec without
access to the code:
• Means tests will be written without preconceptions about how the code works.
• If the code is changed, the same tests are still valid.
White box: testing with access to the code:
• Allows more tests to be done
• Allows tester to apply pressure to those places which look most likely to break.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Quiz(2)
2. You are testing an algorithm which sorts strings into alphanumeric order for a dictionary program. Suggest some of the most important tests you’ll need to do.
3. You are asked to thoroughly test floating point division software which will be burnt into a processor chip. Would you go about this primarily though black box or white box testing, and why?
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Kinds of testing• Unit testing – test one unit (in OO one class) at a
time.
• Integration testing – test that the components of a system (or subsystem) work together correctly
• Regression/smoke testing – check that you haven’t broken it.
• System testing – test that the system works in the context in which it will be required to work.
• Alpha and beta testing – test with real users
• Acceptance testing – get the customer to come up with the dosh.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Unit testing• Testing one unit – class - at a time.
• Relatively simple, but the class you’re testing will usually rely on other classes.
• In general almost all software relies on other software (e.g. Java library classes).
• The search space is generally well defined so techniques like EP ad BVA are most useful here.
• Often possible to be systematic and reasonably confident that a single class is bug-free.
• In Java, often done with the JUnit testing framework.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Integration testing• Testing that the components of a system work
together.
• Harder to define than unit testing; shape of testing space is less obvious.
• Concentrate on important mission-critical features
• Check that the use cases can be performed without problems.
• Don’t get upset when your code causes a problem; don’t get annoyed when somebody else’s does.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Regression/smoke testing
• Regression testing: repeat the tests you did before, to make sure you haven’t broken anything.
• Especially important after significant changes but the more often the better.
• Smoke testing: repeat the most critical tests as often as possible – check that it’s not going up in smoke.
• Integration and Regression/smoke testing are often done in the form of a daily (or nightly) “build” of the system – requires that at least some tests are automated.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
System testing• Testing of the system in the context(s) in which it will
operate.
• This will generally be a lot more varied than the context in which it was developed.
• May involve different hardware, operating systems, performance issues etc.
• Need to check the documentation and procedures as well as the code.
• Many systems which (seem to) work perfectly in a development environment fail in a customer environment.
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Alpha and beta testing• System testing with real users
• Important because they don’t use the software the way you assume.
• Alpha testing: a small group of users, done with SW developers present.
• Beta testing: a wider group, remote from the development team, asked to submit bug reports.
• Better not to start Beta testing until the software will work for most of the users most of the time!
C om b in ing the s tre ngths of U MIS T and
The Victoria U nive rs ity of Manche ste r
Acceptance testingWhere a SW product has a small number of large
customers (e.g. Campus Solutions) the customer(s) may specify a set of tests which the SW must pass before they will accept it and pay the dosh. There are some serious issues with this:
• In general, users don’t really know in advance what they want (the “waterfall fallacy”).
• Who within the customer organisation defines the spec? e.g. Managers and end users will have different views.
• Fixating on passing the acceptance test could result in serious problems being missed.