CMPSC 311- Introduction to Systems Programming Module: Testing · 2016-12-09 · CMPSC 311 -...
Transcript of CMPSC 311- Introduction to Systems Programming Module: Testing · 2016-12-09 · CMPSC 311 -...
CMPSC 311 - Introduction to Systems Programming
CMPSC 311- Introduction to Systems Programming
Module: Testing
Professor Patrick McDanielFall 2016
CMPSC 311 - Introduction to Systems Programming Page
NASA Mariner 1• 1962 – the Mariner 1
spacecraft was set to do a flyby of Venus‣ Guidance system failed
shortly after takeoff
‣ Destroyed just under 300 seconds from takeoff
‣ Specification error
¯Rn ! Rn
CMPSC 311 - Introduction to Systems Programming Page
Why testing ….• When developing software you construct logic to
perform some set of often complex tasks‣ Software bugs happen just as a consequence of process
‣ Identifying the bugs before they cause harm is key to being a good programmer (and viability of the company)
‣ Need to assess the “state” of the code• Figure out what parts of the code need work
• Figure out if changes to code cause problems to arise
CMPSC 311 - Introduction to Systems Programming Page
Validation vs. Verification• Validation is where you run a
program to see if it runs correctly on some input (testing …)
• Verification is used to verify that a program adheres to some invariant (property or behavior) using a proof based system‣ Most non-trivial properties for real
program are undecidable
‣ See CMPSC461 … (very cool!)
CMPSC 311 - Introduction to Systems Programming Page
Kinds of bugs …• Software may fail for lots of reasons …‣ Specification errors (Mariner 1)
‣ Assumption errors
‣ Logic errors
‣ Code errors
‣ Run-time errorsif (cart = 1) {
printf(”Got a one!”);} else {
printf(”Got a non-one!”);}
¯Rn ! Rn
CMPSC 311 - Introduction to Systems Programming Page
Software engineering ….• Every software shop has a process for developing code,
which is defined by the software development model‣ E.g., Waterfall model
Requirements Analysis
Coding
Design
Testing (verification)
Maintenance
Release
Alpha vs. Beta testing:Alpha testing
CMPSC 311 - Introduction to Systems Programming Page
Testing 101• The ability to test depends on a clear and unambiguous
specification (requirements document)‣ Testing with respect to function
‣ If not clear, than untestable
• Note: Industry average:‣ 30-85 errors per 1000 lines of code (LOC)
‣ 0.5-3 errors per 1000 LOC getting through release
CMPSC 311 - Introduction to Systems Programming Page
Types of testing …• Offline ”code” evaluation strategies‣ Code review inspections
‣ Code analysis
• Online testing‣ Workload testing
‣ Black box vs. white box testing
‣ Regression (maintenance, test changes)
• Testing and the phases of the development lifecycle‣ Unit testing validates modules individually
‣ Alpha testing is “acceptance testing” in house
‣ Beta testing typically with external users
Unit Testing
Beta Testing
Alpha Testing
CMPSC 311 - Introduction to Systems Programming Page
Lint (splint on Linux)• Lint uses static analysis of code to check for anything
that may be a problem with code‣ Often creating warnings that are not really problems
‣ This is a great way to really cleanse your code, but may take a while and sometimes not worth the trouble to fix all warnings.
CMPSC 311 - Introduction to Systems Programming Page
Unit testing• Unit testing is a technique where you create code that
exercises the logic and parameters for some module (typically not the entire program)‣ This allows you to create code that validates an
implementation independently from the rest of the program.
‣ Generally, the structure of the code is a single function:• (1) Initialize the module
• (2) Create a loop (of some length)‣ (2a) Exercise each function and validate the return values
• (3) Clean up data
• Note: this approach is used for both functional and performance testing of the module
CMPSC 311 - Introduction to Systems Programming Page
Unit Testing Example (cache)initialize the cache for a given size
Set of cache items = (create N of these, mark all as not in cache)
loop some number of FIXED_ITERATIONS {
select random operation (read/insert from cache)
pick random cache item X
read:
if X marked as in cache
if read from cache says there good, otherwise FAIL
else
if read from cache says there FAIL, otherwise good
insert:
if X NOT marked as in cache
if insert cache success good, otherwise FAIL
walk cache looking for ejected entries, update cached marks
}
close the cache, freeing all of the elements
CMPSC 311 - Introduction to Systems Programming Page
Coverage• Coverage is a measure of how much of a program you
have tested‣ Code coverage – how many lines have do tests exercise
‣ Branch coverage – how many branches do tests exercise
‣ Path coverage – how many paths have do test exercise
Note: we are not counting lines that are non-executable (braces, else, …)
CMPSC 311 - Introduction to Systems Programming Page
Coverage• Code coverage is the amount of code that is executed
when a particular set of inputs is given.• What is the coverage of the following:‣ a=6, b=9 (5/7 = 71%)
‣ a=4, b=5 (3/7 = 42%)
‣ Can you get 100% witha single test case?
int func(int a, int b) {if (a&2) {
b = a<<1;if (b&0x8) {
printf("One");}
} else {b = a^7;
}return(a*b);
}
Note: we are not counting lines that are non-executable (braces, else, …)
CMPSC 311 - Introduction to Systems Programming Page
Test vectors• Test vectors are carefully selected sets of inputs that
get maximal coverage.
• This can be calculated directly from the code (often)‣ (a&2)&(b&8), !(a&2) == 100% coverage
• When constructing tests you want to generate as few test cases as possible that give you the most coverage
int func(int a, int b) {if (a&2) {
b = a<<1;if (b&0x8) {
printf("One");}
} else {b = a^7;
}return(a*b);
}
CMPSC 311 - Introduction to Systems Programming Page
Control flow graphs• A control flow graph is a graph of
the path through the code from entry to (any) exit‣ Path coverage means all paths traversed
‣ However, paths are exponential in the number of branches (worst case)
‣ Meaning intractable in many cases …
START
a&2
b = a << 1; b = a^7;
a&b&0x1000
b = a << 1;
END
int func(int a, int b) {if (a&2) {
b = a<<1;if (b&0x8) {
printf("One");}
} else {b = a^7;
}return(a*b);
}
CMPSC 311 - Introduction to Systems Programming Page
Code reviews• In a professional development organization, you will be
periodically subjected to code review‣ Meeting ”chalk talk”
review process
‣ Offline “expert” review and commentary
CMPSC 311 - Introduction to Systems Programming Page
Lets get real …• Testing is one of the most
important skills you need to become a professional programmer.‣ Identify bugs by reading your code‣ Learn to use unit testing
‣ Create test vectors to test key parts of your code
‣ Take your time—it will pay off in the end