Cs498dm Software Testing Darko Marinov January 17, 2012.
-
Upload
derrick-summers -
Category
Documents
-
view
213 -
download
0
Transcript of Cs498dm Software Testing Darko Marinov January 17, 2012.
![Page 1: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/1.jpg)
cs498dmSoftware Testing
Darko Marinov
January 17, 2012
![Page 2: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/2.jpg)
Teaching Staff
• Instructor: Darko Marinov– Email: marinov AT illinois.edu– Office: SC 3116, 217-265-6117– Office hours: after classes or by appointment
• No TA
![Page 3: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/3.jpg)
Course Overview
• Introduction to software testing– Systematic, organized approaches to testing– Based on models and coverage criteria– Testing is not (only) about finding “bugs”– Improve your testing (and development) skills
• Five problem sets and a project– Centered around testing Java PathFinder (JPF)
![Page 4: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/4.jpg)
JPF in One Slide
• Will have an entire lecture on JPF– You’ll get to learn a lot about Java/JVM/JPF in
this class
• JPF is a tool for systematic testing of Java programs
• JPF is a Java Virtual Machine (JVM) that has support for fast backtracking
• JPF is implemented in Java itself
![Page 5: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/5.jpg)
Administrative Info
• Lectures: TR 12:30pm-1:45pm, 1103 SC
• Credit:– 3 undergraduate hours– 3 or 4 graduate hours (a larger project for 4)
• Prerequisites: recommended software engineering (cs427) and programming languages (cs225, cs421)– Consent of instructor (must if not senior)
![Page 6: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/6.jpg)
Grading
• Points– Project (25%)– Problem sets (5*15%)
• Grades– A*(90%), B*(80%), C*(70%), D*(60%), F(<60%)– For more details, see the syllabus– The instructor may lower the point limits
![Page 7: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/7.jpg)
Project
• Testing a part of JPF
• Deliverables– Proposal (due in three weeks)– Progress report (around midterm)– Final report (by the grade submission deadline)– Bug reports (hopefully you’ll find some bugs)
• Extra bonus points for reporting bugs to me
![Page 8: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/8.jpg)
Collaboration
• You must individually write solutions for the problem sets
• You can collaborate on everything else (unless explicitly stated not to collaborate!)– Discuss problem sets– Do projects in groups, preferably two or three
students
• Testing is a social activity– Communication matters
![Page 9: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/9.jpg)
Course Communication
• Wikihttps://wiki.engr.illinois.edu/display/cs498dmsp12
• Mailing list cs498dm AT cs.illinois.edu
• Instructor: Darko– Email: marinov AT illinois.edu– Office: SC 3116, 217-265-6117– Office hours: after classes or by appointment
![Page 10: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/10.jpg)
Signup Sheet
• Name
• Email address ([email protected])
• Program/Year
• Interests: what would you like to learn about testing?
• Experience: what testing did you do?
![Page 11: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/11.jpg)
Textbook
• “Introduction to Software Testing”by Paul Ammann and Jeff OffuttCambridge University Press, Jan. 2008
• Strongly recommended but not required– Books should be in the bookstore already
![Page 12: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/12.jpg)
This Lecture: Introduction to “Bugs”
• Why look for bugs?
• What are bugs?
• Where they come from?
• How to detect them?
![Page 13: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/13.jpg)
Some Costly “Bugs”
• NASA Mars space missions– Priority inversion (2004)– Different metric systems (1999)
• BMW airbag problems (1999)– Recall of 15,000+ cars
• Ariane 5 crash (1996)– Uncaught exception of numerical overflow– http://www.youtube.com/watch?v=kYUrqdUyEpI
• Your own favorite examples?
![Page 14: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/14.jpg)
Some “Bugging” Bugs
• Smaller issues that give unexpected results
• Your own favorite examples?
![Page 15: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/15.jpg)
Economic Impact
• “The Economic Impact of Inadequate Infrastructure for Software Testing”NIST Report, May 2002
• $59.5B annual cost of inadequate software testing infrastructure
• $22.2B annual potential cost reduction from feasible infrastructure improvements
![Page 16: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/16.jpg)
Estimates
• Extrapolated from two studies (5% of total)– Manufacturing: transportation equipment– Services: financial institutions
• Number of simplifying assumptions
• “…should be considered approximations”
• What is important to you?– Correctness, performance, functionality
![Page 17: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/17.jpg)
Some Motivation for Testers
• An article from SD Times, a magazine for software development managers:“Improving Software Quality”by Lindsey Vereen (page 34/68)
• A slide from Debra Richardson, a professor at UC Irvine:“Analysis and Testing are Creative”(page 26/48)
![Page 18: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/18.jpg)
Terminology• Anomaly• Bug• Crash• Defect• Error• Failure, fault • Glitch• Hang• Incorrectness• J...
![Page 19: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/19.jpg)
Dynamic vs. Static
• Incorrect (observed) behavior– Failure, fault
• Incorrect (unobserved) state– Error, latent error
• Incorrect lines of code– Fault, error
![Page 20: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/20.jpg)
“Bugs” in IEEE 610.12-1990
• Fault– Incorrect lines of code
• Error– Faults cause incorrect (unobserved) state
• Failure– Errors cause incorrect (observed) behavior
• Not used consistently in literature!
![Page 21: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/21.jpg)
Correctness and Quality
Common (partial) propertiesSegfaults, uncaught exceptionsResource leaksData races, deadlocksStatistics based
Specific propertiesRequirementsSpecification
![Page 22: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/22.jpg)
Traditional Waterfall ModelRequirements
Analysis
DesignChecking
ImplementationUnit Testing
IntegrationSystem Testing
MaintenanceRegression Testing
We will look at general techniques, applicable in several phases of testing
![Page 23: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/23.jpg)
Phases (1)
RequirementsSpecify what the software should do
Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness
DesignSpecify how the software should work
Split software into modules, write specifications
Checking: check conformance to requirements, using for example conformance testing
![Page 24: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/24.jpg)
Phases (2)
ImplementationSpecify how the modules work–Unit testing: test each module in isolation
IntegrationSpecify how the modules interact–Integration testing: test module interactions–System testing: test the entire system
MaintenanceEvolve software as requirements change–Regression testing: test changes
![Page 25: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/25.jpg)
Testing Effort
Reported to be >50% of development cost [e.g., Beizer 1990]
Microsoft: 75% time spent testing50% testers who spend all time testing
50% developers who spend half time testing
![Page 26: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/26.jpg)
When to Test
The later a bug is found, the higher the costOrders of magnitude increase in later phases
Also the smaller chance of a proper fix
Old saying: test often, test early
New methodology: test-driven development(write tests even before writing code)
![Page 27: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/27.jpg)
Software is Complex
Malleable
Intangible
Abstract
Solves complex problems
Interacts with other software and hardware
Not continuous
![Page 28: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/28.jpg)
Software Still Buggy
Folklore: 1-10 (residual) faults per 1000 nbnc lines of code (after testing)
Consensus: total correctness impossibleto achieve for complex softwareRisk-driven finding/elimination of faults
Focus on specific correctness properties
![Page 29: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/29.jpg)
Approaches to Detecting Bugs
Software testing
Model checking
(Static) program analysis
…
![Page 30: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/30.jpg)
Software Testing
Dynamic approach
Run code for some inputs, check outputs
Checks correctness for some executions
Main questionsTest-suite adequacy (coverage criteria)
Test-input generation
Test oracles
![Page 31: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/31.jpg)
Other Testing Questions
Selection
Minimization
Prioritization
Augmentation
Evaluation
Fault Characterization
…
Testing is not (only) about finding faults!
![Page 32: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/32.jpg)
Current Status
Testing remains the most widely used approach to finding bugsValidation: are we building the right system?
Verification: are we building the system right?
Testing is gaining importance with test-first development and increased reliability needs
A lot of research on testing (part of mine too)This course is not about research
![Page 33: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/33.jpg)
“Schools” of Software Testing
Bret Pettichord described four schoolsAnalytic (a branch of CS/Mathematics)
Factory (a managed process)
Quality (a branch of quality assurance)
Context-Driven (a branch of development)
This course focuses on artifacts, not process
Do you want a guest speaker from industry?
![Page 34: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/34.jpg)
Topics Related to “Finding Bugs”
How to “eliminate bugs” (localize faults)?Debugging
How to “prevent bugs”?Programming language design
Software development processes
How to “show absence of bugs”?Theorem proving
Model checking, program analysis
![Page 35: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/35.jpg)
Testing Topics to Cover
Test coverage and adequacy criteriaGraph, logic, input domains, syntax-based
Test-input generation
Test oracles
Model-based testing
Testing software with structural inputs
Test automation
Testing in your domain of interest?
![Page 36: Cs498dm Software Testing Darko Marinov January 17, 2012.](https://reader036.fdocuments.us/reader036/viewer/2022062720/56649f125503460f94c25d9e/html5/thumbnails/36.jpg)
Summary of the Introduction
Eliminate bugs to save lives and money
“Bugs” may mean faults, errors, failures
Several approaches for detection: software testing, model checking, static analysis…
Software testing is the most widely used approach for validation and verificationWe will cover systematic approaches to testing, based on coverage criteria for various models
Testing is not (only) about revealing faults