CS527: Advanced Topics in Software Engineering

(Software Testing and Analysis)

Darko Marinov

August 23, 2007

Course Overview

• Graduate seminar on program analysis (for bug finding), emphasis: systematic testing

• Focus on a (research) project– Papers: reading in advance, writing reports,

presenting, and discussing– Project: initial proposal, progress report, final

presentation, paper

• One or two problem sets

Administrative Info

• Meetings: TR 2-3:15pm, 1302 SC

• Credit: 4 graduate hours• Auditors welcome for discussions

– Can come to any lecture, mostly self-contained

• Prerequisites: some software engineering, programming languages, and logic

Evaluation

• Grading– Project [50%]– Presentation [20%]– Participation (reports and discussion) [20%]– Problem set(s) [10%]

• Distribution– Grades will be A- centered– No guarantee for A (or even a passing grade)!

• Project is the most important

Project

• Proposal (due in about a month)• Progress report (around midterm time)• Presentation (around last week of classes)• Paper (by the grade submission deadline)

• Six students who took CS598DM published five papers based on their projects– I’m happy to help, no co-authorship required

Course Communication

• Web sitehttp://agora.cs.uiuc.edu/display/cs527

• Mailing list<course-code>@mail.cs.uiuc.edu

Personnel

• TA: Brett Daniel– Office: 3217 SC, hours: by appointment– Email: b<last-name>3@cs.uiuc.edu

• Instructor: Darko Marinov– Office: SC 3116, hours: by appointment– Phone number: 217-265-6117– Email: <last-name>@cs.uiuc.edu

Signup Sheet

• Name• Netid (please write it legibly)• Program/Year• Group• Looking for advisor/project/thesis topic?• Interests

• Also sign up on Wiki

This Lecture: Overview

• Why look for bugs?

• What are bugs?

• Where they come from?

• How to detect them?

Some Example “Bugs”

• NASA Mars space missions– priority inversion (2004)– different metric systems (1999)

• BMW airbag problems (1999)– recall of >15000 cars

• Ariane 5 crash (1996)– uncaught exception of numerical overflow

• Your own favorite example?

Economic Impact

• “The Economic Impact of Inadequate Infrastructure for Software Testing”NIST Report, May 2002

• $59.5B annual cost of inadequate software testing infrastructure

• $22.2B annual potential cost reduction from feasible infrastructure improvements

Estimates

• Extrapolated from two studies (5% of total)– Manufacturing: transportation equipment– Services: financial institutions

• Number of simplifying assumptions

• “…should be considered approximations”

• What is important for you?– Correctness, performance, functionality

Terminology

• Anomaly• Bug• Crash• Defect• Error• Failure, fault • G…• …

Dynamic vs. Static

• Incorrect (observed) behavior– Failure, fault

• Incorrect (unobserved) state– Error, latent error

• Incorrect lines of code– Fault, error

“Bugs” in IEEE 610.12-1990

• Fault– Incorrect lines of code

• Error– Faults cause incorrect (unobserved) state

• Failure– Errors cause incorrect (observed) behavior

• Not used consistently in literature!

Correctness

• Common (partial) properties– Segfaults, uncaught exceptions– Resource leaks– Data races, deadlocks– Statistics based

• Specific properties– Requirements– Specification

Traditional Waterfall ModelRequirements

Analysis

DesignChecking

ImplementationUnit Testing

IntegrationSystem Testing

MaintenanceVerification

Phases (1)

• Requirements– Specify what the software should do– Analysis: eliminate/reduce ambiguities,

inconsistencies, and incompleteness

• Design– Specify how the software should work– Split software into modules, write specifications– Checking: check conformance to requirements

Phases (2)

• Implementation– Specify how the modules work– Unit testing: test each module in isolation

• Integration– Specify how the modules interact– System testing: test module interactions

• Maintenance– Evolve software as requirements change– Verification: test changes, regression testing

Testing Effort

• Reported to be >50% of development cost [e.g., Beizer 1990]

• Microsoft: 75% time spent testing– 50% testers who spend all time testing– 50% developers who spend half time testing

When to Test

• The later a bug is found, the higher the cost– Orders of magnitude increase in later phases– Also the smaller chance of a proper fix

• Old saying: test often, test early

• New methodology: test-driven development(write tests before code)

Software is Complex

• Malleable

• Intangible

• Abstract

• Solves complex problems

• Interacts with other software and hardware

• Not continuous

Software Still Buggy

• Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)

• Consensus: total correctness impossibleto achieve for complex software– Risk-driven finding/elimination of bugs– Focus on specific correctness properties

Approaches for Detecting Bugs

• Software testing

• Model checking

• (Static) program analysis

Software Testing

• Dynamic approach

• Run code for some inputs, check outputs

• Checks correctness for some executions

• Main questions– Test-input generation– Test-suite adequacy– Test oracles

Other Testing Questions

• Selection

• Minimization

• Prioritization

• Augmentation

• Evaluation

• Fault Characterization

• …

Model Checking

• Typically hybrid dynamic/static approach

• Checks correctness for all executions

• Some techniques– Explicit-state model checking– Symbolic model checking– Abstraction-based model checking

Static Analysis

• Static approach

• Checks correctness for all executions

• Some techniques– Abstract interpretation– Dataflow analysis– Verification-condition generation

Comparison

• Level of automation– Push-button vs. manual

• Type of bugs found– Hard vs. easy to reproduce– High vs. low probability– Common vs. specific properties

• Type of bugs (not) found

Soundness and Completeness

• Do we detect all bugs?– Impossible for dynamic analysis

• Are reported bugs real bugs?– Easy for dynamic analysis

• Most practical techniques and tools are both unsound and incomplete!– False positives– False negatives

Analysis for Performance

• Static compiler analysis, profiling• Must be sound

– Correctness of transformation: equivalence

• Improves execution time• Programmer time is more important

• Programmer productivity– Not only finding bugs

Combining Dynamic and Static

• Dynamic and static analyses equal in limit– Dynamic: try exhaustively all possible inputs– Static: model precisely every possible state

• Synergistic opportunities– Static analysis can optimize dynamic analysis– Dynamic analysis can focus static analysis– More discussions than results

Current Status

• Testing remains the most widely used approach for finding bugs

• A lot of recent progress (within last decade) on model checking and static analysis– Model checking: from hardware to software– Static analysis: from sound to practical

• Vibrant research in the area

• Gap between research and practice

“Schools” of Software Testing

• Brett Pettichord described four schools– Analytic (a branch of CS/Mathematics)– Factory (a managed process)– Quality (a branch of quality assurance)– Context-Driven (a branch of development)

• Brian Marick may give a guest lecture

• We will focus on artifacts, not process

Testing Topics to Cover

• Test coverage and adequacy criteria

• Test-input generation

• Test oracles

• Model-based testing

• Testing GUIs

• Testing software with structural inputs

• Testing in your domain?

Topics Related to Finding Bugs

• How to eliminate bugs?– Debugging

• How to prevent bugs?– Programming language design– Software development processes

• How to show absence of bugs?– Theorem proving– Model checking, program analysis

Next Lecture

• Tuesday, August 28, at 2pm, 1302 SC• Assignment (on Wiki)

– Read a paper on writing papers“Writing Good Software Engineering Research Papers” by Mary Shaw (ICSE 2003)

• If you already read that one, read another

– Read a paper on testing: “Parallel test generation and execution with Korat” by S. Misailovic, A. Milicevic, N. Petrovic, S. Khurshid, and D. Marinov (FSE 2007)