CSE 7314 Software Testing and Reliability Robert Oshana Trip 3 class notes [email protected].
CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 [email protected].
-
Upload
elizabeth-chambers -
Category
Documents
-
view
217 -
download
0
Transcript of CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 [email protected].
![Page 1: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/1.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
Lecture 16
![Page 2: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/2.jpg)
Test execution
Chapter 7
![Page 3: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/3.jpg)
Test execution
• Most visible part of the process
• Occurs at the end of the development cycle– Most other activities have slowed down– Probably on critical path
• Test execution is 40% of the entire effort
![Page 4: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/4.jpg)
Executing the tests
![Page 5: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/5.jpg)
Who should run tests
• Unit tests – probably the developers
• System tests – combination of developer, end user, test team
• Acceptance test – end user
• Look for the right people !
![Page 6: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/6.jpg)
What to execute first
• A strategy decision– Quality of resources– Documentation– Risk analysis
• Run regression test to find problems early
• Then focus on high risk areas
![Page 7: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/7.jpg)
Writing test cases during execution
• Will always think of new test cases
• You are learning more about the system
• Make sure to log these new tests and record them
![Page 8: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/8.jpg)
Record results of testing
• Automation should require that inputs and outputs be logged
• Manual testing can record results right in the test log
• Material to put in the log will vary
• IEEE 829-1998 defines the test log as a chronological ordering of events
![Page 9: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/9.jpg)
Test incident reports
• Incident defined as any unusual result of executing a test (or actual operation)
• May later be categorized as defects or enhancements
• Failure occurs when a defect prevents a system from accomplishing its mission
• Defect tracking becomes an important activity
![Page 10: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/10.jpg)
IEEE Template for test incident reporting
• Identifier
• Summary
• Description
• Impact
• Investigation
• Metrics
• Disposition
![Page 11: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/11.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
End of Lecture
![Page 12: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/12.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
Lecture 17
![Page 13: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/13.jpg)
![Page 14: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/14.jpg)
Writing the test incident report
• Focus on factual data
• Ensure the situation is re-creatable
• Not use emotional judgment
• Not be judgmental
![Page 15: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/15.jpg)
Attributes of a defect tracking tool
• Commercial or custom solutions
• Easy to use and flexible
• Fields should be modifiable to match the organization terminology
• Should facilitate the analysis of data
• Linked to CM system
![Page 16: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/16.jpg)
Testing status and results
• Test status report is often the primary formal communication channel that the test manager uses
![Page 17: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/17.jpg)
Test summary report (IEEE 829)
• Identifier
• Summary
• Variances
• Comprehensive assessment
• Summary of results
• Evaluation
• Recommendations
![Page 18: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/18.jpg)
Test summary report (IEEE 829)
• Summary of activities
• Approvals
![Page 19: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/19.jpg)
When are we done testing?
• “There is no single, valid, rational criterion for stopping. Furthermore, given any set of applicable criteria, how each is weighed depends very much upon the product, the environment, the culture, and the attitude to risk”
![Page 20: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/20.jpg)
If you stop too early
• Many defects left in the product, including show stoppers
• Might be manageable with a small number of customers with expectations set
• May be difficult to switch to new product needs
• Increased employee turnover• Customer frustration with the product
![Page 21: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/21.jpg)
Shipping too late
• Team confidence in product quality
• CS – smaller, predictable
• Loss of revenue, market share
• Greater quality => reputation increases => market share
![Page 22: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/22.jpg)
Defect discovery rate
![Page 23: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/23.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
End of Lecture
![Page 24: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/24.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
Lecture 18
![Page 25: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/25.jpg)
Measuring test effectiveness
• Many organizations do not consciously attempt to measure test effectiveness
• All measures have deficiencies
• Should still develop a method to use in your organization
![Page 26: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/26.jpg)
Categories of metrics for test effectiveness
![Page 27: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/27.jpg)
Defect discovery rates
![Page 28: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/28.jpg)
Bug budget example
![Page 29: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/29.jpg)
Defect removal efficiency (DRE)
Number of bugs found in testing
DRE = ---------------------------------------------
# bugs found in testing + # not found
![Page 30: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/30.jpg)
Defect removal efficiency (DRE)
• Severity and distribution of bugs must be taken into account
• How do you know when the customer has found all the bugs?
• Metrics are “after the fact”• When do we start counting bugs?• Some bugs cannot be found in
testing
![Page 31: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/31.jpg)
![Page 32: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/32.jpg)
![Page 33: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/33.jpg)
![Page 34: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/34.jpg)
![Page 35: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/35.jpg)
# defects weighted by defect age on a project
![Page 36: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/36.jpg)
Formula for defect spoilage
Sum of (# defects X defect phase dis)
Spoilage = -------------------------------------- total number of defects
![Page 37: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/37.jpg)
Formula for defect density
![Page 38: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/38.jpg)
Code coverage
• Not a silver bullet• Tools available• Several weaknesses
– Does not assure code will work
• More effective when used at lower levels
• Global coverage can highlight areas that are deficient
![Page 39: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/39.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
End of Lecture
![Page 40: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/40.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
Lecture 19
![Page 41: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/41.jpg)
The Test Organization
Chapter 8
![Page 42: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/42.jpg)
Test organizations
• No right or wrong way to organize
• Dependent on politics, corporate culture, skill, knowledge of the participants, risk of the product
![Page 43: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/43.jpg)
![Page 44: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/44.jpg)
Independent test teams
• Primary job is testing
• One product or many
• Popularity grown out of frustration
• Could lead to a “brick wall” between developers
![Page 45: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/45.jpg)
Integrated test teams
• Teams made up of developers and testers who all report to the same manager
• May be easier to get buy in earlier if working on a team
• Professional testers• Under pressure, may ship product
prematurely
![Page 46: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/46.jpg)
Developers
• Developers and testers are the same people
• Fewer communication problems• Lack of unbiased look at the system• Reduce risk by having a rigorous test
process, adequate time, business expertise, CM enforced, training, and exit criteria
![Page 47: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/47.jpg)
Other organizational approaches
• Test coordinator
• QA
• Outsourcing
• IV&V
![Page 48: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/48.jpg)
Office environment
• Office space
• Location relative to other participants
• Cube vs office vs common office
• Immersion time
number of uninterrupted hoursE-Factor = ---------------------------------------
Number of body present hours
![Page 49: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/49.jpg)
Office environment
• Quiet time• Meetings
– Start meeting on time– Publish an agenda– Specify who should attend– Keep attendees to a workable number– Limit conversations– Have someone take notes– Urge participation– Choose a suitable location
![Page 50: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/50.jpg)
The software tester
Chapter 9
![Page 51: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/51.jpg)
Characteristics of good testers
• Inquisitive• Functional/business knowledge• Detail oriented• Open minded• Good personality• Technical background• Testing experience• Team player
![Page 52: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/52.jpg)
Finding good testers
• Developers
• Users
• Help desk/technical support
• Technical writers
• QA personnel
• Recent graduates
• Other companies
![Page 53: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/53.jpg)
Hiring testers
• Define job requirements
• Read the resume
• Prepare yourself mentally
• First impressions are important
• Learn how to ask questions
• Ask to demonstrate skills
• Take notes
![Page 54: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/54.jpg)
Hiring testers
• Answer candidates questions
• Check references
• Team interviews
![Page 55: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/55.jpg)
Other test organization issues
• How many testers do you need?
• Retaining staff
• Working overtime
• Software testing certifications
![Page 56: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/56.jpg)
Categories of certifications
![Page 57: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/57.jpg)
![Page 58: CSE 7314 Software Testing and Reliability Robert Oshana Lecture 16 oshana@airmail.net.](https://reader035.fdocuments.us/reader035/viewer/2022062518/56649eb35503460f94bba3d1/html5/thumbnails/58.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
End of Lecture