Class 9

04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved

11

CES 592 Telecommunications CES 592 Telecommunications System Product VerificationSystem Product Verification

Sonoma State UniversitySonoma State University

Class Lecture 9:Class Lecture 9:

Software Testing ProcessesSoftware Testing Processes

Fall 2004Fall 2004


22

InstructorsInstructors France AntelmeFrance Antelme Ario BigattiniArio Bigattini Jaseem MasoodJaseem Masood Steven WoodySteven Woody Coordinator: Prof Ravi KumarCoordinator: Prof Ravi Kumar

WebsiteWebsitehttp://www.sonoma.edu/engineering/courses/CES592.shtmlhttp://www.sonoma.edu/engineering/courses/CES592.shtml


33

Software Testing ProcessSoftware Testing Process

Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature

testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking


44

Software Testing ProcessSoftware Testing ProcessTest Plans and Test CasesTest Plans and Test Cases A test plan is a document that describes the A test plan is a document that describes the

planned test activities. For a large software planned test activities. For a large software project, this may be divided into separate feature project, this may be divided into separate feature test plans.test plans.

A test case is a list of steps which test a specific A test case is a list of steps which test a specific feature. Should not be more than one page long. feature. Should not be more than one page long. Must contain a pass/fail criteria. Must contain a pass/fail criteria.

A test case matrix is commonly used in a test A test case matrix is commonly used in a test plan to identify which combinations/permutations plan to identify which combinations/permutations of conditions will be tested.of conditions will be tested.


55

Software Testing Process:Test PlansSoftware Testing Process:Test PlansFrom IEEE Standard for Software Test Documentation IEEE 829-1998

• Test plan identifier – Name / Number• Introduction – Brief description of product & test strategy• Test items – Description of item to be tested • Features to be tested - List• Features not to be tested – Very important to list, prevents assumptions• Approach – Describe your test strategy• Item pass/fail criteria – Must have pass/fail criteria for tests• Suspension criteria and resumption requirements – (Entry / Exit criteria)• Test deliverables – Results: Performance charts, bug lists, bug charts • Testing tasks• Environmental needs – Lab setup• Responsibilities – Define & agree• Staffing and training needs – Continuous training, eye on future• Schedule – Time to perform test cases vs. time for software to stabilize• Risks and contingencies – Identify real & meaningful risks• Approvals – Management, Customer Account team


66

Software Testing Process: Test PlansSoftware Testing Process: Test Plans

Project Milestones:• Testplan approved• Testbed ready• Scripts complete• Testing Start• Testing Finish• Script transferred to regression


77



88


Schedule estimation is difficult!• Have a target schedule (make your best estimate based on prior experience)• When 75% of that schedule has elapsed, nail down a specific date for test completion, taking into account the progress thus far.• Keep track of your project history, so that you can make more accurate schedule estimates for the next project


99

Software Testing Process:Test CasesSoftware Testing Process:Test Cases

•Test Case identifier – Name / Number• Test case owner – Who wrote it? Who is responsible for updating it?• Item to be tested – Describe• Input/Output Specification• Item pass/fail criteria – Must have pass/fail criteria for test case• Automation – Can it be automated? Is it automated? Point to file.•Testing tasks – one-page procedure• Environmental needs – Lab setup• Special procedural requirements• Inter-case dependencies• Test case priority – changes constantly, depends on many factors• Track which software version this test case is valid for• Schedule – Time to perform the test case – keep track of it• Bugs found by this test case – update every time a new bug is found


1010

Software Testing Process: Test CasesSoftware Testing Process: Test Cases

Start writing test cases as the product requirements are Start writing test cases as the product requirements are being defined.being defined.

Show your test cases to the developer who is doing the Show your test cases to the developer who is doing the coding. Get him/her thinking about how the feature coding. Get him/her thinking about how the feature will be tested. Help prevent the bugs from ever will be tested. Help prevent the bugs from ever getting into the code.getting into the code.

Test cases should give enough information to perform Test cases should give enough information to perform the test, but not so much detail that everyone will the test, but not so much detail that everyone will perform the test in the same way. You want to allow perform the test in the same way. You want to allow some variation and randomness in the test case.some variation and randomness in the test case.


1111

Software Testing Process: Test CasesSoftware Testing Process: Test CasesPositive Test Cases: Positive Test Cases:

Does the feature work as required?Does the feature work as required?

Negative Test Cases:Negative Test Cases:Can the feature be broken?Can the feature be broken?

Network: Disconnected, No Ports available…Network: Disconnected, No Ports available…

Disk Storage: File not found, File in use, Disk Full, Invalid Path, CRC errorDisk Storage: File not found, File in use, Disk Full, Invalid Path, CRC error

Memory: Not enough free memory, fragments too small…Memory: Not enough free memory, fragments too small…

Do the Positive testcases first, to verify the feature is Do the Positive testcases first, to verify the feature is working. Then try to break it with the Negative test working. Then try to break it with the Negative test cases.cases.


1212





1313

Software Testing ProcessSoftware Testing Process Regression versus New Feature testingRegression versus New Feature testing

New Feature Testing: Testing the New Feature Testing: Testing the new features that have been added new features that have been added since the previous test cyclesince the previous test cycle

Regression Testing: Re-testing “old” Regression Testing: Re-testing “old” functionality to make sure that no functionality to make sure that no features were inadvertently brokenfeatures were inadvertently broken


1414

Software Testing ProcessSoftware Testing Process Regression versus New Feature testingRegression versus New Feature testing

New Feature Testing: New Feature Testing: Usually a small percentage of the overall Usually a small percentage of the overall number of tests to be performednumber of tests to be performed

Regression Testing: Regression Testing: Number of tests grows with each new Number of tests grows with each new feature added; can easily consume all feature added; can easily consume all testing resourcestesting resources


1515

Software Testing ProcessSoftware Testing Process Regression test prioritiesRegression test priorities

1.1. Check that the bug was actually fixedCheck that the bug was actually fixed

2.2. Check for related bugsCheck for related bugs3.3. Check that the fix didn’t break something elseCheck that the fix didn’t break something else

Testing Computer Software, Kaner, Falk, NguyenTesting Computer Software, Kaner, Falk, Nguyen


1616





1717


Software Testing ProcessSoftware Testing Process Entrance / Exit CriteriaEntrance / Exit Criteria


1818

Software Testing ProcessSoftware Testing ProcessSoftware Testing Entrance CriteriaSoftware Testing Entrance Criteria Feature list defined & completedFeature list defined & completed Code Review completed, documentedCode Review completed, documented Static Analysis completedStatic Analysis completed Draft user documents readyDraft user documents ready Unit testing plan completedUnit testing plan completed

Does the software meet the entrance criteria? Does the software meet the entrance criteria? If so, then software test will be complete in 12 weeks.If so, then software test will be complete in 12 weeks.If not, then software test will not complete on time.If not, then software test will not complete on time.

Measure the number & severity of bugs found after test start:. Measure the number & severity of bugs found after test start:. If bugs exceeds a limit, then entrance criteria was not met, If bugs exceeds a limit, then entrance criteria was not met, 12 week schedule will not be met.12 week schedule will not be met.


1919

Software Testing ProcessSoftware Testing ProcessExit CriteriaExit Criteria Is the software safe for people, property and data? (“First, do Is the software safe for people, property and data? (“First, do

no harm”)no harm”) All test team members agree that it is fit for use?All test team members agree that it is fit for use? Feature testing: At least one full test cycle with >75% pass rateFeature testing: At least one full test cycle with >75% pass rate Regression Testing completed, >90% pass rateRegression Testing completed, >90% pass rate Code Coverage analysis completed, >70% code coverageCode Coverage analysis completed, >70% code coverage No showstopper or catastrophic bugs openNo showstopper or catastrophic bugs open Final review of all open bugsFinal review of all open bugs

DeliverablesDeliverables Automated regression scriptsAutomated regression scripts Updated test plan, test cases, test scripts Updated test plan, test cases, test scripts (constantly re-evaluate)(constantly re-evaluate) Defect Counts, trend summariesDefect Counts, trend summaries Release notes reviewedRelease notes reviewed User Docs reviewedUser Docs reviewed Customer support trainingCustomer support training


2020

Software Testing ProcessSoftware Testing ProcessA word about Entrance Criteria…A word about Entrance Criteria…

Don’t “burn your lead time” by waiting for the Don’t “burn your lead time” by waiting for the perfect, 100% complete software build to be perfect, 100% complete software build to be delivered. Start testing as soon as you can.delivered. Start testing as soon as you can.

Coding Start

Coding Finished

TestingStart

TestingFinished

OfficialTestingStart – Must meet entrance criteria


2121





2222

Software Testing Process: Version ControlSoftware Testing Process: Version Control

Version control for software under testVersion control for software under test One central source for software to be One central source for software to be

testedtested Version control for test plans, test cases, Version control for test plans, test cases,

test scriptstest scripts

Keep track of what was tested on what Keep track of what was tested on what version of software & hardwareversion of software & hardware


2323

Software Testing Process: Version ControlSoftware Testing Process: Version Control

CVS - Concurrent Versions SystemCVS - Concurrent Versions System RCS - Revision Control SystemRCS - Revision Control System SubversionSubversion

Clearcase - Rational SoftwareClearcase - Rational Software PVCS - MerantPVCS - Merant


2424





2525

Software Testing Process: Defect TrackingSoftware Testing Process: Defect Tracking

Agreed-upon definitions for bug Agreed-upon definitions for bug severity, before the project startsseverity, before the project starts

Quality is everyone’s job – if you see Quality is everyone’s job – if you see a problem, it is your responsibility to a problem, it is your responsibility to enter a bug reportenter a bug report

Weekly bug review meeting - “bug Weekly bug review meeting - “bug scrub”scrub”


2626

Software Testing Process: Defect TrackingSoftware Testing Process: Defect Tracking

BugzillaBugzilla GNATSGNATS Debian Bug Tracking SystemDebian Bug Tracking System

Clear DDTS – Rational SoftwareClear DDTS – Rational Software SilkRadar – Segue SoftwareSilkRadar – Segue Software

http://testingfaqs.org/t-track.htmlhttp://testingfaqs.org/t-track.html


2727

Software Testing Process: Defect ReportsSoftware Testing Process: Defect Reports TitleTitle Severity categorySeverity category InputsInputs Expected resultsExpected results Actual resultsActual results Anomalies ObservedAnomalies Observed Bug Info: Error logs, debug tracesBug Info: Error logs, debug traces Date and timeDate and time Procedure stepProcedure step EnvironmentEnvironment Attempts to repeat / Steps to repeatAttempts to repeat / Steps to repeat TestersTesters ObserversObservers Last version observed to passLast version observed to pass


2828

Software Testing Process: Defect ReportsSoftware Testing Process: Defect Reports The Title is the most important part of the The Title is the most important part of the

Defect Report Defect Report – – make it clear and concise!make it clear and concise!

Find the simplest trigger for the bugFind the simplest trigger for the bug Find the most serious consequences of the Find the most serious consequences of the

bugbug Try to determine if the bug is the tip of an Try to determine if the bug is the tip of an

icecube or the tip of an iceburgicecube or the tip of an iceburg Always give the developer a chance to Always give the developer a chance to

debug it in real time – it may take you debug it in real time – it may take you hours to reproduce it laterhours to reproduce it later


2929

Software Testing Process: Exit CriteriaSoftware Testing Process: Exit CriteriaA word about exit criteria:Watch the Defect trend to know if you are ready to release:

Bug Trend

0

100

200

300

400

500

600

700

11/0

7/0

2

11/2

1/0

2

12/0

5/0

2

12/1

9/0

2

01/0

2/0

3

01/1

6/0

3

01/3

0/0

3

02/1

3/0

3

02/2

7/0

3

03/1

3/0

3

03/2

7/0

3

04/1

0/0

3

04/2

4/0

3

05/0

8/0

3

05/2

2/0

3

06/0

5/0

3

06/1

9/0

3

07/0

3/0

3

07/1

7/0

3

07/3

1/0

3

08/1

4/0

3

08/2

8/0

3

09/1

1/0

3

09/2

5/0

3

10/0

9/0

3

10/2

3/0

3

11/0

6/0

3

11/2

0/0

3

12/0

4/0

3


3030

ReferencesReferencesTesting Computer Software, Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyenby C. Kaner, J. Falk, and H. Nguyen

Managing the Testing Process, Managing the Testing Process, Rex BlackRex Black

Software Test Automation, Software Test Automation, Fewster & GrahamFewster & Graham

Code Complete, Code Complete, Steve McConnellSteve McConnell

““Software Testing and Quality Assurance”, Software Testing and Quality Assurance”, Ross CollardRoss Collard

IEEE Standard for Software Test Documentation, Std 829 - 1998IEEE Standard for Software Test Documentation, Std 829 - 1998

Black-Box Testing: Techniques for Functional Testing of Software and Systems, Black-Box Testing: Techniques for Functional Testing of Software and Systems, Boris Boris Beizer, Wiley, 1995Beizer, Wiley, 1995

Classic Testing Mistakes, Classic Testing Mistakes, Brian MarickBrian Marickhttp://http://www.testing.com/writings/classic/mistakes.pdfwww.testing.com/writings/classic/mistakes.pdf

Software QA / Test Resource CenterSoftware QA / Test Resource Centerhttphttp://www.softwareqatest.com/index.html://www.softwareqatest.com/index.html

Class 9

Documents

Transcript of Class 9