1

Software Quality Assurance

Software Engineering 9

2

Software Warranty - An Example

Limited Warranty on Software and Media KEYSPAN warrants the disks on which the Software is distributed to be free from defects in materials and workmanship and that the Software will perform substantially in accordance with the Documentation for a period of 90 days from your receipt of the Product. ... If the Product fails to comply with the warranty set forth above, KEYSPAN's entire liability and your exclusive remedy will be replacement of the disk or, at KEYSPAN's option, KEYSPAN's reasonable effort to make the Product meet the warranty set forth above. ...

No Liability for Consequential

Damages In no event shall KEYSPAN

or its resellers be liable for any

damages whatsoever (including,

without limitation, damages for loss of

profits, business interruption, loss of

information, or other pecuniary loss)

arising out of the use of or inability to

use this KEYSPAN product, even if

KEYSPAN has been advised of the

possibility of such damages.

3

McCall’s Triangle of Quality

Maintainability

Flexibility

Testability

Portability

Reusability

Interoperability

Correctness

Reliability

Efficiency

Integrity

Usability

PRODUCT TRANSITIONPRODUCT REVISION

PRODUCT OPERATION

4

Internal and external attributes

Reliability

Number of procedureparameters

Cyclomatic complexity

Program size in linesof code

Number of errormessages

Length of user manual

Maintainability

Usability

Portability

5

Why SQA Activities Pay Off?

cost to findand fix a defect

100

10

logscale

1

Req.Design

codetest

systemtest

fielduse

0.75 1.001.50

3.00

10.00

60.00-100.00

6

Quality Concepts

General objective: reduce the “variation between samples” ... but how does this apply to software?

Quality control: a series of inspections, reviews, tests

Quality assurance: analysis, auditing and reporting activities

Cost of quality

• Appraisal costs

• Failure costs

• External failure costs

7

Software Quality Assurance

FormalTechnicalReviews

Test Planning& Review

Measurement

Analysis&

Reporting

ProcessDefinition &Standards

8

Organization of SQA

- an exampleManagement

Industrial Systems

Finance

EC / EDI Key Accounts

Air: OPUS CCIS

Navy: RDN CCIS IMT - Data Modeling

X-Post

EDItoolbox

EDIsec

EC/EDI projects

DSB

Superfos

Realkreditrådet

Post Danmark

Tele Danmark Internet

Landbr. Rådgiv.center

Den Danske Bank

Systematic Software Engineering

Danish Air Force

Danish Air ForceProject Group

F-16 Simulator

Command & Control

Business Process Improvement

Dansk Standard

AQAP Auditor

Defense Systems

IRIS Interoperability

IRIS - Message Format.

Navy: RDN EWare

R&D and Consulting

9

SQA Considerations

• SQA organizations are rarely staffed with sufficiently experienced or knowledgeable people.

• The SQA management team is often not capable of negotiating with development.

• Senior management often backs development over SQA on a large percentage of issues.

• Many SQA organizations operate without suitably documented and approved development standards and procedures.

• Software development groups rarely produce verifiable quality plans.

• Getting good people into SQA is one of the most difficult problems software managers face. One effective solution is to require that all new development mangers be promoted from SQA.

10

Software Quality Assurance CMM Goals

1. Software quality assurance activities are planned.

2. Adherence of software products and activities to the applicable standards, procedures, and requirements is verified objectively.

3. Affected groups and individuals are informed of software quality assurance activities and results.

4. Noncompliance issues that cannot be resolved within the software project are addressed by senior management.

11

Key points

Software quality management is concerned with ensuring that software meets its required standards

Quality assurance procedures should be documented in an organisational quality manual

Software standards are an encapsulation of best practice

Reviews are the most widely used approach for assessing software quality

12

Reviews & Inspections

... there is no particular reasonwhy your friend and colleaguecannot also be your sternest critic.

Jerry Weinberg

13

Software developmentprocess

Quality managementprocess

D1 D2 D3 D4 D5

Standards andprocedures

Qualityplan

Quality review reports

Quality management and software development

14

What Are Reviews?

• A meeting conducted by technical people for technical people

• A technical assessment of a work product created during the software engineering process

• A software quality assurance mechanism

• A training ground

15

What Reviews Are Not

They are not:

• a project budget summary

• a scheduling assessment

• an overall progress report

• a mechanism for reprisal or political intrigue!!

16

The Players

reviewleader

producer

recorder reviewer

standards bearer (SQA)

maintenance oracle

user rep

17

Conducting the Review

be prepared—evaluate product before the review

review the product, not the producer

keep your tone mild, ask questions instead of making accusations

stick to the review agenda

raise issues, don't resolve them

avoid discussions of style—stick to technical correctness

schedule reviews as project tasks

record and report all review results

1.

2.

3.

4.

5.

6.

7.

8.

18

Reviews: An Effectiveness Scale

Formality

MostEffective

Casual conversation

Peer group review

Informal presentation

Formal presentation

Walk through (FTR)

Inspection (FTR)

19

The inspection process

Inspectionmeeting

Individualpreparation

Overview

Planning

Rework

Follow-up

20

Inspection checks

Fault class Inspection check

Data faults Are all program variables initialised before their valuesare used?Have all constants been named?Should the lower bound of arrays be 0, 1, or somethingelse? Should the upper bound of arrays be equal to the size ofthe array or Size -1?If character strings are used, is a delimiter explicitlyassigned?

Control faults For each conditional statement, is the condition correct?Is each loop certain to terminate?Are compound statements correctly bracketed?In case statements, are all possible cases accounted for?

Input/output faults Are all input variables used?Are all output variables assigned a value before they areoutput?

Interface faults Do all function and procedure calls have the correctnumber of parameters?Do formal and actual parameter types match? Are the parameters in the right order? If components access shared memory, do they have thesame model of the shared memory structure?

Storage managementfaults

If a linked structure is modified, have all links beencorrectly reassigned?If dynamic storage is used, has space been allocatedcorrectly?Is space explicitly de-allocated after it is no longerrequired?

Exceptionmanagement faults

Have all possible error conditions been taken intoaccount?

21

Software inspections: Concerned with analysis of the static system representation to discover problems (static verification)

• May be supplemented by tool-based document and code analysis

Software testing: Concerned with exercising and observing product behaviour (dynamic verification)

• The system is executed with test data and its operational behaviour is observed

Static and dynamic verification

22

Software inspections

Involve people examining the source representation with the aim of discovering anomalies and defects

Do not require execution of a system so may be used before implementation

May be applied to any representation of the system (requirements, design, test data, etc.)

Very effective technique for discovering errors

23

Inspections and testing

Inspections and testing are complementary and not opposing verification techniques

Both should be used during the V & V process

Inspections can check conformance with a specification but not conformance with the customer’s real requirements

Inspections cannot check non-functional characteristics such as performance, usability, etc.

24

Metrics Derived from Reviews

• inspection time per page of documentation

• inspection time per KLOC or FP

• inspection effort per KLOC or FP

• errors uncovered per reviewer hour

• errors uncovered per preparation hour

• errors uncovered per SE task (e.g., design)

• number of minor errors (e.g., typos)

• number of major errors

• (e.g., nonconformance to req.)

• number of errors found during preparation

25

Statistical SQA

Product& Process

measurement

• collect information

on all defects• find the causes of the

defects• move to provide fixes

for the process

… an understanding of how

to improve quality ...

26

Inspection rate

500 statements/hour during overview

125 source statement/hour during individual preparation

90-125 statements/hour can be inspected

Inspection is therefore an expensive process

Inspecting 500 lines costs about 40 man/hours effort = £2800

27

Automated static analysis

Static analysers are software tools for source text processing

They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team

Very effective as an aid to inspections. A supplement to but not a replacement for inspections