SAK5102 Software Evaluation

Measuring External Attributes

External Attributes

“those whose measurement must take account both its behaviour and the environment it exists in”

● Internal attributes can be measured objectively, and usually fairly easily

● External attributes can involve aspects that are difficult to quantify, and often introduce subjectivity

● External attributes can usually only be measured toward the end of the development

● We usually measure internal attributes to predict measures of external attributes

● We validate our models (relationships between internal and external attributes) by measuring external attributes

Fitness for purpose

● Does it actually work● Does it contain any “problems”● De facto standard – defect density

The problem with problems

● A human error creates a fault that causes a failure – which to measure?

- Process failures lead to errors - Developers fix faults - testing tries to cause failures - users report failures

Examples● Spelling mistake in document● Incorrect information in documentation that cause valuable data to be

lost● Incorrect information in documentation that causes data to have be re-

entered● One part of source code does not follow coding standard● Screen background is wrong colour● Organisation of controls leads to operator error causing death● Code does not compile due to spelling error of variable name● Code does not compile due to type-mismatch● Application unexpectedly terminates on rare occasions with little data

loss and is quick to restore● Application unexpectedly terminates on rare occasions with

significant data loss and takes significant amount of time to restore● An uncommon key sequence causes the application to terminate in

some circumstances● Code contains function that is never called

Attributes of problems

● Location – where did the problem occur?● Timing – when did the problem occur?● Symptom – what was observed?● End result – which consequences resulted?● Mechanism – how did it occur?● Cause – why did it occur?● Severity – how much was the user affected?● Cost – how much did it cost?

Example: cause of code fault (IEEE draft standard P1044)

● Logic problem - forgotten cases or steps - duplicate logic - extreme condition conditions neglected - unnecessary function - misinterpretation - missing condition test - checking wrong variable - iterating loop incorrectly● Computational problem● Data problem● Enhancement● ….. (table 5.2, pp166)

Defect Density

● Quality as Lack of defects● Defects to be of two types: - known defects : discovered through testing, inspection, and other

techniques - latent defects : may be present but we are as yet unaware● Defect density = number of known defects product size - what is a defect? (error, fault, or failure) - what is product size? - what is product? - what is the process?

Evaluation

● Issues with definition● Is a low defect density good or bad? - is it low because defects aren’t being reported? - does low defect density mean low number of unknown

defects?● Modules with higher than average defect densities are likely to

be risky

Other defect-based metrics

● System spoilage = time to fix post-release defects total system development time

● Time has to be measured carefully and consistently● Otherwise is reasonable metric

Other defect-based metrics

● AT&T Bell Laboratories● Cumulative fault density – faults found internally● Cumulative fault density – faults found by customers● Total serious fault found● Mean time to close serious faults● High-level design review errors per KNCSL● Low-level design errors per KNCSL● Code inspection errors per inspected KNCSL● …..

Defining Quality for Software

● External attribute, does depends on the environment● E.g. “Modifiability” – the ease with which modifications can be

made to the system - how big is the system being changed - how big is the modification to be made - how experienced are the staff in the (language, company,

system,…)

Defining Quality of Software

● But specific notions of quality often also have many characteristics

● E.g. “modifiability” - how well documented is the code – “understandability” - how complex is the structure – “modularity” - the degree to which a change in one place will require

changes in another – “viscosity”● So different people have different ideas of what a given notion

of quality may be

quality criteria Software Quality Models

● Meant to capture the composite nature quality● For the use of the system being considered, consider

the relevant quality factors, which are decomposed into quality criteria, which are measured by quality metrics

Example: McCall Software Quality Model

McCall model

● The McCall quality model is organized around three types of Quality Characteristics:

- Factors (To specify): They describe the external view of the software, as viewed by the users.

- Criteria (To build): They describe the internal view of the software, as seen by the developer.

- Metrics (To control): They are defined and used to provide a scale and method for measurement.

● Since then, various quality models have been defined, adopted and enhanced over the years for example proposed by Boehm

● The need for one recognized standard quality model became more and more urgent. The ISO/IEC 9126 standard is the result of a consensus for a software quality model.

Example

● Consider correctness, which consists of completeness, traceability, consistency

➔ Apply checklist for each criteria for each of Requirement (R), Design (D), Implementation (I). E.g. Completeness

➢ Unambiguous references [R,D,I]➢ All defined functions are used [R,D,I]➢ All defined and referenced calling sequence parameter agree [D,I]➢ Code agree with design [I]➔ For each checklist item i assign 1 to “yes”, 0 to “no”, for each of the artifacts

(Ri, D

i, I

i) and then compute, S

com as

➔ Now compute correctness as

13

Ri

R

Di

D

Ii

I

Scom Stra Scon3

ISO 9126 standard quality model

● Called as Software Product Evaluation: Quality Characteristics and Guidelines for their Use

● As with McCall's this is also based on three levels: - Characteristics (Functionality, Reliability, Usability, Efficiency,

Maintainability, Portability); - Sub-characteristics; - Metrics. ● Each characteristic is refined to a set of sub-characteristics

and each sub-characteristic is evaluated by a set of metrics. Some metrics are common to several sub-characteristics.

ISO9126 - characteristics

● Each type of software has its own quality requirements. For example, if we consider:

ISO9126 : example

Factor criteria metrics

maintainability

correctability

Testability

Expandability

Fault counts

Degree of testing

effort

Change count

Closing timeisolate/fix timefault rate

Statement coveragebranch coveragetest plan completeness

Resource predictioneffort expendature

Change effortchange sizechange rate

usability

● The extent to which the product is convenient and practical to use

● The probability that the operator of a system will not experience a user interface problem during a given period of operation under a given operational profile

● How to measure this?● How to predict this?

Measuring usability

● Evidence of good usability: well-structured manuals, good use of menus and graphics, informative error messages, help functions, consistent interfaces

● Decomposition➔ Entry level➔ Learnability➔ Handling ability

Maintainability

● The ease with which maintenance activities can be performed: corrective, adaptive, preventive, perfective (including new enhancements)

● Decomposition➔ correctability➔ testability➔ expandability

Mean Time To repair (MTTR)

● Problem recognition time● Administrative delay time● Maintenance tools collection time● Problem analysis time● Change specification time● Change time (including testing and review)

Prediction from internal attributes

● “complexity”● Number of changes made (or changes per LOC)● Readability – Fog Index

Reliability

● Probability of failure-free operation● Depends on hardware/software environment and

user● Common metrics include defect density and mean

time to failure

Mean time to failure

● What is failure● Are all failures equal● What is time● How do we predict time

Reliability prediction

● Reliability theory – comes from the hardware world where components eventually wear out

● Create models that describe probability of failure● Software is not hardware● Software does not wear out● As each defect is fixed, reliability should improve

(in long term)