1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay....

Verification & Validation

IS301 – Software EngineeringLecture #30 – 2004-11-10

M. E. Kabay, PhD, CISSPAssoc. Prof. Information Assurance

Division of Business & Management, Norwich University

mailto:mkabay@norwich.edu V: 802.479.7937

Topics

Verification and validation planningSoftware inspectionsAutomated static analysisCleanroom software development

Verification and Validation

Assuring that software system meets user's needs

Objectives

To introduce software verification and validation and to discuss distinction between them

To describe program inspection process and its role in V & V

To explain static analysis as verification technique

To describe Cleanroom software development process

Verification vs Validation

Verification:

“Are we building the product right?” software should conform to its

specificationValidation:

“Are we building the right product?” software should do what user really

requires

V & V Process

Whole life-cycle processV & V at each stage in software process

Two principal objectivesDiscover defects in systemAssess whether system is usable in

operational situation

Static and Dynamic Verification

Software inspections Analysis of static system representationStatic verificationMay be supplemented by tool-based

document and code analysisSoftware testing

Exercising and observing product behavior (dynamic verification)

System executed with test data and its operational behavior observed

Static and Dynamic V&V

Formalspecification

High-leveldesign

Requirementsspecification

Detaileddesign

Program

PrototypeDynamicvalidation

Staticverification

Program Testing

Can revealPresence of errors But NOT their absence

Successful test Discovers one or more errors

Only validation technique for non-functional requirements

Should be used in conjunction with static verification to provide full V&V coverage

Types of Testing

Defect testingTests designed to discover system defectsSuccessful defect test reveals presence of

defects in systemStatistical testing

Tests designed to reflect frequency of user inputs

Used for reliability estimation

V & V Goals

Establish confidence that software is fit for purpose

Does NOT mean completely free of defectsGood enough for its intended use Type of use will determine degree of

confidence needed

V & V Confidence

Depends on system’s purpose, user expectations and marketing environmentSoftware function

Level of confidence depends on how critical software is to organization

User expectationsUsers may have low expectations of

certain kinds of softwareMarketing environment

Getting product to market early may be more important than finding defects in program

Sound familiar

Testing and Debugging

Defect testing and debugging are distinct processes

Verification and validation concerned with establishing existence of defects in program

Debugging concerned with locating and repairing these errors

Debugging involves formulating hypotheses about program behavior then testing these hypotheses to find system error(s)

Debugging Process

Locateerror

Designerror repair

Repairerror

Re-testprogram

Testresults Specification Test

V & V Planning

Careful planning required to get most out of testing and inspection processesStart early in development process

Identify balance between Static verification and Testing

Test planning is about Defining standards for testing process

rather than Describing product tests

V-Model of Development

Requirementsspecification

Systemspecification

Systemdesign

Detaileddesign

Module andunit codeand tess

Sub-systemintegrationtest plan

Systemintegrationtest plan

Acceptancetest plan

ServiceAcceptance

testSystem

integration testSub-system

integration test

Structure of Software Test Plan

Testing processRequirements traceabilityTested itemsTesting scheduleTest recording proceduresHardware and software requirementsConstraints

Software Inspections

People examine representation of system to find anomalies and defectsMay be applied to any representation of

systemRequirements, design, test data . . .

Do not require execution of system Used before implementation

Very effective technique for discovering errors

Inspection Success

In testing, one defect may mask another so several executions are required

Reuse domain and programming knowledge so reviewers are likely to have seen types of error that commonly arise

Thus many different defects may be discovered in single inspection

Inspections and Testing

Inspections and testing are complementary and not opposing verification techniques

Both should be used during V & V process Inspections can check

Conformance with specification But not conformance with customer’s real

requirements Inspections cannot check non-functional

characteristics Performance, usability, etc.

Program Inspections

Formalized approach to document reviews Intended explicitly for defect DETECTION (not

correction)Defects may be

Logical errors, Anomalies in code that might indicate

erroneous condition (E.g. uninitialized variable)

Or non-compliance with standards

Inspection Pre-Conditions

Precise specification must be availableTeam members must be familiar with

organization standardsSyntactically correct code must be availableError checklist should be preparedManagement must accept that inspection will

increase costs early in software processManagement must not use inspections for

staff evaluationsI.e., finding errors does not necessarily

mean programmer is BAD

Inspection Process

Inspectionmeeting

Individualpreparation

Overview

Planning

Rework

Follow-up

Inspection Procedure

System overview presented to inspection team

Code and associated documents distributed to inspection team in advance

Inspection takes place and discovered errors noted

Modifications made to repair discovered errors

Re-inspection may or may not be required

Inspection Teams

Made up of at least 4 membersAuthor of code being inspectedInspector who finds errors, omissions and

inconsistencies Reader who reads code to teamModerator who chairs meeting and notes

discovered errorsOther roles:

Scribe Chief moderator

Inspection Checklists

Checklist of common errors should be used to drive inspection

Error checklist is programming language dependent

‘Weaker’ type checking needs larger checklistExamples:

Initialization Constant namingLoop terminationArray bounds, etc.

Inspection Checks (1)

Data FaultsAre all program variables initialized before

their values are used?Have all constants been named?Should the lower bound of arrays be 0, 1 or

something else?Should the upper bound of arrays be equal to

the size of the array or (size – 1)? If character strings are used, is a delimiter

explicitly assigned?

Control FaultsFor each conditional statement, is the condition

correct? Is each loop certain to terminate?Are compound statements correctly bracketed? In case statements, are all possible cases

accounted for?

Inspection Checks (3) Input/Output Faults

Are all input variables used?Are all output variables assigned a value

before the are output? Interface Faults

Do all function and procedure calls have the correct number of parameters?

Do formal and actual parameter types match?Are the parameters in the right order?If components access shared memory

(globals), do they have the same model of the shared memory structure?

Storage Management FaultsIf a linked structure is modified, have all

links been correctly reassigned?If dynamic storage is used, has space been

allocated correctly?Is space explicitly deallocated after it is no

longer required?Exception Management Faults

Have all possible error conditions been taken into account?

Inspection Rate

Estimating cost of inspection for 500 lines of code: 500 statements/hour during overview = 1 hr per person

x 4 people = 4 person-hours 125 source statements/hour during individual

preparation = 4 hours x 4 people = 16 person-hours 90-125 statements/hour can be inspected in meeting

with 4 people in team = ~5 hours x 4 people = ~20 person-hours

Inspection is therefore expensive process Inspecting 500 lines thus takes ~ 4 + 16 + 20 =

~40 person-hoursEstimate programmer salary $80K/2K hr ~$40/hrMultiply by 2 for extended costs = $80/hrTherefore costs of 40 person-hours effort =

~ $3,200

Automated Static Analysis

Static analyzers = software for source text processingParse program textTry to discover potentially erroneous

conditionsReport to V & V team

Effective aid to inspectionsSupplement to but not replacement for

inspections

Static Analysis ChecksFault class Static analysis check

Data faults Variables used before initialisationVariables declared but never usedVariables assigned twice but never usedbetween assignmentsPossible array bound violations Undeclared variables

Control faults Unreachable codeUnconditional branches into loops

Input/output faults Variables output twice with no interveningassignment

Interface faults Parameter type mismatchesParameter number mismatchesNon-usage of the results of functionsUncalled functions and procedures

Storage managementfaults

Unassigned pointersPointer arithmetic

Stages of Static Analysis (1)

Control flow analysisChecks for loops with multiple exit or entry

points, finds unreachable code, etc.Data use analysis

Detects uninitialized variables, variables written twice without intervening assignment, variables which are declared but never used, etc.

Interface analysisChecks consistency of routine and

procedure declarations and their use

Stages of Static Analysis (2)

Information flow analysisIdentifies dependencies of output

variables. Does not detect anomalies itself but highlights information for code inspection or review

Path analysisIdentifies paths through program and sets

out statements executed in that path. Again, potentially useful in review process

Both these stages generate vast amounts of information. Must be used with care.

LINT Static Analysis

138% more lint_ex.c

#include <stdio.h>printarray (Anarray) int Anarray;{ printf(“%d”,Anarray);}main (){ int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ;}

139% cc lint_ex.c140% lint lint_ex.c

lint_ex.c(10): warning: c may be used before setlint_ex.c(10): warning: i may be used before setprintarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(11)printf returns value which is always ignored

Use of Static Analysis

Particularly valuable for language such as CWeak typing Hence many errors undetected by compiler

Less cost-effective for languages like JavaStrong type checkingCan therefore detect many errors during

compilation

Cleanroom Software Development

Name derived from 'Cleanroom' process in semiconductor fabrication

Philosophy is defect avoidance rather than defect removal

Software development process based on:Incremental developmentFormal specificationStatic verification using correctness

argumentsStatistical testing to determine program

reliability

Cleanroom Process

Constructstructuredprogram

Definesoftware

increments

Formallyverifycode

Integrateincrement

Formallyspecifysystem

Developoperational

profileDesign

statisticaltests

Testintegrated

system

Error rework

Cleanroom Process Characteristics

Formal specification using state transition model

Incremental developmentStructured programming

Limited control and abstraction constructs are used

Static verification using rigorous inspectionsStatistical testing of system (covered in Ch.

21)(but not in this course)

Incremental Development

Formalspecification

Develop s/wincrement

Establishrerquirements

Deliversoftware

Frozenspecification

Requirements change request

Formal Specification and Inspections

State-based model = system specification Inspection process checks program against

this modelProgramming approach defined so that

correspondence between model and system is clear

Mathematical arguments (not proofs) are used to increase confidence in inspection process

Cleanroom Process Teams

Specification teamDevelops and maintains system

specificationDevelopment team

Develops and verifies software.Software NOT executed or even compiled

during this processCertification team

Develops set of statistical tests to exercise software after development

Reliability growth models used to determine when reliability acceptable

Cleanroom Process Evaluation

Results in IBM impressive Few discovered faults in delivered systems

Independent assessment: Process no more expensive than other

approachesFewer errors than in ‘traditional’

development processNot clear how this approach can be

transferred to environment with Less skilled or Less highly motivated engineers

Key Points (1)

Verification and validation are not same thing. Verification shows conformance with specification; validation shows that program meets customer’s needs

Test plans should be drawn up to guide testing process.

Static verification techniques involve examination and analysis of program for error detection

Key points (2)

Program inspections are very effective in discovering errors

Program code in inspections is checked by small team to locate software faults

Static analysis tools can discover program anomalies which may be indication of faults in code

Cleanroom development process depends on incremental development, static verification and statistical testing

Homework

Required by Wed 17 Nov 2004: For 32 points

22.1 – 22.5 (@4)22.9 & 22.10 (@6) (think hard)

Optional by Mon 29 Nov 2004For up to 25 extra points, write detailed

answers for any or all of22.6, 22.7 (@10)22.8 (@5)

DISCUSSION

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay....

Documents

Transcript of 1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay....

Critical Systems Development IS301 – software Engineering Lecture #19 – 2003-10-28 M. E. Kabay, PhD, CISSP Dept of Computer Information Systems Norwich.

1 Copyright © 2010 M. E. Kabay. All rights reserved. Introduction IS 340 – Information Assurance Lecture # 1 M. E. Kabay, PhD, CISSP-ISSMP Assoc Prof Information.

1 Copyright © 2014 M. E. Kabay. All rights reserved. Introduction IS 340 – Information Assurance Lecture # 1 M. E. Kabay, PhD, CISSP-ISSMP Professor of.

1 Notes content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. System Models IS301 – Software.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Process Improvement IS301 – Software.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Managing People IS301 – Software.

1 Copyright © 2007 M. E. Kabay. All rights reserved. Using the TRACK CHANGES Features in MS-Word M. E. Kabay, PhD, CISSP-ISSMP CTO & Program Director,

Introduction to Software Engineering IS301 – Software Engineering Lecture #2 – 2003-08-28 M. E. Kabay, PhD, CISSP Dept of Computer Information Systems.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Software Evolution IS301 – Software.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Application Architectures IS301.

1 Notes content copyright © 2004 Ian Sommerville. NU-specific content © 2004 M. E. Kabay. All rights reserved. Software Processes IS301 – Software Engineering.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Component-Based Software Engineering.

1 Copyright © 2004 Ian Sommerville. NU-specific content © 2004 M. E. Kabay. All rights reserved. User-Interface Design IS301 – Software Engineering Lecture.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Critical Systems Validation IS301.

1 Copyright © 2004 M. E. Kabay. All rights reserved. Phishing Information Systems Security Association New England Chapter Tuesday 16 Nov 2004 M. E. Kabay,

1 Copyright © 2014 M. E. Kabay. All rights reserved. Employment Practices & Policies CSH5 Chapter 45 “Employment Practices & Policies” M. E. Kabay & Bridgett.

1 Copyright © 2010 Jerry Post & M. E. Kabay. All rights reserved. Queries: Part 1 of 2 IS240 – DBMS Lecture # 6 – 2010-02-13 M. E. Kabay, PhD, CISSP-ISSMP.

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Quality Management IS301 – Software.

1 Copyright © 2014 M. E. Kabay. All rights reserved. Email & Internet- Use Policies CSH5 Chapter 48 “Email and Internet-Use Policies” M. E. Kabay & Nicholas.

1 Notes content copyright © 2004 Ian Sommerville. NU-specific content © 2004 M. E. Kabay. All rights reserved. Socio-technical Systems IS301 – Software.