James Nowotarski

10 October 2006

SE 325/425Principles and

Practices of Software Engineering

Autumn 2006

2

Topic Duration

Requirements process recap 30 minutes

V model 45 minutes

Testing techniques 15 minutes

*** Break

Current event reports 30 minutes

Testing techniques 60 minutes

Wrap-up 15 minutes

Today’s Agenda

3

Context

Communication project initiation requirements

Modeling analysis design

Construction code test

Deployment delivery support

Planning & Managing

elicitationRequirementsengineeringtasks (Ch. 7-8)

elaborationspecification

Primarydeliverables

functional reqtsnon-functional reqts

analysis modelsoftware reqts spec

Begin with the end in mind - Sample SRSOverviewRevision HistoryTable of Contents1.0 Introduction

1.1 Purpose1.2 Scope1.3 References1.4 Assumptions and Dependencies

2.0 Use-Cases3.0 Requirements

3.1 Functional Requirements3.2 Non-Functional Requirements

3.2.1 Usability3.2.2 Reliability3.2.3 Performance3.2.4 Supportability

4.0 Online User Documentation and Help System

Requirements5.0 Design Constraints6.0 Purchased Components7.0 Interfaces

7.1 User Interfaces7.2 Hardware

Interfaces7.3 Software Interfaces7.4 Communication

Interfaces8.0 Licensing Requirements9.0 Legal, Copyright, and Other

notices10.0 Applicable StandardsIndexGlossary

5

Functional vs. Non-Functional

A functional requirement (FR) describes what the system needs to do.

Example: ‘The system shall display the current customer balance’.

6

Functional vs. Non-Functional

A non-functional requirement (NFR) describes a constraint upon the solution space.

Examples: Performance, flexibility, reliability, usability, portability, maintainability, safety, and security.Also called “quality” requirements, “ilities”, or even “systemic” requirements. Emergent Properties: An NFR that is realized through the careful implementation of other requirements on which it depends. Example: “The query must return its results in less than three seconds” is only realizable once the architecture and much of the system functionality has been implemented.

7

The Requirements Process

Elicitation: Proactively working with stakeholders to discover their needs, identify potential conflicts, and establish a clear scope and boundaries for the project.

Elaboration (Analysis): Gaining a deeper understanding of the product and its interactions.

Specification: Production of a series of documents that capture the system and software requirements in order to support their systematic review, evaluation, and approval.

Validation: Inspecting requirements to ensure their correctness.

Management: Issues such as software configuration management, traceability, impact analysis, and version control.

8

Key Question: Deliverables

Steps Techniques

What does the system need to do?How well does it need to do it?

Functional requirementsQuality requirements

1. Review as-is system2. Identify requirements of

to-be system

Re-engineering AHPInterviewing PrototypingObservationSurveys/Focus GroupsJoint Application Design (JAD)Benchmarking

Elicitation

Roles Estimating guidelines

Business analyst

9

Joint Application Design (JAD)

10

Elicitation Techniques: AHP

Develop Software

Performance Usability Flexibility

ArchitectureChoice 1

ArchitectureChoice 2

Goal

Alternatives

Qualityreqts

.08 .64 .28

.41.59

11

Requirement Qualities

Each individual requirement should be:

Concise

Correct

Non-ambiguous

Feasible

Verifiable

Traceable

Manageable

12

Topic Duration

Requirements process recap 30 minutes

V model 45 minutes

Testing techniques 15 minutes

*** Break

Current event reports 30 minutes

Testing techniques 60 minutes

Wrap-up 15 minutes

Today’s Agenda

13

Where time is spent on big systems projects

Activity % of time

Planning and Modeling

33%

??? 50%

All other Construction and Deployment

17%

14

Why we test

We test software because we cannot guarantee its correctness – under normal development practices.

Testing is the art of devising and executing test cases that have a high likelihood of finding errors.

A small subset of faults accounts for most failures during operation. We need to ‘test smart’ in order to find these

faults.

15

Verification & Validation

Testing is just part of a broader topic referred to as Verification and Validation (V&V)

Pressman/Boehm: Verification: Are we building the product right? Validation: Are we building the right product?

IEEE standard 1012-1998: Requirements validation is the process of

evaluating an implemented system to determine whether it conforms to the specified requirements.

16

Some overriding principles

DRTFTDo it right the first time

“The first mistake that people make is thinking that the testing team is responsible for assuring quality.”-- Brian Marick.

Stage Containment

17

Stage Containment

Communication project initiation requirements

Modeling analysis design

Construction code test

Deployment delivery support

Planning & Managing

errorError detection defect fault

Error origination

18

V-Model

Requirements

Functional Design

IntegrationTest

Unit Test

Code

TechnicalDesign

DetailedDesign

AcceptanceTest

SystemTest

Testing: Test that the product implements the specification

Flow of Work

Verification

Validation

Legend:

19

Terminology

Testing - Ensures that the components of the application are put together correctly, according to the different levels ofspecification. Testing consists of exercising a newly integrated portion of the application by running a number of test cases in controlled mode. Each test case is designed to test a statement of the specification.

Verification - Verification is the checking of a deliverable against a standard of work set out for the process that produces the deliverable. The sources for this standard include: 1) Exit criteria defined for each task; 2) Design and coding standards set up by the project team; 3) Evidence that the process prescribed for executing the activity has been followed; 4) Consistency and completeness checks; 5) Models and templates described as part of the application architecture; 6) Standards set up for using the technical architecture. Verification is usually done through reviews, inspections, and walk-throughs.

Validation - Validation is the checking of a deliverable against the specification/requirements implemented by that deliverable (e.g., checking a database design against the data model).

Stage Containment - Stage containment is a project management objective driven by the desire to minimize the number of defects or faults discovered after the work has been completed and handed off to the next stage of the development process. Activities to achieve stage containment include verification, validation, and testing.

Entry/Exit Criteria - Entry and exit criteria are predefined standards that deliverables must meet before exiting one development stage and entering another. A team handing work off to another part of the project must fully satisfy their exit criteria, while the receiving team verifies that the work meets their standard entry criteria. The entry criteria for a receiving team are frequently the same as the exit criteria for the delivering team.

V Model - The V model of verification, validation, and testing provides a structured testing framework throughout the development process and ensures that both verification and validation are applied to deliverables within a system.

20

Requirements

Functional Design

IntegrationTest

Unit Test

Code

TechnicalDesign

DetailedDesign

AcceptanceTest

SystemTest

Testing: Test that the product implements the specification

Flow of Work

Verification

Validation

Legend:

May include:• Operational

readiness test• Benefits

realization test

May include:• Performance test• Usability test• Stress test• Inter-Application

integration test• Hardware/Software

integration test• Restart/Recovery

test

V-Model: Optional Tests

21

Requirements

Functional Design

IntegrationTest

Unit Test

Code

TechnicalDesign

DetailedDesign

AcceptanceTest

SystemTest

Testing: Test that the product implements the specification

Flow of Work

Verification

Validation

Legend:

V-Model: The Testing Process

For each activity on left side of V:• Develop test conditions and

cycles• Develop entry and exit criteria

for corresponding test

For each activity on right side of V:• Set up environment and test team• Execute test and capture actual

and expected results, identify errors, correct, and regression test

22

Good tests

High probability of finding error Not redundant “Bang for buck”

23

Statement coverage:Goal is to execute each statement at least once.

Branch coverageGoal is to execute each branch at least once.

Path coverageWhere a path is a feasible sequence of statements that can be taken during the execution of the program.

What % of each type of coverage does this test execution provide?

5/10 = 50%

2/6 33%

¼ 25% Where does the 4 come from?

= tested

= not tested

Test Coverage Metrics

24

Design for testability

Understandability/Simplicity Operability/Controllability Antibugging

25

26

V-Model

Requirements

Functional Design

IntegrationTest

Unit Test

Code

TechnicalDesign

DetailedDesign

AcceptanceTest

SystemTest

Testing: Test that the product implements the specification

Flow of Work

Verification

Validation

Legend:

Unit testing

27

Unit testing Focuses on a single software component or

module. Design description guides test generation to

Ensure coverage of important control paths Test the boundaries of the module.

Focuses on internal processing logic and data structures.

Specific tests/Common errors

28

As no unit operates in a vacuum it is necessary to create stubs and drivers.

Moduleto betested

Stub Stub

Driver

RESULTS

Testcases

InterfaceLocal data structuresBoundary conditionsIndependent pathsError handling paths

Pressman: 6th ed., Figure 13.4

Unit test environment

29

V-Model

Requirements

Functional Design

IntegrationTest

Unit Test

Code

TechnicalDesign

DetailedDesign

AcceptanceTest

SystemTest

Testing: Test that the product implements the specification

Flow of Work

Verification

Validation

Legend:

Integration testing

30

Integration testing Integration testing is a systematic technique

for constructing the software architecture while at the same time conducting tests to uncover errors associated with the interfacing.

BIG BANG integration is not advisable! Incremental, piece-meal approaches:

Top-down Bottom-up Sandwich

Regression testing Ensure changes do not introduce unintended side

effects

31

Modules are integratedby moving downward throughthe control hierarchy.

Depth-first approachincorporates all components on a major control path.Example: M1, M2, M6, M8

Breadth-first approachincorporates allcomponents directly subordinate at each level.Example: M2, M3, M4.

M1

M3M2 M4

M6M5 M7

M8

Top-down integration

32

Ma

Mc

D3

Cluster1

Cluster3

Mb

Cluster 2

D1 D2

Bottom-up integration

33

The re-execution of some subset of tests that have already been conducted to ensure that changes have not introduced unintended side effects.

Whenever change is introduced or existing tests uncover errors that are fixed – there is opportunity for new errors to be introduced.

Supported by capture/playback tools. Regression testing includes:

A representative sample of tests that will exercise all software functions.

Additional tests that focus on software functions that are likely to be affected by the change.

Tests focusing on the software components that have been changed.

Regression testing

34

Topic Duration

Requirements process recap 30 minutes

V model 45 minutes

Testing techniques 15 minutes

*** Break

Current event reports 30 minutes

Testing techniques 60 minutes

Wrap-up 15 minutes

Today’s Agenda

35

White box vs. Black box

36

White box vs. Black box

Inputs designed to test a system function

Outputs (should match intended functionality)

37

White box vs. Black box

White box Black box

• Structural testing • Logic paths• Loops• Internal variables• Error conditions

• Functional testing • Incorrect/missing functions• Interface errors• Performance errors

38

V-Model

IntegrationTest

Unit Test

Code

AcceptanceTest

SystemTest

White box

Black box

39

First proposed by Tom McCabe in 1976.

Enables the test case designer to derive a logical complexity measure of the procedural design.

Uses this measure as the basis for defining an upper bound on the number of execution paths needed to guarantee that every statement in the program ins executed at least once.

Uses a notation known as a flow graph.

Each structured notation has a corresponding flow graph symbol.

Basis Path Testing

40

Sequence

if Case

while until

Where each circle represents one or more nonbranching set of source code statements.

Flow Graph Notation

41

Flow chart and corresponding flow graph.

1

2

3

6 4

57 8

910

11

1

2,3

10

11

6

9

8 4,57

42

Compound logic A compound condition occurs when one or more

Boolean operators (logical OR, AND, NAND, NOR) is present in a conditional statement.

Example:

if a OR bthen do Xelse do Yend if

a

b

xy

x

a

b

xy

43

Any path through the program that introduces at least one new set of processing statements or a new condition.

In terms of a flow graph, an independent path must move along at least one edge that has not previously been traversed.

In the previous example:path 1: 1-11path 2: 1-2-3-4-5-10-1-11path 3: 1-2-3-6-8-9-10-1-11path 4: 1-2-3-6-7-9-10-1-11

The path: 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is NOT an independent path because it does not traverse any new edges.

Independence path

44

These paths constitute a basis set for the flow graph.

Design tests to execute these paths.

Guarantees: Every statement has been executed at least once. Every condition has been executed on both its true and false sides.

There is more than ONE correct set of basis paths for a given problem.

How many paths should we look for? Calculate cyclomatic complexity V(G) V(G) = E-N+2 (E = # edges, N = # nodes) V(G) = P + 1 (Where P = number of predicate nodes) V(G) = R (Where R = number of regions)

Basis Set

45

An Example:Procedure Average (Pressman, p.397)

* This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum of the total number valid.

INTERFACE RETURNS average, total.input, total.valid; INTERFACE ACCEPTS value, minimum, maximum;TYPE value[1:100] IS SCALAR ARRAY;TYPE average, total.input, total.valid;

minimum, maximum, sum IS SCALAR;TYPE i IS INTEGER;

46

Continued…

i = 1;total.input = total.valid = 0;sum = 0;DO WHILE value[i] <> -999 AND total.input < 100

increment total.input by 1;IF value[i] >= minimum and value[i] <= maximum

THEN increment total.valid by 1;sum = sum + value[i]

ELSE skipENDIFincrement I by 1;

ENDDO

IF total.valid > 0THEN average = sum / total.valid;ELSE average = -999;

ENDIF

END average

12

34

56

7

8

9

1011

1213

47

1. Use the design or code as a foundation and draw corresponding flow graph.

1

2

3

4

5

6

78

9

10

13

12 11

Steps for deriving test cases

2. Determine the cyclomatic complexity of the resultant flow graph.

V(G) = 17 edges – 13 nodes + 2 = 6V(G) = 5 predicate nodes + 1 = 6.

48

3. Determine a basis set of linearly independent paths.

1

2

3

4

5

6

78

9

10

13

12 11

Steps for deriving test cases

Path 1: 1-2-10-11-13Path 2: 1-2-10-12-13Path 3: 1-2-3-10-11-13Path 4: 1-2-3-4-5-8-9-2…Path 5: 1-2-3-4-5-6-8-9-2…Path 6: 1-2-3-4-5-6-7-8-9-2…

4. Prepare test cases that will force execution of each path in the basis set.

50

Basis Path Testing

In small groups Derive a data flow graph from the code (see

next slide). Define independence paths (basis set) Derive tests to drive each path in the basis set

51

Activity Code (Valid pins 10000-19999):

Procedure Validate_Pin (Valid_Pin, Return_Code)Valid_Pin = FALSEReturn_Code = GOODPin_Count = 0do until Valid_Pin = TRUE or Pin_Count > 2 or Return_Code = CANCELbegin

get Pin_Number (Pin_Number, Return_Code)if (Return_Code ≠ CANCEL)begin call Validate Pin_Number (Pin_Number, Valid_Pin)

if (Valid_Pin = FALSE) thenbegin

output “Invalid PIN, please re-enter PIN”Pin_Count = Pin_Count + 1

endend

endreturn (Valid_Pin, Return_Code)

52

Read Pressman Chapters 21-23 (Project Planning and Management) Mid-term quiz due (see course home page) Current event reports:

CastiglioneHillHoganSemler

For October 17

53

Extra slides

54

Change Control Process

Create InitialSections

Create/ModifyDraft

Review Draft(V&V)

Create Changes to Incorporate

Changes Needed In Document

DocumentApproved

Create Review Revise ReviewReview Approved

Time

...

Document in Production and Under Formal Change Control

Document in Production and Under Formal Change Control

Document Under Development and User Change Control

Document Under Development and User Change Control

55

Waterfall model

Systemrequirements

Softwarerequirements

Analysis

Program design

Coding

Testing

Operations

Source: Royce, W.  "Managing the Development of Large Software Systems."

56

RUP Artifacts by Phase and Discipline

Discipline Inception Elaboration Construction TransitionBusiness Modeling

RequirementsVisionUse Cases (20-80%)ActorsSoftware Req SpecGlossary

Analysis & Design Software Arch Doc

Implementation

Build PlanBuildTest Results

Test

Test PlanTest ScriptTest DataTest Results

Test Strategy

DeploymentDeployment Plan Training Materials

Support MaterialsAcceptance Test ResultsChange Requests

Product

Executable ArchitectureUser Interface PrototypeUser Interface DesignUse Case RealizationDesign ModelDatabase Design

Business Architecture

57

RUP Artifacts by Phase and Discipline

Discipline Inception Elaboration Construction Transition

Configuration and Change Management

Project Management Risk ListRisk Mgmt PlanBusiness CaseQA PlanSoftware Dev Plan

Environment

Dev Case (Process)ToolsGuidelinesTemplatesSupport

CM PlanCM EnvironmentChange Requests

58

Technology

ProcessPeople

The focus of SE 425 is the process component of software engineering

Core Concepts

Technology

ProcessPeople

… for the delivery of technology-enabled business solutions

59

V-Model

IntegrationTest

Unit Test

Code

AcceptanceTest

SystemTest