Don’t Just “Test”… Validate!!

Agenda

Overview of Testing versus Validation Common Activities and Responsibilities Justification for Validation Discussion

Traditional View of Testing

Project Life Cycle

Design and Coding

Tes

tin

g

Pro

du

cti

on

En

vir

on

men

t

InstallSet For

Year End

Project Life Cycle

Hiring Freeze

Time & Resource Constraints

Design and Coding

Pro

du

cti

on

En

vir

on

men

t

Changes to

RequirementsT

esti

ng Install

Set ForYear End

Risk Factors Examples of Project Risk Factors

SEI identifies at least 10 project risk factors:

1. Over-ambitious Schedule

2. Under-ambitious Budget

3. Over-ambitious or unrealistic expectations

4. Undefined or misunderstood obligations

5. Inadequate software sizing estimate

6. Unsuitable or lack of development process model

7. Continuous requirements change

8. Unsuitable organizational structure

9. Inadequate software development plan

10.Lack of political support

Assess these factors in relationship to the testing effort

Factors Affecting Testing

Short Discussion:

What have you seen?

Testing Objectives

Maximiz

e System Quality

Minimize Testing Effort

Did we build the system right?

What Is Validation?

Providing documented evidence with high degree of assurance the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes

Validation

Risk to Business

Business

Needs

Factors Driving Validation

Critical Success

Factors

Relationship of Risk to ValidationD

egre

e o

f V

alid

atio

n

Degree of Risk

Risk tells you•What to address first•What to address most thoroughly

Validation Objectives

Maximiz

e System Quality

Meet Business Needs

Did we build the right system?

Goals of Validation

Control of system development:—Requirements

meaningful, specified, and approved

—Effective process capability of developer

—Existence of and conformance to SOPs

—Effective, appropriate, documented testing

—Configuration management & change control

Control of system use: —Current accuracy and

reliability

—Continued accuracy and reliability

—Management awareness and control

—Auditability

—Data integrity

—Reviewer independence and credentials

What Makes Up A System?

The Computer System

Hardware

Peripherals

Software

Documentation

The Business Environment

Trained Personnel

Procedures Infrastructure

Testing (traditionally) Encompasses…

Documentation

Trained Personnel

Procedures Infrastructure

The Computer System

HardwareSoftware

Validation Encompasses…

The Computer System

Hardware

Peripherals

Software

Documentation

The Business Environment

Trained Personnel

Procedures Infrastructure

CMM Quality Assurance ProgramProgram Owner Accountabilities

QA Accountabilities occur at every touch point

Idea Generation

GeneralRelease

CustomerRequirement

MarketingDesign

Project Orientation

Initiation

Planning

Executing

Closure

Transition

Implementation Operations Support Retirement

Product Process

Pro

ject

Man

agem

ent

Pro

cess

Test Strategy Test Construction

Test Scope

Test PlanningTest Execution

Test Evaluation

QC Process

Construction

Elaboration

Inception

TransitionSDLC

QA & QC Coming Together

Project Orientation Initiation Planning Executing Closure Transition

Quality AssuranceProcess Consultation

Continuous Improvement

Process Audits Defect Analysis Measurement Analysis

Standards Development

Test & Quality Control

Test Planning

Risk Management Defect Management

Dynamic TestingStatic Testing Test Automation

Measurement & Analysis

Forecasti

ngDefe

ct

Resolutio

n

Metrics Test

Executio

n

Metrics Projec

t Planning

Support Defect

Cost

Analysis

Bench

marking

Common Activities

Each Level is Composed of These Activities

Strategy

Design

Execute

Evaluate

Strategy Development

Determine Validation Strategy Perform Risk Assessment Identify Critical Success Factors Set Validation Objectives Define Validation Activities Estimate Time & Resources Document Validation Strategy

Strategy

Design

Execute

Evaluate

Identify Critical Success Factors

Factor

Correctness

Authorization

Data Integrity

Service Levels

Access Control

Methodology

Reliability

Ease of use

Maintainability

Portability

Performance

Extent to which ...

… system must satisfy stated requirements

… processing requires management authorization

… data stored by the system must be accurate

… schedules must be met

… access must be restricted

… development plan must be followed

… system cannot fail during operation

… effort is required to learn and use system

… effort is required to fix errors in the system

… system can operate in multiple environments

… functions must perform within a specified time

Design

Develop/obtain tools for conducting Validation Activities—Deliverables & Phase Checklists—Procedures describing reviews &

inspections—Validation Matrices—Test Cases—Automated Testing Tools—Audit plans

Design

Strategy

Execute

Evaluate

Examples of Validation Objectives

#

1

2

4

3

Validation Objective

Customer orders processed correctly

System is ready to use

Hold is placed on inventory when QC test fails

Increased through-put

Priority

High

High

Med

Low

Completion Criteria

Enter customer order information as described in the validation test cases and verify the results are correct

No open defects of type SEVERE or HIGHPerformance criteria are metUsers are trained, documentation is ready

Enter FAIL test results for productVerify MRP II indicates status as HOLDVerify product is segregated in warehouse

Measure increase by time study

5 Labels printed as req’d LowPrint labels, verify against approved label copy specifications

Examples of Validation Activities

Activity

Requirements Reviews

Design Reviews

Code Walk-throughs

Code Inspections

Performed by

Business Analysts,

Developers,Users

Developers

Developers

Developers

Description

Study & discuss system requirements to ensure they meet business needs

Study & discuss system design to ensure it will support requirements

Informal analysis of code to find defects

Formal analysis of code to find defects, problem areas, coding techniques, design flaws

Deliverable

Reviewed statement of requirements, ready to be translated into design

Design, ready to be translated into software, hardware, documents, etc.

Software ready to be inspected or tested

Software ready to be tested

Examples of Validation Activities

Activity

Unit Testing

Integration Testing

Acceptance Testing

System Testing

Performed by

Developers

Independent Test Team

Users

Description

Test single units of code. Validates that the software performs as designed.

Test related units, programs, modules. Validates that multiple parts of the system interact according to design.

Conducted at end of Integration Testing. Test entire system. Validates system requirements.

Test system to make sure it works in the environment. Validates business needs.

Deliverable

Software unit ready for testing with other components

Portions of the system ready for testing with other portions

Tested system, based on specification

Tested system, based on user needs

RequirementTest Case

1 2 3 4 5 6Validate sign-on

Validate password

Invalid term ident

Password/ Signon error

Validate Main Menu

Re-enter O, X, M, R

Validate Customer #

x

x

x

x

x

x

x

Customer # Required x

Field entry too long x

Customer ___ not found x

Detailed Validation Matrices

Execution

Organize Validation Tools Train Team Members Execute Validation Plan Track Validation Progress Perform Regression Testing Document Results

Execute

Strategy

Design

Evaluate

Evaluation

Evaluate Compliance of System Development

Evaluate Defects Develop Findings and Recommendations Formalize Report

Strategy

Design

Execute

Evaluate

Who Is Responsible for Validation?

Developer

Business Unit(User)

Quality Assurance

ValidationProcess

Independent Tester

Business Unit Responsibility

Define system requirements

Define data used in the system

Enforce Change Control

Plan & Perform Acceptance Testing

Ensure adequate backup & recovery of data is available

Work with developers to coordinate schedules

Investigate Defects Develop User

Documentation Develop User

Procedures Enforce security Provide audit trail Train personnel

Developer Responsibility

Follow development process

Work with QA to improve the process

Manage changes to system components

Provide adequate backup & recovery

Maintain configuration management

Plan & Perform Unit Testing

Provide Technical Documentation

Maintain communication with QA , Testers & Business

QA Responsibility

Define Development & Validation Process

Verify personnel have adequate training & resources for their job

Review documentation - adherence to standards

Periodically review process activities

Evaluate vendor capabilities & processes

Verify enhancements & maintenance are documented

Conduct Risk Assessments

Verify business unit has established procedures

Ensure system is adequately secure

Independent Tester Responsibility

Follow Validation Process

Working with Users &/or BA’s, perform Requirements Validation

Work with QA to improve the process

Plan & Perform Integration & System Testing

Conduct Risk Assessments

If required, Plan & Perform Stress & Performance Testing

Production

Testing Levels & Validation

Code

Design

Requirements

Business Needs

Unit Testing

Integration Testing

System Testing

Acceptance Testing

Plan for Maintenance

Prepare for Maintenance during Development

Keep Deliverables and Documentation up-to-date—Requirements—Design—Test Cases, Test Data, Test Results

Maintain all related deliverables during error correction—Requirement specifications—Program specifications—Design specifications—Defect reports

Maintenance Deliverables

Risk Assessment Software updates Hardware upgrades System documentation Manuals, procedures Revised/New Test Cases Updated Validation Plan

Validation Issues

Responsibility lies with the user—If outsourced, the responsibility is shared

between the outsourcing organization and the user

Develop deliverables as system is maintained

Group maintenance changes together Schedule maintenance in releases Follow defined methodology

Purchased System Responsibilities

Validation Activity

Feasibility Review

Requirements Review

Design Review

Code Review

Unit Testing

Integration Testing

System Testing

Acceptance Testing

Conversion Testing

Maintenance Testing

Vendor

x

x

x

x

x

x

Customer

x

x

x

x

x

x

System Staff Users

x

x

x

x

x

Cost of Quality The Cost of Correction

Reqmts Design Code Test Prod'n

$

$

AppraisalAppraisalPreventionPrevention

FailureFailure

FailureFailureAppraisalAppraisal

PreventionPrevention

WithoutQuality Process

WithQuality Process

Savings

Cost of Quality Real Benefits

Incremental Benefits

Incremental Costs

Time

Development Maintenance

Net Costs

Net Benefits

Benefits

Costs

Validation Costs and Benefits

Self-Assessment

Are risk assessments being performed on projects?

Is there a methodology in place for system development and validation?

Are verification techniques, such as walk-throughs, being used?

Have all personnel been trained in validation techniques?

Are standards in place for system development and validation?

Are the users active participants in the validation process?

Are the results of the validation used to improve the development and validation processes?

Is the culture ready to accept new processes for validation?

Is there someone responsible for validation of systems?

Y N ?

Validation:Did we build the right system?

Testing:Did we build the system right?

System Validation

Validates BOTH Business & Technical Requirements

Reduces Development Costs by identifying Errors early in Development Cycle

Reduces Risk of Operating Failures after Implementation

Reduces Post Implementation Development Costs by Fixing Defects during Development

Improves Overall Quality of System Greatly improves User Satisfaction

Projects Delivered Without Validation

Projects Delivered With Validation

The Road to Success!!

Summary

Questions / Discussion