Lecture 3: Project Management, Part 2 - University of...

1

Prof. Shervin Shirmohammadi ELG 4912 3-1

Lecture 3:

Project Management, Part 2:Verification and Validation, Project Tracking,

and Post Performance Analysis

Prof. Shervin Shirmohammadi

SITE, University of Ottawa


V & V

• Objective: to find problems before the customer does.

• Verification: verifies whether the product of a given phase satisfies the requirements set at the beginning of that phase

– Reviews & walkthroughs of many things: plans, design, specifications, …

• Validation: actual testing of the system

– module testing, user acceptance testing, regression testing ….

2


Reviews

• It is estimated that 30 to 50% of all project costs are consumed by rework.

– Rework is caused by either change or problems/mistakes.

– We need to catch these problems as early as possible.

• Reviews are one way to catch problems up front.

• Requires substantial initial investment, but returns very well at the end.

• A.k.a static testing


Why Review?• Some figures (different

studies):

– Finds 60 – 90% of all

defects

– Between 2 to 10 times

more efficient than

actual testing

– Catches errors in 1/4th of

the time it takes to catch

them with dynamic

testing

– ROI = Savings/Costs =

4.48 (1997 data of 112

projects)

3


The Ripple Effect• Ripple: affects multiple artifacts

Scenario:

no

Reviews.

Imp

Module

Test

SRS

SDD


The Ripple Effect (…)

Scenario:

with

Reviews.

Imp

Module

Test

SRS

SDD

4


What to Review?

• Everything:

– Proposal

– Contract

– Schedule

– Budget

– SPMP

– SRS

– SDD

– Test Plan

– Component Specs

– Compatibility

– …


The Review Process

• Conduct reviews before exiting each phase, and

also after completion of deliverables

(milestone).

• Gather stakeholders: customer, architects,

analysts, modelers, designers, …

• Assign roles:

– Author

– Moderator

– Presenter

– Scriber

– Inspector (reviewer)

5


Roles• Author: must update the work product, usually

the person who originally constructed it

• Moderator: ensures others perform their roles to the best of their abilities; should have no vested interest in the outcome

• Presenter: explains the functionality of each section of the work product

• Scriber: records all found problems, including suggestions, and assigns severity. Completes the report.

• Reviewer: each member of the team, including the above

• Why have both author and presenter?


Review Attitude

• Review the product, not the person!

• Find problems, don’t solve them.

• Concentrate on finding deviations from the requirements.

• Don’t review more than 1-2 hours straight.

• Limit debate.

• Take responsibility for your review (also credit).

• Prepare at least a day in advance.

• Occasionally, review the review process itself.

6


Dynamic Testing• Dynamic as opposed to static testing (review),

does actual hardware/software testing.

• Purpose: to fix as many errors/problems as possible.– Can we test everything?

– It is important to have a test plan.

• Testing not to be confused with debugging– Debugging: the activity of finding and fixing the

error which is discovered by the test.

• Acceptance testing should not be done by developers– Cognitive Dissonance: subconscious resistance to

admitting one’s own mistake


Types of Tests

• Acceptance: is the final (not always) test based on the customer’s specification.

• Alpha: when development is nearing completion.

• Beta: when development and testing are nearing completion.

• Black-box: based on requirements and functionality, not on internal knowledge of the design and code.

• White-box: based on internal logic (conditions, branches, …).

• Integration: combining all components.

• Regression: retesting after fixes and patches.

• Stress: to test load and performance.

7


Test Plan• A document describing the testing activities and

approach.

• Describes:– Objective

– Scope

– Approach

– Focus

• Identifies:– items to be tested

– the actual test to be done (test cases)

– schedules

– personnel

– the level of acceptable risk

• Can start right away, by specifying how each requirement can be tested.


Test Plan (…)

• PM’s responsibility is to ensure:

– validation criteria has been set correctly (with

traceability)

– Major test phases have been identified correctly

– Main functions (specially the riskier ones) are

tested first

– Test plan maps well to the project plan

– Resources are identified and available

– All required types of tests are included (stress,

regression, integration, …)

8

Example Test Case Table


Test

Case ID

Test Case

Description

Setup Instructions Execution

Instruc-

tions

Expected

Results

Observed

Results

Pass/Fail/

Conditional

Pass

HW-1 Sensor Powers Up

HW-n

SW-1 Sensor Initializes

SW-2 Sensor Sends Data

SW-100 Mainline Program

Initializes

SW-101 Interrupt Received

And Processed

SW-200 Executable Loaded to the Target

SW-250 Blue-Tooth module is

Tx/Rx -ing

Sys-1 GUI comes up

Sys-2 Data I/O is correct

Sys-3 Command accepted, Actuator 1 moves

Sys-20 Aircraft Sends GPS Position

SW-30 Heart Rate waveform is displayed


When to Stop Testing?• We already saw that we can’t afford to test everything. So when to

stop?

• Defect detection follows an S curve (also called Logistic Curve)

• as 1/DefectDensity approaches

C, there is diminishing return:

large number of tests are need

to find smaller number of

remaining defects

• Stop testing when:

• 1/DefectDensity reaches a

certain value we can live with;

or

• set up a testing budget and stop

testing when that budget is

exhausted

9


Project Tracking

• Project management doesn’t finish when the plan is ready

– it has just begun!

• PM must monitor the progress of the project, and control it.

• At any given point in time, PM must have an estimation of two sets of numbers:1. how much time, cost (effort + others)

and how what functions have been done up till now;

2. how much time and cost (effort + others) are needed and what fnctions are left to finish the project.

SCOPE

COST

TIME


PM Process: Typical Iteration

10


Main Control Components

• Throughout the project, PM should monitor and

control:

– Scope and functions

– Quality

– Risks

– Cost

– Schedule


Scope Control

• Mainly controlled through Change Management

• Change is not free!

• Proposed changes are first reviewed!

• Impact of the change is analyzed: function, budget, duration.– New test cases according to new changes

– Verification & validation of new changes

• A decision is made on whether or not to proceed with the change.

Specially important during release time!

• After approval (by who?) the changes are applied.

• This avoids changes to affect the system in an unaccountable

manner.

• If you’re not managing changes, you’re not managing at all!

11


Change Request

Change the look and feel of the user interface according to

attached design.

The current UI has given a bad score on the usability test.


Risk Monitoring

• Maintain and update the top 10 list of risks, at least on a

weekly basis. Take appropriate action accordingly.

12


Cost Monitoring• Budget is determined from the WBS, and the estimates.

• Using accounting data, PM can create a Cost Baseline.

– This is not just salaries; it also includes equipment, material,

supplies, software, hardware, …

• Incorporate change into the Cost.

time

cost

Budget At

Completion

(BAC)Cost

Baseline


Schedule Monitoring

• Update estimates periodically.

– Developers give the PM new estimates periodically.

– Estimates should be in the form of time spent, time remaining. “% complete” is not a useful monitoring metric!

• Why?

• Incorporate changes into the schedule.

• Can use Earned Value Managementtechniques for forecasting and controlling the schedule and the cost.

13


Earned Value Management

• A technique to measure real progress, not

just the effort spent.

• Given the effort spent, it calculates what

was accomplished with that effort, and

how that translates into progress for the

project as a whole.

• Dates back to the 1800s in industrial

engineering.


EVM Components

• 3 major components:

– BCWS - Budgeted Cost of Work Scheduled

• Yearned value: it is the cost baseline.

– BCWP – Budgeted Cost of Work performed

• Earned value: what the project really earned at a given point

in time (how much work is done?)

– ACWP – Actual Cost of Work Performed

• Burned value: what the project actually burned to get to this

point (how much did the work done cost?)

• In the above, typically considers milestones that

are 100% done, partial work is not counted.

14


Measures

• CV: Cost Variance = BCWP – ACWP

• SV: Schedule Variance = BCWP – BCWS

• CPI: Cost Performance Index = BCWP/ACWP

– What does it mean?

• SPI: Schedule Performance Index = BCWP/BCWS

– What does it mean?

• CR: Critical Ratio = SPI × CPI

• ECAC: Estimated Cost At Completion = BAC/CPI

• ETAC: Estimated Time At Completion =

Original Time /SPI


CR Controlling

15


Example

Assume today is May 1st.


Example (…)

• Assume all cost figures in $K

BCWS = 15+20+25+20 = 80

BCWP = 15 + 20 + 25 = 60

ACWP = 10 + 15 + 25 = 50

CV = BCWP – ACWP = 60 – 50 = 10 good

SV = BCWP – BCWS = 60 – 80 = -20 bad

CPI = BCWP / ACWP = 60/50 = 1.2 good

SPI = BCWP / BCWS = 60/80 = 0.75 bad

CR = CPI × SPI = 1.2 × 0.75 = 0.9 OK

• Analysis?

16


Post Performance Analysis• One of the most important indications of maturity is “Continuous

Process Improvement”.

• After a project is finished, we need to:– measure the process, collecting process metrics

– analyze them

– devise strategies for improving the process

– feed them back into the next project

• These activities are known as Post-Performance Analysis (PPA), a.k.a. post-project analysis, or postmortem analysis.

• Note that this is can be done both during the project (iteration PPA) and/or at the end of the project (project PPA)– In fact, the interim observations are the main metrics used in the project PPA

as well.

– Why not wait until the very end?

• PPA activities will lead to a PPA Report, containing metrics summary and process improvement recommendations.


Process• Indicate, at the beginning of the project, what data should

be collected, such as:– Work products data: how many components, size, usage

frequency, power consumption, …

– Risks, their impact, how they were resolved, lessons learned

– Number and frequency of Changes to specs, plan, estimates, …

– Original estimates vs. actual time for various components

• These should be gathered by individuals during the project, while memories are still fresh and reasons for decisions can be clearly remembered.

• At the end, assemble a team for the PPA meeting.

• Do the analysis and recommendations during the meeting– In some cases more than one meeting might be required

• Publish the PPA Report.

17


PPA Meeting• Give people enough advanced notice to prepare.

• For the project PPA (final one), send out questionnaires for people to fill in.

• Keep the meeting short.

• Start off by mentioning the purpose.– Very important for everyone to understand that this is not a

finger-pointing session, but a process analysis and improvement session.

• Go over everyone’s collected data, metrics, and suggestions.– Categorize the data as needed

• Watch out for and avoid finger-pointing!– It is not a personnel review meeting.

• Analyze and recommend improvements

• Publish the PPA report.


PPA Questionnaire• In addition to project specific questions, the following

questions can be very valuable when asked from the individual’s perspective:– Identify things that were done right.

– Identify things that were done wrong.

– Identify any exceptional situations.

– How would you do things differently, if you were to do it all over again?

– Describe the most important 3 things that you could personallyhave done to improve the process.

• Other questions:– Are you proud of the accomplishments of this project?

– Was the communication level effective?

– Was the project well-managed?

– Did we do the best that we can?

Lecture 3: Project Management, Part 2 - University of...

Documents

Transcript of Lecture 3: Project Management, Part 2 - University of...