Two Complementary EVM Cost-Risk Models1. Use of EVM Trends to Forecast Cost Risks

2. Integrated Cost-Risk Model (ICRM)Utilizing ACEIT

For18 MAR SoCal ICEAA Workshop

David R. GrahamConsultant, Salient Federal Solutions

Carlsbad, CAdgmogul1@verizon.net

703-489-6048

Use of EVM Trends to Forecast Cost Risks

Dr. Roy Smoker*MCR LLCrsmoker@mcri.com

(Other Charts Added by David R. Graham)

Some Charts from Original Presentation at2011 ISPA/SCEA Conference, Albuquerque, NM

(C)2011 MCR, LLC

*Roy E. Smoker (2011): Use of Earned Value Management Trends to Forecast CostRisks, Journal of Cost Analysis and Parametrics, 4:1, 31-51

Main Points of Paper1. EVM data is taken from the PMB’s S-curve at it’s most linear section

– The early part of the PMB’s S-curve represents start up so is not linear– The ending part of the PMB’s S-curve represents contract ending so is also not

linear– Linear regression equations work best when data is linear

2. Regression equation developed to forecast BAC– Unique from most regression equations used in EVM performance projections– Basis: BAC grows due to learning more about the nature of the work as the effort

proceeds and additional work is put on contract through ECOs

3. Ending month of contract can also be forecasted using regression equations– Basis: At end of contract BCWP must equal BAC– Regression equations for BCWP and BAC are set equal to solve for months

• BCWP = BAC $86.35M*Months = $4,970.56 + $31.76M * Months• Then just solve for months = 91.06

– NOTE: Calculation made at month 42 - Coefficients change depending on month selected due to amount of available EVM data increasing over time

– Preferred over usual BAC/Avg BCWP or Lipke’s Earned Schedule

Main Points of Paper (cont)

4. Regression equation developed to forecast percent complete (PC)– PC = 0 + 0.013772546*Month (“0” is the intercept, that is, at 0 time there is 0

percent complete)– At month 42, “0.013772546” is the rate of PC/month so, at that rate the

completion month is: Completion Month = 100% Complete/1.3772546 %= 72.6 months• The PC was measured as the raw value of each monthly BCWP divided by the value of

the BAC for month 42– NOTE: Percent complete (PC) using this method understates completion

months due to not taking into account rate at which BAC is growing

5. Variance at Completion (VAC) is a quantification of value of risk that must be burned down– Regression equation developed to forecast EAC– Subtract forecasted EAC from forecasted BAC = forecasted VAC– True significance of forecasted VAC:

• Some risk issues that are part of VAC are known and should be described on Format 5• Unexpected risks have not yet been discovered but are part of this forecasted VAC

18 Months of Data

(C)2011 MCR, LLC5

Equations:

Note: Even with the significance in the parameters in these equations there is a good degree of variability as indicated by their standard errors in parenthesis.

Regressions run in Excel.

Cum Cum Cum LRE

EOM # Months BCWS BCWP ACWP BAC EAC

Sep-95 25 2,034$ 1,988$ 2,017$ 5,750.00$ 5,750.00$ Oct-95 26 2,150$ 2,095$ 2,142$ 5,776.00$ 5,776.00$

Nov-95 27 2,247$ 2,191$ 2,243$ 5,785.00$ 5,785.00$ Dec-95 28 2,358$ 2,294$ 2,353$ 5,822.94$ 5,823.00$ Jan-96 29 2,477$ 2,400$ 2,462$ 5,870.95$ 5,871.00$ Feb-96 30 2,586$ 2,501$ 2,565$ 5,906.54$ 5,917.00$ Mar-96 31 2,705$ 2,617$ 2,694$ 5,961.16$ 5,972.00$ Apr-96 32 2,817$ 2,729$ 2,817$ 6,000.22$ 6,017.00$

May-96 33 2,921$ 2,828$ 2,932$ 6,088.19$ 6,146.00$ Jun-96 34 3,038$ 2,939$ 3,051$ 6,160.98$ 6,350.00$ Jul-96 35 3,152$ 3,047$ 3,169$ 6,156.54$ 6,344.00$

Aug-96 36 3,245$ 3,138$ 3,274$ 6,132.15$ 6,352.00$ Sep-96 37 3,370$ 3,258$ 3,407$ 6,211.78$ 6,401.00$ Oct-96 38 3,479$ 3,373$ 3,535$ 6,173.79$ 6,363.00$

Nov-96 39 3,579$ 3,470$ 3,656$ 6,118.34$ 6,396.00$

Dec-96 40 3,667$ 3,554$ 3,571$ 6,145.90$ 6,424.00$

Jan-97 41 3,765$ 3,647$ 3,869$ 6,292.16$ 6,570.00$

Feb-97 42 3,866$ 3,738$ 3,978$ 6,269.68$ 6,548.00$

(1) BCWS18 = $89.12M * Months (0.7525) T-stat = 118.44 R2 = 0.9988

(2) BCWP18 = $86.35M * Months (0.6925) T-stat = 124.68 R2 = 0.9989

(3) BAC18 = $4,970.56M + $31.76M * Months ( 86.92) ( 2.56) R2 = 0.9056 T-stats 57.19 12.39

(4) EAC18 = $4,393.13M + $52.62M * Months

( 106.65) ( 3.15) R2 = 0.9459 T-stats 41.19 16.73

All Cost are $M

Full 43 Months of EVM Data (months 25 through 67)(from the Excel based EVM Trend Tool)

Essentially Linear Data

$-

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

$7,000

$8,000

$9,000

0 20 40 60 80

TY $

M

Months

EVM Trend Data(Months 25 thru 67)

BCWS

BCWP

ACWP

BAC

EAC

Question: What do you get when you remove the initial start up months and the contract closeout months from the usual S-curve?Answer: A data set that exhibits the graphic forms shown here. Note, there is a hint of an S-curve without the usual tails.

Research: Can we predict the performance of a long term contract from only 18 months of data covering months 25 thru 42 using linear assumptions? Answer: Perhaps apply the approach to other, completed programs to validate the results presented in Dr. Smoker’s paper.

(C)2011 MCR, LLC7

We know: Monthly BAC but not the final BAC Monthly EAC but not the final EAC Monthly VAC but not the final VACWe don’t know the month of the final VAC

BAC & EAC as a Function of Time

BAC= 31.762x + 4970.6R² = 0.9056

EAC = 52.618x + 4393.1R² = 0.9459

$5,600 $5,700 $5,800 $5,900 $6,000 $6,100 $6,200 $6,300 $6,400 $6,500 $6,600 $6,700

0 10 20 30 40 50TY

$M

Months

EAC vs BAC by Month

BAC

EAC

• The uncertainty in the BAC, based on program office decisions to remove work and add work to the contract, also creates uncertainty in the EAC as shown here • However, both equations based on 18 monthly observations are significant and can be used to predict the growth of both BAC and EAC thru time With EAC growing faster than BAC, the

question is when will this contract reach completion so that VAC stops growing?

(C)2011 MCR, LLC8

VAC as a Measure of Risk

• Risk is measured in EVM terms as any deviation from the original baseline. – That is, risk is anything that results in a variance

• Therefore, VAC is the basic measure of risk encountered by the end of the contract effort– Whether the risk is rooted in opportunity with a positive

variance – Or, is rooted in issues related to planning of scope,

estimating, scheduling, or technical criteria that are identified during testing and generally associated with a negative variance

(C)2011 MCR, LLC9

Risk Burn down (1 of 2)

• The final VAC may be estimated as the difference between the linear forecast of BAC and EAC

• Risk burn down may be measured as the amount of VAC that has been worked off

• Therefore, it is possible to show the %risk burn down as a function of the amount of cumulative VAC that has been incurred relative to the final VAC

(C)2011 MCR, LLC10

Risk Burn down (2 of 2)• Here the green line represents the

%Risk that has been burned down and measured as:

1 - Cum VAC/Final VAC

• It is interesting to note that early in this program, risk is being burned down faster than the remaining work is being accomplished.

• Finally, Risk is burned down to zero as remaining work is reduced to zero and percent complete approaches 100%

(C)2011 MCR, LLC11

Summary – Contract Scope

• We have learned– An S-curve with its tails removed exhibits significant

linearity with variability– The scope of a contract grows across time– New work pushes out the expected completion date– There is a future date where

• BCWP will equal BAC• From this equivalency the expected completion date can

be calculated

– Each monthly %complete drops as BAC grows

(C)2011 MCR, LLC12

Summary – Trend Analysis

• We have learned– Normal monthly EACs fall short of final EAC

• Due to same contract scope growth that affects BAC

– Trend analysis • Helps identify the completion date

– Can then estimate the final EAC– Can then estimate the final BAC

– Final VAC • Can be estimated as: (Final BAC – Final EAC)• VAC appears useful in measuring the value of a program’s risks

(planning, estimating, scheduling, technical)• May be used to measure how risks get burned down across the

period of performance from ATP to Estimate Completion Date

(C)2011 MCR, LLC13

Integrated Cost-Risk Model (ICRM)Utilizing ACEIT

David R. GrahamConsultant, Salient Federal Solutions

Carlsbad, CAdgmogul1@verizon.net

703-489-6048

NOTE: Special thanks to Darren Elliott of Tecolote, Inc., for actual ICRM programming

Outline• What ICRM Brings• ICRM Model Overview Narrative • ICRM Model Structure Illustration• Note on Simulation Assumptions Used in the ICRM Model• 5X5 Risk Matrix Rating Scales• Dominant Likelihood Algorithm• DAU EVM 201 LAR Risk Register• ICRM Mechanics General Overview• ICRM WBS Element and Risk Prioritization Tornado Charts & PDF/CDF Graphs• ACEIT Workscreen Examples• ICRM Customization• Summary

What ICRM Brings1. EVM data and risk register results into a probabilistic context

using the DAU Light Assault Reconnaissance (LAR) Vehicle case study as the database for illustrating the ICRM

2. True confidence levels of contractor WBS-level EACs– Through applying a range of WBS-level EVM PFs to create a distribution of possible

‘adjusted’ BCWR values at the WBS element level (cum ACWP + adjusted BCWR = EAC)

– Allowing identification of where contractor WBS-level EACs fall in the distributions

3. WBS-level Risk Register-driven cost-risk distributions– Identifies risk register risks to affected WBS elements– Applies risk likelihoods and cost consequence ranges to WBS element BCWR values

4. Integration of both WBS PF-based and WBS Risk Register-based distributions by statistically summing them through monte carlo simulations in ACEIT producing an overall EAC cost-risk distribution

What ICRM Brings (cont)

5. Enables prioritization– By WBS elements most cost-impacted by risks, and– By risks causing the most significant cost impacts

6. These results provide the basis for an ongoing meaningful dialogue that is not happening today between the EVM analysts, technical risk management teams, cost estimators, schedule analysts, project officers and, ultimately, the program managers based on cost impacts caused by risks

ICRM Model Overview Narrative • Enter in EVM data (e.g., BCWS, BCWP, ACWP cum-to-date)• Non-Probabilistic EAC calculation

– Derive BCWR (BAC-BCWP); adjust by performance factor; develop EAC (i.e., ACWP cum-to-date + adjusted BCWR)

– NOTE: Can use BAC assuming growth derived from EVM Trends Approach

• Performance Factors range data– Use CPI; SPI; CPI*SPI; (can use other PFs) as separate cases or two at a time in a Min & Max

uniform distribution– Utilize ACEIT’s capability to identify minimum/maximum results to construct a PF-based

range distribution relaxing the analyst’s workload• Risk Register data

– Identify risks and their impacts to specific WBS elements• One risk to one WBS; one risk to many WBSs; many risks to one WBS

– Use midpoint of risk likelihoods (e.g., if range=5%-20%, use 12.5%)– Identify risk cost consequence ranges (i.e., low, most likely & high values) and apply

resulting percent impacts on adjusted BCWRs• Incorporate all risks in Latin Hypercube ACEIT simulations and calculate EAC as a total

value that includes all risk effects

ICRM Model Structure Illustration

EAC Calculations Based on Single Performance Factor (PF)

Statistical PF Range & Risk Register-Impacted EAC

Risk Register Risk Inputs (ID WBSs impacted; Cost Consequences; Likelihoods)

Performance Factor Range Impact Calculation (Range * affected WBS items)

Note on ICRM Simulation Assumptions

• Decision Rule on Whether the Risk Actually Happens– Random number generator produces a number

between “0” & “1” that compares against the risk register’s likelihood

• The “Dominant Likelihood” algorithm determines the ‘winner’ of the comparison and either includes the full risk’s cost consequence or none of the consequence

5x5 Risk Matrix Rating ScalesDefinitions From Program’s Risk Management IPT

• Level Likelihood of Occurrence– 1 Not Likely (5% - 20%)– 2 Low Likelihood (21% - 40%)– 3 Likely (41% - 60%)– 4 Highly likely (61% - 80%)– 5 Near certainty (81% - 99%)

NOTES ON COST CONSEQUENCE APPROACHES

Alternative 1: Percent of last approved cost estimate

Alternative 2: Percent of affected WBS element’s BCWR (i.e., S/C, P/L, etc.)

Alternative 3: Percent additional resources taken as a function of burn rate per schedule slip on WBS element(s) affected

5 0 0 0 2

4 0 1 2 1 0

3 1 1 2 0 0

2 0 1 4 1 0

1 2 0 1 0 0

OPP 0 0 0 0 0

1 2 3 4 5Likelihood

Total Risks = 19

High = 5

Medium = 9

Low = 5

Opportunities = 0

Cons

eque

nce

• Cost Consequence Rating (see notes, Alternatives 1,2 &3)

• 5 Critical (23% - 28%)• 4 Serious (15%- 20%)• 3 Moderate (10% - 15%)• 2 Minor (5% - 10%)• 1 Negligible (1% - 5%)• OPP (opportunities) Potential cost

savings (added to matrix)

NOTE: Number of risks in above example are from DAU EVM 201 LAR Risk Assessment

(Note: Percentages from DoD range guidance)

(Note: Percentages from DoD range guidance)

Dominant Likelihood AlgorithmGeneral Process Overview

1). Identify Discrete RisksRisk #1Risk #2Risk #3…

2). Estimate Consequence WBS, Months delay, Phase of Delay WBS % Increase in NRE, REC or both Convert both to $

3). Estimate Likelihood Remote, Unlikely, Likely, Very Likely, Near Certainty

L L

M

L

L

L M

MLL

L

H

M

M

M

M

H H

H

H

H

H

H

H

H

L L

M

L

L

L M

MLL

L

H

M

M

M

M

H H

H

H

H

H

H

H

H

Co

nse

qu

ence

Likelihood1 2 3 4 5

12

34

5N = 1

Max. # RisksLikelihood> Random

# Draw

}Run appropriate

# of iterations

Don’t add consequence

Develop Cumulative Distribution

Add fullconsequence

Yes

No

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60%

Percent Cost Reserve Over Baseline Contract Costs (Without Launch Vehicles)

Pe

rce

nt

Lik

elih

oo

d o

f C

om

ing

in

on

Bu

dg

et (

%)

5x5 Risk DistributionBest CaseMost-Likely ScenarioWorst Case Scenario

NOTE: Random # Draw = > 0 and < 1

25

High:1. Engine2. Cooling3. Exhaust4. Suspension/Steering5. Power Pac/Sub Sys Design

Medium6. Fuel System7. Controls/Instrument8. Subsystem Test9. Armament10. Integration/Assembly

Low11. Frame12. Aux Automotive13. Body Cab14. Communications15. Sys Engineering & PM16. Sys Test & Evaluation17. Training18. Data19. Peculiar Supt Equipment

DAU EVM 201 LAR Risk Assessment

.

.

15

.

9

8

.

4, 5

6, 7

.

17, 18

1411, 1213, 16

19

.

.

2

.

10

.

.

.

.

1, 3

.

1 2 3 4 5

5

4

3

2

1

CONSEQUENCE

LIK

EL

IHO

OD

As of Dec 2003

4,5

MechanicsGeneral Overview

• A basic assumption in using this ACEIT-based ICRM model is that program EVM analysts are not expected to be proficient ACEIT users but can work with their cost estimator counterparts who are proficient

• The ACEIT v7.3a ICRM capability is enabled with the new “Probability of Occurrence” column– This enables the ‘likelihood’ part of the risk register to be a direct

input into ACEIT and function in its Latin Hypercube monte carlo simulations IAW the Dominant Likelihood algorithm

• This particular ICRM model incorporates this new ACEIT capability plus custom ACEIT Dynamic Equation Columns (DECs)– These DECs enable EVM metrics and index information to be used in

calculating an EAC range that is probabilistically derived– These DECs also provide new variables and functions used in ICRM

calculations

MechanicsGeneral Overview (cont)

• The ICRM ACEIT model’s Workscreens enable analysis functions– These analysis functions exist in the interplay between

the Equation/Throughput columns, DECs and within the INPUT VARIABLES sections of each of the Workscreens

– These analysis functions enable the essential calculations for the probabilistic EAC to be produced

• Calculations incorporate both EVM best and worst case EVM Index-derived EACs and risk register likelihoods and cost consequences

Cost Impact Prioritization by WBS

Quantitative Ranking – 70% Tail

ICRM Customization

• Future versions of ICRM can be easily customized – Add or subtract Workscreens to make it more

efficient– Different distributions can be specified for risk

register impacts– Different EVM performance factors can be applied – EVM analysis can be applied to risk register cost

consequences– Cost estimator analysis can be applied to risk register

cost consequences