Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC...

Cost Drivers Learning Event, 2nd November 2005

1

Correlation Tutorial

Raymond Covert, MCR, LLC

([email protected])

Timothy Anderson, The Aerospace Corporation ([email protected])

This tutorial was developed by the authors at:

The Aerospace Corporation

15049 Conference Center Drive,

Suite 600

Chantilly, VA 20151

Copyright © 2004 The Aerospace Corporation

http://www.acoste.org.uk/


16 September 2005 2

Outline

1. Introduction to Correlation in Risk Analysis

2. A Statistical View of Cost Analysis

3. Types of Correlation

4. The Correlation Matrix

5. Deriving Correlation Coefficients



3

Part 1Introduction to Correlation

in Risk AnalysisPurpose of Section - To Answer These 6

Questions About Correlation:1. Who Should Understand It

2. What Is It

3. Why Is It Used

4. Where Is It Used

5. When Is It Used

6. How Is It Used



16 September 2005 4

Who and What

Who Should Understand Correlation ? Correlation should be understood by all cost analysts

performing quantitative cost risk analysis.

What is Correlation?Ref. 1

A measure of association between two variables. It measures how strongly the variables are related, or

change, with each other. If two variables tend to move up or down together, they are

said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated.

The most common statistic for measuring association is the Pearson correlation coefficient, P.

1) www.statlets.com/usermanual/glossary.htm1) www.statlets.com/usermanual/glossary.htm



16 September 2005 5

Correlation in Risk Analysis (2) Why is correlation used?

To quantify the effects of statistical dependence when performing algebra on random variables.

It has a large impact on the statistical properties of the results, particularly when many random variables are involved.

Example: Dice Roll. What happens when we roll 2 dice and add their result? Assume 3 cases:

Case 1: Uncorrelated. Outcome of 1 die is independent from the other.

Case 2: Negatively correlated. Outcome of 1 die relate to the outcome of the other. If one die is a “6”, the other must be “1”.

Case 3: Positively correlated. Outcome of 1 die is same as the other.



16 September 2005 6

Example: Dice Roll Roll of the die gives an equal chance of getting an

outcome (1,2,3,4,5 or 6) Equal, discrete probability Uniform discrete distribution of probabilities Variance, 2 = 3.5

What happens when we sum 2 correlated dice?

Roll of Die

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6

Roll of Die

Pro

bab

ility

Probability of x, P(x) = 1/6Probability of x, P(x) = 1/6



16 September 2005 7

Example: Dice RollSum of Dice: Uncorrelated

0

0.2

0.4

0.6

0.8

1

2 3 4 5 6 7 8 9 10 11 12

Sum of Dice

Pro

bab

ility

Sum of Dice: Correlation =+1

0

0.2

0.4

0.6

0.8

1

2 3 4 5 6 7 8 9 10 11 12

Sum of Dice

Pro

bab

ility

Sum of Dice: Correlation =-1

0

0.2

0.4

0.6

0.8

1

2 3 4 5 6 7 8 9 10 11 12

Sum of Dice

Pro

bab

ility

Case 1: = 0Case 1: = 0

Case 2: = -1Case 2: = -1

Case 2: = +1Case 2: = +1

Triangular, discrete shapeModerate variance, 2=6

Mean = 7

Triangular, discrete shapeModerate variance, 2=6

Mean = 7

P(7) = 1P(<>7)=0

No variance, 2=0Mean = 7

P(7) = 1P(<>7)=0

No variance, 2=0Mean = 7

Uniform, discrete shapeP(each even)=1/6, P(odd) =0

Wide variance, 2=14Mean = 7

Uniform, discrete shapeP(each even)=1/6, P(odd) =0

Wide variance, 2=14Mean = 7

2 + 1 = 32 + 1 = 3

5 + 2 = 75 + 2 = 7

2 + 2 = 42 + 2 = 4



16 September 2005 8

What We Learned From Dice Roll What we learned about the effects of correlation on sums of dice:

It affects the variance and shape It doesn’t affect the mean = 0 changes shape to a discrete triangular distribution =-1 changes shape and removes variance =+1 preserves shape, adds the most variance, and is the same as

multiplying by 2

The sum of dice example used a discretely distributed random variable, but the same rules apply for continuously distributed random variables. Uniform Triangular Normal Lognormal Weibull



16 September 2005 9

Where and When Correlation Is Used

Where is correlation used? When performing algebra on random variables. Quantifying the effects of random variables in cost

estimates. Summing costs of WBS elements. Multiplying costs by random variables (i.e. Inflation). In exponentiation of one random variable with another (i.e.

learning curves).

When is correlation used? Whenever we have random variables in our estimates. When we use Monte Carlo Simulations. In analytic statistical sums.



16 September 2005 10

How Correlation Is Used Directly

Through a correlation matrix,

Indirectly By neglecting correlation, we are defining = 0 By “reusing” random variables, we are defining = 1

Example: We define inflation as a random variable and use the same random variable throughout our cost estimate

By multiplying random variables by a constant, we are defining = 1 Example: We define spacecraft weight as a random variable

and use fractions of it to define weights of different subsystems.

0.11.02.0

1.00.11.0

2.01.00.1




Part 1 Summary All cost analysts performing quantitative cost risk

analysis should understand correlation Correlation measures how strongly the variables are

related, or change, with each other. Correlation affects the variance and shape, but not

the mean We use correlation frequently, but may not even

know it



12

Part 2A Statistical View of Cost Analysis

Purpose of Section: To Understand the following1. Costs are uncertain quantities

2. Costs can be treated as random variables

3. How correlation affects variance




A Statistical View of Cost Analysis

WBS Element Costs are Uncertain Quantities That Have “Probability” Distributions And Statistical Characteristics such as Mean, Median, Mode Costs are Random Variables

Our Goal in Cost Risk Analysis is to Combine Element Cost Distributions to Generate Probability Distribution of Total Cost Use Monte-Carlo or Other Statistical Procedure Quantify Confidence in “Best” Estimate of Total Cost, e.g., Mode Read off Mean, 50th Percentile Cost, 70th Percentile Cost, etc.,

from Cumulative Distribution to Estimate Amount of Risk Dollars Needed




Elements of Risk Model

Cost DriversCost Drivers

Risk DriversRisk Drivers

AssumptionsAssumptions

Cost EstimateModel

Cost EstimateModel

Quantified RiskQuantified Risk

Our Cost / Risk Model Quantifies:Costs

Effects of UncertaintyUncertainty in Program Assumptions

Risks to Program

Our Cost / Risk Model Quantifies:Costs

Effects of UncertaintyUncertainty in Program Assumptions

Risks to Program




Example Cost Drivers

Component, Assembly, Propellant Weights Cooling Requirements Data-Processing Requirements Power Requirements Solar Array Area Orbit Altitude Thrust Requirements Special Mission Equipment




Example Risk Drivers Beyond-State-of-the-Art

Technology Cooling Processing Survivability Power Laser Communications

Unusual Production Requirements Large Quantities (Space

Systems) Toxic Materials Yields

Tight Schedules Undeveloped Technology Software Development Supplier Viability

System Integration Multi-contractor Teams System Testing

Limited Resources Program Funding Stretch-

Out Premature Commitment to

RDT&E Phase Unforeseen Events




Cost-element Probability Distributions

Best Estimate

Low Risk

High Risk

Low Cost, High Risk vs.

High Cost, Low Risk

Best Estimate

Narrow Symmetric distribution: equal

Probability of actual cost higher or lower than best estimate

Narrow Symmetric distribution: equal

Probability of actual cost higher or lower than best estimate

Wide, Right Skewed distributionLower point

estimate, but high probability of actual

cost greater than point estimate

Wide, Right Skewed distributionLower point

estimate, but high probability of actual

cost greater than point estimate

These curves tell two very

different stories

These curves tell two very

different stories

Would you believe both could come

from the same estimate?

Would you believe both could come

from the same estimate?

mode = mean

mode = mean

meanmean

modemode




Correlation Affects the Variance are Costs of WBS Elements (Random

Variables) and n = number of WBS elements

Total Cost =

Mean of Total Cost =

Variance of Total Cost =

=

A Very Important RelationshipA Very Important Relationship




Variance Measures Dispersion Small

Large

X

Area = 1.00

X

Area = 1.00




Does Correlation Matter? If WBS-Element Costs are Uncorrelated (all ij = 0),


If WBS-Element Costs are Correlated,

Variance of Total Cost =– Positive Correlations Increase Dispersion

– Negative Correlations Reduce Dispersion

If (“Worst” Case) All Correlations,


“Ignoring” Correlation Issue is Tantamount to Setting all ij = 0




Yes, Correlation Matters Suppose for Simplicity

There are n Cost Elements

Each

Each

Total Cost

C C Cn1 2, , ,

Var C i 2

Corr C Ci j, 1

Var C Var C Var C Var Cik

n

i

n

j i

n

i j

1 1

1

12

n n n 2 21

n n 2 1 1

Correlation 0 1

VarC n2 n n 21 1 n22




Magnitude of Correlation Impact

Percent Underestimation of Total-Cost Sigma When Correlation Assumed to be 0 instead of is 100% times ...

Percent Overestimation of Total-Cost Sigma When Correlation Assumed to be 1 instead of is 100% times ...

Var C

Var C

n n n

n n

n

n

1 1

1 1 1 11

n n n

n n n

1 1

1 11

1

1 1




Maximum Possible Underestimation of Total-cost Sigma

Percent Underestimated * 100% When

Correlation Assumed to be 0 Instead of

1

1

1 1n

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Per

cent U

nder

estim

ated

n = 10

n = 30

n = 100n = 1000

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Per

cent U

nder

estim

ated

n = 10

n = 30

n = 100n = 1000




Maximum Possible Overestimation of Total-Cost Sigma

Percent Overestimated When Correlation Assumed to be 1 Instead of

0

50

100

150

200

250

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Pe

rce

nt

Ove

res

tim

ate

d

Limit as n

n =10

0

50

100

150

200

250

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Pe

rce

nt

Ove

res

tim

ate

d

Limit as n

n =10




Part 2 Summary

In this section we learned: Costs are uncertain quantities Costs can be treated as random variables Correlation affects variance

Especially when we are summing large numbers of WBS elements

Remember that in the total cost distribution: The means add The standard deviation is the square root of the variance

The variance =



26

Part 3Types of Correlation

Purpose of Section is to learn about:1. Functional and Statistical Correlation

2. Pearson and Spearman Correlation




Types of Correlation

Functional (Causal) Correlation Between cost drivers (Cost Engineering Tools) Between CERs (Cost dependent CERs – SEITPM)

Statistical Correlation Between CER errors

Residual analysis (USCM,SSCM, NAFCOM) Retro-ICE method Estimated based on Number of WBS items

Between Engineering drivers Between complexity, weight, power, etc.




Functional Correlation Between Cost Drivers

Cost drivers are functionally correlated Beginning of Life Power, EPS Weight, Solar Array Weight,

Battery Weight, RCS Weight, etc…

Use Sizing equations from cost engineering tools to examine the causal relationship

Two Good Examples: Electrical Power System Sizing Reaction Control System Sizing




Functional (Causal) CorrelationBetween Cost Drivers

POWERREQUIREMENT

LOAD CONTROL

UNIT POWER /VOLTAGEOUTPUT

PRIMARY BUSVOLTAGE

POWERLEVELS

POWERDISTRIBUTION

POWER /VOLTAGE LOSS

PRIMARY POWERSOURCE

REQUIREMENT

EOL SOLARARRAY POWERREQUIREMENT

ORBITAL DATA

EOL SOLARPOWER OUTPUT

BATTERYISOLATION

DIODEPOWER LOSS

SOLAR ARRAYPARASITIC

LOSSES

BATTERYDISCHARGEVOLTAGE /

POWER

BOL SOLARARRAY POWEROUTPUT

SOLAR ARRAYPOWER LOSS

FACTOR

SOLAR ARRAY SHUNT

DISSIPATIONREQUIREMENTS

THEORETICALBATTERY PEAK

DISCHARGEENERGY REQT.

MISSION CYCLES

DEPTH OFDISCHARGE

REQUIREDBATTERY

DISCHARGECAPACITY

NUMBER OFBATTERIES IN

PARALLEL

BATTERYCHARGER I/OVOLTAGE AND

OUTPUTPOWER

POWERSYSTEMSPECIFICWEIGHT /POWER

ACTUAL BATTERYDEPTH OF

DISCHARGE

Example: Electrical Power System SizingExample: Electrical Power System Sizing

Equations in Design loop are functional correlationsEquations in Design loop are functional correlations

Power Requirement drives EPS WeightPower Requirement drives EPS Weight




Functional CorrelationBetween Cost Drivers

Feedback Example: Reaction Control System Feedback Example: Reaction Control System

Reaction Control SelectionNatural Disturbance Torques SV

Dimensions

Solar Array Pointing ErrorSlew Rate

Antenna Pointing ErrorSlew Rate

Inertia:SV BodyAntennasSolar Array

Reaction Torque

RCS Size

RCS Power

RCS Weight

GimbalTorques

RCS Weight and size drives SV Body InertiaRCS Weight and size drives SV Body Inertia

RCS Power drives Solar Array InertiaRCS Power drives Solar Array Inertia

RCS Size drives SV DimensionsRCS Size drives SV Dimensions

Equations in Design loop are functional correlationsEquations in Design loop are functional correlations




There are two basic types of Cost Estimating Relationships (CERs).

1. Design Parameter Dependent: Subsystem Hardware (HW) CERs which use weight (or other design parameter) as a base.

2. Cost Dependent: Systems Engineering, Integration and Test, and Program Management (SEITPM) CERs which use estimated cost as a base.

CERs are Used Serially in Risk Analysis.1. Subsystem HW estimated costs are driven by subsystem

estimated weights (or other cost drivers).2. SEITPM estimated costs are driven by HW cost estimates.3. So, the variance of the SEITPM cost estimate is functionally

correlated to the HW cost estimates.

Functional Correlation Between CERs




Mathematically Total estimate is a sum of the subsystem and SEITPM

costs:*

Subsystem estimates follow the form:

SEITPM estimate follows the form:

Est

N

iiEstEst SEITPM$SS$TOTAL$

1,

ib

iiiEstiWTaSS$ )(,

N

ii

biiS

bEst

iWTaxaxSEITPM$1

)(where

*Note: The “Total cost” is actually a sum of the subsystem and SEITPM costs. It is not the sum of all costs associated with a spacecraft.

Error terms for SS estimates Error terms for SS estimates

Error term for SEITPM

Error term for SEITPM

Weight Weight




Mathematically

So, the SEITPM estimate is actually represented by:

And the total estimate is represented by:

S

bN

ii

biiEst

iWTaaSEITPM$

1

)(

S

bN

ii

bii

N

ii

biiEst

ii WTaaWTaTOTAL$

11

)()(

Functional correlation Functional correlation

SS CERs SS CERs SEITPM SEITPM




Statistical Correlation Statistical Correlation

Between CER errors. Residual analysis (USCM,SSCM, NAFCOM). Retro-ICE method. Estimated based on Number of WBS items.

Between Engineering drivers. Between complexity, weight, power, etc.

Look at 2 types of statistical correlation Pearson’s correlation Spearman’s correlation

When you have dataWhen you have data

When you have to guess

When you have to guess

When you have data but don’t

know functionalrelationships

When you have data but don’t

know functionalrelationships




Two Types of Statistical Correlation Pearson Product-Moment Linear Correlation

if and only if X and Y are linearly related, i.e., the least-squares linear relationship between X and Y allows us to predict Y precisely, given X

= proportion of variation in Y that can be explained on the basis of a least-squares linear relationship between X and Y

if and only if the least-squares linear relationship between X and Y provides no ability to predict Y, given X

Spearman Rank Correlation if and only if the largest value of X corresponds to the

largest value of Y , the second largest, ... , etc.

if and only if the largest value of X corresponds to the smallest value of Y, etc.

if and only if the rank of a particular X among all X values provides no ability to predict the rank of the corresponding Y among all Y values




Pearson “Product-Moment” Correlation Suppose X and Y are Two Random Variables

are their Expected Values (“Means”)

True Theorem:

False Theorem:

“Covariance” of X and Y

“Variance” of X

“Variance” of Y

“Correlation” of X and Y =

X YE X E Y ,

22, XEXEXXCovXVar

E X Y E X E Y X Y

E XY E X E Y

Cov X Y E XY E X E Y,

22, YEYEYYCovYVar

Corr X YCov X Y

Var X Var Y,

,

Cov X Y XY X Y, Var X Var YX Y 2 2, ,

XY




Pearson Correlation Measures Linearity A Statistical Relationship Between Two Random

Variables X and Y

Realizations of Y, Given Actual Values of X:

X

Y

0X

Y

0




Spearman Rank Correlation Coefficient Data Structure

Statistics Theorem: Spearman Rank Correlation Coefficient Equals Pearson (Linear) Correlation Coefficient Calculated

Between the Two Sets of Ranks

CASE

RANK OF Xi VALUE Xj VALUE

DIFFERENCE

SQUARED DIFFERENCE

#1 r1 c1 d1 = c1 - r1 d12

#2 r2 c2 d2 = c2 - r2 d22

#3 r3 c3 d3 = c3 - r3 d32

.

.

.

.

.

.

.

.

.

. . . . . .

.

.

.

#4 rn cn dn = cn - rn dn2

SUMS nn( )1

2 nn( )1

2 d c r 0 d 2

ss




Linear vs. Rank Correlation

0 0

-0.4

.

.s

076

0..13

.

.s

LINEARLINEAR POWERPOWER “KNEE”“KNEE”

ROOTROOT DECAY w/ OUTLIERDECAY w/ OUTLIER RANDOM w/ OUTLIERRANDOM w/ OUTLIER

More NonlinearMore Nonlinear

Linear Data gives similar and sLinear Data gives similar and s




What Does Correlation Measure?

PEARSON Correlation Measures Extent of LINEARITY of a Relationship Between Two Random Variables

SPEARMAN Correlation Measures Extent of MONOTONICITY of a Relationship Between Two Random Variables

A way to remember:

…L M N O P Q R S…

A way to remember:

…L M N O P Q R S…

Rank

Is Spearman

Rank

Is SpearmanLinear Is PearsonLinear Is Pearson




Part 3 Summary

Two types of correlation Functional and Statistical Correlation Functional correlation affects Cost Drivers and Cost

Dependent CERs Statistical correlation is an observed relationship between

data

Two types of Correlation Statistics Pearson and Spearman Correlation or Linear and Rank Correlation They are similar when the data is linear They are different when data is not linear



42

Part 4The Correlation Matrix

Purpose of Section is to learn :1. Correlation in Risk Rollups

2. Anatomy of a Correlation Matrix

3. How to use a Correlation Matrix

4. Which Common Cost Models Handle Correlation




Cost-Risk Procedure

Use analytic or Statistical sampling methods to arrive at a total cost distribution

WBS-ELEMENTTRIANGULAR DISTRIBUTIONS

MERGE WBS-ELEMENT COST DISTRIBUTIONSINTO TOTAL-COST DISTRIBUTION

BEST ESTIMATE COST

(MOST LIKELY)

70th PERCENTILE

COST

RISKDOLLAR

S

$

Note: Addition of risk dollars brings confidence that total appropriation (best estimate plus risk dollars) is sufficient to fund program.

L1 B1H1 $

L2 = B2 H2 $

L3 B3 H3 $




Total Cost Variance

Remember from Part 1, the Total cost variance,

T Correlation matrix (full matrix)

Vector of standard deviations (cost space)

Excel Commands SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA))

kj

k

jjk

n

k

n

kkTotal

1

121

22 2

Correlation is Essential in calculating variance!




Representing Correlation Matrices

Full Matrix (Have to use this when you use analytic function: )

Upper Triangular:

Lower Triangular:

1 0.2 0.14 0.37 0.20.2 1 0.06 0.15 0.12

0.14 0.06 1 -0.2 0.060.37 0.15 -0.2 1 0.15

0.2 0.12 0.06 0.15 1

1 0.2 0.14 0.37 0.21 0.06 0.15 0.12

1 -0.2 0.061 0.15

1

10.2 1

0.14 0.06 10.37 0.15 -0.2 1

0.2 0.12 0.06 0.15 1

All 3 representations mean the same thingAll 3 representations mean the same thing

TT




Representing Correlation Matrices

Single value shorthand:

This means all of the off diagonal terms are the same value

The Rules: Always positive definite Diagonal terms always 1.0 Off diagonal terms are correlation values Columns and rows are transposed, j,k = k,j

Now for some practical examples

11

11

1




Cost Risk Rollup ProcedureWBS Items Segment

(Launch, Space, Ground)

Want to form a Distribution for

Total Cost of Program

Spacecraft Bus Subsystems

Payload Subsystems

Payload SEITPM(function of payload subsystem cost)

System(Spacecraft Bus, Payload)

Subsystem 1

Subsystem 2

Subsystem N

f($)

Subsystem 1

Subsystem 2

Subsystem N

f($)

Launch

Other Elements

Spacecraft Bus SEITPM(function of spacecraft bus subsystem cost)

Inter-element Correlation needed for All Rollup (Summed)

Costs

Inter-element Correlation needed for All Rollup (Summed)

Costs




Spacecraft Bus: USCM7 Correlation Coefficients

Correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = 0.160

AD

CS

NR

AG

EN

R

CO

MM

NR

EP

SN

R

IAT

NR

PR

OG

NR

ST

RC

NR

TH

ER

NR

TT

CN

R

AD

CS

T1

AK

MT

1

CO

MM

T1

EP

ST

1

IAT

T1

LO

OS

T1

PR

OG

T1

ST

RC

T1

TH

ER

T1

TT

CT

1

ADCSNR 1.000 -0.067 -0.096 -0.035 0.035 0.012 0.413 0.605 0.121 -0.095 0.983 -0.122 0.099 0.564 0.139 0.089 -0.047 -0.057 0.092AGENR 1.000 -0.028 0.525 -0.079 0.127 0.091 -0.230 -0.125 0.416 0.001 0.085 -0.043 -0.163 -0.189 0.033 0.146 0.151 0.232COMMNR 1.000 0.888 0.884 0.966 0.762 0.281 0.850 -0.166 0.305 -0.176 0.157 0.368 0.884 -0.158 0.109 0.037 -0.004EPSNR 1.000 0.265 0.604 0.409 0.003 0.337 0.237 0.011 -0.275 0.076 0.342 0.021 -0.049 0.465 0.123 0.035IATNR 1.000 0.721 0.615 0.331 0.747 -0.037 0.391 -0.133 -0.028 0.501 0.265 -0.145 0.113 -0.014 -0.189PROGNR 1.000 0.697 0.222 0.868 -0.065 0.145 -0.191 -0.044 0.444 0.329 -0.191 -0.000 -0.125 0.019STRCNR 1.000 0.837 0.761 -0.001 0.117 -0.214 -0.113 0.418 0.173 -0.018 0.220 -0.103 0.069THERNR 1.000 0.077 -0.200 0.662 -0.171 -0.053 0.514 0.102 -0.010 -0.063 -0.165 0.092TT CNR 1.000 -0.149 0.475 -0.118 -0.071 0.519 0.294 -0.178 -0.111 -0.095 0.022ADCST1 1.000 -0.100 0.614 0.421 -0.262 -0.354 0.543 0.676 -0.029 0.655AKMT1 1.000 -0.006 0.292 0.855 0.286 0.176 -0.003 -0.027 0.052COMMT1 1.000 0.266 -0.454 -0.088 0.777 0.729 0.126 0.391EPST1 1.000 -0.150 -0.145 0.381 0.388 -0.007 0.520IATT1 1.000 0.448 -0.144 -0.224 -0.014 -0.320LOOST1 1.000 -0.336 -0.097 -0.074 -0.169PROGT1 1.000 0.421 -0.039 0.481STRCT1 1.000 -0.175 0.285THERT1 1.000 -0.140TT CT1 1.000

These correlation coefficients should not be used for all spacecraft cost models




The “Big” WBS Case Study (ISS Risk Estimate) Suppose a risk analyst diligently applies distributions to all costs at

the “level of estimating” – this is good. Assume that:

There are 300 cost elements (N=300) There are about four cost elements in each subsystem (n=4) There are (N/n = 75 subsystems) Correlation is defined between all elements within a subsystem

This means: That

WBS elements are correlated Only about 1% of the cost elements are correlated Risk is very narrow and understated

The correlation appears “just-off-the diagonal” of the correlation matrix – This is bad.

010033.0

89700

900

299*300

3*4)4/300(

)1(*

)1(*)/(

NN

nnnN




“Just-Off-Diagonal” Correlation

Some tools cannot support this function Some nominal statistical correlation does exist Even a few percent makes a big difference with a big WBS

1 0.5 0.51 0.5

1 0.5 0.5 0.51 0.5 0.5

1 0.51 0.5 0.5 0.5 0.5

1 0.5 0.5 0.51 0.5 0.5

1 0.51 0.5 0.5 0.5 0.5

1 0.5 0.5 0.51 0.5 0.5

1 0.51 0.5 0.5 0.5

1 0.5 0.51 0.5

1 0.5 0.5 0.51 0.5 0.5

1 0.51

These Inter-Subsystem WBS Elements are Effectively Uncorrelated

These Intra-Subsystem WBS Elements are Correlated




How to Use Correlation Matrices

Typically, we wouldn’t want to define all of the correlation coefficients for a big WBS (>10 elements)

We can break it up into parts, get the statistics and then sum at higher levels Reduces the size of correlation matrices Provides Risk Breakout by WBS Summary Level

Lets use an example of a “Big” WBS with 40 elements




Yuck 40 Individual WBS Elements and the correlation

matrixSEITPM

Systems EngineeringIntegration & TestProgram ManagementConfiguration ManagementData

SpaceSpace Vehicle SEITPMSpace Vehicle

Spacecarft BusBus Systems EngineeringBus I&TBus PMBus DataStructures & MechanismsThermal ControlAttitude Determination & ControlTTC / C&DHPropulsionElecrical PowerLOOSAGE

PayloadPL SEITPMOptical TelescopePanchomatic SensorMultispectral SensorSpectrometerMagnetometerGravitometerUV Sensor

GroundGround SEITPMGround TerminalMission PlanningSatellite OPS / ControlData Archive and Dissemination

LaunchLaunch VehicleLaunch Systems IntegrationLaunch Vehicle IntegrationOn-Orbit CheckoutLaunch Vehicle SE

OperationsSEITPMMaintenanceMission PlanningMission OpsData Archive and Dissemination

1 0.2 0.4 0.3 0.3 0.1 0.1 0.3 0.2 0.1 0.1 0.1 0 0.2 0.3 0.1 0 0.2 0.4 0.5 0.5 0.1 0.1 0.4 0.4 0.1 0.1 0.4 0.2 0 0.1 0.3 0.2 0.3 0.3 0.4 0.3 0.21 0.1 0.2 0.3 0 0.2 0.4 0.2 0.4 0.3 0.3 0.2 0.5 0.1 0.1 0.1 0.3 0.5 0.3 0 0.4 0.1 0.1 0.4 0.1 0.1 0.5 0 0.3 0.1 0.1 0.4 0.2 0.5 0.2 0.5 0

1 0.3 0.5 0.3 0.3 0.2 0.1 0.4 0.3 0.2 0.5 0 0.1 0.5 0.4 0.4 0.4 0.4 0.5 0.3 0.4 0.5 0.2 0 0 0.4 0.1 0.3 0.5 0.3 0 0.1 0.2 0.4 0.5 0.41 0.5 0.1 0.3 0.5 0.2 0.1 0.1 0.3 0.1 0.2 0.5 0.3 0.5 0.3 0.3 0.3 0.2 0.5 0.3 0.1 0.4 0.3 0.1 0.5 0.4 0.4 0.5 0.4 0.4 0.4 0.2 0.4 0.5 0.2

1 0.2 0.4 0.1 0 0.3 0.3 0.4 0 0.2 0.4 0.5 0.3 0.2 0.1 0.2 0.5 0.2 0.4 0.5 0.2 0.2 0.4 0.4 0.4 0.5 0.3 0.4 0.5 0 0.2 0.3 0.3 0.21 0.1 0.2 0 0.1 0.2 0.3 0 0.5 0.2 0.1 0.3 0.1 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0.4 0.1 0.1 0.2 0.4 0.4 0.2 0 0.5 0.5 0 0.1 0.1

1 0.3 0.2 0.3 0.3 0.1 0.1 0.1 0.4 0 0.1 0.3 0.4 0.2 0.4 0.4 0.2 0.1 0.1 0.3 0 0.4 0.3 0.5 0.1 0.4 0.5 0.5 0.1 0.1 0.2 0.41 0.4 0.4 0.2 0.2 0.5 0.5 0 0.4 0 0.2 0.2 0.3 0.1 0.4 0.2 0 0.1 0.4 0.1 0.1 0.1 0 0.1 0.4 0.4 0 0.3 0.5 0 0.3

1 0.1 0 0.1 0 0.2 0.5 0.4 0.3 0.2 0 0.5 0 0.4 0.3 0.5 0.1 0.1 0 0.4 0.2 0.1 0.4 0.2 0.1 0.1 0.1 0.4 0.3 0.11 0.4 0 0.2 0.5 0.1 0.3 0.5 0.4 0.3 0.1 0.1 0.5 0.4 0 0.5 0.1 0.5 0.1 0.2 0.1 0.2 0.1 0.1 0.2 0.3 0.5 0.1 0.2

1 0.4 0.3 0.3 0.1 0.1 0.1 0.3 0.5 0.3 0.1 0.5 0.1 0.3 0.4 0.2 0.2 0.2 0.5 0.1 0.5 0.2 0.3 0 0.1 0.1 0.2 0.51 0.2 0.1 0 0.1 0.3 0 0.3 0.3 0.4 0.2 0.5 0.1 0 0 0.2 0.2 0.1 0.4 0.3 0 0.3 0.2 0.3 0.4 0.3 0.3

1 0.2 0.1 0.1 0.2 0.4 0.5 0.4 0 0.3 0.5 0.2 0.2 0.3 0 0.4 0.1 0.1 0.2 0.1 0.2 0.4 0.3 0.2 0.1 0.11 0.2 0.1 0.2 0.3 0.1 0.5 0.2 0.4 0.4 0.5 0.5 0.4 0.4 0.2 0.3 0.2 0.4 0.3 0.1 0 0.1 0.5 0.3 0.3

1 0.2 0.1 0.1 0.2 0.4 0.2 0.2 0.3 0.5 0.4 0.2 0.4 0.4 0 0.3 0.4 0.4 0.2 0.3 0 0.4 0.2 0.31 0.3 0.1 0 0 0.2 0.2 0.3 0.4 0 0.1 0.4 0.1 0.3 0.1 0.4 0.1 0.3 0.2 0.2 0.1 0.4 0.1

1 0.3 0.4 0.3 0.4 0.1 0.4 0 0.4 0.1 0.4 0.1 0.4 0.3 0.5 0.2 0.2 0.3 0.4 0.5 0.1 0.41 0.3 0.3 0.2 0 0.3 0.5 0.4 0.4 0.4 0.4 0.2 0.4 0.2 0.1 0.3 0.3 0.5 0.4 0.5 0.4

1 0 0.5 0.3 0 0.3 0.3 0.2 0.5 0.4 0.1 0.3 0.4 0.4 0.4 0.5 0 0.2 0.5 0.31 0.3 0.1 0.5 0.3 0.1 0.2 0.3 0.1 0.1 0.4 0.4 0.2 0.1 0.1 0.3 0.1 0.2 0.1

1 0.1 0.1 0.1 0.2 0.4 0.3 0.5 0.4 0.1 0 0.5 0.2 0.4 0.1 0.2 0.5 0.31 0.3 0.3 0.2 0.5 0.3 0.1 0.4 0.3 0.3 0.2 0.2 0.3 0 0.5 0.3 0.4

1 0.1 0.1 0.5 0.3 0.5 0.4 0.1 0.2 0.3 0.2 0.5 0 0.3 0.1 0.41 0.2 0.1 0.4 0.3 0.3 0.3 0.1 0.4 0.2 0.2 0.2 0.2 0.5 0.3

1 0.5 0.5 0.2 0.3 0.1 0.3 0.4 0.3 0.4 0.3 0.3 0.2 0.31 0.3 0.2 0.5 0.3 0.5 0.1 0.4 0.3 0.1 0.3 0.1 0.2

1 0.5 0.3 0.1 0.1 0.2 0.1 0.4 0.2 0.2 0.5 0.21 0.2 0.4 0.5 0.4 0.1 0 0.5 0.3 0.3 0.1

1 0.1 0.2 0.3 0.1 0 0.4 0.2 0.3 0.51 0.5 0.2 0.2 0.3 0.1 0.4 0.4 0.4

1 0.1 0.1 0.3 0.2 0 0 0.41 0.2 0.2 0.4 0.2 0.1 0.4

1 0.4 0.1 0.2 0.3 0.21 0.4 0.3 0.3 0.1

1 0.1 0 0.21 0 0.4

1 0.21

Imagine 300 WBS Elements!Imagine 300 WBS Elements!




Big Correlation Matrix LayoutAA

BA

CA

DA

EA

AB

BB

CB

DB

EB

AC

BC

CC

DC

EC

AD

BD

CD

DD

ED

AE

BE

CE

DE

EE

A= SETPM (5 elements)

B= Space Segment(20 elements)

C= Ground Segment(5 elements)

D= Launch Segment(5) elements)

E= Operations Segment(5 elements)

Each Block Represents a group of inter element correlationsThe full matrix requires (40*39)/2=780 different correlations

Each Block Represents a group of inter element correlationsThe full matrix requires (40*39)/2=780 different correlations




Multilevel Risk

Look at the problem one small set of pieces at a time

Now do the same for SPACE, LAUNCH, GROUND, O&M

Mean SigmaSEITPM 11.22 2.70

Systems Engineering 1.2 0.24Integration & Test 1.8 0.72Program Management 0.9 0.27Configuration Management 7.2 2.16Data 0.12 0.048

SEITPM Mean = Sum(Means)SEITPM Sigma =SQRT(MMULT(MMULT(TRANSPOSE(sigma),correl_matrix),sigma))

SEITPM Mean = Sum(Means)SEITPM Sigma =SQRT(MMULT(MMULT(TRANSPOSE(sigma),correl_matrix),sigma))

1 0.3 0.3 0.3 0.30.3 1 0.3 0.3 0.30.3 0.3 1 0.3 0.30.3 0.3 0.3 1 0.30.3 0.3 0.3 0.3 1

SEITPMSEITPM




Space Element Risk In the Space Element, first break-out the Bus calculation

Space Vehicle SEITPM is one line item,

Let’s assume we already calculated mean and sigma for the Payload, like we did for the Bus

Mean SigmaSpacecarft Bus 40 7.48

Bus Systems Engineering 3.2 1.12Bus I&T 3.3 1.32Bus PM 1 0.25Bus Data 0.5 0.2Structures & Mechanisms 1 0.3Thermal Control 1 0.3Attitude Determination & Control 8 2.4TTC / C&DH 10 3Propulsion 6 1.8Elecrical Power 3 0.9LOOS 2 0.6AGE 1 0.3

1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1

S/C BusS/C Bus

Mean SigmaPayload 83.0 17.74

Mean SigmaSpace Vehicle SEITPM 43.1 19.38




Space Element Risk

Now we roll-up 3 Items:

Use a small correlation matrix:

The result is:

Mean SigmaSpace Vehicle SEITPM 43.1 19.38Spacecarft Bus 40.0 7.48Payload 83.0 17.74

1 0.4 0.40.4 1 0.40.4 0.4 1

Mean SigmaSPACE 166.1 35.26

Cost

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0 100 200 300 400

PD

F




Economics of Multi-Level Risk After summing all of the elements, we used :

144 Correlation Coefficients vs. 780 (one big matrix)

Views of Risk at all roll-up levels Easier to obtain values for correlation coefficients

We will discuss this in the next part

# Elements # RhosTotal System 5 10SEITPM 5 10Space Element 0 0

12 668 28

Ground 5 10Launch 5 10Ops 5 10

Number of Rhos 144




What We Just DidAA

BA

CA

DA

EA

AB

BB

CB

DB

EB

AC

BC

CC

DC

EC

AD

BD

CD

DD

ED

AE

BE

CE

DE

EE

A= SETPM (5 elements)

B= Space Segment(20 elements)

C= Ground Segment(5 elements)

D= Launch Segment(5) elements)

E= Operations Segment(5 elements)

Relied on AA, BB, CC, DD, and EE correlationRelied on AA, BB, CC, DD, and EE correlation




MathematicallyStep1: Calculate

; Where , and

Step 2: Need correlation coefficients of partition AA, BB and all s to

calculate

BA ,

5

4

3

2

1

B

A

2

1

A

5

4

3

B

BA ,

5445

53354334

522542243223

5115411431132112

25

24

23

22

21

2 2

Tot




Mathematically

Step 3: Calculate total variance using

This is useful when :• We know the correlation between subsystem elements • But not the correlation between subsystems from different elements to

each other (i.e., thermal control SS in spacecraft to ground Command and control CSCIs)

• But do have an idea of correlation between higher-level elements like space to ground.

ABBAABBATot 2222

522542243223511541143113 BAAB

54455335433425

24

232112

22

21

522542243223511541143113

22

AB




Mathematically

11

21

31

41

51

12

22

32

42

52

13

23

33

43

53

14

24

34

44

54

15

25

35

45

55

522542243223511541143113 BAAB




Models That Handle Correlation

PRICE*

NAFCOM*

SSCM*

USCM + OthersCannot

Full

Partial

SimulationAnalyticDiscrete

FRISK

@RISK and CB

Monte Carlo Simulators Cost Model w/ Risk*

SEER*

RI$K

SICM

With Analytic or Simulation Method & correlationStudy

Type of Risk Analysis Method

Lev

el o

f C

orr

elat

ion

Sp

ecif

icat

ion

© 2003 The Aerospace Corporation




Part 4 Summary

Showed how correlation is used in Risk Rollups Provided the anatomy of a correlation matrix

1’s on diagonals Correlation coefficients on off-diagonals Rows and columns are transposes of each other

How to use a correlation matrix Breaking down big risk jobs into smaller pieces Easier to understand Easier to correlate

Showed which common cost models handle correlation



64

Part 5Deriving

Correlation CoefficientsPurpose of Section is to learn how to derive correlation

coefficients:1. When data is available

2. When you have to make an educated guess




Deriving Correlation Coefficients 2 Ways to derive correlation coefficients

Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)

No Data:Educated Guess


ResidualAnalysisResidualAnalysis

Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess

StatisticalStatistical Non-StatisticalNon-Statistical

Effective

Effective

Knee in curve(Steve Book Method)




66

Determining Correlation When Data is Available





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Determining Correlation When Data is Available Statistical Correlation

Residual analysis (USCM,SSCM, NAFCOM) (HARD) Need Database of cost and cost drivers+ CERs+ CER errors Only have good bus correlations with standard models

Retro-ICE method (HARD) Need actual cost data from several similar programs + similar WBS

structure+ total error+ similar models Estimated based on Number of WBS items (EASY)

Need number of WBS items + typical uncertainty Strong function of number of correlated elements Decreases with number of correlated elements

Functional (Causal) Correlation Between cost drivers (HARD)

Need Cost Engineering Tools (CDC, SWAP model) Between CERs (EASY)

Use cost dependent CERs (SEITPM, etc) linked to summary costs in model




68

Residual Analysis Method





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Statistical Correlation From Residual Analysis

Percentage error or standard error are a measure of residual errors

Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals

Cost vs. Weight

0

500

1000

1500

2000

2500

3000

0 20 40 60 80 100

Weight (lbs)

Co

st (

$K)




Deriving Correlation Coefficients Sample calculation using randomly generated

numbers Error Xi and Error Yi represent regression residuals for 2

CERs (X and Y) for 8 programs

22mimi

mimi

jk

yyxx

yyxx

PROGRAM Error, Xi Error, Yi (Xi-Xm) (Yi-Ym) (Xi-Xm)(Yi-Ym) (Xi-Xm)^2 (Yi-Ym)^21 0.5404 0.4224 0.1102 0.0167 0.0018 0.0121 0.00032 0.4943 0.3719 0.0641 -0.0339 -0.0022 0.0041 0.00113 0.4496 0.4340 0.0194 0.0282 0.0005 0.0004 0.00084 0.0088 0.2598 -0.4214 -0.1460 0.0615 0.1776 0.02135 0.5679 0.4291 0.1377 0.0234 0.0032 0.0190 0.00056 0.4486 0.5126 0.0184 0.1069 0.0020 0.0003 0.01147 0.7960 0.5357 0.3659 0.1300 0.0475 0.1339 0.01698 0.1359 0.2804 -0.2943 -0.1253 0.0369 0.0866 0.0157

SUM 0.1513 0.4340 0.0681MEAN 0.4302 0.4057RHO 0.8804 = 0.151 / SQRT(0.434 * 0.068)



71

Retro-ICE Method





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Retro-ICE Method

Another way to look at correlation is the effect on the variance of the total vs. the variance of the components

Example: SPACE Segment Contains 3 elements: Space SEITPM, Spacecraft Bus,

Payload

What you will need: Actuals for some programs (8 or so) Estimates using your “new” cost model or method (You will

be doing a Retro-ICE)

Use the equation for Pearson correlation:

22mimi

mimi

jk

yyxx

yyxx




Retro-ICE Example

Start with a table of actuals and estimates for 8 programs and the 3 WBS elements you wish to determine correlation

8 Programs8 Programs

Actual CostsActual Costs Re-Estimated Costs(From Retro-ICEs)

Re-Estimated Costs(From Retro-ICEs)

Program SV SEITPM Bus Payload SV SEITPMBus PayloadA 33 55 30 28 50 28B 42 40 80 40 30 42C 40 60 45 45 68 47D 28 35 33 28 40 30E 24 42 22 20 35 20F 72 80 100 50 75 45G 16 20 23 8 15 18H 28 60 14 18 50 13

Actuals Estimates




Retro-ICE Example Calculate the Residuals (Actuals – Estimate)

For Each WBS Element (and each program) For the Total

Find the standard deviation, , of each set of residuals For Each WBS Element (8.05, 6.63, 21.26) For the Total (30.53)

Program SV SEITPM Bus Payload TotalA 5 5 2 12B 2 10 38 50C -5 -8 -2 -15D 0 -5 3 -2E 4 7 2 13F 22 5 55 82G 8 5 5 18H 10 10 1 21Stdev 8.048957342 6.631903 21.26029 30.53306

Estimating Error




Retro-ICE Example Construct the correlation matrix

Ones on the diagonal Use Excel “CORREL” function on the off-diagonals Remember the row/column transpose rule

Check your work:

The result should match the standard deviation of the total, 30.53

1 0.525211 0.6394730.525210993 1 0.342461

0.63947252 0.342461 1

8.05 6.63 21.26 1 0.52521 0.63947 8.05* 0.52521 1 0.34246 * 6.63 = 30.53

0.63947 0.34246 1 21.26( )^2( )^2



76

Effective Correlation





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Effective Correlation Effective correlation is different from average

correlation, or the average of the correlation values in the upper (or lower) triangle of the correlation matrix

Effective correlation is weighted by the value of the standard deviation of the constituent elements

The effective correlation may be much different from the average correlation

Look at SSCM as an example

n

k

k

jkj

n

kTotal

eff

2

1

1

1

22

2





Effective Correlation SSCM has an average correlation of 0.04, but an

effective correlation of 0.10 Effective correlation was calculated with SSCM the

following way:1. Calculate the total cost error of SSCM

• For each data point in the database: Calculate the sum of actual cost database Calculate the sum of the estimated costs Determine the percentage error, and take the average

2 For each data point, calculate the SSCM error by multiplying the sum of actual costs by the SSCM percent error, SSCM and square to get

TOT

%24SSCM





3. Then, for each data point, calculate i by multiplying the actual cost for each WBS element by its respective percent error

4. Calculate the dot product of i with itself to get i

5. Now use the following formula to get the effective correlation, eff for that data point.

6. Finally, get the average of the effective correlations to get eff for the model

The eff for SSCM is 0.10 We should figure this out for all of our models!

Effective Correlation

n

k

k

jkj

n

kTotal

eff

2

1

1

1

22

2





Effective Correlation with the Retro ICE Method

Remember the Retro Ice Example?

Use the effective correlation equation to solve for eff

Program SV SEITPM Bus Payload TotalA 5 5 2 12B 2 10 38 50C -5 -8 -2 -15D 0 -5 3 -2E 4 7 2 13F 22 5 55 82G 8 5 5 18H 10 10 1 21Stdev 8.048957342 6.631903 21.26029 30.53306

Estimating Error

508.0)26.21*63.626.21*05.863.6*05.8(*2

)26.2163.605.8(53.30

2

2222

2

1

1

1

22

n

k

k

jkj

n

kTotal

eff




Effective Correlation with the Retro ICE Method

Retro ICE correlation Matrix Average = 0.502

Retro ICE effective correlation eff =0.508

1 0.525211 0.6394730.525210993 1 0.342461

0.63947252 0.342461 1

1 0.508 0.5080.508 1 0.5080.508 0.508 1



82

Determining Correlation When Data is Not Available





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






The Problem

It is not always possible to calculate statistical correlation between WBS elements. May be insufficient data to determine statistical correlation. May be no known functional relationship between WBS

elements.

Yet, there may be reason to believe increases or decreases in the cost of a certain WBS element are likely to cause corresponding increases or decreases in the cost of another WBS element.

In cases such as these, it is still desirable to construct a correlation matrix in order to ensure a truer picture of the total cost variance.




Potential Solutions What can you do if you cannot construct a correlation

matrix from statistical or empirical means? Assume independence

Same as a correlation matrix of zeros. Extremely easy – but, as we have shown, it is WRONG!

Use Knee-in-Curve Method (Steve Book’s Rule of Thumb) When in doubt, assume all correlation values are 0.2. Captures about 80% of the variance compared to assuming

independence. Easy to do. Use the “N-effect”

Modulate our guess at correlation by preserving total error of the estimate

Develop a subjective correlation matrix Excellent results if you can do it accurately. Can fill out an entire correlation matrix this way, but is

somewhat difficult.



85

Knee-in-Curve Method (Steve Book’s Rule of Thumb)





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Steve Book’s Rule of Thumb

According to Dr. Steve Book…

“Ignoring” correlation issue is not a good strategy Tantamount to setting all correlations equal to zero

Problem will not go away

Actual numerical values of inter-WBS-element correlations are difficult to establish, but your estimate and range will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes.

“Ignoring” correlation issue is not a good strategy Tantamount to setting all correlations equal to zero

Problem will not go away

Actual numerical values of inter-WBS-element correlations are difficult to establish, but your estimate and range will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes.

From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA





Dr. Book plotted the theoretical underestimation of percent total cost standard deviation when correlation is assumed to be zero rather than its true value, .

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Per

cent U

nder

estim

ated

n = 10

n = 30

n = 100n = 1000

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Per

cent U

nder

estim

ated

n = 10

n = 30

n = 100n = 1000






For example, given a 30 x 30 correlation matrix in which each actual correlation coefficient is 0.2, if you were to instead assume each correlation coefficient’s value is zero, then you would underestimate the standard deviation of the resulting cost probability distribution by about 60%.

Dr. Book argues that since the “knee” of these curves lies at about 0.2, then it is better to populate an unknown correlation matrix with 0.2’s, or 0.3’s, rather than zeros. Doing so will reasonably capture a substantial amount of

cost variance over doing nothing.





Again, according to Dr. Book…

Repeat: “ignoring” correlation is equivalent to assuming that risks are uncorrelated, e.g., all correlations are zero

If you don’t know the “exact” correlation (and nobody does), you will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes 0.2 is at “knee” of curve on previous charts, thereby providing

most of the benefits at the least loss of accuracy

Higher values are also worth considering

Repeat: “ignoring” correlation is equivalent to assuming that risks are uncorrelated, e.g., all correlations are zero

If you don’t know the “exact” correlation (and nobody does), you will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes 0.2 is at “knee” of curve on previous charts, thereby providing

most of the benefits at the least loss of accuracy

Higher values are also worth considering




90

“N-effect” Correlation





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






“N-effect” Correlation

As N increases, the effective correlation (eff) will decrease in reaction to the central limit theorem. This is the “N-effect”

Why? There is a fundamental limit to the predictive capability of our CERs. Just by breaking the WBS up into more pieces doesn’t improve our estimates.




“N-effect” Correlation Average Correlation* in models seem to be sensitive to number

(N) of CERs

As N increases, decreases


Maximum Possible Underestimation of Total-Cost Sigma

0

10

20

30

40

50

60

70

80

90

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Per

cen

t U

nd

eres

tim

ated

NAFCOM N= 55

USCM7 Bus N= 19

USCM7 FU Bus N= 11

USCM8 Bus N= 17

USCM8 Comm N= 13

SSCM N= 9

•The average correlation is different from the effective correlation, but the effect is similar




Determining Correlation from the Number of WBS Items

There appears to be a trend between the number of WBS Elements (N) in a cost model and the derived average correlation coefficient (AVG) and effective correlation EFF

EFF is a single number used to fill the correlation matrix

As N increases, EFF decreases We looked at the following models:

NAFCOM (NASA/ Air Force Cost Model) USCM7 (Unmanned Space Vehicle Cost Model, Ver. 7) USCM8 (Unmanned Space Vehicle Cost Model, Ver. 8) SSCM (Small Satellite Cost Model)

11

11

1




Determining Correlation from Number of WBS Items

If we see a trend in the chart of percent under-estimation of sigma vs. effective correlation, we have a sound basis for determining correlations when the number of WBS elements grows.

If the actual percent underestimated is k then the N-effect correlation N for a model with N CERs would be:

So, for k=50%:

1

1

1001

12

N

k

N

10 0.333 15 0.214 20 0.158 30 0.103 50 0.061

100 0.030 150 0.020 200 0.015 300 0.010 500 0.006


1

3

1

14

1

15.1

1

1

1

1001

1

2

2

NNNN

k



95

Causal Guess Method of Subjective Correlation(Tim Anderson’s Method)





Retro-ICE

Retro-ICE

Causal Guess

Causal Guess

N-Effect Guess

N-Effect Guess


Effective

Effective






Subjective Correlation

It has previously been shown that it is possible to derive the empirical residual correlation coefficients of a cost model such as USCM, NAFCOM or SSCM. However, this method requires exclusive use of either of

these two cost models to be effective.

One alternative method is to subjectively develop approximate correlation coefficients between WBS elements. This can be as simple as determining whether any two WBS

elements are correlated by a small amount, or by a large amount, and whether that correlation is positive or negative.




Subjective Correlation An example of a subjective correlation decision table

might look like the following:

For example, if you believe two WBS elements have a small amount of positive correlation, then you would choose a correlation value of 0.3.

Positive

correlation Negative

correlation

Uncorrelated 0 0

Small amount of correlation

0.3 -0.3

Large amount of correlation

0.75 -0.75

Subjective Correlation Coefficients




Subjective Correlation

Using this technique, it is only necessary to make the following argument between any two WBS elements:

Thus, if the answer were that a change in the cost of one WBS element might cause a minor, similar change in the other WBS element, then one would assign a correlation value of 0.3 between the two WBS elements.

“If circumstances cause the cost of one WBS element to change, is the other WBS element likely to change also? If so,

how substantially and in what direction?”




But Does It Work? The subjective scoring scheme shown previously is

based on averages. As the figure below shows, the values 0.0, ±0.3, and ±0.75 are the average values in each of their respective ranges.

The idea is this: While one might not know the true correlation between WBS elements, if instead one can subjectively bucket the correlation into one of these five ranges, then a correlation matrix composed of these averages should give approximately the same results as if the true correlations were known.

Negative Correlation Positive Correlation

-0.75 -0.3 0.0 0.3 0.75

0 0.2 0.40.3 0.5 0.70.60.1 0.8 1.00.9-0.9 -0.7-0.8 -0.6 -0.4-0.5-1.0 -0.3 -0.1-0.2




But Does It Work? To test the hypothesis, suppose we substitute the true

correlation coefficients in a typical correlation matrix with their corresponding averages from the subjective correlation table.

If the resulting cost probability distributions are similar, then we can use the technique with some confidence.

An important point, however, is that the subjectively derived values should reflect the actual values.

If this is true, then this method should give a more precise answer than Steve Book’s Rule of Thumb method.

We will see in the following example



101

Application of Correlation Methods When Data is Not Available

Cost Risk Analysis of A Small Earth Orbiting

Visible Imaging Sensor




Example Consider an estimate developed using the following

cost model:Work Breakdown Structure Parameter, X Input data range

RDT&E CER (FY00$K) SE

1. Payload

1.1 Visible Light Sensor Aperture diameter (m) 0.2 - 1.2 128,827 X 0.562 19,3362. Spacecraft

2.1 Structure Structure weight (kg) 54-392 157 X 0.83 38%

2.2 Thermal Thermal weight (kg) 3 - 48 394 X 0.635 45%

2.3 Electrical Power System

X1 = EPS weight (kg), X 2 = BOL

power (wt)

X1: 31 - 491, X 2:

100 - 2400 2.63 (X 1 X2) 0.71236%

2.4 Telemetry, Tracking and Command TT&C weight (kg) 12 - 65 545 X 0.761 57%

2.5 Attitude Determination and Control ADCS weight (kg) 20 - 160 464 X 0.867 48%

3. Integration, Assy. & TestSpacecraft bus + payload total RDT&E cost 2,703 - 395,529 969 + 0.215 X 46%

4. Program Mgmt.Spacecraft bus + payload total RDT&E cost 4,607 - 523,757 1.963 X 0.841 36%

5. Ground Support EquipmentSpacecraft bus + payload total RDT&E cost 24,465 - 581,637 9.262 X 0.642 34%

Work Breakdown Structure Parameter, X Input data range T1 CER (FY00$K) SE1. Payload

1.1 Visible Light Sensor Aperture diameter (m) 0.2 - 1.2 51,469 X 0.562 7,7342. Spacecraft

2.1 Structure Structure weight (kg) 54-560 13.1 X 39%

2.2 Thermal Thermal weight (kg) 3 - 87 50.6 X 0.707 61%

2.3 Electrical Power System EPS weight (kg) 31 - 573 112 X 0.763 44%

2.4 Telemetry, Tracking and Command TT&C weight (kg) 13 - 79 635 X 0.568 41%

2.5 Attitude Determination and Control ADCS weight (kg) 20 - 192 293 X 0.777 34%

3. Integration, Assy. & TestSpacecraft bus weight + payload weight 155 - 1,390 10.4 X 44%

4. Program Mgmt.Spacecraft bus + payload total recurring cost

15,929 - 1,148,084 0.341 X 39%

6. Launch & Orbital Operations SupportSpacecraft bus weight + payload weight 348 - 1,537 4.9 X 42%

RDT&E Cost Model (FY00$K) (less fee)

T1 Cost Model (FY00$K) (less fee)

Reprinted from Space Mission Analysis and Design, 3rd Edition, Wertz and Larson

Reprinted from Space Mission Analysis and Design, 3rd Edition, Wertz and Larson

Reprinted fromSpace Mission AnalysisAnd Design, 3rd Edition.Wertz and Larson




Input Variables

Suppose this spacecraft has the following set of (mean) input variables:

Cost Driver ValueAperture diameter (m) 1.04Payload Wt (kg) 212Structure Wt (kg) 85Thermal Wt. (kg) 11EPS Wt. (kg) 318BOL Power (W) 1061TT&C Wt. (kg) 52ADCS Wt. (kg) 148




Functional Relationships Integration and Test, Program Management and LOOS are

functionally correlated to PL and S/C bus cost The input variables are functionally correlated through the

following sizing relationships with their error terms () Payload Wt. = 200 * (Aperture Diameter)^1.5 + PL

Structure Wt. = 0.4 * Payload Wt. + STR

Thermal Wt. = 0.05 * Payload Wt. + TH

BOLP = 5.0 * Payload Wt. + BOLP

EPS Wt. = 0.3 * BOLP + EPS

TTC Wt. = 50 + 0.01 * Payload Wt. + TTC

ADCS Wt = 0.7 * Payload Wt. + ADCS




Input Parameter Error Terms

Aperture Diameter has a discrete 20% probability of being 1.0m, and a 80% probability of being 1.2m.

Assume the error terms () for the sizing relationships are all triangular probability density functions defined by Low = 0.9 Most Likely = 1.0 High =1.4

1.000.90 1.40




Cost Estimate The resulting cost estimate has the following

(deterministic) value:

Cost Driver ValueAperture diameter (m) 1.04Payload Wt (kg) 212Structure Wt (kg) 85Thermal Wt. (kg) 11EPS Wt. (kg) 318BOL Power (W) 1061TT&C Wt. (kg) 52ADCS Wt. (kg) 148

RDT&E (FY00$K) $M1. Payload 131.7 1.1 Visible Light Sensor 131.72. Spacecraft 77.2 2.1 Structurre 6.3 2.2 Thermal 1.8 2.3 Electrical Power System 22.7 2.4 Telemetry, Tracking and Command 11.0 2.5 Attitude Determination and Control 35.43. Integration, Assy. & Test 45.94. Program Mgt. 59.56. Launch & Orbital Operations Support 24.1TOTAL RDT&E 338.4

T1 (FY00$K) $M1. Payload 52.6 1.1 Visible Light Sensor 52.62. Spacecraft 30.7 2.1 Structurre 1.1 2.2 Thermal 0.3 2.3 Electrical Power System 9.1 2.4 Telemetry, Tracking and Command 6.0 2.5 Attitude Determination and Control 14.33. Integration, Assy. & Test 8.64. Program Mgt. 28.46. Launch & Orbital Operations Support 4.0TOTAL T1 148.8

Total RDT&E + T1 487.2




“Actual” Correlation Matrix Suppose the “actual” correlation matrix is as follows:

Ad

ePL

eST

R

eTH

eEP

S

eBO

LP

eTT

C

eAD

CS

RD

T&

E P

ayload

RD

T&

E S

tructure

RD

T&

E T

hermal

RD

T&

E E

PS

RD

T&

E T

T&

C

RD

T&

E A

DC

S

RD

T&

E IA

&T

RD

T&

E P

M

RD

T&

E G

SE

T1 P

ayload

T1 S

tructure

T1 T

hermal

T1 E

PS

T1 T

T&

C

T1 A

DC

S

T1 IA

&T

Ad 1.000 0.950 0.800 0.600 0.308 0.450 0.600 0.751 0.684 0.134 0.471 0.319 0.399 0.050 0.207 0.025 0.316 0.159 0.204 0.164 0.202 0.476 0.188 0.342ePL 1.000 0.540 0.412 0.247 0.354 0.450 0.654 0.009 0.183 0.066 0.357 0.323 0.423 0.110 0.015 0.083 0.397 0.149 0.069 0.072 0.273 0.259 0.339eSTR 1.000 0.436 0.540 0.423 0.005 0.010 0.267 0.144 0.020 0.491 0.213 0.343 0.078 0.359 0.088 0.043 0.444 0.122 0.123 0.223 0.165 0.023eTH 1.000 0.353 0.258 0.117 0.006 0.337 0.270 0.491 0.242 0.063 0.063 0.036 0.188 0.314 0.016 0.084 0.156 0.192 0.417 0.209 0.118eEPS 1.000 0.840 0.119 0.428 0.468 0.002 0.086 0.289 0.254 0.157 0.415 0.371 0.142 0.038 0.468 0.218 0.101 0.205 0.314 0.294eBOLP 1.000 0.187 0.235 0.358 0.412 0.187 0.245 0.101 0.252 0.427 0.098 0.296 0.477 0.050 0.116 0.160 0.191 0.430 0.020eTTC 1.000 0.334 0.454 0.088 0.224 0.046 0.286 0.186 0.478 0.405 0.459 0.264 0.145 0.142 0.246 0.357 0.093 0.326eADCS 1.000 0.240 0.367 0.078 0.024 0.451 0.367 0.328 0.481 0.003 0.329 0.053 0.342 0.460 0.368 0.387 0.355RDT&E Payload 1.000 0.508 0.071 -0.424 -0.029 -0.681 0.359 -0.565 -0.208 -0.498 0.586 0.181 -0.121 -0.054 -0.571 0.812RDT&E Structure 1.000 0.411 -0.778 0.162 0.381 -0.851 -0.776 -0.612 -0.583 0.493 -0.894 0.392 -0.474 -0.660 -0.071RDT&E Thermal 1.000 -0.059 0.881 0.652 0.705 0.171 -0.566 0.882 0.011 0.415 -0.978 0.816 0.613 0.275RDT&E EPS 1.000 -0.990 0.371 -0.177 0.987 -0.953 0.921 -0.021 0.497 -0.668 0.319 -0.658 -0.696RDT&E TT&C 1.000 -0.180 -0.570 0.216 0.126 -0.216 0.735 -0.701 -0.435 -0.965 -0.986 -0.931RDT&E ADCS 1.000 0.401 0.139 -0.325 0.785 0.251 -0.175 0.559 -0.460 -0.426 0.011RDT&E IA&T 1.000 -0.182 0.600 -0.318 0.055 -0.813 0.068 0.781 0.468 -0.236RDT&E PM 1.000 -0.984 0.695 0.548 0.435 -0.528 -0.735 0.759 0.153RDT&E GSE 1.000 0.400 0.224 0.965 -0.566 0.275 -0.389 0.052T1 Payload 1.000 0.138 -0.893 -0.224 -0.981 -0.264 0.610T1 Structure 1.000 0.186 -0.126 0.049 0.528 -0.989T1 Thermal 1.000 -0.002 -0.542 0.217 -0.681T1 EPS 1.000 -0.039 0.969 -0.399T1 TT&C 1.000 -0.927 0.382T1 ADCS 1.000 -0.314T1 IA&T 1.000T1 PMT1 GSE




Total Cost Distribution: Actual Correlation Values

Total cost distribution with “actual” correlation values:

$M

Lognormal: Mean = $534.7M Std Dev = $126.6M


Note: This is the distribution that followsas a result of using the“actual” correlationcoefficients. The standard deviation is 24% of the mean.

Note: This is the distribution that followsas a result of using the“actual” correlationcoefficients. The standard deviation is 24% of the mean.

Frequency Chart

.000

.027

.054

.081

.109

0

135.7

271.5

407.2

543

0.0 312.5 625.0 937.5 1250.0

5,000 Trials 0 Outliers

Forecast: Total RDT&E + T1




Total Cost Distribution: No Statistical Correlation

We have included functional correlation, but no statistical correlation between the error terms,

Under these circumstances, the total cost distribution has the following appearance and statistics:

Frequency Chart

.000

.031

.062

.093

.124

0

154.5

309

463.5

618

0.0 312.5 625.0 937.5 1250.0



Note: This is a narrow distribution. The standard deviation is 15%of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%.

Note: This is a narrow distribution. The standard deviation is 15%of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%.

$M






Total Cost Distribution: Knee-in-Curve Method

Using Steve Book’s Rule of Thumb, the correlation is set to 0.2, and the total cost distribution has the following appearance and statistics:

$M

Note: This has caused the standard deviation to shift substantially compared to the zero correlation case.

The standard deviation grew by over 76% from the uncorrelated case.


The standard deviation grew by over 76% from the uncorrelated case.

Frequency Chart

.000

.019

.039

.058

.078

0

97.25

194.5

291.7

389

0.0 312.5 625.0 937.5 1250.0

5,000 Trials 1 Outlier







Total Cost Distribution: “N-Effect” Method Using the N-Effect method, the correlation is 0.125,

and the total cost distribution has the following appearance and statistics:


The standard deviation grew by over 59% from the uncorrelated case.This is almost exactly the Standard error of the “actual correlation” case.


The standard deviation grew by over 59% from the uncorrelated case.This is almost exactly the Standard error of the “actual correlation” case.

$M

Frequency Chart

.000

.022

.044

.066

.088

0

110.5

221

331.5

442

0.0 312.5 625.0 937.5 1250.0








“Subjective” Correlation Matrix The corresponding “subjective” correlation matrix

using the causal guess method is as follows:

Ad

ePL

eST

R

eTH

eEP

S

eBO

LP

eTT

C

eAD

CS

RD

T&

E P

ayload

RD

T&

E S

tructure

RD

T&

E T

hermal

RD

T&

E E

PS

RD

T&

E T

T&

C

RD

T&

E A

DC

S

RD

T&

E IA

&T

RD

T&

E P

M

RD

T&

E G

SE

T1 P

ayload

T1 S

tructure

T1 T

hermal

T1 E

PS

T1 T

T&

C

T1 A

DC

S

T1 IA

&T

Ad 1.000 0.750 0.750 0.750 0.300 0.300 0.750 0.750 0.750 0.300 0.300 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300ePL 1.000 0.750 0.300 0.300 0.300 0.300 0.750 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.000 0.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300eSTR 1.000 0.300 0.750 0.300 0.000 0.000 0.300 0.300 0.000 0.300 0.300 0.300 0.000 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300 0.000eTH 1.000 0.300 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.000 0.000 0.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300eEPS 1.000 0.750 0.300 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300eBOLP 1.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.000eTTC 1.000 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300eADCS 1.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300RDT&E Payload 1.000 0.750 0.000 -0.300 0.000 -0.750 0.300 -0.750 -0.300 -0.300 0.750 0.300 -0.300 0.000 -0.750 0.750RDT&E Structure 1.000 0.300 -0.750 0.300 0.300 -0.750 -0.750 -0.750 -0.750 0.300 -0.750 0.300 -0.300 -0.750 0.000RDT&E Thermal 1.000 0.000 0.750 0.750 0.750 0.300 -0.750 0.750 0.000 0.300 -0.750 0.750 0.750 0.300RDT&E EPS 1.000 -0.750 0.300 -0.300 0.750 -0.750 0.750 0.000 0.300 -0.750 0.300 -0.750 -0.750RDT&E TT&C 1.000 -0.300 -0.750 0.300 0.300 -0.300 0.750 -0.750 -0.300 -0.750 -0.750 -0.750RDT&E ADCS 1.000 0.300 0.300 -0.300 0.750 0.300 -0.300 0.750 -0.300 -0.300 0.000RDT&E IA&T 1.000 -0.300 0.750 -0.300 0.000 -0.750 0.000 0.750 0.300 -0.300RDT&E PM 1.000 -0.750 0.750 0.750 0.300 -0.750 -0.750 0.750 0.300RDT&E GSE 1.000 0.300 0.300 0.750 -0.750 0.300 -0.300 0.000T1 Payload 1.000 0.300 -0.750 -0.300 -0.750 -0.300 0.750T1 Structure 1.000 0.300 -0.300 0.000 0.750 -0.750T1 Thermal 1.000 0.000 -0.750 0.300 -0.750T1 EPS 1.000 0.000 0.750 -0.300T1 TT&C 1.000 -0.750 0.300T1 ADCS 1.000 -0.300T1 IA&T 1.000T1 PMT1 GSE




Total Cost Distribution:Causal Guess Subjective Values

Total cost distribution with “subjective” correlation values:

$M

Note: This is the distribution that followsas a result of using the“subjective” correlationcoefficients.

The mean is nearly identical to the “actual” case, and the standarddeviation has increasedby approximately 4%.

Note: This is the distribution that followsas a result of using the“subjective” correlationcoefficients.

The mean is nearly identical to the “actual” case, and the standarddeviation has increasedby approximately 4%.

Frequency Chart

.000

.029

.057

.086

.114

0

142.5

285

427.5

570

0.0 312.5 625.0 937.5 1250.0








Summary of Results

0

0.001

0.002

0.003

0.004

0.005

0.006

200 300 400 500 600 700 800 900 1000

Total Cost $M

Lik

liho

od

mean sigma

Actual 534.7 126.6

Uncorrelated 525.8 79.7

Knee-in-curve 535.3 148.2

N-Effect 532.6 127

Causal guess 531.2 121.2




Summary of Results

Ignoring correlation understated the total cost sigma The N-Effect and Causal Guess methods produced

the best results The Knee-in curve method was close, but provided

the largest variance




Part 5 Review We learned how to derive correlation coefficients:

When data is available When you have to make an educated guess

The Methods: Statistical (Data is available)

Residual analysis Retro-ICE Effective correlation, eff

When data is not available Knee-in-curve method (Steve Book’s Rule of Thumb) N-effect guess Subjective guess (Tim Anderson’s Method)

Example cost risk analysis using subjective methods



117

End


Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC...

Documents

Transcript of Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC...