Bitten

1

Conceptual Design Optimism, Cost and Schedule Growth Effects

Presented at the 2010 NASA Program Management Challenge

9-10 February 2010, Houston, Texas

Claude Freaner, Bob Bitten, Debra Emmons

Used with permission

2

Agenda

• Background/Overview

• Database Description

• Resource Growth– Mass, Power, Cost & Schedule Trends

• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons

• Potential Considerations

3

Background

• Results from a recent study* of 10 NASA missions over the past decade indicated that the average cost and schedule growth of these missions, over and above programmatic reserves, was 76% and 26%, respectively, measured from beginning of Phase B

• One potential causative factor postulated was the inherent optimism in initial concept designs due to competitive pressures

• Inherent optimism can translate to the underestimation of the technical specifications such as mass, power, data rate, and the complexity of a system

• Underestimation of these resources can lead to the underestimation of the final cost of the mission since most cost models use some form of system resources as a predictor of mission cost

• To compound problems, the desire to launch a system as early as possible, in order to obtain science quickly, can lead to a success oriented schedule that may be shorter than historical comparisons would indicate

• This combination of underestimated resources providing an optimistic cost estimate basis combined with a success oriented schedule can contribute to the observed history of cost and schedule growth

* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008

4

Study Approach

• For a set of 20 missions in the study, the mass, power, cost, schedule and other parameters were identified at the beginning of the Preliminary Design phase (NASA Phase B) of a mission*

• These values were then compared to values presented at the Preliminary Design Review (PDR), Critical Design Review (CDR) and at the time of launch to understand the growth over time of each of these resources

• The resource growth is then compared to industry guidelines to understand if these guidelines would have adequately predicted the growth for the mission data set studied

• In addition, resource growth of different mission types and correlation of cost growth in relation to different parameters is also shown

* All parametric and programmatic data were obtained from NASA CADRe data"

5

Agenda






6

Database Description:20 Missions Represent a Wide Range of Recent NASA Missions

• 5 Directed vs. 15 Competed missions

• 7 Planetary missions vs. 13 Earth or near-Earth Orbiters

• 7 Planetary Science vs. 5 Astrophysics vs. 5 Earth Science vs. 3 Heliophysics missions

Planetary? Program Science TypeKey

Center(s)Launch

YearAcquisition

TypeNumber of

Instruments Comments

EO-1 NMP Earth Science GSFC 2000 Competed 5Advanced land imaging technology demonstrator

GENESIS X Discovery Planetary Science JPL 2001 Competed 4Collect samples of solar wind particles at L1 point and return them to Earth

GRACE ESSP Earth Science JPL 2002 Competed 6 Earth Gravity Measurement

SpitzerPhysics of the Cosmos

Astrophysics JPL 2003 Directed 4IR space telescope, the last of the Great Observatories

GALEX Explorers Astrophysics JPL/CalTech 2003 Competed 1 UV space telescope

SWIFT Explorers Astrophysics GSFC 2004 Competed 4 Gamma Ray burst detector

MESSENGER X Discovery Planetary Science APL 2004 Competed 7 Investigate Mercury

MRO X MEP Planetary Science JPL 2005 Directed 7 Investigate history of water on Mars

Deep Impact X Discovery Planetary Science JPL 2005 Competed 3 Comet impactor

Cloudsat ESSP Earth Science JPL 2006 Competed 1 Radar observation of clouds

STEREO STP Heliospheric Science GSFC/APL 2006 Directed 42 spacecraft looking at solar dynamics - Earth leading and trailing orbits

CALIPSO ESSP Earth Science LARC 2006 Competed 3 Aerosols

New Horizons

XNew Frontiers

Planetary Science APL 2006 Competed 7 Investigate Pluto

DAWN X Discovery Planetary Science JPL 2007 Competed 2 Investigate Ceres and Vesta protoplanets

AIM Explorers Heliospheric Science LASP 2007 Competed 3 Aeronomy of Ice in Mesosphere

Fermi (GLAST)

Physics of the Cosmos

Astrophysics GSFC 2008 Directed 2 Gamma Ray Telescope

IBEX Explorers Heliospheric Science GSFC 2008 Competed 2Interaction between solar wind and interstellar medium

Kepler Discovery Astrophysics JPL 2009 Competed 1 Search for Earth-sized exoplanets

LRO XRobotic Lunar

ESMD/Planetary Science

GSFC 2009 Directed 7 Origin of the Moon

OCO ESSP Earth Science JPL 2009 Competed 1Carbon Dioxide Investigation. Mission failed due to launch vehicle failure

7

Updated 20 Mission Results Are Similar to Initial 10 Mission Paper* Results

43% 42%

76%

36%37%41%

56%

38%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Mass Power Cost Schedule

Aver

age P

erce

nt G

rowt

h

10 Mission Study20 Mission Study

1 1 2 2

Note: 1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves

* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008

8

Agenda






9

37%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Mission #1

Mission #2

Mission #3

Mission #4

Mission #5

Mission #6

Mission #7

Mission #8

Mission #9

Mission #1

0

Mission #1

1

Mission #1

2

Mission #1

3

Mission #1

4

Mission #1

5

Mission #1

6

Mission #1

7

Mission #1

8

Mission #1

9

Mission #2

0

Averag

e

Sate

llite

Mas

s G

row

th R

elat

ive

to C

BE

(%)

Summary of Mass Growth from Start of Phase B

Mass Growth Exceeds Typical Guidance:13 out of 20 Missions Exceed 30% Growth from Current Best Estimate

Planetary MissionsAverage

Earth Orbiting Missions

10

Summary of Satellite Mass Growth Over Time

Majority of Mass Growth Occurs Primarily Prior to PDR

37%

0%

28%

19%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

ATP PDR CDR Launch

Sate

llite

Mas

s G

row

th o

ver C

BE

(%)

Mission #1Mission #2Mission #3Mission #4Mission #5Mission #6Mission #7Mission #8Mission #9Mission #10Mission #11Mission #12Mission #13Mission #14Mission #15Mission #16Mission #17Mission #18Mission #19Mission #20Average

11

41%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

Mission #1

Mission #2

Mission #3

Mission #4

Mission #5

Mission #6

Mission #7

Mission #8

Mission #9

Mission #1

0

Mission #1

1

Mission #1

2

Mission #1

3

Mission #1

4

Mission #1

5

Mission #1

6

Mission #1

7

Mission #1

8

Mission #1

9

Mission #2

0

Averag

e

Sate

llite

Pow

er G

row

th R

elat

ive

to C

BE

(%)

Summary of Power Growth from Start of Phase B

Power Growth Exceeds Typical Guidance:8 out of 19 Missions Exceed 30% Growth from Current Best Estimate



12

Summary of Power Growth Over Time

Majority of Power Growth Occurs Primarily Prior to CDR

19%

36%41%

0%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

ATP PDR CDR Launch

Sate

llite

Pow

er G

row

th o

ver C

BE

(%)


13

Summary of Cost Growth from Start of Phase B

Development Cost Growth is Significant:Average Cost Growth, Over & Above Reserves, is 56%

56%

0%

20%

40%

60%

80%

100%

120%

140%

160%

Mission #1

Mission #2

Mission #3

Mission #4

Mission #5

Mission #6

Mission #7

Mission #8

Mission #9

Mission #1

0

Mission #1

1

Mission #1

2

Mission #1

3

Mission #1

4

Mission #1

5

Mission #1

6

Mission #1

7

Mission #1

8

Mission #1

9

Mission #2

0

Averag

e

Dev

elop

men

t Cos

t Gro

wth

ove

r Bas

elin

e (%

)

$-

$40

$80

$120

$160

$200

$240

$280

$320

$360

$400

Dev

elop

men

t Cos

t Gro

wth

ove

r Bas

elin

e ($

M)



Absolute Growth in $M

14

Summary of Cost Growth Over Time

Cost Growth Occurs Primarily After PDR:More than Half of Growth (14% to 56%) is Realized After PDR

21%

56%

14%

0%

-25%

0%

25%

50%

75%

100%

125%

150%

ATP PDR CDR Launch

Mis

sion

Dev

elop

men

t Cos

t Gro

wth

ove

r Bas

elin

e w

ith R

eser

ves

(%)


15

Summary of Schedule Growth from Start of Phase B

Schedule Growth is Significant:Average Schedule Growth is on the order of 18 Months

38%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Mission #1

Mission #2

Mission #3

Mission #4

Mission #5

Mission #6

Mission #7

Mission #8

Mission #9

Mission #1

0

Mission #1

1

Mission #1

2

Mission #1

3

Mission #1

4

Mission #1

5

Mission #1

6

Mission #1

7

Mission #1

8

Mission #1

9

Mission #2

0

Averag

e

Dev

elop

men

t Sch

edul

e G

row

th o

ver B

asel

ine

(%)

0

5

10

15

20

25

30

35

40

45

50

Dev

elop

men

t Sch

edul

e G

row

th o

ver B

asel

ine

(Mon

ths)



Absolute Growth in Months

16

Summary of Schedule Growth Over Time

Schedule Growth Occurs Primarily After PDR:More than Half of Growth (13 of 18 Months) is Realized After PDR

10%17%

38%

-20%

0%

20%

40%

60%

80%

100%

ATP PDR CDR LaunchMis

sion

Dev

elop

men

t Sch

edul

e G

row

th o

ver P

lan

(%)


5 months 8 months18 months

17

Agenda






18

Cost & Schedule Still Show Significant Growth from Baseline Established at PDR

37%41%

38%

15%19%

37%

27%

56%

0%

10%

20%

30%

40%

50%

60%


Aver

age P

erce

nt G

rowt

h

From Phase B StartFrom PDR

1 1 2 2

Note:1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves

Over Half of Uncertainty for Mass & Power is Retired by PDR while 2/3 of Cost & Schedule Uncertainty Remain

19

Data Shows that Programmatic Baseline Maturity Lags Technical Design Maturity

0%

10%

20%

30%

40%

50%

60%

Phase B Start PDR CDR Launch

Aver

age P

erce

nt G

rowt

h

MassPowerCostSchedule

Technical Design Mature by CDR

Programmatic RealizationNot Until Launch

20

Comparison of Competed vs. Directed Missions: No Significant Difference in Results

40%

29%

51%

31%36%

42%

58%

40%

0%

10%

20%

30%

40%

50%

60%

70%


Aver

age

Perc

ent G

rowt

h fro

m P

hase

B S

tart Directed (5)

Competed (15)

1 1 2 2


Growth in All Resources are Similar Between Competed & Directed Missions

21

Planetary vs. Earth Orbiting Missions: Resource Growth is Similar for Mass & Power

40% 37%

25%22%

35%40%

73%

47%

0%

10%

20%

30%

40%

50%

60%

70%

80%


Aver

age

Perc

ent G

rowt

h fro

m P

hase

B S

tart Planetary (7)

Earth Orbiting (13)

1 1 2 2


Cost & Schedule Growth for Planetary Missions is Significantly Less

22

Correlation Between Cost and Schedule Growth Indicates that Schedule Growth Has Strong Influence on Cost Growth

% Cost Growth = 1.4775 * % Schedule GrowthR2 = 0.6166

0%

20%

40%

60%

80%

100%

120%

140%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Schedule Growth from Phase B Start

Dev

elop

men

t Cos

t Gro

wth

from

Pha

se B

Sta

rt

23

Payload Mass and Cost Growth Significantly Greater than Spacecraft Mass & Cost Growth

60%

101%

33%44%

0%

20%

40%

60%

80%

100%

120%

Mass Cost

Aver

age

Perc

ent G

rowt

h fro

m P

hase

B S

tart Payload

Spacecraft

1 1

Note:1) As measured from Current Best Estimate, not including reserves

Data Indicated Payload Resource has Greater Uncertainty than Spacecraft

24

Cumulative Cost Distribution Shows Wide Distribution of Payload Cost Growth vs. Spacecraft Cost Growth

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

-25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250%

Cost Growth from Phase B Start over Initial Estimate without Cost Reserve

Cum

ulat

ive

Dis

trib

utio

n fo

r All

Dat

a Po

ints

Spacecraft Cost GrowthPayload Cost Growth

Payload Cost Growth hasMuch Greater Variancethan Spacecraft Growth

25

Instrument Payload Mass/Cost Growth Data Over Time Also Indicates Payload Cost Lags Payload Maturity

0%

31%43%

60%

0%

18%

37%

101%

0%

20%

40%

60%

80%

100%

120%

Phase B Start PDR CDR Launch

Perc

ent G

rowt

h ov

er C

BE w

/o re

serv

es

Payload MassPayload Cost

Payload Cost Significantly Lags Payload Design Maturity

26

Range of Mass Growth at Instrument & Subsystem LevelCan Provide Guidance for Initial KDP-B Cost Estimates

Range of Mass Growth is Large and Exceeds Typical Industry Guidance

-75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% 275%

Struct & Mech

Power

C&DH

Comm

Thermal

ADCS

Propulsion

Instruments

Range for Mass Distributions

Below Mean

Above Mean

Typical Mass ReserveDesign Guidance 30%

Subsystem Min Mean Max MedianInstrument 11% 60% 202% 51%Propulsion -73% 10% 69% 18%ADCS -21% 29% 108% 24%Thermal -36% 77% 269% 58%Comm -55% 5% 76% 7%C&DH -37% 16% 104% 6%Power -37% 45% 103% 40%Structure & Mech -19% 57% 142% 60%

27

Agenda






28

Potential Considerations• For Project Managers

– Mass and power reserve guidelines for spacecraft and payload could be increased to be more consistent with historical averages

– Schedule growth seems to be a key factor in controlling cost growth

– More emphasis could be placed on early payload designs as there is much greater uncertainty in the payload development than the spacecraft

• For Cost Analysts– Wider ranges of input parameters could be used to provide more

robust initial cost estimates which can address uncertainty in early design

29

References

1) Freaner C., Bitten R., Bearden D., and Emmons D., “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, 2008 SSCAG/SCAF/EACE Joint International Conference, Noordwijk, The Netherlands, 15-16 May 2008.

2) Bitten R., Emmons D., Freaner C., “Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines”, IEEE Aerospace Conference, Big Sky, Montana, March 3-10, 2007.

3) Bitten R.E., “Determining When A Mission Is "Outside The Box": Guidelines For A Cost-Constrained Environment”, 6th IAA International Low Cost Planetary Conference, October 11-13, 2005.

4) Bitten R.E., Bearden D.A., Lao N.Y. and Park, T.H., “The Effect of Schedule Constraints on the Success of Planetary Missions”, 5th IAA International Conference on Low-Cost Planetary Missions, 24 September 2003

5) Emmons D., “A Quantitative Approach to Independent Schedule Estimates of Planetary & Earth-orbiting Missions”, 2008 ISPA-SCEA Joint International Conference, Netherlands, 12-14 May 2008

Bitten

Technology

Transcript of Bitten