Confidence Level Estimating and Budgeting

February 2011

Tom Coonce

Topics

• A brief review of confidence level estimating, budget-setting policy and rationale

• Status of program and project Joint Cost and Schedule Confidence Level (JCL) calculations

• JCL issues and lessons learned• Actions completed or planned to

address lessons learned

2

3

Recap of JCL Policy and Rationale

• NPD 1000.5– Programs are to be baselined at a 70% joint confidence

level– Projects are to be baselined/budgeted at JCL that supports

the program approved JCL– Projects are to be funded at no less than a 50% JCL – JCLs are to be developed and maintained through launch– Program and project proposed cost and schedule baselines

will be assessed by an independent review team

• Policy Rationale– Improve cost and schedule performance– Improve expectations with stakeholders– Improve project planning– Improve understanding of project plans

JCL – Defined by a Product, Refers to a Process

• JCL = Joint Confidence Level – Identifies probability that a given project or

program’s cost will be equal or less then the targeted cost AND the schedule will be equal or less then the targeted schedule date

– Per NPD 1000.5 used at KDP-C to set Program/Project Budget Baseline

HeadquartersOffice of Program Analysis and Evaluation

5/18/09 2

MMS SRB ICAJCL Scatter Plot, Corr=0.646 (ICE 70%=1079)

• JCL is More than the Scatter Plot– It’s a systematic process of integrating cost, schedule, and risk – Provides a cohesive and holistic picture of the project or program’s ability to

achieve cost and schedule goals and to help the determination of reserves (schedule and cost)

– Provides key Decision Support information– Does the project have enough funds?– Can the project meet the schedule?– What are areas of risk toward successful execution of the project? – What risk mitigation strategies provide the best project benefit?– What are project phasing (fund) needs? 4

5

Acceptable JCL Methods

• Resource-Loaded Schedule with Monte-Carlo simulation, with three different techniques– Bottoms-up Resource Loaded Schedule or BURLS technique is

based on resource or cost-loaded schedule that focuses on the project risk lists (known-known risks) and uncertainties (known and unknown unknowns) to set the cost and schedule possibilities

– Cost Estimate/Program Assessment (CEPA) technique is based on a top-level resource or cost loaded schedule that focuses on project performance as well as project risks to set cost and schedule possibilities

– Project Risk Evaluation Process or PREP technique is based on the projects’ top-level resource or cost loaded schedule that focuses on the project’s risk list as well as the views of an Independent panel of experts to set cost and schedule possibilities

• Parametric – Uses cost and schedule estimating relationships, data dispersion,

and relationships between cost and schedule to determine the overall cost and schedule target and associated JCL

An Example of JCL Output

6

Points representcost/Schedule resultpair from each iterationof the simulation

Lower quadrantidentifies iterationssatisfying cost andschedule targets ~ the JCL

Horizontal barindicates Y-axis (Cost) confidencelevel

Vertical bar indicates x-axis (Schedule) Confidence level

Frontier line identifiescost/schedule surface that meets Target JCL

7

Management Councils Revisions the Policies and Expectations

• Eliminated the requirement at KDP-B and D, but stipulated that KDP-Cs JCL must be based on probabilistic resource loaded schedules (May 2009 SMC)– Directed pilot projects be completed for projects at KDP-B

• Eliminated the requirement to maintain the JCL during implementation (November 2009 PMC)– Project are not required to develop a JCL at any other time during

implementation if it is “on plan”– Projects are to be monitored as part of the Baseline Performance Review

(BPR) process– If a project is re-baselined, scored red in the BPR, or if BPR leadership

directs, it must re-compute a JCL– The agency is to use existing metrics and/or establish new factors for

determining when a project turns Red • Eliminated JCL requirement for loosely coupled programs, retained the

requirement for tightly coupled programs, and required MDs to submit program impact assessment whenever a new project enters the program (June 2010 PMC)– Eliminate JCL language from NPD 1000.5; reconstitute and update in NPD 7120.5E

8

Management Councils Revisions the Policies and Expectations (Continued)

• Include cost and schedule range estimates and associated probabilities for all projects at KDP-B with LCC >$250M (June and Aug 2010 PMCs)– Coordinate policy language and place into

7120.5E• Projects must develop, explain and

defend plans and associated probability calculations at KDP-B and C (June 2010 PMC)

9

Final Proposed 7120.5E Language(After the 4 August 2010 PMC)

Under 2.5.4.2 Confidence Level Estimating and Budgeting

2.5.4.2.4 Projects with an expected life cycle cost in excess of $250 million are required to develop cost and schedule estimates and associated confidence levels for meeting those estimates.

(1) At KDP B, a range of cost and schedule estimates and associated confidence levels of meeting those estimates will be provided at KDP-B for projects and single-project programs  (see the footnote for paragraph 2.5.4.2.2). This policy applies to both competed (AO) and directed (non-AO) missions. The decision authority will use this information to guide the formulation activities and to establish the target cost and schedule ranges and resources phasing until KDP-C estimates are developed..

(2) At KDP C, projects are budgeted to a 70 percent joint cost and schedule confidence level or as approved by the decision authority.

10

So, What is Required to Meet the KDP-B Programmatic Requirements?

• For non-AO missions, projects must provide:– A “high” development (thru launch) cost estimate and

associated probability to achieve that cost– A “low” development cost estimate and associate probability

to meeting that cost– Cost estimates must include all cost elements, including

those for which the project managers is not directly responsible, e.g. the launch vehicle and external partners contributions

– High and low schedule durations (thru launch) and associated probabilities of meeting those durations

– A high level schedule of activities associated with the low and high life cycle cost estimates

– High and low life cycle cost estimates– Year-by-year required resources to meet both the low and

high estimates

11

Programmatic Expectations at KDP-B

Non-AO Missions (Concluded)– Basis of Estimates. Examples include:• How the cost and schedules were derived• How risks (known and unknown were addressed in the

estimates)• How the year-by-year resources were determined

• For AO missions, the mission directorate, or designated entity, will provide the same information and calculations specified for the Non-AO missions to the decision authority

• Expect to set budget at higher cost range number, but set development cost and schedule estimates at lower numbers– UFE held by program or mission directorate– AO mission development and life cycle costs based on

winning bid

12

Status of Program and Project JCLs

• During past two years, the following JCLs were completed and presented at PMCs– LDCM, MMS, MSL, SOFIA, GPM*, MAVEN,

LADEE, NuStar, and IRIS,

• Following JCLs are still in process– JWST

• Pilot projects at KDP-B– CxP (cost loaded schedule)– SMAP (parametric)– MAVEN (parametric)

* GPM used the CEPA technique; remainder used BURLS

13

JCL Implementation Issues

• Most projects had not assembled multiple schedules into a top-level electronically integrated schedule. Required extensive time and effort

• Most projects had not quantified their discrete risks nor were they linked to their cost or schedule estimates. Required extensive time and effort

• Few projects had a basis for their cost estimates. Most were based on assigned budgets. Costs were usually not separated into fixed and variable

• Most projects had lower level schedule of activities detail than costs• Project personnel were not staffed or trained to perform the analysis or use the

tools• Use of low cost COTS scheduling and simulation software were inadequate to

perform the task– Microsoft Project with @Risk took too long to perform computations

• Projects and Standing Review Boards had relatively narrow opinions about discrete risks (Known-knowns)

• Consultants were less than successful in convincing projects or SRB members to add other uncertainty (due to known and unknown-unknown risks)

In spite of these issues, most projects developed improved plans and explained them better than without the JCL requirement

14

JCL Lessons Learned

• Improve project control functions– Ensure projects develop credible cost and schedule basis of

estimates– Ensure projects maintain an integrated master schedule that

reflects costs and schedule risks, i.e., maintain resource-loaded schedules

– Ensure projects continuously quantify and monitor risks and their impact on their plans

• Provide projects with historical cost and schedule uncertainly data – To assembly level if possible, but to the subsystem level at a

minimum

• Provide projects with launch vehicle price and schedule uncertainty data

• Provide projects with tools that will permit required turnaround times

15

Action Completed and Planned

• Completed– Created near term optimization techniques for MS Project/@Risk

• Schedule Shorting Tool• Risk Scenario & Automation Tool (R-SAT)

• In-Process– Developing rapid stand alone simulation tools to be used in

conjunction with MS Project (replaces @Risk)• DICE• JACS

– Collecting cost and schedule uncertainties data – Collecting LV price and schedule uncertainty data– Brainstorming program control revitalization plan

• To Be Done:– Deploy new simulations tools– Disseminate LV uncertainty data– Disseminate cost and schedule uncertainty data and educate

project staffs on their use– Decide on program control improvements and implement

Backup

17

Detailed Program and Project Status on JCLs

18

Detailed Program and Project Status on JCLs (Concluded)