2-27-13 The Drive Toward Data Driven Estimating Update …€¦ · The Drive Toward “data...

1

The Drive Toward

“data driven” Estimating

Boeing/Lockheed Martin/Northrop GrummanOriginal March 2011

Updated February 2013

BOE Training Material

2

Background/Objectives

Background: There is a significant industry effort to increase the quality of contractor estimating processes. The industry has applied the term “data driven” estimating to this new method of developing and documenting our basis of estimates(BOEs). “data driven” Estimating methods have certain characteristics that are starting to emerge as standards within our industry.

Objectives:— What is driving our industry toward the use of “data driven” estimating

methods and why is this a business imperative

— Review the current primary estimating methods

— Be able to identify the characteristics of good “data driven” estimating processes and how they are represented in our BOEs

— What are objective-scope-measures and how are they used in a ”data driven” estimate

— Provide examples of “data driven” versus non-data driven BOEs

3

“data driven” Estimating - Contents

BOE - Description

“data driven” Estimating - Definition

The Future – Fundamentals of data driven Estimating

Developing Relationships from History – The Cost Estimating Relationship (CER) Process

“data driven” Estimate Examples – We are the Customer

Summary

Backup Charts/Supplemental Resources

4

Purpose of a BOE

Documents and substantiates effort required to perform tasks — Documents basis and assumptions— Details understanding of requirements— Conveys credibility by clearly defining tasks and assumptions— Demonstrates that estimates are credible, defendable and realistic

(historical data)

Provides evidence that estimate is consistent with other proposal volumes

Time-phased resources consistent with IMP/IMS Allocates skill mix appropriate for task Aligns with Work Breakdown Structure (WBS), SOW, and other

product structures Meet customer funding, timing, and/or scheduling constraints

5

BOEs – The Components A BOE consists of the following main elements:

1. Work Element ID – WBS, SOW, Location, CLIN, NRE/Rec…2. Description of the products or services delivered and the tasks

to be performed3. Estimating method, rationale, assumptions, and computations

used to develop the estimate— Method of quoting

4. Relevant historical data and comparability of the historical data to the current program— Explanation of any adjustments made to historical cost or

programmatic data (i.e., use of modifiers)

5. Estimated labor resources to perform the defined tasks6. Expenditure profile (time-phasing) of labor resources7. Other elements as required by the cost proposal instructions

6

BOE Estimating Methods Labor estimates are developed using one or these typical

estimating methods— Judgment – based on subject matter expert (SME) experience with similar

efforts. — Level-of-Effort/Support - uses resource-based estimates that are often

based on a predetermined level of support for a given period of time — Analogy/Comparison Method – estimate based on historical frame of

reference from a similar effort with adjustments for programmatic or technical differences

— Standard Time Estimating - Labor Standards and Realization/ Performance Factors - time necessary to complete a defined element of work following a prescribed (recognized) technique or method.

— Unit Method – generally a simple ratio (not always statistically validated)— Follow-on Based On Actuals – extrapolation of actuals (CEAC, production)— Parametric Cost Models – use of calibrated, internally developed, or

commercially available cost models — Cost Estimating Relationships (CERs) - equations developed based on

historical, statistically correlated relationships

— Note: Material or Other Direct Costs may be estimated by data such as quotes or catalogs . See FAR for specific cost element and degree of competition requirements

7

Customer Expectations

Customers/DCAA/DCMA expect contractors to begin adopting improved estimating methods to support TINA-compliant proposals and other estimating products

Least Preferred

Judgment

Most Preferred

“Fact Based”Estimating Methods

Move To Fact Based ASAP

8

BOE Methodologies – Credibility Scale

Historical resource based estimates are easier to support than judgmental type estimates.

Customer Credibility LaborHigh Follow-on Projections

based on ActualsAnalogy/Comparison

CER’sParametric Estimates /

Cost ModelsStandards

Unit MethodsLevel Of Effort

Low Judgment

9

Customer Consequences – Poor Estimating

Failure to be responsive to customer estimating processes could be grounds for— Non-compliance with RFP or government regulations— Rejection of proposal and possibly defective pricing— Estimating System Inadequacy

Penalties for bad BOEs include:— Questioned costs— DCAA/Government Deficiency Report— Negotiation losses— Negative factor in competitive evaluations— Defective pricing and fraud prosecution— Suspension of bidding and debarment— Holdbacks

10

“data driven” BOEs – Basic Definition Fact Based Estimating: Estimating processes/BOEs that

provide the supporting information and rationale necessary for accurate, credible and traceable estimates

Information includes:— Use of defined estimating methods

— Historical data from past programs

— Use of objective scope measures and complexity factors where appropriate

Rationale addresses:— Complete description of the estimating approach

— Explanation of the applicability of the historical information

Estimating process must be documented, consistent and repeatable

11

Fact Based BOEs – The Fundamentals

1. Estimate develop using a defined and documentedestimating processes

2. Methods rely on scope measures (attributes) and complexity factors (parametrics) – when applicable

3. Complete estimate development process is explained and documented in the BOE — Professional judgments, modifiers/adjustment factors, and historical

context is adequately explained— Process is consistent and repeatable

4. Estimate relies on relevant historical data and context that is accessible and verifiable — Shows use of latest and most current relevant data— Supports all resource estimates with verifiable data

5. Estimating processes are monitored for learning and improvement as required to support future estimates

12

Objective Work-Scope Measures (OSM)

Scope Measures: estimating predictive measures – technical measures relating to the size of the effort (attributes)— Generally physical and technical measures— Sized at different product levels - system, subsystem, box, etc. — Example:

— Work Products: # drawings by type, sheet counts, # documents & sheet count, SLOC by category, # requirements, # reviews, # test & type, weight, quantities, etc.

Traits/Complexity Factors: characteristics that can improve the accuracy of relationships developed for the OSMs — Example:

— Scope Measures = Drawings— Traits: (numeric and descriptive): # Drawings, Type of Drawing; Drawing

Heritage; Item Next Higher Assembly/Element; Number of Sheets; # of Revisions

— Example: — Scope Measures = Source Lines of Code (SLOC)— Traits: Software Language; New SLOC; Modified SLOC; Reuse SLOC; Actual

Total Delivered SLOC, Deleted SLOC, Auto Generated Code, etc

13

Why Use Scope Measures and Traits in BOEs?

Scope Measures - help to scope the size of the effort to be estimated — Objective quantitative measures of scope to be performed — Directly correlate to (predictors) resource estimates for work products and

tasks — Understanding the size (scope) of the proposed effort is critical to

developing an accurate estimate (BOE) — The BOE author should explain the derivation, source, and application of

the scope measures in the BOE

Traits/Complexity Modifiers – address unique characteristics of the scope measures that can affect required effort to produce— Capture the effects of technical and programmatic differences between

historical programs and the current proposed effort.— The “produceability” can vary widely for any scope measure - traits can

be used to explain the differences— Sometimes requires judgment

14

Objective Scope Measures - Examples

Source Lines of Code (SLOC)

Function points

Computer Software Configuration Item (CSCI)

Number of drawings

Power requirements

Number/type of subsystems, subassemblies, components

Volume (Dimensions)

Weight

Number/type of reviews

Number of end items

Number/type of tools

Number of Contract Data Requirements Lists (CDRLs)

Number of Subcontract Data Requirements Lists (SDRLs)

Number of system requirements

Number of Interface Control Drawings ( ICDs)

Number of payloads

Number of program segments

Number of internal interfaces

Number of external interfaces

Number of risks

Number of architectural trades

Candidate Estimating Attributes

Software

Hardw

are

System

s Engine

ering

15

Traits/Complexity Factors - ExamplesDesign reuse

Performance factors

Complexity

Reuse and pedigree

New technology

Components/functional complexity

Schedule

Requirements maturity

Technology Readiness Level (TRL) requirements/maturity path

Multi‐functional analyses

Testing (qualification; prototype‐protoqualification; delta qualification; static; modal)

Assembly, Integration and Test (AI&T))

Training requirements

Mission profile and environments

Mission design (data rates, pointing stabilities, Technical Performance Measurements (TPMs))

Oversight and analyses of subcontracted elements

Risk

Program maturity/life cycle

Complexity

Schedule

Heritage

Required documentation

Estimating Modifier

Software

Hardw

are

System

s Engine

ering

16

Explaining Adjustment to a BOE

Provided below are some examples of adjustments that may be required to improve the accuracy of the estimate. The list is not meant to be totally inclusive of adjustments an estimator has available in their tool box.— Technical Complexity ~ Labor Estimate

— Improvement Curves ~ Labor Estimate

— Schedule adjustments ~ Labor Estimate

— Quote Decrement ~ Material Estimate

— Quantity Adjustment ~ Material Estimate

— Escalation ~ Material Estimate

— Exchange Rate ~ Material Estimate

In order to use an objective or subjective adjustment, an estimator must explain it thoroughly in the BOE.

17

Using Historical Data in a BOE

Use of Historical Data Must Address:— Source of data – generally from the organization’s certified

historical data repositories— Complete identification of the nature, source and details of

relevant historical data

Ensure compliance with A.C.T.T.R. by documenting the following: — Actual historical hours— Charge numbers included— Technical data used— Time frame of historical data— Repository or source of historical data

18

Developing a Relationship from Historical Data

The statistical analysis process used in the development of a cost estimating relationship (CER) is documented in numerous industry and statistical sources

The objective of the analysis is to validate that a significant statistical relationship exists for the CERs and to demonstrate that the CERs accurately predict effort or costs

CER Strengths

— Past performance reflective of future costs

— Lean and credible estimating method in preparing BOEs

— More scientific and objective than judgement

19

Processs Step:1. Identify Need2. Establish Estimate Structure3. Identification of Effort/Cost Drivers4. Selection of Equation Form5. Selection of Population6. Data Collection7. Data Normalization8. Generate the Cost Estimating Relationship (CER)9. Validate the Results10. Document

Developing a “data driven” Relationship (CER)

numeratordenominator

item to estimateknown item

poolbase

dependent variableindependent variable

20


Regression Analysis: Develop mathematical relationship to make inferences about a future probabilistic event based on a sample of historical data.

If relationship exist between two or more variables, such as weight and hours of effort/cost required, regression analysis is a powerful tool for the cost estimator/analyst.

Regression analysis consists of curve fitting and the analysis of the descriptive and inferential statistics that result from the fit of the data.

21


Y

X

Y

XPositive Linear Relationship Positive Increasing

Curve-linear Relationship

Y

X

Y

XNegative Linear Relationship

Y

XPositive Decreasing Curve-linear Relationship

Negative Increasing Curve-linear Relationship

xbby 10

xbby 10

baxy )1b(

baxy )1b0(

baxy )1b(

22

Preparation of the Historical Data (CER) Step 1: Select the dependent variable to be estimated (e.g. results =

hours, cost)

Step 2: Determine the auditable source of historical data (both cost and technical measures)

Step 3: Define the population(s), subpopulations, and sample(s) of data considering methods of collection (sites, programs, stratifications)

Step 4: Select logical and measurable independent variables and determine their auditable sources of data (SLOCs, Drawings, Pages, Hours)

Step 5: Extract the data and verify its accuracy and completeness using “tests of reasonableness”

Step 6: Prepare the data for use in EXCEL or a statistical tool to perform the regression analysis

Step 7: Test the resulting relationship for accuracy

23

Interpretation Decision Criteria of Statistical Results

Statistical Measure Purpose of Measure Target Value for Good Fit

Correlation Coefficient Measures the fit of the CER model. Gives a quick look at the percentage of the total variation that has been explained by the regression line.

R2 ≥ .80

.50 < R2 < .80

.50 ≤ R2

Coefficient of Variation Measures the target goal for predictability within one standard error (68% of the data). The C.V. is calculated at the mean of Y, allowing the comparison of one CER to another. Calculated from the Standard error which measures the dispersion of the data.

C.V. ≤ 50%

50% < C.V. < 100%

C.V. ≥ 100%

Usability Criterion Judges reliability or usefulness of the CER measured on data points in analysis determining confidence of the results based on F-test when R2 should be .80.

Population ≥ 10

6 ≤ Pop < 10

Pop < 6

Meets guidelines

Below guidelines, acceptable with proper understanding

Below guidelines

24

Data Driven Estimating - Examples

We are all confronted in our daily lives with estimates for products and services— E.g. Home remodel, auto repair

We evaluate the credibility of these estimates based upon the content of the estimate

We need to apply this customer mentality when we review our own proposal cost volumes and BOEs

Example: Everyone has dealt with an automobile repair estimate that we struggle to feel comfortable with. Evaluate

the following BOEs for automobile repair service – how do we as consumers know that we are getting a fair price for these

services

25

D&G Auto – Poor, Fair or Good BOECriteria

• Description of the products or services delivered and the tasks to be performed

• Estimated labor resources to perform the defined tasks

• Time-phasing of labor resources

• The estimating method, rationale, assumptions, and computations used to develop the estimate

• Identification of the relevant historical data and comparability of the historical program to the current program

• Explanation of any adjustments made to the historical data or estimate (i.e., use of modifiers)

26








27








28

“data driven” Estimating – The Challenges Ahead

Focus – Improve estimating products across all estimating stake-holders (e.g. BOEs, Earned Value, Cost Analysis and Program Execution (CAPE), Independent Cost Evaluation (ICE), Business Case Analysis)

The debate as we evolve to our future –— At what level will we be generating estimates (e.g. subsystem, assembly)— What scope measures (OSM) will we be using — How will we document our estimating processes— Same or different approaches for firm business, ROMs/NTE, EAC/ETC?— What measures/approaches will be cost credible with our customers

Historical databases, processes and strategies need to be flexible as we evolve toward the future “improved” estimating products— Estimating, Cost Modeling and Engineering teams working together— As the desired characteristics of these high quality estimating methods

evolve – our methods must evolve also.

Immediate needs:— Documented estimating process — Good historical data — Libraries of BOEs demonstrating data driven estimating approaches

29

Backup Charts/Supplemental Resources

30

Judgment: Least preferred, least credible estimating method –very subject to bias— Generally not based upon historical data, scope measures or other sound

estimating approaches

Relies on Subject Matter Expert (SME) estimate’s of small work packages aggregated to complete hardware, software, etc tasks— State author’s experience from which to make an estimate

— Programs worked; Years in industry— Decompose effort into well defined tasks and subtasks

Disclose if historical methods were investigated and not used, and why they were not used

Estimating Method - Judgment

Rarely are we asked to quote something that does not have some similar historical likeness !

31

Estimating Method - LOE Level of Effort (LOE) - Use when customer specifies effort to

be provide over a stated period of time on work that can be stated only in general terms— Requirement for specific skills/talents that tend to be headcount driven— Used when product cannot be well defined

Contract is for delivery of hours not product. Payment based on effort expended not results achieved.— Restricted LOE Contracts — Statement of Work (SOW) that specifically calls out LOE by customer

What is NOT LOE— Level loading of labor not specifically directed by customer— Staffing a position (i.e. Program Manager) as 160 hrs/month during

period of performance on a completion type contract

32

Estimated Cost (New Design) = Historical Cost (Similar Design) + Complexity or Scaling Factor

Estimating Method – Analogy/Comparison

Analogy/Comparison: This method is based on historical context or frame of reference from a similar effort. Compares items being estimated to items of similar configuration and known cost to produce estimate— Includes use of historical data— Addresses differences between historical data and proposed solution

being estimated— May use relevant estimating attributes and complexity or adjustment

factors (modifiers) to reflect programmatic, technology, requirements or technical differences

— Clearly document in BOE why and how factors are used and validated

33

Estimating Method – Standard Time

Labor standard - time necessary to complete a defined element of work following a prescribed method, for an average (qualified) worker working at a 100% efficiency with no hindrances, breaks, delays or fatigue over extended periods of time— Realization/Performance factors recognize actual experience measured

against the standards

— This method is most often used in manufacturing or production related estimates.

— Examples:

— Manufacturing operation: Time required to complete a drilling operation on a product adapter

— Car servicing: Factory maintenance manual sets a standard number of minutes for each type of repair

34

Estimating Method – Unit Method

Unit Method: Relies on an accumulation of past experience divided by cost driver to produce cost per unit

— Example: Labor cost of developing and fielding entire ground station and costing using number of ground stations to be fabricated

— Example: ABC widget has 11 square feet of surface area. Actual design hours for XYZ widget (10 square feet of surface area) were 80 hours. 80 hours/10 square feet = 8 hours per square foot. Therefore ABC widget will take 88 (8 X 11) hours to design

A Simple Ratio (not statistically validated) of X to Y Is A Unit Method

35

Estimating Method – Follow-on Effort (Actuals)

Follow-on based upon program actuals— Commonly used when developing an estimate of labor or

material for continuing efforts (usually)— Production hardware— Spare parts, Repairs, or Retrofit— Sustaining Engineering— Estimate At Complete (EAC)

Definition: An estimate that uses historical data (hours, material dollars, weights, etc.) to project continuing future efforts of the same product line, process or part.

36

Estimating Method – Parametric Model Parametric Model: Utilizes mathematical calibrated models

that derive project cost or effort from attributes / metrics of project— Source lines of code (SLOC), function points (FPs), COTS

product characteristics, Weight, Size— The measurement is based on the technical, physical, or other

end item characteristics Facilitates alignment of program baselines

(e.g. IMS and Staffing plans) Customer RFP often dictates use Utilized by customer to evaluate BOEs and bid overall Used by ICE to evaluate bid position & performance risk Performed by an experienced cost modeler Commercial parametric models include: PRICE, SEER

37

Estimating Method – Parametric Model, Cont…

Parametric Model: is a group of cost estimating relationships used together to estimate entire cost proposals or significant portions thereof.— These models are often computerized and may include many inter-

related CER’s both cost-to-cost and cost-to-non-cost estimating techniques estimating relationships to predict complex

— Some models use a very limited number of independent estimated values and a series of Parametric inter-related cost-to-noncost estimating techniques relationships to predict complex proposal cost structures

38

Estimating Method - CER Cost Estimating Relationships: Statistical estimating

methods that produce mathematically fitted functions — Developed by mathematically relating effort or time estimates to

OSM, traits or other feature of product or manufacturing environment

Relationship may be based on single or multiple measures (independent variables) representing estimating attributes, complexity measures, or other measures of effort. — The results from application of the CERs (dependent variable) is

frequently a labor estimate expressed in hours/staff-months or an estimate of time in days/weeks/months.

Example: Based on actuals of four projects (design hrs/ sq ft) using linear regression, this estimating

relationship is created:Design Hours = 9.32 * sq ft - 14.3

39

Hardware Cost Modeling

Estimate system acquisition costs based upon:— Quantitative parameters such as complexity, quantity,

weight, and size; and

— Quantitative parameters such as environmental specification, type of packaging, and level of integration; and schedule parameters such as months to first prototype, manufacturing rate, and amount of new design

— Electronic hardware items are characterized by the electronic application and type of componentry

— Mechanical and structural elements can be described in terms of their construction, type of material, functionality, machinability and manufacturing process

40

“data driven” Estimating

Why Basis of Estimate (BOE) Focus?

BOE - Description

“data driven” Estimating - Definition

Fundamentals of “data driven” Estimating

“data driven” Estimate Examples – We are the Customer

Non-Proprietary BOE Training Material Available to All

41

Background/Objectives

Background: There is a significant industry effort to increase the quality of contractor estimating processes. The industry has applied the term “data driven” estimating to this new method of developing and documenting our estimates/BOEs. “data driven” Estimating methods have certain characteristics that are starting to emerge as standards within our industry.

Objectives:— What is driving our industry toward the use of data driven estimating

methods and why is this a business imperative

— Be able to identify the characteristics of good data driven estimating processes and how they are represented in our BOEs

— Provide examples of “data driven” versus non-data driven BOEs

— Raise the bar by providing access to non-proprietary BOE training

42

Responsibilities

4

Mapping of DFARS 252.215.7002 Cost Estimating System Requirements to Major Estimating System Components

Roles

Training

Estimate Data Sources

Estimate Preparation

Estimate Review

Estimate Approval

Internal Controls

Well Supported Data Driven Estimating in Key to Estimating System Health

1. Establish clear responsibility for preparation, review, and approval of cost estimates and budgets.

2. Provide a written description of the organization and duties of the personnel responsible for preparing, reviewing, and approving cost estimates and budgets.

3. Ensure that relevant personnel have sufficient training, experience, and guidance to perform estimating and budgeting tasks in accordance with the established procedures.

4. Identify and document the sources of data and the estimating methods and rationale used in developing cost estimates and budgets.

9. Provide for the use of historical experience, including historical vendor pricing information, where appropriate.

11. Integrate information available from other management systems.

15. Provide procedures that ensure subcontract prices are reasonable based on a documented review and analysis provided with the prime proposal, when practicable.

5. Provide for adequate supervision throughout the estimating and budgeting process.

6. Provide for consistent application of estimating and budgeting techniques.

7. Provide for detection and timely correction of errors.

8. Protect against cost duplication and omissions.

10. Require use of appropriate analytical methods.

14. Provide procedures to update cost estimates and notify the Contracting Officer in a timely manner throughout the negotiation process.

12. Require management review, including verification of the estimating and budgeting policies, procedures, and practices .

13. Provide for internal review of, and accountability for, the acceptability of the estimating system, including the budgetary data supporting indirect cost estimates and comparisons of projected results to actual results, and an analysis of any differences.

16. Provide estimating and budgeting practices that consistently generate sound proposals that are compliant with the provisions of the solicitation and are adequate to serve as basis to reach a fair and reasonable price.

17. Have an adequate system description, including policies, procedures, and estimating and budgeting practices, that comply with the Federal Acquisition Regulation and Defense Federal Acquisition Regulation Supplement.

43

DFARS 252.215.7002 Cost Estimating System Requirements4. Identify and document the sources of data and the estimating methods

and rationale used in developing cost estimates and budgets.

9. Provide for the use of historical experience, including historical vendor pricing information, where appropriate.

11. Integrate information available from other management systems.

6. Provide for consistent application of estimating and budgeting techniques.

10. Require use of appropriate analytical methods.

15. Provide procedures that ensure subcontract prices are reasonable based on a documented review and analysis provided with the prime proposal, when practicable. Provide for adequate supervision throughout the estimating and budgeting process.

Estimate Review

Estimate Data Sources

Estimate Preparation

44

Purpose of a BOE

Documents and substantiates effort required to perform tasks — Documents basis and assumptions— Details understanding of requirements— Conveys credibility by clearly defining tasks and assumptions— Demonstrates that estimates are credible, defendable and realistic

(historical data)

Provides evidence that estimate is consistent with other proposal volumes

Time-phased resources consistent with IMP/IMS Allocates resources such as skill mix appropriate for task Aligns with Work Breakdown Structure (WBS), SOW, and other

product structures Meet customer funding, timing, and/or scheduling constraints

45

BOEs – The Components A BOE consists of the following main elements:

1. Work Element ID – WBS, SOW, Location, CLIN, NRE/Rec…2. Description of the products or services delivered and the tasks

to be performed3. Estimating method, rationale, assumptions, and computations

used to develop the estimate— Method of quoting

4. Relevant historical data and comparability of the historical data to the current program— Explanation of any adjustments made to historical cost or

programmatic data (i.e., use of modifiers)

5. Estimated labor and ODC resources to perform the defined tasks

6. Expenditure profile (time-phasing) of resources7. Other elements as required by the cost proposal instructions

46

BOE Estimating Methods Estimates are developed using one or more of these typical

estimating methods— Judgment – based on subject matter expert (SME) experience with similar

efforts. — Level-of-Effort/Support - uses resource-based estimates that are often

based on a predetermined level of support for a given period of time — Analogy/Comparison Method – estimate based on historical frame of

reference from a similar effort with adjustments for programmatic or technical differences

— Standard Time Estimating - Labor Standards and Realization/ Performance Factors - time necessary to complete a defined element of work following a prescribed (recognized) technique or method.

— Unit Method – generally a simple ratio (not always statistically validated)— Follow-on Based On Actuals – extrapolation of actuals (CEAC, production)— Parametric Cost Models – use of calibrated, internally developed, or

commercially available cost models — Cost Estimating Relationships (CERs) - equations developed based on

historical, statistically correlated relationships(See full training package for more details)

47

Customer Expectations

Customers/DCAA/DCMA expect contractors to begin adopting improved estimating methods to support TINA-compliant proposals and other estimating products

Least Preferred

Judgment

Most Preferred

“Fact Based”Estimating Methods

Move To Fact Based ASAP

48

BOE Methodologies – Credibility Scale

Historical resource based estimates are easier to support than judgmental type estimates.

Customer Credibility LaborHigh Follow-on Projections

based on ActualsAnalogy/Comparison

CER’sParametric Estimates /

Cost ModelsStandards

Unit MethodsLevel Of Effort

Low Judgment

49

Customer Consequences – Poor Estimating

Failure to be responsive to customer estimating processes could be grounds for— Non-compliance with RFP or government regulations— Rejection of proposal and possibly defective pricing— Holdbacks

Penalties for bad BOEs include:— Questioned costs— DCAA/Government Flash Report— Negotiation losses— Negative factor in competitive evaluations— Defective pricing and fraud prosecution— Suspension of bidding and debarment

50

“data driven” BOEs – Basic Definition Fact Based Estimating: Estimating processes/BOEs that

provide the supporting information and rationale necessary for accurate, credible and traceable estimates

Information includes:— Use of defined estimating methods

— Historical data from past programs

— Use of objective scope measures and complexity factors where appropriate

Rationale addresses:— Complete description of the estimating approach

— Explanation of the applicability of the historical information

Estimating process must be documented, consistent and repeatable

51

Fact Based BOEs – The Fundamentals

1. Estimate developed using a defined and documentedestimating processes

2. Methods rely on scope measures (attributes) and complexity factors (parametrics) – when applicable

3. Complete estimate development process is explained and documented in the BOE — Professional judgments, modifiers/adjustment factors, and historical

context is adequately explained— Process is consistent and repeatable

4. Estimate relies on relevant historical data and context that is accessible and verifiable — Shows use of latest and most current relevant data— Supports all resource estimates with verifiable data

5. Estimating processes are monitored for learning and improvement as required to support future estimates

52

Objective Work-Scope Measures (OSM)

Scope Measures = estimating predictive measures – technical measures relating to the size of the effort (attributes)— Generally physical and technical measures— Sized at different product levels - system, subsystem, box, etc. — Example:

— Work Products: # drawings by type, sheet counts, # documents & sheet count, SLOC by category, # requirements, # reviews, # test & type, weight, quantities, etc.

Traits/Complexity Factors: characteristics that can improve the accuracy of relationships developed for the OSMs — Example:

— Scope Measures = Drawings— Traits: (numeric and descriptive): # Drawings, Type of Drawing; Drawing

Heritage; Item Next Higher Assembly/Element; Number of Sheets; # of Revisions

— Example: — Scope Measures = Source Lines of Code (SLOC)— Traits: Software Language; New SLOC; Modified SLOC; Reuse SLOC; Actual

Total Delivered SLOC, Deleted SLOC, Auto Generated Code, etc

53

Why Use Scope Measures and Traits in BOEs?

Scope Measures - help to scope the size of the effort to be estimated — Objective quantitative measures of scope to be performed — Directly correlate to (predictors) resource estimates for work products and

tasks — Understanding the size (scope) of the proposed effort is critical to

developing an accurate estimate (BOE) — The BOE author should explain the derivation, source, and application of

the scope measures in the BOE

Traits/Complexity Modifiers – address unique characteristics of the scope measures that can affect required effort to produce— Capture the effects of technical and programmatic differences between

historical programs and the current proposed effort.— The “produceability” can vary widely for any scope measure - traits can

be used to explain the differences— Sometimes requires judgment

54

Explaining Adjustment to a BOE

Provided below are some examples of adjustments that may be required to improve the accuracy of the estimate. The list is not meant to be totally inclusive of adjustments an estimator has available in their tool box.— Technical Complexity ~ Labor Estimate

— Improvement Curves ~ Labor Estimate

— Schedule adjustments ~ Labor Estimate

— Quote Decrement ~ Material Estimate

— Quantity Adjustment ~ Material Estimate

— Escalation ~ Material Estimate

— Exchange Rate ~ Material Estimate

In order to use an objective or subjective adjustment, an estimator must explain it thoroughly.

55

Using Historical Data in a BOE

Use of Historical Data Must Address:— Source of data – generally from the organization’s certified

historical data repositories— Complete identification of the nature, source and details of

relevant historical data

Ensure compliance with A.C.T.T.R. by documenting the following: — Actual historical hours— Charge numbers included— Technical data used— Time frame of historical data— Repository or source of historical data

56

“data driven” Estimating - Examples

We are all confronted in our daily lives with estimates for products and services— E.g. Home remodel, auto repair

We evaluate the credibility of these estimates based upon the content of the estimate

We need to apply this customer mentality when we review our own proposal cost volumes and BOEs

Example: Everyone has dealt with an automobile repair estimate that we struggle to feel comfortable with. Evaluate

the following BOEs for automobile repair service – how do we as consumers know that we are getting a fair price for these

services

57








58








59








60

“data driven” Estimating – The Challenges Ahead

Focus – Improve estimating products across all estimating stake-holders (e.g. BOEs, EV, CAPE, ICE, Business Case Analysis)

The debate as we evolve to our future –— At what level will we be generating estimates (e.g. subsystem,

assembly)— How will we document our estimating processes

Historical databases, processes and strategies need to be flexible as we evolve toward the future “improved” estimating products— Estimating, Cost Modeling and Engineering teams working

together— As the desired characteristics of these high quality estimating

methods evolve – our methods must evolve also.

Immediate needs: — Documented estimating process — Good historical data — Libraries of BOEs demonstrating data driven estimating

approaches

61

Back Up

62

Estimated Cost (New Design) = Historical Cost (Similar Design) + Complexity or Scaling Factor

Estimating Method – Analogy/Comparison

Analogy/Comparison: This method is based on historical context or frame of reference from a similar effort. Compares items being estimated to items of similar configuration and known cost to produce estimate— Includes use of historical data— Addresses differences between historical data and proposed solution

being estimated— May use relevant estimating attributes and complexity or adjustment

factors (modifiers) to reflect programmatic, technology, requirements or technical differences

— Clearly document in BOE why and how factors are used and validated

63

Judgment: Least preferred, least credible estimating method –very subject to bias— Generally not based upon historical data, scope measures or other sound

estimating approaches

Relies on SME estimate’s of small work packages aggregated to complete hardware, software, etc tasks— State author’s experience from which to make an estimate

— Programs worked; Years in industry

— Decompose effort into well defined tasks and subtasks

Disclose if historical methods were investigated and not used, and why they were not used

Estimating Method - Judgment

Rarely are we asked to quote something that does not have some similar historical likeness !

64

Estimating Method - LOE Level of Effort - Use when customer specifies effort to be

provide over a stated period of time on work that can be stated only in general terms— Requirement for specific skills/talents that tend to be headcount driven— Used when product cannot be well defined

Contract is for delivery of hours not product. Payment based on effort expended not results achieved.— Restricted LOE Contracts — SOW that specifically calls out LOE by customer

What is NOT LOE— Level loading of labor not specifically directed by customer— Staffing a position (i.e. Program Manager) as 155 hrs/month during

period of performance on a completion type contract

2-27-13 The Drive Toward Data Driven Estimating Update …€¦ · The Drive Toward “data...

Documents

Transcript of 2-27-13 The Drive Toward Data Driven Estimating Update …€¦ · The Drive Toward “data...