2015 DOE ACQUISITION AND PROJECT MANAGEMENT WORKSHOP

‘EVMS – FROM DATA TO DIAGNOSIS’

David Kester DCMA EVM Policy and Strategy

March 24, 2015

CONTENTS ***

1. DCMA At a Glance 2. Current EVMS Environment 3. The PM Connection 4. DCMA EVMS Mission Transformation 5. All Things Data 6. Mapping the EVMS Genome – EIA-748 Guideline 9 7. EVMAS – The Technology Solution 8. Role of the ACO in the EVMS Mission

2

DCMA AT A GLANCE ***

Vision One team, one voice delivering global acquisition insight that matters. Mission We are the independent eyes and ears of DoD and its partners, delivering actionable acquisition insight from the factory floor to the front line… around the world.

3

DCMA AT A GLANCE ***

• Number of civilian personnel: 11,473 • Number of military personnel: 528 • Number of contract management offices: 47 • Number of contractors: 20,155 • Number of active contracts: 353,200 with a total obligated

value of more than $1.9 trillion. • Number of countries with operations: 26 • Number of service lines: 15 - Earned Value Management Systems

4 Data as of September 2014

TRANSFORMING FEDERAL PROCUREMENT

*** A New Way Of Governing Is Emerging . . .

5

DEFENSE SPENDING AS % OF GDP

6 Source: usgovernmentspending.com

VALUE FOR MONEY ***

. . . We Measure What We Value, and Value What We Measure

7

VALUE FOR MONEY ***

• Change The Way The Department Does Business By Changing The Way The

Department Thinks About EVMS

. . . Is EVMS A Burden Or Necessity?

8

Strategic Level Inter-Relationships Critical Path to DCMA EVMS Operations

USD (AT&L) PARCA

DoD EVM Policy DPAP

DoD Contractor Business Systems Rule

Office of Management &

Budget

DCMA PMI-V

EVMS Policy

DCMA-O DCMA-I DCMA-SP

EVMS Execution

ANX-205 EVMAA

IPMR Guide

DFARS 252.234-7002

EVMS

DFARS 242.302 Contract Admin

Functions

DoDI-5000.02 EVM Implementation

Policy

OMB Circular A-11 EVMS ANSI-748

Compliance

Title 48 - FAR Subpart

34.2

INST-210 SSI

INST-208 CRI

DFARS 234.201 EVMS Policy CFA

DFARS 234.203 Solicitation

Provisions & Contract Clauses

Execution

DFARS 252.234-7001

Notice of EVMS

ANX-412 CIA

INST-205 MPS

INST-131 CBS

DAES

Performance Indicators (Feedback)

INST-214 EVMCP

DFARS EVMS POLICIES

5 PMI EVMS Policies

CMO utilize Contractor EVMS

data To generate

monthly PARS

242.302 Contract Administration Functions (S-71) DCMA has responsibility for reviewing earned value management system (EVMS) plans and for verifying initial and continuing contractor compliance with DoD EVMS criteria. The contracting officer shall not retain this function.

234.201 EVMS Policy (3) The Defense Contract Management Agency is responsible for determining earned value management system compliance when DoD is the cognizant Federal agency.

DoD EVMSG 9

DFARS 252.242-7005

Contractor Business Systems Rule

SUBPART 234.2 - EARNED VALUE MANAGEMENT SYSTEM

*** One Part Management Strategy, One Part

Management Control System

10

PROJECT MANAGEMENT ***

Project Management Institute, Inc. (PMI) defines project management as "the application of knowledge,

skills, tools and techniques to a broad range of activities in order to meet the requirements of a

particular project."

11

5 PHASES OF PROJECT MANAGEMENT

*** 1. Project conception and initiation 2. Project definition and planning 3. Project launch or execution 4. Project performance and control 5. Project close

12

5 PHASES OF PROJECT MANAGEMENT

*** 4. Project performance and control

Project managers will compare project status and progress to the actual plan, as resources perform the scheduled work. During this phase, project managers may need to adjust schedules or do what is necessary to keep the

project on track.

13

• A System of Systems • Value Comes Through A Smart Design and An Even

Smarter Implementation Strategy

• Requires Data That Is Timely, Trusted, And Relevant

Anatomy Of An EVMS

14

• Minimum allowable acceptance conditions:

• Ensures a clear definition of work prior to beginning that work

• Presents a logical plan for accomplishing the work • Provides an objective measure of accomplishments • Early and accurate identification of trends/issues • Accurate picture of contract status • Basis for course correction

. . . Better Data (Ingredients), Better Decisions (Pizza)

Defining EVMS Success

15

1

4

9

2 1

3

9

1 1 2 2

7

2 2 2 1

5

2

5

1 1 1

15

5 5

2 2

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Num

ber o

f Rev

iew

s

Guideline

FY13-14 Total Non Compliant EVMS GLs

• Total of 93 Guidelines were found to be Non Compliant in the 23 major reviews • GL 6 (schedule with network logic) – 9 of 23 reviews (40%) • GL 10 (create work/planning packages) – 9 of 23 reviews (40%) • GL 16 (record direct costs) – 7 of 23 reviews (30%) • GL 27 (revise estimate at completion) – found non compliant in 15 of 23 reviews

(65%) S P

A

EAC

16

∂

BIG DATA - Transforming How DCMA Executes Its DFARS EVMS Compliance

Mission ***

1. Provide Clarity 2. Ensure For Consistency 3. Reduce or Eliminate Costs

17

Initiative 1.3.3 – Alternative Approaches to EVMS Compliance

*** Revise current EVMS Policy and CONOPS to streamline how DCMA conducts the EVMS mission (Compliance, Surveillance, and Analysis), and target cost savings of 20%

18

PRIMARY GOALS ***

• Influence industry partners to field effective (and efficient) EVMS that are compliant with the 32 ANSI-748 guidelines

• Improving Program Management by the way we think about (and use) EVMS

• Strengthen the purchasing power of the Department

19

BLUF – DATA DRIVEN ***

• Rethinking the way in which EVMS compliance (initial) and surveillance (continuing) can be performed to simultaneously test the reliability of core management processes, and to analyze data sets to summarize, detect patterns, and draw conclusions about contract performance • Data-driven compliance to help predict when and where

issues will occur

• Targeted Compliance and Surveillance • Predictor variables identify and localize the issue, optimizing

surveillance and minimizing disruptions • Surveillance driven by data /emerging issues, not by the

calendar • Consistent application regardless of EVMS specialist

20

A Drop of Blood (Data), An Instant Diagnosis – Early Detection,

Prevention

Source: Theranos - Elizabeth Holmes, President and CEO

21

THERANOS = Aligning

Technology with Strategy

PREMISE ***

• The success of the data-driven EVMS compliance approach is reliant upon the availability and quality of the data gathered and the effectiveness of its analysis and interpretation.

• Human error can creep into the EVMS compliance process at any stage of the endeavor and serious consequences can result when they do.

• A data-driven approach to EVMS compliance places value on decisions that can be backed up with data that can be verified.

22

WHY A DATA DRIVEN APPROACH? ***

• Yields a more structured and disciplined analytical approach to evaluating the assessment

• Highlights potential risks in EVMS implementation that might otherwise be unknown

• Better allocates resources where pay-back is greatest • Reduces (and works towards) eliminating unnecessary

and inconsistent data requests and CAM discussions • Projected time (cost) savings for the enterprise (and aligns

with the AT&L Better Buying Initiative)

23

Source: From Keeping Up with the Quants: Your Guide to Understanding and Using Analytics (HBR Press)

A Focus On The Quantitative Analysis Of Systems And Processes

*** Developing numerical (or quantitative) techniques for the

analysis and verification of system properties such as reliability (trustworthiness) and relevance (usefulness) of data

24

It Is A Capital Mistake To Theorize Before One Has Data

- Sherlock Holmes

25

DATA - The New Raw Material ***

• More Data Doesn’t Guarantee Better Decisions • More Data Is Useless If We Aren’t Asking The Right

Questions • Asking The Right Questions Requires Looking At The

Right Data In The Right Way • Knowing How Long Data Must Be Retained • Data Access - 252.234-7002 Earned Value

Management System as prescribed in 234.203 • (f) The Contractor shall provide access to all pertinent records and data

requested by the Contracting Officer or duly authorized representative as necessary to permit Government surveillance to ensure that the EVMS complies, and continues to comply, with the performance criteria referenced in paragraph (a) of this clause

• Data Formatting 26

COMMON DATA FORMATTING ISSUES ***

• Inconsistent data content • Tied to paper based forms and reports that are useless in a digital environment (scanned documents or PDFs) • High volume of non-value added data content • Lots of human effort to collect, enter, sort through, or manipulate the data

27

UN/CEFACT /XML ***

• Allows for the combination of more efficient processing, more accurate searching, and more flexible linking that will significantly reduce costs

• Toolset neutral –program management software vendors are supporting as a common means to import or export data

28

MAPPING THE EVMS GENOME ‘Cracking The Code’

• 32 ANSI-748 Guidelines

• 73 Guideline Attributes • 127 Tests

• 189 Metrics

• Tailored Protocols • Engineering and Development • Construction • Production Assembly

29

30

MAPPING THE EVMS GENOME ***

DCMA role for executing compliance and implementing remains unchanged by the OSD EVMSIG as stipulated : • Section 1.2: “…when DoD is the cognizant federal agency,

the Defense Contract Management Agency (DCMA) is responsible for determining EVMS compliance.”

• Section 1.1: “… agencies and organizations charged with conducting initial and continuing EVMS compliance activities will establish amplifying agency procedures and/or guidance to clarify how the respective agencies and organizations are implementing this guide to include the development of evaluation methods for the attributes associated with each of the 32 guidelines.”

Guidelines 32

EIA-748 GUIDELINE CLASSIFICATIONS ***

2.31 Per Guideline

Attributes* 155 Before

63 After 73 EVMSIG

Test Steps 330 Before

136 After 127 EVMSIG

1.72 Per Attribute

*Attribute - A Quality Or Characteristic Of A Guideline That Makes It Distinct From Other Guidelines

52% Reduction In Attributes 62% Reduction In Test Steps

31

Testing Algorithm ***

Example: Write an algorithm to determine a student’s final grade and indicate whether the grade is ‘passing’ or ‘failing’. The final grade is calculated as the average of four marks.

32

Testing Algorithm ***

Step 1: Input M1,M2,M3,M4

Step 2: GRADE ← (M1+M2+M3+M4)/4 Step 3: if (GRADE < 60) then Print “FAIL” else Print “PASS” end

33

Testing Algorithm Flowchart

PRINT “PASS”

GRADE←(75+65+80+70)/4

IS GRADE<

60

PRINT “FAIL”

STOP

Y N

START

INPUT 75+65+80+70

73

TAKE ACTION

34

ARTIFACT

CALCULATION

METRIC

THRESHOLD

ROOT CAUSE

EIA-748 Guideline 9 Testing Protocol

*** • EVMS Category: Planning, Scheduling, and Budgeting

• EIA Standard Guideline: 2.2d - Authorize and Budget by

Cost Elements

• Intent of Guideline: Establish budgets for authorized work with identification of significant cost elements (labor, material, etc.) as needed for internal management and for control of subcontractors

35

• Distributed Budgets reflect the contractually authorized efforts allocated to WBS elements

• Distributed Budgets include budgets assigned to control accounts during baseline establishment as well as work packages and planning packages

36

• Are the distributed budgets assigned to control accounts planned and identified in terms of the following cost elements:

(1) Direct labor dollars and/or hours? (2) Material and/or subcontract dollars? (3) Other direct dollars?

• Accomplishing the cost account resource plan is the crux of this criterion question. It requires that the budgetary resource plan be devised by element of cost; i.e. labor (in terms of dollars or hours), material (dollars), and other direct charges (dollars)

• This type of budget segmentation adds discipline and integrity to the control account's resource plan and ensures that the correlation is made between every budget dollar and the entity that dollar is supposed to buy

• The work authorization system is what translates the contractor's plans into practice

37

38

EVMAS – Earned Value Management Analytics System Is Intended To Empower

Program Teams (PMs, ACOs, IPTs) ***

39

EVMAS – Aggregate Data To Identify (And Predict) EVMS Non-compliance

*** • Correlation and Causation

• Identifying The Degree And Type Of Relationship Between Any Two (Or More) Quantities, And Then Understanding The Reasons Why

• Predictor For The Likelihood Of An Issue

40

GL 1 0101

0102

0103

GL 17

1701

G

GL 2

0201

0202

GL 18

1801

G

GL 3 0301a

0301b

0301c

0301d

0301e

0301f

0301g

0301h

0301i

0303a

0304a

0304b

GL 19

1901a

1902a

G G G G G G G G G

GL 4 0401

0402

GL 20

2001

2002

GL 5

0501a

0502a

0503a GL 21

2101a

2102a

2102b

2102c

2103a

2104a

2105a

G G G G G G G R

GL 6

0601a

0602a

0603a

0604a

0605a

0606a

0607a

0608a

0609a

0610a

0611a

0612a

0613a

0614a

0615a

0616a

0617a

0618a

0622a

0619a

0620a

0620b

0621a

0621b

GL 22

2201a

2202a

2202b

2202c

2202d

2202e

2202f

2202g

2202h

2202i

2202j

G G G G G G G G R G G G G G G G G G G G G G G G G G G G G G

GL 7

0701a

0702a

0702b

0703a

0704a GL 23

2301a

2304a

2306a

R G

GL 8 0801

0802

0803

0804

0804

0805

GL 24

2401

2403

G G G

GL 9

0901a

0902a

0903a

0905a GL 25

2501a

2501b

2501c

2501d

2501e

2501f

2501g

2501h

2502a

2502b

2502c

2502d

2502e

2502f

2502g

2502h

G G R G G G G G G G G G G G G G G G G G

GL 10 1001

1002

1003

1004

1004

1005

1006

1006

1007

1008

1009

1010

1011

1012

1013

GL 26

2601

2602

2603

G G R G G G R G G G G G R G G

GL 11 1101

1102

GL 27

2701

2702

2702

2702

2702

2703

2704

2705

2706

2707

2708

2709

2710

2711

2712

G G R G R R G R G G G G G G

GL 12 1201

GL 28

2801

GL 13 1301

GL 29

2901

2901

G G

GL 14

1401a

1402a

1403a

1404a

1405a

1406a

1407a

1408a

1409a

1410a

1411a

GL 30

3001a

3001b

3001c

3001d

3001e

3001f

3002a

3003a

G G G G G G G G G G G G G G

GL 15

1501

1502

GL 31

3101

G G

GL 16

1601

1602

1602

1603

1603

1604

1605

1605

1605

1606

1606

1606

1607

1608

1609

GL 32 32

01

3202

3203

3204

3205

3206

3207

G G G G G G G G G G G G G G R

DCMA PROOF OF CONCEPT

41

… An Instant Diagnosis – Early Detection, Prevention

MARKERS

The use of an EVMS maturity model will allow for a contractor to have its methods and processes categorized against a clear set of criteria

Maturity levels could be used to determine ongoing Government surveillance requirements

Mechanism to communicate EVMS risks to customers and stakeholders

Results Matter

Source: Credit Karma

42

Role of ACO in EVMS Mission ***

• Identify EVMS Contract requirements through Contract Receipt and Review (CRR)

• Maintain the Contract Business Analysis Repository (CBAR) • Review EVMS functional reports submitted in accordance

with Contractor Business System (CBS) policy when a system review is conducted or material deficiencies are identified

• Approve/Disapprove EVMS based upon specialist findings • Progress Payment Administration

• Percent of physical completion; Estimates-at-completion 43

• DFARS 252.242-7005 clause added to contracts requiring contractor’s to certify that they have no major (or significant) deficiencies in their systems:

1. Cost Accounting 2. Purchasing 3. Estimating 4. Property Management 5. Earned Value Management System 6. Material Management

• Contracting officers would have the authority to withhold payments on cost reimbursement, incentive-type, time-and-materials, and labor-hour contracts

• Required for one or more 16 high-risk (guidelines 1, 3, 6, 7, 8, 9, 10, 12, 16, 21, 23, 26, 27, 28, 30, or 32), discretionary for the other 16 guidelines

• Enforcement relies on auditors or other functional specialists to document any business system deficiencies

Contractor Business Systems Rule

44

Step 1 EVMS Compliance Review & Report Step 2 DACO Initial Determination Step 3 Supplier Written Response Step 4 DACO Board of Review Step 5 DACO Final Determination (2-5%) Step 6 DACO Initiates Withholds Step 7 Supplier Response (CAP) Step 8 DACO Verify and Validate Corrections Step 9 DACO Considers Withhold Reduction Step 10 DACO Removes Withholds

CMO Functional Specialists working through the COO

are to Assess the Contractor’s corrections within 90 Days of receipt

Contractors are Required to Respond to the ACO

within 30 Days

COO Functional Specialists are to generate a

Business System Analysis Summary within 45 days

DCMA Business Systems Process ***

45

WHAT IS A SIGNIFICANT DEFICIENCY?

*** • DFARS 252.234-7002 - Earned Value Management

System • DFARS 252.242-7005 - Contractor Business Systems

“Significant deficiency means a shortcoming in the system that materially affects the ability of officials of the Department of Defense to rely upon information produced by the system that is needed for management purposes.”

46

UPCOMING College of Performance Management (CPM) ARTICLE

***

47

Questions ***

48