Surveillance Planning Team

Presentation to the

Engineering Management Council

February 21, 20001

Greg Robinson

Revision 4

Purpose of team

• To improve probability of mission success for NASA projects by improving Agency guidance for risk-based, functionally integrated surveillance.• Surveillance suitable to the project• Surveillance proportional to the risk• Surveillance that meets project management needs

Team Products: Policy Direction Procedural Guidance Streamlined Simplified Tools to Do the Job

Team Products: Policy Direction Procedural Guidance Streamlined Simplified Tools to Do the Job

Surveillance as a method to Surveillance as a method to improve likelihood of mission improve likelihood of mission successsuccess

Key objectives:Key objectives: Integrate surveillance approach:Integrate surveillance approach:

Engineering, S&MA, Finance, ProcurementEngineering, S&MA, Finance, Procurement Improve effective utilization of surveillance resourcesImprove effective utilization of surveillance resources Enhance communication of surveillance resultsEnhance communication of surveillance results Surveillance to proactively mitigate risk (part of RBAM)Surveillance to proactively mitigate risk (part of RBAM) Make surveillance more response to project managementMake surveillance more response to project management

Background (PBC Report)

• July 1999, Administrator tasked the AA for Procurement to assess how the Agency could improve implementation of Performance Based Contracting (PBC)

• June 2000, the PBC Team released their report.

A Significant Finding: Many NASA Centers have not developed or implemented contract surveillance plans. This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.

A Significant Finding: Many NASA Centers have not developed or implemented contract surveillance plans. This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.

Background (NIAT)

• October 20, NASA Integrated Action Team (NIAT) Report, Enhancing Mission Success - A Framework for the Future

NIAT-13 Action Summary: Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.

NIAT-13 Action Summary: Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.

Surveillance Planning Team: Chartered by AA for Procurement and

AA for OSMA, November 2000 Also sponsored by the Office of the

Chief Engineer

Representation:

NASA Centers:• JSC• KSC• MSFC• SSC• GSFC

NASA Headquarters:• Code Q• Code H• Code M• Code AE• Code S• Code Y

Team Sponsorship

Current Members:• JSC - Jim Nise (SOMO Engineering Management) • JSC - Lee Norbraten (Management Systems Office)• KSC - Larry Tucci (S&MA)• KSC - David Culp (Procurement)• KSC- Terry Smith (Quality)• MSFC - Barry Musick (Engineering) (for Jim Kennedy) • MSFC - Charlie Chesser (S&MA)• SSC - Mike Smiles (S&MA)• GSFC - Mary Clark (EVM)• S - Nancy Porter (Program Office - Space Science)• Y - Tom Magner (Program Office - Earth Science)• M - John Castellano (Program Office - HEDS)• Q - Steve Newman (PBMA)• Q - Tom Whitmeyer (Quality)• H - Jeff Cullen (Procurement)

Team Membership

Tools To Be UsedTools To Be Used

Root Cause AnalysisRoot Cause Analysis

Customer CoordinationCustomer Coordination

Functional AnalysisFunctional Analysis

Benchmarking of Best PracticesBenchmarking of Best Practices

• NIAT and PBC findings analyzed to determine underlying contributing factors.

• EMC and PMC review.

• Team determines Engineering, Procurement, S&MA objectives for surveillance, and identify methods for improved implementation and integration.

• Team validates approach against best-of the best in surveillance activity.

Lack of meaningful guidance for conducting contract surveillance

Existing NASA Policy, Procedural Guidance, and FAR Supplementation inadequate

Need for continuous risk evaluation of contractor performance(not a one-time plan)

“Stove pipe” approach to performing surveillance

Contributing Factors• Lack of common definition• Range of contract mechanisms• Involvement of multiple functional organizations• Differences in approaches across programs and centers• Lack of agency wide training

Root Causes Currently Identified

Team Strategy

CHALLENGES:Large/Diverse Project Base Large/Diverse Functional Support Organizations

CHALLENGES:Large/Diverse Project Base Large/Diverse Functional Support Organizations

Define surveillance needs

Provide “risk based” options

Utilize Existing Guidance Where Possible: NPG 7120.5 NPG 8735.2 NASA FAR and Supplement Other (EVM etc)

Schedule and ProductsSchedule and Products

Initial Meeting

Agency Objectives

Outline for NPD

Draft NPD

One Week

Benchmark Study

Benchmark Report

Draft NPG ForSurveillance

Publish Summary of Guidance Examples

Jan-00 Feb-00 April-00Telecon

1/30/00

July-00

Final Report for

Chief Engineer

Recommended Follow On Actions

(Training, Etc.)

MeetingMeeting

1/9/00

EMC Meeting

2/22/00MeetingTelecon

Current Status - Next StepsCurrent Status - Next Steps Draft NPD Draft NPD Survey Survey NPG Initial Approach (see attached)NPG Initial Approach (see attached) Benchmarking Plans Benchmarking Plans

IndustryIndustry AgenciesAgencies

Status meeting with PMCStatus meeting with PMC

• AE/Office of the Chief Engineer is responsible for the NPD, project managers are responsible for initiating and maintaining the surveillance plans.

• Plans are required for contracts, grants, cooperative agreements and MOUs unless otherwise specified by the Center Director

• The surveillance plan will:• Establish and maintain direct linkage with risk of the contract• Have a level of surveillance penetration that is proportional to risk areas and to the

scope and risk of work to be performed.• Be a collaborative integrated effort with procurement, engineering, S&MA, and finance.• Be initiated during development of the SOW • Provide definition of government surveillance by type, responsibility, and frequency. • Specify if or how sub contract work is addressed.• Provide for the communication of surveillance plans and results in a defined manner to

appropriate levels of management.

• NASA engineering, procurement, safety and mission assurance, and other technical organizations shall provide surveillance support to program/project management through collaborative involvement in the development and maintenance of surveillance plans and performance of associated levels of surveillance penetration.

Surveillance NPD - Key Elements

EMC BACKUP CHARTSEMC BACKUP CHARTS

“Prime” (PBC) Contractor

NASA as Integrator

In-House

Increasing level of integration responsibility andaccountabilityassigned tocontractors

80% PBC Goal

Changing Environment

Project Management/Risk Management

ProjectManagement

Cost Schedule

TechnicalPerformance

ContractManagement

MissionAssurance

Safety/Security

Risk Management is a Project Management

Tool to Balance Requirements/Resources

S&MA

ProcurementEngineering

Functional Organizations Provide Expertise/Information to

Support this process

Finance

Elements of a PBC Contract

Performance Requirements and Standards

Completion Form Contract Type(FFP, FPIF, CPIF, CPAF -- but not LOE)

Well Defined Metrics

Contractor Quality AssurancePlan

Comprehensive Surveillance Plan

Fee for Results Not Best Efforts

Surveillance Methods and Tools

Control • Cause and Effect Analysis• Closing a Risk• Cost-Benefit Analysis • List Reduction• Mitigation Status Report• Multivoting• PERT Chart• Problem-Solving Planning• Risk Information Sheet• Spreadsheet Risk Tracking • Stoplight Chart

AnalyzePlan

Track

• Baseline Planning• Planning Decision Flowchart• Planning Worksheet

- Brainstorming- Cause and Effect Analysis- Cost-Benefit Analysis- Gantt Charts- Goal-Question-Measure- Interrelationship Digraph- List Reduction- Multivoting- PERT Chart- Work Breakdown Structure

• Risk Information Sheet

• Bar Graph• Mitigation Status Report• Risk Information Sheet• Spreadsheet Risk Tracking• Stoplight Chart• Time Correlation Chart• Time Graph

Surveillance PlanA Surveillance Plan documentshow risks will be monitored on aproject: the process, activities,milestones, and responsibilities

associated with surveillance. It is a subset of theproject plan and is written before the project begins.

Identify

• Baseline Identification and Analysis• Contractor provided metrics• Contractor Monthly Reports• Technical Interface Meetings• Milestone Reviews• Inspections• Assessments

• Test Reports• Financial reports• EVM• Failure reports

• Project Metrics

• Project Metrics

Identify

Analy

ze

Plan

Track

Control

Communicate

Risk Drives Areas of Surveillance

W e a k Im p le m e n ta t i o n

R i s k M a n a g e m e n t P r o c e s s

C o m m u n i c a t i o n

R e q u i r e m e n tD e f i n i t i o n

In fo r m a l o r R e s t r ic t e d

C o n t r o l l e d P ro c e s s

P l a n n in g

T e c h n o l o g yR e a d i n e s s

E x p e r ie n c e o fM a n a g e m e n t T e a m

C o s t

R e v i e w s

E x te n s i v e , P e e r &In d e p e n d e n t

L im i te d P ro je c t In te r n a l

P ro v e nT e a m

O J T

T R L 5 - 6

T R L 1 - 3

E x is t i n g

E x te n s i v e , U p - F r o n tR e a c t i v e

C le a r , F i x e d ,P a re n t - C h i ld

D e v e lo p e d a s N e e d e d ,F re e F lo a t

D y n a m ic ,In te r a c t i v e

T e c h n i c a l T e a mC o h e s i v e , A u th o r i t y

W e a k l y C o u p le d

T ra d a b le R e s o u rc e

P e r fo r m a n c e th eD o m in a n t V a r ia b le

L o w e s tR i s k

S a f e t y a n d M i s s i o nA s s u r a n c e

L im i te dIn v o l v e m e n t

E x te n s i v e In v o l v e m e n t

Surveillance Proportional To Risk

Surveillance Approach

Evaluated Risk

Based on “Risk Management”

Tailored toselected Contractor

Tailored toselected Contractor

Developed prior to awardDeveloped

prior to award

Not incorporated into the contract

Not incorporated into the contract

Insight and Oversight, commensurate with riskInsight and Oversight, commensurate with risk

DynamicDynamic

Surveillance Suitable to the Project

Based on “Risk Management”

DynamicDynamic

Low Dollar Amount

Non Flight

No Hazardous Operations

High Dollar Amount

Flight

Hazardous Operations

Surveillance Dynamics (RBAM)

Final Surveillance Plan

(High/MediumRisk) based on

selected contractor)

ACQUISITION STRATEGY PROCESS

EngineeringProcurement

S&MA, Other

REQUIREMENTS POST AWARDSOURCE

SELECTIONSOLICITATION DEVELOPMENT

ACQUISITION STRATEGY

DEVELOPMENT

Contract Performance

RISK RANKING

HIGH RISKMEDIUM RISK

LOW RISK

22

Project ManagerRisk Management

Surveillance Results

PreliminarySurveillance Plan

Level of Surveillance Penetration

Consequence of Failure

5

4

3

2

1

Likelihood

1 2 3 4 5

Did they do th

e

right th

ings?

Did they do th

ings

right an

d did they do

them rig

ht?

Increasi

ng technica

l penetr

ation

Review of

Processes

Review of P

rocess

and Implem

entation

Mapped on Risk Chart

Surveillance Continuum

Did they do the right things?

Did they do the right things and did they do them

right?

Increasing technical penetration

Review of Processes

Review of Process and

Implementation

Level 0 Level 1 Level 2 Level 3 Level 4

No Penetration

Level of surveillance contingent on defining an acceptable risk:Level of surveillance contingent on defining an acceptable risk:

• Technical risk levels

• Amount of trust in performing organization’s abilities

• How well processes are defined

• Level at which NASA is performing Task Agreements for the program

• Human rating of vehicle

• Program visibility and impact of failure

• Design complexity, manufacturing complexity, producibility

• Value of asset

Surveillance Evolves as Risk is Better Defined

WB

S

Formulation ImplementationSRR PDR CDR FRR OperationsProduction

WB

S

WB

S

Top-Level

Rqmts.

Trades

Concepts.

SystemSpec.

EndItem

Specs.

Analysis

Prelim.Drawings

DetailedDrawings

RiskPlan

Production/Manufacturing

Package

H/W & S/WVerification

Plan

Acceptance/COFR

Risk PenetrationSurveillance Matrix

Identified RiskIdentified Risk

No Penetration• Accept performing organization’s tasks at face value (based on assessment that no penetration is required)• Contractor develops and implements verification plan

Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve issues• Review verification plan, its implementation, and selected verification closure data

In-depth Penetration • Includes intermediate penetration with addition of:

• Methodical review of details• Independent models to check and compare vendor data, as required

• Review verification plan, implementation, and concur in all verification closure data

Total Penetration • Perform a complete and independent evaluation of each task• Independent review of all verification documentation (including closure data) and witness verification testing

Technical Penetration Levels (by discipline areas)

Low Penetration • Participate in reviews and Technical Interchange Meetings and assess only the data presented• Perform periodic audits on pre-defined process(es)• Chair board or serve as board member, or RID writer, at a formal review• Participate in resolution and closure of issues• Review verification plan and its implementation

Level 0

Level 1

Level 2

Level 3

Level 4

Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve evolving issues• Institute specific, regularly planned status and open action/issues reviews

In-depth Penetration • Includes intermediate penetration with addition of methodical review of details• Government Senior Business Managers may participate in reviews

Business Systems

Penetration Levels

Low Penetration • Participate in or perform periodic audits on pre-defined process(es) and/or systems• Participate in resolution and closure of issues• Hold periodic status and open actions/issues reviews

Level 0

Level 1

Level 2

Level 3

Level 4

Total Penetration • Perform a complete and independent evaluation of each task• Independent review of all Contractor final contract documentation • Regularly scheduled meetings of Senior Business Managers (Contractor and Government) to review current/anticipated issues and overall status

No Penetration• Accept performance at face value (assess that no penetration is required)

Technical Surveillance Plan

AREA OF PENETRATION EN

GIN

EE

ER

ING

SE

RV

ICE

Ele

ctr

ical P

ow

er

Instr

um

enta

tio

n a

nd

Co

ntr

ol

Co

mp

ute

rs a

nd

Data

Syste

ms

Flig

ht S

oft

ware

Avio

nic

Syste

ms

EE

E P

art

s a

nd

Packag

ing

Co

ntr

ol E

lectr

onic

s

Rad

io F

req

ue

ncy

Avio

nic

s S

imula

tio

n

Str

uctu

ral and

Dynam

ic L

oad

s

Str

eng

th A

naly

sis

Str

uctu

ral D

esig

n

GS

E a

nd

Me

chanis

ms D

esig

n

The

rmo

dynam

ics a

nd

He

at T

ransfe

r

The

rmal and

Flu

id S

yste

ms

Str

uctu

ral and

Dynam

ics T

esting

Enviro

nm

enta

l E

ffe

cts

No

n-D

estr

uctive

Eval &

Trib

olo

gy

Me

tallic

Mate

rials

and

Pro

ce

sse

s

No

n-M

eta

llic

Mtls &

Pro

ce

sse

s

Pro

ject E

ng

ine

ering

Che

mis

try

Manufa

ctu

ring

Se

rvic

es

Sp

ecia

l T

est E

quip

me

nt D

esig

n )

Sys E

ng

r S

up

po

rt

Co

nfig

ura

tio

n a

nd

Data

Mg

mt

Enviro

nm

ents

Structural Test Rqmts (Proof Loads) 1 1 1 2 1Peroxide Issues (Comp,Clean,Venting) 2 2MPS/RCS Propellant TransferC/SiC Performance 2 2 2 1SIGI 2 2Retractable Solar Array 2 2 2WLE Surface Temperatures 2Li Ion Batteries 3CADS 2Software V & V 3Communication System 2High Temperature Bearings 2 1Laser Initiated Pyrotechnic FTS 1Composite Materials 2Systems Engineering/Integration Issues 1 1 1 1AL 5254 Tank 2 2Component Qualification Rqmts 2 1 1 1 1Structural Verificaton & Testing 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1Environmental Assessment/Effects 1 1 2Star Tracker 1Actuator Control Electronics 2Computers/Data Systems 2Level 0 - No Penetration Level 1 - Low Level Penetration - Participate in reviews and TIMs, assess only the data presented, perform periodic audits Level 2 - Intermediate Level of Penetration - Level 1 plus daily or weekly involvement to identify and resolve issues Level 3 - In-depth Level of Penetration - perform independent assess. and run independent models to check and compare vendor dataLevel 4 - Total Penetration - perform a complete and independent evaluation of each task

X-37 Engineering Directorate

Insight Matrix(ED Interpretation Plus Additional

Recommended Areas)

Business Surveillance Plan

Estimating System

Accounting SystemProperty Mgmt. SystemMajor Sub-K Approval

Periodic Floor Audits

Proposal development and timely negotiation

Procurement System

Socio-Economic Performance

Labor Relations

Special Contract Schedules (LOE or T&M)

Performance Measurement Reviews or Audits

Corporate Financial Health

Performance of audits delegated to DCMA

Performance of audits delegated to DCMA

Performance of audits delegated to DCMA

Performance of audits delegated to DCMA

Performance of audits delegated to DCMA

NASA review & consent on all Sub-K action over $1M

Union contract negotiations scheduled for this summer.NASA Labor Relations Specialist involved.

Fortune 500 Company. No formal surveillance required.

Excellent performance to date. Reviewed every 6 Months

Monthly review with Senior Managers on status of all Proposal Development activity and negotiation status

Performance of audits delegated to DCMA

Not applicable

Surveillance ApproachSystem/Area