Enterprise Risk and Opportunity Mangemetn (EROM) Workshop › docs › default-source › sma... ·...

Enterprise Risk and Opportunity Management (EROM) Workshop #1

October, 30, 2014, 8:30AM – 4:30PM

Room 1Q39, NASA Headquarters

Minutes

Attendees (on-site):

Dezfuli, Homayoon NASA HQ, Office of Safety and Mission Assurance (OSMA)

Bauder, Stephen P. NASA HQ, Exploration Systems Division (ESD)

Colon, Alfredo NASA HQ, OSMA

Comstock, Doug NASA HQ, Office of Evaluation (OoE)/Cost Analysis Division (CAD)

David, Samina NASA HQ, ESD

FitzSimonds, Chris NASA HQ, Office of the Chief Financial Officer (OCFO)

Freeman, Veronica NASA HQ, OCFO

Gallagher, Katie NASA HQ, Office of the Chief Technologist (OCT)

Groen, Frank NASA HQ, OSMA

Hill, Arnold NASA HQ, OoE

Jambulingam, Nat NASA HQ, Mission Support Directorate (MSD)

Lehnhardt, Emma NASA HQ, OCFO/SID

Lodge, Cynthia NASA HQ, OCFO/SID

Mexcur, Paul NASA HQ, OSMA

Nelson, John NASA HQ, Space Technology Mission Directorate (STMD)

Southwell, Jessica NASA HQ, OCFO/SID

Walsh, Jon NASA HQ, Office of the Chief Information Officer (OCIO)

Wennerberg, Linda NASA HQ

Allan Benjamin Information Systems Laboratories (ISL)

Everett, Chris ISL

Rutledge, Pete Quality Assurance & Risk Management Services, Inc. (QA&RMS)

Attendees (off-site):

Gawdiak, Yuri Aeronautics Research Mission Directorate (ARMD)

Irvine, Lynn HQ

Robinson, Frank GRC (on detail to OoE/Cost Analysis Division (CAD))

Thomas, Sharon JSC

Pollitt, Julie QA&RMS

1

Executive Summary

Homayoon Dezfuli explained how the proposed EROM concepts were an evolution of the work

done by OSMA in the area of risk management (RM) since the mid-1990s. He explained how

risk management at NASA began as Continuous Risk Management (CRM) around 1995. In 2008

NPR 8000.4, Agency Risk Management Procedural Requirements, was updated, at which time

RM was enhanced by making it the combination of Risk-Informed Decision Making (RIDM) plus

CRM, instead of CRM alone. As early as 2008, the NPR called for RM on an Agency-wide basis.

Homayoon described some of the details of the RM process (CRM+RIDM) for background. Then

he explained what EROM is and why NASA needs it—agency-wide and at every level of the

organizational hierarchy. Homayoon was followed by Allan Benjamin who provided an

overview of the EROM Concept Paper, which had been distributed to workshop attendees for

review in advance of the meeting. Following that, Jessica Southwell and Emma Lehnhardt from

OCFO presented information on their organization, NASA’s performance framework and

Strategic Objective Annual Review process, and the challenges and opportunities they saw in

the proposed EROM concepts. It was learned that OCFO handles much more data than is

readily apparent in their publically released documents such as the NASA Strategic Plan. More

work will have to be done to determine where in their processes, EROM can play a valuable

role. The afternoon of the workshop was devoted to discussion of the viewpoints and concerns

heard during the morning and reaching consensus on next steps. It was agreed that an EROM

working group should be formed and continue to work on the development of EROM; a

number of participants offered to be part of it. It was also agreed that before engineering a

detailed EROM process for NASA, the group should hear more from the mission directorates,

Centers, and programs/projects including examples of what they are already doing in enterprise

risk management. The next workshop will be devoted to this purpose; it will be held in the

January 2015 timeframe; Nat Jambulingam will help in organizing it using the Lean Six Sigma

(LSL) approach. Homayoon Dezfuli will continue to be actively involved in the development of

the EROM process.

Workshop Details

The following detailed minutes are divided into sections according to the agenda (Attachment

1).

Administrative Remarks and Introductions

Dr. Homayoon Dezfuli opened the workshop shortly after 8:30AM EDT with 28 attendees both

on- and off-site. The participants (see list on previous page), both on- and off-site introduced

themselves. Homayoon went over the workshop agenda, which can be found at Attachment 1.

He stated that workshop minutes would be provided, that this was only the first workshop, and

that he hoped there would be one or more follow-up workshops on this topic in the future. He

mentioned the EROM Concept Paper that had been circulated to all attendees in advance of the

2

meeting and said that its purpose was to put ideas on the table and to start conversation. He

said he believed the concept paper had served that purpose as evidenced by the people who

were present to begin discussing Enterprise Risk and Opportunity Management (EROM).

He asked the participants to 1) try to look at the proposed EROM concepts strategically, looking

down the road, perhaps as far as 10 years from now and 2) to discuss the concepts generically

and not try to match them to current practices. In approaching the concepts this way,

Homayoon said he hoped that the end result would be less vulnerable to future organizational

changes. Beyond that he expressed the opinion that NASA tends to be a leader and that what

we were doing in this workshop might also help other government agencies in the long-run.

Linda Wennerberg pointed out that there are a lot of changes going on at Headquarters at the

present time and that these changes need to be considered in any discussions of EROM.

Homayoon stated that in discussing Enterprise Risk Management (ERM), we need to look at the

entire Agency and not just Headquarters.

Homayoon provided some background on his role in risk management—that he is the primary

OSMA person in the area of risk management (RM) and is the lead for NPR 8000.4, Agency Risk

Management Procedural Requirements, the document in which he started making

improvements to RM at NASA in the 2008 timeframe.

Evolution of Risk Management at NASA – Homayoon Dezfuli

The presentation slides are provided at Attachment 2. The following comments are keyed to,

and meant to supplement the content of the slides that were being presented when the

comments were made.

Slide 2: The EROM Concept Paper is consistent with NPR 8000.4A. It was intended to start us

talking about concepts. Debate is encouraged. Homayoon said he was looking forward to the

input from the Strategic Improvement Division of OCFO. He acknowledged and thanked Jessica

Southwell for the helpful discussions he had already had with her and her staff on the topic.

Homayoon made the point that OSMA has been involved in RM for many years and that some

of that background was not covered in the Concept Paper. He said he would communicate

some of that history to the participants to give them a better understanding of why OSMA was

involved.

Slide 4: Continuous Risk Management (CRM) has been in use at NASA since the mid-1990s,

mainly during the implementation phase of projects.

Slide 5: In the 2008 timeframe, when NPR 8000.4 was being revised, NASA was moving toward

retiring the Space Shuttle. What was missing in NASA’s RM up to that point in time was a role

in impacting decision making to help make the right choices. There was a need to make RM

3

more proactive. An evolutionary approach was taken to address the gap. A new process called

Risk-Informed Decision Making (RIDM), which complimented CRM, was added to the RM

process. RIDM was essentially Analysis of Alternatives (AoA) in a risk management framework

or, more specifically, risk AoA. In the overall RM process, once RIDM took place, CRM followed

to focus on assuring that requirements were met.

Slide 6: Among the motivations for changing NPR 8000.4 in 2008 was the need to formalize risk

acceptance at NASA. There was also the desire to make project requirements risk-informed.

Slide 7: This slide shows that RIDM works to flow requirements down the NASA organizational

hierarchy, while CRM works to elevate risk information and risk decisions, as needed. CRM is

now intended to work in the context of requirements, which was not always the case before

2008.

Slide 8: The figure on this slide is from NPR 8000.4A.

Slide 9: Note on this slide that risk is defined in terms of requirements. Homayoon pointed out

that the key words here are: shortfalls, requirements, and future.

Slide 10: Homayoon admitted that he did not do a good job of explaining the relationship of

AoA with RIDM. RIDM is a deliberative process. It is important to understand that RIDM is risk-

informed and not risk-based decision making. It includes the development of probability

distributions for various performance measures and establishing risk tolerances on

performance measures. Low risk tolerances on performance measures that have imposed

constraints assure a high likelihood of program/project success. The concept of “Performance

commitment” was introduced to support a risk-normalized comparison of decision alternatives,

at a level of risk tolerance determined by the decision maker..

Yuri Gawdiak said that risk tolerance needs to be derived and dynamic and that it might be

expressed in terms of expected values. Homayoon replied that risk tolerance should be

specified by the decision maker.

Doug Comstock pointed out that NPG 7120.5E defines cost and schedule in terms of a 70% joint

confidence level (JCL) and that this is in line with Homayoon’s description of RIDM. Homayoon

added that he had worked with Tom Coonce with respect to the JCL.

Slide 11: CRM makes sure that someone has responsibility for requirements and their

associated risk. CRM now ensures that we go after the “risk drivers” and not just individual

risks—it causes us to look at the forest and not just the trees. It helps identify cross-cutting

issues.

Slide 12: Homayoon said he is providing background because the existing concepts he is

showing, which have been applied at NASA for years, are the building blocks for EROM.

4

Objective hierarchies are used all the time within the decision analysis community;

they provide a bird’s eye view of what is important. A trade tree is analyzed in relation to the

objectives.

Slide 14: Risk Analysis is Performance Assessment of decision alternatives supported by

Probabilistic Modeling. It is multi-disciplinary and requires a high level of coordination among

analysts to maintain consistency in the modeling of alternatives across disciplines.

Performance Measures are characterized in terms of probability distributions that reflect

relevant uncertainties, both in implementation and in the performance of the realized product.

Characterizing the correlations among the domains of safety, technical, cost, and schedule is

challenging. Since performance measures are typically not independent, correlation between

performance measures should be preserved. For example, cost and schedule tend to be highly

correlated. High costs tend to be associated with slipped schedules.

Slide 15: Sharon Thomas commented how in her experience, “performance commitments” was

at first an unfamiliar term. Homayoon said he struggled with it at first, too. He explained that if

one were to commit to a “performance commitment,” it would become a requirement [which

is what happens once the alternative selection is made].

Slide 16: CRM uses risk statements. They help provide additional rigor to the RM process. The

Condition is factual; the departure from the baseline plan is the basis for assigning likelihood.

The risk statement provides the discipline and structure that helps decision makers act on the

risks. The Departure is expressed as a change relative to a baseline plan (requirement.)

Doug Comstock asked where the likelihood appears in the risk statement. Homayoon replied

that likelihood is in the word “possibility” that is a standard part of the wording of all risk

statements. The statement needs to be analyzed by the “analysis” step of the CRM to

determine the likelihood value.

Slide 17: The “planning” step of the CRM which is about the development of risk responses,

which can be tactical or strategic depending on the timeframe for action.

Slide 18: The Aerospace Safety Advisory Panel (ASAP) criticized NASA for the large number of

“yellow” risks present during the Space Shuttle’s return to flight after the Columbia mishap.

They said that NASA must pay attention to the totality of risk and not just the individual risks.

Note on this slide that timeframe and uncertainty affect the ranking of risks. Sharon Thomas

commented that what they look at in the field is what is on the “critical path.”

Slide 19: Taxonomies can help find commonality among risks for the purpose of “binning”

them, which helps identify cross-cutting risks. Homayoon said he had talked to Jesscia

Southwell about taxonomies; the problem will be how to employ them at the highest levels; we

will need to spend some time on that.

5

EROM and Why We Need It – Homayoon Dezfuli

Slide 21: Homayoon said we are not yet where we want to be in RM for programs and projects.

Even so, we now have the push from GPRAMA, A-11, and A-123 that calls for building on the

work we have done to extend RM to the enterprise level at NASA.

Slide 22: It should be noted that when NPR 8000.4A was written in 2008, it contained the

wording that applied RM to all levels of the Agency.

Slide 23: A-11 calls for agency-wide application of RM. ASAP has called for formal RM and

accountability for risk.

Slide 24: The EROM Concept Paper may not yet be perfect but it started by looking at a variety

of pertinent internal and external documents. Many concepts exist in the external documents,

but the vast majority are intended for application to financial organizations rather than

technical ones like NASA. EROM won’t be realized overnight at NASA.

Slide 25: Note that institutional RM is seen here as the foundation of enterprise-wide RM.

Slide 26: The enterprise level of the Agency is where the “big-ticket choices” are made; the

combination of RIDM + CRM remains applicable for managing risks at this level. In the past the

emphasis of RM has been on identifying risks and then mitigating them. Some say the Agency

has become risk-averse, which raises the question of how far to push risk reduction measures in

our endeavors. It is appropriate to bring in the “risk tolerance levels” and new factor of

opportunities or benefits; these can be game changers for RM. We have seen the case for

bringing risk and opportunity together expressed in a letter from the Administrator.

Nat Jambulingam asked if ERM was only at the Agency level. Homayoon replied saying there

are two terms to consider: EROM-level and EROM-wide. We need to apply EROM at the

Agency level but not lose sight of the need for it Agency wide. The EROM Concept Paper

focused more on the application of EROM at the executive level.

Alfredo Colon commented that the same EROM concepts are applicable down the NASA

organizational structure. Another participant asked if the NASA Centers are doing ERM. Emma

Lehnhardt said we saw some evidence of that at Sharon Thomas’s recent workshop at GSFC.

Sharon added that the Centers have unique assets, resulting in differing portfolios; some

Centers are beginning effective institutional RM. Linda Wennerberg said that we need to take

an Agency-wide look versus a Center look at unique assets and capabilities; having institutional

RM as a foundation is okay, but there will be differences at every level. Sharon added that

there are two tiers of RM needed at the Centers: institutions and programs/projects; she asked

how can we combine/integrate these?

Slide 30: Homayoon said we have found that the COSO process is not totally applicable for

NASA because NASA is a technical organization. 6

This slide illustrates the major partitions of NASA that have to work together. EROM

has to have this view of the Agency.

Nat Jambulingam asked about the Concept Paper having applicability to other government

agencies. Homayoon said we struggled with the document—whether to make it for NASA or to

make it more generic; in the end the paper contains the necessary elements [principles] that all

agencies need for effective EROM. Jessica Southwell said that there are other agencies, like the

Department of Homeland Security, that have robust RM, but with mixed success.

Slide 34: Cynthia Lodge commented that the implementation of program/project plans is not

done at the executive level and that she is seeing a mixing of governance with implementation

and evaluation in the material presented. Allan Benjamin pointed out that the processes

illustrated are not all done by the same people, but they are done at the same level. Cynthia

added that not every agency has the same governance structure.

Jessica Southwell asked what are the perceived gaps? Homayoon replied that there are not

necessarily gaps and that the Program Management and Mission Support councils evaluate

programs/projects and institutions. Jessica asked what the perceived needs of these councils

and senior leadership are. She suggested that we need to ask them before we engineer the

process.

Steve Bauder said that ERM is not new; he has set up ERM to do integrated RM in his

organization. He reinforced the idea that we need to find out what people are doing in ERM

before engineering a new process. Homayoon said that trying to develop a single EROM

process that applied overall was impossible, so the approach was taken to lay out the principles

that are essential to any version of EROM.

Jessica said that there are different interpretations of what OMB is asking agencies to do in A-

11, but it is clear they are looking for better decision making. Homayoon said he fully agreed

and that is what we are trying to bring about in what we’ve done thus far. Sharon added that if

the EROM process is developed at HQ and then levied on the Centers, there may be problems if

the Centers already have a process in place. She asked if the EROM concepts are to be

guidance or requirements. Homayoon said that his intent is to offer ideas and expectations,

not requirements; if the ideas are useful, then they should be used. Sharon acknowledged

receipt of Homayoon’s message. Nat added that ERM is not a requirement from OMB, just a

best practice.

Slide 35: The definitions seen here have been customized for use at the agency-level.

Doug Comstock commented that with regard to risk and opportunity, we have fixed resources

at NASA. If one trades cost against the likelihood of success, one can do more missions if one is

willing to accept a higher risk of failure. Homayoon said that was a good example of the kinds

of trades NASA does. Yuri Gawdiak commented that perhaps that was mixing things, as failures

7

are already accounted for in the 70% JCL; he sees it as an expected value problem. He said

NASA doesn’t say “no risk.” He said we know the value placed on human life, the expected

value of discovery, and the probability of failure, and that these factors need to be considered

dynamically every time. Homayoon said we need an integrated view risks and benefits.

Slide 39: Homayoon said that while all the terminology in this slide may not be current, NASA’s

strategic planning structure can be viewed in the form of a tree like this. We have to look at the

implications of this; we have to develop an EROM scheme that has this kind of view.

Slide 42: This slide is part of developing the building blocks of a process for better decision

making. It has an integrated, agency-wide perspective. It can be how we put ourselves on the

right trajectory.

Homayoon concluded his presentation by saying that he wanted to show where OSMA had

come from in the field of RM. The solution to EROM will require “a village.” Some principles

and rules are clearly needed, along with the right interfaces. We can get where we want to go

incrementally and gradually. Yuri commented that for us all to be on the same page we need a

workshop where the mission directorates and the Centers present their examples of ERM, so

that we can see where we all agree. Homayoon agreed and said that this is only the first

workshop and we may need many. He also said we need co-organizers to make EROM a real

joint effort. Frank Groen said that we had some discussions like that at Sharon’s workshop with

regard to people’s struggles with RIDM. Homayoon said we would discuss RIDM more later; he

knows people struggle with it; RIDM is not for CRM people to implement as it has to do with

deciding on direction setting decisions. RIDM is invoked for key decisions such as architecture

and design decisions, make-buy decisions, and budget reallocation (allocation of reserves),

which typically involve requirements-setting or rebaselining of requirements. There is a need

to engage a broader community than just Risk Management experts. RIDM is intended to be an

integral part of systems engineering and should involve systems engineers. RIDM demands a

level of coordination among domain-specific analysts (e.g., safety, performance, cost, schedule)

that may be more than what the system is used to. Emma said we haven’t heard about RM

from the mission directorates and we need to do that. Homayoon agreed. Doug Comstock said

that would help us identify the gaps in the way people are doing ERM.

The workshop took a break from 10:35 to 10:45AM.

Overview of EROM Concept Paper – Allan Benjamin

The presentation slides are provided as Attachment 3. The following comments are keyed to,


comments were made.

Slide 2: In the existing NASA RIDM and RM Handbooks, RIDM starts with a hierarchy for a

project. Here we apply the objectives hierarchy construct to strategic performance.

8

The roll-up process requires integration.

Slide 4: This slide is about balancing risks and opportunities.

Nat Jambulingam commented that the weighing of risks and opportunities appears very

subjective and asked about the guidelines for doing this. Allan replied that there is not an

engineering answer. Rather, the answer is a psychological one; it’s a situation of “pain vs.

gain.” It’s personal. Homayoon added that it’s not the job of the risk analyst to decide on the

criteria; it’s the decision maker’s call. Yuri said we have the value of human life, hardware,

work effort, etc.; it becomes a problem of the expected value of risk and benefits. Allan said he

had done it that way with nuclear reactors; it can be controversial. He said that polling the

decision makers is a good way, as it puts the onus on the people responsible for risk. He also

said there is more than one way to do it. Homayoon said this would be a good topic for a

future workshop.

Slide 5: Note that there are risks, opportunities, and indicators for each objective.

Opportunities may introduce new risks—added cost, delays, etc. One can judge risks based on

past experience, if it applies.

Slide 6: The indicators shown in this example table are meant to be quantifiable. They signal

the need for a response.

Slide 8: These definitions were presented earlier by Homayoon. Note how risk is distinguished

from opportunity. These definitions are for strategic planning. They provide confidence in the

achievement of objectives. They provide arguments for and against objectives that may be

levied from the higher level organization.

Slide 10: This is the asteroid retrieval example. A detailed example is hard to do at this point.

What kinds of analysis and thinking would be needed? If we have an objectives hierarchy in

place in our strategic and performance plans, what would we have to do to add the asteroid

retrieval mission?

Slide 11: This slide is meant to illustrate how we look at the effects of this new project on other

existing objectives.

Slide 16: Steve Bauder asked how leading indicators fit it. Allan referred to his slide 5 to answer

the question. He said that leading indicators show a trend toward [or away from] success.

Chris Everett said one can find leading indicators addressed in the COSO report. Cynthia said

that this concept doesn’t work in every case; that mission directorates have many other

indicators. Steve said he wasn’t sure why we were discussing leading indicators in EROM. Allan

replied that there is a correlation between leading indicators and the risk of not meeting

objectives; EROM is more qualitative; leading indicators are quantitative but reflect on

qualitative risks and opportunities.

9

Homayoon asked Steve if he could share some of his ERM-related documents with him. Steve

cited on example of an institutional opportunity at KSC dealing with pad design, but it was in

conflict with the rocket being designed; there were performance and cost interactions. He

added that the asteroid retrieval mission example is real to his office. He said his office has

been working on ERM for about 2-1/2 years.

Slide 18: This slide shows the connection between risks, opportunities, and internal controls.

NASA’s Performance Framework; EROM’s Challenges and Opportunities – Jessica Southwell &

Emma Lehnhardt

The presentation slides are provided at Attachment 4. The following comments are keyed to,


comments were made.

Slide 4: Jessica explained that her people are the subject matter experts on external

requirements.

Slide 5: A lot of the information her office works with does not go into public reports. The

NASA Strategic Plan is their most visible document.

Slide 7: Below the dashed line are many “fluid” measures.

Slide 8: The SOARs were initially envisioned as a way to identify poor programmatic

performance for Congress—for cutting programs. OMB implements GPRAMA via Circular A-11.

Jessica’s office has a mandate from OMB to rank and put 10-20% of NASA’s objectives in each

colored box.

Slide 9: This methodology is for use over a ten year timeframe. Jessica’s office was aware that

OMB wanted to know about risks and opportunities, so they are both reflected in the

methodology.

Slide 11: Jessica said she sees a lot of funding-related risks.

Slide 13: At this time Emma Lehnhardt took over; she is the SOAR process leader in OCFO. She

said the entire Strategic Planning and Performance Management team read the EROM paper

closely, and the comments reflect inputs from the whole team.

Slide 14: Allan Benjamin asked if ERM would be more of a requirement in the coming update of

A-123. Jessica said that OMB got a lot of pushback when they previously proposed to make

ERM a requirement and that it will probably remain a best practice.

Emma said that her main issues with EROM can be placed in four bins: 1) How to integrate

performance and risk management communities; 2) Timing; 3) Operationalizing; and 4) Burden.

10

Speaking for her organization, Jessica said that risk is in their culture, but risk is a sensitive

matter and that we have to be careful about our strategic messaging. Katie Gallagher said that

if you communicate risk to Congress, you are essentially asking them to do something about it.

Slide 17: Jessica said there is no clear mapping of objectives to Centers.

Homayoon commented that everything developed in EROM would not automatically go to

OMB. Emma said that the strategic framework is itself influenced by political factors; there is a

lot of sensitivity to what is reported externally. Jessica added that what you see is only a small

part of the data her office deals with. Allan asked if there is any way to define units of analysis

that lie “below the tip of the iceberg.” Jessica said maybe, but we would have to get the

mission directorates involved; the “Ecosystem” (slide 10) has only been done once. Jennifer

Kerns said that more discussion is needed with the mission directorates; the Strategic Plan is

not the right level.

Jessica Southwell said there are mandatory timing requirements on the PGs and APIs; there are

190+ measures, which may be too much data to deal with. Emma commented on the burden

that would entail and that it might well grow in subsequent years.

Slide 18: Allan asked if current methods were too subjective. Jessica said “without a doubt,” but

that she believed in incremental progress; it would have been hard to do more the first time.

Slide 20: Sharon said she hears a lot of questions about TCAT.

Slide 21: The timeframe for the 2015 SOAR process will be Dec 2014 until its submission to

OMB in mid-May 2015. Allan asked if Jessica/Emma expected their process to be more rigorous

in five years. They responded “sooner.”

In concluding this presentation, Jessica commended Homayoon for bringing the group to the

table. Emma said they are very sensitive to putting more burden on their points of contact.

Jessica reiterated that the PGs and APIs are not the right units of analysis. Steve Bauder said he

was seeing more emphasis on new risks, but that his organization knows what will affect the

strategic objectives. Sharon said that the Centers know their risks but have difficulty translating

them to the HQ level as there are no consistent criteria for what becomes an agency-level risk.

Steve said that ESD knows what to do because they have worked on it.

Homayoon asked if anyone on-line had anything they wanted to present. No one did.

The workshop took a lunch break from 12:45-1:45PM.

Group Discussions:

Understanding Viewpoints and Concerns and Reaching Consensus on EROM Principles

and

Strategies for Continuing Development of EROM at NASA

11

What had been planned as two separate sessions for the afternoon ended up being a single

group discussion.

Based on the morning’s presentation, Homayoon observed that NASA’s strategic goals and

objectives appear to be mandates. Jessica said they are for a four-year period. Cynthia Lodge

said they have to align with the budget. Allan asked if they would be mandated after four

years. Jessica said the terminology might change. Congress and the Executive Branch might

want to see certain things, but NASA still won’t have full autonomy. Allan asked what fraction

of full autonomy we would have. Cynthia said that the content is developed differently each

time. Allan observed that perhaps NASA has quite a bit of autonomy in the content.

Homayoon asked if this activity would be within EROM. Cynthia said no, but there are other

ways to view it; we formulate a budget annually; issue papers are requested from within the

Agency; we could ask for risk statements or risk ratings. This would fall in line with the Strategic

Review process. She said we could obtain risk input from the Centers; they could identify the

strategic goals or objectives that the risks affect. Jessica said that EROM could work in budget

deliberations. Cynthia said that opportunities would be associated more with near-term

mitigations (1 to 5 years).

At this point Steve said he wasn’t hearing an ERM conversation. He said the biggest challenge

he sees is communication. Not all risks require assessment. Are there risks with external

interaction? Cross-cutting risks need to be looked at the Enterprise level. Chris Everett said

that is why we propose the taxonomy to categorize risks for the purpose of identifying cross-

cutting risks. Cynthia said that in the Baseline Performance Review (BPR) she sees a lot of risks

that are cross-cutting; it would be an opportunity to highlight them; the BPR deals with more

than APIs.

Homayoon referred back to Jessica’s slide 7. He said he thought that the 5- and 10-year plans

were what the Agency wants done. What are the risks to these plans? What would be wrong

with identifying these risks? Cynthia said there are other indicators that you don’t see on this

slide. Emma cited example risks such as the psychological effects of long-duration spaceflight,

the health effects of radiation exposure in space, etc.

Katie Gallagher asked what it is we are trying to do. Are we trying to instill more rigor? She

said she is struggling with the link to this framework. Steve Bauder agreed and said his

organization tried to develop a division risk portfolio but found they wouldn’t be able to

mitigate the risks because there was not enough money. He said that most of the reported

risks were really problems. In the end they gave up, as the process appeared to have no value.

Subsequently they focused on cross-cutting risks.

Homayoon referred to his slide 23 and asked how OCFO would satisfy what A-11 asks for.

Cynthia said that hopefully we’re not just trying to generate new risks [with the proposed

process].

12

Homayoon asked if we can envision an end goal that we can work toward. Jessica said that if

the A-11 words about ERM became a requirement, then we’re already covered adequately; we

don’t want to over-engineer it. Cynthia said she would like to hear from other mission

directorates to learn how they are managing risks; she proposed we ask them. Steve Bauder

said that some project risks will directly impact agency goals; the question is which ones are

that significant? He cited an example risk in the possibility of not meeting mission objectives

for EM-2. He said that’s an Enterprise risk—in every year’s budget cycle.

Nat Jambulingam said that OCE was doing RM under the leadership of Hal Bell [who is now

working in OSMA]. He added that SMD, ESMD, and HEOMD are doing it, but what is the

current status of their activities? Do we want a more formalized approach? What would be the

value of doing that?

Homayoon asked the group about the best way to look at how the Agency is working today.

Nat said he would assist in organizing another workshop for that. Homayoon asked if we

should confine that to HQ organizations first. Jessica said it probably doesn’t matter if we do

HQ or the Centers first. Cynthia asked if there is anything here to leverage for the BPR process.

Frank Groen said that we should have a gradual approach to this and that we want to identify

risks that really affect the Agency. Jessica said she will be interested to see what data is out

there. Cynthia said we need to deal with risks today and down the road.

Homayoon asked Katie how AoA is done at OCT, i.e., what the expectations are for AoA. Katie

said that is handled by the Agency Mission Planning Council. She added that what the GAO is

talking about with respect to AoA has to do with acquisition sourcing.

Homayoon asked the group about next steps. He said we need to hear from the mission

directorates. He said that maybe before that we can hear more from Steve Bauder about his

RM work in ESMD. Steve agreed. Jessica asked who should be in our working group. Steve

said that mission directorates, Centers, and programs/projects should all be involved. Jessica

said she would work with Homayoon to make this happen. Homayoon said he certainly wants

to make EROM more than just about him (OSMA). Sharon said she would help bring the

Centers into the activity. Steve said he would help, too. Homayoon suggested the January

2015 timeframe for the next workshop. Nat said that the proper owner for the ERM process is

OSMA, but Homayoon said he wants the Agency to drive it. Nat said that we still need

somebody to lead the team of experts and come up with the process.

Homayoon asked if the group wanted to discuss the basic principles of EROM now or after the

next workshop. [Consensus was that it would be better discussed later.]

Homayoon concluded the discussion by saying the success in EROM would require “champions”

or “leaders,” [with emphasis on the plural nature of those words, i.e., not just Homayoon].

Homayoon said that minutes of this workshop would be provided shortly to all participants.

13

Enterprise Risk and Opportunity Management (EROM) Workshop #1

October, 30, 2014, 8:30AM – 4:30PM

Room 1Q39, NASA Headquarters

Agenda

8:30AM Administrative Remarks and Introductions All

8:45AM Evolution of Risk Management at NASA Homayoon Dezfuli

9:15AM EROM and Why We Need It Homayoon Dezfuli

9:45AM Break

10:00AM Overview of EROM Concept Paper Homayoon Dezfuli &

Allan Benjamin

11:00AM NASA’s Performance Framework; EROM’s Challenges

and Opportunities

Jessica Southwell &

Emma Lehnhardt

12:00PM Lunch

1:00PM Group Discussion – Understanding Viewpoints and

Concerns and Reaching Consensus on EROM Principles

All

2:15PM Break

2:30PM Group Discussion – Strategies for Continuing

Development of EROM at NASA

All

4:00PM Closing Remarks and Next Steps Homayoon Dezfuli

4:30PM Adjourn

Ground Rule:

This first EROM workshop is time-limited to only a single day. While we are looking for everyone’s

opinions, our relatively large group needs to be focused at a high-level in order to stay on time.

For participation from off-site:

Audio: 1-844-467-6272 passcode 920573#

Presentations: To view presentation slides in real-time via Adobe Connect, go to

https://ac.arc.nasa.gov/eromw/

Note: You will need Adobe Connect software on your computer to view the presentation materials. If you

do not already have it, it will load automatically when you go to the above web site to log in as a guest; it

may take a few minutes.

Attachment 1

https://ac.arc.nasa.gov/eromw/

NASA Workshop for Enterprise Risk and

Opportunity Management (EROM)

Introduction

Homayoon Dezfuli, Ph.D.

Office of Safety and Mission Assurance

NASA Headquarters

October 30, 2014

NASA Headquarters

Washington, DC

PRE-DECISIONAL

NASA EROM Workshop – Purpose & Structure

• The purpose of this workshop is to:

‒ Present the proposed EROM framework that NASA Office of Safety

and Mission Assurance (OSMA) has been developing as part of a

larger project of Agency-wide Risk Management (RM) conceptual

guidance deriving from NPR 8000.4A, Agency Risk Management

Procedural Requirements

‒ Solicit stakeholder feedback, ideas, issues, and lessons learned

from the NASA strategic management community and other

stakeholders that can be used to develop further the framework

• The structure of this workshop is interactive and participatory:

– The morning session is split between presentations by the OSMA

and the NASA Office of the Chief Financial Officer / Strategic

Investments Division (OCFO/SID)

– The afternoon session is reserved for group discussion

o Understanding viewpoints and concerns along the way to

consensus on EROM principles

o Strategies for continuing development of EROM at NASA

2

Evolution of Risk Management

at NASA

3

Historical Perspective on NASA Risk

Management (RM) • No “formal,” systematic RM process in

NASA until mid-1990s.

• Then came the “Continuous Risk Management” or “CRM” process:

– Originally developed by Carnegie Mellon University for the Department of Defense

– Brought increased attention to risk over the next decade

– Initially applied to program/projects and later to institutions

– Stressed management of individual risk issues during implementation/operation phases

– Risks were identified via brainstorming

– Individual risks were analyzed qualitatively and arrayed on a “risk matrix” of severity vs. likelihood

• RM ≡ CRM

Communicate and

Document

Id nt ye if

An

lye

az

Pnla

acT

kr

Con

lort

4

RM Approach After 2008

• In 2008, we took the next step in the evolution of RM by

revising NPR 8000.4A, Agency Risk Management Procedural

Requirements

• NPR 8000.4A evolved NASA’s risk management to entail two

complementary processes: Risk-Informed Decision Making

(RIDM) and Continuous Risk Management (CRM)

RM RIDM + CRMRM RIDM + CRM

– RIDM informs systems engineering decisions

through better use of risk and uncertainty

information in selecting among alternatives

and establishing baseline performance

requirements

– CRM manages risks over the course of the

development and implementation phases of

the life cycle to assure that requirements

related to safety, technical, cost, and schedule

are met

` 5

Motivations for Changing RM in 2008

• To promote a RM approach that is holistic and coherent across the

Agency:

– Agency strategic goals explicitly drive RM activities at all levels

– All risk types and their interactions are considered collectively during

decision-making

– Focusing on “forest-level” risk picture, from which the tree-level “individual

risks” should be derived, and within the context of which the “tree-level”

risks are prioritized and managed

– RM activities are coordinated horizontally and vertically across the Agency

• To increase rigor in the technical basis for direction-setting, requirement

setting, and risk response/acceptance decisions:

– Better and more useful treatment of uncertainty

– Having an integrated perspective on risks when analyzing competing

alternatives

– Better characterization of the risk that a decision-maker is accepting when

making commitments to stakeholders

• To match better the stakeholder expectations and the “true” resources

required to address the risks to achieve those expectations through

‒ More stress on development of credible performance requirements

6

The RM Process Begins with NASA

Strategic Goals

• Within NASA’s organizational

hierarchy, high-level

objectives (NASA Strategic

Goals) flow down in the form

of progressively more

detailed performance

requirements, whose

satisfaction assures that

objectives are met

• RIDM is designed to maintain

focus on strategic goals as

decisions are made

throughout the hierarchy

• CRM is designed to manage

“risks” in the context of

requirements

RID

M P

roce

ss

7

NASA RM Framework

• RM as RIDM+CRM operates at each level of the NASA hierarchy, with

interfaces for the flowdown of requirements, the elevation of

decisions, and the communication of risk information

Performance Requirements DevelopmentPerformance Requirements Development

8

NPR 8000.4A Definition of Risk is Based on

Meeting Performance Objectives

“Risk is the potential for performance shortfalls, which may be

realized in the future, with respect to achieving explicitly

established and stated performance requirements.”

• Performance shortfalls may be related to institutional support for

mission execution or to any one or more of the following mission

execution domains:

– Safety (e.g., avoidance of injury, fatality, destruction of key

assets, environmental damage)

– Technical (e.g., thrust or output, amount of observational data

acquired)

– Cost (e.g., execution within allocated cost)

– Schedule (e.g., meeting milestones)

9

The RIDM Process and its Themes

Risk-Informed Decision Making (RIDM)

Identification of Alternatives Identify Decision Alternatives (Recognizing

Opportunities) in the Context of Objectives

Risk Analysis of Alternatives Risk Analysis (Integrated Perspective) and

Development of the Technical Basis for

Deliberation

Risk-Informed Alternative Selection Deliberate and Select an Alternative and

Associated Performance Commitments

Informed by (not solely based on) Risk

Analysis

To Requirements Baselining

• The importance of considering multiple

objectives across all mission execution

domains (safety, technical, cost, schedule)

• The importance of close ties between the

selected alternative and requirements

derived from it

– Match commitment levels with the decision

maker’s risk tolerance limits

– Develop achievable requirements

• The importance of a documented decision

rationale

10

The CRM Process and its Themes

Communicate and

Document

Id nt ye if

An

lye

az

Pnla

acT

kr

Con

lort

• CRM process is oriented

toward keeping the

potential for performance

shortfalls within tolerable

limits

• At the micro level, the process is largely

unchanged

– However, the context within which CRM

operates is now defined explicitly

– All “risks” managed within an organizational

unit are pegged to the performance

objectives or requirements that that unit is

working to

• There is new emphasis on

– More formalism and technical rigor

– Consideration of aggregate risk for risktradeoff (when feasible)

– “Institutional” risk management

– Cross-cutting risks

– Formalizing risk acceptance and decisionelevation

• Risk responses are to be based on

addressing the most important causes of

risk (i.e., the risk drivers)

11

For Background:

Some details of RIDM and CRM

12

RIDM Process Identification of Objectives and Alternatives

• An objectives hierarchy (OH) is constructed by subdividing the top-level objectives into more detailed objectives, thereby clarifying the intended meaning.

• At the first level of decomposition, the top-level objective is partitioned into the mission execution domains of Safety, Technical, Cost, and Schedule.

• Within each domain, the objectives are further decomposed until appropriate quantifiable performance objectives are generated.

• Alternative design solutions are generated as part of the Systems Engineering process

Scientific Orbiter for

Collection of Atmospheric

Data at Planet X

Propulsive Deceleration into

Planetary Orbit

Aero-capture Maneuver to

Decelerate into Planetary

Orbit

High Fidelity Science

Instrumentation

Low Fidelity Science

Instrumentation

Small Launch Vehicle

Medium Launch Vehicle

Large Launch Vehicle




High Fidelity Science

Instrumentation

Low Fidelity Science

Instrumentation






Large Launch Vehicle 13

RIDM Process Risk Analysis of Alternatives (Risk AOA)

• The goal is to develop a risk analysis framework that integrates domain-specific performance assessments and quantifies the performance measures

– Risk Analysis - probabilistic modeling of performance

Risk Analysis

of an Alternative

Uncertain Conditions

Performance Measure 1

Performance Measure n

Probabilistically - Determined

Outcomes

Funding

Environment

Technology

Development

Limited

Data

Operating

Environment

Etc.

* Performance measures depicted for a single alternative

Design, Test &

Production

Processes

…• Safety Risk

• Technical Risk

• Cost Risk

• Schedule Risk

Product of

Risk

Analysis

• Establishing a transparent framework that:

– Operates on a common set of performance parameters for each alternative

– Consistently addresses uncertainties across mission execution domains and across

alternatives

– Preserves correlations between performance measures

14

RIDM Process Deliberation of the Merits of Each Alternative in the Context of

Performance Commitments (notional)

Imposed

Constraint

Notional Risk Tolerances: High Moderate Low

Direction of Goodness

Alternative

A

Alternative

B

Alternative

C

Payload

CapabilityReliability Cost & Schedule

Performance Measures*

* These are arbitrary, notional choices

Performance commitments are set at performance measure values that

correspond to given risk tolerances

Risk tolerances given by the shadedareas under the pdfs, on the “bad”

side of the performance commitments

PCA1 PCA2 PCA3

PCB1 PCB2 PCB3

PCC1 PCC2 PCC3

15

CRM The Risk Statement - Structure

• The Risk Statement

– “Given that [CONDITION], there is a possibility of [DEPARTURE]

adversely impacting [ASSET], thereby leading to [CONSEQUENCE].”

• Example:

Given that

[CONDITION: the state of knowledge of Planet X’s atmosphere is

limited; the fact that it is difficult to ascertain more information

about Planet X’s atmosphere from Earth; and the fact that the

spacecraft contains radioactive material], there is a possibility of

[DEPARTURE: unanticipated atmospheric characteristics during

the aerocapture maneuver at Planet X leading to a less-than-

optimal trajectory] adversely impacting

[ASSET: the spacecraft], thereby resulting in

[CONSEQUENCE: spacecraft breakup and radioactive

contamination of Planet X].

16

The Tactical and Strategic Dimensions of CRM

IDENTIFYIndividual Risks

ANALYZEIndividual Risk Criticality and Risk to Performance

Requirements Using Quick Look Approach

ANALYZE Risk to Performance

Requirements Using Graded Analysis Approach

PLAN For Tactical Response

PLANFor Strategic

Response

TRACKRisk Drivers

CONTROLRisk Drivers

No

Yes

Any Near Term Actions

Required?

Any Risk that Cannot Be

Controlled with Existing Plan?

Yes: New Risk

NoYes: Change to Existing Risk

Strategic Planning

Successful?

Reevaluate Performance Requirements (RIDM & SE)

Yes

No

Elevation Needed?

Elevation Needed?

Elevate & Return to Identify

Elevate & Return to Identify

Yes

Yes

No

No

COMMUNICATEDOCUMENT

RIDM Handoff to

CRM

SE Performance Requirement

s

Define & Maintain Risk Management Plan

17

CRM Tactical and Strategic Criticality Rankings of Individual Risks

Ranking by Attributes

• Likelihood and Severity Attribute: The

likelihood that the individual risk could

cause one or more performance risks to

cross over one or more tolerability

thresholds (i.e., from tolerable to marginal,

from tolerable to intolerable, or from

marginal to intolerable)

• Uncertainty Attribute: The degree to which certain

qualitative uncertainty factors (e.g., uniqueness,

complexity, detectability) are inherent in the

individual risk

• Timeframe Attribute: The amount of time available

before a response must be initiated

Prioritizing the Attributes

• Tactical (Near-Term) Ranking: A ranking based on

amalgamating the three criticality attributes in the

following order: timeframe first, likelihood second,

and uncertainty third.

• Strategic (Long-Term) Ranking: A ranking based on

amalgamating the three criticality attributes in the

following order: likelihood and severity first,

uncertainty second, and timeframe third.

18

Categorizing Risks with Taxonomies

• Taxonomies can help categorize

risks for a more efficient RM

process

– They can be helpful when

deciding on RM responses to

identified risks

– With taxonomies, it may be

possible to craft more efficient

responses that simultaneously

address all or most of the

elements within a given

taxonomic category

• The CRM process involves

three distinct taxonomies

related to the Risk Statement

structure:

– Condition/departure taxonomy

– Asset taxonomy

– Consequence taxonomy

PotentialDepartures *

TechnicalDepartures

ProgrammaticDepartures

ProcessControl Issues

EngineeringIssues

DesignIssues

Subcontractorand Supplier

Issues

HumanResource

Issues

InstitutionalResource

Issues

BudgetResources

ScheduleResources

DesignStaffing

ManufacturingStaffing

ProductAssuranceStaffing

RequirementsFlowdown

SuppliedItem Defects

PerformanceRequirements

Design V&V

ReliabilityDesign

ManufacturingSpecifications

AssemblySpecifications

Inspectionand TestSpecifications

PartsQualityControl

Manufacturingand AssemblyControl

Inspectionand TestControl

MarginManagement

Coordination

TimelinessResearch &Analysis Staffing

Qualification TestStaffing

Launch SupportStaffing

Flight OperationsStaffing

InformationTechnology& Management

Facility &EquipmentResources

ExternalDepartures

PoliticalIssues

EconomicIssues

Public RelationsIssues

Acts of God

*Note: The taxonomy for “Departures” is the same as to the taxonomy for “Conditions”

19

Where we are today and where

we need to go with NASA RM

20

RM at NASA Today • The following NASA RM-related documentation exists:

NPR 8000.4A December 16, 2008

NASA/SP-2010-576 April 2010

NASA/SP-2011-3422 November 2011

• RM continues to evolve for NASA programs/projects; it has not

yet reached the levels described in these documents, but…

• In light of Government requirements and policies such as

GPRAMA and OMB Circulars A-11 and A-123, among others…

• It is time to work on extending RM to the Agency (Enterprise)

level … which brings us to EROM

21

Internal Pull Functions for EROM at NASA

• NPR 1000.5B, Policy for NASA Acquisition:

o “It is NASA policy to incorporate a risk-informed decision-making process

that includes the identification, analysis, and management of programmatic,

institutional, technical, cost, schedule, environmental, safety, management,

industry, and external policy risks that might jeopardize the successful

execution of the Agency's acquisition strategies.” (1.e.(9))

• NPR 8000.4A, Risk Management Procedural Requirements:

o “This NPR establishes requirements applicable to all levels of the Agency.”

(P.1.b.); “This NPR applies to all Agency activities, including NASA

Headquarters…” (P.2.a.)

o “Risk management at the Agency level addresses risks identified at the

Agency level, as well as risks elevated from Mission Directorates and

Mission Support Offices.” (1.2.1.f)

o “Risk management at the Agency level integrates the full spectrum of risks.

(1) Dealing with risk as a strategic issue, from a high Agency-level/corporate

perspective.

(2) Engaging all functions and line management levels in the process.

(3) Bridging the gaps between domains of risk management (e.g., safety, technical,

financial/cost, institutional).” (1.2.1.g)

o “At the Agency level, emphasis is placed on optimizing and improving the

Agency's mission objectives and goals versus individual project or program

goals/objectives.” (1.2.1.h)

22

External Pull Functions for EROM at NASA • OMB Circular A-11, Preparation, Submission, and Execution of the

Budget:

o “All agencies should implement the enterprise risk management guidance as

appropriate for the agency mission and in accordance with agency-specific

programs.” (270.1)

o “Agencies… should identify, measure, and assess challenges related to

mission delivery, to the extent possible. Enterprise risk management (ERM)

is an effective agency-wide approach to addressing the full spectrum of the

organization’s risks by understanding the combined impact of risks as an

interrelated portfolio, rather than addressing risks only within silos. ERM

provides an enterprise-wide, strategically-aligned portfolio view of

organizational challenges that, when brought together, provides better

insight about how to most effectively prioritize and manage risks to mission

delivery.” (270.24)

• Aerospace Safety Advisory Panel (ASAP), Annual Report for 2013:

o “NASA should consistently provide formal versus ad hoc processes for

managing risk with clear accountability. ” (Recommendation 2014-AR-05)

“We have observed that NASA often relies on the quality and integrity of

its personnel to ‘do the right thing,’ which makes risk management

personality-dependent rather than part of formal processes.”

23

Development of NASA/SP-2014-615 • The draft NASA/SP-2014-615, Enterprise Risk & Opportunity

Management: Concepts for NASA’s Consideration, has been developed

to be:

o Responsive to internal/external pull functions

o Consistent with the RM framework OSMA has been developing over the past

several years, as presented in the NASA RM Handbook

o Consistent with ERM best practices such as those in COSO* and ISO/FDIS-

31000 guidance

NASA RM Framework ERM Best Practice NASA/SP-2014-615

• This draft publication is intended to be generally applicable to NASA

and other government/non-profit agencies and organizations

• Some terminology and process assumptions might not be fully consistent

with current NASA practice

• Where inconsistencies need to be addressed, your feedback is encouraged 24

*Committee of Sponsoring Organizations of the Treadway Commission

Effective RM is Enterprise-Wide RM

• EROM is a necessary element of an integrated, enterprise-wide

RM function

o Objectives and risk tolerances flow down from the Agency level to Programs,

Projects, and Institutions

o Objectives and risk tolerances flow from Programs/Projects to Institutions

o Risk reporting protocols

assure appropriate situational

awareness of subordinate

organizations’ risk statuses at

each level of the NASA

organization

o Risk elevation protocols

assure that risk management

decision-making occurs at the

appropriate level of the NASA

organization

o Each level’s RM function

oversees subordinate RM

functions to:

Ensure their effectiveness

Identify systemic or cross-

cutting risks

EROM

(Enterprise Level)

RM

(Programs/

Projects)

RM

(Institutional)

25

Core Principles Underpinning EROM per

OSMA’s EROM Concepts Paper

• EROM is an instance of RM

o EROM specializes application of NPR 8000.4A to the Agency level

o RM = RIDM + CRM

• EROM is the Enterprise-level component of an integrated,

Enterprise-wide RM process

o Integrates with program/project and institutional RM processes

• The principle purposes of EROM are:

o To risk- and opportunity-inform the development of the Agency’s

strategic plan

o To risk- and opportunity-inform the development of the Agency’s

portfolio plans

o To manage risks and opportunities that relate to the achievement of

the Agency’s strategic objectives (Internal, External, Elevated from

Programs/Projects/Institutions, Cross-Cutting)

o To facilitate communication of risks and opportunities within NASA

and with other agencies of the Federal Government 26

Enterprise Risk and

Opportunity Management

(EROM)

27

What is EROM?

• The overall objectives of Enterprise Risk and Opportunity

Management (EROM) are to facilitate the successful development

of the Agency’s Strategic Plan, to promote an optimal means for

implementing the plan, and to evaluate performance with respect

to the plan

• The method for doing this is to seek an integrated, optimal

balance between minimizing the potential for loss (risk) while

maximizing the potential for gain (opportunity) with respect to the

Agency’s overall mission

o The focus on the Agency’s overall mission is the reason for the “E” in

“EROM”

o The overall mission emphasis implies an integration of risk and opportunity

management over all programs, projects, initiatives, and activities in the

Agency’s portfolio

o Achievement of an optimal balance implies the involvement of the Decision

Maker(s) in setting maximum tolerable levels for risk and minimum desirable

levels for opportunity 28

Historically, what have been the defining

aspects of EROM (a.k.a. ERM)?• According to the Committee of Sponsoring Organizations of the

Treadway Commission (COSO), Enterprise Risk Management

encompasses:

o Aligning risk appetite and strategy

o Enhancing risk response decisions

o Reducing operational surprises and losses

o Identifying and managing multiple and cross-enterprise risks

o Seizing opportunities

o Improving deployment of capital

• The COSO framework has been incorporated within an international

standard (ISO/FDIS-31000, “Risk Management - Principles and

Guidelines,” 2008), and has become a best practice for large

commercial enterprises

• The COSO – ISO/FDIS-31000 framework focuses principally on financial

risks and opportunities 29

How would EROM for agencies like NASA differ from

EROM for commercial enterprises?

• For EROM to be effective at agencies like NASA, it must focus on the objectives

and constraints that NASA is required to satisfy, including:

o Achievement of scientific and technical gains in the public interest, over both short-

term and long-term horizons

o Exploration of new frontiers and knowledge development

o Partnerships with other nations and with commercial enterprises

o Public education and involvement

o Objectives common to both commercial and nonprofit enterprises, including

institutional development and maintenance, legal and reputational protection, and

financial health

o Specific annual outcomes mandated by Congress and the White House

o Satisfaction of Government requirements and policies such as those prescribed within

GPRAMA, OMB Circular A-11, and OMB Circular A-123, among others

• These objectives must be met within financial, schedule, and political

constraints that are subject to periodic change due to changing Administrations

in Washington and changing public priorities

• The EROM framework for NASA should adhere to the basic principles in NPR

8000.4A and in the NASA Risk Management Handbook concerning the roles of

Risk-Informed Decision Making (RIDM) and Continuous Risk Management (CRM)

30

EROM in Relation to the Organization

• The EROM framework supports decisions made within the strategic

management, program/project management, and mission support

management functions that already exist within the agency

31

EROM in Relation to the Organization (cont’d)

• Each level has responsibilities for planning, plan implementation,

performance evaluation, and for communicating the results of these

activities with the other levels

32

EROM at the Executive Level

• Information flows into and out of the executive level from/to several

entities

Internal: Mission support (institutional) and program/project entities

External: Governance and marketplace entities

33

Operationalization of EROM

• EROM is operationalized within an organization through the introduction of risk-

and-opportunity-informed decision making and continuous risk and opportunity

management

• Risk-and-opportunity-informed decision making enters into the planning stage

for each management function (including strategy and implementation

planning)

• Continuous risk and opportunity management enters into the evaluation stage

for each management function (including implementation corrections and re-

planning when necessary)

Planning

EXECUTIVE LEVEL, INSTITUTIONAL LEVEL,

OR PROJECT LEVEL

ImplementationEvaluation

34

Some Important Definitions Concerning Risk

and Opportunity • Risk is defined as the possibility for future performance shortfalls with respect

to achieving explicitly established and stated strategic objectives

• Opportunity is defined as the possibility for future performance improvements

with respect to achieving the explicitly established mission of the Agency

o Some opportunities reduce the risk of not meeting one or more already-stated strategic

goals or desired outcomes

o Other opportunities create an opening for changing strategic objectives or desired

outcomes to align them better with the agency’s vision and mission

• Potential risk (or risk projection) refers to the possibility of a goal, objective, or

desired outcome not being met when a desired functionality has been

conceptualized but the architecture has not yet been selected from among

various alternatives.

• Potential opportunity (or opportunity projection) refers to the possibility of an

existing goal, objective, or desired outcome being met more effectively, or a new

goal, objective, or desired outcome becoming feasible, when a desired

functionality has been conceptualized but the architecture has not yet been

selected

o It also refers to an opportunity for which the actions required to realize the benefit may

be available in the future depending on factors that are presently unknown 35

Some Important Definitions Concerning Risk

and Opportunity (cont’d)

• A risk scenario is an individual concern that, if it becomes a reality,

could present a risk to the ability to achieve a strategic objective

o A risk scenario is expressed in terms of a risk statement

• An opportunity scenario is an individual opening that, if it is acted

upon, could lead to an opportunity to either increase the likelihood of

achieving a strategic objective or open the possibility of defining a

new objective that coincides with the Agency’s mission

o An opportunity scenario is expressed in terms of an opportunity

statement

• Cumulative risk refers to the cumulative effect of all the risk

scenarios in producing a decrease in the likelihood of being able to

meet a strategic objective

• Cumulative opportunity refers to the cumulative effect of all the

opportunity scenarios in producing an increase in the likelihood of

being able to meet a strategic objective or in opening the possibility of

defining a new objective that coincides with the Agency’s mission

36

In what areas does EROM facilitate strategic

planning, plan implementation, and evaluation of

performance for an agency like NASA?

• The EROM approach assists in allocation decisions for the Agency’s budgets,

facilities, and human resources by balancing the opportunity for success against

the risk of failure and the cost versus the potential gain.

• EROM helps provide focus for institutional and mission support functions and

initiatives by identifying Agency-level risks and opportunities that pertain to

staffing requirements, the qualifications of the staff, test facility requirements,

information technology needs, and other program/project support needs.

• The EROM approach facilitates consistency and coherence in the treatment of

risks and opportunities across different entities and organizational units.

• EROM facilitates the rollup of risks and opportunities from various entities within

NASA to Agency level, and enables an agile response to risks and opportunities

that require immediate action.

• The benefits that derive from using an EROM approach within NASA are

particularly significant for missions that are complex and involve difficult choices

between alternative pathways (e.g., the proposed flexible path to Mars)

37

Interfaces between EROM Activities and Strategic

Management Activities: Strategic Planning

38

The Elements of NASA’s Strategic Objectives

Hierarchy Strategic Goal 1.

Strategic Objective “A” Strategic Objective “B”

Multi-Year Performance Goal “a” Multi-Year Performance Goal “b”

Annual Performance Indicator “x”

Strategic goals pertain to general direction-defining accomplishments that cover the next 10 or

20 years and beyond. (Example: expand the frontiers of knowledge, capability, and opportunity

in space)

Strategic objectives cover up to a 10-year timeframe. (Example: expand human presence into

the solar system and to the surface of Mars)

Strategic outcomes cover up to a 10-year timeframe. (Example: sustain the operation and full

use of the International Space Station)

Multi-year performance goals (MYPGs) cover up to 5 years. (Example: study Earth from space

to understand human impact on our planet by launching at least two missions by 2015.)

Includes Agency Priority Goals and Cross-Agency Priority (CAP) Goals.

Annual performance indicators (APIs) pertain to a 1-year time period. (Example: complete the

Orbiting Carbon Observatory-2 (OCO-2) Systems Integration Review.) 39

Risk AOA during Strategic Planning: Analysis of Potential Risks and Opportunities

• The strategic planning process may be viewed as a selection of a set

of objectives (i.e., strategic goals, strategic objectives, MYPGs, and

APIs) from among a variety of alternative sets of objectives

• A principal product that EROM provides through the process is

analysis of each alternative set of objectives in terms of the potential

for risks and opportunities that could imperil or aid the successful

achievement of the Agency’s mission

• The EROM approach identifies and evaluates potential risk scenarios,

potential opportunity scenarios, and potential introduced risk

scenarios (i.e., additional risks that might occur if an opportunity

were exploited)

40

Risk AOA during Strategic Planning (cont’d)

Strategic Goal 1.

Risk

Projection

Opportunity

Projection

Strategic Objective “A”

Risk

Projection

Opportunity

Projection

Strategic Objective “B”

Risk

Projection

Opportunity

Projection

Potential Opportunity Scenarios

Multi-Year Performance Goal “a”

Risk

Projection

Opportunity

Projection

Multi-Year Performance Goal “b”

Risk

Projection

Opportunity

Projection

Potential Introduced Risk

Scenarios


Risk

Projection

Opportunity

Projection

Potential Risk Scenarios

Potential

RiskTBD Tolerable Marginal Intolerable

Potential

Opportunity

Level

TBD Insignificant Marginal Significant

41

Risk AOA during Strategic Planning (cont’d) Strategic Goal 1.

Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Potential Risk

Scenarios

Key Risk Indicators

Risk Indicator

Status

Cumulative Risk

Projection

X X X

X X X X X X



Key Opportunity Indicators

Opportunity Indicator

Status

Cumulative Opportunity Projection

X X X

X X X X

X X

Potential Introduced Risk Scenarios

Potential Introduced

Risk Scenarios

Key Risk Indicators

Risk Indicator

Status

Cumulative Introduced

Risk Projection

X X X

X X X X X X

PotentialRisk

TBD Tolerable Marginal Intolerable

Potential

Opportunity Level TBD Insignificant Marginal Significant 42

NASA Workshop on Enterprise Risk and

Opportunity Management

Technical Considerations and Asteroid

Retrieval Example

Dr. Allan S. Benjamin

October 30, 2014

PRE-DECISIONAL

Not for Distribution unless Approved by Homayoon Dezfuli, NASA Office of Safety & Mission Assurance

1

Objectives Hierarchy (a.k.a. Strategic Performance Framework), Risk and Opportunity Scenarios, and Leading Indicators

• The identification of strategic objectives and development of performance goals and indicators are done in different processes and in different timelines

• Risk and opportunity scenarios and leading indicators are identified at each level of the objectives hierarchy by the EROM analysts working with SMEs and TAs

• Some scenarios and indicators at higher levels are rolled up from lower levels

• More on leading indicators later

Strategic Goal 1.

Risk Scenarios & Leading Indicators

Opportunity Scenarios & Leading Indicators







Multi-Year Performance Goal “X”



Multi-Year Performance Goal “Y”






Annual Performance Indicator “y”



Information provided by EROM analysis team (together with SMEs and TAs)

Information provided by EROM analysis team and also rolled up from lower levels

Not for Distribution unless Approved by Homayoon Dezfuli, NASA Office of Safety & Mission Assurance 2

Example of an Opportunity Scenario that is Specified Directly at a Higher Level and a Risk Scenario that is Rolled-up from Lower Levels

• Opportunity for 10-Year Strategic Objective a: There is a possibility that new technology in the

area of electric propulsion may become available within a ten-year time frame, making it

possible to gain a far greater knowledge of the outer solar system over the next decade.

• Risk for 5-Year Performance Goal X: If milestone slippages that have occurred during the past

year in Program X are not corrected, there is a possibility that System X will not be ready for

launch in five years.

• Risk for 5-Year Performance Goal Y: If milestone slippages that have occurred during the past

year in Program Y are not corrected, there is a possibility that System Y will not be ready for

launch in five years.

• Roll-up Risk for 10-Year Strategic Objective a: If Systems X and Y are not successfully

launched in five years, there is a possibility that exploration of the outer solar system will be

severely impaired over the next decade.


The Goal of EROM is to Find a Near-Optimal Balance between Risks and Opportunities

Examples of Risk and Opportunity Parity Statements

• [Example Risk Tolerance Statement 1]: A risk scenario is considered to reach the risk tolerance

boundary if the likelihood of failure to land humans on Mars by 2035 increases from its targeted

value of 10% to 20%.

• [Example Risk Tolerance Statement 2]: A risk scenario is considered to reach the risk tolerance

boundary if the targeted date of 2035 for landing humans on Mars increases to 2045.

• [Example Opportunity Appetite Statement]: An opportunity scenario is considered to reach the

opportunity appetite boundary if the launch system for landing humans on Mars will also be

capable of being used for exploratory missions to the moons of Jupiter and Saturn.

4 4

The Role of Leading Indicators in Evaluating the Status of Strategic

Objectives and Performance Goals

• Risk and opportunity leading indicators are used to infer the likelihood that each strategic objective and performance goal in the objectives hierarchy will be successfully achieved within the assigned timeframe

• During the strategic planning process, they are used to help decide from among various candidate strategic objectives and performance goals

• During the performance evaluation process, they are used to assess how the likelihoods of success based on current conditions stand with respect to the initial projections

• Leading indicators should possess the following characteristics:

o Quantifiability o Correlatability o Actionability

5

Strategic Goal 1.

Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Risk

Projection

Opportunity

Projection


Potential Risk

Scenarios

Leading Risk

Indicators

Leading Indicator

Status

Cumulative Risk

Projection

X X X

X X X X X X



Leading Opportunity Indicators

Opportunity Indicator

Status

Cumulative Opportunity Projection

X X X

X X X X

X X


Potential Introduced

Risk Scenarios

Leading Risk

Indicators

Leading Indicator

Status

Cumulative Introduced

Risk Projection

X X X

X X X X X X

Potentia

l Risk

Level


Potential

Opportunity Level TBD Insignificant Marginal Significant

5

Organizing Risks and Opportunities by Category Helps to Ensure that All Important Scenarios and Associated Leading Indicators are Identified

Category Sub-Category

Owner (Entity)

Example Risk/Opportunity Events

Affected Strategic

Outcomes

Internal Leading Indicators (Examples)

External Leading Indicators (Examples)

New technology opportunities

Mission performance

Specify one

Opportunity to enhance mission performance through application of new technology

Specify one or more

Initiation of and results from Internal state-of-the-art assessments

Technology trends in areas pertinent to NASA missions (propulsion, materials, etc.)

Number of patents applied for

TRL rate of progress

Institutional capability

Specify one

Opportunity to enhance institutional capability or reducing institutional cost through new technology

Specify one or more


Technology trends in areas pertinent to NASA institutional capabilities

Information technology

Specify one

Opportunity to enhance IT capabilities

Specify one or more


IT trends

Education and partnerships

Public outreach

Specify one

Failure to meet public education goals

Specify one or more

Missed milestones

Low enrollment in educational programs

Domestic technology transfer

Specify one

Failure to meet technology transfer goals

Specify one or more

Missed milestones Lack of interest or progress from potential commercial partners

Number of technology transfer agreements

Trends regarding the sharing of sensitive information and materials

International partnerships

Specify one

Failure to meet international partnership goals

Specify one or more

Missed milestones for which NASA is responsible *

Lack of interest or progress from potential international partners

New regulations regarding sensitive info

Competition from a foreign country 6

Organizing Risks and Opportunities by Category Helps to Ensure that All Important Scenarios and Associated Leading Indicators are Identified

Category Sub-Category

Owner (Entity)

Example Risk/Opportunity

Events

Affected Strategic

Outcomes

Internal Leading Indicators (Examples) External Leading Indicators (Examples)

Financial Funding Specify one

Funding cut Specify one or more

Economic indicators

Congressional makeup

Changes in national priorities

Budgeting Specify one

Insufficient contingency

Specify one or more

Contingency compared to other programs / projects *

Rate of spending compared to other programs / projects *

Unresolved assignment of roles and responsibilities

Costs Specify one

Increased cost of materials and/or purchased services

Specify one or more

Price trends

Threats of foreign conflicts or political changes (affecting rare material costs, e.g.)

Supplier financial problems

Specify one

Increased cost of operations

Specify one or more

Monthly cost reports *

Low scores on self assessments and audits *

Specify one

Milestone slippage costs

Specify one or more

Earned value reports * Government shutdown

Specify one

Accident costs Specify one or more

Precursors, anomalies, mishap reports *

7

Some Important Definitions Concerning Risk and Opportunity

• Risk is defined as the possibility for future performance shortfalls with respect to achieving

explicitly established and stated strategic objectives

• Opportunity is defined as the possibility for future performance improvements with respect to

achieving the explicitly established mission of the Agency

o Some opportunities reduce the risk of not meeting one or more already-stated strategic goals or

desired outcomes

o Other opportunities create an opening for changing strategic objectives or desired outcomes to align

them better with the agency’s vision and mission

• Potential risk (or risk projection) refers to the possibility of a goal, objective, or desired

outcome not being met when a desired functionality has been conceptualized but the

architecture has not yet been selected from among various alternatives.

• Potential opportunity (or opportunity projection) refers to the possibility of an existing goal,

objective, or desired outcome being met more effectively, or a new goal, objective, or desired

outcome becoming feasible, when a desired functionality has been conceptualized but the

architecture has not yet been selected

o It also refers to an opportunity for which the actions required to realize the benefit may be available in

the future depending on factors that are presently unknown 8


Some Important Definitions Concerning Risk and Opportunity (Cont.)

• A risk scenario is an individual concern that, if it becomes a reality, could present a risk to

the ability to achieve a strategic objective

o A risk scenario is expressed in terms of a risk statement

• An opportunity scenario is an individual opening that, if it is acted upon, could lead to an

opportunity to either increase the likelihood of achieving a strategic objective or open the

possibility of defining a new objective that coincides with the Agency’s mission

o An opportunity scenario is expressed in terms of an opportunity statement

• Cumulative risk refers to the cumulative effect of all the risk scenarios in producing a

decrease in the likelihood of being able to meet a strategic objective

• Cumulative opportunity refers to the cumulative effect of all the opportunity scenarios in

producing an increase in the likelihood of being able to meet a strategic objective or in

opening the possibility of defining a new objective that coincides with the Agency’s mission

9 Not for Distribution unless Approved by Homayoon Dezfuli, NASA Office of Safety & Mission Assurance 9

Example Asteroid Retrieval Mission

• A high-level, limited-scope, hypothetical example of how the EROM analysis might be conducted to evaluate:

The potential risks and opportunities associated with capturing an asteroid and redirecting it to lunar orbit

The effect on the cumulative risks and opportunities associated with the strategic objectives in the existing strategic plan

• The example illustrates these activities within the context of strategic planning, but the same general principles

would apply to strategic performance evaluation

The principal difference is that the scenarios and leading indicators would be based not only on historical experience and

expert judgment, but also on experience gained during the design, implementation, and performance of the mission

• The candidate design involves a robotic capture system and an electric propulsion (EP) system, both of which

require development and proof of capability

The robotic capture system has to be able to grab a spinning asteroid that is about 20 meters along its longest dimension

(larger than currently being considered within NASA) and that has at present an undefined shape, hardness, and spin rate

The capture system must weigh less than 1000 kg

The EP system has to operate for at least 3.5 years, and must be able to achieve a specific impulse of at least 6000 sec.

Some of the parts are provided by suppliers

• The example is for illustration purposes only

All entries and values for this example are invented and are not intended to represent an actual design or situation.


Excerpt of an Example Objectives Hierarchy Incorporating the Asteroid Retrieval Mission

This example postulates that a new strategic objective 1.X, called “Retrieve an Asteroid,” has been proposed for addition to the current strategic plan

The objective is to determine whether the opportunities provided by this candidate addition justify the risks

The EROM approach identifies and evaluates potential risk scenarios, potential opportunity scenarios, and potential introduced risk scenarios (i.e., additional risks that might occur if an opportunity were exploited)

Note that the identification of strategic goals and objectives and the development of performance goals are done in different processes and in different timelines


Example Ranking of Potential Risks: Electrically Powered Asteroid Redirect System

All rankings are notional and hypothetical

Potential risk scenarios concern insufficient specific impulse, grid erosion, and supplier costs

“Response” and “watch” trigger values for the leading indicators are determined from:

(a) System performance requirements

(b) Spacecraft weight allocations (c) P(LOM) thresholds and goals

established by the agency (d) Constraints on total system cost (e) The budget organization’s view

of how the cost should be allocated

• Current values for the leading indicators are determined from:

(a) Test results (b) Historic trends (c) Engineering analyses (d) Related operating experience (e) Expert judgment

Multi-Year Performance Goal 1.X.2 Develop an EP asteroid redirect system

Potential Risk Potential Opportunity


Potential Introduced Risk Scenarios Potential Risk Scenarios

Scenario Description Leading Indicator

Description Response

Trigger Watch Trigger

Observed Current Value

Overall Rating of Potential

Risk

May not be able to achieve sufficient impulse

within weight requirement

Specific impulse test results 6,000 sec 8,000 sec 5,000 sec

Intolerable

Historic weight growth 10 % 8 % 7 %

Erosion from the sputter grid may impact the solar

panels or other components causing LOM

Lowest operating temperature 100 C 50 C 20 C

Supplier costs or time-to-complete may be too high

Market-place labor inflation rate 6 % 4 % 5 %

Potential Risk Level


12

Example Ranking of Potential Opportunities and Introduced Risks:

Electrically Powered Asteroid Redirect System All rankings are notional and hypothetical

Multi-Year Performance Goal 1.X.2 Develop an EP asteroid redirect system

Potential Risk Potential Opportunity


Scenario Description

Leading Indicator Response


Expert Judgment of Current

Value


Opportunity

Breakthrough in alternate electric

thruster technology

Experts’ judgment of likelihood of achieving ISP >

8,000 sec

20 % 50 % 50 % Significant


Scenario Description

Leading Indicator Response


Observed Current Value


Introduced Risk

Development costs of new

technology or time-to-complete may be too high

Historical development time

for new electric thruster

technology

5 years 3 years 3 years

Tolerable (Note 2)

LOM probability with new

technology may be too high

P(LOM) from PRAs and historical experience for

systems involving new EP technology

Mean: 1/20

90% Conf.: 1/10

Mean: 1/40

90% Conf.: 1/20

Mean: 1/50

90% Conf.: 1/15

Note 2: Although the predicted 90% confidence level of P(LOM) is of marginal concern, there is reason to believe that it will more closely approach the mean value in time as uncertainties are reduced

Potential Opportunity Level

TBD Insignificant Marginal Significant

Potential Risk Level


13

Importance of the Asteroid Retrieval Mission to Other Strategic Objectives

EXPAND THE FRONTIERS OF KNOWLEDGE, CAPABILITY, AND OPPORTUNITY IN SPACE

1.1 Expand human presence into the solar system and to the surface of Mars …

1.2 Conduct research on the ISS to enable future space exploration …

1.3 Facilitate and utilize U.S. commercial capabilities to deliver cargo and crew to space

1.4 Understand the sun and its interactions with earth and the solar system …

1.5 Ascertain the content, origin, and evolution of the solar system and the potential for life …

1.6 Discover how the universe works, explore how it began and evolved, and search for life …

1.7 Transform NASA missions … by maturing crosscutting and innovative space technologies

ADVANCE UNDERSTANDING OF EARTH & DEVELOP TECHNOLOGIES TO IMPROVE THE QUALITY OF LIFE

2.1 Enable a revolutionary transformation for safe and sustainable U.S. and global aviation …

2.2 Advance knowledge of earth as a system to meet the challenges of environmental change …

2.3 Optimize agency technology investments, foster innovation, facilitate technology infusion …

2.4 Advance the nation’s STEM education and workforce pipeline …

… EFFECTIVELY MANAGE OUR PEOPLE, TECHNICAL CAPABILITIES, & INFRASTRUCTURE

3.1 Attract and advance a highly skilled, competent, and diverse workforce, …

3.2 Ensure the continued advancement of strategic, technical, and programmatic capabilities …

3.3 Provide secure, effective, and affordable Information technologies …

3.4 Ensure effective management of NASA programs and operations …

The degree to which a risk or opportunity associated with the candidate asteroid mission affects another strategic objective depends on:

• The tolerability of the risk or significance of the opportunity for the asteroid mission

• The importance of the asteroid mission for the other strategic objective

14

Candidate Ranking of the Importance of the Asteroid Retrieval Mission with Respect to Various Strategic Objectives


• The likelihood of success of any strategic goal that is supported by the asteroid retrieval mission depends on:

o The likelihood of success of other initiatives that affect the strategic goal in question

o The importance of the asteroid retrieval mission relative to the other initiatives for the strategic objective in question

o How dependent the other initiatives are on the success of the asteroid retrieval mission (and vice versa)

o The likelihood of success of the asteroid retrieval mission

No. Objec. Type

Narrative Relation of Objective to Asteroid Retrieval

Mission

Importance of Asteroid Retrieval Mission Relative

to the Objective

EXPAND THE FRONTIERS OF KNOWLEDGE, CAPABILITY, AND OPPORTUNITY IN SPACE

1.1 SO Expand human presence into the solar system and to the surface of Mars to advance exploration, science, innovation, benefits to humanity, and international collaboration

1.1.1 MYPG Complete critical milestones in the development of the Space Launch System, Orion, and Exploration Ground Systems for the human exploration of deep space

It is planned for the SLS to lift astronauts and hardware to the lunar orbiting asteroid

HIGH: Failure of the asteroid retieval mission would retard the ability to meet a critical SLS milestone

1.1.2 MYPG Complete System Requirements Reviews by FY 2015 for the Resource Prospector Mission and In-Situ Resource Utilization (ISRU) Demonstration Experiment on Mars 2020 mission

One of the ISRU Demos is to be performed on the lunar orbiting asteroid

HIGH: Failure to demonstrate ISRU on an asteroid will jeopardize some of the objectives of the Mars 2020 mission and the affordability of extraterrestrial exploration and operations in general


Candidate Ranking of the Importance of the Asteroid Retrieval Mission with Respect to Various Strategic Objectives (Cont.)


Number Objective Type

Narrative Relation of Objective to Asteroid Retrieval Mission

Importance of Asteroid Retrieval Mission Relative to the Objective

SERVE THE AMERICAN PUBLIC & ACCOMPLISH OUR MISSION BY EFFECTIVELY MANAGING OUR PEOPLE, TECHNICAL CAPABILITIES, & INFRASTRUCTURE

3.1 SO Attract and advance a highly skilled, competent, and diverse workforce, cultivate an innovative work environment, and provide the facilities, tools, and services needed to conduct NASA’s missions.

3.1.1 MYPG Define and build diverse workforce skills and competencies needed for the Agency’s technology development and deep space exploration

The asteroid retrieval mission helps to define the technical competencies that need to be developed at NASA through hiring and training

MODERATE: If the asteroid retrieval mission fails, time spent in developing the technical competencies is still of value (though arguably less so than if it succeeds)

3.1.4 MYPG Between 2012 and 2016, support the demolition and elimination of obsolete and unneeded facilities.

The testing needs of the asteroid retrieval mission helps identify certain facilities that are still needed

LOW: The cost of retaining facilities needed to support the asteroid retrieval mission is a small percentage of the total mission cost

3.1.5 MYPG Manage coordination of NASA’s international and interagency activities in conjunction with the NASA mission directorates

NASA is seeking international partnerships for the asteroid retrieval mission

MODERATE: NASA already has over 100 cooperative agreements with foreign countries


Level of Risk & Opportunity Contributed by the Asteroid Retrieval Mission to Existing Strategic Objectives

1. EXPAND THE FRONTIERS OF KNOWLEDGE, CAPABILITY, AND OPPORTUNITY IN SPACE A B C

1.1 Expand human presence into the solar system and to the surface of Mars …

1.2 Conduct research on the ISS to enable future space exploration …

1.3 Facilitate and utilize U.S. commercial capabilities to deliver cargo and crew to space

1.4 Understand the sun and its interactions with earth and the solar system …

1.5 Ascertain the content, origin, and evolution of the solar system and the potential for life …

1.6 Discover how the universe works, explore how it began and evolved, and search for life …

1.7 Transform NASA missions … by maturing crosscutting and innovative space technologies

2. ADVANCE UNDERSTANDING OF EARTH & DEVELOP TECHNOLOGIES TO IMPROVE LIFE A B C

2.1 Enable a revolutionary transformation for safe and sustainable U.S. and global aviation …

2.2 Advance knowledge of earth as a system to meet the challenges of environmental change …

2.3 Optimize agency technology investments, foster innovation, facilitate technology infusion …

2.4 Advance the nation’s STEM education and workforce pipeline …

3. … EFFECTIVELY MANAGE OUR PEOPLE, TECHNICAL CAPABILITIES, & INFRASTRUCTURE A B C

3.1 Attract and advance a highly skilled, competent, and diverse workforce, …

3.2 Ensure the continued advancement of strategic, technical, and programmatic capabilities …

3.3 Provide secure, effective, and affordable Information technologies …

3.4 Ensure effective management of NASA programs and operations …

Column A: Importance of the asteroid mission relative to the objective

Column B: Level of risk contributed by the asteroid mission to the objective

Column C: Level of opportunity contributed by the asteroid mission to the objective

Importance:

Low

Moderate

High

Potential Risk:

Tolerable

Marginal

Intolerable

Potential Opportunity:

Insignificant

Marginal

Significant 17

Example Identification of Risk Mitigation and Opportunity Exploitation Options and Risk-Informed Internal Controls based on Asteroid Retrieval Mission Results

• (Hypothetically,) the biggest risk to the objective of retrieving an asteroid is the possibility that current EP technology may not provide sufficient specific impulse

• The most promising opportunity is the potential for a breakthrough in an alternate EP technology

• A response action consistent with that information would be to accelerate attempts to develop an alternate EP technology that has higher specific impulse potential

• If this response action were pursued, a high priority would have to be assigned to developing and implementing internal controls related to the development of the new technology

Qualifications of the staff in EP technology Quality of associated testing Verification of results Documentation of results Meeting of schedules for the new technology development Tracking of associated budgets Robust cross-organizational communication

• It is equally important to institute strong controls for risks that are presently tolerable but that have a potential to become intolerable and thereby pose an emerging threat to a strategic objective

Mass growth controls Operating temperature controls


Enterprise Risk Management: Implementation Challenges and

Opportunities

Jessica Southwell & Emma Lehnhardt Strategic Investments Division

Strategic Planning and Performance Management Branch

October 30, 2014

OCFO/Strategic Investments Division – Strategy and Performance Branch For NASA Internal Use Only

Purpose

• Provide a high-level overview of our organization

• Explain NASA’s performance framework and Strategic Objective Annual Review process

• Discuss challenges and opportunities associated with ERM concept paper

• Discuss potential next steps


2

Organization: SID’s Team Structure

Strategic Investments Division (SID)

Director: Cynthia Lodge,*

Performance Improvement Officer

Strategic Planning & Performance Management (SPPM) Branch

Chief: Jessica Southwell

Emma Lehnhardt

(Strategic Plan/SOAR/AFR/ERM)

Ellen Gersten

(GAO QLB and High Risk)

David Walters

(Perf Mgmt. Systems/Perf Rept)

Chris FitzSimonds

(Performance Planning/APG/CAP)

SPPM supports the PIO and Chief Operating Officer (COO) in executing GPRA Modernization Act of 2010 and

other requirements

We are the “PIO Staff”

Program Analysis Branch

Chief: Mary Beth Zimmerman

Tracy Osborne

(Astro, Earth, Helio, Reimbursables)

Kevin Gilligan

(Planetary & Facilities)

Derek Hodgins

(HEOMD, STMD, ARMD)

Alesyn Lowry

(JWST, AMPM, R&D Tracking)

Lewis Dotson

(Cost & Schedule Reporting/Database, CoP)

Justin Oliveira (on leave)

(Integrated Analysis, SIP, Center/FTE modelling, Pro model)

Robert Giannini (ISS, Commercial, LSP)

Detailees • Sabrena Yedo (Cross-cutting,

Trending/historical analysis) • Benjamin Studenski (Project

Support)

*SID’s Director (Cynthia Lodge) is also NASA’s PIO. In her PIO capacity, she reports to NASA’s COO (Robert Lightfoot) as required by the GPRA Modernization Act of 2010. In addition, as a SID Director, she reports to Deputy CFO (Andrew Hunter) and CFO (David Radzanowski).


3

Performance Requirements – External Factors

NASA Strategic Planning,

Performance Management &

External Reporting

GPRA Modernization Act

of 2010

OMB Circular A-11

OMB Circular A-136

Congressional Authorization/Acts

& GAO Audits

Other Executive Branch Guidance & External Advisory

(NRC, NAC)

Both the Executive and Legislative branches leverage a variety of mechanisms to address perceived performance issues government wide, and to require performance management,

reporting and budget/performance integration for all CFO Act agencies. Ability to demonstrate and communicate value added to the public is a key driver.


4

Requirements Coordination

External Requirements

SPPM staff works with the mission directorates and mission support offices to collect, analyze, develop, edit, and finalize performance data and resulting products. Products are reviewed by NASA’s PIO and COO before being released to external stakeholders.

COO/PIO/CFO & Survey

Data

Determine performance

product requirements

Develop plan and schedule

Work with contacts to complete

performance information

Obtain approvals from PIO and COO

Finalize performance products and

make available to external

stakeholders

NASA Governance and Reviews (BPR, SMC,

EC)

NASA delivers electronic copies to OMB and Congress, posts copies of performance reports on www.nasa.gov, and other performance products like Agency Priority Goals on www.performance.gov. NASA delivers hard copies of reports to Congress on request.


5

http://www.nasa.gov

http://www.performance.gov

Big Mission, but Streamlined Public Reporting


Current Strategy-Performance Framework

NASA Performance

Framework

2014 Strategic Plan

Strategic Goal

Timeless (3 total)

Strategic Objective

Up to 10 years (15 total)

Performance Goal

Up to 5 years

(72 total for FY 14)

Annual Performance

Indicators

1 year – report 3rd and 4th

quarter (120 total for FY14)

Agency Priority Goal

2 years – report quarterly

Cross-Agency Priority Goal

Up to 5 years – reporting

schedule varies

Agency Priority Goals*: Target areas

where agency leaders want to achieve

near-term performance acceleration

through focused senior leadership

attention

CAP Goals*: Presidential priority areas

that require active collaboration between

multiple departments and agencies

because they address long-standing

challenges for which no one agency has

sole responsibility.

Strategic Objective Annual Review

(SOAR)*: Starting with the 2014 strategic

plans, every agency will be required to

conduct annual reviews of their strategic

objectives. These reviews will highlight

those areas where the agency has made

“noteworthy progress” or has “focus areas

for improvement”. These annual reviews

will provide input into budget formulation

and require COO/PIO to make final

categorizations. (Note: SOAR reviews

also allow us to conduct a “pulse check”

of PG/API progress in Q2.)

*Requirements mandated by the GPRA

Modernization Act of 2010 and OMB

Circular A-11


7

SOARs: Introduction and Requirements

• SOARs: Strategic Objective Annual Reviews – An annual assessment of each Strategic Objective, starting in FY14

– An analysis on the progress toward our strategic direction

– Required by Congress (GPRAMA) and implemented by OMB (A-11) for all major Federal agencies

• OMB expecting to use this information to understand how we justify our budget

• The COO, PIO, and other Agency leaders stress that this process should be useful for NASA

• Compulsory categorization of all 15 Objectives and OMB’s quota:

– Categories: Noteworthy Progress

Satisfactory Performance

Focus Area for Improvement

– 10 - 20% quota per OMB means between 1 and 3 Objectives identified as “Noteworthy,” and between 1 and 3 identified as “Focus Area(s) for Improvement”


8

OCFO/Strategic Investments Division – Strategy and Performance Branch

General Assessment Methodology Impact Implementation Risks & Challenges Opportunities

Are your strategies having the intended impact?

Are your performance goals and indicators being met?

What risks, challenges or external factors may impact success in the future?

Have new innovations emerged that may enhance future progress?

Areas for Evaluation

• Assessment of strategies and effectiveness

• Key indicators

• Performance Goals & Indicators • Other Internal Indicators (Technical

indicators, Efficiency Measures, etc.) • Cost and Schedule performance for

projects

• Technical, Cost, Schedule and Programmatic Risk

• External – Legislative, Policy, Industry, Partnerships

• Internal – Capacity and Skills Gap, Policy Analysis, Financial, Cross-Agency Dependencies

• Identification of External Best Practices & Innovations that can be leveraged

• Partnerships/collaboration • Cross-Agency dependencies

Potential Sources of Evidence (Suggested, Not Exhaustive)

• Mission Directorate Plans and Roadmaps

• Internal & External Studies • NASA Governance Reviews (PMC,

MSC) • Program Evaluations (BPR, DMPC,

Center Management Reviews, etc.) • Mission Directorate Program Review

Guidance • Research Studies

• Performance Management Systems • Program Evaluations (BPR, DMPC,

Center Management Reviews, etc.) • Evaluation Studies – IPAO • Technical Indicators – OCE • NASA Governance Reviews (PMC,

MSC) • Operating Plans • Mission Directorate Program Review

Guidance • Research Studies

• Decadal Surveys • NRC Reports • NAC Reports • Internal and External Studies • Cross-government surveys &

Studies – OSF • GAO Audits • OIG Audits • Other sources TBD by MD

• SIP Guidance • Internal & External Research Studies • Industry & Partner breakthroughs • NAC Reports • Other sources TBD by MD

Summary of Findings

Rating as “Noteworthy Progress,” “Satisfactory Performance,” or “Focus Area(s) for Improvement” Overall assessment of strategies effectiveness on outcomes and success criteria

Overall assessment of YTD performance against plan Identify changes in strategy

Identify gaps in evidence Identify key risks and mitigation approach

Proposals to address challenges (internal & external) Next steps to continue progress

For NASA official use only 9

1.1 – Assessment Ecosystem FY14 Initial Op Plan FY15 Request FY15-19 Trend % Change, FY15 to FY19

$3,268 M $2,967 M ↑ 12%

AMPM Elements

2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030

EFT-1 EM-1^ EM-2^ EM-3^ EM-4^ EM-5^ EM-6^

NOTIONAL OUT YEARS, TENTATIVE

Portfolio-wide Cost and Schedule Assessment

Cost: N/A Schedule: N/A Notes: Orion and SLS not yet confirmed.

Objective-wide past performance trending

Goal Type FY11 FY12 FY13 FY14

Multi-Year Performance Goals 3 4 4 4 -- On track

Annual Performance Indicators 2 3 3 8 -- On track

Latest BPR Ratings

Program Rating Explanation

Orion Dec 2013 March 2014 T: Yellow; C: Yellow; S: Yellow; P: Green

SLS Dec 2013 March 2014 KDP-C memo in work

EGS Dec 2013 March 2014 GSDO green across the board

AES Feb 2014 March 2014 On track to exceed plan for completed milestones

Recent GAO Reports Recent IG Reports Other Reports

• Orion & SLS included in Quick Look Book (link) o Orion: Identifies Funding and Design as project

challenges o SLS: Funding, Schedule, and Integration of Existing

Hardware as challenges • Sep 2013: Defense and Civilian Agencies Request

Significant Funding for Launch-Related Activities (link)

• Orion status, 15 Aug 2013 ( ) – “Although we believe MPCV Program officials are managing the Program as effectively as they can within a constrained budget, we are concerned about the future of the Program given the risks associated with incremental development and dependencies on the SLS and GSDO programs and the ESA for delivery of the Service Module.”

• Forthcoming: Audit of NASA's Launch Support and Infrastructure Modernization Efforts

link • NAC report forthcoming in May 2014 (link) • 2012 NRC report on Strategic Direction (link); “Lack of

national consensus” on human spaceflight goals; not compelling

• ASAP Annual Report 2013 (link): “Progress has been made on establishing risk acceptance authorities, but there is still no official policy on program- vs. element-level risk decisions. There are still no overall LOC thresholds and goals for the complete mission.” – Rated YELLOW overall

10 For NASA official use only

http://www.gao.gov/assets/670/662571.pdf

http://www.gao.gov/products/GAO-13-802R

http://oig.nasa.gov/audits/reports/FY13/IG-13-022.pdf

https://www8.nationalacademies.org/cp/projectview.aspx?key=49488

http://www.nap.edu/catalog.php?record_id=18248

http://oiir.hq.nasa.gov/asap/documents/2013_ASAP_Annual_Report.pdf

2014 SOAR Cycle Approach to Risk

• Self-assessment risks and challenges by type (grouped by PIO analyst):

2014 SOAR Cycle: Self-identified risks and challenges for Objectives Access to space

5% Commercial forces/comptetition

3%

Funding 21%

Intra-NASA culture and collaboration

5%

Mission cadence 3%

Mission management 3%

Mission support dependencies 8%

Other 20%

Partnerships 9%

Policy or Politics 12%

Schedule 3%

Systems/Data 5%

Technology development 3%

Examples in “Other” type group: • Pu-238 availability • Long mission durations • Growing airspace system

demands • “Inherent risk” • Measuring technology

transfer benefit to the nation



Performance Products Integration,

through February 2016



EROM Discussion

OCFO/Strategic Investments Division – Strategy and Performance Branch For NASA Internal Use Only 13

ERM in 2014 update of A-11

• There is no firm requirement for ERM.

• The ERM section of A-11 only states the expectation for agencies to manage risk.

• In terms of strategic reviews, ERM is suggested as a tool that can help.

• ERM expectations will be also communicated in A-123(Internal Controls)

Intent

ERM improves agency capacity

to prioritize efforts, optimize resources, and

assess of changes in the environment.

Requirements/ Suggestions

Agencies are not required to have a CRO

or enterprise risk management function

Agencies “are expected to manage

risks to mission, goals, and objectives of the

agency.”

Agencies “are encouraged to

consider instituting ERM”

Deliverables

None, beyond currently existing

requirements to discuss

risk/challenges and

opportunities in Strategic Review

deliverables.


14

EROM Challenges – Strategic Planning

• Strategic planning/development efforts include requirements outside of our control, incorporating several inputs, mandates, and perspectives that go beyond performance management information – It’s infeasible to expect that EROM outputs would be

primary driver for identifying goals and objectives

• Strategic Goals can be timeless based on Agency mission and Congressional/Executive direction, and can only be modified every 4 years – EROM feedback loops would not provide the opportunity

for amending or changing Goals and/or Objectives earlier

• No clean mapping of Strategic Goals/Objectives to Centers – Integration of Center risks posture would be difficult


15

EROM Challenges – SOAR

• Given SOAR assessment methodology leverages performance ratings as only one input for implementation, EROM could not replace current methodology

– An alternative will need to be developed to enhance current assessment of risks and opportunities

– EROM could be an input to the assessment

• SOAR assessments are done annually, so may not present the best mechanism for Continuous Risk Management at the Agency level


16

EROM Challenges – Performance Plans

• Using Performance Goals (PG) and Annual Performance Indicators (API) to determine enterprise risk posture and opportunities may be limiting: – API and PG map to programs and budget but are not complete representation

of activities

– Centers and/or MDs often generate unique implementation plans and roadmaps (Tier 3)

– Performance Plans are mostly generated by HQ

• EROM approach does not distinguish project-specific risk with organizational risk, as there is not a clear method for incorporating institutional risk even though the Center risk managers are envisioned to provide the scenarios and information – Collecting risk information for approximately 200 measures will represent

significant workload for our stakeholders and will require coordination between HQ performance community and Center risk managers

– PGs and APIs are “living measures,” sometimes modified based on budget and other factors - they are not lagging indicators o Changes can take place yearly and require OMB approval


17

EROM Challenges – Performance Plans (cont.)

• Given their diversity, measures could have limitless list of potential risks and scenario statements, thus becoming a time-intensive compulsory exercise – EROM taxonomy is very detailed, addressing most/all

of risks defined will represent significant burden

• Scores could become arbitrary, and/or subject to personal biases to achieve desired ratings

• Overall ratings for indicators does not spell out how opportunities and risks are being calculated, results may not be useful to decision makers


18

EROM Challenges – GAO and Governance

• EROM can result in additional requirements for projects in terms of how they are being asked to track risks and opportunities.

– For example, if adopted, would projects be expected to track their risks differently than they have done in the past?

– Given GAO's interest in project risk lists, EROM can result in Congressional (GAO) oversight.

– Some of the examples cited for EROM are already considered as part of projects' independent cost analyses, such as the time to develop new technology. What is the proposed linkage between these different efforts?

– EROM does not account for APMC/DPMC role in approving projects or issuing direction to projects if there are issues

• EROM approach for selection processes conflict with the GAO Analysis of Alternatives (AoA) best practice guide that is in work

– In an audit setting, NASA would have to demonstrate that the AoA best practice was met

– EROM paper does not focus on mitigation strategies, which is a key component of the AoA


19

EROM Challenges – GAO and Governance (cont.)

• Approach does not leverage current risk inputs provided by Centers as part of the SoA process

• Approach does not address the need to have a standardized risk lexicon/identification risk matrix from Centers

• Not clear what EROM effect would be on TCAT and other efforts


20

EROM Opportunities

• EROM can represent a first step towards an agency view of risk

• EROM can provide one perspective to enhance future strategic plan development efforts

• EROM can be a tool for the SOAR assessments, and factor into objective ratings

• EROM can help bring risk and performance communities together

• EROM can help meet OMB’s requirements from a A-11 and A-123 (statement of assurance process) perspective


21

Potential Next Steps

• Invite Center risk managers to Performance Community of Practice monthly meetings – Transparency and collaboration among communities will be key and represent

a first step towards change management

• Incorporate risk community in 2015 SOAR cross-cutting assessment • OICMS could consider approaches for integrating Center’s risk information

(provided as part of the Statement of Assurance process) to provide an agency-wide perspective of risk

• Conduct focus interviews with Agency leadership to understand their specific vision and needs as it relates to BPR and other governance councils

• PIO and staff will be developing an incremental approach to assess risk as part of the strategic objectives (perhaps as in interrelated portfolio) for the 2015 SOAR. We will work with the strategic objective leaders and risk community over the next few months

• Continue integrating SOAR risks and opportunities in our performance management systems to capture and report data


22

BACKUP

OCFO/Strategic Investments Division – Strategy and Performance Branch For NASA Internal Use Only 23

Organization: SID’s Location within OCFO

Sue Romans

[Vacant]

Frank Peterson

Associate Administrator & COO

Robert Lightfoot CFO

David Radzanowski

Deputy CFO, Budget & Performance

Andrew Hunter

Budget Division Director

Strategic Investment Division Director & PIO

Cynthia Lodge

Program Analysis Branch Chief

Mary Beth Zimmerman

Performance Branch Chief

Jessica Southwell

Deputy CFO Finance

Systems Division Director

Beverly Veit

Financial Mgmt. Division Director

Terence Alfred

Quality Assurance Division Director

Policy Division Director

Kevin Buford


24

Strategic Planning & Performance Management Branch Team

Activity Primary POC Alternate POC GAO Quick Look Ellen Gertsen Emma Lehnhardt

GAO High Risk Ellen Gertsen Emma Lehnhardt

EVM Ellen Gertsen Emma Lehnhardt

Evidence and Evaluation Ellen Gertsen Emma Lehnhardt

Strategic Reviews Emma Lehnhardt Ellen Gertsen

Enterprise Risk Management Emma Lehnhardt Ellen Gertsen

Agency Financial Report & Summary of Performance and Financial Information

Emma Lehnhardt David Walters

Strategic Projects (Strategic Plan,

special analysis) Emma Lehnhardt Ellen Gertsen

Lab-to-Market CAP Goal Emma Lehnhardt Chris FitzSimonds

Agency Priority Goals & Cross Agency

Priority Goals Reporting Chris FitzSimonds David Walters

Annual Performance Plan/Annual

Performance Report Chris FitzSimonds David Walters

Benchmarking Chris FitzSimonds Emma Lehnhardt

GPRAMA Audits Chris FitzSimonds Ellen Gertsen

WBS Charts Chris FitzSimonds David Walters

Newsletter David Walters Chris FitzSimonds

Community of Practice David Walters Chris FitzSimonds

Management & Performance Report David Walters Chris FitzSimonds

Performance Management Systems David Walters Chris FitzSimonds

Communications Plan David Walters Chris FitzSimonds


25

SID’s Performance Management & Reporting Roles

within OCFO Ex

tern

al P

erfo

rman

ce R

epo

rtin

g P

rod

uct

s

Strategic Investments Division (SID)

•Strategic Plan (development and production)*

•Strategic Objective Annual Reviews (SOARs)*

•Annual Performance Plan and Annual Performance Report (concurrent with CJ)*

•Agency Priority Goals (APG) Quarterly Reporting*

•Cross-Agency Priority (CAP) Goals*

•Bi-annual assessment of performance measures*

•Agency Financial Report (AFR)’s Performance Highlights and Ratings Summaries*

•GAO Quick Look and High Risk reports*

•Baseline, Major Program Annual Reports (MPAR), and other cost reports**

•NSPD-49 Implementation Action: Space Acquisition Program/Project Cost and Schedule Growth**

•Issue Papers**

•Project Technical Reviews**

Financial Management Division

•Agency Financial Report (Document Management, Financial Statements)

Quality Assurance Division

•Agency Financial Report (Audit)

•Congressional Justification (CJ)

Budget Division

*Managed by the Strategic Planning and Performance Management Branch within SID. This branch also serves as the PIO staff. **Managed by the Program Analysis Branch within SID; not addressed in this presentation.


26

2014 NASA Strategic Plan:

Goals and Objectives


27


SOARs Process and Assessment Overview

• Process:Self-

Assessments

•Due end of March 2015

•Analyze progress using guiding questions from PIO

•Provides self-rating

•Will generate a substantial amount of materials and supporting evidence that remain internal

Cross-Cutting Assessment (PIO Review)

•Identifies major cross-Agency themes, issues, and actions

•Analyzes each Strategic Objective (with additional evidence/sources)

•Provides independent rating recommendation

COO Briefing & Decision

•Presented summary of information from self-assessment and cross-cutting assessment

•Decides on final categories

•Final decision validated by the SMC

•Materials used to generate Summary of Findings for OMB

Reporting & Budget

Integration

•Summary of Findings (May)

•Summary of Progress draft (Sept)

•PPBE decision making integration

•SOAR elements of the APP/APR (published concurrent with CJ, in February 2016)

• Assessment methodology: Assessment Components

Impact Implementation Risks & Challenges Opportunities

Long-term outlook… Are your strategies

having the intended impact?

Near-term performance… Are your

performance goals and indicators being met?

What risks, challenges or external factors

may impact success?

Have new innovations emerged that may

enhance future progress?

Diverse sources of evidence underpin all elements of the assessment 28

SOAR Self-Assessment Summary: 1.1 (Exploration)

Objective Statement Strategic Objective 1.1: Expand human presence into the solar system and to the surface of Mars to advance exploration, science, innovation, benefits to humanity, and international collaboration.

MD/Office HEOMD

Contributing Programs Orion Multi-Purpose Crew Vehicle Program, Space Launch System Program, Exploration Ground Systems Program, and Advanced Exploration Systems

Leader(s) Greg Williams

Assessment Components

Impact Implementation Risks & Challenges Opportunities

Long-term outlook… Are your strategies having the intended impact?

Near-term performance… Are your performance goals and indicators being met?

What risks, challenges or external factors may impact success?

Have new innovations emerged that may enhance future progress?

Self-Assessment Results

• Will implement an iterative, strategic approach to prioritize activities that support human exploration beyond LEO

• Will develop a new human deep-space exploration architecture with SLS, Orion, and other high-priority capabilities needed for human exploration/pioneering

• Strategies are demonstrating success in achieving the objective.

• Progressing well towards completion of near-term Program milestones with no schedule slips

• EFT-1 is on target for launch.

• Satisfactory Performance has been demonstrated in all assessment areas. Implementation of Performance Goals and APIs are 100% on track for FY 14 and FY 15

• Developing the technologies and capabilities for humans to safely explore and pioneer our solar system.

• Maintaining key schedules for hardware and software in development while funding ongoing and new research activities necessary to design future human exploration systems in an environment of continuing national budget pressures.

• Selecting an asteroid • Long mission durations • Radiation effects on Human

explorers/pioneers

• International collaboration • Industry led partnerships and/or Interagency

partnership opportunities

Self-Assessment Category

Satisfactory Performance

Next Steps No adjustments to the strategy, budgets, performance measures, or

organization are anticipated at this time.



Strategic Planning, Performance Management & Reporting

Requirements

• NASA’s strategic planning and performance management/reporting requirements are communicated through several sources: – GPRA Modernization Act of 2010 (GPRAMA)

– OMB Circular A-11, Part 6

– OMB Circular A-136

– Annual Budget Guidance

– Various OMB Memoranda

– NASA Leadership (COO, PIO)

– Congressional Feedback

– Internal/External Stakeholder Feedback

– GAO Audits


30

Products Span Multiple Fiscal Years

Cross-Agency Priority Goals

Reported on quarterly

Covers length of performance as defined in each CAP Goal

FY 2016 Performance Plan

Submitted with the FY 2016 Congressional Justification

Complements the FY16 budget request

FY15-FY16 Agency Priority Goals

Reported on quarterly

Covers 2 years of performance

FY 2015 Performance Plan Update

Submitted with the FY 2016 Congressional Justification

Provides revision for current (execution) fiscal year performance measures

FY 2014 Annual Performance Report

NASA’s performance for the fiscal year

Based on measures included in the FY 2014 Performance Plan as well as results from the Strategic Reviews process

GAO Quick Look Audit

Reports on NASA’s large-scale projects for FY 2014

20

14

Throughout the fiscal year, the PIO staff is working on products spanning different fiscal years. This includes the FY 2016 Performance Plan for the forthcoming budget year, the FY15 Performance Plan Update for the execution year, and the FY14 Annual Performance Report. These tasks have recently been augmented with the Strategic Reviews


31

New Era of Performance Management:

PMM & PMMe

• These systems were incrementally developed since 2011 to automate performance management, produce reports and enable analyses. To date, these systems manage hundreds of performance goals and indicators from FY 2006 to FY 2016

• NASA’s “performance warehouse”

• This is where our performance partners enter their ratings

• Limited reporting capability

• Inter-agency agreement

• Generates XML exports for upload to performance.gov

• Additional features & reports “glossies”

• Unique to NASA

• Generates APP Review Report

• Generates combined APP/APR MACRO report

• These systems can support RMO efforts in developing CJ narratives (in particular, Achievements in FY 2014 and Key Achievements Planned for FY 2016 sections) by providing a holistic view of program performance across all documents

PMM

PMMe

(NEACC)


32

Performance Management Systems History

• Late 2011 ~ Initial planning and requirements development • A hybrid-systems approach was selected that utilized an inter-agency

Performance Measure Management (PMM) System while leveraging NASA’s internal IBM Cognos Business Warehouse

• Work was initiated in 2012 on PMM system configuration and the Cognos PMM Extension (PMMe).

• PMM was first utilized to collect FY12 Q3 data • System functionality upgrades continual under annual development

phases; currently in PMMe Phase 4 • Series of internal and external mandates drive the direction of future

development • Next frontier includes making performance information available to

enhance knowledge sharing across the agency, promote analysis, and enhance decision making – Developing useful, intuitive reports is a key component


33

Overview of the Performance Management Systems

• PMM – Developed by Department of

Treasury – Performance data warehouse – Contains strategic framework

performance measure information

– Web Portal for POCs to input performance data

– Machine readable reporting

• PMMe (BI/Cognos) – Developed by the NEACC – Performance dashboard – Performance plan report

generation – Generates APP Review Copy,

Measure Data Sheets, M&P Content, etc.


34

Measure Data Sheet Development

• Development began as a way to give NASA’s performance community greater access to measure data and to support annual performance planning activities

• The development process focused on the availability of measure data across multiple reporting products in multiple formats

• Updates to the data models were made to provide additional context and support for increasingly robust data element information

• Work initiated in June 2014, and was delivered on schedule in July 2014

• Future enhancements planned to include strategic objective annual review, budget, high-risk, and other programmatic information


35

Performance Measure Data Sheets

• Allows users easy self-serve access to all performance measure data and Mission Directorate program and project activities

• New reporting capability will facilitate budget development & execution, performance planning, performance reporting, program analysis, and auditing

• Measure Data Reports can facilitate analysis by providing insight into performance measures for the Agency across multiple years

36


Measure Data Sheet – PG View

37

Measure Data Sheet – PG View (cont.)

38

Measure Data Sheet – API View

39

Enterprise Risk and Opportunity Mangemetn (EROM) Workshop › docs › default-source › sma... ·...

Documents

Transcript of Enterprise Risk and Opportunity Mangemetn (EROM) Workshop › docs › default-source › sma... ·...