Risk Management Literature Survey

Risk Management Literature Survey

An overview of the process, tools used and their outcomes

Report to: Dutch Space Stork Product Engineering TNO/TPD Verhaert

Authors

Maarten G.H. Bijl Robbert J. Hamann

Delft University of Technology Aerospace Engineering

August 2002

Risk Management Literature Survey Delft University of Technology

Distribution Name Company P.R. van Nugteren Dutch Space W. Pasteuning Stork Product Engineering W. Gielesen TNO/TPD D. Vrancken Verhaert


- 3 -

Table of Contents Introduction ......................................................................................................................... 4 What is Risk? ...................................................................................................................... 5 Risk Management process................................................................................................... 7

Step 1: Risk Planning ...................................................................................................... 9

Step 2: Identification of risk events and risk items ......................................................... 9

Step 3: Risk Assessment ............................................................................................... 11

Step 4: Risk mitigation and tracking ............................................................................. 14

The place of risk management in the project life cycle............................................. 15

Problems with the implementation of risk management........................................... 17 Risk Management Tools.................................................................................................... 18 Remarks for future risk managers and tool developers..................................................... 23 Reference List ................................................................................................................... 25 Appendix A ....................................................................................................................... 27


- 4 -

Introduction This report describes the outcome of a literature review performed as part of a master thesis study at the Technical University of Delft, department of Aerospace Engineering within the research group of System Integration / Design and Analysis of Space Systems. The title of the thesis study is “Technical risk assessment techniques for small to medium sized space companies”. The objective of the thesis study is to define a technical risk assessment method that can be implemented in these companies. Before the definition of such a method is done a literature survey is performed. The scope of the survey is to find out what (technical) risk really is, what kind of assessment tools and methods there are available, and what results they produce. Since risk assessments are part of the risk management process and some tools aid several steps of the process, I have taken the liberty to broaden the scope of the survey to the risk management process. Extra attention is given to the technical aspects of this process. The report is structured as follows: first definitions of risk and risk management are given and discussed. After the definitions, an overview of the total risk management process is given, this is done in a step by step manner, from planning up to mitigation and tracking. The place of risk management in the project life cycle and problems with the implementation are briefly discussed to give a more complete view on risk management. Next risk management tools that are used in the process are described. In total three different categories of tools are dealt with, namely: risk visualising tools, risk tracking tools, and integrated risk assessment tools. Finally some extra remarks are made as a lesson for future implementers of the risk management process and developers of tools. Since this literature study will be used as a starting-point for further research, all articles are classified to their contents. This classification is done with respect to eight criteria to select the articles that will be useful for further research and to give the reader of this literature study an overview on the main articles that deal with technical risk assessment.


- 5 -

What is Risk? Risk is part of everybody’s daily life, its part of the decisions we make. The word risk is commonly used to express a certain probability that an undesired event will happen. How big is that probability? What are the consequences and what degree of influence do they have on their environment? These matters are very subjective to each individual, e.g. a race driver has another perception of risk in normal traffic than an old lady. This is not different in technical projects; engineers classify risk dependent on their background, experience and their role in the project. Difference in perception on this issue can lead to different identifications of risks and therefore different outcomes of a risk assessment, and can be a threat to the technical performance of a project. Paradoxically it seems that the assessment of risks can be considered as an risk of its own when the perception of risk differs among the assessors or the assessed. Fortunately this risk can be mitigated by choosing a clear definition of risk. Some definitions that can be found in literature are stated below

− Literal definition of Risk and Safety Risk [dictionary]: o The chance of injury, damage or loss; dangerous chance; hazard o Insurance: the chance of loss; the degree of probability of loss; the amount

of possible loss to the company

− Risk – Any occurrence likely to adversely affect the attainment of project objectives56.

− Risk is a probability, a mathematical quantity that can be measured, calculated or estimated 56.

− Risk – The probability of an undesired outcome56.

− Risk – Probability of failure times the severity of the consequences56

− Risk is either a condition of, or a multi-dimensional measure of, exposure to unpredictable loss or losses 53.

− Risk is an undesirable situation or circumstance that has both a likelihood of occurring and a potential negative consequence on the project 45.

− Technical Risk denotes the risk that a project will fail to meet its performance criteria26.

− Performance Risk, or technical risk, is uncertainty that a product design will satisfy technical requirements and the consequences thereof 7.

It’s not only the subjective character of risk that makes it hard to classify but also its diversity. When someone studies risk, it should also be clear what type of risk is mainly looked into. Technical, schedule, cost and management risk are a few examples of what risk can describe in technical projects. In other projects you can also find political, environmental or financial risks.


- 6 -

The difficulty is that the different types of risks are strongly connected, the chance that e.g. a sensor fails to meet its requirement during testing, is a technical probability but one of the consequences can be a schedule overrun, so is it therefore also a schedule risk? This is one of the questions someone who is occupied with risk should consider and should be clear of in his definitions of risk. In this paper the focus will mainly be on the risk that influences the technical performance of a project. Another aspect of risk is that it can describe more dimensional phenomena. An example of the two dimensional view on risk is the belief that risk equals probability times severity, this view is a technical view on risk. Another way to view the multidimensionality is the view of Tedd Yellmann53. He says that risk contains of three facets namely: Expected Loss, Variability of Loss Values and Uncertainty about the Mental Models used. The difference between the two views on risk is mainly that the consequences of a risk are considered to be variable with a certain maximum in case of the multidimensional view. This maximum value, or worst possible consequence, is the starting-point of the two dimensional view. Of course it is dependent on the way in which risk is used, if risk should be considered as a two or as a multidimensional entity. One thing that seems to be normal is that risk is always considered to be something negative. Roger van Scoy disputes this and states “Risk in itself is not bad; risk is essential to progress, and failure is often a key part of learning. But we must learn to balance the possible negative consequences of risk against the potential benefits of its associated opportunity” 56. What can be deduced from this quote is that there is a certain level of risk, where risk becomes acceptable. The ECSS standard states that risk is acceptable when: Its magnitude is less then a given threshold, defined in the risk management policy45. The balance between opportunities and risks has got to be carefully maintained by project member(s) responsible for the performance of the project. It depends on project structure if this is done by a project manager or for instance by the whole team of engineers. The process of handling the risks in a project is called the Risk Management Process. Some definitions are:

− Risk Management – An organised process to identify what can go wrong, to quantify and assess associated risks, and to implement/control the appropriate approach for preventing or handling each risk identified 56.

− Risk Management – The systematic and iterative optimisation of the project resources, performed according to the established project risk management policy, which describes the organisation’s attitude towards risks45.

− The Risk Management Process consists of all the project activities related to the identification, assessment, reduction and acceptance of risks45.

− Risk Management is the identification, analysis, prioritisation, mitigation planning, and monitoring and control “of events which have a potential of causing unwanted change”51.

Part of this process is the assessment of the risks involved. How the risk management process is structured and how assessments are done will be described in the following chapters as well as the tools used to assess them and their outcomes.


- 7 -

Risk Management process This chapter describes the process of risk management and how it is structured. Some techniques on how to perform certain steps; the place of risk management in the project life cycle, and what actions should be taken are discussed. Tools that aid the risk management process are dealt with in the next chapter. Large, complex and time consuming projects involve considerable risks. By identifying these risks before they actually take place, they can be controlled and managed by taking preventive actions. The process that identifies, assesses, mitigates and controls the risk is the risk management process. Looking back at the definitions given in the previous chapter we can conclude that Risk Management is a systematic and iterative process to optimise the performance of the project within the given requirements. The iterative part of the risk management process should be emphasised at all times. As with any type of analysis, it is unlikely that decision makers, analysts, and other participants do everything right the first time. Therefore, it is necessary to repeat periodically (parts of) the risk management process. In this manner new critical risks can be identified and rearranged. To a large extent, this may be due to new information acquired as the program progresses. In other words, a specific risk that originally was deemed non critical might turn out to be a major bottleneck in the project development and may require proper attention. “Risk management is about value adding to the product development 7”, is an often heard statement, but what can be the benefits to a project in case of conducting the risk management process in the right way? Some benefits are 25:

- Better and more detailed perception of the content of the risks, what their consequences are , and how they interact.

- Better planning, taking into account the imposed risks; better responding on risks occurring, and a better approach in minimising the effects of a risk.

- Feedback during the design and planning process about the level of risk of risk events, this can be used to avoid risks.

- As a consequence of the previous aspects: a general reduction of the total project risk.

- A possibility to check the sensitivity for changes in the decisions made.

- Documentation and integration of knowledge, which normally is only known by individual experts.

- Insight and knowledge to take better decisions.


- 8 -

The risk management process generally consists of four main steps, which can be repeated until the risks are under an acceptable level. These steps are

1. Planning

2. Identification of risk events

3. Risk assessment

4. Risk mitigation and tracking Below the iterative process through the project life cycle is visualised, the purpose of the picture is creating awareness that risk management is a process that continues through the whole life cycle.

Figure 1: The iterative risk management process modified ref. 45

Of course this classification is a generalisation and one will find variations on this classification. It is dependent on the characteristics of the project and the risk management plan if other steps are introduced or seperated.


- 9 -

Step 1: Risk Planning Before risks can be identified, assessed, and mitigated, a decent strategy or plan must be developed. This plan involves defining detailed activities to be accomplished by specific individuals within an established schedule. Risk planning is the detailed formulation of a program of actions for the management of risk. It is the process to: (1) develop and document an organised, comprehensive, and interactive risk management strategy; (2) determine the methods to be used to execute the risk management strategy and (3) plan for adequate resources. Risk planning is iterative and includes describing and scheduling the activities and process to assess, handle, monitor and document the risk associated with a program. This result is the Risk Management Plan (RMP) 37. To construct the Risk Management Plan, data must be gathered from the work breakdown structure, the system requirements and performance standards, configuration data, schedule, program, and project management plans 18,39. The reason for this is twofold: first, to understand the project in enough detail to perform the technical risk and most-likely cost and schedule estimate. Secondly, to evaluate the projects maturity, in order to assess management risk 18. As the project matures through time, the information at hand becomes more detailed. With this increment of detail new versions, or updates, of the RMP are made. Although management is responsible for high level program planning, everyone plans with respect to accomplishing their assigned task. Making a risk plan includes developing a risk-management policy and procedures. The risk management plan is documented to respond to the goals established for the risk-management initiative. Technical staff and integrated product teams develop more detailed action plans to resolve technical risk. Action plans are needed to delegate responsibility and authority for managing risks to the lowest possible level within the organisation 2. The responsible project members should periodically review the plan and revise it, if necessary. The Risk Management Plan can be changed at events such as: (1) a change in (acquisition) strategy; (2) preparation for a major decision point; (3) technical audits and reviews; (4) an update of program plans, and (5) preparation of a program objective memorandum, which is a document that expounds the scope of a project37.

Step 2: Identification of risk events and risk items Identification of risk events is the act of determining and describing events that could impact the program. These events can be coupled to (sub)-systems or components within the program, these are called risk items or risk elements. Below an overview is given from ways in which some projects identified their risks.

− Risk identification techniques to baseline existing risks include the use of risk taxonomy (ranking), interviews with experts in the field, brainstorming, and analysis of similar programs. Special techniques must be established to introduce new risks that are identified after the baseline has been established; this includes setting up email addresses or risk folders where potential new risks are identified 51.


- 10 -

− A way to identify risk events is by organising a workshop for the stakeholders. Possible risk events are presented and an open discussion follows. After this discussion a list with significant threats to operability can be compiled 47.

− The identification of risks was done by review, analysis, and constant interrogation to discover risk items. This was in the beginning done by facilitated brainstorm sessions, but in the latter phase by the project risk manager 35.

− The methodology of identifying the risk items was that of interviewing the project’s key staff. This approach was regarded as the most suitable, despite being more time consuming than structured brainstorm techniques (e.g. the Meta plan technique). It also circumvents negative group-dynamic situations in which project team members tend to be less vocal than on a one-to-one basis. The initial interviews were conducted by non project team members who were sufficiently familiar with the project and the environment. When the interviewer is not a team member there is always a chance that the team members might be reluctant to expose any shortcomings. The question of using an external person should therefore be treated very carefully, with due respect for the project’s feelings and wishes. One possibility is to have the outsider conduct only the initial risk identification exercise, leaving the recurring task to someone within the project itself. To help identifying the risks a checklist can be composed, for use as an interviewing guide 38.

− Historically, US DoD (Department of Defence) programs have defined and categorised the sources of risk into seven areas: threat, requirements, technology, design engineering, support, manufacturing, cost and schedule. Risks were initially identified on a top down basis from a system level perspective during the concept development phase of the program. In subsequent efforts, the technical risks were identified on a bottom-up approach based on the work brake-down structure. The list produced is the foundation of all subsequent technical assessment and mitigation efforts 49.

What draws the attention is that all projects use an interview based, or brainstorm based, approach to identifying risks. It is however thinkable that more systematic techniques as Failure Modes and Effects Analysis (FMEA) and fault trees can be used. Each part, or subsystem, can be examined individually to determine its failure modes. What can be observed is that these techniques are used, but in a different way then to just identify risk events, they’re used to aid the probabilistic assessment process (step 3). The reason for this is probably the fact that these techniques display the interdependencies between the different elements of a system, and overall probabilities can be computed more easily. After the risk events are identified, a risk watch list should be initiated. This list must be further refined during the later phases of the project 37.


- 11 -

Step 3: Risk Assessment After the identification of the risk events, it has to be determined in what way they form a threat to the success of the project. The level of risk for each event has to be defined to rank the risks and to filter out the risk drivers for the project. In this way a clear overview of the stakeholder risks can be given. Risk assessment techniques are used to perform this classification. In case of single point failures, this may require testing a sample of the component to determine a probability distribution 26. If the problem is more complex and testing can’t provide a solution, the use of extensive risk assessment techniques is the way to proceed, but it should be noted that general practice is not to accept single point failures. There are two main types of assessment techniques, namely quantitative and qualitative techniques. In case of qualitative techniques the resulting numbers are only used for distinction or rank ordering6. Another description states: qualitative methods produce descriptive estimates for risks, for example “very high” risk 15. Quantitative methods produce numeric exposure estimates for risk 15. With quantitative techniques the interval between the resulting numbers and the ratio of the resulting numbers is used to interpreted or rank the different risk items6. As stated earlier the focus of this paper is on the technical aspect of risk, therefore technical risk assessment will be discussed here. The defining relationship, or definition of risk, for any method of technical risk assessment is Risk = Uncertainty x Consequence of failure. Usually this relationship is applied in all areas of technical risk in a project, such as hardware, software, and integration. During the identification step, risk events are identified. In assessments however, subsystems or parts are assessed and not single events. So first the assessor has to connect the events, when the events weren’t defined on a system level, with the appropriate (sub)systems and make a list of the (sub)systems he wants to assess. After the (sub)-system list is constructed, for every risk event the probability of failure has to be determined. This can be done in several ways. Quantitative techniques One method is the statistical determination of the probability of that item, known as the traditional probabilistic assessment method. This method calculates the system failure probability based on initiating event frequencies, test and maintenance data on component failures in case of hardware items, and software test data and bug reports of comparable software in case of software items . Fault tree modelling is then usually applied for calculating top event frequencies consisting of subsystem failures and event tree modelling is applied to calculate both frequencies and the various scenarios and the possible consequences 4. The Accident Sequence Precursor method (ASP)4 is used to improve traditional Probabilistic Risk Analysis (PRA) calculations. ASP uses failure probabilities derived from operational experience of existing systems to calculate generic accident probabilities for a homogeneous population of systems per operating year. Since subsystem failure probabilities and interdependencies are of most interest, the system has to be modelled such that the data collection from existing systems is capable of deriving those failure probabilities 4.


- 12 -

Monte Carlo Simulation is not actually a risk assessment technique, but a uncertainty analysis method, often applied to find probability distributions of estimates. In the case of complex subsystems, Monte Carlo Simulation can be used to evaluate a model that represents the subsystem. This technique is also used to visualise the spread of risks, because it is unlikely that all risks, or none, will occur 25. Qualitative techniques Given that most projects are unique, and that the systems in it are composed of hardware and software, it is unusual to find relevant statistical data with which probabilities or probability distributions can be quantified 21. Most assessments are therefore based on expert judgement. In the case of an expert based assessment the probability of failure will be determined by estimating this probability in a number of risk categories. An example set of hardware risk categories is 42: • State of technology • Personnel resource status • Technical difficulty • Material resource status • Manufacturing process difficulty • Test resource status • Production equipment status Categories that are thinkable in the case of a software assessment 42 : • State of technology • S/W personnel resource status • S/W implementation difficulty • S/W engineering tools status • Requirements stability status • Test resource status • S/W engineering environment status • S/W engineering process Integration risk is a function of technical difficulty and maturity of design, or State of Technology 49. The stated categories are just an indication of what kinds of categories are possible; it is totally dependent on the characteristics of the project which categories influence the project risk. For each (sub) system the assessor has to define the rating per category. This rating corresponds with a certain probability of failure for that item, as can be seen in table 1. The total probability is determined by multiplying and summing the probabilities of the different categories with respect to their interdependencies. An example of a rating table which can be used for a category is given below50,52:

State of technology Rating Probability Qualified item which meets all requirements 0 0.01 Existing item which requires acceptance 1 0.1 Existing item requiring minor modification 2 0.2 Existing item requiring major modification, technology demonstrated

3 0.4

New design, can be done with existing parts or software modules

4 0.5

New design, requires new h/w, s/w 5 0.7 New design requiring state of the art advance 6 0.9

Table 1: State of technology risk category


- 13 -

After each (sub) system is classified in ways of probability of failure, it has to be defined in ways of consequence of failure. The consequence of failure coefficients (Cf) can be determined in the same way as the probabilities of failure. It should be noted that in case of a total project assessment, so also including schedule and cost risks, the total consequence of failure coefficient is the weighted value of the three. Since the focus lies on technical risk assessment the table gives a possible division between technical consequences 50,52.

Impact Cf No effect on item or system performance, including support systems 0 Negligible: failure to meet requirements would create inconvenience or non operational impact. Reduction in technical performance

0.1

Marginal: failure to meet requirements would result in degradation of the secondary mission. Small reduction in technical performance

0.5

Critical: failure to meet requirements would degrade the system to a point where mission success is questionable. Some reduction in technical performance

0.7

Catastrophic: failure to meet requirements would result in a mission failure. Significant degradation/ non achievement of technical performance

0.9

Table 2: Impact factors

After both the probability and the consequence of failure coefficients are determined, the risk can be calculated. The expert based assessments visualise the risk in graphs and divide the risks into three groups; low, moderate, and high risks. Therefore this is a qualitative technique. The methods of visualisation are described in the chapter about tools. The results acquired with the expert assessment are of course subjective of nature, it is up to the assessor’s previous experiences how he ranks risk elements or events. Sensitivity analysis can be done to check the influence of selecting a different weighting for the risk categories. The sensitivity of the total risk to a change in probability or consequence of failure coefficient is checked and can also be plotted in graphs. FMECA: in general terms a Failure Modes, Effect and Criticality Analysis is an ongoing procedure by which potential system failures are analysed to determine the effect on the system, and to classify each potential failure mode according to its severity47. This method is an extension on FMEA. The identified risk events are translated to modes with two states per mode and an effect. Using a system tree the total risk of the system is determined. Although FMECA may result in quantitative figures, for example risk priority numbers, this technique is also considered to be qualitative in nature 6. Similarity analysis can be used during the concept phase of the design. It includes performing affinity diagramming of identified risks with risk items under investigation. This is done to assign a probability of occurrence and impact rating to each risk 23.51. The simple risk map technique is a simplification of the expert based method. The difference lies in the fact that here only State of Technology is considered as input for


- 14 -

probability of failure, hence no numerical processing is required to obtain overall probability of occurrence of a risk event. The impact of failure is equal to the expert method. The risk elements or (sub)-systems are plotted in a risk map, and the inputs are given by the technical team of a project 14.

Step 4: Risk mitigation and tracking The technical risk assessment process places each of the risk events in one of the three conditions (high, medium or low). The boundaries of these categories are determined by the requirements of the project or stated in the Risk Management Plan. The appropriate risk mitigation action varies with the level of risk. This involves selecting one out of four strategies. These strategies are (1) reduction (reducing the risk’s impact or probability, shifting the risk’s timeframe, or changing the risk’s consequence); (2) investigation (conducting research until a decision on a mitigation approach can be made); (3) acceptance of the risk by doing nothing (with the anticipation that the risk may have to be managed in the future); (4) watching the risk by tracking the risk and its attributes for early warnings signs of critical changes10. Risk items rated high are given special treatment. They are usually assigned to a group of engineers within the responsible team, or if severity warrants, a so called tiger team of subject experts. In either case, the assigned engineers develop alternative mitigation approaches and then prepare a Risk Mitigation Plan. This plan is a comprehensive and detailed report which clearly indicates the actions that will be taken to mitigate the risk49. Mitigating the risk is reducing the likelihood of occurrence or severity of its consequences, or both. There is usually a trade-off to be made between the magnitude of the risk and the resources needed to reduce that risk to an acceptable level 38. In the beginning there is usually a focus on reducing the likelihood, but as the project matures the design becomes less flexible and therefore it will be more difficult to change the characteristics of a (sub)-system. Items rated medium, or moderate, also require a risk mitigation plan. However, the level of effort required to prepare this will generally be less and the plan itself will comprise only a few pages. Low risks are tracked, but no plans are made. An important aspect of these plans is that they must show a prediction of expected results, including when any alternate approach should be adopted to ensure the projected technical risk reduction49. Monitoring and control, or tracking, involves watching the risk elements identified and the mitigation results both internal and external to a project. Monitoring is the early warning system that triggers a revised mitigation plan, initiates a work around, or possibly identifies a new risk10. The basis of monitoring is the evaluation of the progress, and effectiveness, of risk mitigation actions by comparison of the predicted results with the actual performance. Control is managing the risk mitigation plans and being prepared to react to changes in the performance of risk mitigation activities 10. To maximise visibility and early problem identification, it is useful, as stated above, to track the execution of all risk mitigation plans Since the actual risk assessment for a given item may remain at a given level for an extended period of time, it is not practical to use risk alone as a measure of progress. As a result, a system of risk reduction metrics should be applied to each plan. These tailored parameters are developed in the product


- 15 -

development teams by reviewing the risk assessment for each item needing a plan and devising metrics which characterise one or more of the risk sources identified 49.

The place of risk management in the project life cycle Most of the focus of risk management activities must lie in the early phases of a program’s life cycle, because effective treatment of risk early in a program has the highest payoff. It isn’t so that the later phases are not addressed anymore. Some arguments for risk management activities in these phases are : first, there are still issues which can be costly, and second, there is an important opportunity for lessons learned from the actual outcomes of decisions based on uncertain information available in the early phases that can provide a basis for improved risk management procedures in future. When we look at the activities through time we see the following. In the early phases of a program there is often very little information about the system being built or the processes used to design, develop, manufacture, operate, and dispose (if applicable) of the system. Effective risk management in that case must identify and bound the sources of uncertainty in the analyses used to justify the program. As the program moves into later development phases, attention must focus on timely and accurate reporting of information related to risks to ensure that program management can limit the impacts of the problems that actually develop. In the operations and support phase, there may be additional data collection requirements to confirm the impacts generated from risk tracked in the development phases. For a long term program, the risk management procedures applied in the development phases might be applied during the operations and support phase for the development of product improvement options, for use in other projects.

Figure 2: The effect of information on risk through time 19

In short per phase 19: Pre concept phase: assessing that critical items are within performance is the main action. In this phase, engineering studies and analyses on the effectiveness of various candidate systems are performed. The engineering team and program management will identify critical item product and process technologies. A review of the initial technology strategy is performed to establish the potential performance variations for each technology.


- 16 -

Concept phase: actions: develop requirements; identify and assess risks; define demonstration/validation risk reduction. The most promising concepts are developed in more detail in this phase. Program management and engineers are now able to be more specific in describing significant risk areas of the program, including critical items, new processes, and new technologies. Demonstration/validation phase: actions: test of critical risk areas; requirements/ risk trades. During this phase component, subsystem, and system risk are assessed through breadboard or brassboard testing, prototype testing, and for those items that can not be individually tested, analysis and simulation. The engineering data becomes the risk data, since the purpose of the activity is to select the design approach most likely to satisfy the objectives of the program. Engineering/manufacturing phase: actions: identify and assess risks; risk mitigation planning; Risk tracking and reporting. In this phase, program management must work with the information which defines the state of development of the system and potential and real problems. Risk tracking is very important in this phase since the earlier a risk is resolved, the lower the cost. Production / Deployment: risk planning; risk monitoring and reporting The emphasis in this phase is to monitor production process progress to determine both product and production deficiencies. Risk management can add consideration of possible variations around the nominal performance numbers. Operations and support: risk monitoring and reporting. In this phase the environment in which the system operates is leading. In unmanned space projects only risk or system monitoring and reporting can be done. The only possibility here to mitigate risks is risks concerning collisions and software risks.


- 17 -

Problems with the implementation of risk management Implementation of the risk management process in programs is not without challenges. Most difficulties may be classified as cultural, or having to do with attitudes and perceptions, rather than with the mechanics of the implementation. First is the inevitable resistance to change. Most people prefer the status quo and resist the introduction of new concepts, processes etc. An attempt that can be made to meet this challenge is by writing a series of memos explaining the process, its purpose, and its implementation as well as making personal contact with the key participants. Explaining to someone what you are going to do , why, and what benefit it will have for him or her can have a tremendous impact on the other person’s attitude toward the process. Another problem can occur during the initial assessment of risk. Some people on a program will react negatively to the connotation of having a risk in their area of responsibility rated as “High”. To overcome this problem a clear template for rating is necessary and periodically communication with the responsible person about the true rating of the risk. Further more it is important to introduce new metrics one at a time, to give people time to assimilate and see the benefits of the new processes and concepts. In this way you create a certain awareness which leads to not only acceptance of the process, but also active “customer” participation in the process 39.


- 18 -

Risk Management Tools This chapter describes tools that are, or can be, used during the risk management process and in particular during the risk assessment. A division can be made between tools that visualise risks, tools that aid a part of risk management process (e.g. risk monitoring tools), and tools that are software applications which perform integrated risk assessments. Risk visualising tools Risk matrix A risk matrix arranges the risk events in combinations of consequence of failure and probability. In this way it can easily be seen ,which risks are the risk drivers of a project. This type of tool is commonly used in combination with an expert based risk assessment

Figure 3: Risk matrix26

The boundaries of the different levels of risks are not symmetrical, one reason for this is the fact that independent of probability a catastrophic consequence of failure is always related with a extremely high risk, since it singly jeopardises the whole project. The tool is very easy to use, and it can also provide clear insight in the results of risk mitigation, as the shift of the elements over the matrix can be visualised. It should be noted that a risk matrix doesn’t provide a numerical relative importance of each rating, so the tool divides the risks in groups, but does not say anything about the ranking within such a group. Absolute risk graph The absolute risk graph can be used to visualise the number of risks currently present in a project and the status of them. Status is expressed in terms of “Retired” up to “High”. In this way project members can always see the status of the project their working on. When the graph is distributed during, for example work meetings, this tool is very useful to create awareness and commitment among the project members. Because historical data of


- 19 -

the project stays in the plot, it is easy to see the progress made in a certain amount of time.

Figure 4: Absolute risk numbers in a project through time 35

Risk Map The risk map, like the risk matrix, arranges the risk events in combinations of consequence of failure and probability. The difference between the risk matrix and the risk map is the scale of the axis. The qualified terms have made way for a quantified scale, although there is a relationship between the two. The risk drivers of a project are still very easy to identify, but now the relative importance between the different risk items can be deduced from the risk map. This type of tool is also very often used in combination with an expert based risk assessment.

Figure 5: Risk map plot of the tool Risk$


- 20 -

Risk Index A risk index is a histogram plot of the risk values (probability times severity) for each risk item in a project. The plot contains a red line indicating the thresholds from low to moderate and from moderate to high risk areas, and divides in this way the risk in different groups. The horizontal scale of the plot is occupied by the risk items or subsystems of the project, so it can easily be seen if an item is a high risk or not. An advantage of the risk index is that actual risks are plotted per item, and not the probability of failure against the consequence of failure. A disadvantage of the risk index is that it doesn’t lend itself to see the progress of mitigation actions since previous data is not taken into account. Integrated risk assessment tools There are numerous COTS (commercial of the shelf) integrated risk assessment tools. They are usually based on Microsoft Excel worksheets, database interfaces (like Microsoft Access), system architecture programs, or a combination of these. The differences in the different tools lie in the degree of support with respect to the risk management process. Below are two examples of commercial available tools. Risk$ 42. RISK$ is a tool to assess the risk in a hardware and/or software (conceptual) design by means of quantified scoring of constituent parts of a system for a number of aspects. It is a modified sub-set of CASETS (Computer Aided Systems Engineering Tool Set)42. The user interface is shown in the figure below.

Figure 6: Input data sheet of Risk$

Assessing risks with Risk$ works as follows: the first step is to select the definition of “State of Technology” that is specific to the project and import the risk items in the Excel sheet. The second step is selection of the risk categories that will be included in the assessment. Then, the ratings for each item per item can be determined. The next step is


- 21 -

to select the impact categories you want to use and rate the items to get the Impact Index. After recalculation of the worksheet Risk$ generates a risk map, risk index , and risk chart as outputs 42. Prophet 28. Another integrated risk management software tool is Prophet. The tool aids projects in identifying, assessing, mitigating and tracking risks by integrating risk management in the day to day activities and decisions associated with defining the systems, and with planning and managing programs. Prophet is actually a tool set, comprised of intuitive, user friendly Excel input forms, and a database application. The excel input forms are used by teams to enter risk data, and to document proposed mitigation or contingency plans. The excel sheets require virtually no tool training. The database that is used can integrate data from multiple teams, prioritise risk across a program, maintains risk assessment history and rationale, and generates risk reduction charts as outputs. All techniques that are used by the engine are based on DoD requirements.

Figure 7: Prophet’s main interface window 28

Risk monitoring tools Risk waterfall chart Areas requiring risk reduction may be anticipated and tracked using a “risk waterfall” chart. The example in figure 8 shows an assessed risk level for a particular case in an operational scenario. The goal is to plan and track a chain of activities intended to mitigate this risk. As when building a TPM chart, one solicits expert opinion on the appropriate activities to reduce the risk and the amount of reduction anticipated for each, which are indicated in the chart as step functions of risk reduction. In figure 8 , the information produced by activities 1, 2, and 3 directly contributes to reducing the risk that the product will not conform to requirements in this area. The expected combined effect of the information created by these activities is to decrease the risk to a level deemed acceptable 7.


- 22 -

Figure 8: Risk waterfall chart 7

TPM tracking chart A Technical Performance Measurement (TPM) tracking chart forecasts and monitors an evolving TPM relative to its requirement. Initially, experts with applicable project, system, and technology experiences may forecast a “planned profile” for the TPM. The profile is projected based on a number of factors, including technology risk, planned test and validation activities, historical data, experience, and expert opinion. The planned profile integrates information from these sources into a format that helps planners and managers make decisions. As the project unfolds, demonstrated measures are recorded periodically. Ideally, the actual profile will meet or exceed the requirement (in case of a minimum performance requirement), and uncertainty will decrease. TPM tracking also enables margin and risk management methods. In Figure 9 the requirement is represented by the dashed horizontal line. The circles represent a point estimate or measure of the most likely level of performance delivered by design at various times. Each estimate conveys uncertainty with the high-low bars showing the optimistic and pessimistic possibilities. In practice however many projects omit the uncertainty bars.

Figure 9: Technical Performance Measurement through project time 7


- 23 -

Remarks for future risk managers and tool developers The risk management process has clearly its benefits to a project or program, as with any method it has also its pitfalls. The following list of remarks are deduced from several case studies (9, 20, 26, 35, 38, 39, 47, 49, 55). The meaning of the remarks is twofold; first to learn from past failures, and second to create awareness at what degree people should be involved in risk management process. Some remarks seem obvious, but fact remains that errors are made how obvious some things may seem.

− Know the requirements.

− Look at all the risks, the greatest risk drivers are often overlooked.

− Look at risk relationships.

− Give every risk driver the proper attention.

− Often a risk driver will impact all facets of risk and the integrated result will be improperly estimated.

− Risk identification is the most critical step in risk management, yet is poorly done.

− Often risk are managed by lists that are ranked by subjective qualitative measures resulting in excessive expenditure of risk management resources.

− Faster, better, cheaper, or other competitive initiatives, have an negative effect on the risks.

− Know that the probabilities that are determined during the assessment phase also have a certain uncertainty, and probability distribution. It is however very often beyond the scope of the projects requirements to look into this uncertainty.

− Bring in the right expertise (if affordable) and keep the team size manageable.

− A group of people working together does not make a team. An individual working on a project form a threat since communication is poorly done.

− Make sure that experts really are experts. For a extended list of expert characteristics see 13.

− Bad documentation leads to loss of knowledge.

− Plans should not include status in their text.

− Risk management requires a change of mindset, it does not happen without strong support from top-down.

− “Do not shoot the messenger”, or verbally attacking the messenger of bad news has a negative effect on the further co-operation of project members in the process.


- 24 -

When designing a tool, one has to reckon with very specific customer requirements; this can be done in the case of a design for a particular project or company. Designing for commercial use is therefore more difficult, every customer has other needs and therefore other requirements. To try to find out the customer requirements, a questionnaire will be used in the next steps of the thesis study. Research on customer requirements has been performed before. The article “Factors in the selection of a risk assessment method 15” gives a good overview on considerations to be made when designing a risk assessment method. This together with the other references has led to a preliminary set of requirements that can be used as a guideline for developers. It should be noted that this is not a complete set of requirements and before one starts with the development or definition of a risk assessment method all the requirements should be known.

− A method should be capable of operating at any desired level of granularity.

− A method should be adaptable for different type systems.

− A method should be able to customised for a given situation.

− A method should provide a variety of quantitative as well as qualitative methods.

− A method should produce reliable and accurate results.

− A method should be easy to use.

− A method should be inexpensive to purchase and use.

− A method should be fast to use.

− A method should integrate with existing system development methods.

− A method should be able to give clear, precise and well justified .recommendations.

− A method should have an acceptable level of automation.


- 25 -

Reference List 1. A Process Driven Approach to Integrated Environments, Malcolm, N.E. and McKennon, J.,Proceedings INCOSE International

Symposium 1997,1997

2. A Sixth Discipline for Future Awareness,Hall, E and Gorsuch, T.,Proceedings INCOSE International Symposium 1997,1997

3. Application of ANSI Standard to Space Station Resources,Taylor, I and Bassler, J.,Proceedings INCOSE International Symposium 1995,1995

4. Application of some Risk Assessment Techniques: Formal Expert judgement and accident sequence precursors,Goossens, L.H.J. and Cooke, R.M., Safety Science vol 26 No 1/2 1997,1997

5. Applying Programmatic Risk Assessment to Nuclear Materials Stabilisation R&D Planning,Kenley, C.R. and Brown-Van Hoozer, S.A.,Proceedings INCOSE International Symposium 1997,1997

6. Comparing safety analysis techniques, Rouvroye, J.L. and Bliek, E.J. van den,Reliability Engineering And System Safety #75 2002,2002

7. Complex System Product Development: Adding Value by Creating Information and Reducing Risk,Browning, T.R., Deyst, J.J., Eppinger, S.D. and Whitney, D.E.,Proceedings INCOSE International Symposium 2000,2000

8. Denver International Airport: How could System Engineering Principles Have Prevented Disaster?,Cook, R.H.,Proceedings INCOSE International Symposium 2000,2000

9. Establish a Baseline Assessment to Manage Risks Using Risk Matrix, Willhite, A.M. and Norton D.R.,Proceedings INCOSE International Symposium 2000,2000

10. Evaluation of Risk Management Strategies for a low-cost, High-Risk Project, Shishko, R. and Jorgensen, E.J.,Proceedings INCOSE International Symposium 1996,1996

11. Experiences in using PRA as an operational tool, Perryman, L.J.; Foster, A.S. and Nicholls, D.R.,Int. Journal Pressure Vessels and Piping #61 1995,1995

12. Expert Decision Making, Hutton, RJ and Klein,G,Systems Engineering # 2 1999,1999

13. Expressing and interpreting the results of quantitative risk analysis. Review and discussion,Aven, T and Pörn, K,Reliability Engineering and System Safety # 61 1998,1998

14. Factors in Technical Risk Assessment, Hamann, R.J. and Zandbergen, B.T.C.,Proceedings 3rd European Systems Engineering Conference,2002

15. Factors in the selection of a risk assessment method, Lichtenstein, S, Information management end Computer Security 4/4 1996,1996

16. Facts and Values in risk assessment, F.B.Cross,Reliability Engineering and System Safety #59 1998,1998

17. Human error Identification for risk assessment of high risk systems Part 1: review and evaluation of techniques,Kirwan, B,Applied Ergonomics Vol 29 No 3 1998,1998

18. Integrated Risk Management: A Case Study, Roberts, B.B. and Winterlin, R.C.,Proceedings INCOSE International Symposium 1996,1996

19. Life Cycle Risk Management, Brekka, L.T. and Vlay, G.J.,Proceedings INCOSE International Symposium 1995,1995

20. Managing Research and development Projects: A Systems engineering Approach in the early stages of design,Brink, JR and Peisert,GD, Proceedings INCOSE International Symposium 1999,1999

21. Mathematical tools for probabilistic risk assessment, Bedford, T and Cooke, R., Delft University of Technology, 2001

22. Meaning and contextualisation in risk assessment, Jones, T.H., Reliability Engineering and System Safety #59 1998,1998

23. Methods and techniques for risk prediction of space shuttle upgrades, Hoffmann, C.R. ; Pugh, R. and Faysall, S.,AIAA Paper 98-25195,1998

24. On Comparing PRA results with operating experience, Martz, HF and Picard, R.R.,Reliability Engineering and System Safety #59 1998,1998

25. Onderzoek naar een Mogelijke Risk Management Methode, Baan, J.,Stork, Fokker Aerostructures report FAE/EMP/99-10-005,1999

26. Principles and guidelines for project risk management, Pennock, MJ and Haimes, Y.Y.,Systems Engineering Vol 5 2002,2002

27. Program Risk: the Balancing of Performance, Schedule and Cost Risks, May, C. and Olson, R.,Proceedings INCOSE International Symposium 1995,1995


- 26 -

28. Prophet- The engine for integrated risk management ,Huff, D.S., Proceedings INCOSE International Symposium 1997,1997

29. Providing a Framework for Effective Software Quality Measurement: Making a Science of Risk Assessment, Martin, R.A. and Shafer, L.H., Proceedings INCOSE International Symposium 1996,1996

30. Risk and Opportunity Management and the Project Life cycle, Forsberg, K. and Mooz, H., Proceedings INCOSE International Symposium 1995,1995

31. Risk Assessment and Risk Management, Heste, R.E. and Harrison, R.M,ISBN 0-85404-240-7,1998

32. Risk Indicators as A tool for Risk control, Oien, K, Reliability Engineering and System Safety #74 2001,2001

33. Risk Influence Analysis A method for identification and assessment of risk reduction strategies ,Rosness,R, Reliability Engineering And System Safety #60 1998,1998

34. Risk Management, Anon., KLM Manual,1999

35. Risk Management for the NASA/JPL Genesis Mission: A Case Study, Roberts, B.B. and Bennett, R.B., Proceedings INCOSE International Symposium 2000,2000

36. Risk Management for the new millennium, Dr D Hillson, Proceedings INCOSE International Symposium 1999,1999

37. Risk Management Guide for DoD Acquisition, Anon., Defense Systems Management College Press, Virginia, USA,2000

38. Risk Management in ESA's Scientific Directorate: A Case studty, Schroeter, J.,ESA Bulletin 107, pp. 64-71,2001

39. Risk Management on the Advanced Tomahawk Weapon Control System: a Practical Application of Risk Management in Today's Defense Environment, Ford, W.C., Proceedings INCOSE International Symposium 1995,1995

40. Risk Management: A Systems Paradigm ,Hessami, A.G., Systems Engineering # 3 1999,1999

41. Risk Reduction Through Changing Success Criteria, McKinney, D., Proceedings INCOSE International Symposium 2000,2000

42. Risk$, Fokker Space corp., User's Manual Risk$,1999

43. Safety risk assessment and management - The ESA approach, Preyssl, C, Reliability Engineering and System Safety # 49 1995,1995

44. Sources of Schedule Risk in Complex System development, Browning, T.R., Systems Engineering #3 1999,1999

45. Space Project Management; Risk Assessment, ESA/ESTEC,ECSS standard ECSS-M-00-03A,2000

46. Space Systems Engineering (Issue 2), Hamann, R.J., Delft University of Technology, 2001

47. Summary of the Results from the Risk Management Program for the Mars Microrover Flight Experiment, Shishko, R. and Matjevic, J.R., Proceedings INCOSE International Symposium 2000,2000

48. Systems engineering and Cost as an independent variable, Brady, J, Systems Engineering Vol 4 No 4 2001,2001

49. Tailored Risk Management; a Successful Application to a Major Program, Palmer, J.A. and Dechoretz, J.A., Proceedings INCOSE International Symposium 2000,2000

50. The evaluation Of Risk Management Tools, Elseth, B.O. and Hamann, R.J., Proceedings 15th Int. Cost Engineering Congress,1998

51. The Relationship of Technology Change Management to Risk Management, Mosier, S.P., Guenterberg, S.A. and Rphael, R.R., Proceedings INCOSE International Symposium 2000,2000

52. The Use of an Automated System Engineering Tool to support the Risk Management Process, Sadler, G.G. and Baker, L., Proceedings INCOSE International Symposium 1995,1995

53. Three Facets of Risk, Yellman, T.W., Proceedings 2000 World Aviation Conference,2000

54. Uncertainty in probabilistic risk assessment, Winkler, RJ, Reliability Engineering and System Safety # 54 1996,1996

55. Using a directed graph methodology for integrated risk assessment on space station freedom program, Gantzer, DJ,AIAA Paper 93-4675-CP,1993

56. What is " Risk", Hall, D.C., Risk Management Working group INCOSE,2002

57. What is "Risk"? Results from a survey exploring definitions, Dr D Hillson, Risk Management Working group INCOSE,2002


- 27 -

Appendix A


- 28 -

Article classification As can be seen in the reference list, almost sixty articles or books have given input for this literature survey. Are these references all of the same importance? This is obviously not the case. To select the references that can be used as input for next steps in the thesis work, all the references are examined with respect to eight criteria or questions, namely:

- Are the contents of the reference safety related?

- Does the reference deal with Risk Analysis or Risk Assessment?

- Are the described methods quantitative or qualitative of nature?

- Do the described methods make use of external or internal assessors?

- Is the Project Manager or the Systems Engineer the responsible person in case of the described method?

- Do the described methods focus on Technical Risk or on Cost/Schedule Risk?

- Can the described method be applied on small or large projects?

- What kind of tools are described or are there other remarks to be made? As stated earlier, the thesis study will focuses on technical risk assessment techniques that can be applied in small to medium sized space companies. This implicates the following for the selection of the references:

- The references should not be safety related.

- Risk Assessment should be of main interest and not Risk Analysis.

- It is more likely for the type of companies to have an internal assessor since external assessors are much more expensive.

- References that describe Technical Risk are of higher interest than those which describe Cost/Schedule Risk?

All the references that are categorised as important are written bold. The classification can also be used as a guideline for people who like to go more deeply into the subject of risk management and technical risk assessments.

Risk

Man

agem

ent L

itera

ture

Sur

vey

D

elft

Uni

vers

ity o

f Tec

hnol

ogy

- 2

9 -

Cat

egor

ies

R

efer

ence

Safety related

analysis

assessment

quantitative

qualitative

external evaluator

internal evaluator

PM

Engineer

Cost/Schedule

Technical

Large Project

Small project

Tool

s an

d re

mar

ks

1 A

Proc

ess

Driv

en A

ppro

ach

to In

tegr

ated

Env

ironm

ents

x

x

2

A Si

xth

Dis

cipl

ine

for F

utur

e Aw

aren

ess

x

3 Ap

plic

atio

n of

AN

SI S

tand

ard

to S

pace

Sta

tion

Res

ourc

es

x

x

x

x

4 Ap

plic

atio

n of

som

e R

isk

Asse

ssm

ent T

echn

ique

s: F

orm

al

Expe

rt ju

dgem

ent a

nd a

ccid

ent s

eque

nce

prec

urso

rs

x x

5 Ap

plyi

ng P

rogr

amm

atic

Ris

k As

sess

men

t to

Nuc

lear

Mat

eria

ls

Stab

ilizat

ion

R&D

Pla

nnin

g

x

x

x

x

Tech

nica

l mat

urity

mod

el

6 C

ompa

ring

safe

ty a

naly

sis

tech

niqu

es

x x

x x

x

FM

EA, F

TA, M

arko

v 7

Com

plex

Sys

tem

Pro

duct

Dev

elop

men

t: Ad

ding

Val

ue b

y C

reat

ing

Info

rmat

ion

and

Redu

cing

Ris

k

x

TPM

, Ris

k w

ater

fall

8 D

enve

r Int

erna

tiona

l Airp

ort:

How

cou

ld S

yste

m E

ngin

eerin

g Pr

inci

ples

Hav

e Pr

even

ted

Dis

aste

r?

x

x

9 Es

tabl

ish

a B

asel

ine

Asse

ssm

ent t

o M

anag

e R

isks

Usi

ng R

isk

Mat

rix

x

x

x

R

isk

mat

rix

10 E

valu

atio

n of

Ris

k M

anag

emen

t Sta

tegi

es fo

r a L

ow-C

ost,

Hig

h-R

isk

Proj

ect

x x

x

x

x x

11 E

xper

ienc

es in

usi

ng P

RA

as a

n op

eria

tiona

l too

l x

12

Exp

ert D

ecis

ion

Mak

ing

x

x x

x

x

x ex

pert

base

d as

sess

. 13

Exp

ress

ing

and

inte

rpre

ting

the

resu

lts o

f qua

ntita

tive

risk

anal

ysis

. R

evie

w a

nd d

iscu

ssio

n

x

x

baye

sian

met

hod

14 F

acto

rs in

Tec

hnic

al R

isk

Asse

ssm

ent

x

x

x

x C

ASET

S, s

impl

e ris

k m

ap

15 F

acto

rs in

the

sele

ctio

n of

a ri

sk a

sses

smen

t met

hod

x

se

lect

ion

crite

ria u

sefu

ll fo

r co

nstru

ctin

g re

q.

16 F

acts

and

Val

ues

in ri

sk a

sses

smen

t x

x

17

Hum

an e

rror I

dent

ifica

tion

for r

isk

asse

ssm

ent o

f hig

h ris

k sy

stem

s Pa

rt 1:

revi

ew a

nd e

valu

atio

n of

tech

niqu

es

x

ev

alua

tion

tool

Risk

Man

agem

ent L

itera

ture

Sur

vey

D

elft

Uni

vers

ity o

f Tec

hnol

ogy

- 3

0 -

Cat

egor

ies

R

efer

ence

Safety related

analysis

assessment

quantitative

qualitative

external evaluator

internal evaluator

PM

Engineer

Cost/Schedule

Technical

Large Project

Small project

Tool

s an

d re

mar

ks

18 In

tegr

ated

Ris

k M

anag

emen

t: A

Cas

e St

udy

x x

x

x

x x

x

x

19 L

ife C

ycle

Ris

k M

anag

emen

t

x

x

20

Man

agin

g R

esea

rch

and

deve

lopm

ent P

roje

cts:

A S

yste

ms

engi

neer

ing

Appr

oach

in th

e ea

rly s

tage

s of

des

ign

x x

ex

pert

opio

n, z

eta

met

hod

21 M

athe

mat

ical

tool

s fo

r Pro

babi

listic

Ris

k An

alys

is

x

x

22 M

eani

ng a

nd c

onte

xtua

lisat

ion

in ri

sk a

sses

smen

t

psyc

ho s

ocia

l arti

cle

23 M

etho

ds a

nd te

chni

ques

for r

isk

pred

ictio

n of

spa

ce s

huttl

e up

grad

es

x

x

24 O

n C

ompa

ring

PRA

resu

lts w

ith o

pera

ting

expe

rienc

e

x

x

25

Ond

erzo

ek n

aar e

en M

ogel

ijke

Ris

k M

anag

emen

t Met

hode

x x

x

x

x x

x

exam

ple

asse

ssm

ent m

etho

d 26

Prin

cipl

es a

nd g

uide

lines

for p

roje

ct ri

sk m

anag

emen

t

x

x

x

x x

x

x x

risk

mat

rix

27 P

rogr

am R

isk:

the

Bala

ncin

g of

Per

form

ance

, Sch

edul

e an

d C

ost

Ris

ks

x x

x

x

28 P

roph

et- T

he e

ngin

e fo

r int

egra

ted

risk

man

agem

ent

x

x

x

x x

x x

x Pr

ophe

t 29

Pro

vidi

ng a

Fra

mew

ork

for E

ffect

ive

Softw

are

Qua

lity

Mea

sure

men

t: M

akin

g a

Scie

nce

of R

isk

Asse

ssm

ent

x

x

30 R

isk

and

Opp

ortu

nity

Man

agem

ent a

nd th

e Pr

ojec

t Life

cyc

le

x

x

V

mod

el

31 R

isk

Asse

ssm

ent a

nd R

isk

Man

agem

ent

x

po

litic

al p

olic

y m

akin

g 32

Ris

k In

dica

tors

as

A to

ol fo

r Ris

k co

ntro

l

x

x

33

Ris

k In

fluen

ce A

naly

sis:

A m

etho

d fo

r ide

ntifi

catio

n an

d as

sess

men

t of

risk

redu

ctio

n st

rate

gies

x x

x x

34 R

isk

Man

agem

ent

x x

x

x

x

x qu

estio

nnai

re

35 R

isk

Man

agem

ent f

or th

e N

ASA/

JPL

Gen

esis

Mis

sion

: A C

ase

Stud

y

x x

x x

x

x x

x

36R

isk

Man

agem

ent f

or th

e ne

w m

ilenn

ium

x

Risk

Man

agem

ent L

itera

ture

Sur

vey

D

elft

Uni

vers

ity o

f Tec

hnol

ogy

- 3

1 -

Cat

egor

ies

R

efer

ence

Safety related

analysis

assessment

quantitative

qualitative

external evaluator

internal evaluator

PM

Engineer

Cost/Schedule

Technical

Large Project

Small project

Tool

s an

d re

mar

ks

37R

isk

Man

agem

ent G

uide

for D

oD A

cqui

sitio

n

x

x

x

x x

DoD

sta

ndar

d 38

Ris

k M

anag

emen

t in

ESA'

s Sc

ient

ific

Dire

ctor

ate:

Cas

e st

udy

x

x x

x

ris

k m

atrix

39

Ris

k M

anag

emen

t on

the

Adva

nced

Tom

ahaw

k W

eapo

n C

ontr

ol

Syst

em: a

Pra

ctic

al A

pplic

atio

n of

Ris

k M

anag

emen

t in

Toda

y's

Def

ense

Env

ironm

ent

x

x

x

x x (S

)x

x

40 R

isk

Man

agem

ent:

A Sy

stem

s Pa

radi

gm

x

x

x

ris

k m

atric

es

41 R

isk

Red

uctio

n Th

roug

h C

hang

ing

Succ

ess

Crit

eria

supe

rfici

al ri

sk a

rticl

e 42

Ris

k$

x

x

x x

x

x

x in

tegr

ated

tool

, map

, mat

rix,

char

t 43

Saf

ety

risk

asse

ssm

ent a

nd m

anag

emen

t - T

he E

SA a

ppro

ach

x

44 S

ourc

es o

f Sch

edul

e R

isk

in C

ompl

ex S

yste

m d

evel

opm

ent

x

x

x

45 S

pace

Pro

ject

Man

agem

ent;

Ris

k As

sess

men

t

x

x

x

x x

x x

ECSS

sta

ndar

d 46

Spa

ce s

yste

ms

engi

neer

ing

x

x

x

x le

ctur

e no

tes

47 S

umm

ary

of th

e R

esul

ts fr

om th

e R

isk

Man

agem

ent P

rogr

am fo

r the

M

ars

Mic

roro

ver F

light

Exp

erim

ent

X

x

x x

x

x

48 S

yste

ms

engi

neer

ing

and

Cos

t as

an in

depe

nden

t var

iabl

e

x

49

Tai

lore

d R

isk

Man

agem

ent;

a Su

cces

sful

App

licat

ion

to a

Maj

or

Prog

ram

x

x

x

x

x x

x

50 T

he e

valu

atio

n O

f Ris

k M

anag

emen

t Too

ls

x x

x

C

ASET

S, M

onte

Car

lo

51 T

he R

elat

ions

hip

of T

echn

olog

y C

hang

e M

anag

emen

t to

Ris

k M

anag

emen

t

x

x

x

inte

rvie

ws

52 T

he U

se o

f an

Auto

mat

ed S

yste

m E

ngin

eerin

g To

ol to

sup

port

th

e R

isk

Man

agem

ent P

roce

ss

x x

x x

x

x x

R

isk$

like

tool

53 T

hree

Fac

ets

of R

isk

x x

54 U

ncer

tain

ty in

pro

babi

listic

risk

ass

essm

ent

x x

Risk

Man

agem

ent L

itera

ture

Sur

vey

D

elft

Uni

vers

ity o

f Tec

hnol

ogy

- 3

2 -

Cat

egor

ies

R

efer

ence

Safety related

analysis

assessment

quantitative

qualitative

external evaluator

internal evaluator

PM

Engineer

Cost/Schedule

Technical

Large Project

Small project

Tool

s an

d re

mar

ks

55 U

sing

a d

irect

ed g

raph

met

hodo

logy

for i

nteg

rate

d ris

k as

sess

men

t on

spa

ce s

tatio

n fre

edom

pro

gram

x

dire

cted

gra

ph to

ol

56 W

hat i

s "

Ris

k"

de

finiti

on s

urve

y 57

Wha

t is

Ris

k?R

esul

ts fr

om a

sur

vey

expl

orin

g de

finiti

ons

de

finiti

on s

urve

y

Risk Management Literature Survey

Documents

Transcript of Risk Management Literature Survey