A classified bibliography of recent research relating to project risk management

ELSEVIER European Journal of Operational Research 85 (1995) 18-38

EUROPEAN JOURNAL

OF OPERATIONAL RESEARCH

T h e o r y a n d M e t h o d o l o g y

A classified bibliography of recent research relating to project risk management

Terry Williams Department of Management Science, Strathclyde University, Glasgow G1 1XH, UK

Received June 1993; revised November 1993

A b s t r a c t

This document contains a bibliography of recent research relating to project risk management, bringing together relevant research scattered across a range of publications. It considers how success or failure can be defined for a project (more than simple time/cost/technical target achievement). It looks at the historical evidence of projects, illustrating failure to achieve targets. What risk means to a project, and how a project team perceive, identify and quantify risk is considered - often the crucial credibility-point in practice. Techniques are discussed for the analysis of risk, to schedule (including analytical and more generally applicable simulation techniques), cost and technical achievement, both separate analyses and the first steps towards an integrated analysis. Success for project participation depends on who bears the risks, and the vital role of risk analysis in informing the contractual allocation of risk is explored. Finally, the management structures and procedures needed to manage risk are discussed.

Keywords: Project management; Risk analysis

O . I n t r o d u c t i o n

Science and Engineering increasingly pro- gresses by major projects, many of them high-risk. However, the need to identify a project's uncertainties, estimate their impact, analyse their inter- actions and control them within a risk-management structure has only in recent years been realised, mainly within the defence, construction and oil industries. Thus there is a need to integrate this work within a cohesive academic framework. This paper provides a first step in this process, by providing a classified bibliography of research relating to Project Risk Management,

0377-2217/95/$09.50 © 1995 Elsevier Science B.V. All rights SSDI 0377-2217(93)E0363-3

which is scattered across a wide range of publications.

This field involves interdisciplinary research, with input from

- Management Scientists: reviewing projects and management structures;

- Operational Researchers: analysing combi- nations of uncertainties;

- Engineers: estimating and forecasting uncertainties;

- Psychologists/decision analysts: looking at engineers' estimates. Where related areas are called upon in this paper, therefore, the emphasis is on quoting review

reserved

T. Williams / European Journal of Operational Research 85 (1995) 18-38 19

papers, so that this paper can act as a source document for further study.

The structure of the paper follows the logic of Williams (1993), who provides most of the following categories and discusses each of them in turn. These categories are somewhat heterogeneous, reflecting the fragmented nature of the field; the need for a 'synthesis of ideas' is well-recognised (Chapman 1990b). The paper is in eight sections, divided into three parts.

(i) What is risk? Section 1 gives a general introduction by looking at major projects, what 'success' means, and the need to study project risk. Section 2 looks at some historical evidence of project out-turns, providing the motivation for risk management. Section 3 considers the psycho- logical questions of what risk is, how uncertainties can be elicited from engineering/scientific experts, and what their attitudes to risk are.

(ii) Risk Analysis. 'Success' in a project can be regarded as provision on time, on budget, of a required performance or achievement (see Sec- tion 1). Thus the uncertainties identified in Sec- tion 3 need to be combined in three different sets of analyses:

- temporal analyses (Section 4); - cost analyses (Section 5); - performance analyses, and integrating the

three aspects (Section 6). (iii) Risk Management. Having analysed the

risks, the next stage is the writing of a contract to state and partition the risks between the contracting parties; Section 7 considers the progress made in such contractual mechanisms. Then, once the project starts, risk management needs to be an on-going process. Section 8 looks first at some relevant work on project management, then at the role formal risk-management can (or should) play in project management.

1. D i s c u s s i o n o f p r o j e c t r i s k

"One aspect of the future is obvious: all new undertakings will be accomplished in an increasingly complex technical, economic, political and social environment. Thus project management

must learn to deal with a much broader range of issues, requirements and problems in directing their projects to successful conclusions. Certainly, project management in every field will be called upon to address complexities and risks beyond anything experienced in the past" (Turnan, 1986). The field of risk management has grown up to analyse and manage these uncertainties, as managers seek to achieve their objectives.

Projects have to be managed to achieve their objectives (Turner et al., 1988), and it:is important that these objectives be defined and speci- fied (Knoepfel, 1990; Ireland and Shirley, 1986; also see Larson and Gobelli's survey, 1989, discussed below). Even then, however, the concept of project 'success', essential to understanding the reasons for a project 'failing', is not well defined.

Steiner in 1969 defined a project as "an organization of people dedicated to a specific purpose or objective. Projects generally involve large, ex- pensive, unique or high risk undertakings which have to be completed by a certain date, for a certain amount of money, within some expected level of performance". This three-fold criterion of success - meeting cost, schedule and performance targets - has become widely used. It catches the essential task of the Project Manager, and the essential trade-offs which he must make (Kohrs and Welngarten, 1986, report seeing a sign: "Good! Fast! Cheap! Pick any two"), and some authors (e.g. McCoy, 1986) have tried to develop an integrated success criteria based on the three aspects (see also Section 6).

However, many authors feel that defining success is not that easy - indeed, the Project Man- agement Institute in USA dedicated a whole conference to its definition and measurement (Pro- ject Management Institute, 1986). de Wit (1986) discusses the difficulties of measuring success as the attainment of project objectives, including two elements:

(i) Firstly the changes to project objectives between project phases. At its most simplistic, Avots (1984) suggests that schedule is most important early in the project, during the project cost becomes most important, and after the project, only technical performance is remembered.

20 T. W'dliams / European Journal of Operational Research 85 (1995) 18-38

(ii) Secondly, the number of stakeholders seeking success. Salapatas and Sawle (1986) define success to have been achieved only when three groups perceive success: the Client (from the view point of performance, budget and reputation), the Builder (profitability, reputation, client and public satisfaction) and the Public (environment, reliability, cost). Baker et al. (1988) require satisfaction from the parent, the client, the users/clientele and the project team itself.

Even post-completion success evaluation procedures are unreliable - see for example the discussion of prudency audits carried out on US nuclear plants in Hadzi-Pavlovic and Bissett (1986). Morris (1989) shows that there are much wider issues than the three-fold definition, including the technical and commercial definition of a project, and the consideration of strategic and external factors. Even without these method- ological considerations, it is found in experience that 'success' and 'failure' can be very close (Potter, 1987), and that "many large projects are saved from disaster by fortuitous circumstances" (Sykes, 1982).

The need for project risk management has been widely recognized. This is particularly so in the case of Major Projects. Fraser (1984) says that 'Normal' projects have the characteristics (amongst others) that "risk assessment can follow well established procedures as all risks are visible", "there are no catastrophic risks", "the scale of individual risks is small compared with the size of the parties involved and therefore there is no completion problem", but that "none of these characteristics is true of the largest projects"; "in general, beyond a certain size, the risks of projects increase exponentially and this can either be appreciated at the beginning or discovered at the end". Limited reviews of project risk management are given in Chapman and Cooper (1985) and Stringer (1987, 1992).

Risk management is practised formally particularly in the defence industry (discussed separately below). Construction is also a major user, because of the size of the projects undertaken (Perry and Hayes, 1985a, b; Habison, 1985), and Information Technology projects (Charette, 1989; Computing, 1990; Wolff, 1989; Humphries, 1989;

Morgan, 1989). R&D is an obvious field where risk is the main differentiator between projects (White, 1982). The analysis of risk is also increasingly playing a role in strategic decision-making (Bettis and Thomas, 1990)..The oil industry is also an important user (e.g. Baker, 1986; Hall, 1986; Clark and Chapman, 1987), but as well as being notoriously secretive, their requirements are somewhat unusual as there is more empirical data about two main areas of uncertainty - weather and geology.

The defence industry is slightly different in that risk management has now been made mandatory by the sole customer (in the UK, the MoD) and thus particularly espoused by contractors in defence tenders (Hayes, 1987). Some references to defence matters will be made in the Sections below, but further examples of practice can be found from the point of view of contractors in Bradley (1989), Clark et al. (1989, 1990), Telfer (1987), Bowers (1989) and Hopkins (1987). From the point of view of the customer, Chief Scientific Advisor (undated) gives the guidelines from which to work (see also PERAC (undated) and Pugh et al. (1989), Humphries (1989) and Hull, (1989, 1990, 1992) describe some of the outworkings in practice (see also Roberts, 1987). In the US defence industry, risk is also mandatory, and is indeed more visibly built into the acquisition process. For example, the US Army has for almost 20 years used an acquisition strategy called Total Risk Assessing Cost Estimating (TRACE), which seeks to add a factor to baseline estimates to allow for uncertainty (see Cocker- ham, 1979). See also Brabson (1983) for a discussion of US procurement.

2. Historical evidence

Before considering risk to the projects, we must consider whether or not the evidence shows that projects are meeting their success criteria. If they are, then clearly project managers are being successful in identifying and avoiding or amelio- rating risk, and there is little need to advance the field. Anecdotal evidence is available; for exam-

T. Williams/European Journal of Operational Research 85 (1995) 18-38 21

ple, Cleland and King (1988) cites half a dozen American examples, including Forbes magazine's comments on the US nuclear power programme, and the well-known case of the $8 billion Trans- Alaskan Pipeline, of which the State of Alaska claimed that $1.6 billion was ' imprudent ' . How- ever, this evidence is not sufficient. There has, however, been a certain amount of work collecting data on historical project out-turns, beginning with work such as Marshall and Meckling (1959), collecting data to try to predict overruns. In a similar vein, Ruskin and Lerner (1972) on 73 US Air Force projects and carried out regression to see the factors related to overruns.

The key text in summarising the historical evidence, at least up to 1987, is Morris and Hough (1987). They list 33 references containing data- bases of project out-turns: the reader is referred there to details of these, and they will not be replicated here (this part is usefully summarised in Morris, 1986). One data-base that is perhaps worth mentioning and expanding, as it is an important source of world data, is the World Bank Tenth Annual review of project performance audit results (World Bank, 1984), showing a gradual decline in performance, partly because projects are getting more risky, with cost overruns shown up to 560%, with cost often contained at the expense of scope (however, their over-run figure includes inflation (see discussion below) and the effect of the oil-crisis); time over-runs average 61% (again including changes to scope) (more discussion is given in Baum and Tolbert, 1985). Morris and Hough's preface to their list of data- bases states that "curiously, despite the enormous attention project management and analysis have received over the years, the track record of projects is fundamentally poor, particularly for the larger and more difficult ones. Overruns are common. Many projects appear as failures [refer- enced to a Financial Times article], particularly in the public view. Projects are often completed late or over budget, do not perform in the way expected, involve severe strain on participating in- stitutions or are cancelled prior to their completion after the expenditure of considerable sums of money". In summarizing their data-base, they state that " there are hardly any reports showing

underruns . . . . In all the other cases, represent- ing some 3500 projects drawn from all over the world in several different industries, overruns are the norm, being typically between 40 and 200 per cent, although greater percentage overruns are found in a number of groupings, particularly certain defence projects and in the US nuclear industry". (This last figure relates to cost;overruns.)

Morris and Hough point out that these figures must be treated with care, and give a number of caveats to the cost over-runs; these are worth considering in detail, as they must be taken into account in looking at all the data-bases quoted in this Section. Firstly, some of the 'overruns' relate to increased order quantities (thus over-running the original budget). This, or a n y unforced client-proposed project changes which raises the contract level (i.e. a Contract Change or Varia- tion Order), is not an overrun in the accepted sense, but rather a feature of the data-bases. Regulatory changes are considered as part of customer-requested changes, such as in the US nuclear industry, causing "a substantial proportion of the cost growth in this industry". How- ever, this author thinks that this is too simplistic - for semi-public or mixed pr ivate /publ ic projects, which increasingly make up mega-projects (see below), regulation changes are possibly the major risk (see the discussion of Merrow, 1988) below), and it is not always clear whether the customer or prime contractor takes the risk. This also comes into an early Depar tment of Energy (1975) report, which highlights the additional costs to North Sea projects due to additional regula- tions, and relate this to Concorde's noise and pollution constraints. The second most important caveat is the treatment of escalation. Most gov- ernment projects specifically disallow any allowance for inflation in the tender price, and escalate payments in accordance with some accepted index; an example quoted in Morris and Hough is that the "Central Electricity Generating Board (CEGB) discounts all costs back to the project's budget base dates. This makes comparison of overruns on UK nuclear power l~lants with those experienced by the US nuclear planes, for example, almost impossible to make accurately - US plant costs include not only infliation but

22 7". Williams/European Journal of Operational Research 85 (1995) 18-38

generally the finance charges for funds used during construction (AFUDC: Amortized Funds Used During Construction)". Thirdly, the treatment of contingencies differs from datum to datum: quoting again, "The Apollo programme, for example, came in at $21b, only $1b over its original estimate. Few know that the initial estimate included $8b of contingencies . . . . Very few public projects have even semiformal contingency budgets". Finally, Morris and Hough point out that overruns are not necessarily a measure of project success, such as when market conditions change to make an apparently over-running project profitable. Perhaps one further caveat is Fox (1984) who warns against hindsight in post-project evaluations, because big projects are difficult to manage.

A major study carried out since Morris and Hough is described in Merrow (1988). This is an analysis carried out with RAND of a data-base of 52 projects, all over $500m. It addresses four points in particular. Firstly, have the projects met their targets? The data-base shows that many projects met their time-target - the average slip- page was 17% - but there was a clear over-run on cost - the average over-spend was 88%; however, the caveats made by Morris and Hough must be made here, since it is not clear what the original budgets contained for the projects. Sec- ondly, Mega-projects (defined here as $1b) were compared with merely 'big projects' (i.e. those $0.5-1b), with performance being (proportion- ally) roughly similar. Thirdly, the main problem causing over-run was found to be regulatory problems; a regression equation is developed giving

cost growth = 1.04 + 0.78

* (no. regulatory problems) + " .

where the 'number of regulatory problems' varied between 0 and 5. (There is a similar equation derived for time-growth, although this is less de- pendent upon regulation.) Finally, some planning actions are suggested on the basis of the figures.

Three major studies have been carried out to see if there is any relationship between the management structure used in the project and the success of the project. First of these is the study described in Larson and Gobelli (1989) (reported

in a number of papers, see Section 8). 547 companies, mainly concerned with development projects, were studied. The results in terms of management structure are discussed in Section 8, but the results on out-turns are not explicitly given; indeed, the authors explain that "Data collection procedures . . . and the need to generate a large enough sample to draw meaningful comparisons prohibited the use of multirater evaluations of success which have been employed in other studies [refs]. Respondents were simply asked to eval- uate their project . . . . Although other researchers have relied on similar perceptual measures of success [refs], individual appraisals do not provide a strong basis of measurement". The second is Might and Fischer (1985), who studied 103 development projects. Again, the findings on management structure are discussed in Section 8, but again no explicit results on out-turns are given; however there is an interesting correlation matrix between the various measures of success, showing a 2 / 3 correlation between cost and schedule success, but only a 1 /3 correlation between the 'technical success relative to the original plan' and either the cost or technical success. Finally, Myers and Devey (1984) (which also occurs in the Morris and Hough data-base), who studied the Pioneer [Process] Plant Study data-base, showing cost out-turns of 90-300% of planned, time over-runs of 0-30 months and performance out- turns of 0-105% of planned.

The defence world, each nation with a single customer carrying out risky development projects, is well-suited for collecting data. In fact, for the UK, there is surprisingly litte available summary data, although a great wealth of anecdotes which give valuable insights to the success or failure of defence projects (e.g. Bryson, 1982, 1986). The key turning-point in effective UK defence project management was the Downey Report (Ministry of Technology, 1969). This referred back to an early Ministry of Supply report in 1958, which studied about 100 projects and found an average ratio of final costs to initial estimates of 2.8 (and also quotes an analysis of US programmes giving an average cost ratio of 3.2 and an average time ratio (actual divided by original estimate) of 1.36). However, only anecdotal data is given on the


1960's; indeed, Downey states that "the need throughout the Ministry of Aviation for systemi- cally recorded information on past projects has been recognized for some time, but our investiga- tions have disclosed that unfortunately much of the information required is not readily available, and where it exists is not suitably recorded. Moreover, little progress has been made towards devising a suitable system of recording". Downey follows this statement by, "It is true that in the Controllerate of Aircraft, the Project Time & Cost Analysis Branch (PTC(An)), formed from the old Technical Development Plans, can provide some information about airframe development, but this is not in a form in which comparisons can easily be made . . . " . This work has continued, and this department now has a data- base of military aircraft (and guided weapons) project outturns. Pugh (1987) gives a discussion of the results on time-scale out-turns, showing (inter al) that time- and cost-overruns are not highly correlated, and that development timescales have not significantly increased since the war.

The key auditors of the Ministry of Defence's expenditure are the National Audit Office. Their report on the control and management of the development of major equipments (National Au- dit Office, 1986) examined twelve projects, and found real increases of almost £1b (91%) since the Staff Requirement was approved (see also their previous procedural report, National Audit Office, 1985). The House of Commons Select Committee also review the Ministry of Defence's expenditure. Their Report on the procurement of major defence equipment (House of Commons, 1988) discusses ten programmes currently in procurement, so without any out-turn data, but they do show some data on the changing nature of the contracts (the percentage of contracts that are cost-plus having halved over the previous 5 years) and that the Downey proposals were not being achieved (e.g. only 8% of expenditure was on early (risk-reduction) phases of the project rather than Downey's 15%).

More formal data-capture has occurred in the USA. Acker (1980) show how the American De- partment of Defence (DOD) procurement process had matured to that point from 1960. He

quotes the Harvard Weapons Acquisition Re- search Project of 1962, who showed that systems tended to exceed their specification, but at the expense of time (development time being on average 36% longer than predicted) and cost (on average 7 times the original estimate)i DARPA (America's advanced defence research organisation) commissioned a report to analyse the out- turns on projects in 1977-1980 (Meridian, 1981). This aggregated the projects into 'high',:'medium' and 'low' risk programmes; high-risk programmes had year-end cost factors (actual/predicted) of 1.5 to 3.0; also given are five NASA pr0grammes, which for the financial year 1980 had cost factors around 1.5. A survey of 246 US Army programmes by Arbogast and Womer (1988) had cost over-runs of -21% to +437% (mean 15%)and time over-runs of - 8 to +74 weeks ~(mean 7) (baseline estimates not given). A history of the data available on DOD programmes is given in Baker et al. (1988), who also describe a survey of 646 projects, 2/3 being private sector and 1/3 public sector, with conclusions about the particular difficulties of the public sector. The key auditor of the US DOD is the General Accounting Office; their general report (General Accounting Office, 1983), notes the policy of identifying risks and reducing them before production, but then analyses the causes of production problems in six (missile and aircraft) programmes; more particularly, the report on the Trident and! nuclear- powered submarine projects (General Accounting Office, 1982) describes considerable overspends (each Trident boat costing $1.3b in 1982 prices).

Most of the references noted in this paper contain some evidence of success or failure in projects. Many papers cover just one country (e.g. Arditi et al. 1985, which describes a data-base of Turkish projects, with 40-110% cost over-runs - although with anecdotes of big projects with up to 386% over-run - and time over-runs of 34-44%; furthermore, evidence suggested performance to be declining) or one industry (e.g. Statman and Tyebjee, 1984, on R&D, showing these projects have been no riskier than any other type of project). Other papers are vehicles for a lot of interesting anecdotes of big projects (e.g. Stall- worthy and Kharbanda, 1987, who discuss the

24 T. Williams/European Journal of Operational Research 85 (1995) 18-38

North Sea, nuclear construction, chemical plants, etc).

3. Definition and quantification of risk

Risk analysis and management can only be as good as the perception and quantification of risk by the project team, and it is at this point that the credibility of risk analysis often falls down. This section considers the questions of what risk is, how uncertainties can be elicited from engineering/ scientific experts, and what their attitudes to risk are. "[The] identification and assessment of environmental uncertainty then becomes critical to project success and lies at the heart of the decision-making process" (Adams and Martin, 1982), this decision-making not being aimed at eliminating all risk, but "the purpose of risk analysis and risk quantification is always to provide input to an underlying decision problem which involves not just risks but also other forms of costs and benefits ... that risk is acceptable which comes along with the optimum decision problem" (Kaplan and Garrick, 1984).

There has been a considerable amount of research and publication, on the meaning of 'probability'. Schafer (1976) defines two types of probability: aleatoric probability, relating to the outcome of an intrinsically uncertain situation (from the Latin alea, meaning dice), and epistemic uncertainty, relating to a measure in belief in a proposition, or more generally to a lack of complete knowledge (this debate is excellently ex- pounded by Oakes, 1986).

The word 'risk' generally has implications of negative or adverse results from an uncertain event: Ansell and Wharton (1992) discuss the origins of the word and some modern definitions, all referring to the uncertainty of the event, and the adverseness of the effect. (However, both of these aspects must be present; the casual reader will be confused by a number of definitions of risk in the literature that (wrongly) define risk as simply a bad event, for example Fishburn (1984), who calls a certain bad event 'risky', Statman and Tyebjee (1984) who define risk as 'a high probability of failure', or in the IT field, Bunyard, who

defines software risks as software defects.) The division between epistemic and aleatoric can be found to underlie a lot of the discussions of uncertainty. For example, the popular idea of likelihood/impact grids (where risks are plotted as points on a graph of likelihood versus impact, Williams, 1993) is extended by Charette (1989) into 3-dimensional graphs with independent axes he labels severity (i.e. impact), frequency (i.e. likelihood) and 'predictability' (i.e. the extent to which the risk is aleatoric rather than epistemic). Wynne (1992) takes this distinction further, by distinguishing between risk (where the 'odds' are known), uncertainty (where the odds are not known, but the main parameters may be), igno- rance (where we don't know what we don't know) and indeterminacy (described as 'causal chains or networks open' - so presumably implying an element of unknowability). Busby (1992) uses 'con- trollability' rather than frequency and 'significance' rather than 'impact'. This paper will take the word risk to refer to an adverse event which is uncertain, either aleatorically or epistemically (although of course these two types of uncertainty require different attitudes), which in practice covers most of the main requirements of projects. In general, epistemic risk forms the biggest problem in planning and prediction, where there is little historical evidence on which to base the prediction (so work in combining experts' opinions based on past evidence, such as by Bhola et al, 1992, provides less assistance than might be expected).

The definitions of uncertainty and risk obvi- ously have implications for how these are per- ceived by estimators and decision-makers. There is a fair amount of work undertaken on the perception of uncertainty, a key early paper being Tversky and Kahnernan (1981), who consider the significant effects of question-framing and formulation on choice and uncertainty evaluation. The now classic text on work carried out since that time is Kahneman, Slovic and Tversky (1986), to which the reader is referred for further references.

The implications for the perception of risk within management problems is discussed in MacCrimmon and Wehrung (1986), who provide a full literature survey, and describe a study of

7". Williams/European Journal of Operational Research 85 (1995) 18-38 25

managers' attitudes to taking risks and how they choose options and risk action/outcome benefits in standardized risk situations; it also compares various risk measures in these standardized situations (e.g. Prospect Theory, Kahneman and Tver- sky, 1979). One further step taken by this text (building on MacCrimmon and Wehrung, 1984) is consideration of managers' adjustment of risk, by hedging or delaying - this will be discussed further in Section 8. Further work on management perspectives on risk is described in an important paper by March and Shapira (1987), showing that managers' decisions are affected more by performance targets than probability-estimates (also providing a good literature list). However, a warn- ing is given in Jackson and Carter (1992) that good assessment of individual probabilities does not necessarily imply the ability to predict system failures. (See also work on propensity profiles (Hull, 1980).)

Before risks can be quantified, it first needs to be decided what the risks are. Little structured work has been done either on ensuring complete- ness or on eliciting such risks from experts. Charette (1989) does attempt a structured approach for software projects, describing a 'Risk Estimate of the Situation' and an associated information gathering process. Noonan and Tham- hein (1986) mention a process during the project of identifying risks throughout the project. One technique that has some value is Influence Dia- grams (e.g. Ashley, 1987), but other than the occasional unsuccessful attempt to model engineer's mental process (e.g. Albino, 1988) this is an area needing further study.

Quantifying the risks has been studied more, and its importance is well-known, both in initial planning (Smith 1985) and in revising initial plans (Hughes 1986). There are two main approaches:

(i) Direct encoding, where individual probability distributions are elicited from experts. The classic treatment is Merkhover (1987), who describes the six stages of motivating, structuring, conditioning, encoding, verifying and aggregating, and discusses the various pitfalls that can occur, such as subjects' tendency to underweight distri- butional information, or regression towards the

mean (similarly Zelany, 1979). Again, Kahneman, Slovic and Tversky (1986) also go into these issues in depth (Keeney and Winterfeldt, 1989, provides a good but easier introduction).

(ii) Pragmatic approaches, recognizing the time-constraints under which such studies often operate, attempt to provide such quantification as is possible. The simplest such approach catego- rizes into High/Medium/Low probability and impact (a very common device, but although these classes can be defined arbitarily for probability, the definition of 'High' impact is problematic, although Ireland and Shirley (1986) provide some useful definitions which require further testing). The other main device is to estimate plus-or- minus percentages; however, the actual values to give to these -t-% figures is still a matter of considerable debate (de Wit, 1987). One example of an attempt was Hackney's Definition Rating method (Hackney, 1986) (based on Hackney, 1985), which tried to specify the degree of definition of a project (and thus to some extent the limitation of epistemic uncertainty), both top- down and in its elements; this is used to fit very simple Normal distributions and to suggest some +% figures. Saunders (1990) similarly gives generic + % figures based on parameters such as 'the knowledge of the job' (again, essentially epistemic considerations).

A better answer is perhaps to combine the pragmatic approach of (ii), but learning all of the lessons gained from (i) about debiassing etc.

One further complication to the study of initial estimates is the feed-back the estimates have on outcomes, in particular the 'Parkinson's rule' effect that durations will expand to fill the time allotted, and while much more work is needed to ensure that this effect is taken into account (following the work of Guitierrez and Kouvelis, 1991, and Littlefield and Randolph, 1991), it can at least be pointed out that this effect implies bias in the original estimates.

Finally, the area of Technological Forecasting has relevance to the problem of predicting the success or otherwise of developmental projects. In this context, Wright (1987) provides a good summary of the state of the art.


4. Analysis of time risk

The area of project risk most studied is the risk to achieving timescale. Indeed, in much of the project management literature, 'risk analysis' means ' the analysis of risk to the programme', and such risk analysis has been practised in some form for some time (e.g. Hopkins, 1987, in the defence industry, Hall, 1986, in the oil industry, and Cooper and Chapman, 1986, a general text- book).

The temporal analyses are all based on the network (PERT or CPM) concept (see, e.g. Moder, 1988). This is a well-established analytical tool when the project is deterministic (although Blankevort, 1987, claims that " the emphasis on network planning techniques may have made a full turn in the past two decennia. It started with PERT, a typically goal-oriented technique. This evoluted [sic] to more activity-oriented methods (CPM, precedence, linked bar-charts) to the suggestions of today to explore new ways of defining project goals (CAD-systems?)".)

However, there are divergent trends of analysis when there are uncertainties in the network. There is an extensive literature on the solution of stochastic networks, but a full discussion of this is out of place in such a paper. The first line of attack on stochastic networks is of course analytic, and there are excellent reviews of this work in Adlakha and Kulkarni (1989) and Ritchie (1985). However, there is no simple analytic solution of the stochastic network (other than specific cases, e.g. exponentially-distributed networks; see Kulkarni and Adlakha, 1986). Of more promise in the analytical solution is the derivation of exact bounds (see for example Dodin, 1985, Devroye, 1979, and Kamburowski, 1985, 1986). Also of promise in the analytical solution is the derivation of approximate solutions (e.g. Anklesaria and Drezner, 1986, using stochastically dominat- ing paths).

These analytical techniques are for networks with no resource-constraints. There is no work done for the frequent practical case when there are resource constraints - indeed, work is not complete on deterministic networks with resource constraints. In practice, however, these and complex uncertainties need to be modelled. Williams

(1985, 1990) and Kidd (1991) (describing VERT) make the point that simulation is essential in such cases. Ragsdale (1989) gives a good summary of the literature of network simulation; Golenko-Ginzburg (1988) gives some of the history of PERT leading to GERT leading to VERT (see also llumoka, 1987). Examples of such complexities include unusual distributions (e.g. combining 0-1 and continuous uncertainties); resource availabilities or requirements that are uncertain (and varying over time), effects that operate across a range of activities a n d / o r resources; uncertainties in the project network structure (i.e. probabilistic branching, where the effect is out- side management control, or conditional branching, where management makes a decision based on the progress of project parameters), domain- specific uncertainties (such as the random failure of test-rigs etc). Although straight simulation is more frequently used in practice, there have been theoretical advances in improving the information derived from simulations, generally using either Variance Reduction (e.g. Avramidis, 1991) or Conditional simulation (e.g. Adlakha, 1986). It should be pointed out, however, that Chapman (1990) says that "always using Monte Carlo simulation is a simple general solution, but it is often inefficient 'using a sledgehammer to crack a nut'. More important, to continue the analogy, it en- courages an approach which makes the walnuts very unattractive eating, and difficult to find". There are also still considerable misunderstand- ing about the technique (for examples, see Badiru, 1991, and Pugh and Soden, 1986).

Most of this analysis has centred on analysing the duration of the project. In practice, management are interested in two questions: the total duration, and which activities are critical in determining that duration. There has been some research into finding those activities most likely to fall on the critical path (e.g. Dodin and E1- maghraby, 1985). However, Williams (1992a) shows that this definition of criticality is not sufficient to highlight those activities which require management attention, and that the concept of cruciality is required as well as criticality.

During most analyses, certain moments of activities' durations can be estimated, but rarely are there full definitions of duration distributions.


Therefore, some a priori distribution is needed. For analytical studies, it is helpful if the distribution is such that PERT calculations provide a random variable of the same distribution (e.g. Golenko-Ginzburg, 1989). As well as work on the classic Beta distribution (e.g. Farnum, 1987), some authors have proposed their own distributions (e.g. Berny, 1989). Williams (1992b) for practical simulations looks for the simplicity, in particular the Triangular distribution. (See also the work using extreme value theory in Dodin and Sirvanci (1990).) An additional complication, pointed out in Section 3 above, is the Parkinson Law effect (Gutierrez and Kouvelis, 1991) which suggests durations should be treated as targets rather than analytical inputs (Littlefield and Randolph, 1991).

The use of computer packages for deterministic project management is well-established (Le- vine, 1988). Lists and summaries of the available packages are given in Morrison (1985), Thamhain and Wilemon (1986), Wasil and Assad (1988) and Wit and Herroelen (1990) (this latter providing a useful bibliography), and Bowers (1985) gives some guidance on desirable features. However, in the real, uncertain, world, "users clearly consider they are not given the tools they need" (Gabriel, 1987), and there is no generally accepted software approach to uncertain PERT. "The PERT/CPM package plus data-management is accepted as the target model" says Harris (1987), although he goes on to say that no package comes near that target as yet. Although advances are being made (for example, Assad and Wasil look at What-If analyses (1986) and resource levelling (1988)), clearly more work is needed here, although the temporal analysis must be integrated with the cost and performance analysis (see Sec- tion 6), and the whole integrated into the management structure of the project (see Section 8).

As well as these technical issues, there are questions about what to measure when judging temporal progress, and some suggestions are made by in Balthazar (1987).

5. Analysis of cost risk

The use of probabilistic sums to calculate ranges of cost-estimates is well-established and is

not unusual. It has been established for some time in making individual business investments (Hull, 1980), particularly in risky situations such as projects involving multiple currencies (Anti, 1980), with coverage applied to the whole of a company's risk exposure (Shapiro and Titman, 1980). This is what is often termed 'Risk Analysis' in the financial world. The problem is inherently easier than the analysis of time risk because of the additivity of cost-elements. The use of this type of Risk Analysis for major projects at a high level, i.e. Capital Budgetting, has been claimed to be very successful (see for example:early in- stances by Coats and Chesser, 1982; Hertz and Thomas, 1983). However, Ho and Pike (1992, and Pike and Ho, 1991) describe a survey of its use in large UK firms which suggests that only around a quarter of such firms use any formal risk analysis. Such analysis is important at the start of a project, since although cost is spread throughout the project, cost causation is concentrated near the beginning of the project (see, for example, the graphs in Hirzel, 1986).

Many such 'analyses' used in practice are of the most simple type, merely giving probabilistic values to cost elements then taking a s~ochastic, rather than deterministic, sum. This sort of analysis is carried out by the Treasury (Treasury, undated) and Ministry of Defence (Hull, 1992); Bradley et al. (1990) also describe an example using simulation to carry out the summation. Hodder and Riggs (1985) go a little further by putting such analysis into Discounted Cash Flow. Gehrung and Narula (1986) sum costs when the costs are subjective and vague in the earlY stages of a project. Yeo's (1990) useful small classification of cost risks, used to provide a contingency allowance calculation, uses a straightforward summation. Cooper and Chapman (1985) also describe a slightly more complex summation.

There are more sophisticated methods for carrying out full probabilistic financial appraisals of projects. Burke and Ward (1988) give a full account of such methods, with references. The paper above by Ho and Pike (1992) also provides a useful summary of methods, both carrying out a simple risk adjustment and full probabil!istic risk analysis.

28 T. Williams / European Journal of Operational Research 85 (1995) 18-38

6. Analysis of performance risk and overall 3-way analysis

There is little in the literature about the analysis of total performance risk. However, "research has shown time and again projects fail because the technical content of the program is not controlled strictly enough or early enough" (Morris, 1988, who gives a literature list to back up this point). While the addition of time elements is much more complex than the addition of cost elements, the summation is at least feasible, being the summation of a similar type of item. For performance risks, the different measures making up the overall required technical performance will not be commensurate with each other; for example, the contract for a warship might have technical targets for speed, weight, reliability and infra-red signature; these measures, while inter- acting in their achievement or otherwise, cannot be combined together. To carry out an overall analysis, a common measure must be used, which can be either

(i) probability: that is, calculate the total probability of achieving the technical target (i.e. all of the individual measurements)

(ii) cost: where available, the combined distribution of the sum of Liquidated Damages or (the reverse) Incentives (as suggested in Kohrs and Welngarten (1986)); this is discussed further in Section 7. Franke (1987), for example, makes the suggestion that all risks be put into cost (although it is questionable whether this is feasible).

In both cases, it must be remembered that the distributions or probabilities will not be independent, but rather will be highly correlated, sometimes negatively (i.e. the achievement of one target - say, weight - is likely only to come with a lower probability of the achievement of the other target - say, speed). The calculation of the risk- profiles of the individual measures themselves will be different depending on the type of measure. Simply additive measures, such as weight, might be carried out exactly as the simple cost summations in Section 5. More complex summations, such as for infra-red signature, must be carried out in ways specific to that measure.

Once the technical performance measures

themselves have been made commensurate, there is a further problem of making the (overall) technical performance commensurate with the temporal and financial performance. Only then can a proper measure be provided of the 'success' (and hence risk to, or progress towards, success) of a project, as discussed in Section 1.

The first steps towards this aim, combining the time and cost factors, have been taken. Wollmer (1985) starts to add cost to the activities by as- suming activity durations are linear functions of investment. Russell (1986) puts cash-flow considerations into resource-constrained networks. Smith-Daniels and Aquilano (1987) and Schtub (1986) both try to maximise the Net Present Value of a project by trading-off late start of activities with the probability of the project as a whole being late. Elmaghraby (1990) suggests a very simple model to aid bidding for projects, on the basis of the time to, and cost to, Key Events.

The movement towards a full three-fold success analysis has been slower (Williams, 1993). Ireland and Shirley (1986) make some small steps towards this goal. Hazelrigg and Husband (1985) in evaluating R & D projects propose a model which combines the probability density functions of 'technological advancement' , time and cost, although it is not clear whether this would trans- pose to the analysis of individual projects success- fully. The most probable way forward is to make the measures commensurate based on the common measure of cost, incorporating amortization, Liquidated Damages, Incentives, etc. For this we must consider contractual aspects of projects and the incorporation of risk into contractual instru- ments, as in the following section.

7. Contractual aspects

A most important aspect of risk for either a contractor or client is who is liable for the risk, and who thus has the motivation to avoid or vitiate the risk (Curtis et al., 1991). Therefore, the contract is at the heart of any analysis of risk ( "A prime function of any contract is to identify, access and allocate risk", NEDO, 1982). Many standard forms of contract such as FIDIC (Jaynes,


1985) hardly mention risk, and frequently, contracts are not clear on who bears the risk (Robin- son, 1987, claims clarity on this point as the strength of Design and Build contracts).

Not surprisingly, it is firstly the high-risk industries that have advanced the rigorous distribution of risks in contract. For example, Kahn (1981) gives a description of the cost-reimbursement type of contractual schemes used in the European Space Agency (ESA) (and discusses technical and time performance bonuses, and risk premiums paid to a 'prime contractor'). The second industry here is the construction industry, which holds probably the biggest projects. For example, CI- RIA work on target and cost-reimbursable contracts is described in Perry and Thompson (1982); Nahapiet and Nahapiet (1985) give the results of a survey of the use of different construction contractual arrangements; others describe particular schemes, such as Tiong (1990) on Build-Operate- Transfer. Most recently, a conference was held on Construction Contract Policy (Uff and Cap- per, 1989), which included a paper by Abraham- son (1989) pointing out inter al that changing the contract does not change the actual risks faced and that the prices quoted to a client are generally for a risk-reduced world, and have to be seen as such; and another paper by Chapman et al. (1989), who point out the implied problems with choosing the lowest fixed price bid, including that this bidder might be the least aware of risk.

A prime contractor takes on all the risks to achieving some written description of the project, and is seen by many laymen as the answer to a client's risk-management problem (so is wel- comed by, e.g. House of Commons, 1988, 1991); however, there are additional risks, particularly the risk that the client does not write exactly and unambiguously what he requires in the specification, and commercial risks such as the contractor going bankrupt. The place of risk within the p r i n c i p a l / a g e n t or p r i m e / s u b c o n t r a c t o r relationship is a 'next step' that "researchers should focus on" in the research being done into this relationship (Eisenhardt, 1989).

There are a number of possible payment schemes that can be used within a contract. A

good overall summary of the main types of payment scheme is given in In't Veld and Peeters (1989) (following Herten and Peeters, 1986), who describe Cost-reimbursable schemes (Cost plus percentage fee, cost plus fixed fee, cost plus incentive fee) and fixed-price schemes (firm fixed price and fixed price with incentives) and different equations for incentive schemes (Department of Energy, 1975, gives some early guidance). When to use which scheme is slightly less clear, although Martin and Webster (1986) attempt to relate the choice to uncertainty profiles. More recently, Chapman et al. (1988) discuss overall risk curves and how much cost risk ~o transfer, with guidelines as to what is feasible and rational.

The key to ensuring coverage of risks and making decisions on when and whether to transfer is often the Risk Register (Williams, 1993) described in Section 8. Busby (1992) similarly links contractual decisions to 'Centres of Risk Evaluation', and discusses the relationship between the 'level of criticality' and tl~e 'level of acceptability'. Perry (1986) proposes a list of factors to be taken into account on taking this decision, which covers the overall project environment, as well as specific points such as which party can best control the events leading to the risk, which party can best control the risk, whether the premium is likely to be reasonable, whether the transfer is likely to be able to Sustain the consequences of the risk etc. Further advice is given in Barnes (1983) (suggesting that the party who deems the risk to be lower pays, etc).

Further discussion on who bears the risk is given in South (1985), who makes a plea for insurance cover to remove the legal friction between the parties - especially for large risks, since "the insistence of companies bearing obli- gations which they will not in fact be able to meet is a theoretical comfort of no practical value" (Fraser, 1984). The legal position when certain risks occur is simplified by a 'force majeur' clause (see Srinivasau and Sassoon, 1982, for a definition in the international contracting domain). Ockman (1986) discusses what to do !when projects go wrong and claims for Delay and Disrup- tion.

30 T. Williams /European Journal of Operational Research 85 (1995) 18-38

8. Risk Management

Once the project starts, risk management needs to be an on-going process - risk analysis alone is not sufficient (as in, say, Ward and Chapman, 1991). This section looks at the role formal risk- management can (or should) play in project management.

Before that, the foundation must be laid in the relevant work on project management. There are many good Project Management handbooks and textbooks available, Ritz (1990) and Cleland (1988) being perhaps the most comprehensive and up-to-date. Project Management techniques are well-established (Gouse and Stickney, 1988, provide a history and overview of project management in the USA to the mid-1980's), and the Project Manager's job is recognized and established (Stallworthy and Kharbanda, 1987).

The basis of Project Management is the division of the project into parts, by task (i.e. the Work Breakdown Structure, or WBS - see La- void, 1988), by time (i.e. using PERT, as discussed above), and by cost (frequently also based on the WBS) (see Stoddart-Stones, 1988). Huot (1979) proposes that the WBS be used to integrate time, cost and performance measurement. Bent (1988), in introducing the concepts of control says that "the fundamental elements of control are the cost element and the project schedule" (as functions of the progress of the technical performance), although when Might (1984) carried out a survey of the effectiveness of project control systems on high-risk US (Department of Defense (DoD) and NASA) projects, the findings were unclear. Moves are being made to integrate the three aspects above into a single project control system, such as the DoD's C/SCS system (Farld and Karshenas, 1986). Vanesse (1986) rec- ognizes the need for, and makes a start on, a project management data-base including subpro- jects, activities/networks, resources, procurement/materials (see also comments by Tuman, 1988).

The Project Manager must also set up a project management infrastructure, and it is into this that a risk management structure must fit. There are three basic forms of management structure:

functional, project and (various forms of) matrix, the latter having received the most attention lately. Matrix management is described in an introduction by Adams (1984) and a full handbook by Cleland (1984). A full survey was carried out by Gobelli and Larson (1987a, b; Larson and Gobelli, 1989), who describe the three different forms of management structure and give literature surveys, and then describe their survey of 547 companies primarily concerned with development projects. Statistics were taken comparing project structure with whether cos(/time/technical targets were met; functional management (and the 'functional matrix') turned out worst on all of these aspects, and matrix structures (the balanced matrix and 'project matrix') best. Gray et al. (1990) and Might (1985) also describe similar surveys, the former with similar results (the latter's findings are rather unclear). Dinsmore (1984) discusses the particular suitability of matrix methods for superprojects ($1b plus): "The matrix within a project structure ... lends itself well to meeting such internal superproject requirements, Such a matrix gives organisational flexibility to the gigantic project that, if not bro- ken down into smaller operational groups, runs the risk of becoming a giant project bureaucracy".

To utilise some of these ideas for the management of risk, the Risk Register is often the start- ing-point (see Williams, 1993a). A full discussion of how the Risk Register can assist, and indeed is central to, the three types of analyses described above, and to two sets of plans: the contractual allocation of risk and the preparation of Risk Management Plans, is given in Williams (1993b). The first set, structured contractual preparation, is becoming more established (although Chapman et al. 1988, say that "the implied systematic approach to risk allocation and pricing is unusual in current practice"); Busby (1992) gives a good example, and such analysis is important for any company aiming to be a prime contractor (Brad- ley, 1989). However, the second set is also important, and Chapman et al. (1988) warn that "much of the current difficulty with risk allocation ... may arise because of contractual parties' preoc- cupation with transferring risk to other parties. As long as parties believe that risks can be trans-


ferred or offioaded onto someone else, perhaps inappropriately, then inadequate attention may be given to risk avoidance or reduction measures". These measures are generally termed 'risk reduction actions' and 'contingency plans' (e.g. Wil- liams, 1993a), although equivalent expressions are found (e.g. Alter and Ginzburg's, 1978, 'inhibiting strategies' and 'compensating strategies' (respec- tively)).

This analysis structure has to be embedded within a risk management infrastructure (Wil- liams, 1993a), which generally has to fit into the matrix organisation discussed above. Key to this is often some central body, called a 'Risk Man- agement Committee' (Williams, 1993a), or an 'overview team' (Fraser, 1984) or an 'Expert Management Unit' (Hirzel, 1986). This retains an overview of the whole project, and enables rational decision-making (e.g. Trade-off studies, Starr and Whipple, 1980). Morris (1988), in discussing integration across project subsystem interfaces, describes the use of liaison positions, or task- forces, or co-ordinators (each of which have different levels of authority). However, he points out that major projects usually start with a centralized structure, become decentralized, and end centralized (quoting as examples the infamous Trans-Alaskan Pipeline and the Acominas steel plant), and that during the decentralized phase, a large management superstructure is needed to maintain project integrity. (For a discussion of internal risk management roles, see Laufer, 1990, and Wearne, 1992).

Although no general structure is available for all projects, Busby (1992) provides some useful pointers, and Ireland and Shirley (1986) make some comments on an integrated risk management system. Charette (1989) describes in detail sets of pro-formas and reporting methods for software projects. Further than that, Charette also describes (and stresses the importance of) the general process of risk management throughout the project, as does Humphries (1989). This is particularly important in projects in which phases run in parallel or 'concurrently' (see literature quoted in Morris, 1988).

An important feature of managing risk is the maintenance of risk data-bases to retain knowl-

edge of where risks can occur. Data-bases, either formal or informal, do exist for specific domains, and are used for planning and tendering. Many of these, of course, are commercial-in-confidence, particularly in highly competitive industries such as the oil industry. Two examples of data-bases of project risks are Niwa and Okuma (1982), who describe a well-structured data-base (with a structure reminiscent of a Risk Register)in use at Hitachi, and its value of the data-base for knowledge-transfer on project risk; and Ashley (1987), who describes a number of examples of expert systems based on risk-knowledge. An article in Computing (1990) also describes some examples for software projects, particularly from Charette. (On risk expert systems, see also Seiler, 1990, Probst and Worlitzer, 1987, 1988, and Smith, 1987).

References

Abrahamson, M. (1989), "Risk problems relating to construction", in: J. Uff and P. Capper (eds.), Construction Con- tract Policy: Improved Procedures and Practice, Centre of Construction Law and Management, King's College, Lon- don, 1989, 21-35.

Acker, D.D. (1980), "The maturing of the DOD acquisition process", Defense Systems Management Review 3/3, 7-77.

Adams, J.R., and Kirckof, N.S. (1984), "The practice of matrix management", in: D.I. Cleland, Matrix Manage- ment Systems Handbook, Van Nostrand Reinhold, New York, 13-30.

Adams, J.R., and Martin, M.D. (1982), "A practical approach to the assessment of project uncertainty", in: Proceedings of the Project Management Institute, Toronto, Canada, IV-F 1-11.

Adlakha, V.G. (1986), "An improved conditional Monte Carlo technique for the stochastic shortest path problem", Man- agement Science 32/10, 1360-1367.

Adlakha, V.G., and Kulkarni, V.G. (1989), "A classified bibliography of research on stochastic PERT net~works: 1966- 1987", INFOR 27/3, pg 272-296.

Albino, V., Cavallone, S., and Mummolo, G. (1988), "A new approach in the engineering project analysis: The aggrega- tion game", International Journal of Project Management 6/1, 45-48.

Alter, S., and Ginzberg, M. (1978), "Managing uncertainty in MIS implementation", Sloan Management ReL,iew, Fall, 23-31.

Anklesaria, K.P., and Drezner, Z. (1986), "A multivariate approach to estimating the completion time for PERT networks", Journal of the Operational Research Society 37, 811-815.


Anti, B. (1980), Currency Risk and the Corporation, Eu- romoney Publications, London.

Arbogast, G.W., and Womer, N.K. (1988), "An error compo- nents model of cost overruns and schedule slip on army R&D programs", Naval Research Logistics 35, 367-382.

Arditi, D., Akan, G.T., and Gurdamar, S. (1985), "Cost overruns in public projects", International Journal of Project Management 3/4, 218-224.

Ashley, D.B. (1987), "Knowledge-based approaches to construction project planning", in: New Trends in Project Management, Proc. 11th International Expert Seminar, April 1987, International Project Management Association Zurich, 163-188.

Ashley, D.B., and Avots, I. (1984), "Influence diagramming for analysis of project risk", Project Management Journal 15/1, 56-62.

Assad, A.A., and Wasil, E.A. (1986), "Project Management using a microcomputer", Computers & Operations Research 13, 231-260.

Assad, A.A., and Wasil, E.A. (1988), "Project management on the PC. Software, applications and trends", Interfaces 18/2, 75-84.

Avots, I. (1984), "Information systems for matrix organisa- tions", in: D.I. Cleland (ed.), Matrix Management Systems Handbook, Van Nostrand Reinhold, New York.

Avots, I. (1986) "The coming impact of artificial intelligence on project management", in: M.C. McGrool, J. Vister, W.J. Vriethoff and G. Wijnen, Project Management in Progress - Tools and Strategies for the 90s, North-Holland, Amsterdam, 1986.

Avramidis, A.N., Bauer, K.W., and Wilson, J.R. (1991), "Simulation of stochastic activity networks using path control variates", Naval Research Logistics 38/2, 183-201.

Badiru, A.B. (1991), "A simulation approach to PERT network analysis", Simulation 57/4, 245-255.

Baker, B.N., Fisher, D., and Murphy, D.C. (1988), "Project management in the public sector: Success and failure patterns compared to private sector projecs", in: D.I. Cleland and W.R. King (eds.), Project Management Hand- book, 2nd ed., Van Nostrand Reinhold, New York, 1988, 920-934.

Baker, B.N., Murphy, D.C., and Fisher, D. (1988), "Factors affecting project success", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nos- trand Reinhold, New York, 1988, 902-919.

Baker, R.W. (1986) "Handling uncertainty", International Journal of Project Management 4/4, 205-210.

Balthazor, L.R. (1987), "So you think you know where you are? A review of some techniques for evaluating and predicting development schedules", Paper given at 'Devel- opment Time Scales: Their Estimation and control', Royal Aeronautical Society Symposium, 12th Feb., 1987.

Barnes, N.M.L. (1983), "How to allocate risks in construction contracts", International Journal of Project Management 1, 24-28.

Baum, W.C., and Tolbert, S.M. (1985), Investing in Develop- ment, Oxford University Press.

Bent, J.A. (1988), "Project control: An introduction", in: D.I.

Cleland and W.R. King (eds.), Project Management Hand- book, 2nd ed., Van Nostrand Reinhold, New York, 1988, 559-596.

Berny, J. (1989), "A new distribution function for risk analysis", Journal of the Operational Research Society 40, 1121- 1127.

Bettis, R.A., and Thomas, H. (eds.) (1990), Risk, Strategy and Management, Vol. 5 in Strategic Management Policy and Planning, JAI Press, Greenwich, CT.

Bhola, B., Cooke, R.M., Blaauw, H.G., and Kok, M. (1992), "Expert opinion in project management", European Jour- nal of Operational Research 57, 24-31.

Blankevort, P.J. (1987), "Project Management tools and organization concepts (workshop report by' the Moderator)", in: New Trends in Project Management, Proc. 11th Interna- tional Expert Seminar, April 1987, International Project Management Association Zurich, 117-119.

Bowers, J. (1985), "Network analysis on a micro", Journal of the Operational Research Society 36/7, 609-611.

Bowers, J. (1988), "Assessing risk in major projects", in: Proceedings of the International Society of Parametric Ana- lysts, 10th annual conference, Brighton, 13-26.

Bowers, J. (1989), "Risk analysis applied to project schedule forecasting", in: Project Risk Analysis in the Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989.

Brabson, G.D. (1983), "The defence acquisition improvement program", Program Manager, Nov/Dec, 5-13.

Bradley, R.A. (1989), "The application of risk analysis techniques to support the management decision process", in: Project Risk Analysis in the Aerospace Industry, Proceedings of the Conference of the R. Ae. Soc., 8th March, 1989.

Bradley, R.M., Powell, M.G., and Soulsby, M.R. (1990), "Quantifying variations in project-cost estimates", Journal of Management Engineering 6, 99-106.

Brunies, R.A., and Brophy, R. (1986), "Minimizing construction claims under the project management concept", in: Measuring Success, Proceedings of the 18th Annual Seminar /Symposium of the Project Management Institute, Mon- treal, Canada, September 1986, 198-212.

Bryson, L. (1982), "Large-scale project management", Pro- ceedings of the IEE. Part A 129, 625-633.

Bryson, L. (1986), "All at sea", Inaugural address as Presi- dent, Proceedings of the IEE. Part A 133/1, 1-15.

Bunyard, J.M. (1982), "Today's risks in software development - Can they be significantly reduced?" Concepts 5/4, 73-94.

Burke, C.M., and Ward, S.C. (1988), "Project Appraisal - Finance approaches to risk", in: N.B. Cook and A.M. Johnson (eds.), Developments in Operational Research 1988, 45 -70.

Busby, I.C. (1992) "An approach to strategic risk management for defence contractors", Paper given at the Defence Manufacturers' Association seminar on The Analysis and Management of Risk in Defence Products, 28th April, 1992.

Chapman, C.B. (1989), "Future developments in risk analysis techniques", in: Project Risk Analysis in the Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989.

Chapman, C.B. (1990), "Risk analysis, a view from 1990",


IMA Journal of Mathematics. Applications in Business and Industry 2/2, 79-95.

Chapman, C.B., and Cooper, D.F. (1985), "Risk analysis", in: G.K. Rand and R.W. Eglese (eds.), Further Developments in Operational Research, Pergamon Press, Oxford.

Chapman, C.B., Phillips, E.D., Cooper, D.F., and Lightfoot, L. (1985), "Selecting an approach to project time and cost planning", International Journal of Project Management 3.

Chapman, C.B., Ward, S.C., and Curtis, B. (1989), "Risk theory for contracting", in: J. Uff and P. Capper (eds.), Construction Contract Policy: Improved Procedures and Practice, Centre of Construction Law and Management, King's College, London, 1989, 74-109.

Charette, R. (1989), Software Engineering Risk Analysis and Management, McGraw-Hill, New York.

Chief scientific advisor (undated), "CSA guidelines for technical scrutiny", Ministry of Defence.

Chilstrom, K.O. (1988), "Project needs and techniques for management audits", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nos- trand Reinhold, New York, 1988, 620-637.

Clark, P., and Chapman, C.B. (1987), "The development of computer software for risk analysis: A decision support system development study", European Journal of Opera- tional Research 29, 252-261.

Clark, R.C., Pledger, M., and Needler, H.M.J. (1989), "Risk analysis in the evaluation of non-aerospace projects", in: Project Risk Analysis in the Aerospace Industry, Proceedings of Conference of the R.Ae.Soc., 8th March, 1989.

Clark, R.C., Pledger, M., and Needler, H.N.J. (1990), "Risk analysis in the evaluation of non-aerospace projects", In- ternational Journal of Project Management 8/1, 17-24.

Cleland, D.I. (1984), Matrix Management Systems Handbook, Van Nostrand Reinhold, New York.

Cleland, D.I. and King W.R. (1988), "Project owner strategic management of projects", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nos- trand Reinhold, New York, 1988, 165-188.

Cleland, D.I. and King W.R. (1988), Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988.

Coats, P.K., and Chesser, D.C. (1982), "Coping with business risk through probabilistic financial statements", Simulation 38, 111-121.

Cockerham, J.M. (1979), "Army RTDE Funding, planning and management: 'Total Risk Assessing Cost Estimate (TRACE)' concept", Army Research Development and Ac- quisition Magazine, Jul, 24-26.

Computing (1990), "Getting to grips with IT failure before the event", Computing, 29th Nov, 28-29.

Cooper, D.F., and Chapman, C.B. (1985), "Risk analysis of a construction cost estimate", Project Management 3/3, 141-149.

Cooper, D.F., and Chapman, C.B. (1986), Risk Analysis for Large Projects: Models, Methods and Cases, Wiley, New York.

Curtis, B., Ward, S., and Chapman, C. (1991), "Roles, respon- sibilities and risks in management contracting", Construc- tion Industry Research and Information Association (CIRIA) Special Publication 81.

Davies, R.M.G., and Saunders, R.G. (1988), "Applying systems theory to project management problems", Interna- tional Journal of Project Management 6/1, 19-26.

Department of Energy, Peat Marwick Mitchell and Atkins Planning (1975), "North Sea costs escalation study", HMSO.

de Wit, A. (1987) "New methods of estimating and controlling project costs" in: New Trends in Project Management, Proc. llth International Expert Seminar, April 1987, Interna- tional Project Management Association Zurich, 219-226.

de Wit, A. (1986), "Measuring project success: An illusion", in: Measuring Success, Proceedings of the18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 13-21.

Devroye, L.P. (1979), "Inequalities for the completion times of stochastic PERT networks", Mathematics of Operera- tional Research 4, 441-447.

Dietrich, J.A. (undated), "The Expert Project Manager: XPM", Expert Management Systems Inc, 'Phoenix, AZ 85029.

Dingle, J. (1991), "Cultural issues in the planning and development of major projects", International Journal of Project Management 9/1, 29-33.

Dinsmore, P.C. (1984), "Whys and why-nots of matrix management", in: D.I. Cleland, Matrix Management Systems Handbook, Van Nostrand Reinhold, New YOrk, 394-411.

Dodin, B. (1985), "Bounding the project completion time distribution in PERT networks" Operations Research 33, 862-881.

Dodin, B., and Elmaghraby, S.E. (1985), "Approximating the criticality indices of the activities in PERT networks", Management Science 31, 207-223.

Dodin, B., and Sirvanci, M. (1990), "Stochastic networks and the extreme value distribution", Computers & Operations Research 17/4, 397-409.

Eisenhardt, K.M. (1989), "Agency theory: An assessment and review", Academic Management Reviews 14/1, 57-74.

Elmaghraby, S.E. (1990), "Project bidding under deterministic and probabibilistic activity durations", European Journal of Operational Research 49, 14-34.

Farid, F., and Karshenas, S. (1986), "C/SCS Criteria for measuring progress under inflation", in: Measuring Suc- cess, Proceedings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 139-144.

Farnum, N.R., and Stanton, L.W. (1987), "Some results con- cerning the estimation of beta distribution parameters in PERT", Journal of the Operational Research Society 38/3, 287-290.

Fishburn, P.C. (1984), "Foundations of risk measurement. 1: Risk as probable loss", Management Science 30/4, 396- 406.


Fox, J.R. (1984), "Evaluating management of large, complex projects: A framework for analysis", Technology in Society 6/2, 129-139.

Franke, A. (1987), "Risk analysis in project management", International Journal of Project Management 5/1, 29-34.

Fraser, D.C. (1984), "An approach to major projects", Major Projects Association, Templeton College, Oxford, 1984.

Gabriel, E. (1987), "Analysis of the future needs and software producers' response (workshop report by the moderator)", in: New Trends in Project Management, Proc. 11th Interna- tional Expert Seminar, April 1987, International Project Management Association Zurich, 19-27.

Gehring, H., and Narula, S. (1986), "Project cost planning with qualitative information,", International Journal of Project Management 4/2, 61-66.

General Accounting Office (1982), "Cost growth and delivery delays in submarine construction and electric boat are likely to continue", General Accounting Office, Washing- ton DC; MASAD-82-29, 19th April 1982.

General Accounting Office (1983), "Status of major acquisi- tions as of September 30th 1982", General Accounting Office, Washington DC; GAO/NSIAD-83-32, 7th Sep- tember 1983.

Gilmore, M.L. (1989), "Knowledge-based systems in construction and civil engineering", International Journal of Project Management 7/3, 147-153.

Gobeli, D.H., and Larson, E.W. (1987), "Relative effectiveness of different project structures", Project Management Journal 18/2, 81-85.

Gobeli, D.H., and Larson, E.W. (1987), "Project structures versus project performance", in: New Trends in Project Management, Proc. 11th International Expert Seminar, April 1987, International Project Management Association Zurich, 49-72.

Golenko-Ginzburg, D. (1988), "Controlled alternative activity networks for project management", European Journal of Operational Research 37, 336-346.

Golenko-Ginzburg, D. (1989), "The activity-time distribution in PERT", Journal of the Operational Research Society 40/4, 389-393.

Gouse, M.K. and Stickney, F.A. (1988), "Overview of project management applications", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988, 869-901.

Gray, C., Dworatschek, S., Gobeli, D., Knoepfel, H., and Larson, E.W. (1990), "International comparison of project organization structures: Use and effectiveness", Interna- tional Journal of Project Management 8/1, 26-32.

Gutierrez, G.J., and Kouvelis, P. (1991), "Parkinson's law and its implications for project management", Erratum in Management Science 37/11, 1507, Management Science 37/8, 990-1001.

Habison, R. (1985), "Time and cost securing in complex large-scale projects", Project Management 3/3, 178-181.

Hackney, J.W. (1986), "Contingency allowances and overrun probabilities", in: Proceedings of the 9th international cost engineering congress Oslo, Norway, Paper B3.

Hackney, J.W. (1985), "Applied contingency analysis", Trans- actions of the American Association of Cost Engineers B2.

Hadzi-Pavlovic, V., Bissett, B., and Razdan, K. (1986), "Success or failure? - The answer should not be in the eye of the beholder" in: Measuring Success, Proceedings of the 18th Annual Seminar~Symposium of the Project Man- agement Institute, Montreal, Canada, September 1986, 29- 34.

Hall, J.N. (1986), "Use of risk analysis in North Sea projects", International Journal of Project Management 4/4, 217-222.

Harris, R. (1987), "Project Management Systems - Past developments, future trends, user requirements", in: New Trends in Project Management, Proc. llth International Expert Seminar, April 1987, International Project Manage- ment Association Zurich, 11-18.

Hayes, R. (1987), "An introduction to project planning and costing risk management", in: Risk Analysis and Manage- ment, Conference Friday 13th March 1987, Chartered Institute of Management Accountants and the Opera- tional Research Society.

Hazelrigg, G.A., and Husband, F.L. (1985), "RADSIM - A methodology for large-scale R&D program assessment", IEEE Transactions of Engineering Management 32/3, 106- 115.

Herten, H.J., and Peeters, W.A.R. (1986), "Incentive contracting as a project management tool," International Jour- nal of Project Management 4/1, 34-39.

Hertz, D.B., and Thomas, H. (1983), "Decisions and risk analysis in new product and facilities planning problems", Sloan Management Review, Winter, 17-31.

Hirzel, M. (1986), "Holistic decision-making organization in project management", International Journal of Project Management 4/1, 14-16.

Ho, S.S.M., and Pike, R.H. (1992), "The use of risk analysis techniques in capital investment appraisal" in: J. Ansell and F. Wharton (eds.), Risk: Analysis, Assessment and Management, Wiley, New York 71-94.

Hodder, J.E., and Riggs, H.E. (1985), "Pitfalls in evaluating risky projects", Harvard Business Review, ed. 63/1, 128- 135.

Hopkins, D. (1987), "Planning for risk in defence development projects", Paper given at: "Development time scales: Their estimation and control", Royal Aeronautical Society Symposium, 12th Feb, 1987.

House of Commons (1988), "The procurement of major defence equipment. 5th report, Session 1987-1988", HMSO.

House of Commons (1991), "Procurement of Upholder class submarines", Ninth Report of the House of Commons Defence Committee Defence Committee, 17th July 1991, HMSO.

Hughes, M.W. (1986), "Why projects fail: The effects of ignoring the obvious", Industrial Engineering 18/4, 14-18.

Hull, J.C. (1980), The Evaluation of Risk in Business Invest- ment, Pergamon, Oxford.

Hull, J.K. (1989), "The application of risk analysis techniques in proposal assessment", in: Project Risk Analysis in the

T. l¥illiams / European Journal of Operational Research 85 (1995) 18-38 35

Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989.

Hull, J.K. (1990), "Application of risk analysis techniques in proposal assessment", International Journal of Project Management 8/3, 152-157.

Hull, K. (1992), "Risk analysis techniques in defence procurement", Prepared by MOD (AEPS), and distributed by the EEA (Annex C ER&M/1/92) .

Humphries, D.E. (1989), "Project risk analysis in the aerospace industry", in: Project Risk Analysis in the Aerospace Indus- try, Proc. Royal Aeronautical Society Conf., 8th March, 1989.

Humphries, P. (1989), "Risk analysis tools and techniques in project management", in: Project Risk Analysis in the Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989.

Huot, J.C. (1979), "Integration of cost and time with the work breakdown structure", Transactions of the American Asso- ciation of Cost Engineers H, 3, 1-6.

Ilumoka, A.A. (1987), "A tolerance analysis approach to network scheduling for engineering project management", International Journal of Production Research 25/4, 531- 547.

In't veld, J., and Peeters, J. (1989), "Keeping large projects under control: the importance of contract type selection", Project Management 7/3, 155-162.

Ireland, L.R., and Shirley, V.D. (1986), "Measuring risk in the project environment", in: Measuring Success, Proceedings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 150-156.

Jackson, N., and Carter, P. (1992), "The perception of risk", in: J. Ansell and F. Wharton (eds.), Risk: Analysis, Assess- ment and Management, Wiley, New York, 55-70.

Jaynes, G.L. (1985), "Can the contract help?" in: Manage- ment of International Construction Projects, Proc. Conf. Inst. Cir. Eng., London, Nov. 1984, Thomas Telford, Lon- don, 85-94.

Kahneman, D., Slovic, P., and Tversky, A. (1986), Judgement under Uncertainty: Heuristics and Biases, Cambridge Uni- versity Press, Cambridge.

Kahneman, D., and Tuersky, A. (1979), "Prospect theory: An analysis of decision under risk", Econometrica 47, 263-291.

Kahn, S.G. (1981), "Partners in risk. Cost incentive development contracts", ESA Bulletin 26, 48-53.

Kamburowski, J. (1986), "Normally, distributed activity durations in PERT networks", Journal of the Operational Re- search Society 36/11, 1051-1057.

Kamburowski, J. (1985), "An upper bound on the expected delay of completion time of PERT networks", European Journal of Operational Research 21/2, 206-212.

Kangari, R., and Riggs, L.S. (1989), "Construction risk assessment by linguistics", 1EEE Transactions on Engineering Management 36/23, 126-131.

Kaplan, S., and Garrick, B.J. (1984), "On the quantitative definition of risk", Risk Analysis 1, 11-28.

Keeney, R.L., and Winterfeldt, D.V. (1989), "On the uses of

expert judgement on complex technical problems", IEEE Transactions on Engineering Management 36/2, 83-86.

Kidd, J. (1987), "A comparison between the VERT program and other methods of Project duration estimation", OMEGA 15/2, 129-134.

Kidd, J. (1991), "Do today's projects need powerful network planning tools?", International Journal of Production Re- search 29/10, 1969-1978 (see also similar work in OMEGA 17/2 (1989) 103-111, and International Journal of Project Management 8/2 (1990) 109-115).

Knoepfel, H. (1990), "Cost and quality control in the project cycle", International Journal of Project Management 7/4, 229-235.

Knoepfel, H., and Burger, R. (1987), "Project organization and contract management", in: New Trends in Project Management, Proc. l l th International Expert Seminar, April 1987, International Project Management Association Zurich, 89-98.

Kohrs, R.H., and Welngarten, G.C. (1986), "Measuring successful technical performance - A cost/schedule/technical control system", in: Measuring Success, Proceedings of the 18th Annual Seminar/Symposium of the Project Man- agement Institute, Montreal, Canada, September 1986, 158-164.

Kulkarni, V.G., and Adlakha, V.G. (1986), "Markov and Markov-regenerative PERT networks", Operations Re- search 34/5, 769-781.

Larson, E.W., and Gobeli, D.H. (1989), "Significance of project management structure on development success", IEEE Transactions on Engineering Management 36,/2, 119-125.

Laufer, A. (1990), "Decision-making roles in project planning", Journal of Management Engineering 6, 416-430.

Lavold, G.D. (1988), "Developing and using the Work Break- down Structure", in: D.I. Cleland and W.R; King (eds.) Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988, 302-323.

Levine, H.A. (1988), "Computers in project management", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988, 692-735.

Levitt, R.E., and Kunz, J.C. (1985), "Using knowledge of construction and project management for automated schedule updating", International Journal of Project Man- agement 16/5, 57-76.

Littlefield, T.K., and Randolph, P.H. (1991), "PERT duration times: Mathematics or MBO", Interface 21/6, 92-95.

Maccrimmon, K.R., and Wehrung, D.A. (1984), "The risk in-basket", Journal of Business 57, 367-387.

Maccrimmon, K.R., and Wehrung, D.A. (1986), Taking Risks: The Management of Uncertainty, Free Press, New York.

March, J.G., and Shapira, Z. (1987), "Managerial perspectives on risk and risk taking", Management Science 33/11, 1404-1418.

Marshall, A.W. and Meckling, W.H. (1959), "Predictability of the costs, time and success of development", Rand Corpo- ration Report P-1821.

Martin, M.D. and Webster, F.M. (1986), "Contract type and


the measurement of project success", in: Measuring Suc- cess, Proceedings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 165-172.

McCoy, F.A. (1986), "Measuring success: Establishing and maintaining a baseline", in: Measuring Success, Proceed- ings of the 18th Annual Seminar/Symposium of the Project Management Institute, Montreal, Canada, September 1986, 47-52.

Merkhover, M.W. (1987), "Quantifying judgemental uncertainty: Methodology, experiences and insights", IEE Transactions on Systems, Management, and Cybernetics' 17/5, 741'52.

Merrow, E.W. (1988), "Understanding the outcomes of megaprojects", RAND/R-3560-PSSP Rand, Santa Mon- ica CA, March 1988.

Might, R. (1984), "An evaluation of the effectiveness of project control systems", IEEE Transactions on Engineer- ing Management 31/3, 127-137.

Might, R.J., and Fischer, W.A. (1985), "The role of structural factors in determining project management success", IEEE Transactions on Engineering Management 32/2, 71-77.

Ministry of Technology (1969), "Report of the steering group on development cost estimating" (the Downey Report), HMSO.

Moder, J. (1988), "Network techniques in project management", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988, 324-373.

Morgan, D.G. (1989), "Risk analysis in the evaluation of major software projects", in: Project Risk Analysis in the Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989; paper published separately.

Morrison, D. (1985), "The new breed of project management", Computers and Decisions' 17/15, 32-36.

Morris, P.W.G. (1986), "Research at Oxford into the precon- ditions of success and failure of major projects", in: Mea- suring Success, Proceedings of the 18th Annual Seminar/ Symposium of the Project Management Institute, Montreal Canada, September 1986, 53-66.

Morris, P.W.G. (1988), "Managing project interfaces - Key points for project success", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nos- trand Reinhold, New York, 1988, 16-55.

Morris, P.W.G. (1989), "Initiating major projects: The unper- ceived role of project management", International Journal of Project Management 7/3, 180-185.

Morris, P.W.G. and Hough, G.H. (1987), The Anatomy of Major Projects: A Study of the Reality of Project Manage- ment, Wiley, New York.

Myers, C.W., and Devey, M.R. (1984), "How management can affect project outcomes: An exploration of the PPS database", Rand Corporation N-2196-SFC.

Meridian (1981), "Preliminary analysis of technical risk and cost uncertainty in selected DARPA programs", Meridian

report for DARPA, DARPA order no. 4287, Meridian Corp, Falls Church, VA.

Nahapiet, H., and Nahapiet, J. (1985), "The management of construction projects. Case studies from the USA and UK", The Chartered Institute of Building, London.

National Audit Office (1985), "Ministry of Defence: Produc- tion costs of defence equipments in non-competitive contracts", Report by the Comptroller and Auditor General 505.

National Audit Office (1986), "Ministry of Defence: Control and Management of the development of major equipments", Report by the Comptroller and Auditor General 568, HMSO, 26th July, 86.

NEDO (1982), "Target cost contracts: A worthwhile alternative", National Economic Development Office, HMSO.

Niwa, K., and Okuma, M. (1982), "Know-how transfer method and its application to risk management for large construction projects", IEEE Transactions on Engineering Manage- ment 29/4, 146-153.

Noonan, F., and Thamhain, H. (1986), "Identifying and assessing risks for projects", in: Measuring Success, Proceed- ings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 173-176.

Oakes, M. (1986), Statistical Inference: A Commentary for the Social and Behavioural Sciences, Wiley, New York.

Ockman, S. (1986), "Measuring success with claims management", in: Proceedings of the 18th Annual Seminar/ Symposium of the Project Management Institute, Montreal, Canada, September 1986, 266-273.

Perac (undated), "Guidelines for risk assessment and risk management in MOD procurement programmes", Min- istry of Defence.

Perry, J.G. (1986), "Risk management - An approach for project managers", International Journal of Project Man- agement 4/4, 211-216.

Perry, J.G., and Hayes, R.W. (1985), "Risk and its management in construction projects", Proceedings of the Institute of Ovil Engineers 78/1,499-521.

Perry, J.G., and Hayes, R.W. (1985), "Construction projects - know the risks", Chartered Mechanical Engineers 32/2, 42-45.

Perry, J.G., and Thompson, P.A. (1982), "Target and cost-reimbursable construction contracts. Parts A: A study of their use and implications", Construction Industry Re- search and Information Association (CIRIA) Special Pub- lication 85 Part A.

Pike, R.H., and Ho, S.S.M. (1991), "Risk analysis in capital budgeting: Barriers and benefits", OMEGA 19/4, 235-245.

Poiaga, L. (1989), "Operations management in project management and cost engineering: An outlook for new operational developments", European Journal of Operational Research 41/1, 1-14.

Potter, S. (1987), "On the right lines? The limits of technological innovation", Frances Pinter, London.


Probst, A.R., and Worlitzer, J. (1987), "Project management and expert systems", in: New Trends in Project Manage- ment, Proc. llth International Expert Seminar, April 1987, International Project Management Association Zurich, 139-162.

Probst, A.R. and Worlitzer, J. (1988), "Project management and expert systems", International Journal of Project Man- agement 6/1, 11-17.

Project Management Institute (1986), Measuring Success, Pro- ceedings of the 18th Annual Seminar/Symposium of the Project Management Institute, Montreal, Canada, Septem- ber 1986.

Pugh, L.A., and Soden, R.G. (1986), "Use of risk analysis techniques in assessing the confidence of project cost estimation and scheduling", International Journal of Pro- ject Management 4/3, 158-162.

Pugh, P. (1987), "Where does the time go? A review of development time scales for aircraft and guided weapons", Paper given at: "Development time scales: Their estimation and control", Royal Aeronautical Society Symposium, 12th Feb., 1987.

Pugh, P., Acord, A.G., and Ball, D. (1989), "Who can tell what might happen? A review of methods for risk assessment", in: Project Risk Analysis in the Aerospace Industry, Proc. Conf. R. Ae. Soc., 8th March, 1989.

Ragsdale, C. (1989), "The current state of network simulation in project management theory and practice, OMEGA 17/1, 21-25.

Ritchie, E. (1985), "Network base planning techniques: A critical review of published development", in: G.K. Rand and R.W. Eglise (eds.), Further Developments in Opera- tional Research, Pergamon, Press, Oxford.

Ritz, G.J. (1990), Total Engineering Project Management, Mc- Graw-Hill, New York.

Roberts, C.J.U. (1987), "Experiences of programme planning and management from the customer's viewpoint", Paper given at: "Development time scales: Their estimation and control", Royal Aeronautical Society Symposium, 12th Feb. 1987.

Robinson, J. (1987), "Comparison of tendering procedures and contractual arrangements", International Journal of Project Management 5/1, 19-24.

Ruskin, A.M., and Lerner, R. (1972), "Forecasting costs and completion dates for defense research and development contracts", IEEE Transactions on Engineering Management 19/4, 128-33.

Russell, R.A. (1986), "A comparison of heuristics for scheduling projects with cash flows and resource restrictions", Management Science 32, 1291-1300.

Salapatas, J.N., and Sawte, W.S. (1986), "Measuring success of utility projects past, present and future", in: Measuring Success, Proceedings of the 18th Annual Seminar~Sym- posium of the Project Management Institute, Montreal, Canada, September 1986, 67-76.

Sathi, A., Morton, T.E., and Roth, S.F. (1986), "Callisto: An intelligent project management system", AI Magazine, Winter, 34-51

Saunders, R.G. (1990), "Project management in R&D: The Art of estimating development project activities", Interna- tional Journal of Project Management 8/1, 45-50.

Schtub, A. (1986), "The trade-off between the net present cost of a project and the probability to complete it on schedule", Journal of Operations Management 6/4, 461- 470.

Seiler, R.K. (1990), "Reasoning about uncertainty in certain expert systems: Implications for project management", In- ternational Journal of Project Management 8 / 1, 51-59.

Shafer, G. (1976), A Mathematical Theory of Evidence, Prince- ton University Press, Princeton, NY.

Shapiro, A., and Titman, S. (1990), "A integrated approach to corporate risk management", in: J.M. Stern and D.H. Chew (eds.) The Revolution in Corporate Finance, 215-229.

Smith-Daniels, D.E., and Aquilano, N.J. (1987), ~'Using a late start resource-constrained project schedule to improve project net present value", Decision Sciences 18, 617-630.

Smith, J.U.M. (1987), "Expert systems in project management", International Journal of Project Management 5/1, 7-10.

Smith, L.A., and Mandakovic, T. (1985), "Estimating: The input into good project planning", IEEE Transactions on Engineering Management 32/4, 181-185.

South, A.J. (1985), "Risk analysis and allocation", in: Man- agement of International Construction Projects, Proceedings of the Conference Institute of Civil Engineers, London, Nov. 1984, Thomas Teiford, London, 15-22.

Srinivasu, R., and Sassoon, D.M. (1982), "International contracting and procurement for development projects", In- ternational Law Institute, Georgetown University Law Centre, Washington, DC.

Stallworthy, E.A., and Kharbanda, O.P. (1987), "The project manager in the 1990s", International Journal of Project Mangement 7/5, 3-67.

Starr, C., and Whipple, C. (1980), "Risks of risk decisions", Science 208, 1114-1119.

Statman, M., and Tyebjee, T.T. (1984), "The risk of investment in technological innovation", IEEE Transactions on Engineering Management 31/4, 165-171.

Steiner, G.A. (1969), "Top management planning", Macmil- lan, New York; quoted on 481 of D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nostrand Reinhold, New York, 1988.

Stoddart-Stones, R. (1988), "Development of project management systems for major projects", International Journal of Project Management 6/1, 34-38.

Stringer, J. (1987), "Risk and uncertainty in the formulation of major projects", in: Risk Analysis and Management, Conference Chartered Inst. Mgmt. Acc. and Opl. Res. Soc., Friday 13th March 1987.

Stringer, J. (1992), "Risk in large projects", Tutorial Papers from Operations Research Conference 1992; Operational Research Society, Birmingham, 29-46.

Sykes, A. (1982), "Reducing neglected risks on giant projects", in: A.J. Kelley (ed.), New Dimensions of Project Manage- ment", Lexington books, Lexington, MA.

38 T. Williams ~European Journal of Operational Research 85 (1995) 18-38

Telfer, P.J. (1989), "Risk analysis during project definition", in: Project Risk Analysis in the Aerospace Industry Proc. Conf. R.Ae.Soc., 8th March 1989.

Thamhain, H.J., and Wilemon, D.L. (1986), "Computer-aided project monitoring is coming of age", in: Measuring Suc- cess, Proceedings of the 18th Annual Seminar ~Symposium of the Project Management Institute, Montreal, Canada, September 1986, 446-453.

Tiong, R.L.K. (1990), "Comparative study of BOT projects", Journal of Management Engineering 6, 107-122.

Treasury (undated), "Green Book Guide", Treasury, 57-64. Tuman, J. (1986), "Success modeling: A technique for build-

ing a winning project team", in: Measuring Success, Pro- ceedings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, Septem- ber 1986, 94-108.

Tuman, J. (1988), "Development and implementation of project management systems", in: D.I. Cleland and W.R. King (eds.), Project Management Handbook, 2nd ed., Van Nos- trand Reinhold, New York, 1988, 652-691.

Turner, J.R., Grude, K.V., Haug, T., and Andersen, E.S. (1988), "Corporate development: Balancing changes to people, systems and organization", International Journal of Project Management 6/1, 27-32.

Tversky, A., and Kahneman, D. (1981), "The framing of decisions and the psychology of choice", Science 211, 453-458.

Uff, J., and Capper, P. (1989), "Construction contract policy: Improved procedures and practice", Centre of Construc- tion Law and Management, King's College, London, 1989.

Vanasse, J.R. (1986), "Integrating project and organizational information with relational data base management system", in: Measuring Success, Proceedings of the 18th Annual Seminar~Symposium of the Project Management Institute, Montreal, Canada, September, 1986, 454-462.

Ward, S.C., and Chapman, C.B. (1991), "Extending the use of risk analysis in project management", International Journal of Project Management 9/2, 117-123.

Wasil E.A., and Assad, A.A. (1988), "Project management on the PC: Software, applications and trends", Interfaces 18, 75-84.

Wearne, S.H. (1992), "Contract administration and project risks", International Journal of Project Management 10/1, 39-41.

Wharton, F. (1992), "Risk management: Basic concepts and general principles", in: J. Ansell and F. Wharton (eds.),

Risk: Analysis, Assessment and Management, Wiley, New York, 1-14.

White, W. (1982), "A risk/action model for the differentia- tions of R and D profiles", IEEE Transactions on Engi- neering Management 29/3, 88-93.

Williams, T.M. (1985), "Risk analysis using an embedded PERT package", YARD Internal Memorandum No. 503, YARD Ltd, Glasgow.

Williams, T.M. (1990), "Risk analysis using an embedded CPA package", International Journal of Project Manage- ment 8, 84-88.

Williams, T.M. (1992), "Criticality in stochastic networks", Journal of the Operational Research Society 43/4, 353-357.

Williams, T.M. (1992), "Practical use of distributions in network analysis", Journal of the Operational Research Society 43/3, 265-270.

Williams, T.M. (1993), "Risk management infrastructures", International Journal of Project Management 11 / 1, 5-10.

Williams, T.M. (1993), "Using the risk register to integrate risk management in project definition", International Jour- nal of Project Management 12, 17-22.

Wit, J.D., and Herroelen, W. (1990), "An evaluation of microcomputer-based software packages for project management", European Journal of Operational Research 49, 102- 139.

Wolff, J.G. (1989), "The management of risk in system development: 'Project SP' and the 'New Spiral Model"', Soft- ware Engineering Journal, 4, 481-491.

Wollmer, R.D. (1985), "Critical path planning under uncertainty", Mathematical Programming Studies 25, 164-171.

World Bank (1985), "Tenth annual review of project performance audit results 1984", World Bank, 1818 High Street NW, Washington, DC 20433.

Wright, G., and Ayton, P. (1987), Judgemental Forecasting, Wiley, New York.

Wynne, B. (1992), "Science and social responsibility", in: J. Ansell and F. Wharton (eds.), Risk: Analysis, Assessment and Management, Wiley, New York, 137-152.

Yeo, K.T. (1990), "Risks, classification of estimates and contingency management", Journal of Management Engineer- ing 6, 458-70.

Zelany, M. (1979), "Intuition - Its failures and merits", in: B. Persson (ed.), Surviving Failures. Patterns and Cases of Project Mismanagement, Almqvist & Wiksell International, Stockholm, 172-183.

A classified bibliography of recent research relating to project risk management

Documents

Transcript of A classified bibliography of recent research relating to project risk management