· Web viewApplied Information Economics ______________ The AIE Reference Manual. Version 3.1....

Le contexte d’utilisation de RAVI.

Compliance

Strategic

Economic

FF. 0.1 m

FF 1m

FF. 10m

FF. 100 m

Size of the initial investment

Development

costs

Development

and deployment

costs

Applied Information Economics

______________

The AIE Reference Manual

Version 3.1

DRAFT

TABLE OF CONTENTS

Introduction to the AIE MethodPage 7

The AIE Procedure:

1 Describe & Classify The IT investmentPage 11

2 Clarify The Decision ModelPage 15

3 Measure VariablesPage 23

4 Conduct Value of Information Analysis (VIA)Page 35

5 Conduct Risk & Return AnalysisPage 39

6 Make RecommendationPage 41

Appendixes

A. Custom DataPage 47

B. AIE Assessment Project PlanningPage 49

C. Workshop Guidelines Page 57

D. AIE Assessment Document Templates Page 59

E. Intangibles Checklist Page 71

F. Spreadsheet Templates Page 75

G. Equations Reference Page 77

H. Example Calibration Tests Page 79

I. Presentation Templates Page 83

J. Glossary Page 111

K. Bibliography Page 115

L. AIE Feedback Surveys Page 117

Introduction To The AIE Method

The AIE method (Applies Information Economics) is a set of tools, procedures and organizational structures to provide decision-makers with a quantitative evaluation of IT investments.

General objectives and scope

The aim of the Applied Information Economics (AIE) is to improve the IT decision process by providing decision-makers with sound economic evaluation tools specifically designed for the features of IT investments. The AIE method consists of a set of tools, procedures and organizational structures that can be used to generate and supply this decisionsupport data.

Features of IT investments

An “IT investment” is a mobilization or acquisition of information technology resources (hardware and software) and manpower for a limited period to alter the way that part of the enterprise operates. The benefits expected from this change will not accrue immediately, but over a fairly lengthy period of time. IT investments can be characterized by the following features:

· IT investments represent a growing portion of companies' investments, and IT resources are most often used by companies to develop strategic opportunities.

· IT benefits are increasingly difficult to measure since the improvements in productivity generated by automating manual clerical tasks have generally already been realized, and any further benefits are now likely to be made in more subtle “intangible” areas such as communication, information management or expert systems.

· IT investments are risky. Examples abound of partial failure (schedule and cost overruns) and complete failure (project cancellation). These common risks are rarely considered in the decision-making process.

AIE Benefits

Using the AIE method can help quantify the risks and intangible benefits of IT investments. The primary benefits AIE offers are:

· The risks can be stated in a manner consistent with other financial and actuarial measures

· The benefits can be quantified no matter how intangible they may seem.

· The AIE method can also go beyond the simple “yes or no” recommendation for investments. AIE can also identify specific implementation considerations to optimize the investment. Some considerations will include methods for managing risks or means for accelerating the accrual of benefits.

· Finally, some of the methods of AIE are applied to the AIE method itself. AIE calculates the value of additional information about an investment so that data-gathering efforts for IT investment analysis are optimized.

A Practical Method

The AIE method was designed specifically with practical implementation in mind. For example, although certain stages involve sophisticated concepts and statistical techniques, the method itself can be implemented by filling out easytouse spreadsheets.

The method has also been designed to fit in squarely with the set of IT reporting activities for companies.

AIE will be used by a variety of people. The hands-on users will adopt the method for producing decision-support documents, while the indirect users, i.e. the senior management of the company (executive management, business unit managers and IT managers) will base their decisions on these documents.

Realism and Flexibility

It should be possible to apply the proposed method in all companies. The method is therefore flexible enough to be tailored to a variety of companies for a wide range of IT investments.

This flexibility is provided through the customizable pieces of AIE. Some sections of AIE are “fill-in-the-blank” procedures that would vary among organizations.

Uses Of The AIE Method Over Time

The AIE method can be applied several times during the life of a particular project - e.g. at the end of the feasibility study to decide whether or not to proceed with detailed studies, at the end of the detailed studies to start up development work or even during actual implementation as part of a project review to decide whether or not the project needs to be reorganized or even canceled. The AIE method can be used to assess the various options for a particular project and hence provide selection criteria.

The AIE method can be used in strategic planning to compare and sequence IT investment projects, to periodically review this sequence in light of possible external or internal changes and, if necessary, reassign priorities and resources.

AIE links in with strategic planning in one other way: the assessment takes account of how each individual project contributes to the strategic plan (alignment).

Scope

AIE cannot, on its own, guarantee to improve the project failure rate or senior management satisfaction. AIE is an investment evaluation tool not an IT-project management tool. AIE will assist with project success only by identifying good IT investments at the initial decision stage. Other project management factors need to be applied to improve the degree of satisfaction.

The AIE method deals specifically with uncertainty, providing a big picture of the project. AIE can assist the IT project

management effort by identifying the investment project parameters where uncertainty is most likely to jeopardize the project.

Key Principles

The key principles are:

· in terms of assessment project organization, to separate the various roles to limit the effects of bias,

· to clarify intangibles,

· to express uncertainty in explicit, statistical terms,

· to determine the factors where more extensive analysis is justified to reduce the associated uncertainty.

AIE Implementation

The AIE method is first introduced to an organization through an implementation project. This project will start with training of all AIE users within the organization. Then certain parts of AIE will be customized for the specific organization and plans for implementation will be developed. AIE will often be initiated with a pilot project as hands on experience for the new users of AIE and as a test of the new decision making process within the organization. A detailed discussion of the AIE implementation procedure is part of another document.

The Stages of AIE

Once AIE has been implemented each proposed IT investment will be evaluated according to the following AIE phases.

Describe & Classify

This stage classifies an IT investment according to size and type so that the appropriate level of analysis can be applied. For example, if an investment is small and is mandated by government then virtually no analysis is required to make the investment. On the other hand, if an investment is large as well as optional, then more extensive analysis is justified. The high-level costs, benefits and risks identified in this stage are also key input to the clarification stage of AIE.

Clarify

The aim of this stage is to transform all the benefit, risks and cost intangibles into tangibles, i.e. into parameters that can be associated with a unit of measurement. The ultimate output of this stage is to express the investment decision problem as a spreadsheet model. All of the “intangible” benefits, costs, and risks will be variables in a quantitative decision model.

Measure

This is initially the explicit statement of the uncertainty about a quantity in statistical terms. Subsequently, the uncertainty can be reduced through observation and analysis. Measurement should not be confused with the arbitrary generation of exact figures. Various techniques are proposed for obtaining these measures: standard estimation techniques, calibrated estimates, search for information in external databases, scientific observations.

Optimize

Optimization is the general term for choosing the best of a defined set of possible choices with a given amount of information. This stage comes in three parts:

· Conduct Value of Information analysis (optimizing the measurement process)

· Conduct Risk & Return Analysis (optimizing the decision for risk and return)

· Make Recommendations (identifying what factors to manage to optimize the implementation of the decision)

This stage uses a simulation to combine uncertainties for actual implementations of the different investment project variables and to provide an overview of the probability distribution for the return on investment (or other financial criteria). This profile can then be used to deduce the mean expected Return On Investment (ROI) and the risk of negative ROI.

Investment criteria specific to the company can then be used to approve or disapprove the investment and even to alter the initial definition of the project. At this stage, it is possible to go

beyond the single binary question (yes or no) and implement a sensitivity analysis technique to determine those variables for which it would be profitable to reduce uncertainty by expanding the search for information.

Overview of the Major AIE Tasks

*Steps 4, 5 & 6 are all part of “Optimize”

AIE Method Components

The AIE method takes the form of documents that provide a general, but detailed description of the models, concepts and procedures involved. These documents will, in particular, include a summary and a presentation for senior management. The AIE method comes with supporting documents and is backed up with training sessions, workshops and the associated training materials.

The AIE method also comes with various tools:

· checklists for itemizing costs, benefits and risks,

· document templates for producing the business case that will be submitted to the decision-makers,

· spreadsheets for performing the calculations required by the method.

1. Describe & Classify The IT Investment

The objective is to provide a brief description of the missions of the investment project, the type (compliance, strategic or economic), the size, and a list of tangible and intangible elements of costs, benefits and risks. The classification of the IT investment is used to determine the necessary level of AIE analysis.

Summary Procedure Data

Responsibilities

Sponsor

Estimator

Auditor

Judge

Project Manager

Provide benefits and risks factors description

Respond to Questionair

Conduct Questionair and respond to some questions on it

Approve classification results

Develops AIE project plan if required

Time Required

2 hours to 2 days

Prerequisites

- appropriate training

- IS strategic plan

- customized AIE

Tools

- benefit and cost elements checklists

- risk factors checklists

- Project Definition Template

Required/Optional

Required

Reference(s):

Appendix A

Deliverable

1-3 written pages

Purpose

Different types of IT investments require different levels of analysis. This section deals with the determination of analysis requirements based on a classification of the investment. There are three objectives for the Classification step of AIE :

· Provide a high-level desciription of the investment project that must be evaluated.

· Describe & Classify the investment by type and size.

· Agree on the plan for the remainder of the project based on the analysis requirements indicated by classification.

Approach

To classification is done by plotting the proposed investment on a "Classification Chart" (see chart on next page). Depending on what region of the chart the point plots to we may take one of 5 actions:

· The investment is too small to even bother with classification, so make a judgement call.

· Accept the investment based on classification results alone

· Reject the investment based on classification results alone

· Proceed with computing the risk/return analysis with an abbreviated report

· Proceed with computing risk return analysis and generate a full report

The chart consists of 2 dimensions: the "Confidence Index" and "Estimated Investment Size". Each of these values are measured and the result shows what region of the chart the investment falls in and, consequently, which of the previously mentioned actions should be taken.

The Confidence Index

The Confidence Index (CI) is an initial assessment of how certain the decision maker is that the investment will have a positive value. But the decision maker is not actually directly involved in each assessment of the CI. Instead a model is developed that uses known characteristics of the proposed investment to estimate the CI and - some cases - user focus group or user survey responses. A short questionaire is filled out for the investment and the CI is computed from the responses.

Typical questions on the CI questionaire would be:

· "Does the investment involve internally developed software or is it only purchased technology?"

· "If it involves custom software development is it likely to be finished in 60 days or less?"

· "Did the User Focus group give it a review of 'very useful' or 'critically needed'?"

Your organization's specific CI questionaire has been developed and is provided in the "Classification Worksheet" on your AIE diskette and an example is shown in Appendix A. This worksheet computes the range for the CI (conservative and optimistic) based on the responses to this questionaire.

The CI can be roughly interpreted as answering the question "What is the probability that this investment has a positive net value or is otherwise necessary?". This quantity is the vertical dimension on the Classification Chart. Note that if the investment is below a certain size, as indicated on the Classification Chart, then the CI is not required and a purely subjective judgement can be made.

Investment Size

The size of the investment is initially estimated subjectively or by whatever data is quickly available. Even very broad ranges are acceptable at this point (for example, $2M to $8M).

What qualifies as the "Investment size" must be defined by the decision makers of an organization. One example of the definition of investmen size might be "All costs incurred before first benefits are realized". For the specific definition of what is included in the investment size in your organization refer to the "Classification Worksheet" included on your AIE diskette.

Interpreting the Results

Once the CI and the investment size have each been given a range of values - that is, an upper bound and a lower bound - then together they form an elipse on the classification chart. The greater your uncertainty about the values the large the elipse will be.

Depending on what region the elipse falls in the decision maker can decide to accept or reject the investment or to proceed with additional analysis. The output is a simple one or two page report that includes the following:

1. Objective of the investment

2. Responses to the questionaire

3. A classification chart

4. Recommended action

See Appendix D or the MS Word file on the AIE diskette for this document template.

Example Classification Chart

(See Appendix A for the classification chart for your organization)

Activities

1.1. Write title, main purpose/mission statement,

1.2. Write list of benefits, cost elements and risk factors,

1.3. Fill out questionair

1.4. Compute confidence index

1.5. Plot the ellipse which represents the investment project on the classification chart

1.6. Plan the remainder of the project with a timeline and resources required

1.7. Validate the project definition document

The deliverable must contain each of the items listed in the subtask steps. Include a classification chart as a graphic. Use the templates shown in Appendix D (you should have the electronic files as well). Check your classification chart to be sure it is up to date with the one provided in Appendix Since part of this deliverable includes the plan for the remainder of the project, refer to Appendix B for project planning help. It will be helpful to include any high-level timelines and resource requirements as part of this deliverable.

2. Clarify The Decision Model

Developing a quantitative decision model (in the form of a spreadsheet) of the benefit/cost/risk analysis, forces us to clarify many of the “softer” issues in the value of the proposed investment. In this section we resolve the “intangibles” and formulate a decision model.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:Facilitator

Financial Analyst

Workshop Participant


Optional Involvement


Facilitate the Workshop

Expert Review

Time Required

1 to 5 Days

Prerequisites

1. Project Description and Classification

Tools

Excel Templates

Required/Optional

Required

Reference(s):

App. C: Workshop Guidelines

App. E: Intangibles Checklist

Deliverable

Excel Spreadsheet with Cost/Benefit/Risk Model (no numbers, yet); 1-2 written pages

Purpose

The objective of this task is to reduce all (intangibles( to unit-of-measure variables and to formulate a comprehensive spreadsheet-based model of the costs and benefits of the proposed system investment.

We cannot answer a question we do not entirely understand. However, our understanding is of a very limited kind when we are not able not able to express the problem quantitatively. Just like any scientific endeavor, we must express ourselves quantitatively in order to get a handle on the problem.

There are many tasks that depend on this task as input. We are developing a foundation during this task that will be critical to the rest of the analysis.

Approach

The Clarification stage is usually conducted in a series of workshops that apply specific facilitation tools. These tools will help the participants state the effects of the investment in a more tangible manner. Part of the approach involves focusing on the development of a spreadsheet based decision model as a very precise and unambiguous expression of the decision problem.

By focusing on the development of a specific formula for the decision we force ourselves to think of all factors in the decision as quantities. Most factors perceived to be “intangibles” will fade away and all we will be left with is a set of formulas in a spreadsheet.

Activities

This is an overview of the major subtasks of the Clarification task.

2.1 Resolve Intangibles*

2.2 Develop Structure for Cost/Benefit/Risk Data Sheet*

2.3 Determine Need for Additional Analysis or Reclassification – State in your deliverable if you are recommending a reclassification or if you should move on to measurement.

2.4 Summarize Clarification – Use the template in Appendix D to help organize your findings into a document. Part of your deliverable will be the spreadsheet itself.

*Additional detail on 2.1 and 2.2 is provided in the following sections

2.1. Resolve Intangibles

The Intangibles of an IT investment often only seem immeasurable because they are not clearly defined. Ambiguity can be removed by applying the Clarification tools.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:Facilitator

Financial Analyst






Expert Review

Time Required

1 to 2 Days

Prerequisites

1. Project Description

Tools/References

App.C: Workshop Guidelines

App. D: Document Templates

App. E: Intangibles Checklist

Required/Optional

Required

Deliverable

List of quantitative variables

Purpose

Most IT investments seem to have multiple “Intangible” benefits or costs. These intangibles may seem difficult to include in the decision process. Yet, if intangibles are not represented many of the key benefits of IT may be under-represented.

The objective of this task is to find one or more quantitative factors (tangibles) underlying what seems to be a list of “intangible” or “soft” benefits or costs.

Approach

The AIE approach uses a very successful method to deal with this problem. For starters, AIE simply assumes that, in reality, there are no intangibles. Furthermore, we only think that some benefit or cost is an intangible because we are unclear about what the cost or benefit really is.

AIE uses a method called the “Clarification Workshop” to coach the participants into defining what they really mean by a given intangible. This approach focuses on the practical, observable consequences of a supposed “Intangible”. In effect, we deal with ambiguity by removing it. Eventually, clearer definitions begin to emerge and the participants invariably think “Yes, that is what I meant all along” and the units-of-measure will be more obvious.

Examples of “Intangibles”:

· Orgainizational Flexibility

· Employee Empowerment

· Strategic Alignment

· Customer Satisfaction

· Better Access to Information

Tasks

This task consists of three main parts:

2.1.1.Develop Clarification Approach

2.1.2.Conduct Clarification Workshop

2.1.3.Develop Clarification Deliverable

These are described in more detail in the following paragraphs.

Task 2.1.1 Develop Clarification Approach

Perhaps the resolution issues that need to be resolved are fairly simple. Perhaps they are more elaborate and require a more deliberate facilitated workshop. Here we will determine what we need to do and prepare for it.

2.1.1.1. Conduct Initial Reviews

Have a meeting among the individuals involved in the development of the Description deliverable. Perhaps there are some simple ways to express the “Intangibles” in a measurable way. Refer to Appendix E (Intangibles checklist) for help. Also, refer to some of the resolution tools in subtask 2.1.2 Conduct Clarification Workshop. Perhaps some of the intangibles can be easily converted to tangibles with only a little extra thought. If it seems that there are several difficult intangibles then consider step 2.1.1.2

2.1.1.2. Conduct Cursory Research

It is important at this step to answer the “What is out there” question. Perhaps a little research will turn up an article or report on the value of such investments. This should give you ideas the types of possible questions to focus on in a workshop. Refer to the section 3.2 (Conduct Secondary Research” for specific information. Be sure to limit the amount of time you spend on secondary research at this stage to less than 2 hours. If it is required, more extensive research will be done later.

2.1.1.3. Prepare Workshop

If several of the intangibles require additional thought to resolve then a deliberate facilitated workshop is required to work out the issue. Refer to the Appendix C (Workshop Guidelines) for additional information about preparing for facilitated workshops

· Identify participants: A clarification workshop should consist of one facilitator and 3 to 6 other participants. The participants should represent individuals with a stake in the proposed IT investment (sponsors) as well as those who will be making some of the quantitative estimates (estimators). The facilitator must be a trained facilitator and the participants must have received a briefing on the AIE approach.

· Schedule/prepare facilities: The facilities should consist of a conference room with flip charts and/or white boards.

· Prepare workshop tools: You could make slides or handouts of the “Clarification Chain” and “Thought Experiment” (shown in 2.1.2.) and perhaps the Intangibles Checklist in Appendix E. It is also useful to prepare feedback forms to hand out at the end of any workshop.

Task 2.1.2 Conduct Clarification Workshop

The Clarification Workshop is a facilitated session that uses free-form techniques to help the participants translate intangibles into measurable quantities. For additional information on workshops see Appendix C “Workshop Guidelines”.

Note: By facilitated, we mean that the workshop is run by a facilitator who is trained in clarification methods and that the workshop is structured with the following guidelines.

2.1.2.1.State Agenda and Goals

Let the participants know exactly why they are attending the workshop, how long it should take, and what the expected outcome is. Also, be sure to introduce the roles of the various participants it necessary.

2.1.2.2. Review Project Description

Review the purpose, the expected benefits, the costs, and the risk factors identified in the project description (from 1. (Describe & Classify the IT Investment(. List those issues that have not yet been converted into tangibles. The participants may desire to alter the description somewhat. Come to an agreement among the participants on the description of the project and the list of intangibles to review.

You may need to brainstorm additional intangibles until the participants agree that the list of intangibles is complete. This may start slow but if the facilitator engages the group properly the ideas will soon start flowing. Remember, brainstorming is a process of idea generation, not idea evaluation. Reserve evaluation of the ideas until later.

2.1.2.3. Apply Clarification Methods

For each of the intangibles listed in the previous subtask step apply one or more of the following tools until the intangible is replaced by measurable quantities.

First, try asking “What do you mean by______?”. Sometimes people volunteer resolutions to intangibles with such simple prompting. The facilitator encourages the group to focus on the practical consequences of the intangible.

For more difficult problems try the “Clarification Chain” and the “Thought Experiment” explained below.

The Clarification Chain:

· If it is better, it is different in some relevant way

· If it is different in some relevant way, it is observable

· If it is observable, it is observable in some amount

· If it is observable in some amount, it can be measured

A Thought Experiment:

Imagine that you have made a duplicate of your organization that is precisely the same in every respect except that one company has more of "the intangible" than the other. What do you actually observe to be the difference between them? If there is no observable difference then perhaps the “intangible benefit” is not really a benefit at all.

Consider this an iterative method. You may want to reapply these same tools more than once until specific measurements (as shown below) are identified.

Examples of Clarified Intangibles:

Intangible

Tangible

Unit-of-measure

“Employee Empowerment”

Reduced management overhead per employee

Hours management/employee

Improved claims adjusting

% of accurate claims

“Customer Satisfaction”

Repeat business

% of new customers who make another purchase

For additional examples of clarified intangibles see Appendix E “Intangibles Checklist”.

2.1.2.2. Record Tangibles

When the facilitator believes that a specific and unambiguous measurable has been defined then the variable name and unit of measure must be recorded onto the Parameter Table (Reference Appendix Y for Template). Also, be sure to record newly discovered intangibles and their resolution to the “Intangibles checklist” for use by future facilitators.

Task 2.1.3 Develop Clarification Deliverable

You must present the findings of the clarification workshop in a concise summary that will be made part of the final deliverable for the AIE analysis.

It would be helpful to show a table that presents each of the original intangibles and how they were ultimately resolved into measurable quantities. The “Example of Clarified Intangibles” on this page might be used as a guideline.

It would also be uselful to the reader to list the participants, the date of the workshop(s), and any feedback from the participants regarding there confidence in the completeness of the list.

Refer to the template in Appendix D as a guideline for developing this document.

2.2. Develop Cost/Benefit/Risk Data Sheet

Once the relevant factors in an investment decision have been redefined as measurable quantities, we can insert those variables into a decision model constructed in a spreadsheet. This will be the basis of the more advanced analysis that comes later.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:Facilitator

Financial Analyst






Expert Review

Time Required

1 to 5 Days

Prerequisites

2.1. Resolve Intangibles

Tools/References

Excel Templates

App. G: Equations Reference

Required/Optional

Required

Deliverable

Excel Spreadsheet with Cost/Benefit/Risk Model (no numbers, yet)

Purpose

After we know what variables make up the model, we must create an spreadsheet model that correctly represents the Cost Benefit Analysis. We have to be specific about how the variables actually “add up” to compute the ROI or NPV of a project. Obviously, errors in the basic financial formula will cause error in the decision. Therefore it is critical to make sure we develop a rational, economically and financially valid spreadsheet model.

The objective of this task is to develop a spreadsheet model that will compute the NPV and/or ROI. At this point the focus is developing the formula - not generating specific numbers to put into your model. So, initially, the model will only contain “test” data to ensure your formulas are working

Approach

We will be building a basic spreadsheet which is not too different from other Cost/Benefit spreadsheets you may have developed in the past (with only a few additional features). The model will consider the benefits, the costs and your formulas for discounted cash flow or return on investment and perhaps taxes. To ensure that the formulas do not contain the errors that many cost/benefit spreadsheets typically have, quality assurance steps must be performed on the spreadsheet. In other words, it will make sense economically.

Unlike most CBA’s, however, we will compute the effect of project cancellation on the expected value of the investment. Uncertainties regarding benefits and costs will also be expressed quantitatively.

Tasks

We will not go into detail about how to develop simple financial models on a spreadsheet. We must assume that you have some experience in constructing basic CBA spreadsheets. If not, then you should enlist the help of someone who has. There are some things fairly unique to AIE, however, that we need to cover.

Calculating “Expected Values” with a Chance of Cancellation

One of the most influential factors in the net value of an information system investment is the chance that it might be cancelled prior to implementation but after a significant expenditure. Later, we will assess the probability of cancellation but at this point we will focus on simply including the variable in the CBA formula.

Our formula will calculate what is called the “Expected Value” of the outcome. An expected value is simply probability-weighted averages of all the possible outcomes. In the most simplified situation we will only consider two possible outcomes: the project is cancelled or it isn’t.

If a project is cancelled then you will incur some costs but you will not incur the benefits. If it is not cancelled you will incur costs but you will also get the benefits (of course even if it is not cancelled costs can still exceed benefits). The Expected Value calculation requires us to use the mean values for the costs and the benefits. It also requires us to use two different costs (one if cancelled and one if not).

See App. G (Equations Reference) for detailed information on Excel formulas for cancelation models.

The simplest version of the cancelation model just considers the binary cancelled/not cancelled question. The more elaborate cancelation models consider exactly when a project may be cancelled and what might have caused it. On projects with a long expected duration we should also include the effect of project delays on the realization of benefits.

Marginal vs. Loaded Labor Costs

The cost of labor to compute benefits (productivity improvements) or costs (development labor) is common in CBA’s.

Usually, labor costs simply assume some full salary plus “loaded” expenses.

These numbers, however, do not represent the actual business impact of productivity improvements. This assumes that if you save one hour of labor through productivity improvements, that you will actually save one hour worth of salary and its allocated administrative costs. This is not the economically sound method for dealing with the value of labor.

But you do save something when you save some labor, even if you do not actually reduce the labor pool. What you do save is the “Marginal Value of Labor”. The value you should use may be provided in Appendix A “Custom Data”.

Reference Tools

In App. G (Equations Reference) you will find a helpful reference of basic financial calculations. Also, below is a “QA checklist” for CBA’s. It is a good idea to review this in detail as you are developing your spreadsheet.

QA Checklist for the cost/benefit model

· Are you using the spreadsheet templates provided where they are appropriate ?

· Did you consider the chance of cancellation or other catastrophic failure in your cost benefit formula?

· Does your model implicitly assume that the usage rate for your system is 100% of the target users? Should you explicitly include a “usage rate” variable in order to capture your assumptions about this quantity?

· Does the model need to reflect the possibility of future changes in business volumes? If so, should you explicitly include growth rates (positive or negative) to capture your assumptions?

· Have you conferred with the appropriate internal expert regarding possible tax considerations?

· Are you using real marginal costs instead of artificial loaded costs for labor? Have you checked the Standard Metrics in App. A ?

· Do costs include training, implementation, maintenance, vendor fees, required hardware/software upgrades, “help desk” costs, and user involvement?

· For longer projects, does the model delay benefits until the project is completed? Or does it assume that benefits start at a fixed time regardless of the duration of development and implementation

· Are the benefits realized as soon as the system is implemented or do they become realized more gradually?

· Are you double-counting anything?

· Is the math correct? Did you check the calculations of the spreadsheet with test numbers ?

· Are you adjusting for the time value of money correctly? Did you check your formulas against the reference in App. G ?

Example Spreadsheet Set Up

Always start with the AIE template Cost/Benefit/Risk spreadsheet provided on the AIE diskette. This is a simplified business model that should suffice for most smaller investments that require risk/return analysis. Additional benefit, cost and/or risk sections can always be added as the investment requires.

Using Time Series

A common requirement is to show cash flows over each of several years. Often, the time series in shown in the columns of a spreadsheet. But AIE already uses multiple columns (and worksheets) for all variables. So to make modeling time series as simple as possible the AIE spreadsheet has some additional features dedicated to making time series modeling simpler.

The AIE spreadsheet has a new function called a "Time Series" function. This represents in a single sell values for several time periods. Time series cells can be added , multiplied, etc. to eachother or to normal cells. The time series cell can then be used to create the proper formulas for all the periods in the time series. It can show the details for each period in the time series or it can hide them for simplicity.

The syntax of the time series function is as follows:

=timeline(formula)

This function returns the string "Time Series" to show that it consist of several cells of various values. The formula is string that is written just like any Excel formula with some exceptions:

1. Exclude the "=" (equals sign) at the beginning of the formula. For example, type "=timeline(C$10+C$11)" to add cells C10 and C11 - not "=timeline(=C$10+C$11)".

2. References to cells that are other time series must be relative (that is, no "$" in the cell reference). References that are not other time series must be anchored on the row. For example in the function "=timeline(C$10*C20)" an non-time series cell (C10) is multiplied by each period in another timeseries (C20)

3. The word "Period" is a reserved label that means a number that represents a specific period in a time series. For example in "=timeline(C15*C$5^Period)" each period in the time series at C15 is multiplied by the non-time series cell at C5 taken to the power of the period. The cell at C5 may represent a growth factor so that in the first year of the series is taken times factor^1, the second year by factor^2, etc. The first period in a time series always has "Period" = 1.

Once you have typed in the timeline function, select that cell, and click on the "Insert Timeline" button. This will create a number of rows equal to the number of periods in the "planning horizon" cell. The first period is always the value in the "first year" cell. Make sure you have blank rows beneath the time cell before you do this or it will overwrite the cells below it.

All the time series can be hidden or expanded to show each period by clicking on the "Expand/colapse Timeline" button. The spreadsheet will look much simpler with the time series colapsed.

You can still make time series manually if you wish. But these rows will not be recognized by the "expand/colapse" macro and so will always stay in the expanded mode.

It is helpful to show the actual formulas used in the spreadsheet in the Source Reference column. This is done easily by clicking on the "Show Formula Text" button at the top of the Source Reference column. This pastes the text of all formula cells in column C to column F. Any future source references added to non-formula cells are not overwritten when clicking on the Show Formula Text button.

Example Spreadsheet for the Clarification Phase

The objective at this point is to create a spreadsheet with valid formulas. The numbers are still just test values and we don't need to make any measurements yet.

3. Measure Variables

Initially, this stage will focus on simple quantifying the current level of uncertainties about variables. Subsequently, further analysis can reduce uncertainty further.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:Facilitator

Review

Workshop participant

Review

None


TimRequired

Initially, 1-2 days

Subsequent, variable

Prerequisites

2. Clarified Parameters

Tools/References


Excel Templates

Required/Optional

Required

Deliverable

Excel Spreadsheet with Cost/Benefit/Risk Model with quantities and uncertainties

1-2 written pages

Purpose

If our analysis is going to be realistic we have to have realistic numbers.

The objective of this task is to employ one of several methods for assessing the quantities of variables in the cost benefit model.

Approach

The AIE approach to measurement is the scientific approach to measurement. We only use data that has been directly observed, is estimated by individuals with a “calibrated” track record at estimating, or is obtained from other valid sources that use these methods. The AIE definition of measurement is likewise the scientific definition.

Definition: Measurement is an observation that results in a reduction of uncertainty about a quantity.

This definition has the strict requirement of being observation-based and dealing strictly with quantities (anything else is not a measurement). But this definition removes the constraint that most people put on measurement – that it has to be exact to be a measurement. Any reduction in uncertainty about a quantity is a measurement and can be a value to our analysis. If, through additional analysis, reduce the probable range of the development costs for a system – even by just half or less – then I have reduced uncertainty about the cost of the system.

We use levels of measurements. The first set of measurements is the most cursory with high uncertainties. Further measurements will tend to be more deliberate and will reduce uncertainty much

Activities

For more detail on each of these steps, refer to the sections in the following pages

3.1.Choose Measurement Method

3.2.Conduct Secondary Research

3.3. Calibrated Uncertainty Assessements

3.4. Scientific Observations

3.5. Update Worksheet with Measurements

3.1. Choose Measurement Method

In order to optimize further measurements, we must first determine how much we currently know. Often, this involves the “Calibrated Uncertainty Assessment” method but it can also include other types of expedient research


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:Facilitator

Review


None

None


Time Required

½ to 1 Day

Prerequisites


Tools/Reference

Guidelines in 3.1.1.

Required/Optional

Required

Deliverable

Identified measurement methods for each variable

Purpose

Every variable will require different types of measurements depending on two factors: the type of the variable and the stage of analysis.

The objective of this task is to choose which of the methods will be employed to reduce uncertainty about the quantity and to what extent it should be employed.

Approach

We will first make a distinction according to the stage of analysis. The first time the measurements task is executed in a AIE analysis the measurements are called “Initial Estimates”. At this stage all measurements will be characterized by a relatively small investment in the measurement process and a correspondingly high degree of uncertainty about the measurement.

Additional measurements will only be made according to the findings of the next major AIE task – the Value of Information Analysis (VIA). If a variable requires further

measurements (as dictated by VIA) then there will be additional measurements of the appropriate level of analysis.

We also determine the type of analysis needed according to the type of variable. For example, if a variable is one that is captured in a standard metric, then we use the standard metric. If it is a quantity that can be observed under the right conditions then it may require a controlled experiment. If it is a quantity that may currently be tracked somehow in your organizations existing databases then we may use queries.

Tasks

For more detail on each of these steps, refer to subsequent sections.

3.1.1.Apply Guidelines Pertaining to the Stage of Analysis

3.1.2.Apply Variable-type Guidelines

Task 3.1.1. Apply Guidelines Pertaining to the Stage of Analysis

These guidelines are driven by whether this is the first or subsequent stage of measurement in this AIE project. The effect is to give you some idea of the size of the measurement effort you should be considering and in some cases will direct you to the type of method you should use.

Is the first time in this particular AIE analysis project that you have come to the Measurement section? If so, go to the following “Guidelines for Initial Estimates”. Otherwise, go to “Guidelines for Additional Analysis”

Guidelines for Initial Estimates

For initial estimates (when the measurement task is executed for the first time in a AIE analysis) we will be “timeboxing” the analysis. That is, we will set a maximum period of time that we will spend in this stage of analysis and stick to it. Our objective at this point is to provide some broad, quick estimates. We will determine where and if additional analysis is needed after we have conducted the “Value of Additional Analysis” task. Use the following guidelines to limit the amount of time you spend in this stage.

Guidelines for Magnitude of Initial Analysis

(Only pertains to analysis prior to first VIA)

Investment Project Size

Effort

Duration

Probably under $50,000

1-2 work-days

2-5 hours

Probably not over 1mill.$ nor under $50,000

3-6 work-days

1-2 days

Probably over 1 mill.$

7-14 work-days

3-4 days

Major surveys or controlled experiments are obviously not practical given these constraints. You should focus on using mostly standard metrics (where you have them), secondary research, or calibrated estimates.

Guidelines for Additional Analysis

If you have already done initial estimates and have conducted the Value of Information Analysis then use the following table as a guideline for how much time you can spend analyzing a variable.

Magnitude of Analysis Guidelines:

(Expected analysis cost should be 2%-20% of the EVPI)

EVPI Range

Effort

Examples Possible Types of Analysis

Under $20k

0

No deliberate analysis is justified

$20k to $50k

Up to 2 work-days

Informal phone surveys of randomly selected subjects, more extensive research of secondary data or internal databases

$50k to $200k

2 to 5 work-days

Small controlled experiments (asking vendors to participate in benchmarks), phone surveys, e-mail surveys,

$200k to $1000k

5 to 25 work-days

Mail, e-mail, phone surveys of significant number of subjects (50-250), deliberate controlled experiments, possibly the use of a software metrics service

Over $1000k

One or more work-months

Use of software metrics, large deliberate studies with professional assistance, pilot projects with entire divisions or branches

Task 3.1.1. Apply Variable-Type Guidelines

Now that you know how much effort is justified in analysis, you can determine the method of analysis required for each of the variables (Note that if this is an initial estimate you have already been given some guidelines regarding the type of analysis). Here are a series of questions about the variable that should help you identify a method of measurement.

· Is the variable regarding the cost, duration, or chance of successful implementation of the system? If this applies to the variable then you should probably be referring to the standard metrics (Appendix A) where you may find estimation models for these. If, for some reason, these models do not apply to the cost, duration, or chance of cancellation for this particular project (or if the standard metrics are incomplete at this point) then you should consider calibrated estimates backed up by secondary research.

· Is the variable a quantity that about some current activity that may leave a trail? Just about everything leaves some kind of trail. If this variable is one that leaves a trail, then perhaps the answer is in your company’s databases or paper files. For example, if you are trying to measure the average current processing time for a claim (supposing that your proposed system will reduce this time and, therefore, this original quantity must be part of your CBA) then perhaps this can be derived just from doing queries on existing records. See the secondary research task for details.

· Is the quantity about a current activity that is not currently tracked but could be tracked if you made a deliberate effort to do so? Perhaps you need to do some surveys of people involved in this activity. Perhaps they will have some idea. Or, better yet, you could somehow record their activities while they do it. There are practical ways to do this. See the section on random samples for details.

· Is the quantity not about some current activity but about possible effects of some change? If you are trying to estimate the effect on productivity or the effect on customer retention due to some new system then you do not have the luxury of simply following some existing trail. In this case you have to deliberately arrange for the effect to take place. This is what a controlled experiment is – specifically setting up an environment where you can watch some specific phenomenon take place (unlike sampling which only observes existing phenomenon).

· If you simply asked enough people, would you know more about this quantity? In case you are trying to measure something like the chance that a customer would be encouraged to stay with your firm if they had the benefit of some new billing system, information system, etc. then you might try conducting surveys (i.e. opinion polls). See the section on random sampling for details.

· Is there a reasonable chance that someone else has already conducted such measurements and reported them? If so, then spending some time doing some secondary research might turn out to be fruitful. See the secondary research section for details.

· Are any of the above, for any reason, not applicable or practical but you have SOME idea about what the quantity may be? If this is the case then perhaps you (or someone else) should become calibrated probability assessors and make a calibrated assessment of the value. See the section on calibrated estimates for details.

Hint for Generating Ideas:

How would others do it?

Believe it or not, the measurement problems you have right now are probably not the most difficult measurement problems anyone ever had. Other people routinely measure things equally or even more difficult. How would each of the following people research the item you are looking for?

· A detective

· An archeologist

· An analyst at marketing research firm

· A stock analyst

· A political analyst

· An actuary

· A doctor making a diagnosis

· A journalist working on a story

· A librarian

· A military intelligence officer

· A hunter tracking an animal

· A scout seeking a trail

· A scientist

· A spy

· A graduate student working on a thesis

Helpful Prompting Thoughts

“Everything relevant is observable” – what you are attempting to measure leaves a footprint somehow. What observable and/or recorded consequences does the thing you are measuring have? Do those consequences have any observable and/or recorded consequences?

“Someone knows something about it” – Who are you not asking for input that would be vital to the information research?

“All information is cumulative” - let one finding adjust your approach to further research, or go back and redo research if you found something useful. Use findings to identify entirely new research paths.

“Don’t make it harder than it is” – There is a simpler way to do it, what is it? If its not measurable one way, its measurable another way.

3.2 Conduct Secondary Research

Someone else has probably already measured many of the items you may attempt to measure, perhaps in another organization for a different purpose. Perhaps the information can be derived from data you are already tracking within your firm.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other: DBA

Corporate Librarian

Industry Analyst

Review

Primary responsibility

None

None

Expert assistance

Expert assistance

Expert assistance

Time Required

½ to 2 Days

Prerequisites

3.1. Choose Measurement Method

Tools/References

Required/Optional

Required

Deliverable

Excel Spreadsheet with Cost/Benefit/Risk Model and initial estimates

Purpose

In order to be as efficient as possible in are analysis we should attempt to utilize existing information whenever possible. This “secondary research” (meaning it is from another source as opposed to primary, directly measured data) can be a time saver and it is usually a fruitful effort.

The objective of this section is to provide guidelines for the use of secondary research.

Approach

At the beginning of your analysis try doing a set of searches on the type of system for which you are doing the AIE analysis. It may give you ideas on variables to consider in the cost/benefit model. Some level of secondary research should be attempted whenever possible.

Secondary research cannot be as highly directed as doing your own surveys and other direct measurements. You will get a lot of data related to what you need but not exactly what you need. However, even if search results don’t specifically answer a question the findings can still usually effect your assessment of the quantity. For example, suppose you are trying to find the percentage improvement in productivity due to a specific network management tool. None of the research tells you specifically the percentage improvement but you found an independent survey that says most network managers did not experience a productivity improvement. How does this effect your knowledge of the possible ranges of this quantity?

Tasks

3.2.1.Search the Internet

3.2.2.Search library resources

3.2.3.Search internal databases

For additional detail, see the sections below.

Tasks 3.2.1 Searching the Internet

They Internet is getting to be a very productive place to search for information on just about anything. Its almost certain that you’ll find something that will be helpful but it can also be a big waste of time if you don’t know when to quit. So set a time limit on this of 30 to 120 minutes.

First, just go to a major search engine and try a search. Next try some of the periodicals on the Web (Usually, articles within a periodical will not show up on a web-wide search. You have to go into their own search engines)

Using Search Engines

Search engines keep getting better. The following are search engines with large databases. Some will ask you to specify whether you want to search only web sites or Newsgroups. Be sure to check both in order to get a thorough search. There may be one or more newsgroups that have something to tell you about your question.

· www.hotbot.com

· www.altavista.com

· www.lycos.com

Keep in mind that when you conduct this type of search that you will get a lot of undesired “hits”. Don’t let that discourage you. You can quickly sift through the responses you received and disregard the unwanted hits. If there are too many hits to examine then you should try “narrowing” your search by adding more conditions.

Example

If you get too many hits with the search “ ‘Document management systems’ ”) then you might try “ ‘Document Management Systems’ and insurance and productivity”

Search Periodicals/Services on the Web.

Often, articles within a periodical will not come up on a generic search from a search engine. You have to visit their web sites and use their “archival search engines”. These three periodicals have a lot of articles on ROI-related topics and have search engines for finding the important articles.

· www.informationweek.com

· www.computerworld.com

· www.datamation.com

Also if your firm subscribes to Giga Information Group, The Gartner Group, or The Meta Group, you may find what you are looking for with searches on their web-sites.

Task 3.2.2 Search Library Resources

There is still a lot of information that is not on the Internet. Fortunately, this information is also easily searchable. You just can’t do it while sitting at your desk. You’ll have to go to a library.

If you don’t know your way around a library, just ask a librarian for help. Libraries that will be the most fruitful for your purposes will tend to be large corporate or university libraries.

Libraries tend to be a good source for government studies, academic journals, and special associations. In the insurance industry, you might try researching, LOMA and LIMRA publications. Also, the periodical “Insurance & Technology” tends to be a good source of information about the effects of IT in the insurance industry. Soon, more of this information will also be accessible through the Internet.

Task 3.2.3 Search Internal Databases

Your current databases may already have what you are looking for. You might be surprised what you can infer just by looking at the data in your firms databases – even if you already work for the IT department.

Individuals at your level (probably some management or analyst or planning position) generally do not have intimate familiarity with all of the information on your databases. But, at least you have people in your firm who do know. DBA’s , DA’s, and even data entry people can be generally aware of recorded data

Some information you can probably derive from searches of internal databases include the following:

· Internal Productivity, Business volumes – query total volumes for a time period

· Response/cycle Times – you may track time-stamps on records created at different stages of a process

· Growth rates – look at a changes in business volumes over time

· Differences between groups – for example, comparing the retention of customers that get never had a late claim versus customers who did

· Costs of various activities – the “table of accounts” may have some expenses broken down to this level

· Distributed searches – perhaps what you are looking for cannot be found with a single query on client server or mainframe but you can find it by querying the hard-drives on several PC’s

· Reports that have been done in other departments

3.3. Calibrated Probability Assessments

Observation based measurements are really just the use of the scientific method. We use controlled experiments or random sampling methods to measure some quantity which is relevant to the analysis.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other: Facilitator

Workshop participant*


Observer*

None

Facilitate a Workshop

*optional

Time Required

3 to 6 Hours

Prerequisites

3.1 Choose Measurement Method

Tools/References

Appendix H: Example Calibration Tests

Required/Optional

Optional (but almost always utilized)

Deliverable

Probability distributions for selected variables

Purpose

If the subjective judgement of the business or IT person can be tested, measured, and refined then asking the opinion of an expert can be one of the most cost effective methods of measurement.

The objective of this section is to explain the mechanism of calibrated uncertainty assessments and how to apply them.

Approach

Assessing one’s uncertainty about a quantity is a general skill that can be taught. In other words, experts can measure whether they are systematically “underconfident”, “overconfident” or have other biases about their estimations of quantities – regardless of the specific type of uncertainties they are attempting to estimate (project costs, market forcasts, etc.). Once this self-assessment has been conducted they can learn several techniques for achieving a measurable improvement in assessing uncertainty. This initial “calibration” process is critical to the accuracy of the estimates later received about the project.

Definition

Overconfidence: The individual routinely puts too small of an “uncertainty” on estimated quantities and they are wrong much more often then they think. For example, when asked to make estimates with a 90% confidence interval much fewer than 90% of the true answers fall within the estimated ranges.

Underconfidence: The individual routinely puts too large of an “uncertainty” on estimated quantities and they are correct much more often then they think. For example, when asked to make estimates with a 90% confidence interval much more than 90% of the true answers fall within the estimated ranges.

Tasks

3.3.1.Choose estimators

3.3.2.Calibrate estimators

3.3.3.Estimate the quantity

Additional information is in the following sections.

Task 3.3.1. Choose Estimators

The estimators should be individuals who are actually in a position to have some idea about the quantities in question. Different variables may require estimates from different people so separate sessions may be required.

Task 3.3.2. Calibrate Estimators

Few individuals tend to be naturally good estimators. Most of us tend to either be biased toward over or under confidence about our estimates.

Academic studies have proven that you can receive better estimates by putting proposed estimators through a workshop designed around removing personal estimating biases. This workshop begins by asking the participants to make a 90% confidence interval to describe their personal knowledge about a given set of general knowledge questions.

Since the original estimates were made with a 90% confidence, an average of 1 in 10 should be incorrect. By reviewing the participants answers to these questions we can derive and illustrate their over or under confidence. By performing this process of answer and review several times, participants become “calibrated” to the level of their personal confidence that corresponds to a 90% level of statistical confidence.

Task 3.3.3. Estimate the Quantity

After your chosen estimators have been calibrated, it is time to estimate the quantity in question. Simply ask them to estimate the quantity as they did in the calibration tests

In order to protect against “overconfidence”, make sure your probabilities reflect the appropriate amount of uncertainty about each consideration in the following checklist:

· Availability of resources

· the scale of the required features

· uncertainties in the organization

· untried technologies

· problems organizing across departments

· lack of familiarity with the development platform

· uncertainty about change in business volumes that effect the ROI

· uncertain government or market factors

One more advanced method you might try for binary probabilities (like the chance that a system will be cancelled or not) is called “Baysian Decomposition”. It works by identifying “conditional probabilities” (the probability given some condition, like a merger taking place) and the chance of those conditions.

Here is a simple example:

Example for Baysian Decomposition

The chance of cancellation (original assessment)

32%

The chance of cancellation given that the merger does not take place

20%

The chance of cancellation given that the merger took place

40%

The chance of the merger taking place

85%

Adjusted chance of cancellation

(multipy each conditional probability times the chance of that condition and add them up)

37%

Example 90% Confidence Intervals after Calibration (for a document management system)

90% Confidence Intervals

Cost/Benefit Variable

Lower 5%

Mean

Upper 5%

Percentage Time Spent Searching

5%

15%

25%

Percentage Reduction in Time Spent

5%

30%

50%

Number of Annual Occurrences of lost documents

10

50

90

Clerical Hours to Reproduce

60

150

240

Annual Cost of litigation lost due to unfound docs

100,000

400,000

1,500,000

Annual Number of New Documents

38,000

50,000

62,000

Number of Documents

42

50

58

3.4. Scientific Observations

Measurements of how the real world works is often best done by recording data about observation and analyzing that data with quantitative methods.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other: Statistical Support

Help with the planning and exectution of the study

Time Required

Variable

Prerequisites

3.1 Choose Measurement Method

Tools/References

Required/Optional

Depends on VIA

Deliverable

Probability distributions for selected variables recorded in the Excel

Purpose

The objective of this section is to help AIE users identify possible applications of scientific observations in the assessment of the IT investment. A detailed discussion about methods of scientific observation are beyond the scope of this document. However, it is possible to describe the apects of such methods as they pertain to AIE usage and to help those who may be familiar with scientific methods to apply them to AIE.

NOTE : If you have no experience with sampling methods or controlled experiments then seek support internally or from external sources.

Approach

Just about anything you want to measure can and probably already has been measured scientifically. Yet IT rarely avails itself of these powerful methods when performing a cost benefit analysis. Although it is outside of the scope of AIE to discuss scientific method in detail, the AIE method strongly encourages the use of such methods.

It is often assumed (incorrectly) that scientific studies must be large and expensive undertakings. In fact, scientifically valid data can be gathered in a very short period of time by very few people.

Example 1: Measure the effect that a new billing system has on customer retention by:

· Including randomly selected customers in a pilot project and measure the increase in retention against other customers;

· Sending a questionnaire to randomly selected customers and asking if they would be more likely to stay because of the changes.

Example 2: Measure the improvement in productivity of a claims adjuster due to a new claims system by:

· Doing a spot sample of claim adjuster activities (thereby measuring the size of activities that would be eliminated by automation)

· Conduct a pilot project with several randomly selected claims adjusters on the new system and compare them to other claims adjusters.

Example 3: Measure the shortest possible time it takes to prepare a bid by asking the CEO to have a few bids expedited through the process

Tasks

This is not a detailed description of the scientific methods but it is worth mentioning a few basic characteristics of all scientific measurements:

· Identify the quantity to be observed

· Identify a method by which the quantity can be observed and recorded. All observation can be split into two types : observing a phenomenon that already occurs or artificially creating a situation to observe the phenomenon (an experiment)

· Determine the statistical method for interpreting the recorded data

· Standard pitfalls to be addressed in scientific measurements include interdependencies of unmeasured variables, observer bias and the effect of the observer on the phenomeno

3.5. Update Worksheet with Measurements

At this point we know probability distributions for all variables. We need to capture this information in the worksheet. Specifically, in the way random scenarios are generated.


Responsibilities

Sponsor

Estimator

Auditor

Judge

AIE Project Mgr

Provides some measures

QA the spreadsheet


Time Required

1 to 2 Hours

Prerequisites

Measurements

Tools/References

AIE Excel Template

Required/Optional

Deliverable

Updated Excel worksheet

Purpose

We need to state our uncertainty about measurements by describing the properties of the probability distribution for the number. This effects the risk as well as the value of information computations that come later.

The objective of this section is to update values in The Lower Bound, Best Estimate, Upper Bound, Distribution Type and Source Reference columns.

Approach

We will be building on the worksheet you have already started on. The formulas are listed in Appendix G "Equations Reference" for your reference but may also be copied from templates (formula reference to the left).

Each of the following columns should be filled in at this point:

· The Best Estimate – In the Clarification Phase this column was filled with formulas and test values. Now, for some of rows, the values need to be specified depending on the distribution type. See the Distribution Data Table.

· Upper and Lower Bounds - These columns should have been empty up to this point. Depending on the distribution type they may need to be given values now. See the Distribtution Data table for the meaning of these values

· Distribution Type - This column should have been empty up to this point. For those rows that contain measurements with uncertainty we need to specify the distribution type. Any row with Upper and Lower Bounds must have a distribution type (although one distribution - binary - requires a distribution type but not upper and lower bounds). Any row that contains a formula does not need a distribution type. See thje Distribution Data Table for more information. Distribution types can be entered manually as a number or can be chosen from the pull-down menu in the distribution type column. Select the cell that you need to specify the type for and then select a distribution type from the list.

· The Source Reference – This column provides information about the source of quantity. It can be a short note or it can refer to a more detailed reference (perhaps an appendix). For variables that are calculated from other numbers (i.e. formula cells) the formula can be pasted here for reference.

See the figures on the following two pages for details.

Distribution Data Table

What each column contains

For rows that contain:

Dist. Type

Upper & Lower Bound

Best Estimate

Source Reference

A measurement with a Normal distribution

1

Represents the "90% confidence interval"

Test value only, not used in the distribution calculations

Specifies source of the measurement

A measurement with a Lognormal distribution

2

Represents the "90% confidence interval"; the absolute lower bound of a lognormal is always 0



A measurement with a Uniform distribution

3

Represents the absolute (100% certain) upper and lower bounds



An event with a certain chance of occuring (Binary)

4

Not applicable; should be empty

Represents the % chance of the event occuring

Specifies source of the event probability

A measurement with a Split Triangle distribution

5


Represents the median; the point where there is equal chance of the quantity being higher or lower


A measurement with a Right Triangle distribution

6




A measurement with a Left Triangle distribution

7




A fixed value (it has no uncertainty) as specified by some standard

8


Represents the fixed value of a number

Specifies the source of the standard that requires this fixed number

Custom distributions (measurements with distributions not listed above)

9

Depends on the specific nature of the distribution; could be 90% or 100% intervals or empty

Depends on the specific nature of the distribution; could be a median, mode, mean, percent chance or other


A spreadsheet calculation

NA


Contains the formula for the cell

Shows the text version of the formula

Example Spreadsheet for the Measurement Phase

The spreadsheet developed in the Clarification Phase is given more information in the Measurement Phase. We fill in the Lower Bound, Upper bound, Distribution Type and the Source Reference Columns.

4. Conduct Value of Information Analysis

We can optimize further measurements by determining the value of additional information about each of the variables in cost/benefit/risk analysis. Many variables, even though their uncertainty is seems large, may not justify additional measurements.

. Summary Procedure Data

Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:

None

Review analysis

Conduct analysis

None

None

Time Required

½ to 1 Day

Prerequisites

3. Conduct Measurements

Tools

The Monte Carlo Wizard

Excel Templates

Required/Optional

Required

Deliverable

Excel Spreadsheet with Cost/Benefit Model and initial estimates updated with EVPI (MVPI optional)

Purpose

The reduction of uncertainty always has some value in a decision but often it does not have enough value to justify the cost of additional analysis. Care must be taken so that time is not wasted by analyzing less important variables and critical information is not missed by failing to analyze more important variables.

The objective of this section is to calculate the value of additional analysis and make a decision to proceed with analysis or to terminate it.

Approach

The AIE method optimizes the analysis of variables in the cost/benefit model by determining which variables are most likely to alter the decision if additional information were available.

We will use a method from decision theory that requires only that we have defined the probability distributions for each of the variables in the CBA (which by this point, we have)

Procedure for computing the VIA

4.1. Go to the "Monte Carlo" worksheet

4.2. If you have added/moved any rows to Sheet 1, changed any distribution types, or modified values that effect the "Initial Investment" value then you must recreate the Monte Carlo Model before conducting the VIA:

4.2.1. If you have any custom distribution formulas already in the Random Scenario column they should be temporarilly saved to another column

4.2.2. Click on "Clear Monte Carlo Model" (optional). This is useful if you want to ensure you are starting with a clean slate.

4.2.3. Click on the "Create Monte Carlo Model" button. This creates a model with all the proper distribution formulas and it also resets all VIA flags to 0.

4.3. If you did not have to recreate the Monte Carlo Model, then be sure all VIA Flags are set to 0 (recreating the Model does this automatically)

4.4. Click on the "Clear VIA" if you want to clear old VIA values (optional). If you haven't moved any rows in the model then the next step will just overwrite old values. But if there are old values in rows that no longer have measures in them then these will still be there unless you clear the model.

4.5. Click on the "Compute VIA" button. This may take a one or two minutes while it computes the VIA data for each row.

Interpreting the results of VIA

The VIA tool creates 3 columns of data for each row that has a distribution type specified.

· The "Individual EVPI" (Expected Value of Information) column. This represents the value of perfect information about specific quantity. It only puts an extreme upper limit on what you should be willing to pay for the information. What you should realistically be willing pay is between 5% and 25% of the EVPI.

· The "Individual Threshold". This is the value that the quantity in this row must have for the investment to breakeven assuming all other quantities are held at their mean value.

· The "Threshold Probability". This is the chance that the quantity in that row could alter the decision (by changing the Net Present Value from positive to negative or vice versa) independently. Note that when the Threshold Probability is 0 then the Individual EVPI is 0. When the EVPI is large the Threshold Probability tends to be large.

The most important data to look at will be the Individual EVPI. The Individual Threshold and the Threshold probability are really just used to help calculate the EVPI. But sometimes it is helpful to condiser all the columns. For example:

· When you are using a distribution with absolute bounds (that is, upper and lower bounds that cannot be exceeded like a uniform or triangular distribution) you will get an EVPI of 0 anytime the Individual Threshold is outside of these bounds. Even if the threshold is very close. So when the threshold is close to, but just outside of, the bounds make sure you are absolutely certain the bounds cannot be exceeded. If you are not sure then either increase the range of the bounds or change to a distribution that does not have hard boundaries (like the normal distribution).

· When the EVPI is very large and it apparently justifies a major measurement effort for that one variable, do a reality check with the Individual Threshold and the Threshold Probability. Sometimes the EVPI will be large simply because the scale of the investment is large even though the Threshold Probability shows a very low chance of that quantity effecting the outcome. Small changes to the upper and lower bounds of numbers may have a significant effect on the EVPI in this case.

Task 4.5 Summarize Findings

The results of the VIA section can be assimilated as comments in section 3 (Measurement).

Be careful to record the findings in the spreadsheet in a different version than the one you will use for updated measurements.

The final results should simply be a clear statement about what, if any, additional measurement is required. If no additional measurement was required then you should clearly state that you have terminated analysis and will move on to section 5 – Risk/Return Analysis.

The 5. Conduct Risk & Return Analysis

No matter how many measurements we make, we must still make the final decision under some uncertainty. We will assess the uncertainty that we are actually making the wrong decision (i.e. risk) and plot the result on our firms Risk/Return profile.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:

Review


Time Required

1 to 2 Hours

Prerequisites

Excel spreadsheet completed through Monte Carlo columns

Tools/References

App. A: Customization Data (the risk/return profile)

Monte Carlo Wizard


Required/Optional

Depends on Classification

Deliverable

A risk return plot

Purpose

Even after an economically justified amount of analysis is completed we will still have uncertainty in our variables and we will still have to make a decision. The objective of this section is to assess the risk vs. the return of the proposed investment and base a decision on this foundation.

Approach

The method we will use is similar to the risk return analysis sometimes conducted by portfolio managers. We will start with your firm’s risk/return profile specified in Appendix A. This “risk/return boundary” for your firm specifies what chance of loss is acceptable for a given expected return.

Example Risk/Return Profile

Activities

5.1. Calculate the Risk/Return Boundary

5.2. Calculate the risk of the investment

5.3. Plot the risk and return

Activity 5.1 Calculate the Risk/Return Boundary

We should already have data on how much risk your firm is willing to take for a given expected return and a given investment size. We have to calculate from this the risk/return profile of your firm for the size of the investment under consideration.

Simply go to the AIE worksheet called Profile.xls and enter the expected cost of the investment. This will compute the acceptable risk for a series of different expected returns. You might wish to make a graph of these columns so that you have a visual aid like the one to the left.

Activity 5.2 Calculate the Investment Risk

5.2.1. Open the Monte Carlo Wizard in the project worksheet by clicking on this toolbar button:

5.2.2. Select the “Monte Carlo Cell” as the cell where the ROI is calculated

5.2.3. Check the “Histogram to Risk Report Page” box

5.2.4. Set “# of Iterations” to 10,000

5.2.5. Click "Run"

Setting the Monte Carlo Wizard for The Final Risk Return Analysis

Subtask 5.3 Plot the Risk and Return

We are defining the risk as the chance of a negative return. This quantity is reported in step 6.2.7. in the output from the Monte Carlo Wizard. You can visualize what this looks like with a histogram like the following.

Example Distribution of an IRR

The area under the curve represents a probability (total area = 100%). The area under the curve to the left of “0” is about 10% in this picture. Hence, the risk is stated as “a 10% chance of a negative return”

The average of all possible returns – the “expected” return – is about 75% ( a little right of the peak). This quantity is also generated as output by the Monte Carlo Wizard.

Now you just plot the Risk on the vertical axis of the profile graph you made in 6.1. and the expected ROI on the horizontal axis. In this example, the investment is acceptable even though there is about a 10% chance of a negative return.

Example of a Risk/Return Plot

For an IT Investment

6. Make Recommendation

The result of all this analysis is a concisely and clearly stated set of recommendations regarding the proposed investment.


Responsibilities

Sponsor

Estimator

Auditor

Judge

Other:

Review


Input on Follow-up measurement section

Review

None

Time Required

½ to 2 Days

Prerequisites


Tools

App.D : Document Templates

Required/Optional

Required

Deliverable

Excel Spreadsheet with Cost/Benefit Model and initial estimates

Purpose

The result of all our analysis is a clear and concise recommendation about the investment. We need to answer the following questions

Key Questions To Be Answered in the Recommendation

1. “Should we invest in this proposed project or not?”

2. “If you didn’t do a full analysis, why?”

3. “If I should invest in this project, what are the key implementation considerations?”

4. “If I should not invest in the project, can I change some of the parameters that may make a more viable project?”

5. “If I invest, How can I manage the key risks in this investment?”

6. “How will I know the investment turned out to be successful?”

Steps

1. Copy a blank template for a recommendations sections to your project document

2. In the headline portion of the template (see template description for details) write the recommendation clearly and conscisely – Tell the reader right away, should we invest? Yes or No.

3. If this was an expedient recommendation, state why

4. Paste a copy of the risk/return graph

5. Follow-up measurements

6. Implementation issues – preferably, a small Gantt chart

7. Managing the risk

8. Suggestions for modification of the project definition

Details for step 6. Managing Risk

Most of the “uncertainty reduction” so far was done through measurement of uncertain quantities. But another type of uncertainty reduction method can be utilized during the project itself. These methods are all proactive steps to alter the project development effort itself or the environment of the project. Instead of merely assessing the uncertainty about many of these factors we can take deliberate steps to effect the risk of the project. Any such steps should be incorporated into your recommendation as part of your “Managing the Project Risk” section.

· Could the project be reduced in size in some way that meets most of the requirements but significantly reduces the amount of work or chance of cancellation?

· Can project resources be confirmed for this project if there is uncertainty about resource availability?

· Is executive support (both users and IT) for the project uncertain? If so, is it feasible to confirm support for the project formally or request a delay until the project finds support?

· Could the project be delayed until after uncertain organizational changes have come to pass?

· Would scheduling periodic “Continuation Reviews” of the project reduce the chance that the project would only be cancelled after a lot of resources have been spent?

· Is it feasible to use purchased or existing software that captures most of the required fun

· Web viewApplied Information Economics ______________ The AIE Reference Manual. Version 3.1....

Documents

Transcript of · Web viewApplied Information Economics ______________ The AIE Reference Manual. Version 3.1....