A Detailed Comparison Between

Vose Software BVBA www.vosesoftware.com 1 | P a g e

Detailed comparison of

ModelRisk, @RISK and Crystal Ball

Crystal Ball is a product of Oracle Corporation. @RISK is a product of Palisade Corporation. Disclaimer: All reasonable attempts have been made to ensure that the comparison between ModelRisk and competitive products described in this document are factually based, fair and accurate. All software was used according to vendor specifications and versions tested were those available at the time of publication (July 2014). If you are aware of any inaccuracies or omissions in this text that should be corrected in order to maintain a fair and accurate comparison, please contact [email protected].


Introduction This technical review compares the three most common risk analysis Excel add-in software products:

Product Link to official intro video Company web site

Crystal Ball: Here www.oracle.com

ModelRisk: Here www.vosesoftware.com

@RISK: Here www.palisade.com

Summary of conclusions The best product choice depends on the factors you consider most important:

Lowest cost for similar level of features ModelRisk

Largest choice of distributions ModelRisk

Offers a choice of correlation tools ModelRisk

Most closely follows Excel protocol ModelRisk

Easiest to share results ModelRisk

Widest range of technical tools ModelRisk

Includes insurance and finance modeling ModelRisk

Widest range of results plots ModelRisk, @RISK

Linking to Microsoft Project in Excel - @RISK

Language other than English - @RISK or Crystal Ball

Approach The primary data source for this review is this1 independent and up-to-date overview of Excel risk

analysis add-ins on Wikipedia. This review only includes the Wikipedia information where there is a

difference between the software products. Aside from which technical capabilities each product has,

user-experience is also extremely important and a strong measure of the quality of the software as a

whole so, where possible, links have been provided to official videos produced by each developer so you

can compare the approaches. Please do take a look at them. This review is split into the following

sections:

Functionality

Technical specifications

Simulation controls

Reporting results

Help file, technical support and training

Implementation within Excel

Precision of the algorithms used in the software

Pricing

1 Last viewed 13 July, 2014


Functionality The following chart is plotted from the Wikipedia article which gave a numerical summary of the

number of features each product offered2

The three types of random variables that you tend to see in risk analysis modeling are:

Distributions

Correlation structures

Time series

Distributions - there are something like 150 distribution types that have a practical use and are not

simply a rehashed or renamed version of another one. You could say that 90% of users only need to use

10% of the types of distribution that are known, but which 10% varies enormously between the types of

problems that the users focus on. You can view videos of how the different products fit distributions to

data here: Crystal Ball3, ModelRisk, @RISK.

Correlation structures (copulas) are a very important and much overlooked component of risk modeling.

Failure to incorporate appropriate correlation will usually result in an underestimation of the risk. All of

the products reviewed effectively use a Gaussian copula which allows simulating a correlation structure

between any number of variables4. ModelRisk also provides the Clayton, Gumbel, Frank and Student

2 The table was removed from general view because of Wikipedias policy of not including original research (you can track the discussion that resulted in its removal on this page). 3 Official video not found 4 In fact, @RISK and Crystal Ball uses rank order correlation, a simulation technique from 1982 that closely resembles a Gaussian copula. See

Iman, R. L. and Conover, W. J., (1982). 'A Distribution-Free Approach to Inducing Rank Order Correlation Among Input Variables', Commun

Statist-Simula Computa 11(3) 311-334.


copulas. Bivariate versions can be rotated and all can be statistically fit to data. ModelRisk also offers an

empirical copula that will statistically reproduce unusual data patterns across any number of variables.

Correlating two Uniform distributions using the Gaussian copula with, for example, a 90% correlation

would produce a scatter pattern that looks like this:

But the different copulas will produce markedly different patterns for the same 90% correlation, for

example (reading from the top left: Clayton, inverted Clayton, Frank, Gumbel, inverted Gumbel,

Student):

For example, the Clayton copula (top left plot) is often used by banks for loan defaults because these are

strongly correlated in times of recession, but not in better times. The selection of correlation shape can

have a very significant effect on the tails of results when the correlation level is moderate (e.g. 10% to

50%). You can view the way ModelRisk fits copulas to data here, and for unusual patterns here.

Time series models are used to forecast variables across several periods, like exchange rates, inflation,

sales volumes. They incorporate at a minimum the trend and the level of randomness around that trend.

More sophisticated models incorporate characteristics like sudden jumps, cycles of high and low periods

of volatility, etc. The more sophisticated models are usually fit to data using software. You can view

videos of how the different products fit time series to data here: Crystal Ball Suite, ModelRisk, @RISK.


Technical specifications This section compares the key differences in the modeling tools that each product offers.

Source: Wikipedia

Notes

Distribution fitting a fundamental requirement unless your risk analysis model is only based on subjective estimates of uncertainty

Correlation fitting another fundamental requirement unless your risk analysis model is only based on subjective estimates of

uncertainty, or all your variables are unconnected (independent)

Time series fitting a fundamental requirement if you want to make forecasts based on historic data

Optimizer allows you to determine the optimal values of variables you control based on some target(s) like maximizing the mean

profit. All three products use the industry leader OptQuest optimizer, and @RISK includes other optimizers

Database connectivity useful if you have databases with large datasets that you want to fit distributions, correlation structures or

time series and that will update as new data are added to the database

VBA calls useful if you want to automatically run simulations, or functions that will internally perform simulations or probability

calculations, etc.

C++ calls for very advanced modelers who want to build their own specialist, high performance simulation programs

Six Sigma functionality a moderately useful feature for Six Sigma specialists that analyzes simulation results using Six Sigma

metrics

Probability calculations a very useful capability for advanced probability modelers because, amongst other reasons, one can

combine calculation and simulation together to create more precise and faster models.

User Defined Function error analysis an error message appears in the spreadsheet cell. For example:

Extreme vale modeling very useful in certain circumstances - many risk analysis issues are driven by extremes, like predicting the

size of the largest wave or wind gust, the lowest temperature, the largest insurance claim, the longest blackout, etc.

Expert elicitation tools to help the modeler get the most precise uncertain estimates from subject matter experts. If this is useful

to you, you should investigate each tools capabilities because there is a great deal of variation in what they offer

Data previsualizer allows you to review and explore the data before you start fitting distributions, time series, or correlations.

Again, a large variation in what is offered, so worth investigating the differences if this is important

Differentiation and integration a very advanced capability for either (i) building a system of ordinary differential equations with

uncertain inputs; or (ii) performing one-dimensional numerical integrations. Useful for scientists, engineers and finance/insurance

people mostly

Technical specification@RISK

Standard

@RISK

Professional

@RISK

Industrial

Crystal

Ball

Crystal Ball

Suite

ModelRisk

Standard

ModelRisk

Professional

ModelRisk

Industrial

Distribution fitting No Yes Yes Yes Yes No Yes Yes

Correlation fitting No No No No No No Yes Yes

Time series fitting No No Yes Yes Yes No Yes Yes

Optimizer included No No Yes No Yes No Yes Yes

Database connectivity No No No No No No No Yes

VBA calls to functions No Yes Yes Yes Yes Yes Yes Yes

C++ calls to functions No Yes Yes No No Yes Yes Yes

Six sigma supported Yes Yes Yes Yes Yes No No Yes

UDF error analysis No No No No No Yes Yes Yes

Extreme value modeling No No No No No No Yes Yes

Expert elicitation tools Yes Yes Yes No No No Yes Yes

Data previsualizer No No No No No No Yes Yes

ODE and numerical integration No No No No No No No Yes

Assumption library No Yes Yes No No No No Yes

Converters for CB CB CB No No CB, @RISK CB, @RISK CB, @RISK


Assumption library allows the user to store key assumptions that are used in various models. Again, a very wide range of

capabilities offered, so worth investigating the differences if this is important

Converter used if one changes to another software product, and wishes to keep using models developed in the old product (CB =

Crystal Ball)

Simulation controls This section compares the controls that are available to run simulations.

Source: Wikipedia

Notes:

Lock/unlock sampling switches off the Monte Carlo simulation, replacing random values with a predictable value like the median.

Useful in model auditing as one can compare values between different versions of a model

Apply sample to model allows the user to review the results data and force Excel to show a particular selected sample. This lets

you check, for example, why a particularly high or peculiar value was produced by the model. It also demonstrates to an auditor that

specific scenarios appear within the results, which is especially useful for banks and insurance companies

Random number generator advances in random number generators have made this essentially irrelevant. Nearly every product

uses the Mersenne Twister by default.

Spreadsheet interpreter this creates a compiled version of the Excel model and then runs this instead of within Excel. The result is

a much faster simulation time. Mostly useful if one has a large model that needs optimizing. The drawback is that not all Excel

models can be compiled.

Sampling method MCS = Monte Carlo simulation, LHS = Latin Hypercube sampling. We describe in a LinkedIn blog why LHS is no

longer important here.

Reporting results Risk analysis is a decision-making tool. That means that it is vital to be able to share the results of a risk

analysis effectively. The simulation results from a risk analysis model can be complex for a decision-

maker to understand, so a wide variety of customizable graphical reports is essential.

The greatest difference between the products is how they organize the storage and presentation of

results, and particularly how these results can be shared with people who are not licensed users of their

products, which is explained below.

Simulation controls@RISK

Standard

@RISK

Professional

@RISK

Industrial

Crystal

Ball

ModelRisk

Standard

ModelRisk

Professional

ModelRisk

Industrial

Lock / unlock random variables Yes Yes Yes No Yes Yes Yes

Apply specific sample in model Yes Yes Yes No Yes Yes Yes

Random number generator Several Several Several MCG Twister Twister Twister

Spreadsheet interpreter No No No Yes No No No

Sampling method MC, LHS MC, LHS MC, LHS MC, LHS MC MC MC

Reporting results@RISK

Standard

@RISK

Professional

@RISK

Industrial

Crystal

Ball

ModelRisk

Standard

ModelRisk

Professional

ModelRisk

Industrial

Free report viewer No No No No Yes Yes Yes

Spider Yes Yes Yes No Yes Yes Yes

Box Yes Yes Yes No Yes Yes Yes


Crystal Ball

Crystal Ball opens a new window for each graph after clicking a ribbon button. Examples:

Image source: Oracle Axoft

Results files can only be opened by another Crystal Ball user, though the charts can be copied as bitmaps

into Word, PowerPoint, etc.

ModelRisk

Results are shown in a separate application called the Results Viewer which opens at the end of the

simulation. The user can create and save an electronic report and that file can then be distributed to

others, without the need to share the model itself. The file can be opened by anyone who has installed

the ModelRisk Results Viewer, which is available free of charge from Vose Softwares web site here.

Image source: Vose Software

Aside from saving on licenses, the main advantages of this approach is that the report is electronic and

completely customizable labels and color schemes can be edited by the reviewer, new graphs can be

created, plots can be interrogated for say the 95th percentile instead of the 90th, and the whole

report can be resaved if desired. Plots can also be copied into Word, PowerPoint and other applications.


@RISK

@RISK opens a new window for each graph after clicking a ribbon button. Examples:

Image source: Palisade

Results files can only be opened by another @RISK user, though the charts can be copied as bitmaps into

Word, PowerPoint, etc.

Help file, technical support and training Technical support is a vital part of software delivery. Installation of software, particularly ad-ins to other

products, is not always simple because of access restrictions and security policies within a company and

sometimes requires liaising with the IT department. Moving software to a new computer can also be

very frustrating if one does not have ready access to technical support.

For the majority of risk analysis problems, the modeling techniques are not that complex, but some

basic training is really essential to avoid making the most common mistakes. Online videos that teach

the basics of driving the software are a good start, and a comprehensive help file saves a lot of time, but

if you are making multi-million dollar decisions based on a risk analysis model then we recommend

investing in 1-2 days of face-to-face, or at least online, training. We strongly recommend that the

training you purchase is not just about how to use the software. Instead, the course should focus on the

types of problems that you face, and the content tailored to show you the features of the software, the

particular distributions, etc. that you will need. The examples really need to be tailored to your industry

and problems to be effective, since people find it difficult to extrapolate from an example in another

field.

Help file, support

and training@RISK

Standard

@RISK

Professional

@RISK

Industrial

Crystal

Ball

ModelRisk

Standard

ModelRisk

Professional

ModelRisk

Industrial

UDFs linked to help file Yes Yes Yes No Yes Yes Yes

Online videos Yes Yes Yes No Yes Yes Yes

Online training Yes Yes Yes No Yes Yes Yes

Onsite training Yes Yes Yes No Yes Yes Yes

Language versionsZH, EN, FR, DE,

JA, PT, ES

ZH, EN, FR,

DE, JA, PT, ESEN, JA, ES EN EN

ZH, EN, FR,

DE, JA, PT, ESEN


Notes

UDFs linked to help file this means that when you open a dialog for a function, there is a link that will take you directly to the

relevant help file topic. Excel makes this quite easy for developers to do, and automatically adds a hyperlink to the dialog box, so it is

hard to see why this feature should not be implemented

Implementation within Excel Risk analysis add-ins for Excel are popular because Excel is essentially ubiquitous across businesses and

government agencies, and the modeling environment is intuitive and easy to use. At the same time,

there are many features like database connectivity, the VBA macro language, pivot tables, etc. that are

used by a small minority of power users but make it possible to develop very sophisticated models if

needed. Microsoft also encourages developers to produce add-ins and have a programming structure to

ensure that the add-ins will perform well with Excel. Excel add-ins for risk analysis therefore make a lot

of sense. Monte Carlo simulation add-ins for Excel reviewed here take two approaches to adding

uncertainty to the model.

Approach 1: Covering a cell with a distribution

The first approach, used in Crystal Ball, is to add an invisible layer to the spreadsheet. One selects a

spreadsheet cell that contains a value (not a formula), selects a distribution from the menu perhaps

with links to other cells, and clicks OK to enter the distribution. The cell background turns green, but the

original number that was in that cell remains unchanged. Some people like this because one can share

the model with non-Crystal Ball users and it still works as the original unrisked spreadsheet model,

although the simulation component is lost unless the other user has Crystal Ball too. It has its dangers,

however. You can enter a value of 100 in the cell, for example, and cover it with a uniform distribution

between 0 and 1. The incompatibility is not detected by the software and the distribution is only visible

if you click on a Crystal Ball button. The lack of visibility also makes it more difficult to audit and check a

model. Links within Crystal Ball are invisible to Excel, and Excels auditing tools do not work.

Approach 2: Inserting a random sampling function

The second approach is to provide user-defined functions, or UDFs. This is the standard method of

creating extra Excel functionality envisioned by Microsoft and employed by ModelRisk and @RISK. For

example, the following UDFs will take a random sample from a Triangular distribution with minimum,

most likely and maximum of 0, 40 and 100 respectively:

ModelRisk: =VoseTriangle(0,40,100)

@RISK: =RiskTriang(0,40,100)

The use of UDFs makes it possible to trace through the logic of the model using Excels Formula Auditing

tools as illustrated below:


A fundamental Excel rule is that there is a hierarchy of calculation in any formula. For example, in this

formula:

=SUM(SQRT(A1), A2)

Excel will evaluate A1 and A2, then evaluate SQRT(A1), and finally evaluate the SUM.

These rules are critical to having a predictable calculation outcome. However, this presents problems for

developers in creating functions that perform some operation on a distribution. For example, consider

the following @RISK function:

=RiskCompound(5, RiskLognorm(10,1))

The function sums 5 independent samples from a Lognormal(10,1) distribution. However, if we follow

Excels rules, the RiskLognorm function which is a random sampling function - must be evaluated

before the RiskCompound function. So, wed get something like this as an intermediary calculation:

=RiskCompound(5, 1.032)

in which case the RiskCompound would just be adding five 1.032 values together. To avoid this, @RISK

suppresses the evaluation of the RiskLognorm function. This causes problems of predicting how the

function behaves. For example, it is not immediately evident how the RiskCompound function should

behave in these circumstances:

=RiskCompound(5, RiskUniform(RiskUniform(2,3),4))

=RiskCompound(5, A2), where A2: =RiskNormal(A1,2)

ModelRisks approach is to use different functions when defining a distribution, called Objects, not the

same function used for sampling. Their equivalent function works as follows:

= VoseAggregateMC(5, VoseLognormalObject(10,1))

The LognormalObject function is used to define the distribution we are intending to use. Object

functions allow ModelRisk to comply with Excels evaluation rules. They are used in a variety of

ModelRisks tools.


Pricing All software vendors offer different editions of their products. In the following table, the products are

sorted according to price in US$ for a single perpetual (i.e. not time limited) license including the first

years maintenance and support. These prices appear on the following web sites, accessed on 10 July

2014: Crystal Ball5, ModelRisk6, @RISK7.

Notes

Some additional factors have been included that are not already described in previous sections. The choice is somewhat subjective

but focused on features useful to a modeler.

5 https://shop.oracle.com/pls/ostore/product?p1=oraclecrystalball&p2=&p3=&p4=&sc=ocom_crystalball 6 www.vosesoftware.com/purchasing.php 7 www.palisade.com

Price comparisonModelRisk

Standard

@RISK

StandardCrystal Ball

@RISK

Professional

ModelRisk

Professional

@RISK

Industrial

ModelRisk

Industrial

Crystal Ball

Suite

Perpetual license + 1st year support $895 $1,195 $1,213 $1,595 $1,645 $2,195 $2,295 $2,433

Cost of additional year's support $161 ? $218 ? $296 ? $413 $438# CPUs supported All 2 ? 2 All All All ?

Single click function view Yes No No No Yes No Yes No

Distribution fitting No No Yes Yes Yes Yes Yes Yes

Links to Microsoft Project No No No Yes No Yes No No

Aggregate modeling No No No Yes Yes Yes Yes No

Markov Chain simulation No No No No Yes No Yes No

Multivariate time series No No No No Yes No Yes No

Dynamic sorting No No No No Yes No Yes No

Optimizer No No No No Yes Yes Yes Yes

Stochastic data sharing No No No No No No Yes No

Specialist financial tools No No No No No No Yes No

Specialist insurance tools No No No No No No Yes No

Bayesian model averaging No No No No No No Yes No

A Detailed Comparison Between

Documents

Transcript of A Detailed Comparison Between