Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48,...
-
date post
21-Dec-2015 -
Category
Documents
-
view
220 -
download
0
Transcript of Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48,...
Automating the Analysis of Simulation Output Data
Stewart Robinson, Katy Hoad, Ruth Davies
OR48, September 2006
Outline
The problem
A prototype automated output Analyser
Findings from prototype Analyser
The AutoSimOA Project
Current work - Collecting and characterising real and artificial models
The Problem
Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts.
Simulation software generally have very limited facilities for directing/advising on simulation experiments.
Main exception is directing scenario selection through ‘optimisers’.
With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.
The Problem
Despite continued theoretical developments in simulation output analysis, little is being put into practical use.
There are 3 factors that seem to inhibit the adoption of output analysis methods:
• Limited testing of methods• Requirement for detailed statistical knowledge• Methods generally not implemented in simulation
software (AutoMod/AutoStat is an exception)
A solution would be to provide an automated output ‘Analyser’.
A Prototype Analyser
Simulationmodel
Warm-upanalysis
Run-lengthanalysis
Replicationsanalysis
Use replicationsor long-run?
Recommendationpossible?
Recommend-ation
Output data
Analyser
Obt
ain
mor
e ou
tput
dat
a
Masters Project (3 students).
The Analyser looked at:
• Warm-up
• Run-length
• Number of replications
Scenario analysis could be added.
A Prototype Analyser
A prototype Analyser has been developed in Microsoft Excel.
At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.
Illustration: Warm-up
Load Analyser into Excel.
Enter name of SIMUL8 model.
Specify initial number of replications and run-length to use.
Welch's Method: Plot of Moving Average (Window = 12 )
0.00
20.00
40.00
60.00
80.00
100.00
120.00
1 84 167 250 333 416 499 582 665 748 831 914
Observation
Mo
vin
g a
vera
ge
Illustration: Warm-up
Illustration: Replications
Findings from Prototype Analyser
It is possible to link an Automated Analyser in Excel to a simulation software tool.
This was just a proof of concept.
Key issues to address:• More thorough testing of output analysis methods for their
accuracy and their generality.• Adaptation of methods to sequential procedures and to
minimise the need for user intervention.
A 3 year, EPSRC funded project (GR EP/D033640/1) in collaboration with SIMUL8 Corporation.
The AutoSimOA Project
Objectives
• To determine the most appropriate methods for automating simulation output analysis
• To determine the effectiveness of the analysis methods• To revise the methods where necessary in order to
improve their effectiveness and capacity for automation• To propose a procedure for automated output analysis of
warm-up, replications and run-length
Only looking at analysis of a single scenario
The AutoSimOA Project
CURRENT WORK:
1. Literature review of warm-up, replications and run-length methods.
2. Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…)
3. Collection of ‘real’ simulation models.
Use models / data sets:
Provide a representative and sufficient set of models / data output for use in discrete event simulation research.
Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..
Categorising Output Data Sets by Shape & Characteristics
Group A
…Group NGroup B
Auto Correlation Spread round mean
NormalityTrend
Cycling/Seasonality
Terminating
Non-terminating
Steady state
In/out of control
Transient
Model characteristics
Deterministic or random
Significant pre-determined model changes (by time)
Dynamic internal changes i.e. ‘feed-back’
Empty-to-empty pattern
Initial transient (warm-up)
Out of control trend ρ≥1
Cycle
Auto-correlation
Statistical distribution
Output data characteristics
ARTIFICIAL MODELS
Create simple models where theoretical value of some attribute is known.
E.g. M/M/1: mean waiting time.
Create simple models where value of some attribute is estimated but model characteristics can be controlled.
E.g. Single item inventory management system: Number-in-stock.
Construct output, which closely resembles real model output, with known value for some specific attribute.
E.g. AR(1) with Normal errors
Create different output types Transient
Steady state
Steady state cycle
Trend + Initial transient (warm-up)
Example artificial models:
1. Auto-Regressive (2) series
BiasFnXXX ttt 21 5.025.0
1005.010 te
220010 1005.0 t
Sine t
Exponential
Under Damped oscillations
Mean shift 2
Initial Bias Functions:
Run1 ~ AR(2) + "underdamped oscillations" initial bias
-30
-20
-10
0
10
20
30
40
0 100 200 300 400 500
t
Run1 ~ AR(2) + "mean shift" initial bias
-6
-4
-2
0
2
4
6
8
10
12
14
0 100 200 300 400 500t
Run1 ~ AR(2) + "exponential" initial bias
-10
-5
0
5
10
15
20
25
30
35
40
0 100 200 300 400 500t
Run1 ~ AR(2) with no initial bias
-4
-3
-2
-1
0
1
2
3
4
5
0 100 200 300 400 500
t
mean 1.8
Example artificial models:
2. E4 ~ Erlang(4) / M / 1 Queue
Traffic Intensity = 0.8
Queuing time for each customer in a E4/M/1Queuing System
0
2
4
6
8
10
12
14
16
1 432 863 1294 1725 2156 2587 3018 3449 3880 4311 4742
index
REAL MODELS
Models created in “real circumstances” that cover each
general type of model and output encountered in real life modeling.
e.g. Call centre: percentage of calls answered within 30
secs
e.g. Production Line Manufacturing Plant:
through-put / hour
e.g. Fast Food Store:
average queuing timee.g. Swimming
Pool complex: average
number in system
TransientSteady State
Cycle
Steady State
With or without warm-up
Trend
0
2
4
6
8
10
12
14
16
18
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480
time (mins)
num
ber
of custo
mers
0
5
10
15
20
25
30
35
40
45
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480
time (mins)
num
ber
of custo
mers
Example ‘ real ’ models:1. Argos – Number of customers in queue to pay
Stochastic model with changing arrival rates.
Empty to empty; transient; autocorrelated; non-normal output.
0
50
100
150
200
Time (hours)
Num
ber
of ite
ms
Example ‘ real ’ models:2. Leggings Manufacturing Plant – Through-put / hour
Stochastic model.
Steady state with warm-up; not autocorrelated; normal output.
0
50
100
150
200
Time (hours)
Num
ber
of ite
ms
0
10
20
30
40
50
60
1 5 9 13 17 21 25 29 33 37 41
Time (hours)
Num
ber
of
com
ple
ted ite
ms
Example ‘ real ’ models:3. Sanitory Towel Packing Plant – Through-put / hour
Stochastic model with changing productivity in work stations.
Steady state daily cycle.
40
41
42
43
44
45
1 10 19 28 37 46 55 64 73 82 91 100 109 118
Time (days)
Mea
n th
roug
h-pu
t for
the
day
Series of means of each cycle:
autocorrelated; non-normal output.
Use this representative and sufficient set of models/outputwhen
The AutoSimOA Project
• determining the most appropriate methods for automating simulation output analysis
• determining the effectiveness of the analysis methods
• revising the methods where necessary in order to improve their effectiveness and capacity for automation
In order to propose a procedure for automated output analysis of warm-up, replications and run-length.
Automating the Analysis of Simulation Output Data
Stewart Robinson, Katy Hoad, Ruth Davies
OR48, September 2006