Simulation Modeling and Analysis

Session 12

Comparing Alternative System Designs

Outline

• Comparing Two Designs

• Comparing Several Designs

• Statistical Models

• Metamodeling

Comparing two designs

• Let the average measures of performance for designs 1 and 2 be 1 and 2.

• Goal of the comparison: Find point and interval estimates for 1 - 2

Example• Auto inspection system design

• Arrivals: E(6.316) min

• Service: – Brake check N(6.5,0.5) min– Headlight check N(6,0.5) min– Steering check N(5.5,0.5) min

• Two alternatives: – Same service person does all checks– A service person is devoted to each check

Comparing Two Designs -contd

• Run length (ith design ) = Tei

• Number of replications (ith design ) = Ri

• Average response time for replication r (ith design = Yri

• Averages and standard deviations over all replications, Y1* = Yri / Ri and Y2* , are unbiased estimators of 1 and 2.

Possible outcomes• Confidence interval for 1 - 2 well to the

left of zero. I.e. most likely 1 < 2.

• Confidence interval for 1 - 2 well to the right of zero. I.e. most likely 1 > 2.

• Confidence interval for 1 - 2 contains zero. I.e. most likely 1 ~ 2.

• Confidence interval

(Y1* - Y2*) ± t /2, s.e.(Y1* - Y2*)

Independent Sampling with Equal Variances

• Different and independent random number streams are used to simulate the two designs.

Var(Yi*) = var(Yri)/Ri = i2/Ri

Var(Y1* - Y2*) = var(Y1*) + var(Y2*)

= 12/R1 + 2

2/R2 = VIND

• Assume the run lengths can be adjusted to produce 1

2 ~ 22

Independent Sampling with Equal Variances -contd

• Then Y1* - Y2* is a point estimate of 1 - 2

Si2 = (Yri - Yi*)2/(Ri - 1)

Sp2 = [(R1-1) S1

2 + (R2-1) S22]/(R1+R2-2)

s.e.(Y1*-Y2*) = Sp (1/R1 + 1/R2)1/2

= R1 + R2 -2

Independent Sampling with Unequal Variances

s.e.(Y1*-Y2*) = (S12/R1 + S2

2/R2)1/2

= (S12/R1 + S2

2/R2)2/M

where

M = (S12/R1)2/(R1-1) + (S2

2/R2)2/(R2-1)

• Here R1 and R2 must be > 6

Correlated Sampling• Correlated sampling induces positive

correlation between Yr1 and Yr2 and reduces the variance in the point estimator of Y1*-Y2*

• Same random number streams used for both systems for each replication r (R1 = R2 = R)

• Estimates Yr1 and Yr2 are correlated but Yr1 and Ys2 (r n.e. s) are mutually independent.

Recall: Covariance

var(Y1* - Y2*) = var(Y1* ) + var(Y2* ) -

2 cov(Y1* , Y2* ) =

12/R + 2

2/R - 2 12 1 2/R = VCORR

= VIND - 2 12 1 2/R Recall: definition of covariance

cov(X1,X2) = E(X1 X2) - 1 2 =

= corr(X1 X2) 1 2 =

= 1 2

Correlated Sampling -contd• Let Dr =Yr1 - Yr2

D* = (1/R) Dr = Y1* - Y2*

SD2 = (1/(R - 1)) (Dr - D*)2

• Standard error for the 100(1- )% confidence interval

s.e.(D*) = s.e.(Y1* - Y2* ) = SD/ R

(Y1* - Y2*) ± t /2, SD/ R

Correlated Sampling -contd

• Random Number Synchronization Guides– Dedicate a r.n. stream for a specific purpose

and use as many streams as needed. Assign independent seeds to each stream at the beginning of each run.

– For cyclic task subsystems assign a r.n. stream.– If synchronization is not possible for a

subsystem use an independent stream.

Example: Auto inspection

An = interarrival time for vehicles n,n+1

Sn(1) = brake inspection time for vehicle n in model 1

Sn(2) = headlight inspection time for vehicle n in model 1

Sn(3) = steering inspection time for vehicle n in model 1

• Select R = 10, Total_time = 16 hrs

Example: Auto inspection

• Independent runs

-18.1 < 1-2 < 7.3

• Correlated runs

-12.3 < 1-2 < 8.5

• Synchronized runs

-0.5 < 1-2 < 1.3

Confidence Intervals with Specified Precision

• Here the problem is to determine the number of replications R required to achieve a desired level of precision in the confidence interval, based on results obtained using Ro replications

R = (t /2,Ro SD/

Comparing Several System Designs

• Consider K alternative designs

• Performance measure i

• Procedures– Fixed sample size– Sequential sampling (multistage)

Comparing Several System Designs -contd

• Possible Goals– Estimation of each i

– Comparing i to a control 1

– All possible comparisons

– Selection of the best i

Bonferroni Method for Multiple Comparisons

• Consider C confidence intervals 1-i

• Overall error probability E = j

• Probability all statements are true (the parameter is contained inside all C.I.’s)

P 1 - E

• Probability one or more statements are false

P E

Example: Auto inspection (contd)

• Alternative designs for addition of one holding space– Parallel stations– No space between stations in series– One space between brake and headlight

inspection– One space between headlight and steering

inspection

Bonferroni Method for Selecting the Best

• System with maximum expected performance is to be selected.

• System with maximum performance and maximum distance to the second best is to be selected.

i - max j i j

Bonferroni Method for Selecting the Best -contd

1.- Specify , and R0

2.- Make R0 replications for each of the K systems

3.- For each system i calculate Yi*

4.- For each pair of systems i and j calculate Sij2 and select the

largest Smax2

5.- Calculate R = max{R0, t2 Smax2 / 2}

6.- Make R-R0 additional replications for each of the K systems

7.- Calculate overall means Yi** = (1/R) Yri

8.-Select system with largest Yi** as the best

Statistical Models to Estimate the Effect of Design Alternatives

• Statistical Design of Experiments– Set of principles to evaluate and maximize the

information gained from an experiment.

• Factors (Qualitative and Quantitative), Levels and Treatments

• Decision or Policy Variables.

Single Factor, Randomized Designs

• Single Factor Experiment– Single decision factor D ( k levels)– Response variable Y

– Effect of level j of factor D, j

• Completely Randomized Design– Different r.n. streams used for each replication

at any level and for all levels.

Single Factor, Randomized Designs -contd

• Statistical model

Yrj = + j + rj

whereYrj = observation r for level j

= mean overall effect

j = effect due to level j

rj = random variation in observation r at level j

Rj= number of observations for level j

Single Factor, Randomized Designs -contd

• Fixed effects model– levels of factors fixed by analyst rj normally distributed

– Null hypothesis H0: j = 0 for all j=1,2,..,k

– Statistical test: ANOVA (F-statistic)

• Random effects model– levels chosen at random j normally distributed

ANOVA Test

• Levels-replications matrix

• Compute level means (over replications) Y.i* and grand mean Y..*

• Variation of the response w.r.t. Y..*

Yrj - Y..* = (Y.j* - Y..*) + (Yrj - Y.j*)

• Squaring and summing over all r and j

SSTOT = SSTREAT + SSE

ANOVA Test -contd• Mean square MSE = SSE/(R-k) is unbiased estimator of

var(Y). I.e. E(MSE) = 2

• Mean square MSTREAT = SSTREAT/(k-1) is also unbiased estimator of var(Y).

• Test statistic

F = MSTREAT / MSE

• If H0 is true F has an F distribution with k-1 and R-k d.o.f.

• Find critical value of the statistic F1-

• Reject H0 if F > F1-

Metamodeling

• Independent (design) variables xi, i=1,2,..,k

• Output response (random) variable Y

• Metamodel– A simplified approximation to the actual

relationship between the xi and Y

– Regression analysis (least squares)– Normal equations

Linear Regression

• One independent variable x and one dependent variable Y

• For a linear relationship

E(Y:x) = 0 + 1 x

• Simple Linear Regression Model

Y = 0 + 1 x +

Linear Regression -contd• Observations (data points)

(xi,Yi) i=1,2,..,n

• Sum of squares of the deviations i2

L = i2 = [ Yi - 0

’ - 1(xi - x*)]2

• Minimizing w.r.t 0’ and 1 find

0’* = Yi /n

1*

= Yi (xi - x*)/ (xi - x*)2

0* = 0

’* - 1*

x*

Significance Testing• Null Hypothesis H0: 1 = 0

• Statistic (n-2 d.o.f)

t0 = 1*

/(MSE/Sxx)

where

MSE = (Yi - Ypi)/(n-2)

Sxx = xi2 - ( xi )2/n

• H0 is rejected if |t0| > t/2,n-2

Multiple Regression

• Models

Y = 0 + 1 x1 + 2 x2 + ... + m xm +

Y = 0 + 1 x + 2 x2 +

Y = 0 + 1 x1 + 2 x2 + 3 x1 x2 +