1

2

2kr Factorial Designs with Replications

• r replications of 2k Experiments– 2kr observations.– Allows estimation of experimental errors

• Model:y = q0 + qAxA + qBxB +qABxAxB+e

e=Experimental error

3

Computation of Effects• Simply use means of r measurements

I A B AB y Mean y

1111

-11

-11

-1-111

1-1-11

(15, 18, 12)(45, 48, 51)(25, 28, 19)(75, 75, 81)

15482477

16441

8621.5

389.5

205

TotalTotal/4

Effects: q0 = 41, qA = 21.5, qB = 9.5, qAB = 5

4

Estimation of Experimental Errors• Estimated Response:

yi = q0 + qAxAi + qBxBi +qABxAixBi

• Experimental Error = Estimated-Measured

eij = yij - yi

= yij - q0 - qAxAi - qBxBi - qABxAixBi

• Sum of Squared Errors:

SSE =

r

iie

22

1

2

^

^

0,

ji

ije

5

Experimental Errors: Example

Estimated Response:y1 = q0 - qA - qB + qAB = 41 -21.5 -9.5 +5 = 15

Experimental errors:e11 = y11 - y1 = 15 - 15 = 0

^

^

i I A B AB Response Responses Errors41 21.5 9.5 5 yi yi1 yi2 yi3 ei1 ei2 ei3

1 1 -1 -1 1 15 15 18 12 0 3 -32 1 1 -1 -1 48 45 48 51 -3 0 33 1 -1 1 -1 24 25 28 19 1 4 -5

4 1 1 1 1 77 75 75 81 -2 -2 4

Effect Estimated Measured

SSE = 02 + 32 + (-3)2 + (-3)2 + ... + 42 = 102

^

6

Allocation of Variation

Total variation or total sum of squares:SST =

jiij yy

,

2..)(

ijBiAiABBiBAiAij exxqxqxqqy 0

ji

ijABBAji

ij erqrqrqyy,

2222222

,

2 222..)(

SST = SSA + SSB + SSAB + SSE

7

DerivationModel:

ijBiAiABBiBAiAij exxqxqxqqy 0

Since x’s, their products, and all errors add to zero

jiij

jiBiAiAB

jiBiB

jiAiA

jijiij

exxqxq

xqqy

,,,

,,0

,

02

,0

,

2 rqqyjiji

ij

8

Derivation (cont’d)Mean response:

0,

22

1.. qy

ry

jiij

Squaring both sides of the model and ignoring cross product terms:

jiij

jiBiAiAB

jiBiB

jiAiA

jijiij

exxqxq

xqqy

,

2

,

222

,

22

,

22

,

20

,

2

SSY = SS0 + SSA + SSB + SSAB + SSE

9

Derivation (cont’d)Total Variation:

SSESSABSSBSSA

SSSSY

yy

yySST

jijiij

jiij

0

..

,

2

..,

2

,

2

One way to compute SSE:

22220

22 ABBA qqqqrSSYSSE

10

Example: Memory-Cache Study

27204

81757545121815 2222222

SSY

703220172272040

102)55.95.2141(3227204

3005122

1083)5.9(122

5547)5.21(122

20172411220

22222

222

222

222

220

2

SSSSYSST

SSE

rqSSAB

rqSSB

rqSSA

rqSS

AB

B

A

11

Example: Memory-Cache Study(cont’d)

SSA + SSB + SSAB + SSE=5547 + 1083 + 300 + 102= 7032 = SST

Factor A explains 5547/7032 or 78.88%Factor B explains 15.40%Interaction AB explains 4.27%1.45% is unexplained and is attributed to errors.

12

Review: Confidence Interval for the Mean

Problem: How to get a single estimate of the population mean from k sample estimates?Answer: Get probabilistic bounds.

Eg., 2 bounds, C1 & C2 There is a high probability, 1-,

that the mean is in the interval (C1, C2 ):

Pr {C1 C2} = 1 - • Confidence interval (C1, C2 )

• Significance Level• 100 (1-) Confidence Level• 1- Confidence Coefficient.

13

Confidence Interval for the Mean (cont’d)

Note: Confidence Level is traditionally expressed as a percentage (near 100%); whereas, significance level , is expressed as a fraction & is typically near zero; e.g., 0.05 or 0.01.

14

Confidence Interval for the Mean (cont’d)

Example:Given sample with:

mean = x = 3.90SD = s = 0.95n = 32

A 90 % CI for the mean = 3.90 + (1.645)(0.95)/= (3.62, 4.17), used the central limit theorem.

Note: A 90 % CI => We can state with 90 % confidence that the population mean is between 3.62 & 4.17. The chance of error in this statement is 10 %

32

15

Testing for a Zero Mean

Difference in processor times of two different implementations of the same algorithms was measured on 7 similar workloads. The differences are:

{1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}Can we say with 99 % confidence that one implementation is superior to the other

16

Testing for a Zero Mean (cont’d)Sample size = n = 7mean = x = 1.03sample variance = s2 = 2.57sample deviation = s = 1.60CI = 1.03 t x 1.60/ = 1.03 0.605t 100 (1- ) = 99, = 0.01, 1- /2 = 0.995

From Table, the t value at six degrees of freedom is:

t[0.995; 6] = 3.707 & the 99% CI = (-1.21, 3.27).

Since the CI includes zero, we can not say with 99%

confidence that the mean difference is significantly

different from Zero.

7

17

Type I & Type II ErrorsIn testing a NULL, hypothesis, the level of significance is the probability of rejecting a true hypothesis.

HYPOTHESIS

Actually True Actually False

Correct Error (Type II)

Correct Error (Type I)

To Accept

To Reject

DECISION

Note: The letters & denote the probability related to these errors

18

Since q0 = Linear combination of normal variables=> q0 is normal with variance

Variance of errors:

Effects are random variables.Errors ~ N(0,σe) => y ~ N( y.., σe)

Confidence Intervals For Effects

)2/( 22 re

MSEr

SSEe

rs

ijije

)1(2)1(2

12

22

2

19

Confidence Intervals For Effects (cont’d)

Denominator = 22(r - 1) = # of independent terms in SSE=> SSE has 22(r - 1) degrees of freedom.Estimated variance of q0 :

Similarly,

Confidence intervals (CI) for the effects:

CI does not include a zero => significant

)2/( 222

0rss eq

r

Sqqq

e

ABBAsss

22

iqri stq)1(2;2/1 2

20

Example

For Memory-cache study:Standard deviation of errors:

57.375.128102

)1(22 r

SSEes

Standard deviation of effects:

03.112/57.32/ 2 rss eqi

For 90% Confidence : 86.18,95.0 t

21

Example (cont’d)Confidence intervals:

)91.6,08.3(

)41.11,58.7(

)41.23,58.19(

)91.42,08.39(

92.1)03.1)(86.1(

0

AB

B

A

ii

q

q

q

q

qq

No zero crossing=> All effects are significant.

22

Confidence Intervals for Contrasts

Contrast Linear combination with coefficients = 0

Variance of hiqi:

r

hss ie

qh ii 2

222

2

For 100 ( 1 - ) % confidence interval,use .)1(2;2/1 2 r

t

23

u = qA + qB -2qAB

Coefficients = 0,1,1, and -2 => ContrastMean u = 21.5 + 9.5 - 2 x 5 = 21

Variance

Example: Memory-cache study

375.632

62

22

eu

ss

Standard deviation 52.2375.6 ust[0.95;8] = 1.8690% Confidence interval for u :

52.286.121 utsu (16.31, 25.69)

24

Mean response y :y = q0 + qA xA + qB xB + qABxA xB

The standard deviation of the mean of m response:

CI for Predicted Response

^^

21

11ˆ mney effm

ss

neff = Effective deg of freedom

=Total number of runs

1 + Sum of DFs of params used in y

5

22 r

25

CI for Predicted Response (cont’d)

21

225

ˆ:)(rey ssm

100 ( 1 - ) % confidence interval:

myrsty ˆ)1(2;2/1 2ˆ

A single run (m = 1) : 21

21

12

5ˆ

rey ss

Population mean

26

Example: Memory-cache Study

For xA = -1 and xB = -1:•A single confirmation experiment:y1 = q0 - qA - qB + qAB

= 41 - 21.5 - 9.5 + 5 = 15Standard deviation of the prediction:

^

25.4157.31 125

25

ˆ2

1

21

rey ss

Using t[0.95;8]=1.86, the 90% confidence interval is:

)91.22,09.8(25.486.115

27

Example: Memory-cache Study (cont’d)

•Mean response for 5 experiments in future:

80.257.3 51

1251

25

ˆ2

1

21

mrey ss

The 90% confidence interval is:

)29.20,79.9(80.286.115 •Mean response for a large number of experiments in future:

30.257.3 125

25

ˆ2

1

2 rey ss

The 90% confidence interval is:

)28.19,72.10(30.286.115

28

Example: Memory-cache Study (cont’d)

•Current mean response: Not for future.(Use the formula for contrasts):

06.212475.12

2ˆ 2

22

r

hs

yies

90% confidence interval:

)83.18,17.11(06.286.115 Notice: Confidence intervals become narrower.

29

Assumptions

1. Errors are statistically independent.2. Errors are additive.3. Errors are normally distributed4. Errors have a constant standard deviation e.5. Effects of factors are additive.

=> observations are independent and normally distributed with constant variance.