Design of Screening Experiments with Partial Replication

David J. EdwardsDepartment of Statistical Sciences & Operations Research

Virginia Commonwealth University

Robert D. LeonardDepartment of Information Systems and Analytics

Miami University

DATAWorks

March 21, 2018

Screening Experiments• Screening Experiments are frequently utilized in

the early stages of experimentation• Goal of screening is to identify important main

effects and perhaps some important two-factorinteractions

• Typical screening designs are• Unreplicated strength 2 or 3 orthogonal arrays• Designs constructed using variance-based optimality

criteria (such as D-optimality)

To Pool or Not to Pool … That is the Question? • Screening designs are usually unreplicated

• No unbiased estimate of experimental error.

• Inference is carried out by:• A robust estimator for the error variance based on a fitted

saturated model (e.g., Lenth’s Pseudo Standard Error)• Pooling sums of squares from terms identified as not active

• If statistical inference is an integral part of the analysis,one view is that inference should be performed using apure error estimate versus the residual mean squarefrom the fitted model.

Partial Replication

• From Lupinacci and Pigeon (2008, JQT): “…evenin the initial stages of an experimental program,it might be desirable or even necessary for theexperimenter to have a realistic estimate of theexperimental error.”

• Since full replication of an experiment may bequite costly, partial replication offers a cost-saving alternative for obtaining a modelindependent error estimate.

Motivation

Gilmour, S.G. and Trinca, L.A. (2012). “Optimum Design of Experiments for Statistical Inference”. Journal of the Royal Statistical Society, Series A, 61(3), 345-401.

DP-optimality• Based upon the formulation for the volume of a

confidence interval region for휷, which is proportional to

퐹 , ;/푿 푿 / ,

where p is the number of parameters in the assumed model, 푑 is the pure error degrees of freedom, and 퐹 , ; is the 1− 훼 -quantile of the 퐹-distribution.

• The so-called DP-optimal design is then defined as that which maximizes

푿′푿 / 퐹 , ; .

DP-optimality

As inference is rarely performed on the intercept, it can be treated as a nuisance parameter and hence, the DP-optimal design maximizes

푿 푸푿 / 퐹 , ;

where 푿 is the model matrix without a column of ones, 푸 = 퐈 − 푱 , 푰 is the 푛 × 푛 identity matrix, and 푱 is the 푛 × 푛 matrix of ones.

DP-optimal Designs• DP-optimal designs tend to

have an excessive number of replicate runs and seem inefficient regarding the use of available degrees of freedom in screening situations.

- - - + - + -- - - + - + -- - - + - + -- - + - - - +- - + - - - +- - + - - - +- + - + + - +- + - + + - +- + - + + - +- + + - + + -- + + - + + -- + + - + + -+ - - - + + ++ - - - + + ++ - - - + + ++ - + + + - -+ - + + + - -+ - + + + - -+ + - - - - -+ + - - - - -+ + - - - - -+ + + + - + ++ + + + - + ++ + + + - + +

7-factor, 24-run DP-optimal design with a priori main effects model.

No lack-of-fit degrees of freedom!

Compound DP-optimality• To overcome this potential deficiency and encourage the

availability of degrees of freedom for testing lack-of-fit,Gilmour and Trinca (2012) propose to incorporate degreeof freedom efficiency (푑푓 = (푛 − 푑)/푛 ) into theformulation of a compound criterion.

• For example, combining the DP-criterion with degrees offreedom efficiency, we maximize

푿 푸푿 푛 − 푑퐹 , ;

where 휿 = (휅 , 휅 ) are weights.

Compound DP-optimality

• This modification helps…• 휿 = 0.5, 0.5 produces a design with 8

degrees of freedom for pure error (dfpe) and 8 for lack-of-fit (dflof).

• 휿 = 0.2, 0.8 has 5 dfpe and 11 dflof

• Despite this, design construction still critically depends on the choice of a priori model.

Bayesian D-optimality• DuMouchel and Jones (1994)• Primary terms are those effects assumed to be active

while potential terms may or may not be active• Joint prior distribution of the unknown model

parameters is휷|휎 ~푁 ퟎ,휎 푹 , where푹 = 푲/휏is the prior covariance matrix and

푲 =ퟎ × ퟎ ×ퟎ × 푰 ×

• The posterior distribution for 휷 given 풚 is given as휷|풚~푁 푿 푿+ 푹 푿 풚,휎 푿 푿 + 푹

Bayesian D- and DP-optimality

• A Bayesian D-optimal design, 휉∗, maximizes|푿 푿 + 푹|

• Extending the formulation of the DP-optimality criterion within the Bayesian framework, we suggest the Bayesian 퐷푃-optimality criterion which maximizes

푿 푸푿 + 푲∗

휏퐹 , ;

푲∗ =ퟎ( )×( ퟎ( )×

ퟎ ×( 푰 ×

where

Some Comparisons

풅풇풑풆 풅풇푳풐푭 D-eff DP-eff

Design I (D-opt)

0 16 1.00 0.00

Design II (BD-opt)

0 16 0.94 0.00

Design III (DP-opt)

16 0 1.00 1.00

Design IV (BDP-opt)

6 10 0.90 0.57

cDP(0.5, 0.5)

8 8 1.00 0.76

cDP(0.2, 0.8)

5 11 1.00 0.54

Protecting Against Model Misspecification

푨 = 푿 푿 푿 푿 is the so-called alias matrix and trace(AA’) (to be minimized) can be viewed as a measure of the potential impact of bias

풅풇풑풆 풅풇푳풐푭 D-eff DP-eff Primary & Primary

Correlation

Primary & Potential

Correlation

Potential & Potential

Correlation

풕풓(푨푨′)

Design I (D-opt)

0 16 1.00 0.00 0.000 0.109 0.071 5.333

Design II (BD-opt)

0 16 0.94 0.00 0.095 0.027 0.092 1.113

Design III (DP-opt)

16 0 1.00 1.00 0.000 0.143 0.100 21.000

Design IV (BDP-opt)

6 10 0.90 0.57 0.175 0.041 0.228 1.873

cDP(0.5, 0.5) 8 8 1.00 0.76 0.000 0.116 0.081 10.333

cDP(0.2, 0.8) 5 11 1.00 0.54 0.000 0.157 0.100 7.667

Additional Comparisons• Example 1: Two-levels, 7-factors (MEs primary, 2FIs potential)

Additional Comparisons• Example 2: 3-levels, 7-factors (MEs and Pure Quadratics

primary and 2FIs potential)

Design Choices via Multiple Criteria

• It is possible for multiple designs to satisfy anygiven optimality criterion, in which caseadditional criteria may be used to furtherdiscriminate between choices of optimal designs.

• Lu et al. (2011, Technometrics) have shown thata suite of designs can be reduced to a smaller setof Pareto-optimal choices by combining multipledesign criteria as a weighted sum of scaledcriteria values.

Multiple Criteria• Four sets of 100 designs are created using each of D-, DP-, Bayesian

D-, and Bayesian DP-optimality. • Criterion values (dfpe, D-eff, and 푡푟(푨푨 )) are calculated for all non-

dominated designs and scaled so that the maximum value is one and minimum value is zero.

• Next, 퐿 norms are calculated as

푤 푓 휉 − 푓 ,

where 푓 휉 , 푓 휉 , 푓 휉 are the scaled values, 푭 = 1, 1, 0 since we wish to maximize both dfpe and D-eff and

minimize 푡푟 푨푨 ; and 푤 are weights• For each choice of weight, the design which minimizes the L1 norm is

selected.

Revisiting Example 1

Revisiting Example 1

20

풅풇풑풆 풅풇푳풐푭 D-eff DP-eff Primary & Primary

Correlation

Primary & Potential

Correlation

Potential & Potential

Correlation

풕풓(푨푨 )

Design I (D-opt)

0 16 1.00 0.00 0.000 0.109 0.071 5.333

Design V (D-opt)

1 15 1.00 0.01 0.000 0.082 0.081 4.000

Design VI(D-opt)

0 16 1.00 0.00 0.000 0.075 0.081 3.667

Design IV (BDP-opt)

6 10 0.90 0.57 0.175 0.041 0.228 1.873

Design VII (BDP-efficient)

6 10 0.94 0.59 0.111 0.057 0.205 2.550

Revisiting Example 2

풅풇풑풆 풅풇푳풐푭 D-eff DP-eff Primary & Primary

Correlation

Primary & Potential

Correlation

Potential & Potential

Correlation

풕풓(푨푨′)

Design VIII (D-opt)

0 9 1.000 0.000 0.054 0.149 0.148 15.755

Design IX (DP-opt)

9 0 0.870 1.000 0.116 0.223 0.237 45.927

Design X (BDP-efficient)

6 3 0.904 0.494 0.100 0.140 0.239 25.120

Worth mentioning?• Common practice in response surface experiments to

replicate center point runs in order to obtain a modelindependent estimate of error.

• To generate the designs in this study, the full 37 factorialwas utilized as the candidate set of points.

• Out of the 400 24-run designs, none of them include acenter point run.

Simulation Protocol For each of 2000 iterations:1. Randomly assign columns of the design matrix to be active (2-5). 2. Randomly assign two-factor interactions to be active based on the weak effect heredity principle (i.e., a two-factor interaction is active only if one or more of its parent main effects are active) (0-7).3. Select the coefficient vector (β) for the active effects. Two scenarios are considered:

a. Equal: Randomly assign coefficients for each of the active effects from (±1, ±1.5, ±2, ±2.5, ±3, ±3.5).b. Smaller: Randomly assign coefficients for the main effects from (±2, ±2.5, ±3, ±3.5) and the two-factor interactions from (±1, ±1.5, ±2). More likely to represent what is observed in practice (effect hierarchy).

4. Generate the response vector as 풚 = 푿푨휷 + 흐, where 푿 is the matrix consisting of the active effect columns and each 휖 is simulated from a standard normal distribution.

Simulation Protocol (cont.)5. Fit the primary model (in this case, a main effects only model) to the simulated data and determine the set of active effects. This is accomplished by:

a. If a pure error estimate is available, statistical inference is performed using the mean square pure error (MSPE). Usual t-

statistics are computed as 훽 / 푣푎푟 훽/where 푣푎푟 훽 =

푀푆푃퐸 × 푿 푿 ퟏ풊풊and p-values computed based on a t-

distribution with dfpe ; α=0.05 is used for selection of active effects.

b. If a pure error estimate is not available, statistical inference is performed using the residual mean square from the fitted model.

6. If a pure error estimate is available, perform a lack-of-fit test via the usual F- statistic with dfLoF numerator and dfpe denominator degrees of freedom.

Simulation Protocol (cont.)

• At the end of the 2000 iterations, the average proportion of correctly identified active main effects (power) and the average proportion of main effects declared active which were actually inactive (false discovery rate) are calculated.

• Also calculate the power for detecting lack-of-fit by reporting the proportion of simulations in which lack-of-fit was correctly identified.

25

Some (Abbreviated) Results

An Aside: Pure Error and False Discovery (Variable Selection)

An Aside: Pure Error and False Discovery (Variable Selection)

Final Comments• Proposed a Bayesian modification to the DP-criterion to

allow for the construction of optimal designs that provide some safeguard against misspecification of the a priori model as well as the ability to obtain a model independent estimate of 휎 for conducting statistical inference.

• Examples illustrate that the proposed designs are a compromise between traditional Bayesian D-optimal designs, which rarely provide replicate points, and DP-optimal designs, which tend to over-replicate.

• The Pareto front approach and multiple design criterion also help to demonstrate the compromising effect of Bayesian DP-optimal designs.

Thanks for your attention!

Any questions?