Practical (Real-life)Implementation of Definitive Screening … · 2019-03-29 · Full factorials...

Practical (Real-life) Implementation of Definitive Screening Designs Made Easy(ish)

Dr Paul Nelson

Prism Training & Consultancy Ltd

2018

The Plight of the Flight of the Phoenix

2© Prism Training & Consultancy Ltd

“Essentially, all models are wrong, but some are useful”

George BoxBrad Jones Chris NachtsheimDesign-Ease & then Design-Expert

the early years


in the game

Design-Ease & then Design-Expert the early years

Yates, Plackett& Burman 1940s – 50s

What are the practicalities of…

• Implementing DSDs vs Classical sequential (SCO) strategy in real-life

• Managing their Analysis – handling multiple (likely correlated) responses, when the design possesses a desirable combinatorial structure

• Managing their supplementation/augmentation – be (pro)activefor new DSDs, or (re)active for existing DSDs, when it comes to potentially active factors…

• Managing their augmentation – to block or not to block, centre points & randomisation

4

Standing on the shoulders of Giants

© Prism Training & Consultancy Ltd

I’m not saying I’ve (re)solved these practicalities in the context of DSDs and their supple/augmentation, but hopefully raised some interesting research ideas and challenges

5

Choose a Design or Strategy


Characterization

including Verification

Screening

Optimisation

6

Customer Requirements / Criteria

Identify the vital few factors (culprits), from 6 suspects, causing an unexpected 30% drop in yield and propose alternative ranges likely to provide a robust optimum capable of routinely meeting customer specifications or QTPP

Solvent, Reagent, AcidCurrent process

Purity after isolation 99.9%

Early Stage Development

NC

O CO2H

NC

O CO2Me

Solution yield (by HPLC) 97a/a%

Example 1: Esterification of an Acid to Robustly Maximise Quality & Productivity

Screening: part of a sequential SCO strategy


7

2k-p Fractional Factorial Screening Design:


Fractionation & Resolution

Full factorials (white – no aliases). All possible combinations of factor levels are run. Provides information on all effects.

Resolution III (Red Warning) Designs: Main Linear Effects (MLEs) aliased with 2-factor interactions (2-FIs).

Resolution IV (Amber Caution) Designs: MLEs aliased with 3-FIs & 2-FIs are aliased with other 2-FIs.

Resolution V (Green Safe) Designs: MLEs are aliased with 4-FIs & 2-FIs with 3-FIs.

Alternative Minimum-Run Designs

Caution: partial aliasing – more on this later

8

26-0 Full Factorial Design:


Full Factorial

9

26-1 Fractional Factorial Design:


Half Fraction: Resolution V

10



Quarter Fraction: Resolution IV

Aliased Terms[A] = A + BCE + DEF [B] = B + ACE + CDF[C] = C + ABE + BDF [D] = D + AEF + BCF[E] = E + ABC + ADF [F] = F + ADE + BCD[AB] = AB + CE [AC] = AC + BE[AD] = AD + EF [AE] = AE + BC + DF[AF] = AF + DE [BD] = BD + CF[BF] = BF + CD

Design-Expert® Software

Product

Error estimates

Shapiro-Wilk test

W-value = 0.989

p-value = 0.950

A: Solvent

B: Water Spike

C: Reagent

D: Acid

E: Temperature

F: Time

Positive Effects

Negative Effects

0.00 2.95 5.90 8.86 11.81

0

10

20

30

50

70

80

90

95

Half-Normal Plot

|Standardized Effect|

Hal

f-N

orm

al %

Pro

bab

ility

D-Acid

E-Temperature

F-Time

11



Eighth Fraction: Resolution III

Aliased Terms[A] = A + BD + CE + BEF + CDF[B] = B + AD + CF + AEF + CDE[C] = C + AE + BF + ADF + BDE[D] = D + AB + EF + ACF + BCE[E] = E + AC + DF + ABF + BCD[F] = F + BC + DE + ABE + ACD[AF] = AF + BE + CD + ABC + ADE

+ BDF + CEF

12

Sequential Assembly of Fractions – Characterization


Resolving ambiguities with a 2nd augmented foldover fraction

Fold over Res III designs to decouple ME’s from 2FI’s – fold over pairs

Check Alias List: Res IV design

13



Resolving ambiguities with a 2nd augmented Semi-fold fraction

Semi fold over Res IV designs to decouple 2FI’s

14



Resolving ambiguities with a 2nd optimal augmentation

Augment, build or repair using Alphabetic-optimal design – save resources

15

Testing, Recognising & Handling Curvature


Screening & Characterisation studies identified key effects to be D, F and DF interaction…

but with strong evidence of curvature – suggests design is in the region of an optimum:

16

Optimization & RSM Designs


96

Optimisation

17

Added Bonus of Propagation of Error



Risk Assess: part of a sequential SCO strategy

Risk Assessed Design Space

Knowledge Space

Control Space

Control Space

“Parameter movements that occur within the design space are not subjected to regulatory notification. However, movement out of the design space is considered to be a change and would normally initiate a regulatory post approval change process.”

Optimisation


Resources: part of a sequential SCO strategy

36 Rxs!

20

Alternative Minimum-Run Characterization Designs


Resolution V for characterizing the vital few factors in more depth

Test for curvature but not estimable

Main effects not independent

but small correlation (±0.091)

2FI correlations

negligible (±0.183)

(±0.084& 0.099)

1 intercept + 6 MLEs + 15 2FIs = 22 runs + 2 CPs = 24 runs

Compared to 26-1 or 32 runsVI

21

Alternative Minimum-Run Screening Designs


Resolution IV for screening the factors to identify the vital few

Allowing for botched runs

0.1 – 0.55 .73 & .75

-0.1

–-0

.55

-.7

3&

-.7

5

2FI correlations

not all negligible

Main effects not independent

but orthogonal to 2FIs

Test for curvature but not estimable

(2x6)+2

22

What more do you want from Screening?


Small/minimum number of runs (2k… +1, for the intercept, at a minimum)

Main effects orthogonal & independent of potentially important 2nd order effects, such as 2FI’s & Quadratic terms. But, it’s a screening design!

Partial aliasing of 2nd order terms (not completely aliased) –small, ideally negligible – better than Res IV… don’t forget, it is supposed to be a screening design!

Drop/ignore unimportant factors – DSD retains properties, but greater power to detect effects of the retained factors (synonymous with fractional factorial design projective properties) – later… adding Fake Factors

Project on to any 3 active factors to enable fitting of a full second order RSM model, since 3 settings per factor… the definitive in Definitive Screening Design!

If Only...

Design

≥18 factors, a full RSM model is estimable for any 4 factors ≥24 factors, a full RSM model is estimable for any 5 factors

23

Definitive Screening Design Structure


6 foldoverpairs

& a centre point run

Model-oriented Designs “Design for the experiment, don't experiment for the design”

Jones & Nachtsheim Quality Technology, 43(1), 1-15, 2011

Near-minimal run (2K+1) DSD with 3 (not 2) levels

Design

0

0

±0.25

±0.5

±0.4655

+0.133

24

Definitive Screening Design Structure


Building a general minimum run DSD

A minimum run DSD is constructed using a kxk conference matrix C, such that

A C6 6x6 conference matrix

k foldover runs

k runs

centre point

k even

= 2k+1 runs

… k odd, C (k+1) matrix used with last column deleted

K

KK

k odd, 2*(k+1) + 1 or 2k+3 runs

Categorical factors, 2 centre runs: k even 2k+2, K odd 2k+4 runs

CTC=(k−1)Ik×k

To construct a DSD with more than the minimal number of runs, use a conference matrix with c > k columns and do not assign the last c – k (fake factor) columns to factors… later

Design

25



Full 2nd order Response Surface Model (RSM) consists of:An intercept (1 of those)All main effects (=k for k factors)All main quadratic effects (=k)All 2-FIs (=k(k-1)/2)

#terms in a full RSM 1 + 2k + k(k-1)/2 or (k+1)(k+2)

2

Ability to fit a full RSM model involving any 3 active factors

Minimum design is saturated by ML & Q effects. If you include the 2FI’s… supersaturated. Only estimate at most 13 model terms in the case of k=6 minimum run DSD and we need an estimate of σ. Is a simple analysis possible?

k=6 full (RSM) model consists of:1 intercept + 6 MLE’s + 6 QE’s + 15 2FI’s= 28 model terms

Simpler analysis when: Effects are >> σ & hereditableNo active 2FIs (eek! unlikely), so strong 2nd order terms don’t inflate σ or bias other important 2nd order effects… Sparsity of effects – # active factors → # effects ≤ ½ # runs > n/2, automated selection procedures tend to struggle

Definitive Screening

Analysis – aren’t we are asking a lot

Useful Effects Model Main Linear Effects Model

MLEs orthogonal/unbiased by 2nd order terms, but σ for a MLEs only model inflated by strong 2nd order terms. Separating the vital few difficult and the reliability of the coefficient SEs questionable

Can be Challenging. DSDs are supersaturated: # model terms > # runs e.g., k=6, # terms 28 > # runs 13: estimate at most 13 model terms & need to estimate σ.

Insufficient df to estimate full RSM model for 4 or more factors as well as σ. DSDs rely on sparsity, size of effects relative to noise, hierarchy & heredity.

26

Definitive Screening Design Analysis


(Multi-)Univariate or BORAT Analysis

27



# active effects < n/2 – just a few 2nd order effects – otherwise automated model selection procedures tend to struggle. We cannot fit a majority of the 28 terms

Power low to detect moderate 2nd order effects. Effects must be large. If many terms appear, or are likely to appear active… augment, or supplement

Anticipates the region might be ‘curvy’ so a RS model should be investigated

Previously Recommended Analysis Procedure: specify a RS model and use forward variable selection with stopping based on the AICc criterion to find out which effects are active and which are not

(Multi-)Univariate Analysis

Definitive Screening

28



Not designed to be a 1-step RSM

BUT can happen if ‘true’ set of active effects (6 here) <1/2 #Runs (13), so has gained in popularity

DSDs with more than 5 factors project onto any 3 factors to allow fitting the full quadratic model

(Multi-)Univariate Analysis

A problem of partial aliasing

Can fit full RSM model for any 3 active factors

29



New Recommended Analysis Procedure: use DSD foldover structure & correlation properties – odd/even 1st / 2nd order effects are orthogonal

Design-oriented Models Jones & Nachtsheim, Technolmetrics, 59(3), 319-329, 2017

Design Oriented (Multi-)Univariate Analysis

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Response 1 Response 2 Response 3

A:Solvent B:Water SpikeC:Reagent D:Acid E:TemperatureF:Time Product Y_ME Y_2nd

vol %w/w eq eq deg C hour %

0 -1 -1 -1 -1 -1 76.23 -9.78 86.01

0 1 1 1 1 1 95.78 9.78 86.01

-1 0 -1 -1 1 1 95.26 -0.56 95.83

1 0 1 1 -1 -1 96.39 0.56 95.83

-1 -1 0 1 1 -1 97.68 0.91 96.78

1 1 0 -1 -1 1 95.87 -0.91 96.78

-1 -1 1 0 -1 1 94.99 6.25 88.75

1 1 -1 0 1 -1 82.5 -6.25 88.75

-1 1 1 -1 0 -1 73.45 -11.59 85.04

1 -1 -1 1 0 1 96.63 11.59 85.04

-1 1 -1 1 -1 0 98.12 3.27 94.86

1 -1 1 -1 1 0 91.59 -3.27 94.86

0 0 0 0 0 0 95.82 0.00 95.82

6 foldover pairs &

centre point run

Strong heredityWeak heredity

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Response 1 Response 2 Response 3

A:Solvent B:Water SpikeC:Reagent D:Acid E:TemperatureF:Time Product Y_ME Y_2nd

vol %w/w eq eq deg C hour %

0 -1 -1 -1 -1 -1 76.23 -9.78 86.01

0 1 1 1 1 1 95.78 9.78 86.01

-1 0 -1 -1 1 1 95.26 -0.56 95.83

1 0 1 1 -1 -1 96.39 0.56 95.83

-1 -1 0 1 1 -1 97.68 0.91 96.78

1 1 0 -1 -1 1 95.87 -0.91 96.78

-1 -1 1 0 -1 1 94.99 6.25 88.75

1 1 -1 0 1 -1 82.5 -6.25 88.75

-1 1 1 -1 0 -1 73.45 -11.59 85.04

1 -1 -1 1 0 1 96.63 11.59 85.04

-1 1 -1 1 -1 0 98.12 3.27 94.86

1 -1 1 -1 1 0 91.59 -3.27 94.86

0 0 0 0 0 0 95.82 0.00 95.82

Combined ANOVA

30



New Recommended Analysis Procedure: use DSD fold-over structure & correlation properties – odd & even 1st & 2nd order effects are orthogonal

Fit All MLEs excl. Intercept to Y data. Then split into two orthogonal columns…

Predicted Y (YME) and Residual Y or Y – YME (Y2nd) are orthogonal (correlation = 0), so sum = Y

Use Forward Selection & p-value (0.1, 0.05, 0.01) to identify MLEs from YME

Use All Hierarchical & Adj R-Squared on Y2nd to identify 2nd order effects – start by including only 2nd order terms associated with the active MLEs to maintain hierarchy / heredity

Select combined YME & Y2nd model terms for Y to fit the design-oriented RSM model

Note: you can use this 2 response decomposition analysis for any fold-over design

Design Oriented (Multi-)Univariate Analysis

Additional Backward Elimination Step to tidy up

31

Multiple Responses & Multivariate Analysis


Exploiting DSD foldover structure, centring, scaling & correlation of MV data, along with PCA/PCR/PLS orthogonal projection properties:

PCA score & loadings Bi-plot displays correlation between inputs & outputs – correlation between Y’s and of Acid & Time with Y’s clearly evident from 1st PC alone

Model relationship between orthogonal Y PCs & X inputs as previously described, or…

Only 1st PC strong

Correlated responses benefitting from the DSD structure

Fit PLS MLEs models to correlated MV Y data to identify the vital few MLEs

Fit PLS models including All 1st & 2nd order terms associated with active MLEs; to maintain heredity & identify active 2FI & Quadratic effects – PLS orthogonal projection method

DSDs provide a definitive approach to screening in that main effects are not biased by any second-order effect and all quadratic effects are estimable. As screening designs they have many desirable characteristics. In the presence of sparsity in the number of active factors, our designs project to highly efficient response surface designs. Their most appropriate use is in the earliest stages of experimentation when there are a large number of potentially important factors that may affect a response of interest and when the goal is to identify what is generally a much smaller number of highly influential factors. DSDs work best when most of the factors are continuous. That is because each continuous factor has three levels, allowing an investigator to fit a curve rather than a straight line for each continuous factor. However, I want to make it clear that using a DSD is not a panacea. In other words, a DSD is not the solution to every experimental design problem.

DSD Farside

DSDs blah blah a definitive approach to screening blah blah blah blah blah blah blah blah blah blah blah blah blah all quadratic effects are estimable. Blah Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah highly efficient response surface designs. Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blahl blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah allowing an investigator to fit a curve rather than a straight line for each continuous factor. Blah blah blah blah blah blah blah blah blah using a DSD is blah a panacea. Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah.

What Brad Jones says about DSDsWhat is sometimes heard…


33



What if >3 factors are active? Return to Design

Be Proactive add Fake Factor columns to improve estimate of σ2

Adds 4 extra runs & 2 error df (2 fold-over pairs).

If many terms are likely to be active / appear active, then Proactively /Reactively… consider DSDs with > the minimal # runs, or augmenting, to estimate 2nd order terms. Useful if you can retain the special DSD foldover pair structure with desirable properties

Analyse YME:

Analyse Y2nd:

Runs Error DF Power Potential MLEs

When in doubt, build it stout

MSq = 0.39/3 = 0.13SD=√(0.39/3) = 0.361

34

DSD: Proactive Fake Factor Supplementation


MSq = 0.39/3 = 0.13SD=√(0.39/3) = 0.361

Analyse Y2nd: 17 rows, but only 9 df (independent values) as each fold-over pair are the same. Intercept accounts for 1 df, so 8 dfleft to estimate 2nd order effectsUse heredity assumption, All Hierarchical & Adj R-Squared on Y2nd

to identify 2nd order effects

Fake factors are orthogonal to MLEs & 2FIs involving the real factors. When deleted, the resulting design is still a DSD, but fake factor df help create an unbiased estimate of σ2 (RMSE) → greater power to detect the effects of the real factors.

Analyse YME: 17 rows, but only 8 df (independent values) as centre point & each fold-over pair sum = 0. Six ‘real’ MLEs, 5 significant, so 8 – 5 = 3 df to estimate σ2

35



Term Estimate Forced correct model using DSD data

Intercept 93.32

B Water Spike %w/w(0.2,2) -1.25 Note: MLEs orthogonal to 2nd order terms

D Acid eq(0.1,0.4) 4.82 so are not partially aliased with BF & DF

F Time hour(1,4) 6.04 i.e., there alias coeffs = 0

BF Water Spike %w/w*Time hour 1.59

DF Acid eq*Time hour -4.77

FF Time hour*Time hour -2.46

Term Estimate Fake Factor DSD combined model with increased sensitivity

Intercept 91.297 includes additional MLEs & consequently 2nd order terms

A Solvent vol 0.2686 Aliased Matrix Coefficient BF FF

B Water Spike %w/w -1.251 Solvent vol*Temperature deg C 0.00 0.81

D Acid eq 4.8186 Water Spike %w/w*Acid eq -0.14 -0.81

E Temperature deg C -0.436 Acid eq*Temperature deg C 0.66 -0.81

F Time hour 6.0407

AE Solvent vol*Temperature deg C -1.732 0.00 -1.40 2nd order terms in red have been magnified

BD Water Spike %w/w*Acid eq -1.464 0.21 1.19 due to the partial aliasing, which is the hidden

DE Acid eq*Temperature deg C 2.2918 1.51 -1.86 cost of screening experiments in general

DF Acid eq*Time hour -4.411

SumProduct 1.72 -2.07

BF FF

“Essentially, all models are wrong, but some are useful”

George Box

36



Design Oriented Multivariate Modelling & Analysis

37

DSD: Proactive Active Factor Supplementation


Supplement – suspect any 4 or more subsets of active factors & 2nd order effects

Intelligent search for additional fold-over pairs (6 = 12 runs) to help fit a full RSM model involving potentially any subset of 4 active factors for each response. No need to unrealistically expand to ≥18 factors (37+ runs) for a full RSM model is estimable for any 4 factors

Additional fold-over pairs preserve fold-over structure, enabling estimation of the required model using previous methods

OPh

H

+

I

Ph

O

H

Pd(OAc)2 (0.03 eq)TBA-Cl (1.0 eq)NaOAc (1.7 eq)

DMF/60 oC/ 3.5 h

59% conversion56% yield

1.5 equiv

Cacchi et. al. Tetrahedron 1989, 813

Example 2: Improve Conversion of a Heck Reaction

SCO

38



Simulation Model & Fit DS Model & Analysis

Forced Model Coefficients

RSM Model Coefficients & Plot Default 17 run DSD with Fake Factors

Reasonable Model Agreement, Fit, Makes Practical Sense?

39

DSD: Multiple Responses & MVAnalysis


Round 1: PLS fitting orthogonal MLEs to correlated Ys to identify the vital few MLEs

Orthogonal MLEs & Components – 6 factors. Only 4 VIPs!

Large # correlated responses, but also benefits from the DSD structure

Round 2: Fit PLS models incl. all 1st & 2nd order terms associated with active MLEs Validation possible given FFs & removal of non-significant terms

If only we had the luxury of knowing the model in real-life!

Block Augmented DSD* – assure each block is composed of fold-over pairs &…

40



Augmented DSD any subset of 4 factors Un-blocked (Completely Randomised)

Main effects model

MLEs & Quad model

Design: when you don’t have luxury of knowing model at outset

Add centre run to second block

*Run original DSD & REACT if any subset of 4 potentially active factors appear after analysis of first 13 runs. This can only happen in the case of Multiple Responses

Additional 6 foldover pairs

41



Simulation Model & Fit DS Model & Analysis25 run Augmented DSD any subset of 4 factors

26 Run Block Augmented DSD – assuring each block is composed of fold-over pairs (same data) except additional CP Block 2

Valid randomisation of a block design: randomise the order of the blocks and the units within each block when proactivelysupplementing a DSD.

If reactively augmenting, then original DSD comes first and…

42

DSD: Reactive Active Factor Sequential Augmentation


Augment for a specific set of 4 factors – you must’ve run the first DSD

Sequential reactive addition of 4 foldover pairs to preserve fold-over structure while enabling estimation of the required model

Intelligent search to identify an additional 4 fold-over pairs (8 runs) to fit a full RSM model for 4 identified active factors

The likelihood is the 2nd block of 8 augmented runs will be performed on a separate occasion, may be with different operators, equipment etc. The two blocks of runs should ideally include a blocking factor.

Another benefit of maintaining the foldover pair structure is that the two blocks are entirely composed of foldover pairs. Assuming fixed blocks, add a centre run to the augmented block to retain the ability to fit all quadratic effects.

Return to Design: we know the specific subset of 4 active factors

43

DSD: Reactive Active Factor Sequential Augmentation


Simulation Model & Fit DS Model & Analysis21 run Augmented DSD specific subset of 4 factors

22 Run Block Augmented DSD – assuring each block is composed of fold-over pairs (same data) except additional CP Block 2

Straight to Round 2: Fit PLS models incl. all 1st & 2nd order terms associated with the subset of 4 active factors

44

DSD: Multiple Responses & MVAnalysis


Large # correlated responses, but also benefits from the DSD structure

Brad Jones


– Renaissance Men– Jimmy Stewart– Frank Towns, Lew & Heinrich

46

Where to Go – Blogs and Articles


Practical (Real-life)Implementation of Definitive Screening … · 2019-03-29 · Full factorials...

Documents

Transcript of Practical (Real-life)Implementation of Definitive Screening … · 2019-03-29 · Full factorials...