Post on 05-Jan-2016
Applying Change Point Detection in Vaccine Manufacturing
Hesham Fahmy, Ph.D.
Merck & Co., Inc.West Point, PA 19486
hesham_fahmy@merck.com
Midwest Biopharmaceutical Statistics Workshop (MBSW) - MAY 23 - 25, 2011
2
Outline
Definitions Detection methods CUSUM and EWMA estimators
Case studies CUSUM and EWMA SSE
Conclusions
3
Methods of Detection
Visual (Simple but Subjective) Raw data; run chart CUSUM chart EWMA chart
Analytical (Complicated but Objective) Change-Point estimators; i.e. CUSUM,
EWMA Mathematical Modeling; i.e. MLE, SSE
4
Types of Variation
Common Causes – natural (random) variations that are part of a stable processMachine vibrationMachine vibrationTemperature, humidity, electrical current fluctuationsTemperature, humidity, electrical current fluctuationsSlight variation in raw materialsSlight variation in raw materials
Special Causes – unnatural (non-random) variations that are not part of a stable processBatch of defective raw materialBatch of defective raw materialFaulty set-upFaulty set-upHuman errorHuman errorIncorrect recipeIncorrect recipe
5
Cumulative Sum Control Chart CUSUM: cumulative sum of deviations from
average
A bit more difficult to set up More difficult to understand Very effective when subgroup size n=1 Very good for detecting small shifts Change-point detection capability Less sensitive to autocorrelation
10,0max ttt CKyC
0 wabove/belo deviations daccumulate tC
6
Exponentially Weighted Moving Average
EWMA: weighted average of all observations
A bit more difficult to set upVery good for detecting small shiftsChange point detection capabilityLess sensitive to autocorrelation
“EWMA gives more weight to more recent observations and less weight to old observations.”
-1t
0j
-1 01 EyE tjt
jt
CUSUM Shewhart0 1
7
Process Shifts
Step Linear Nonlinear
Others
8
Process Model
in-control state
out-of-control state
t
Change point, "Unknown"
9
Change-Point Estimation Procedures
CUSUM Change-Point Estimation Procedure (Page 1954)
HCTtiHCCt TitCUSUM ,1,....1 0 ,0:̂
EWMA Change-Point Estimation Procedure (Nishina 1992)
TTiitEWMA UCLETtiUCLEEt ,1,....,1 ,:ˆ 00
0 wabove/belo deviations daccumulate tC
10
Example:
HCTtiHCCt TitCUSUM ,1,....1 0 ,0:̂
y
8
9
10
11
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
t
Process Data
10,0max ttt CKyC C
+
0
1
2
3
4
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
t
Upper CUSUM Chart
5H
5.0K
Most Recent Reintialization at t =22
Actual Change-Point= 20
11
Example: y
8
9
10
11
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
t
Process Data
TTiitEWMA UCLETtiUCLEEt ,1,....,1 ,:ˆ 00
11 ttt EyE 2.0
1000 E
E
9.5
10.0
10.5
11.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
t
EWMA Chart
UCL=11
LCL=9
tE at t =19Most Recent time
Actual Change-Point= 20
12
Real Example
29.0
30.0
31.0
32.0
33.0
Run
Cha
rt o
f Pro
dx
25 50 75 100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
Sample
Run Chart of Prodx
13
CUSUM vs. EWMA
-10
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
Cum
ulat
ive
Sum
of P
rodx
25 50 75 100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
Sample
CUSUM of Prodx
31.06
31.07
31.08
31.09
31.10
31.11
31.12
31.13
31.14
EW
MA
of P
rodx
25 75 125
175
225
275
325
375
425
Sample
Avg=31.0870
EWMA of Prodx, Lamda=0.001
14
5.0
2.0
29.5
30.0
30.5
31.0
31.5
32.0
32.5
EW
MA
of P
rodx
25 50 75 100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
Sample
Avg=31.087
EWMA of Prodx, Lamda= 0.2
29.5
30.0
30.5
31.0
31.5
32.0
32.5
EW
MA
of P
rodx
25 50 75 100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
Sample
Avg=31.087
EWMA of Prodx, Lamda= 0.5
15
Simulated Case Studies
To test different methods' (control charts/analytical) capability for identifying change-points Case # 1: small shifts/drifts Case # 2: mirror image of case # 1 Case # 3: large shifts/drifts
16
Case #1
-6
-5
-4
-3
-2
-1
0
1
2
3
4
Run C
hart
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Run Chart of Y
-6
-5
-4
-3
-2
-1
0
1
2
3
4
Run C
hart
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Run Chart of Y
17
Case # 1
-7
-6
-5
-4
-3
-2
-1
0
1
2
3
4
Y
1 2 3 4 5 6
0 25 50 75 100 125 150 175 200 225 250
Obs
Individual Measurement of Y
1 & 6: In-control process; N(0,1)
2 : -ve linear trend; 0.1 sigma
3 : Step shift; N(-3,1)
4 : Step shift; N(-1.5,1)
5 : +ve linear trend; 0.1 sigma
18
-100
-75
-50
-25
0
25
50
75
100
Cum
ula
tive S
um
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y (Alpha=0.0027)
CUSUM (V-Mask)
-6
-5
-4
-3
-2
-1
0
1
2
3
4
Run
Cha
rt of
Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Run Chart of Y
Detection Criterion: Slope Change
-3
-2
-1
0
1
2
3
Run
Cha
rt o
f Y
40 42 44 46 48 50 52 54 56 58 60
Sample
Samples 40 to 60
Diagnostic Sequence Plot
19
-100
-75
-50
-25
0
25
50
75
100
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y (Alpha=0.01)
CUSUM; 01.0
0027.0
Exact Profile-100
-75
-50
-25
0
25
50
75
100
Cum
ula
tive S
um
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y (Alpha=0.0027)
20
CUSUM – Target Adjusted
-300
-200
-100
0C
umul
ativ
e S
um o
f Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y - Target= 0
-150
-100
-50
0
50
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150
Sample
CUSUM of Y - Target= 0
-300
-250
-200
-150
-100
Cum
ula
tive S
um
of Y
50 75 100 125 150 175 200
Sample
CUSUM of Y - Target= 0
21
EWMA;
-5
-4
-3
-2
-1
0
1
2
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=-1.11
EWMA of Y (Lamda=0.5)
5.0
-3
-2
-1
0
1
2
3
Run
Cha
rt o
f Y
40 42 44 46 48 50 52 54 56 58 60
Sample
Samples 40 to 60
Detection Criterion: Slope Change
22
EWMA;
-1.20
-1.18
-1.16
-1.14
-1.12
-1.10
-1.08
-1.06
-1.04
-1.02
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=-1.1080
EWMA of Y (Lamda=0.001)
001.0
-100
-75
-50
-25
0
25
50
75
100
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y (Alpha=0.01)
CUSUM
Same Profile
23
Case # 2 “Mirror Image”
-4
-3
-2
-1
0
1
2
3
4
5
6
7
Y
1 2 3 4 5 6
0 25 50 75 100 125 150 175 200 225 250
Sample
Individual Measurement of Y
1 & 6: In-control process; N(0,1)
2 : +ve linear trend; 0.1 sigma
3 : Step shift; N(3,1)
4 : Step shift; N(1.5,1)
5 : -ve linear trend; 0.1 sigma
24
0
100
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y
CUSUM Chart
-100
-75
-50
-25
0
25
50
75
100
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y (Alpha=0.01)
Mirror Image
Case # 2
Case # 1
25
0
50
100
150
200
250
300
350
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y - Target= 0
CUSUM – Target Adjusted
26
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=0.992
EWMA of Y-Lamda=0.01
EWMA Chart; 01.0
27
EWMA Chart; 2.0
-1
0
1
2
3
4
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=0.99
EWMA of Y - Lamda=0.2
-1
0
1
2
3
4
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
µ0=0.00
EWMA of Y - Target= 0, Lamda= 0.2
28
EWMA Chart;
-2
-1
0
1
2
3
4
5
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=0.99
EWMA of Y-Lamda=0.5
5.0
29
Case # 3
-5
-3
-1
1
3
5
7
9
11
13
15
17
Y
1 2 3 4 5 6
0 25 50 75 100 125 150 175 200 225 250
Sample
Individual Measurement of Y
1 & 6: In-control process; N(0,1)
2 : +ve linear trend; 0.5 sigma
3 : Step shift; N(12.5,1)
4 : Step shift; N(9.5,1)
5 : -ve linear trend; 0.38 sigma
30
CUSUM
-300
-200
-100
0
100
200
300
400
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y
31
CUSUM – Target Adjusted
0
1000
Cum
ulat
ive
Sum
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
CUSUM of Y - Target=0
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
Cum
ulat
ive
Sum
of Y
50 75 100 125 150 175 200
Sample
CUSUM of Y - Target=0
-500
-400
-300
-200
-100
0
100
200
300
400
500
600
Cum
ulat
ive
Sum
of Y
-25 0 25 50 75 100 125
Sample
CUSUM of Y - Target=0
32
EWMA;
-2
0
2
4
6
8
10
12
14
16
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=5.45
EWMA of Y, Lamda=0.5
5.0
33
EWMA;
-2
0
2
4
6
8
10
12
14
16
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=5.45
EWMA of Y, Lamda= 0.2
2.0
34
EWMA;
-2
-1
0
1
2
3
4
5
67
8
9
10
11
12
13
14
15
EW
MA
of Y
0 25 50 75 100 125 150 175 200 225 250
Sample
Avg=5.45
EWMA of Y, Lamda=0.1
1.0
35
Detection by SSE Pick a window of about 30 points
including the “investigated point” Fit a two-phase regression using all
possible change-points & calculate the SSE
Plot possible change-points vs. their SSEs
36
-3
-2
-1
0
1
2
3
Y
20
21
22
23
24
25
26
27
28
29
30
31
32
SS
E
30 35 40 45 50 55 60 65 70
Obs
Left Scale: Y
Right Scale: SSE
35 to 65 Window
90% CI
-1
0
1
2
3
4
5
6
Y
28
30
32
34
36
38
40
42
44
SS
E
55 60 65 70 75 80 85 90 95
Obs
Left Scale: Y
Right Scale: SSE
60 to 90 Window
-1
0
1
2
3
4
5
Y
10
11
12
13
14
15
16
SS
E
105 110 115 120 125 130 135 140 145
Obs
Left Scale: Y
Right Scale: SSE
110 to 140 Window
-1
0
1
2
3
4
5
Y
105 110 115 120 125 130 135 140 145
Obs
Overlay Plot
Examples from Case # 2
37
-5
-2.5
0
2.5
5
7.5
10
Y
10
20
30
40
50
60
70
SS
E
30 35 40 45 50 55 60 65 70
Obs
Left Scale: Y
Right Scale: SSE
35 to 65 Window
2.5
5
7.5
10
12.5
15
17.5
Y
20
30
40
50
60
70
80
90
100
SS
E
55 60 65 70 75 80 85 90 95
Obs
Left Scale: Y
Right Scale: SSE
60 to 90 Window
7
8
9
10
11
12
13
14
15
Y
10
15
20
25
SS
E
105 110 115 120 125 130 135 140 145
Obs
Left Scale: Y
Right Scale: SSE
110 to 140 Window
3
4
5
6
7
8
9
10
11
Y
15
20
25
30
35
40
45
50
55
60
SS
E
155 160 165 170 175 180 185 190 195
Obs
Left Scale: Y
Right Scale: SSE
160 to 190 Window
Examples from Case # 3
38
Conclusions
Change-point problem is general and can be applied in many applications such as 4 parameter logistic regression and degradation curves.
Another application in manufacturing processes includes detection of the change-point for process variance.
It is preferred to combine both analytical and visual techniques; in addition to process expertise; to get accurate results.
39
References
• Fahmy, H.M. and Elsayed, E.A., Drift Time Detection and Adjustment Procedures for Processes Subject to Linear Trend. Int. J. Prod. Research, 2006, 3257–3278.
• Montgomery, D. C., Int. to Stat. Quality Control, 1997, (John Wiley: NY).
• Nishina, K., A comparison of control charts from the viewpoint of change-point estimation. Qual. Reliabil. Eng. Int., 1992, 8, 537–541.
• Pignatiello, J.J. Jr. and Samuel, T.R., Estimation of the change point of a normal process mean in SPC applications. J. Qual. Tech., 2001, 33, 82–95.
• Samuel, T.R., Pignatiello Jr., J.J. and Calvin, J.A., Identifying the time of a step change with X control charts. Qual. Eng., 1998, 10, 521–527.
40
Acknowledgements
Lori Pfahler Julia O’Neill Jim Lucas