Altekar-Giacoletti Process Validation Statistical Tool ... · PDF fileBethesda, MD 1 PROCESS...

ISPE Process Validation Conference12 – 14 September 2017

Bethesda, MD

1

PROCESS VALIDATION –STATISTICAL TOOL OVERVIEW

Maneesha Altekar, Katherine GiacolettiISPE Statistics Validation ConferenceSeptember 2017

ispe.orgConnecting Pharmaceutical Knowledge

Stage 2 (PPQ)• Statistical Intervals

• Confidence Intervals

• Tolerance Intervals

• (Prediction Intervals – not often used)

• Visualization Tools

• Scatterplots, run charts, box plots, histograms, variability charts

• Sampling Strategies

• Types of sampling

• Sample size

• Capability (preliminary)

• Variance Components Analysis

• ASTM

Statistical Methods Used in Process Validation Stages 2 & 3 (PPQ & CPV)

Stage 3 (CPV)• Acceptance Sampling

• Attributes & Variable

• Operating Characteristics

• Control Charts

• Types of charts

• Setting Limits

• Run rules & assumptions

• Capability

• Cpk & Ppk

• Discrete data?

• Assumptions

Beyond the scope of today’s session


Bethesda, MD

2


Basic Statistical Concepts – Not Covered

This “refresher” assumes an understanding of some basic statistical terms & concepts:

• Sample vs. Population

• Summary statistics – mean, standard deviation, range, median, etc.

• Scatter plots, box plots, histograms

… and at least a high-level familiarity with statistical methods such as:

• Statistical intervals

• Acceptance sampling

• Control charts

3


How to get in trouble with statistics

… or better yet, let’s find out how to avoid trouble!

4

This Photo by Unknown Author is licensed under CC BY-SA


Bethesda, MD

3


The critical assumptions for many statistics that often are overlooked

• Randomness & Independence – the most critical assumptions for many statistical tools

• Randomness means a lack of a pattern

• Technically, it means that the data arise from a (single, common) probability distribution (e.g. Normal, binomial)

• Independence means the value of any given data point does not depend on the value of any others

• Lack of randomness/independence makes many statistical calculations invalid

• … meaning they can lead to misleading conclusions & the wrong decisions

5



Examples of non-randomness: Location Effect

• Location effect – consistent pattern of lower values at end of a batch. The repeated pattern indicates that this is likelynon-random

• Calculations of the mean and SD, control limits, statistical intervals, etc. will all be misleading

6

332925211713951

105.0

102.5

100.0

97.5

95.0

Observation

Indi

vidu

al V

alue

1 2 3 4

1 2 3 4

1

1

… and how to detectGraph your data!!If data are collected from multiple batches and/or multiple locations, plot them in time order to look for non-random patterns


Bethesda, MD

4



Examples of non-randomness: Two Populations

• Two sides of press resulting in different tablet weights

• Therefore different API content

• Other related examples: multiple filling lines, different filling nozzles, etc.

• Combining them to calculate any statistics will be invalid and misleading

7



8

Two populations: how to detectGraph your data!!• Hint for 2 populations: Run/control chart

will show alternating high/low values• To confirm: side-by-side individual

value/box plots, run/control charts by press side… even just a histogram (sometimes) – but again, if data are collected with ordering or in groups, you must look for signs of non-randomness before doing any statistics

20148216104181261

1500

1400

1300

1200

1100

1000

900

800

700

600

Sorted Loc

Sort

ed w

eigh

t

123

BatchSorted

Time Series Plot of Sorted weight

20148216104181261

106

104

102

100

98

96

94

92

Sorted Loc

Sort

ed C

U (%

LC)

123

BatchSorted

Time Series Plot of Sorted CU (%LC)

Batch

Press Side

Loc

321

RLRLRL

201918171615141312111098765432120191817161514131211109876543212019181716151413121110987654321201918171615141312111098765432120191817161514131211109876543212019181716151413121110987654321

106

104

102

100

98

96

94

92

CU (%

LC)

Individual Value Plot of CU (%LC)

Batch

Press Side

Loc

321

RLRLRL

201918171615141312111098765432120191817161514131211109876543212019181716151413121110987654321201918171615141312111098765432120191817161514131211109876543212019181716151413121110987654321

1300

1200

1100

1000

900

800

wei

ght

Individual Value Plot of weight

120011251050975900

20

15

10

5

0

weight

Freq

uenc

y

Histogram of weight

104102100989694

25

20

15

10

5

0

CU (%LC)

Freq

uenc

y

Histogram of CU (%LC)


Bethesda, MD

5


What about Normality?

“Normality” – that data come from a Normal distribution – is an assumption of many statistical tests and calculations

9

107.5105.0102.5100.097.595.0

0.20

0.15

0.10

0.05

0.00

X

Den

sity

Distribution PlotNormal, Mean=100, StDev=2



• A Normal distribution is symmetrical around the mean and has a specific steepness and thickness of “tails” (how fast the ends taper off on either side)

• In fact, failure to meet these precise conditions has little practical impact on most statistics

• Tests for Normality (Anderson-Darling, etc.) are not good at detecting non-Normality for small samples (for which it may matter more) and overly sensitive for large samples (for which it matters even less)

10


Bethesda, MD

6


Why are you testing your data for normality?

• For large sample sizes the normality tests often give a meaningful answer to a meaningless question

• For small samples they give a meaningless answer to a meaningful question

- Greg Snow, Intermountain Healthcare

11




If one or more of these things is not true, ask

• Is the thing being measured expected to be non-Normal?

• If not, what is causing it? Multiple populations? Non-randomness?

• What is the practical implication on the statistics to be done?

12

104102100989694

25

20

15

10

5

0

CU (%LC)

Freq

uenc

y

Histogram of CU (%LC)

What is important for valid statistics is that the data are

• Unimodal• Symmetrical, at least roughly• Continuous (not “chunky”)


Bethesda, MD

7


Representativeness: If I need randomness, isn’t a simple random sample always best?

Types of sampling:

• Simple Random Sample: every unit has an equal chance of being selected

• Stratified Random Sample: a random sample is selected from each pre-designated group or “stratum”

• Systematic Sample: every nth unit is sampled

13


Representativeness: If I need randomness, isn’t a simple random sample always best?

• Another critical assumption for making valid inferencesabout a population from a sample, is that the sample be representative

• The best sampling strategy is the one that guarantees a representative sample

• Covers the full range of values in the population

• In the same proportions as the population

• Earlier in the lifecycle, stratified or systematic sampling to learn about homogeneity may justify simple random sample later

14


Bethesda, MD

8


Statistical intervals – misconceptions & abuses

• Why use a statistical interval?

• Quantifies the uncertainty of an estimate (e.g. the mean)

• Gives a range of plausible values

• This is only true for tolerance intervals (with some caveats) and Bayesian intervals

• What “confidence” really means

• The confidence level is the probability of the interval including the true value.

• Think of it as the probability of being right.

15



Ok, so we have to do a little math… Here is the formula for a Confidence Interval:

16

n

stx

Confidence factor, t , depends on sample size and the level of risk, α (the percentage of time that the confidence interval will not cover the true mean, µ)

Point estimate of mean, µ

Standard deviation -Point estimate of variation, σ

Sample size

16

Estimate ± Margin of Error


Bethesda, MD

9

ispe.orgConnecting Pharmaceutical Knowledge 17

1st Quartile 2.0000Median 3.00003rd Quartile 4.0000Maximum 5.0000

2.1740 3.8260

2.0000 4.0000

0.7942 2.1080

A-Squared 0.35P-Value 0.402

Mean 3.0000StDev 1.1547Variance 1.3333Skewness -0.0000000Kurtosis 0.0803571N 10

Minimum 1.0000

Anderson-Darling Normality Test

95% Confidence Interval for Mean

95% Confidence Interval for Median

95% Confidence Interval for StDev

54321

Median

Mean

4.03.53.02.52.0

95% Confidence Intervals

Summary Report for Precipitation

90% CI = [2.33, 3.67] 95% CI = [2.17, 3.83]99% CI = [1.81, 4.19]




• Confidence, variability, sample size & uncertainty

• A wider interval does not mean that data are likely to be as extreme as the interval boundaries

• Keep in mind “probability of being right” – that’s what statistical confidence is, and why intervals get wider with

• Less information (smaller n)

• Higher variability

• Higher confidence

When using them to make decisions, interpretation must keep these things in mind

18


Bethesda, MD

10


Putting some of this together – Statistics for Stage 2 PPQ Planning & Analysis

• Stage 2 Planning

• Sample Size (intra-batch)• Quantitative attribute: statistical interval within specs,

given assumptions from Stage 1

• Qualitative attribute: statistical interval for probability of non-conformance – tie to routine AQL

• ASTM (CU), ANSI with routine AQL as RQL or tightened/higher level than routine

19



• Stage 2 Planning

• Number of batches – most companies doing risk-based

• Careful with capability or other estimates of or based on between-batch variability in Stage 2 – why?

Statistical cautions• Estimate of between batch standard deviation very unreliable

with small n

• Translates to very unstable capability estimate

Practical cautions• Even with large n to give statistically stable capability estimate, it

would have little practical use – haven’t seen long-term sources of common cause variability yet (more on this later)

20


Bethesda, MD

11



• Stage 2 Analysis & Conclusions

• Use statistics to help understand risk

• Keep goal in mind – PPQ should not fail due to statistics

• TIs or CIs outside of specifications – why?

• LOOK AT THE DATA

21


• The most critical assumptions for valid statistical analyses are randomness, independence, and representativeness.

• Normality is not as important as the data being unimodal, symmetrical (or close to), and continuous (not “chunky”). • Lack of apparent Normality is often a hint that there is a lack of

randomness or independence• Graph your data

• To check assumptions• To see what you expect statistical analyses to show• To look for unexpected responses, relationships, etc.

• When using statistical intervals to make decisions, understand the impact of sample size, variability, and confidence level on the width (i.e., uncertainty) – and what confidence really means

Key Messages


Bethesda, MD

12


Acceptance Sampling

• The process of selecting a representative part of a larger quantity for inspection or analysis to determine if the larger quantity can be approved as acceptable.1

• Why should we care about it?

– FDA has made it clear that it expects people to not simply execute a sampling plan, but also to understand and justify

• what it means (AQL, RQL, OC curves)

• zero defects in sample does not mean zero defects in batch

• its implications (when acceptance criteria are not met)

• its risks (how well can you confidently characterize the process, are the risks acceptable), etc.

1 The Handbook of Applied Acceptance Sampling, K. Stephens


Acceptance Sampling – “Fit for Purpose”

Sampling used for different purposes throughout the PV lifecycle

• Stage 1: Estimate defect rate, factors that affect it, batch uniformity

• Stage 2: Confirm desired reliability levels, tied to severity of harm

• Stage 3: Monitor for change, establish inter-batch consistency



Bethesda, MD

13


Acceptance Sampling

• Two types of data• Attribute - each result is Pass/Fail, qualitative

• Variable - each result is numerical, quantitative

• Attribute: Characteristic or property of an item– Glass vial (cracked, not cracked)

– Cap color (correct, incorrect)

– Printing on box (legible, Illegible)

• Variable: Numerical measurement– Dimensions of a component

– Injection force for a device


Acceptance Sampling

5 things to know about a sampling plan

• Lot size

• Sample size

• AQL – acceptance quality limit

• RQL – rejectable quality limit, also LTPD or lot tolerance percent defective

• Acceptance number



Bethesda, MD

14


• AQL - the quality level that is the worst tolerable, as a process average. Often it is defined as the percent defective with a 95% chance of acceptance.• Associated with producer’s risk

• RQL – the worst quality level that would be acceptable in a single lot. Often it is defined as the percent defective with a 10% chance of acceptance.• Associated with consumer’s risk

• Acceptance number – if number of observed defects is at or below this number, lot will be accepted

Acceptance Sampling, cont.


• Risk acceptance


Type I Error() – 5%

Type II Error() – 10%

TRUE DEFECTIVE RATE:

< AQL > RQL (LTPD)

Accept lot DECISION:

Reject lot

Consumer’s risk

Supplier’s risk

The probabilities for Acceptance/Rejection associated with a sampling plancan be graphically displayed in an Operating Characteristic Curve


Bethesda, MD

15

ispe.orgConnecting Pharmaceutical Knowledge 29

Percent Defective

Pro

b. o

f A

ccep

tanc

e (P

a)

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

Operating Characteristic (OC) Curves are often used to illustrate the performance of the sampling plan. These curves provide a way to compare the performance of different plans.

High probability means lots will typically be found acceptable. Steepness of the

curve indicates the discrimination sampling plan.

Low probability means lots will typically be found unacceptable.

Operating Characteristic (OC) Curve


Operating Characteristic (OC) CurveIncreasing the Sample Size Improves the Discriminatory ability of the test

30

OC (Operating Characteristic) Curves for AQL 0f 0.1% for various batch sizes

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 2 4 6 8 10 12 14 16

13;0

50;1

80;2

125;3

200;5

315;7

500;10

800;14

1250;21

Risk of rejecting good quality = 1- Pr (acc Lot)

Risk of accepting poor quality

Prob

of

Acc

Defect Rate (%)

N, acc #


Bethesda, MD

16


• During PPQ (Stage 2), sampling plans based on RQL

• Emphasis on consumer protection (beta risk)

• Tightened or increased confidence/coverage vs routine levels

• During CPV (Stage 3), sampling plans based on AQL/RQL

• Monitor for shifts

• Adjust confidence/coverage to allow reduced sampling (focus on alpha risk)



• Attribute Acceptance Sampling Standards– ANSI/ASQ Z1.4-2003 (R2013)

– ISO 2859-1:1999

– MIL-STD-105E (1989), cancelled 1995

• Variable Acceptance Sampling Standards– ANSI/ASQ Z1.9

– ISO 3951

• These plans are AQL based and designed to control the producer’s risk



Bethesda, MD

17


How to use the ANSI Attribute Sampling Table

• Need 3 Values– AQL (selected from choices above)

• Percent Nonconforming

– Batch Size Being Inspected

– General Inspection Level

• Typically use Level II

• Determine sample size code letter• Table I

• Determine Sample Size / AC• Single sampling plan

• Normal Inspection

• Table IIA


Determining the Sampling Plan

Sampling plan: inspect 315 units, if more than 5 defectives are observed reject the lot.


Bethesda, MD

18


• Sample plan: sample 315, Reject if 6 or more defectives observed

• AQL = 0.65%

• P(X≥6|N=315 and p=0.0065) = 1.8%

• The Producers risk that a batch with an acceptable quality (worst tolerable) will be rejected is 1.8%. False Reject rate is 1.8%

• P(X<6|N=315 and p=0.0331) = 5.0%

• The Consumers risk is that “poor” quality batches will be accepted is 5% for p =0.0331. False Accept rate is 5% for p = 0.0331

• OC curve will visually show this for ALL p’s

Statistical Conclusions


Operating Characteristic (OC) Curve

Can be compared to determine an appropriate sampling scheme

36


Bethesda, MD

19


• Sampling carries risks – understand what it is and be able to justify it in the sampling plan

• AQLs and RQLs should be chosen according to risk to patient, GMP compliance (customer complaints)

• Sampling plans should not

– be the same throughout the lifecycle

– chosen blindly from ANSI / ISO tables

• Sampling plans should

– be linked to risk management

– evolve through the lifecycle

– evaluated and chosen based on their operating charateristics

Key Messages


Questions?

38


Bethesda, MD

20


• Statistical methodology to graphically monitor results, and ensure that process is stable and in statistical control

• Why should we care?

– FDA has made it clear that companies need to demonstrate that their process continues to remain in a validated state, and to do so with the use of data and statistics

– CPV is not a checkbox, intended to motivate continuous improvement

Control Charts


• Graphically monitor process against pre-established limits

• Ensure process is stable and in statistical control

– Only common cause variation

• Quickly detect special cause variation

Control Charts


Bethesda, MD

21


Control Charts

Control Chart displaying only Common Cause Variation

21191715131197531

104

102

100

98

96

94

92

Batch

_X=98.47

UCL=104.04

LCL=92.90

Avg Dissolution

4137332925211713951

104

102

100

98

96

94

92

90

Batch

_X=98.47

UCL=104.04

LCL=92.90

1

1

55

5

1

55

Avg Dissolution

Control Chart displaying Special Cause Variation


• Data can be continuous or discrete

• Continuous – assay, dissolution, tablet weight

• Discrete – number of defects, proportion defective

• Not all control charts apply to all types of data

Control Charts


Bethesda, MD

22


• X-Bar and R charts (Normal dist)• Multiple measurements (reported values) per

batch

• Batch means, range,

• Example, tablet weights, dissolution, CU

• X and MR chart (Normal dist)

• Single measurement (reported value) per batch

• Example, water content, pH, assay

Various Control Charts – Continuous data


• P chart

• Proportion of defective units (Binomial dist)

• NP chart

• Number of defective units (Binomial dist)

• C chart

• Number of defects (Poisson dist)

• U chart

• Number of defects per unit (Poisson dist)

Various Control Charts – Discrete data


Bethesda, MD

23


Control Charts – Discrete data

Common mistake

Correct chart

Jul 14Apr 14Jan 14Oct 13Jul 13Apr 13Jan 13Oct 12Jul 12Apr 12Jan 12

2.5

2.0

1.5

1.0

0.5

0.0

Month

_C=0.469

UCL=2.523

LCL=0

C Chart of Number of Defects per Month

Jul 14Apr 14Jan 14Oct 13Jul 13Apr 13Jan 13Oct 12Jul 12Apr 12Jan 12

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

-1.5

Month

_X=0.469

UCL=2.270

LCL=-1.333

I Chart of Number of Defects per Month


Control Charts – Discrete data

272421181512

25

20

15

10

5

0

Normal Number of Broken Tablets

454035302520151051

30

25

20

15

10

Batch

11

29

20

Number of Broken Tablets

I Chart

Good Enough


Bethesda, MD

24


• Data from historical batches – 25-30 most recent• If not enough, can calculate preliminary limits

• Check that data are from a “stable” process• Allow for variability from raw material batches

• Remove any special cause result

Control Charts – Calculating Limits


Control Charts – Calculating Limits

37332925211713951

103

102

101

100

99

98

Batches

_X=100.3

UCL=102.7

LCL=97.9

I Chart of Assay

645750433629221581

105

104

103

102

101

100

99

98

97

96

Batch

_X=100.19

UCL=104.45

LCL=95.93

I Chart of Assay


Bethesda, MD

25


• Western Electric rules - Decision rules for detecting an out-of-control process

Control Charts – Responding to Signals

16/08/

2015

14/07/

2015

17/06/

2015

17/06/

2015

29/0

5/20

15

11/0

5/20

15

11/0

5/20

15

28/0

4/20

15

05/0

2/20

15

01/0

1/20

15

115

110

105

100

95

90

85

80

Date Tested

_X=98.04

UCL=113.64

LCL=82.44

2

2

2

I Chart of Results

Tested on 14/07/15

464136312621161161

115

110

105

100

95

90

85

80

Batch

_X=98.04

UCL=113.64

LCL=82.44

I Chart of Result


• Process not always truly stable• Shift up or down with raw material batches

• Successive batch results are not always independent• Testing performed in groups• Batches manufactured in campaigns

• Data not always normal• CQA truly non-normal or is there an outlier?

• Response to signal should be risk based• Signal should trigger response, but commensurate with

risk to patient, process, compliance

Key Messages


Bethesda, MD

26


Questions?

51


• Process Capability is a measure of a process’ ability to meet expectations. This is typically expressed in the form of an Index, usually Cpk or Ppk.

• Motivate continuous process improvement

Process Capability

52


Bethesda, MD

27


• Expectations are typically specifications, but may be ANY set of limits deemed important by the business.

• Process Capability is intended to describe what the process is expected TO DO, not what it did.• Process capability < 1 can be expected to produce OOS

results with some probability• A process must be Stable and Predictable (in a state of

statistical control) to use capability measures.

Process Capability

min ,3 3

USL x x LSLCpk Ppk

s s

53


• Assumes normality, stability• at least approximately

• Impacted by number of results used in calculation• how well are mean and SD estimated from data

• 1-sided lower confidence bound is a better indicator of process capability

Process Capability


Bethesda, MD

28


Interpreting Cpk and Ppk

55

-1-2-3 1 2 3

LSL USL

-1-2-3 1 2 3

LSL USL

Cpk or Ppk ~ 1Process on target

Cpk or Ppk < 1Process off target

-1-2-3 1 2 3

LSL USL

Cpk or Ppk < 1Process not capable.

-1-2-3 1 2 3

LSL USL

Cpk or Ppk > 1Process highly capable.


Interpreting Cpk and Ppk

56

Sigma Level Ppk/Cpk % OOS

2 0.67 4.55

3 1.00 0.27

4 1.33 0.01

5 1.67 0.0001

6 2.00 0.0000002

For a two-sided specification


Bethesda, MD

29


Cpk versus Ppk (If only this was Easy!)

• Cpk is for short term, Ppk is for long term.• Cpk tells you what the process is capable of doing in the future. Ppk

tells you how the process has performed in the past.• Cpk includes only common cause variation, whereas Ppk includes

both common and special cause variation.• Cpk uses as “estimated sigma” value in its formula. Ppk uses the

“actual sigma” value in its formula.• Unlike Cpk which uses time as a factor (because of its control chart

roots), Ppk ignores time.• Ppk is an estimate of process capability during its initial set-up,

before it has been brought into a state of statistical control.

Here are some direct quotes from the web showing HOW confusing it is out there:

57


Difference Between Cpk and Ppk

• There is none! During this conference, regardless of whether presentations use Cpk or Ppk it means the same thing. Same formula, same intent; maybe different names.

• Potential differences DO exist in how “s” is determined

• In this conference pay attention to how the standard deviation is estimated.

2 4 2

, , , , within between

R S MRS S S

d c d

58


Bethesda, MD

30


Process Capability Example

59

Particle size data from a high volume drug substance. 341 lots included in the analysis.

PS MV

50

100

150

200

LSL

USL


• Not just a number!• Motivate process improvement• Requires process to be stable and normal• Critical to look at the confidence interval!

Key Messages


Bethesda, MD

31


Questions?

61


• FDA (and other regulators) expects companies to demonstrate that their processes remain in a validated state throughout the lifecycle. . . and do so with the use of statistical methods

• Everyone in the company needs to understand their data (not just statisticians), the risks assumed, the interpretation of results. . . know when you can do it yourself and when you need a professional statistician’s help

Final thoughts . . .

Altekar-Giacoletti Process Validation Statistical Tool ... · PDF fileBethesda, MD 1 PROCESS...

Documents

Transcript of Altekar-Giacoletti Process Validation Statistical Tool ... · PDF fileBethesda, MD 1 PROCESS...