Statistical Approach to Medical Device Verification and ... · PDF fileSTATISTICAL APPROACH TO...
Transcript of Statistical Approach to Medical Device Verification and ... · PDF fileSTATISTICAL APPROACH TO...
STATISTICAL APPROACH TO MEDICAL DEVICE VERIFICATION AND VALIDATION
Yes, there is going to be some math (but not much)
Medical Device Verification and Validation In the medical device world, verification and
validation are the backbone of design control This is how you prove you have produced
what you said you would and what the customer wants! It’s also required
What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Verification means confirmation by examination and
provision of objective evidence that specified requirements have been fulfilled
Design verification. Each manufacturer shall establish and maintain procedures for verifying the device design. Design verification shall confirm that the design output meets the design input requirements. The results of the design verification, including identification of the design, method(s), the date, and the individual(s) performing the verification, shall be documented in the DHF
What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Validation means confirmation by examination and
provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled
Design validation means establishing by objective evidence that device specifications conform with user needs and intended use(s)
What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Design validation. Each manufacturer shall establish and
maintain procedures for validating the device design. Design validation shall be performed under defined operating conditions on initial production units, lots, or batches, or their equivalents. Design validation shall ensure that devices conform to defined user needs and intended uses and shall include testing of production units under actual or simulated use conditions. Design validation shall include software validation and risk analysis, where appropriate. The results of the design validation, including identification of the design, method(s), the date, and the individual(s) performing the validation, shall be documented in the DHF
This is NOT the same as Test Method or Process Validation
Where Does V&V Fit Into the Design Control Picture?
A Few Quick Points
Design V&V studies are NOT experiments! Y0u should know the outcome before
you start.
Require predefined acceptance criteria Which is why they are not
experiments
Before you start V&V, you should have enough characterization data to understand your system (product)
Ok… Enough Introduction
But before we get into the nerdy stuff a few hints to make it easier: Ensure you have clearly defined and measurable
product requirements Use standards when available (more in a minute) Hire a good statistician!!
Standards and Guidelines There are several good guidelines
and standards published to help CLSI publishes a variety of
guidelines for statistical analysis useable for verification studies Normally protocols based on CLSI
guidance is accepted by the FDA Common Technical Specifications
(CTS) for in vitro diagnostics medical devices These CTS requirements are
associated with the IVD Directive 2009/108/EC of the European Parliament and the Council of 3rd February 2009 on in vitro diagnostic medical devices
Standards and Guidelines
A few words on CLSI EP Protocols � Provide estimates for individual parameters (limit
of detection, precision, bias …) � CLSI does not have a standard to combine parameters
CLSI standards in many cases proscribe sample sizes and outliers �routines
Two Requirements for CLSI EP Protocols “Doable” Don’t need a Nobel prize to figure out how to do it �
“Right” As in right enough. If CLSI method deviates from a
more complicated gold standard method after the 7th decimal place, that’s ok
An EP rule The least burdensome of several alternatives will
always be used – if the minimum sample size is 40, 40 will be the only sample size used
Speaking of Sample Size…
Three ways to get sample size From a standard This is the easiest
Math Good enough
Most common questions I get (how many???)
Number of Replicates
Math for Replicate Number: SD is the standard deviation estimate based on
characterization delta is the allowable error limit n is the number of samples α is the significance level, usually 0.05 1-β is the power of the test z1-α/2 and z1-β/2 are the z values corresponding to the
probability of 1-α/2 and 1-β/2 respectively on a normal distribution table (z table)
𝒏𝒏 ≥ ��𝒛𝒛𝟏𝟏−𝜶𝜶/𝟐𝟐 + 𝒛𝒛𝟏𝟏−𝜷𝜷/𝟐𝟐� ∗ 𝑺𝑺𝑺𝑺/𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅�𝟐𝟐
Number of Replicates
Good Enough for Replicate Number: You will find that unless you have unusual
circumstances, the calculation on the last slide will normally give you a small number of replicates
Three (3) is the minimum number you need to see a possible outlier and get a good mean
Power calculations will tend to give you an answer which is more than you have time or money for
Unless y0ur system has a lot of rep to rep variability, a smaller number of reps will work Variability can be determined through good
characterization and used as justification for number of reps.
Speaking of Sample Size…
The same type of logic applies for how many samples are needed: From a standard Again, this is the easiest
Math Good enough
Speaking of Sample Size…
Math: Some definitions: Margin of Error (Confidence Interval) No sample will be perfect, so you need to decide how much
error to allow. The confidence interval determines how much higher or lower than the population mean you are willing to let your sample mean fall
Confidence Level How confident do you want to be that the actual mean falls
within your confidence interval? The most common confidence intervals are 90% confident, 95% confident, and 99% confident
Speaking of Sample Size… Math:
Confidence Level Your confidence level corresponds to a Z-score. This is a
constant value needed for this equation. Here are the z-scores for the most common confidence levels 90% – Z Score = 1.645 95% – Z Score = 1.96 99% – Z Score = 2.576
You will find this will give you pretty large sample numbers
You also have to know the SD
𝒏𝒏 =𝒛𝒛𝟐𝟐 ∗ 𝑺𝑺𝑺𝑺 ∗ (𝟏𝟏 − 𝑺𝑺𝑺𝑺)𝑴𝑴𝒅𝒅𝑴𝑴𝑴𝑴𝑴𝑴𝒏𝒏 𝒐𝒐𝒐𝒐 𝑬𝑬𝑴𝑴𝑴𝑴𝒐𝒐𝑴𝑴𝟐𝟐
Speaking of Sample Size… Good Enough “Large” vs. “Small” Sample Numbers Larger sample numbers are more likely to approach
a closer model to the total population than smaller samples numbers
Larger sample numbers are more likely to approach a “normal” distribution than small sample numbers
A rule of thumb is to have 16 degrees of freedom (D.F.) for any average standard deviation used in building a model "Understanding Industrial Designed Experiments"
(Schmidt & Launsby) This would be 17 samples
Speaking of Sample Size… "The closeness of the approximate
probability 1- alpha to the exact probability depends on both the underlying distribution and the sample size.....as the underlying distribution becomes 'less normal' (i.e., badly skewed or discrete), a larger sample size might be required to keep a reasonably accurate approximation. But, in all cases, an n of at least 30 is usually quite adequate.“ Probability And Statistical Inference 2nd Edition"
(Hogg & Tanis)
Speaking of Sample Size…
As a rule of thumb, you can consider n=30 observations or more to constitute a ‘large' sample
So… If n=30 is a large sample, and n-17 is a reasonable number to get enough degrees of
freedom…
Therefore, by using n=20 there is a compromise in dropping the theoretical value (n=30) to a value that is more practical to use in industrial settings
Speaking of Sample Size…
A few final notes on sample size If SD is known, then use z distribution.
If SD is not known: If n is large, then use z distribution. If n is small, then use t-distribution
Design Verification Examples
There are so many different types of product requirements and potential ways to verify… You really don’t want to be here all night, do you?
Here some suggestions of product requirement verifications and what elements you should have in your protocols and reports
Precision and Accuracy
These are sometimes confused You need to be sure you are measuring the
correct parameter And that parameter meets you needs and
requirements
Evaluation of Imprecision and Inaccuracy Imprecision Refers to random analytic error Lack of repeatability and/or reproducibility
Inaccuracy Refers to systematic analytic error Lack of trueness Constant Proportional
Total Error Combined error for a single result
Precision and Accuracy
How Much Error is Ok??
Performance is acceptable when Observed error < allowable error
Performance is not acceptable when Observed error > allowable error
How much error is acceptable? Based on medical and or clinical need User needs traced to product requirements which
are what you verify/validate
Protocols Should contain (at a minimum) Purpose including design input, design output being
verified Rationale for the verification design Measurable acceptance criteria. Materials required Verification Method/Procedure. Include procedure or
reference for evaluation of unexpected test results (e.g. no-tests, invalids)
Boundary values, must be assessed and included Data Analysis including validity criteria, if applicable Statistical methods with references to published
methods, if applicable Include method for handling exceptions and outliers.
Reports Should contain (at a minimum) Materials used (including part number, lot number,
expiration date, storage condition, as applicable) Equipment used and calibration status Revisions of any reference procedures Date and location of testing Individual(s) performing the verification Results including or referencing raw data Summary of procedural errors and investigation of failures
or unexpected results, if any Summary of excluded data, if any Disposition, i.e., whether or not the acceptance criteria was
met. If acceptance criteria are not met, a discrepancy report should be
initiated
Design Validation
I did not spend a lot of time (or any) on design validation In many cases, the number of sites, samples and
statistics required are dictated by Regulatory agencies Cost of external studies Availability of clinical samples Time (mainly cost and management driven)
You still have to justify what you do and have the data analyzed statistically
In Summary
You always need to have some sort of statistical justification for your studies
You need to be able to defend your justification
There are many guidelines and standards available to reference
HIRE A GOOD STATISTICIAN
Almost Last Slide
That’s All Folks!
My Contact Information: Alan Golden Manager QA, Operations
Support Abbott Molecular [email protected] 224-361-7159