Risk Analysis QC Plans:Risk Analysis QC Plans:Principles and PrioritiesPrinciples and Priorities

James O. WestgardJames O. WestgardUniversity of Wisconsin Medical SchoolUniversity of Wisconsin Medical School

Westgard QC, Inc.Westgard QC, Inc.Madison, WIMadison, WI

www.westgard.comwww.westgard.com

11

ObjectivesObjectives Review principles of industrial risk analysisReview principles of industrial risk analysis Identify priorities when applying risk analysis for Identify priorities when applying risk analysis for

analytical quality management in medical labsanalytical quality management in medical labs Assess importance of method validation for dealing Assess importance of method validation for dealing

with with ““safety characteristicssafety characteristics”” Assess importance of Statistical QC for providing Assess importance of Statistical QC for providing

known detection capabilitiesknown detection capabilities Recognize the advantages of integrating SQC Recognize the advantages of integrating SQC

procedures and Risk Analysis control mechanisms procedures and Risk Analysis control mechanisms in a QC Planin a QC Plan

22

Oct 25, 2011: CLSI EP23A Oct 25, 2011: CLSI EP23A Risk Analysis and Risk Analysis and ““The Right QCThe Right QC””! !

33

What’s the Right QC?

44

The Right QC is what I always do! Why would I do the wrong QC? 55

WhatWhat’’s s ““The Right QCThe Right QC””according to CLIA? according to CLIA?

493.1256 Standard: Control procedures.493.1256 Standard: Control procedures. (a) Laboratory is responsible for having control (a) Laboratory is responsible for having control

procedures that procedures that monitor the accuracy and precision of monitor the accuracy and precision of the complete analytical process.the complete analytical process.

(b) (b) ……must establish the number, type, and frequency must establish the number, type, and frequency of testing control materialsof testing control materials…”…”

(c 1) The control procedures must (c 1) The control procedures must detect immediate detect immediate errors that occur due to test system failure, adverse errors that occur due to test system failure, adverse environmental conditions, and operator performanceenvironmental conditions, and operator performance;;

(c 2) (c 2) Monitor over time the accuracy and precision of Monitor over time the accuracy and precision of test performancetest performance ……

66

Where learn about Where learn about Risk Analysis and QC? Risk Analysis and QC?

77

ISO 14971

CLSI EP18

CLSI EP23

CLSI C24

ISO 15198

ISO 22367

Manufacturers

Laboratories

CLSI EP22

What is risk?What is risk?Industrial ModelIndustrial Model

RiskRisk -- function of 3 factors:function of 3 factors: OCCurrenceOCCurrence –– probability or frequency of probability or frequency of

failurefailure DETectionDETection –– probability that failure will be probability that failure will be

detected before harm detected before harm SEVeritySEVerity –– consequence or harm consequence or harm

88

How estimate risk?How estimate risk? Rank each factor Rank each factor

Scale of 1Scale of 1--10 common in industry10 common in industry Scales of 1Scales of 1--5 and 15 and 1--3 common in healthcare3 common in healthcare

Combine factorsCombine factors Industry calculates Risk Priority NumberIndustry calculates Risk Priority Number Risk = OCC * DET * SEV = RPNRisk = OCC * DET * SEV = RPN Healthcare often use Healthcare often use ““Risk MatrixRisk Matrix””

FMEA is the common industrial toolFMEA is the common industrial tool Failure Modes and Effects AnalysisFailure Modes and Effects Analysis

99

How mitigate risk in industry?How mitigate risk in industry?

OCCURENCE OCCURENCE -- ““Design for safetyDesign for safety”” to to eliminate failureeliminate failure--modes modes Disclose Disclose ““safety characteristicssafety characteristics””

DETECTION DETECTION -- Build in Build in Control MechanismsControl Mechanisms Provide alerts to identify problemsProvide alerts to identify problems

SEVERITY SEVERITY -- Provide instructions for Provide instructions for ““safe safe useuse”” and precautions for and precautions for ““use errorsuse errors”” Inform laboratory of limitations and advise Inform laboratory of limitations and advise

how to monitor performance (QC)how to monitor performance (QC)1010

OCCURRENCEOCCURRENCEKey Guidance from ISO 14971Key Guidance from ISO 14971

First, design for safety!First, design for safety! ““IVD medical devices have performance IVD medical devices have performance

characteristics that determine the accuracy of characteristics that determine the accuracy of examination results. Failure to meet the examination results. Failure to meet the performance characteristics required for a performance characteristics required for a specific medical use could result in a specific medical use could result in a hazardous situation that should be evaluated hazardous situation that should be evaluated for risk to patients.for risk to patients.””

1111

Method Validation Method Validation A prerequisite to risk analysisA prerequisite to risk analysis Safety Characteristics Safety Characteristics –– performance performance

characteristics that determine the characteristics that determine the accuracy of IVD devices accuracy of IVD devices -- precision, precision, trueness or bias, analytical specificity, trueness or bias, analytical specificity, quantitation limit, etc.quantitation limit, etc. Laboratory must verify/validate performance Laboratory must verify/validate performance

is satisfactory for intended useis satisfactory for intended use

1212

SEVERITYSEVERITYKey guidance from ISO 14971Key guidance from ISO 14971

H2.5.2 H2.5.2 Estimating the severity of harm Estimating the severity of harm requires an understanding of the medical requires an understanding of the medical use of the IVD examination results, the use of the IVD examination results, the analytical performance requirements for analytical performance requirements for each application and the extent to which each application and the extent to which medical decisions are based on IVD medical decisions are based on IVD examination results. For this reason, examination results. For this reason, qualified medical input to the risk estimation qualified medical input to the risk estimation process is essential.process is essential.

1313

What should a laboratory do What should a laboratory do about severity?about severity?

““Detect immediate errors that occur due Detect immediate errors that occur due to test system failure, adverse to test system failure, adverse environmental conditions, and operator environmental conditions, and operator performanceperformance”” (CLIA 493.1256)(CLIA 493.1256)

Perform corrective actions to Perform corrective actions to ““recoverrecover””before reporting of test resultsbefore reporting of test results

Provide information for safe useProvide information for safe use

1414

DETECTIONDETECTIONKey Guidance from ISO 15198Key Guidance from ISO 15198

For existing IVD medical devices, For existing IVD medical devices, conventional statistical quality control conventional statistical quality control procedures (e.g., as described in procedures (e.g., as described in CLSI CLSI C24C24) are considered adequate unless ) are considered adequate unless evidence from riskevidence from risk--monitoring activities monitoring activities indicates [other] quality control procedures indicates [other] quality control procedures are essential for maintaining risk at an are essential for maintaining risk at an acceptable level. In that case, acceptable level. In that case, the quality the quality control procedures shall be validated. control procedures shall be validated.

1515

CLSI C24CLSI C24--A3 (2006)A3 (2006)Statistical Quality Control for Statistical Quality Control for

Quantitative Measurement ProceduresQuantitative Measurement Procedures

Describes QC planning processDescribes QC planning process Provides Provides ““SigmaSigma--metric QC selectionmetric QC selection”” tooltool

Calculate Calculate Sigma =Sigma = (TEa (TEa –– Bias)/SDBias)/SD Where TEa is quality required for test,Where TEa is quality required for test, Bias represents inaccuracy of methodBias represents inaccuracy of method SD represents imprecision of methodSD represents imprecision of method

1616

CLSI C24A3 QC Planning ToolCLSI C24A3 QC Planning ToolRelates Sigma to QCRelates Sigma to QC

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

1.65 2.65 3.65 4.65 5.65

13s/2of32s/R4s/31s/6x 0.07 0.07 6 113s/22s/R4s/41s 0.03 0.03 4 112.5s 0.04 0.04 4 112.5s 0.03 0.03 2 113s/22s/R4s 0.01 0.01 2 113s 0.00 0.00 2 113.5s 0.00 0.00 2 113s 0.00 0.00 1 1

Pfr Ped N R

Pro

babi

lity

for

Rej

ecti

on (

P)

3 4 5 6

DesirableError

Detection

DesirableFalse

Rejection

Systematic Error (SE, multiples of s)

Sigma Scale

1717

How should a medical laboratory address Detection?

Right SQC is essential in any QC PlanRight SQC is essential in any QC Plan Define the quality needed for intended Define the quality needed for intended

medical use or clinical application medical use or clinical application Account for the observed precision and bias of Account for the observed precision and bias of

the measurement procedure the measurement procedure Account for the known performance Account for the known performance

characteristics (sensitivity, error detection) of characteristics (sensitivity, error detection) of the QC procedurethe QC procedure

Account for Expected and Unexpected EventsAccount for Expected and Unexpected Events

1818

Expected and Unexpected Events Expected and Unexpected Events are Key to Risk Analysis QC Plansare Key to Risk Analysis QC Plans

Expected EventsExpected Events What changes or failures might occur?What changes or failures might occur? What is the probability of failure?What is the probability of failure? What is the severity of harm from a failure?What is the severity of harm from a failure? What control mechanisms can be What control mechanisms can be

implemented to detect failures?implemented to detect failures?

Unexpected EventsUnexpected Events How verify the attainment of the intended How verify the attainment of the intended

quality of test results?quality of test results?1919

CLN November 2011CLN November 2011Risk Management for QCRisk Management for QC

2020

CLSI EP23A CLSI EP23A –– Laboratory Quality Laboratory Quality Control based on Risk ManagementControl based on Risk Management

Quality Control Plan Quality Control Plan –– ““a document that a document that describes the practices, resources, and describes the practices, resources, and sequences of specified activities to control sequences of specified activities to control the quality of a particular measuring the quality of a particular measuring system or test process to ensure system or test process to ensure requirements for its intended use are metrequirements for its intended use are met””

2121

Example QC PlanExample QC Plan

2222

QC Plan Frequency Recovery DisclosureAnalyst/operator controlsStandard Operating Procedure Yearly SOP review Director review NoOperator training Every operator Supervisor review NoOperator checklists Daily Supervisor review NoSystem maintenance Manuf. Schedule Manuf. Repair NoOperator competency Yearly Re‐train NoBuilt‐in analyzer controlsElectronic checks Manuf. Manuf. Instructions NoFunction tests Manuf. Manuf. Instructions Sample conditionProcess tests Manuf. Manuf. Instructions NoCalibration checks Manuf./Reg. Supervisor review NoStable control materialsStatistical QC Startup + Monitor TS guidelines NoTrueness control Calibration TS guidelines NoPeriodic EQA, PT 3/year CA plan NoPatient data analysisImplausible values Each sample Repeat test Yes

TS = Trouble-Shooting; CA = Corrective Action

CLSI EP23A CLSI EP23A –– Laboratory Quality Laboratory Quality Control based on Risk ManagementControl based on Risk Management RiskRisk –– combination of the probability of combination of the probability of

occurrence of harm and the severity of occurrence of harm and the severity of that harmthat harm Risk = OCC * SEV Risk = OCC * SEV

OCCOCC -- ““Probability of occurrence of harmProbability of occurrence of harm””is not the same as is not the same as ““probability of probability of occurrence of a failureoccurrence of a failure””

2323

CLSI EP23A Occurrence RatingCLSI EP23A Occurrence RatingEP23 Rating Definition Ratio Defect rate

Frequent Once per week 1/7 0.1429

Probable Once per month 1/30 0.0333

Occasional Once per year 1/350 0.0029

Remote Once every few years 1/1000 0.0010

Improbable Once in lifetime of system 1/2000 0.0005

2424

What is Occurrence when observe 1 What is Occurrence when observe 1 hemolyzed sample per day?hemolyzed sample per day?

Does it make any difference if the lab Does it make any difference if the lab workload/day is 10 or 1000 samples?workload/day is 10 or 1000 samples?

Occurrence Occurrence –– 1/What?1/What? 1 failure with a sample1 failure with a sample 1 failure with an analytic run1 failure with an analytic run 1 failure per shift1 failure per shift 1 failure per day1 failure per day 1 failure per bottle of reagent1 failure per bottle of reagent 1 failure per lot of reagent1 failure per lot of reagent 1 failure per calibration1 failure per calibration 1 failure per calibration cycle1 failure per calibration cycle 1 failure per proficiency testing event1 failure per proficiency testing event

2525

Occurrence Occurrence -- 1/What? 1/What? Problem with ranking scalesProblem with ranking scales

TimeTime is not the right reference for is not the right reference for estimating occurrenceestimating occurrence

Number of patient test results Number of patient test results provides a better reference to provides a better reference to account for workloadaccount for workload

Defect rate Defect rate provides a better provides a better perspective for estimating occurrenceperspective for estimating occurrence

2626

Estimating OccurrenceEstimating Occurrence1. Describe Lab Process1. Describe Lab Process

Lab Process Description Parameters

Samples/run 50

Runs/day 2

Workdays/week 6

Weeks/year 52

Months/year 12

Workdays/year 312

Samples/year 31200

3 year factor 0.33

5 year factor 0.20

2727

Estimating OccurrenceEstimating Occurrence2. Expand ranking scale2. Expand ranking scale

Ranking Description

Very frequent 1 sample/day

Very frequent 1 run/day

Frequent 1 sample/week

Frequent 1 run/week

Probable 1 run/month

Probable 1 day/month

Occasional 1 day/year

Remote 1 day/3 years

Improbable 1 day/5 years

2828

Estimating OccurrenceEstimating Occurrence3. Calculate Defect Rate & DPM3. Calculate Defect Rate & DPM

2929

Ranking Defects/Year

Defect Rate

Defects/Million

1 sample/day 312 0.0100 10,000

1 run/day 15,600 0.5000 500,000

1 sample/week 52 0.0017 1,667

1 run/week 2,600 0.0833 83,333

1 run/month 600 0.0192 19,231

1 day/month 1,200 0.0385 38,462

1 day/year 100 0.0032 3,205

1 day/3 years 33 0.0011 1,058

1 day/5 years 20 0.0006 641

WhatWhat’’s the point?s the point? Risk analysis is more complicated Risk analysis is more complicated

than it first appears!than it first appears! CLSI guidelines simplify risk analysis, but also CLSI guidelines simplify risk analysis, but also

make it qualitative, subjective, even arbitrarymake it qualitative, subjective, even arbitrary

Laboratories must prioritize activities for Laboratories must prioritize activities for defining quality requirements, validating defining quality requirements, validating method performance, and selecting SQC method performance, and selecting SQC procedures to provide a procedures to provide a ““safety netsafety net”” for for catching errorscatching errors

3030

WhatWhat’’s the point?s the point? Risk analysis should be integrated into Risk analysis should be integrated into

existing existing ““error frameworkerror framework”” for managing for managing analytical quality of testing processesanalytical quality of testing processes

Development of Analytic QC Plan should Development of Analytic QC Plan should complement Statistical QC by adding complement Statistical QC by adding control mechanisms for specific failure control mechanisms for specific failure modesmodes Should identify control mechanisms for preShould identify control mechanisms for pre--

analytic and postanalytic and post--analytic failureanalytic failure--modesmodes

3131

SQC SQC vsvs Risk AnalysisRisk Analysis

3232

(7) Result Report

(3) Amplification

(1) Specimen collection

(2) Sample processing

(4) Calibration

(5) Detection

(6) Readout

TraditionalStatistical

QC

QC1

QC2

QC3

QC4

QC5

QC6

QC7

RiskAnalysisQC Plan

SQC SQC andand Risk AnalysisRisk Analysis

3333

(7) Result Report

(3) Amplification

(1) Specimen collection

(2) Sample processing

(4) Calibration

(5) Detection

(6) Readout

QC1

QC2

QC7

SQC

Potential Benefits Potential Benefits of Risk Analysis QC Plansof Risk Analysis QC Plans

Address preAddress pre--analytic and postanalytic and post--analytic analytic problemsproblems Laboratory may change these processes more Laboratory may change these processes more

readily than changing analytic processes readily than changing analytic processes

Address specific problems in highly Address specific problems in highly complex analytic systemscomplex analytic systems Need to monitor critical control pointsNeed to monitor critical control points

Address QC for Lab Developed TestsAddress QC for Lab Developed Tests

3434

ThatThat’’s my diagnosis!s my diagnosis!

Dr. Parvin, what do you think?Dr. Parvin, what do you think?WhatWhat’’s the probability of s the probability of

occurrence of harm?occurrence of harm?