FOCUS 2011 NATIONAL AUDIT OF INTERNAL QUALITY CONTROL PRACTICE in UK. David Housley. Luton and...

FOCUS 2011NATIONAL AUDIT OF INTERNAL

QUALITY CONTROL PRACTICE in UK.

David Housley.

Luton and Dunstable NHS Trust.

Teresa Teal.

National ACB Audit Group.

INTRODUCTION

• A previous audit of IQC within the Thames Region in 2006 showed significant variation in practice.

• 29 returns were received.• Some areas of very poor performance were identified.• 9 audit standards were developed and published 2008.

Housley D G et al; Ann Clin Biochem; 45: 135–139.

• AIM: To re-audit IQC practice nationally and compare to previous audit data from a single area.

METHODS

• Questions based on the previous audit, with minor modifications to improve clarity, facilitate electronic data gathering and to increase focus on audit standards.

• Questionnaire designed using SurveyMonkeyTM

• Distributed by e mail in March 2011 via ACB office with a hyperlink to the questionnaire.

• Circulated to one individual in each UK hospital.• Respondents were instructed to restrict answers

to routine chemistry systems only (not endocrine or esoteric).

Q. 1 & 2

• 86 responses received.

• ANALYTICAL SYSTEMS IN USE;– 41% Roche– 16% Beckman - Coulter* – 8% Olympus* – 14% Abbott– 13% Siemens– 8% Ortho

• *note change of ownership since previous audit.

Q.3

• How frequently is internal quality control material run through your analyser (assume standard core hours (weekday) with no analytical, technical or procedural problems that require extra analysis)?

• Please answer for serum / plasma assays only.

• 53% At set time intervals• 64% At fixed times of the day • 6% After a set number of samples or tests

– Responses ranged from hourly through to 24 hourly. – Most common responses ranged from 4 to 8 hourly.

86 answered question

Q.4

• Please state what your lab considers to be its core working hours and whether you operate a shift system or on-call system outside of this period?

• Core working hours of labs are variable but most common answers approximated too:– 62% 09:00 – 17:30– 26% 08:00 – 20:00– 3% 08:00 – 18:00– 5% 07:00 – 22:00– 4% 24/7 shift


Q.5

• Does frequency of IQC outside of weekday core hours differ to the frequency within core hours?

24.5% YES (21% previously)

75.5% NO

Mostly this is a reduction in the frequency of IQC, but significant evidence of discretionary testing still remains.

Q. 6

• Does your laboratory have a policy for IQC practice

outside of weekday core hours ?

• YES = 58% (30% previously)– NO = 42%


Q.7

• According to your laboratory sample throughput, what is longest run length (number of patient specimens processed between IQC

assessment) for a plasma / serum sodium measurement ?

0

200

400

600

800

1000

1200

1400

Roche Beckman Abbott Olympus Siemens Ortho

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Roche Beckman Abbott Olympus Siemens Ortho


Q.8

• Does your laboratory preferentially utilise IQC material

from a third party source if it is available ?

• YES = 85%

• NO = 15%

• Unchanged from previous regional audit.


Q.9

• Which of the following are used routinely (every IQC assessment)?

• 44% Unassayed (no values assigned) material• 27% Assayed (quoted values used)• 61% Assayed (values assigned by lab)

Replicates 100 80 50 40 30 25 20 10

Number of labs

3 1 3 1 10 1 23 6

If you assign your own values, how many measurements are made to derive the target and SD ?

Q.10

• How are the target concentrations for IQC material chosen ?

– 60% What is available on the market– 81% Clinically important decision levels– 86% To cover low, normal and high values – 42% Detection limits of assay– 35% To assess areas of poor assay performance – 5% Other reasons

– custom made material, stability, lot size, cost.


Q.11

• Are multi-rules (e.g. Westgard rules) used as the primary form of QC data evaluation for accepting / rejecting QC ?

YES = 60.5%

NO = 39.5%

Free text replies showed a lot of confusion.Very similar percentages to previous regional audit.


Q.12

• If you do use multi-rules (e.g. Westgard rules), do you apply the same rules to all analytes, or do you use method (analyte)specific rules ?

• 89.5% Same for all• 10.5% Method specific

• Free text replies showed a lot of confusion.

Q.13

• Do you use single 2 SD rules ?

• 55% YES (76% in previous regional audit)• 45% NO

• Fewer laboratories using single 2SD rules.• Free text replies varied from ‘all analytes’ to single

analytes, most commonly sodium / ISE.


Q.14

• How are your control limits set ?

– 92% Data generated within laboratory – 46% Cumulative data– 31% Manufacturer derived ranges – 11% Peer group or EQA data

– Varies with different analytes.

Q.15

• Do you consider the IQC data on your main chemistry analyser easy to access ?

• 84% YES• 16% NO


Q.16

• What is the typical response when IQC data suggests a method is out of control (e.g. failed QC) ?

– 67% Repeat using existing control material– 73% Re-run using new control material– 87% Re-calibrate then re-run control– 6% Over-ride then continue with patient samples

– Free text comments from all.– Would depend on whether 2SD being used as a warning or multi

rules are violated.– Question could have been more specific.

Q.17• What grade / type of staff make decisions about whether to accept or reject

IQC data in real time ?

6% Trainee / MTO 5% Qualified (not registered)95% BMS 196% BMS 282% BMS 344% BMS 426% Locum BMS20% Quality manager36% Clinical Scientist32% Consultant / Specialist registrar

Very similar results to previous regional audit.


Q.18• If IQC data showed that a method was out of control (i.e. QC had

failed), what grades of staff would make the decision to continue processing patient samples ?

18% WOULD NEVER HAPPEN IN MY LAB

0% MTO / Qualified (not registered)1% Trainee15% BMS 161% BMS 265% BMS 340% BMS 4 3.5% Locum BMS 20% Quality manager49% Clinical Scientist50% Consultant / Specialist registrar


Q.19

• How often is out of control (non-ideal) IQC accepted (e.g. in order to ensure work is completed) ?

– 7% Daily– 7% Weekly– 2% Monthly– 55% Rarely– 26% Never– 5% Other

– Other reasons were ‘known problem’; ‘reagent or calibrator change’; ‘depends on clinical significance’; ‘only for non-critical tests such as protein or bilirubin’; ‘we know we get fliers’.


Q.20

• What grade of staff monitors overall IQC performance ?

– 29% Quality manager– 86% Senior in charge of section – 53% Clinical scientist– 61% BMS3 or BMS 4– 26% Consultant


Q.21

• Do you formally compare performance between multiple analysers (same site or across sites)?

• 71% YES• 29% NO

• Very similar to previous regional audit.• Very wide range of statistical processes in use.


Q.22

• If your service is split between geographically distant sites (e.g. different hospitals) are there barriers that prevent IQC performance being compared for the same type of analysers located on the different sites ?

• 20% YES (34% in previous regional audit)• 80% NO

• Barriers include lack of time, poor IT, staff pressures, time pressures, poor cooperation.


Q.23

• If an assay is judged not to be performing adequately (poor IQC performance), what systems

are used to prevent patient results being released ?

– 79% Chemistry is disabled– 33% Data not transferred to LIS– 32% Results not validated– 39% Relies on operator not to report


Q.24

• Please state how long your laboratory stores IQC data for. If the data is not readily retrievable for the whole duration, please additionally state how long it is in a readily retrievable form?

– 7% <1 year– 10% <3 years– 13% 5 years– 1% 7 years– 24% 10 years– 1% 30 years– 28% Indefinite– 16% For lifetime of analyser

76 answered question.x

Q.25

• Does your audit trail allow you to identify retrospectively for any patient sample, the IQC data/performance relating to analysis?

• 89% YES• 11% NO

• Free text comments gave as little as 2 hours and one week and several said a month.

• Many said indefinitely or for lifetime of analyser.

Q.26

• Does your laboratory use six sigma ?

• 8% YES (3% in previous regional audit.)

• 92% NO


Q.27

• Does your laboratory hold regular dedicated IQC meetings ?

• 68% YES• 32% NO

• Very similar results in previous regional audit.Frequency ranged from daily up to 6 monthly.

Monthly meetings are the most common scenario.

Q.28

• Is IQC a standing agenda item on governance or management meetings ?

• 81% YES (72% in previous regional audit.)• 19% NO

• Is IQC discussed at dedicated meetings or general senior management meetings ?– 55% Both – 14% Dedicated meeting only– 26% Senior management meeting only– 6% Neither dedicated meeting or senior management meeting


STANDARD 1• IQC policies should cover a 24 hour period.

– 25% of labs still show clear evidence of having IQC practices that vary according to the time of day or whether samples are being run during core hours versus ‘on-call’ or other shift.

There is an apparent improvement compared to the previous audit in which 70% had no policy, BUT -

Remaining labs should develop their policies for 24/7.

Labs should review the validity of IQC testing practice out of hours if it differs to the routine day.

Discretionary IQC testing out of hours MUST BE discouraged.

STANDARD 2

• Target values and ranges for IQC material should be assigned locally for each instrument.

• Manufacturer target levels and ranges should not be used.

• Significant variability remains in the way that labs establish the target value and SD ranges for IQC materials.

• This is unchanged since the previous audit.

STANDARD 3

• Laboratories should use IQC material from a third party source.

• The proportion of labs not using third party QC material remains unchanged since the previous audit.

STANDARD 4

• Use EP5-A2 or equivalent to determine IQC values within each individual laboratory.

• There is clear evidence of significant variability in the way that labs assign target values and ranges to QC material. Since this is vital to the ability of a IQC process to detect true error, this should be addressed.

STANDARD 5

• Encourage method specific rules (single and multi) based on the required quality of the assay, and known method bias and coefficient of variation.

• There has been an apparent increase in the number of labs that utilise method specific rules, but scope for significant further work.

• Other concerns (see later)

STANDARD 6

• Discourage inappropriate use of single rules, but when used, ensure they have the required degree of error detection.

• 55% stated they use single 2SD rules, compared to previous 76%. This is an apparent improvement, but again concerns (see later).

STANDARD 7

• Review grades of staff accepting and rejecting IQC. • The use of trainee and other staff who are not state-

registered to accept and reject QC should be discouraged or they must be supervised.

• Still approximately 10% of labs are allowing either trainees or non-registered staff to make key decisions. This is unchanged since the previous audit.

• A weakness of the audit is that we do not know about the degree of supervision, but we suggest labs review and ensure appropriate supervision is in place or change of practice occurs.

STANDARD 8:• If IQC policies are robust, and the correct

IQC with appropriate error detection is used, failed IQC should not be accepted (e.g. to continue processing patient samples).

• ONLY 18% of respondents said continuing analysis would never happen.

• This remains very disappointing. The reasons quoted are as unconvincing as they were in the original audit.

STANDARD 9

• For analysers that have multiple modules, each module should be assessed individually with IQC on each assessment.

• Not measured in either audit, but principle still proposed.

DISCUSSION POINTS

IQC testing intervals are still very variable. It is unclear how much relates to lack of confidence in

methodology versus poor IQC planning. This is reflected in the large variation in the number of

sodium measurements that labs are making between IQC measurements. Scope for standardisation.

There is a need for our professions to work with manufacturers of IQC material to refine the target concentrations used.

DISCUSSION POINTS

Some free text answers imply a lack of understanding of IQC. For example, some of the labs that stated they use single 2SD rules are using multi-rules. Most of the labs that state they use method specific rules are not.

• Would there be any value in the ACB and/or IBMS designing a questionnaire to evaluate core knowledge and understanding of IQC within the professions ?

There is lack of clarity on what statistical tests best allow services operating on multiple sites to compare data across their whole service see recent paper.

DISCUSSION POINTS

40% of labs still rely on human intervention to prevent patient results being released when IQC indicates a method is out of control. This should be minimised in favour of automated IT systems.

Several labs to need to review the length of time that IQC data is stored for – RCPath guidance exists.

6% of labs that do not formally discuss IQC at either a dedicated meeting or management / governance meetings should review the feasibility of doing this.

COMMENTS FROM WESTGARD

• Analytical failures can be invisible to the laboratory until they escalate to become serious. Maintaining quality in the light of workload pressures is essential.

• The UK are doing more for IQC than in the US. • What the UK have done is very helpful and we would be

keen to ensure labs around the world can learn from it.• Interested to use their website as a forum to disseminate

and discuss findings in the form of articles and essays.• They have data from a UK lab and a European lab that

shows introduction of a robust IQC programme can reduce the number of calibrations by 50% and can reduce expenditure on IQC material by 50% (reduction in number of controls, repeat runs and calibrations).

IQC: planning and implementation strategies. James Westgard.

• Ann Clin Biochem. 2003. 40. 593-611.

• Do not use 2SD control limits.• Do not use the same control rules for all tests.• Do select IQC procedures on the basis of the quality

required and precision and accuracy of the method.• Do minimise false rejections in order to maximise

response to real problems when they occur.• Do build in the error detection necessary to detect

medically important errors by selection of appropriate control rules and numbers of control measurements.

Clinical Biochemistry.Leeds General Infirmary. UK.

• Ann. Clin. Biochem. March 2011. P136-146.• Jassam N, Lindsay C, Harrison K, Thompson D,

Bosomworth MP, Barth JH.• The implementation of a system for managing

analytical quality in networked laboratories.• Defined quality requirements for 71 analytes and

developed software to monitor analytical indices.• 5 sites, single manufacturer.• 9 general chemistry, 7 immunoassay analysers.

The way forward

• Adopt a total IQC programme. • IQC policies should cover 24 hours.• Establish IQC procedures that minimise

false rejections and maximise the detection of medically important errors.

• Encourage method specific IQC rules.• Define control limits based on own data

and reject results which are our of control.• Preferentially use IQC from a third party.

THANK YOU

Thank you for listening.

A big thank you to the 86 labs who responded to the survey and to Westgard for helpful comments.

FOCUS 2011 NATIONAL AUDIT OF INTERNAL QUALITY CONTROL PRACTICE in UK. David Housley. Luton and...

Documents

Transcript of FOCUS 2011 NATIONAL AUDIT OF INTERNAL QUALITY CONTROL PRACTICE in UK. David Housley. Luton and...