1

ModuleModule

PANEL TESTINGPANEL TESTING

Intermediate level

2

Content OverviewContent Overview

• What is panel testing?• What is panel testing used for? • Organization of a panel testing round• Analysis of results; scoring system• Forms• Feedback

3

What is Panel Testing?

• One of EQA methodsOne of EQA methods

• System of sending stained and/or System of sending stained and/or unstained smears from NRL to unstained smears from NRL to peripheral laboratories to check peripheral laboratories to check proficiency in performing AFB proficiency in performing AFB smear microscopy and reporting smear microscopy and reporting AFB resultsAFB results

• Tests individual performance of a Tests individual performance of a laboratory worker, not the laboratory worker, not the laboratory overalllaboratory overall

4

• Rapid assessment of performance of a laboratory Rapid assessment of performance of a laboratory staff to prioritize training and supervisory activitiesstaff to prioritize training and supervisory activities

• Quick detection of problems associated with very Quick detection of problems associated with very poor performancepoor performance

• Evaluation of competency of laboratory technicians Evaluation of competency of laboratory technicians prior to and following trainingprior to and following training

• Supplement to rechecking programsSupplement to rechecking programs• Investigation of excessive errors found in recheckingInvestigation of excessive errors found in rechecking

What is Panel Testing Used For?

5

Panels’ Composition

• The composition of a panel set is determined by NRLThe composition of a panel set is determined by NRL

• 10 slides per panel sets: stained and unstained 10 slides per panel sets: stained and unstained smearssmears

• Unstained smears:Unstained smears:• Evaluate staining technique; provide information about stain Evaluate staining technique; provide information about stain

preparation and qualitypreparation and quality

• Different degrees of difficultyDifferent degrees of difficulty• degree of difficulty is usually determined by number of scanty degree of difficulty is usually determined by number of scanty

and low-positive smearsand low-positive smears

6

National reference laboratory

Intermediate laboratory

Peripheral laboratories

A POSSIBLE SCHEME OF A A POSSIBLE SCHEME OF A PANEL TESTING ROUNDPANEL TESTING ROUND

Peripheral laboratories

7

Getting Started: Issues to Consider

•System for sending slides

•Frequency of testing

•Forms to record and report results

•Time allowed for technicians to complete PT

•Availability of microscopes

•Scoring system, performance criteria

•Analysis of PT results, feedback

8

Sending Slides

• Delivery system based on services, regulations, resources available:• mail/post• courier• supervisory visit

• Turnaround time

• Safe package to prevent breakage of slides:• strong plastic slide holders

9

Performing a Panel Test Round• Frequency: at least one to two times a yearFrequency: at least one to two times a year

• A standardized PT reporting form / an accompanying letter to A standardized PT reporting form / an accompanying letter to provide instructionsprovide instructions

• Individual, not group workIndividual, not group work

• No incentives or punitive actions as a result of the PT exerciseNo incentives or punitive actions as a result of the PT exercise

• Time allowed to complete the PT exercise, Time allowed to complete the PT exercise, maximum:maximum:• 2 hours for a stained slide set2 hours for a stained slide set• 3 hours for an unstained slide set3 hours for an unstained slide set

10

Individual Results of Panel Testing / Feedback

Form

Central Laboratory Use Only:

Test slide set No: Passsing score:

Date of sending set: Date results received:

Peripheral Laboratory:

District:

Date PT conducted:

Name of technician reading test smears:

Slide Number Result Expected result Error Type Points

Total Points: Pass / Fail:

HFP: HFN: LFP: LFN: QE:

Recommended Action:

Supervisor:

Central Laboratory Only

Feedback:

Form PT3: PANEL TESTING INDIVIDUAL RESULTS AND FEEDBACK FORM

Note: If more than one technician performs AFB microscopy in the laboratory, each technician should read all 10 smears and record their results on a separate form. Technicians should not discuss results or share forms until all results have been sent back to the central laboratory. Forms for all technicians should be sent to the central laboratory for evaluation.

11

Management of PT During a Supervisory Visit

• Administration of PT during on-site visits:Administration of PT during on-site visits:• can be effective in some circumstances can be effective in some circumstances • provides direct observation of work under PT exerciseprovides direct observation of work under PT exercise• corrective action may be easily facilitatedcorrective action may be easily facilitated• BUT: may be impractical in routine conditions BUT: may be impractical in routine conditions • can be done in a special surveycan be done in a special survey

• Important:Important: PT must not disrupt routine patients’ PT must not disrupt routine patients’ examinationsexaminations, therefore consider:, therefore consider:

• Careful planning of a supervisory visit Careful planning of a supervisory visit • Allocating sufficient time for a visitAllocating sufficient time for a visit

12

Analysis of PT Results

• Scoring system to distinguish major and Scoring system to distinguish major and minor errors, for example:minor errors, for example:

Set of 10 slides, each slide is worth 10 points, total Set of 10 slides, each slide is worth 10 points, total possible score = 100possible score = 100• HFP and HFN scores 0HFP and HFN scores 0• LFP, LFN and QE scores 5LFP, LFN and QE scores 5

(QE = 2 grades difference) (QE = 2 grades difference) • Passing score = 80 – 90Passing score = 80 – 90

13

Result of Controller Result of Technician Negative 1-9 AFB/100 f 1+ 2+ 3+ Negative correct LFN HFN HFN HFN 1-9 AFB/100 f LFP correct correct QE QE 1+ HFP correct correct correct QE 2+ HFP QE correct correct correct 3+ HFP QE QE correct correct

Types and Classification of Errors

Correct: No errorsQE Quantification error Minor errorLFN Low False Negative Minor errorLFP Low False Positive Minor errorHFN High False NegativeMajor errorHFP High False Positive Major error

14

Analysis of PT Results

• Study the aggregate results from all laboratories Study the aggregate results from all laboratories

• Post-validate panel slides/batchesPost-validate panel slides/batches• Assure that poor performance is not due to panel slide Assure that poor performance is not due to panel slide

problems in NRLproblems in NRL

• If a majority of technicians fail to report correct results for the same slide/batch it may represent a problem with slide preparation at NRL:

→ exclude this slide from scoring exclude this slide from scoring

→ check returned discrepant slidescheck returned discrepant slides

→ detect problems in preparation of panel smearsdetect problems in preparation of panel smears

→ undertake measures to improve the quality of panel undertake measures to improve the quality of panel smears preparationsmears preparation

15

An Example of Post-Validation

25 26 27 28 29 30 31 32 33 34

A B C D E F G H I J

Slide 1 42 stained 3+ 1+ 2+ 3+ 1+ 2+ 3+ 3+ 3+ 3+ 2+

Slide 2 46 stained neg 1/100 neg neg neg neg neg neg neg neg neg

Slide 3 38 stained 1+ 8/100 1+ 1+ neg 1+ 1+ neg 8/100 1+ 1+

Slide 4 49 stained neg neg neg neg neg neg neg neg neg neg neg

Slide 5 56 stained 5/100 3/100 neg 5/100 neg 3/100 5/100 neg 4/100 neg 3/100 many erors with the same batch

Slide 6 49 stained neg neg neg neg neg neg neg neg neg neg neg

Slide 7 56 stained 5/100 neg 5/100 5/100 neg 3/100 3/100 neg 5/100 neg 3/100 many erors with the same batch

Slide 8 46 stained neg neg neg neg neg neg neg neg neg neg neg

Slide 9 52 stained neg neg neg neg neg neg neg neg neg neg neg

Slide 10 45 stained 2+ 1+ 1+ 1+ 1+ 1+ 1+ 1+ 1+ 2+ 8/100batch is consistent but AFB quantity is at low acceptable limit

Peripheral Laboratories' ResultsSlide No Batch No

Stained or unstained

Expected result

Slide Sets Numbers

Comments

16

Interpretation of Results

• False positive and negative errors should be False positive and negative errors should be considered separatelyconsidered separately

• False positives - lack of proficiency / faulty False positives - lack of proficiency / faulty microscopemicroscope

• False negatives - poor stain / inadequate False negatives - poor stain / inadequate examination time / poor microscopeexamination time / poor microscope

17

Feedback to Laboratories on PT results

• Timely and confidentialTimely and confidential

• Individual and aggregate test results Individual and aggregate test results

• Criteria for acceptable performanceCriteria for acceptable performance

• Reports to TB program coordinator should provide Reports to TB program coordinator should provide appropriate background information and appropriate background information and recommendations and not simply scores recommendations and not simply scores

• Poor performance often requires a visit to laboratoryPoor performance often requires a visit to laboratory

18

PT Aggregate Results of Multiple Laboratories

District: District Supervisor:

Supervising Laboratory: Period PT conducted:

Panel test slide set(s): Passing score:

Peripheral Lab Annual volume SPR, %Technician(s)

participated in PTPT score HFP HFN LFP LFN QE Total errors

District Averages

Report submitted by: Date:

SPR: slide positivity rate; PT - panel testing; Annual volume - annual volume of smear examinations; HFP - high false positives; HFN - high false negatives; LFP - low false positives; LFN - low false negatives; QE - quantification errors

Form PT4: PANEL TESTING REPORT OF MULTIPLE LABORATORIES FOR DISTRICT SUPERVISOR AND NTP

19

PT Aggregate Results Report: Example

20

Key Messages:

• PT is an effective method when it is necessary to quickly obtain information about capabilities of individual laboratory technicians to read smears and report results according to standards approved by NTP.

• A well functioning system should be established to A well functioning system should be established to distribute panels, collect and analyze data; provide distribute panels, collect and analyze data; provide timely feedback to peripheral laboratories.timely feedback to peripheral laboratories.