Putting the A in ADCL – the work of the Calibration ... · work of the Calibration Laboratory...
Transcript of Putting the A in ADCL – the work of the Calibration ... · work of the Calibration Laboratory...
1
Putting the A in ADCL Putting the A in ADCL –– the the work of the Calibration work of the Calibration
Laboratory SubcommitteeLaboratory Subcommittee
Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens
AAPM Calibration Laboratory Sub-committee
2
What is an ADCL?What is an ADCL?
Accredited Dosimetry Calibration LaboratoryProvides calibrations to users for instruments and radioactive sources for dosimetry in radiotherapy and diagnostic imagingCurrently three ADCLs:
University of Wisconsin (Madison) K&S Associates (Nashville) MD Anderson Cancer Centre (Houston)
http://uwrcl.medphysics.wisc.edu/http://www.kslab.comhttp://rpc.mdanderson.org/adcl/
3
What does the CLA do?What does the CLA do?
“Oversee all activities regarding the operations of the ADCLs. The Subcommittee’s task is to accredit, supervise and maintain the highest level of confidence in the quality of the ADCL system, with sufficient capacity in the system to prevent undue delays in satisfying the membership’s calibration needs while providing a choice of ADCLs.”
Main forum for discussion of issues relating to calibration of ion chambers and brachytherapy sources
4
Some historySome history
In 1971 the AAPM formed a task group to develop guidelines for the establishment of a system of secondary standard calibration laboratories.
The laboratories would be accredited by the AAPM to provide high precision dosimetry calibrations.They would be outside of the National Institute of Standards andTechnology (National Bureau of Standards as it was then) but traceable to NIST.
Purpose:To reduce the time required for precision calibrations To create a system of secondary standard laboratories, To establish a technical resource for the membership of the AAPM
5
Time progresses … Time progresses …
A sub-committee of the Radiation Therapy Committee of AAPM has overseen the development of the program to this day.Names have changed but the focus is still the sameAt the summer meeting of the AAPM in 1995, the Subcommittee initiated a major revision of the accreditation protocol which has resulted in the present criteria that now conform to ISO standard 17025:2005.
6
What does the CLA do?What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment visits
Monitors performance
Makes recommendations
7
What does the CLA What does the CLA notnot do?do?
Does not set prices Does not recommend any one ADCL over anotherDoes not distinguish between commercial and academic organizationsDoes not try to “spread the business around”
8
Registration Registration vsvs AccreditationAccreditation
We all see signs saying “An ISO9001 registered company” but what does it mean?Registration:
Procedure used to give written assurance that a system conforms to specified requirements. Such systems include those established for the management of product, process or service quality, and environmental performance
9
Quality System RegistrationQuality System Registration
ISO 9000 quality systems are genericApplies to all types of organizationsRecognition is of a quality system that’s defined in terms the output (products/services) of the quality systemRegistration does not certify or guarantee the quality of products or service for compliance with specific technical specifications
10
AccreditationAccreditation
Accreditation:Procedure by which an authoritative body gives formal recognition that a body or person is competent to carry out specific tasks
11
Laboratory AccreditationLaboratory Accreditation
Formal recognition that a testing laboratory is competent to carry out specific tests or types of testsKey Words:
competentspecific tests (scope of testing)
Accreditation is having a Quality System anddemonstrating competency.Requires assessment (audit) visits to the laboratory to verify competency.
12
What does the CLA do?What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment visits
Monitors performance
Makes recommendations
13
Laboratory Accreditation:Laboratory Accreditation:System AuditingSystem Auditing
Based on a course by:PROFESSIONAL QUALITY SYSTEMS
Gary D. Scalise, Instructor Technical Lead Assessor A2LA
Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens
AAPM Calibration Laboratory Sub-committee
14
What Really Is An Audit?What Really Is An Audit?
Audit or Assessment: to verify conformancebased on objective evidence
15
Types of AuditsTypes of Audits
First PartyInternalThe laboratory’s own audit
Second PartyExternalUsually by a client to their own specific requirements
Third PartyExternalBy independent organization, e.g. AAPMFull quality system evaluationObjective – done by trained and qualified assessors
16
ISO17025 and the AAPMISO17025 and the AAPM
ISO standard 17025:2005 is the international standard covering calibration laboratories The AAPM has a set of Criteria which are used for the accreditation of ADCLsThe Criteria are based on the requirements set out in ISO17025A lab accredited according to the AAPM Criteria is, by implication, following ISO17025But – the Criteria have additional requirements that must be met. Therefore ISO17025 accreditation is not enough on its own to satisfy use of the ADCL logo
17
The AAPM CriteriaThe AAPM CriteriaThe Criteria are based on the requirements set out in ISO17025In addition there are detailed technicalrequirements for each specific area that will be covered by the scope of accreditation:
Co-60 absorbed dose to waterLDR and HDR brachytherapyDiagnostic radiology dosimetryElectrometer calibration, etc
18
Audits Audits –– the details the details
The slides will focus on the AAPM criteria and what the CLA does
BUTThey can be applied to any assessment of a quality system, quality assurance program, procedure, etcTo both:
external audits (such as carried out by the CLA)internal audits (checking before someone else does)
19
First First -- an audit needs Auditors!an audit needs Auditors!
Understanding of quality system philosophiesIn-depth knowledge of standardsTechnically competent on test methodsExcellent communication skills (written and oral)Ability to probe and analyze – not obnoxiousGood logic, judgement and integrityPoise, tact, persistence and maturityEasy going and positive attitudeNot easy to find!
20
Auditors need to be trained on…Auditors need to be trained on…
The standard (AAPM Criteria, ISO17025…)The organization’s auditing proceduresInformation gathering techniquesThe human relation aspects of auditing
It unfortunately takes more than 1 lectureIt unfortunately takes more than 1 lecture
21
Phases of an AuditPhases of an Audit
Determine ifQuality SystemDocumentationmeets intent ofthe standard
(AAPM Criteria)
Reading beforehand
Evaluate how well Quality System
Policies and Proceduresare put into practice
“Live”
22
PrePre--Audit PreparationsAudit Preparations
Review all pertinent quality system documents• Goal: awareness of specific policies and procedures
Prepare a detailed gap analysis vs. requirements • Goal: understanding of what is there compared to what
should bePrepare questions and checklists for use during auditReview all documents once again• Goal: complete familiarity with the laboratory procedures• Don’t start on the plane
23
Scheduling the AuditScheduling the Audit
Contact the auditeeProvide results of your document review (if possible)Request any additional documentsEstablish the date and duration of the auditProvide checklists you want to have filled outEnsure there is a correct understandingSend auditee an agenda of the audit
24
Typical Audit SequenceTypical Audit SequenceOpening Meeting
Introduction of each auditee
Gathering the information and evidence
Summary with each auditee
Compiling Report
Closing Meeting
25
Opening MeetingOpening Meeting
Gather all the major participantsLaboratory director and quality manager
Supervisors, key personnel, analysts as needed
Discuss the scope of the auditReview the agenda of the auditDiscuss how deficiencies will be handledNo surprises at the closing meeting
26
What’s being looked at? What’s being looked at?
1. Management Requirements From Section 4 of AAPM CriteriaOrganizational Structure, Quality Policy, Technical Management, Records Management, Purchasing Policy, Internal Audit, etc.
2. Technical RequirementsFrom Section 5 of the Criteria (and Appendices)Test methods, Equipment, Analysis, Reporting, etc.
27
Gathering Information and Gathering Information and EvidenceEvidence
Asking QuestionsInspecting Facility and Equipment Examining DocumentsExamining RecordsObserving Activities
28
Limitations of QuestionsLimitations of Questions
Can’t always take answers at face valueAnswers may not be the truthMay be intentionally misleadingCould be coached responsesBut remember:
Auditors are in the evidence businessAuditors are in the evidence businesshunches and opinions don’t counthunches and opinions don’t count
29
Technical ProficiencyTechnical Proficiency
Beyond system auditing, an audit of a laboratory requires demonstrated technical proficiency and competency (Section 5.0)Auditors can verify (see) this four ways:1. Verify suitable equipment2. Verify appropriate procedure/test method3. Interview personnel to verify knowledge4. Observe the test to verify competence
30
Inspect Facility & EquipmentInspect Facility & Equipment
Evaluate design layout of laboratoryProper environmental conditions and monitoring?Sufficient lighting? Temperature?Effective separation between areas?Good housekeeping? Appropriate security?Suitable equipment – range, precision?Status of equipment – maintenance, calibration?
31
Examining DocumentsExamining Documents
Documents available for use? Implemented?Is there proper document control for
Internal documents?External documents?
Document changes properly made?Draft procedures in use?Obsolete documents removed?Verify that testing follows written procedures
32
Examining RecordsExamining Records
Quality and technical records in place?Records allow traceability and repetition?Procedures being followed for archiving?Records include identity of all personnel?Contemporaneous records? Complete?Error corrections properly made?
33
Observation of ActivitiesObservation of Activities
Witness demonstrations of procedures by staffConfirms what we heard from our questionsConfirms that analysts are following proceduresDetermines status of equipment and setupEnables us to judge technical skillsEstablishes the level and depth of trainingGains information on supervision
34
Aside Aside –– CLA surveillance CLA surveillance
There is no requirement in a surveillance visit to witness a procedure (calibration)However, it may be useful:
if there is any confusion with documentationif it’s a simple procedureto help familiarize the auditor with the lab facilities and personnel if the lab requests it
35
Compare Audit Findings to…Compare Audit Findings to…
AAPM Criteria requirements (or relevant standard)
Need specific clause #The internal quality system requirementsTechnical laboratory proceduresAgreed methods with the clientCompare reality to requirements,
But remember the rule…
36
#1 Rule of Auditing #1 Rule of Auditing
If you cannot express the non-conformance in the words of the quality system standard,
the appropriate technical specification or test method, or the company’s own policies
and procedures,
THEN YOU DO NOT HAVE A THEN YOU DO NOT HAVE A NONNON--CONFORMANCECONFORMANCE
37
Audit ReportingAudit Reporting
Reporting has more than one aspectVerbal reporting
Feedback to the auditee at time of finding
Closing meetingSummarize findings to management at close of audit
Written deficiencies statementsDocumented audit report
Documented as soon as possible (same day)
38
RecommendationRecommendation
Don’t keep tight-lipped until the presentation of the report because:
Shock and Awe are not legitimate assessor toolsInformation may be power but you are really there to assist not controlIf a finding is simple to rectify then it can be cleared before you leave (and that’s easier for everyone) You may be mistaken than it’s a non-conformance
39
When Something Seems WrongWhen Something Seems Wrong
Is it really wrong?Do the lab personnel know its wrong?What is their explanation?Is it an isolated event, or a symptom of a deeper problem?Why didn't the quality system detect it?What lapse in the quality system allowed this to happen?
40
Compiling the Audit ReportCompiling the Audit Report
Writing the Non-Conformances (Deficiencies)Assembling the complete audit report
Scope of Audit: What, How, WhyAgenda of Audit: When, Where, How LongIdentification of auditor(s) and auditee(s): WhoNon-ConformancesCompleted checklistsTime frame for corrective action responses
Make sure you leave time for this!Make sure you leave time for this!
41
What is a nonconformity?What is a nonconformity?
Definition:
“non-fulfillment of a requirement”
An equivalent term is “deficiency” “Non-compliance” should be avoided as it has a legal implication
42
What is an “audit finding?”What is an “audit finding?”
Definition:“result of the evaluation of the collected audit evidence against audit criteria”
Findings can be good or “not so good”Can indicate conformity, nonconformity or opportunities for improvement
44
Writing a DeficiencyWriting a Deficiency
Structure of a deficiency statement
The Requirement
Description of the deficiency
The objective evidence
45
Writing DeficienciesWriting Deficiencies
Directly related to the relevant quality system standard, technical procedures, laboratory policies and procedures, etc.And based on a clear statement of objective evidence recorded during the audit
46
RecommendationsRecommendations
Auditors are free to make recommendations that result from audit findings or general observations of the laboratoryLaboratories are free to ignore any recommendations!But recommendations can be a real help to the development and improvement of a QS (the lab is effectively getting free consultancy) and therefore should be considered
47
Closing MeetingClosing Meeting
Gather participants – everyone if possibleReview the compliance areas – firstReview the non-compliance issuesBe careful to distinguish between non-compliances and recommendationsComplete any recordsThank people for their assistance and time
48
Corrective ActionsCorrective Actions
A deficiency triggers the need for corrective actionThe auditee is responsible for corrective action
Conduct a root cause investigation for eachIdentify potential corrective actionsDetermine the significance and resourcesEvaluate possible effects on other departments or other elements of the quality systemEstablish a time frame for implementationReference each corrective action to one deficiency
49
FollowFollow--Up on DeficienciesUp on Deficiencies
Auditee should first monitor that the corrective action brought desired resultsVerify that the agreed corrective action has been effective in the long termAdditional audits may be warranted if the laboratory is still out of compliance
50
Audit ClosureAudit Closure
An audit is formally closed when…All reported nonconformities are addressed by corrective action plan
Responsibility of auditee
Corrective action is implementedResponsibility of auditee
Corrective action is verified as effectiveResponsibility of audit program manager (CLA)
51
Audit summaryAudit summary
Audits are a valuable tool for any process-based organizationAudits are compliance basedAn auditor only writes deficiencies when objective evidence has been gatheredAll deficiencies must directly relate to the relevant standard, test method, etc.Remember the #1 Rule of Auditing
52
What does the CLA do?What does the CLA do?
Develops criteria
Accredits laboratories
Carries out assessment visits
Monitors performance
Makes recommendations