1 Annual Assessor Training 2015 PER Program Vijay Krishna.

1

Annual Assessor Training 2015PER Program

Vijay Krishna

2015 ANSI Annual Assessor Training Slide 2

Activity

Take a minute to write the following:One new thing that you learnt/discovered about

the standard/accreditation process since the last annual training.

One thing that you believe will improve consistency among assessors in assessing the standard.

One thing that you want to get in the next three days.


Progress towards 17024:2012 compliance

ISO/IEC 17024: 2012 completed

ISO/IEC 17024: 2012: 2012 in progress

ISO/IEC 17024: 2012: 2012 in first quarter of 2014

23 20 8


Progress towards 17024:2012 compliance

ISO/IEC 17024: 2012 completed

ISO/IEC 17024: 2012: 2012 in progress

ISO/IEC 17024: 2012: 2012 in first quarter of 2014

23 20 9

23 NCRs 55 NCRs


Requirements generating most NCs

4.3 Management of impartiality

4.3.6 The certification body shall identify threats to its impartiality on an ongoing basis. This shall include those threats that arise from its activities, from its related bodies, from its relationships, or from the relationships of its personnel.



9.6 Recertification process 9.6.3 The recertification period shall be based upon the scheme requirements. The rationale for the recertification period shall take into account, where relevant, the following: regulatory requirements; changes to normative documents; changes in the relevant scheme requirements; the nature and maturity of the industry or field in which the certified person is working; the risks resulting from an incompetent person; ongoing changes in technology, and requirements for certified persons; requirements of interested parties; h) the frequency and content of surveillance activities, if required by the scheme.



9.6.5 In accordance with the certification scheme, recertification by the certification body shall consider at least the following: on-site assessment; professional development; structured interviews; confirmation of continuing satisfactory work and work experience records; examination; checks on physical capability in relation to the competence concerned.



10.2.5 Management review

10.2.5.2Review Input

10.2.5.3 Review Output


Minutes from 2014 AAT

4.3.6b) Provide an impartiality threat analysis report for all the related bodies and steps taken by the certifying agency to eliminate/minimize them. What might a threat analysis look like?Content of threat analysis needs to cover the 5 types of threats (listed in annex ownership, governance, management, personnel, shared resources, finances, contracts, marketing (including branding) and payment of a sales commission or other inducement for the referral of new applicants, etc. Relationships of personnel and related bodies; actual and perceived threats The report could be presented in different form. It could be part of management report, minutes, matrix/table, or a formal threat analysis report.


4.3.6 Impartiality threat analysis

Issues to consider Identified threats and what is done to address

threatsSMEs on committees who are also being allowed

to teach cert scheme related coursesMeeting minutes or other documentation that

demonstrates an “ongoing basis”


Discussion question 8.2: What are the issues relating to job/practice analysis that needs further clarification?

The scope of the JTA should be clearly defined.The JTA should involve sufficient stakeholders to

represent the field.The JTA should be validated with a wider group. If

the job analysis involved a process such as DACUM with sufficiently large number of SMEs, then an survey may not be required.


Discussion question 9.3.1: What are the issues relating to standard setting that need further clarification?

The cut score panel should attempt to represent the field. Even though it is not possible to come up with a fixed minimum number of experts for a cut-score study, it was decided to go with the generally acceptable minimum number of six. If the CB uses less than six members for cut-score study, it would have to provide a rationale for doing so and the assessors would have to determine if the rationale is valid and acceptable.

When the cut score is adjusted within the acceptable SEM, the CB should document the rationale.


Discussion question 9.1.2: Does the standard require a certification body to review each application to confirm meeting the eligibility requirements? Yes.

Discussion question 9.1.3: Would self attestation be acceptable means of meeting eligibility requirements such as work experience and/or education? Just checking a box indicating meeting the eligibility requirements would not be acceptable. The CB would have to provide some objective evidence.


Discussion question: What part of the management system requirement needs further clarification? The scope of management system should include all certification related activities.When a certification body uses ISO 9001, it needs to ensure that all the certification activities are covered within its scope. The CB would have to provide the necessary documentation and it would not be acceptable if the CB just provided a certificate indicating compliance with ISO 9001.


9.4.9 The certificate shall be designed to reduce the risks of counterfeiting.

Discussion question 9.4.9: What are some ways to meet this requirement?This could include unique identification number, holograph, special ink, non-duplicable paper etc. The CB would have to show how they have reduced the risks of counterfeiting. The CB’s process for issuing duplicate certificates should also be looked into.


Separation of Training

5.2.3 d) not require the candidates to complete the certification body's own education or training as an exclusive prerequisite when alternative education or training with an equivalent outcome exists; What evidence would be acceptable to establish that equivalent outcome does not exists? The CB would be required to establish equivalence of training content. The CB is not required to conduct a psychometric analysis to establish equivalence of training outcomes.


Minutes from 2012

Job Task Analysis

Minimum required: The CB has to attempt to represent the field which is the scope and purpose of the certificationThe CB has to validate the JTA with either a sample of SMEs or a survey of practitioners. When a CB uses an existing standard (like ASME, OSHA, ISO, ANS), it still has to conduct a validation study. The JTA must document elements of the study The JTA must document the outcome: A blueprintIt is not required that JTA should be presented in a single report as long as all the elements of the study are documented and provided to ANSI


Standard Setting Study

Minimum required:The CB must document the standard-setting processThe CB must document the people involved in setting the standard and the rationale for their selection The standard setting process must be a criterion-referenced methodThe CB must document the way in which the field was represented fairly by the process


Exam Analysis

Minimum required:CB must keep candidate data and aggregate it as appropriate CB must evaluate / review the dataCB must do item analysisCB must establish the reliability of the exam The CB must use the analysis to ensure the health of the exam.CB must establish that exams are fair


Management System

Minimum required:The management system has to function as a system Document controlWhat’s controlled and what’s not controlled should be identified. Must be fully implemented & reflected in submissionMust catalog documents systematicallyMust document a change request processAssurance that the current version of a document is accessed


Internal audit Audit plan must be documented At a minimum, all the certification processes and procedures must be audited

with an accreditation cycle of 5 years. An auditor cannot audit his/her own work A CB must have completed one internal audit before being accredited. A CB must have completed a management review of the internal audit before

being accredited. A management review must happen at least once every year.

The auditor must be sufficiently knowledgeable about the procedures/process to conduct the audit.

During annual surveillances, the CB must provide evidence of corrective and preventive actions for any deficiencies identified during the internal audit. The CB should do a root cause analysis to identify underlying issues.

As part of internal audit, psychometric functions (process) need to be audited, however, the standard does not require auditing the psychometric results/content (unless CBs policies requires it)


Monitoring Subcontractors

Minimum required:The CB should ensure that the provisions of the contract are being fulfilled. The depth of review depends on the criticality of the function/s performed by the subcontractors and the potential to impact the integrity of the certification. It is not required that the subcontractor or testing delivery vendor should a have management system that covers the entire scope of 17024. However, all the elements relevant to the performance of the contract must be reviewed by the CB.


Financial Statements

Minimum required:Must have a financial statement including balance sheet & income statement for 3 years or less if the organization has not been in operation The financial statement should have a breakout of certification income and expenditure; however, break out by scopes is not required.The CB must show sufficient revenue &/or reserves to maintain the integrity of the program


Other Issues:

It is not the role of an assessor to recalculate the test statistics. During the review, if there are any red flags relating to the calculations in the overall data, it should be reported.

Recalculation of data should only be done to pinpoint problems suggested by overall analysis of the data.

It’s not required that we categorize items into concepts, knowledge or skills


Other Issues:

Assessors should cite a NC once if it occurs repeatedly. Cite it under the major section that subsumes it, and note the other areas it impacts. The question is what action would it take to close out the citation? If it takes other actions to close out the nonconformity, then there are multiple nonconformities.


Overview of accreditation process

• Agenda two week prior to the onsite• 2 day onsite + performance test site• Initial report within 2 weeks• Corrective actions within 4 weeks• Response to NCs within 2 weeks

• Approval of the findings

• Up to 12 months to close the NCs


Annual surveillance

4-6 weeksReport 2 weeksResponse to NCs 2 weeksResponse to corrective actions 2 weeks


Expected timelineInitial accreditation

Surveillance

Days per assessor Total days

Desk review 2-3 4-6

Onsite 2 days 4

Travel time

Total report writing 1

Corrective action review 1

Total 11-13 days

Days per assessor Total days

Desk review 2 4

Onsite 1

Travel time

Total report writing 1

Corrective action review ½

Total 6- 7 days


Closing a non-conformity

What constitutes implementation?


New PCAC guidance document on test administration

1 Annual Assessor Training 2015 PER Program Vijay Krishna.

Documents

Transcript of 1 Annual Assessor Training 2015 PER Program Vijay Krishna.