2017-01-23

1

Assessment Assessment

MethodologiesMethodologiesWorkshopWorkshop

Pro Board Annual ConferencePro Board Annual Conference

San Diego, January 20/21, 2017San Diego, January 20/21, 2017

Assessment MethodologiesAssessment Methodologies

Presenters:Presenters:

• Fred Piechota

• Brian Brauer

• Chris Senaratne

• Craig Richardson

Assessment MethodologiesAssessment Methodologies

Objectives:Objectives:

• Overview of the Pro Board assessment

methodologies

• Discuss some examples of each

• Application for change in accredited process

• Group dialogue on project methodology

direction

• Q&A

Assessment MethodologiesAssessment Methodologies Applicable CriteriaApplicable Criteria

TD2:TD2:

• The agency shall have the ability to completely

test the level(s) of each applicable standard

with justifiable test methodologies.

2017-01-23

2

Applicable CriteriaApplicable Criteria

TD3:TD3:

• Procedures shall be in place in the test

development, administration, review, and

improvement processes to ensure the

production of valid and reliable test

instruments and test results.

Applicable CriteriaApplicable Criteria

TA5:TA5:

• Procedures shall be in place to ensure that skills tests are assessed by qualified persons who are not involved in the training of the candidate in the skill being tested; that qualified persons administer (proctor) cognitive tests; and that qualified persons evaluate projects, portfolio submissions, or other methodologies (if used).

Applicable CriteriaApplicable Criteria

TA6:TA6:

• Procedures shall be in place to ensure that

skills tests, written exams, projects, portfolio

submissions, or other evaluation

methodologies are administered in a

consistent and impartial manner.

Written CognitiveWritten Cognitive Written CognitiveWritten Cognitive

Definition:Definition:

• The cognitive (knowledge) assessment methodology is used to assess a candidate’s subject matter knowledge by means of a written or oral test in which the candidate is required to provide specific answers to specific questions related to the JPRs of the standard/level to which the candidate is seeking certification.

2017-01-23

3

Written CognitiveWritten Cognitive

• Assessment made through a Test Instrument –

a set of questions used to evaluate the

candidate

• Test Instrument is made up of a multiple

number of Test Items (individual questions)

from a Test Bank

Written CognitiveWritten Cognitive

• The test bank must include test items that are designed to assess, and correlated to, each of the areas of the standard that are going to be tested cognitively.

• Random selection from that correlated bank of test items can be used to generate a test instrument.

• Another methodology of selection is stratified or targeted random selection, where the random selection is done from the subsets of test items for each section of the standard for which cognitive evaluations are used.

Written CognitiveWritten Cognitive

• Test Banks can be developed locally or

purchased

• Development processes of NFPA, PTS, IFSTA,

Jones & Bartlett have been approved by

ProBoard

• Local validation by subject matter experts is

necessary for both locally developed or

purchased test banks

Psychomotor SkillsPsychomotor Skills

Psychomotor SkillsPsychomotor Skills

Definition:Definition:

• The psychomotor (skills) assessment methodology is used to assess a candidate’s ability to perform a physical task by means of a real-time skills performance evaluation. The candidate is required to correctly perform the physical task/skill identified by the JPR and is evaluated on his/her performance in real-time as the candidate performs the task.

PortfolioPortfolio

2017-01-23

4

PortfolioPortfolio

Definition:Definition:

• The portfolio method of evaluating personnel for certification may be utilized to evaluate job performance requirements, requisite knowledge, requisite skills, and objectives of standards when other methods of evaluation are not applicable, or when the accredited agency identifies that the portfolio method is an appropriate method of evaluation on the standard-to-test correlation matrix. The portfolio process is intended to measure the candidate’s accomplishments with respect to specific job performance requirements (JPR) or objectives within the various levels of certification that are part of the National Professional Qualifications System.

PortfolioPortfolio

Definition:Definition:

• The portfolio method of evaluation shall not

be confused with other processes used to

identify or verify that a candidate has met a

single job performance requirement or

objective, such as letters of proof from a

supervisor, certificates, transcripts, etc.

PortfolioPortfolio

Definition:Definition:

• The Portfolio process is based on the

evaluation of a candidate’s experience,

training, and education to ensure compliance

with the job performance requirements or

objectives of the appropriate standard.

PortfolioPortfolio

• Different from assessing pre-requisites or establishing an entry point into the certification system

– because certification can be granted upon completion

• Must be a documented, valid and reliable process that correlates the candidate’s submitted accomplishments to the JPRs in the level to which certification is sought

Other MethodologiesOther Methodologies

Definition:Definition:

• The other assessment methodology column in

the Assessment Methodology Matrix is used

to identify a valid and reliable assessment tool

for all or part of a certification assessment

that is not described above.

ProjectProject

2017-01-23

5

ProjectProject

Definition:Definition:

• The project-based assessment (PBA) method of evaluating personnel for certification may be utilized to evaluate job performance requirements, requisite knowledge, requisite skills, and objectives of a standard. The instrument used to evaluate the certification candidate using the PBA method is known as a project.

ProjectProject

Definition:Definition:

• A project is an assignment that requires the application of knowledge to perform a task that is not assessed/evaluated as it is being performed. The completion of the task must include a requirement for the candidate to submit an output product that can be assessed by a project evaluator to determine success of the candidate.

ProjectProject

Definition:Definition:

• An assessment is likely a project if: – the evaluator is not required to witness the application of the

knowledge for the task taking place (e.g., skill set to develop a budget cannot be witnessed by an evaluator), or

– the assessment has a “take home” component or a component that is completed without direct supervision of a proctor or evaluator (e.g., receive an assignment during class and have to take it home to complete it), or

– the assessment has several plausible outcomes and the evaluation tools must be flexible enough to allow for that (e.g., develop a capital improvement plan that could be accomplished several correct ways), or

– has 6 months to complete a task and submits a product for final evaluation)

ProjectProject

Definition:Definition:

• Typically, a PBA is an evaluation of a product (such as a budget, long range plans, reports, or letters) rather than a process (such as donning, tying knots, raising ladders, or decontamination). If the evaluator has to watch someone complete each individual step, it is likely a psychomotor skill; however, if an evaluator does not have to witness each step but rather evaluates a completed product, it is likely a PBA.

ProjectProject

Definition:Definition:

• The Self-Assessment has several criteria that must be addressed if an agency elects to use PBA. There must be a process for evaluating assigned projects, as found in the Test Administration and Test Development sections of the Self-Assessment, that accurately measure compliance with a specific job performance requirement, requisite knowledge, requisite skill or objective. It is preferred that the instructor is not the evaluator; however, where an agency has established an approved evaluation tool to score the project (i.e. a rubric, scoring criteria, or model example used by the evaluator to assess all the candidates in a consistent manner), it may be acceptable that the instructor evaluates the candidates. This tool must have been developed in accordance with the agencies policies as defined in the Test Development section of the Self-Assessment.

Change in Accredited ProcessChange in Accredited Process

2017-01-23

6

Change in Accredited ProcessChange in Accredited Process

• Similar to the application to extend to a new standard or level

• Agencies are accredited based upon the processes submitted in the application and self assessment document; and at a site visit

• Your work is dynamic and your business model often changes to remain responsive to constituents

• New process must be approved

Change in Accredited ProcessChange in Accredited Process

Examples:Examples:

• Changing test development procedures (i.e. switching test bank providers or beginning to develop test questions in-house)

• Beginning the use of a new assessment methodology

• Introducing remote testing when not previously approved

• Online testing

Change in Accredited ProcessChange in Accredited Process

How it works:How it works:

• Complete the application form

• Assess the new process for how it differs from existing approved processes and identify the criteria that relate to the new process

• Answer the appropriate sections of the self assessment

• Provide a detailed summary of the new process and how it is administered

ProjectProject

2016 Conference Information:2016 Conference Information:

• Compliance with definition upon any of the

following occurring after July 1, 2016:

– Next scheduled site visit

– Application to extend to new levels

– NFPA Standard new edition

ProjectProject

Where we are now:Where we are now:

• COA meeting recap

• Some gaps to address

• Engage agencies in establishing a process to

move forward

• Testing JPR vs Testing RK & RS

COA Opinion 18COA Opinion 18

• The Committee on Accreditation (COA) has ruled that when completing a Standard to Test Correlation Matrix, the completion of either the Job Performance Requirement (JPR) section of the matrix or the Requisite Knowledge (RK) and the Requisite Skill (RS) sections of the matrix meets the COA’s requirements. It follows therefore, that the evaluation process shall measure either the JPR or the RK and RS for each objective in the standard.

2017-01-23

7

NFPA 1033 Annex B

• B.3.1 Certification. JPRs can be used to

establish the evaluation criteria for

certification at a specific job level. When used

for certification, evaluation must be based on

the successful completion of JPRs.

NFPA 1033 Annex B

• First, the evaluator would verify the

attainment of requisite knowledge and skills

prior to JPR evaluation. This might be through

documentation review or testing.

NFPA 1033 Annex B

• Next, the candidate would be evaluated on

completing the JPRs. The candidate would

perform the task and be evaluated based on

the evaluation parameters and/or

performance outcomes. This performance-

based evaluation can be either practical (for

psychomotor skills such as “ventilate a roof”)

or written (for cognitive skills such as

“interpret fire patterns”).

NFPA 1033 Annex B

• Note that psychomotor skills are those

physical skills that can be demonstrated or

observed. Cognitive skills (or mental skills)

cannot be observed but are evaluated on how

one completes the task (process-oriented) or

on the task outcome (product-oriented).

Which Methodology?Which Methodology? Which Methodology?Which Methodology?

2017-01-23

8

Which Methodology?Which Methodology? Which Methodology?Which Methodology?

Which Methodology?Which Methodology? Which Methodology?Which Methodology?

Which Methodology?Which Methodology?