August 2007 FFP 2741 1

www.floridastatefirecollege.org

Testing and Evaluation Testing and Evaluation Techniques Techniques

Chapter 7Chapter 7

Florida State Fire College

Ocala, Florida

August 2007 FFP 2741 2

Testing and EvaluationTesting and Evaluation

You had the opportunity to cover part of this chapter in Fire Service Course Delivery

August 2007 FFP 2741 3

Terminal ObjectiveTerminal Objective

The student will be able to construct, administer, and evaluate an assessment instrument

August 2007 FFP 2741 4

Enabling ObjectivesEnabling Objectives

The student shall be able to:Define the four levels of evaluationDifferentiate between summative and formative evaluationDefine the different kinds of testsDiscuss the difference among the various types of testsList various sources for tests

August 2007 FFP 2741 5

Four Levels of Evaluation Four Levels of Evaluation

Level I – reaction

Level II – learning

Level III – transfer

Level IV – business results

August 2007 FFP 2741 6

ReactionReaction

How did the student react to the class?

Evaluation form

Focus groups (rarely used)

Required by most academic institutions

August 2007 FFP 2741 7

LearningLearning

What has the student learned?

Oral examination

Written test

Skills assessment

August 2007 FFP 2741 8

TransferTransfer

How much did the student retain?

Measured six weeks to six months post coursework – on the job

Based upon tests, observations, surveys, and interviews

Not regularly done within the fire service

August 2007 FFP 2741 9

Business ResultsBusiness Results

An assessment of the financial impact of the training or the return on investment.

Six months to two years post coursework

Most difficult to measureTraining courses do not have business

objectives – ie: reduction in accidentsMethodology for assessment is not yet refined

August 2007 FFP 2741 10

Do we ever get a final evaluation?Do we ever get a final evaluation?

In teaching you cannot see the fruit of a day's work.  It is invisible and remains so, maybe for twenty years. 

~Jacques Barzun

August 2007 FFP 2741 11

EvaluationEvaluation

“An evaluation is a process of making a value judgment based upon one or more sources”

Evaluation processes examine two components:

Instruction from the teacher

Performance of the student on objectives

August 2007 FFP 2741 12

Purpose of EvaluationPurpose of Evaluation

Provides feedback to students

Provide students gratification and motivation

Measures the effectiveness of the instructor

Measures the effectiveness of the program in meeting objectives

August 2007 FFP 2741 13

Formative EvaluationFormative Evaluation

Ongoing evaluation to change or adapt the program

Compares the objectives to the testing strategy

Occurs during development of course or testPilot testing is formative evaluation

August 2007 FFP 2741 14

Summative EvaluationSummative Evaluation

Typically performed at end of programProvides students feedback on mastery of subjectProvides feedback on effectiveness of teaching strategySummative toolsCourse evaluation formsFinal exams

WrittenPractical

August 2007 FFP 2741 15

Formal EvaluationFormal Evaluation

“the test”Did the student attain

the course objectives?Gives a gradeAdds stress to the

student If required to pass, it

should not be the first testing of the material

August 2007 FFP 2741 16

Informal EvaluationInformal Evaluation

Provides student feedbackWith or without a

recorded gradeHelps ID weaknesses

and strengthsUse caution when

presenting

August 2007 FFP 2741 17

TestsTests

Instructor’s last chance to emphasize the important information the student needs to retainWrittenPractical Oral

August 2007 FFP 2741 18

Written TestsWritten Tests

Multiple Choice

True/False

Matching

Completion or fill in the blank

Essay

August 2007 FFP 2741 19

Guidelines for Written TestsGuidelines for Written Tests

Be sure questions relate to objectivesAllow appropriate amount of timeSimple multiple choice allow about one minute

per questionScenarios and essays require more time

Clear complete directionsBe sure it has proper grammar and punctuation

August 2007 FFP 2741 20

Multiple Choice TestsMultiple Choice Tests

Common method for formal and informal evaluationNormally used for national and state certification examinationsEasy to grade and be objective

Test construction miscuesUsing previous questions informationNegatively worded stemsFill in the blank in middle of stemUsing ‘all of the above” or “none of the above”

August 2007 FFP 2741 21

True/False TestsTrue/False Tests

Limited to two answers, no gray area

Difficult to construct in positive voice

Avoid always or never statements

Useful tool as a study guide

August 2007 FFP 2741 22

Matching TestsMatching Tests

Works best with definitions and terms

Difficult to design

Cautious of multiple matches

Test directions must be clear

August 2007 FFP 2741 23

Completion TestsCompletion Tests

Fill in the blankStatements must be clear as to intent of questionNeed to be grammatically correctBe aware of size of blankAvoid having blank at beginning of sentence

August 2007 FFP 2741 24

Essay TestsEssay Tests

May require long or short answerTime consuming and difficult to gradeRecommended to grade in a group formatRubric is a useful grading toolHand written exams must allow sufficient timeShotgun approach.. As much information as possible in hopes of hitting the target

August 2007 FFP 2741 25

What is a Rubric? Heidi Goodrich, a rubrics expert, defines a rubric as "a scoring tool that lists the criteria for a piece of work or 'what counts.'" So a rubric for a multimedia project will list the things the student must have included to receive a certain score or rating. Rubrics help the student figure out how their project will be evaluated. Goodrich quotes a student who said he didn't much care for rubrics because "if you get something wrong, your teacher can prove you knew what you were supposed to do."

Generally rubrics specify the level of performance expected for several levels of quality. These levels of quality may be written as different ratings (e.g., Excellent, Good, Needs Improvement) or as numerical scores (e.g., 4, 3, 2, 1) which are then added up to form a total score which then is associated with a grade (e.g., A, B, C, etc).

Many rubrics also specify the level of assistance (e.g., Independently, With Minimal Adult Help; With Extensive Adult Help) for each quality rating.

Rubrics can help students and teachers define "quality". Rubrics can also help students judge and revise their own work before handing in their assignments.

http://rubistar.4teachers.org/index.php?screen=WhatIs&module=Rubistar

August 2007 FFP 2741 26

Oral ExamsOral Exams

Requires verbal answers by studentsAdvantagesEvaluate quick reaction of studentAssesses the student thought process

DisadvantagesLimited number of students examined at one timeDifficult to standardizeTime consuming and labor intensiveUnexpected distractionsUnfair emphasis on repeated mistakes

August 2007 FFP 2741 27

Project AssignmentsProject Assignments

Gets students working outside the class

In groups, helps develop people skills

NegativesHard to standardizePotential plagiarismMay measure only end product and not

consider the process

August 2007 FFP 2741 28

Practical ExamsPractical Exams

SituationalDemonstration of a skill in the context of a

scenarioFFII Exam for forcible entry

Rote“Demonstration of steps of performing a skill”FFII Exam donning of SCBA

August 2007 FFP 2741 29

Practical Skills EvaluationPractical Skills Evaluation

Rote mechanical skills

No real world stresses

Situational skills test asks the student to think through a situation

August 2007 FFP 2741 30

Simple Skill EvaluationSimple Skill Evaluation

Define required skill

Determine efficiency expected

Student needs to know what is expected

Skills check lists are helpful

August 2007 FFP 2741 31

Performance EvaluationsPerformance Evaluations

Determine and defineExpected outcome

Is performance or decision making more important than the situation?How stressful a situation is student prepared to handle?

August 2007 FFP 2741 32

Performance EvaluationsPerformance Evaluations

Determine standards to be evaluated

Situation shouldRepresent desired

outcomeRealistic scenariosRealistic environmentRealistic in the real

world

August 2007 FFP 2741 33

Performance EvaluationsPerformance Evaluations

Keep situation in perspectiveRemember safetyRemember legal

ramifications

List all activities to be completed

August 2007 FFP 2741 34

ReliabilityReliability

Multiple testsCompare resultsThe closer the scores, the more reliable

Four questionsDoes it measure consistently on different occasionsAny influence of the environmentAny difference with administratorsDoes it discriminate against anyone

August 2007 FFP 2741 35

Test ResourcesTest Resources

Written examination resources:NFPAPublishers’ test banksFire textbooksFire textbook instructor guidesTextbooks of practice certification examinationsOn-line and computer-based practice certification testsFire Internet sites

August 2007 FFP 2741 36

Test ResourcesTest Resources

Practical examination resources:NFPAFire Internet sitesFire textbooksFire continuing education programs

August 2007 FFP 2741 37

Test ResourcesTest Resources

Oral examination resources:NFPAFire Internet sites

August 2007 FFP 2741 38

ConclusionConclusion

As part of the learning process, the instructor needs a mechanism to evaluate the

student’s learning and identify whether or not the student is achieving the objectives

and goals of instruction.

August 2007 FFP 2741 39

ConclusionConclusion

EvaluateFour LevelsSummativeFormative

TestsKinds of testDifferences among the various typesSources for testing materials

August 2007 FFP 2741 40

Multiple choice tests should avoid what?“Which choice is not correct”“All of the choices are correct”“None of the choices are correct”Maybe I just broke the rule!!!!!!

August 2007 FFP 2741 41

Which type of test is most commonly used for state or national certifications?True/FalseMatchingMultiple choiceFill in the blank

August 2007 FFP 2741 42

Unexpected distractions would likely be a disadvantage in what type of testing?RoteSituationalWritten Oral

August 2007 FFP 2741 43

Of the four levels of evaluation, which one would likely look at job performance 6 months to a year down the road?Reaction Business resultsLearningTransfer