Effective Training: Strategies, Systems and Practices, 2nd Edition

Chapter Eight

Evaluation of Training

Chapter 8 2

Session Overview Define evaluation Describe why many organizations

don’t evaluate training Explain who is interested in

evaluation data Describe the various levels of

training evaluation

Chapter 8 3

Evaluation Defined Evaluation: judgment of training

programs worth or value Formative Evaluation (process)

examines how the training was designed,

developed, and carried out Summative Evaluation (outcome)

determines if training accomplished its objectives

Chapter 8 4

Resistance to Training Evaluation

A few reasons why training managers

don’t evaluate training:

Nothing to evaluate No one really cares Evaluation is a threat to my job

Chapter 8 5

Types of Evaluation Data

The following are types of evaluationData that should be collected:

Reaction Questionnaire Learning Job Behavior Organizational Results

Chapter 8 6

Who Is Interested Outcome Data

Reaction Learning Behavior Results

Training Department

Trainer Yes Yes Yes No

Other Trainers Perhaps Perhaps Perhaps No

Training Manager Yes Yes Yes Yes

Customers

Trainees Yes Yes Yes Perhaps

Trainees’ Supervisor Not really Only if no transfer

Yes Yes

Upper Management No No Perhaps Yes

Chapter 8 7

Kirkpatrick Level 1: Reaction Questionnaire

Reaction Questionnaires are:

“Used to determine what the trainees thought about the training.” (p. 353)

Affective Questionnaire: - measures general feelings about

training

Utility Questionnaire: - measures the trainee's belief about

the value of the training

Chapter 8 8

Reaction Questionnaire: Types of Questions

The following is obtained from a reaction

questionnaire: Facilities issues Trainer issues Perceived value of training Relevance of training

(example: pp. 355-356)

Chapter 8 9

Timing of Reaction Questionnaire

The following times can be used toadminister a reaction questionnaire?

During training (after each segment) Immediately following training One or two months after training

Chapter 8 10

Steps in Developing a Reaction Questionnaire (1 of 2)

1.  Determine what you want to find out (consider training objectives).

2.  Develop a written set of questions to obtain the information.

3.  Develop a scale to quantify respondents’ data.

4.  Make forms anonymous so participants will feel free to respond honestly.

5.  Ask for information useful in determining differences in reactions by subgroups (e.g., young vs. old; minority vs. non-minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment.

Chapter 8 11

Steps in Developing a Reaction Questionnaire (2 of 2)

6. Allow space for “Additional Comments” in order to allow participants the opportunity to mention things you might not have considered.

7. Decide the best time to give the questionnaire to get the information you want.

a. If right after training, have someone other than the instructor administer and pick up the information.

b. If some time later, develop a mechanism for obtaining a high response rate (e.g., encourage the supervisor to allow trainees to

complete the questionnaire on company time).

Chapter 8 12

Kirkpatrick Level 2: Learning

Assessing Learning:

Declarative knowledge – paper/pencil multiple choice test

Procedural knowledge – paired comparison method (p. 359; 8-4)

1. Determining relationships among a number of concepts

Strategic knowledge – Probed Protocol Analysis 1. Define a problem2. Explain step-by-step how to solve problem

Chapter 8 13

Kirkpatrick Level 3: Job Behavior

Job Behavior:

“Measuring if training actually transferred to the job”

Chapter 8 14

Kirkpatrick Level 3: Job Behavior

Data collection approaches: Behavior data

Interviews Direct observation Archival records of performance

Scripted situation - a process for assessing transfer of training to the job by providing rater with a number of responses to rate trainee (p. 363)

Attitudes – attitudinal surveys

Chapter 8 15

Kirkpatrick Level 4: Organizational Results

Organizational Results:

“This measurement would be used to determine any change after training was complete” (p. 364)

Did grievances go down? Did scrap rates diminish?

Chapter 8 16

Kirkpatrick Level 4: Organizational Results

Data collection: Consistent tracking of

organizational performance should take place at intervals before, throughout, & after training.

Scrap rate after training – scrap rate before training = results

Chapter 8 17

Evaluation: Kirkpatrick

Perceived match between trainee Exceptions and what training provided

Reactions

Trainee readiness for the course Trainee motivation to learn Design, materials, and content Trainer(s) behaviors

Learning

Transfer of training Motivational forces in the job setting Opportunity to applying training on the job KSAs

Job Behavior

External environment of the organization: Economy, regulations, suppliers, etc Internal environment of the organization: Policies, procedures, systems Employee performance, KSAs, and needs

Organizational

Results