Measuring Training

8/2/2019 Measuring Training

1/49

1VRAJEEV

MeasuringROI of

Training


2/49

2VRAJEEV

Agenda

1. Measuring the Effectiveness of TrainingProgram

2. Measuring Return on Investment of Training

3. Enhancing the Effectiveness and ROI of

Training


3/49

3VRAJEEV

A.Measuring the Effectivenessof Training Program


4/49

4VRAJEEV

The Four Levels of Evaluation

Level 1 - Reaction

Level 2 - Learning

Level 3 Behavior

ApplicationLevel 4 Performance Impact

Four Levelsof Training

Effectiveness


5/49

5VRAJEEV

Level 1 - Reaction

Evaluate trainees

reactions to the program:

Did they like the program?Did they like the

facilitators?

Did they like the training

accommodation and

facilities?

Level 1 -Reaction


6/49

6VRAJEEV

Guidelines for Evaluating Reaction

1. Determine what you want to find out

2. Design a form that will quantify reactions

3. Encourage written comments and suggestions

4. Get 100 percent immediate response

5. Get honest response

6. Develop acceptable standards

7. Measure reactions against standards, and takeappropriate action

8. Communicate reaction as appropriate


7/497VRAJEEV

Please give us your frank reactions and comments. They will help us to evaluate

this program and improve future programs.

Program :

Facilitator :

1. How do you rate the subject (interest, benefit, etc)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

2. How do you rate the facilitator? (knowledge, ability to deliver and communicate?)

a. Excellent

b. Very Goodc. Good

d. Fair

e. Poor

3. How do you rate the facilities? (comfort, convinience, etc)

a. Excellent

b. Very Good

c. Goodd. Fair

e. Poor

4. What would have improved the program?

Sample ofReaction

Form


8/498VRAJEEV

Level 2 - Learning

Measuring learning means

determining one or more of

the following :

What knowledge was

learned?

What skills were

developed or improved? What attitudes were

changed?

Level 2 -Learning


9/499VRAJEEV

Guidelines for Evaluating Learning

1. Use a control group if practical

2. Evaluate knowledge, skills and/or attitudes bothbefore and after the program

3. Use a paper-and-pencil test to measure knowledge

4. Use a performance test to measure skills

5. Get 100 percent immediate response

6. Use the results of evaluation to take appropriateaction


10/4910VRAJEEV


Experimental Group Control Group

Pre Test Score 45 46Post Test Score 55 48

Gain 10 2

Pretest and Posttest Scores on Change Management TrainingExample :


11/4911VRAJEEV

Level 3 Behavior Application

The frequency ofapplication of newskills/knowledge/ attitudes (on the job)

The effectiveness of theskills/knowledge/ attitudes (as applied onthe job)

Level 3 BehaviorApplication


12/4912VRAJEEV



2. Allow time for behavior change and application to takeplace

3. Evaluate both before and after the program if practical

4. Survey and/or interview one or more of the following :trainees, their immediate supervisor, their subordinates,and others who often observe their behavior

5. Get 100 percent response or a sampling

6. Repeat the evaluation at appropriate times

7. Consider cost versus benefits


13/4913VRAJEEV

Example of Survey to Measure Behavior Application

Understanding and Motivating

1. Trying to understand my subordinates 5 4 3 2 1

2. Listening to my subordinates 5 4 3 2 1

3. Praising good work 5 4 3 2 1

4. Talking with subordinates about

their family and personal interests

5. Asking my subordinates for their ideas 5 4 3 2 1

6. Applying "Management by Walking Around" 5 4 3 2 1

Time and energy spent after the prgram compared to time and energy spent before

the program

5 4 3 2 1

Instruction: The objective of this questionnaire is to determine the extent to which those whoattended the recent program on Leadership have applied the principles and techniquesthat they learned there to the job.

Circle the answer that you consider appropriate for each question.5 = Much more 4 = More 3 = Same 2 = Less 1 = Much less


14/4914VRAJEEV

Level 4 Performance Results

Indicate the extent to which youthink this program hasinfluenced each of these

measures in your work unit,department, or business unit: Productivity Quality Customer Response Time Cost Control Employee Satisfaction Customer Satisfaction Other

Level 4 PerformanceResults


15/4915VRAJEEV



2. Allow time for results to be achieved

3. Measure both before and after the program if practical4. Repeat the evaluation at appropriate times

5. Consider cost versus benefits

6. Be satisfied with evidence if proof is not possible


16/4916VRAJEEV

Performance Indicators

HARD DATA INDICATORS

Duration involved

Number of defect products Sales volume

Customer satisfaction index Response time to orders

Others


17/49

17VRAJEEV

Performance Indicators

SOFT DATA INDICATORS (intangible impacts)

Job satisfaction

Conducive working relationship Effective communication Stress rate Quality in decision-making


18/49

18VRAJEEV

Example : Measuring Training Results

0.00

50.00

100.00

150.00

200.00Program : TQM

Training

Results after

3 months oftraining, number

of defects

dropped to 80

units/day

120 units

80 units

Before training After training


19/49

19VRAJEEV

0.00

10.00

20.00

30.00

40.00

50.00

Example : Measuring Training Results

Program :

Sales Training

Results after 3

months training,

number of sales

per salesman

increase to 30

units/month.

20 units

30 units

Before training After training


20/49

20VRAJEEV

B.MeasuringReturn on Investment of

Training


21/49

21VRAJEEV

Benefits of ROI of Training

Measure contribution

Set priorities

Focus on results

Alter management perceptions of training


22/49

22VRAJEEV

ROI of Training Model

CollectData

Isolate theEffects ofTraining

Convert Datato Monetary

Values

Calculate ROIof Training

TabulateProgram

Costs

IdentifyIntangibleBenefits


23/49

23VRAJEEV

Net Program Benefits

Program CostsX 100

Return on Investment Formula

ROI =

Example :

Costs per program (25 participants) Rs 88,500

Benefits per program (1st year) Rs 230,625

Rs 230,625 88,500

Rs 88,500ROI = X 100

ROI = 161 %


24/49

24VRAJEEV

Collecting Data

Identify appropriateperformance indicators

Develop a collection plan

Collect

Data


25/49

25VRAJEEV

Example of Performance Indicators

Units produced Items sold Work backlog New accounts opened

Productivity Inventory turnover Etc.

Output

Equipment downtime Overtime Time to project

completion

Processing time Repair time Lost time days Etc

Time


26/49

26VRAJEEV

Example of Hard Indicators

Unit costs Variable costs Overhead costs Operating costs

Number of costreduction

Etc.

Cost

Scrap Waste Rejects Error rates

Rework Product defects Product failure

Quality


27/49

27VRAJEEV

Example of Performance Results

Some performance results after trainingprogram:

Rejections were reduced from 11 % to 7.4 %

Absenteeism was reduced from 7 % to 3.25 %

The annual turnover rate was reduced from 30% to 16 %

Customer complaints were reduced by 65 %


28/49

28VRAJEEV

Isolating the Effects of Training

Methods toIsolate theEffects ofTraining

UsingControl Group

TrendLines

Participants

EstimateSupervisors of

ParticipantsEstimate


29/49

29VRAJEEV

UsingControlGroup

A control group arrangement can beused to isolate training impact.

With this strategy, one group

receives training, while another,similar group does not receivetraining.

The difference in the performance ofthe two groups is attributed to thetraining program.



30/49

30VRAJEEV

TrendLines

Trend lines are used to project thevalues of specific output variables iftraining had not been undertaken.

The projection is compared to theactual data after training, and thedifference represents the estimateimpact of training.



31/49

31VRAJEEV

Trend Lines Analysis

Jan Feb Mar Apr JulJunMay Aug

At the beginning ofMay, a Sales training

Program session washeld

V o

l u m e o f

S a

l e s

The difference

represents the estimate

impact of training.

Trend Projection

Actual sales performance


32/49

32VRAJEEV

Participants andSupervisors of

ParticipantsEstimate of

Trainings Impact

This method rests on the assumption thatparticipants (and their supervisors) arecapable of estimating how much aperformance improvement is related to the

training program.

Because their actions have produced theimprovement, participants (and theirsupervisors) may have very accurate input

on the issue. They should know how much of the

change was caused by applying what theyhave learned in the program.



33/49

33VRAJEEV

Participants andSupervisors of

ParticipantsEstimate of

Trainings Impact

Typical Questions to Estimate :

What percent this improvementcan be attributed to the application

of skills/techniques/knowledgegained in the training program?

What confidence do you have inthis estimate, expresses as a

percent?

What other factors contributed tothis improvement in performance?



34/49

34VRAJEEV

1 Training Program 50% 70%

2 Change in Procedures 10% 80%

3 Adjustment in Standards 10% 50%

4 Revision to Incentive Plan 20% 90%

5 Increased Management Attention 10% 50%

6 Other - -

PercentImprovement

Caused by

ConfidenceExpressed as

a PercentFactors Which Influenced

ImprovementNo.


Example of a Participants Estimation

The confidence percentage is multiplied by the estimate (50 % x 70 %) to produce a usable training factor value of 35 %


35/49

35VRAJEEV

1 Training Program 50% 70%

2 Change in Procedures 10% 80%

3 Adjustment in Standards 10% 50%4 Revision to Incentive Plan 20% 90%

5 Increased Management Attention 10% 50%

6 Other - -

PercentImprovement

Caused by

ConfidenceExpressed as

a PercentFactors Which Influenced

ImprovementNo.


Example of a Participants Estimation

The confidence percentage is multiplied by the estimate (50 % x 70 %) toproduce a usable training factor value of 35 %

This adjusted percentage is then multiplied by the actual amount ofimprovement (post-program minus pre-program value) to isolate the portionattributed to training

The adjusted improvement is now ready for conversion to monetary values,

and used in the return on investment


36/49

36VRAJEEV

Converting Data to Monetary Values

ConvertingData to

Monetary

Values

Steps to Convert Data to MonetaryValues

1. Focus on a unit of improvement

2. Determine a value of each unit

3. Calculate the change in performancedata

4. Determine an annual amount of change

5. Calculate the annual value ofimprovement


37/49

37VRAJEEV

Steps to Convert Data to Monetary Values

An example to illustrate the steps to convert data to monetary values

1. Focus on unitimprovement

One grievance reaching step two in thefour-step grievance resolution process

Steps Illustration

2. Determine avalue of each

unit

Using internal experts, the cost of anaverage grievance was estimated to beRs,500 when considering time and directcosts


38/49

38VRAJEEV


3. Calculate thechange in

performancedata

Six months after the program was completed,total grievances per month reaching step two

declined by ten. Seven of the then grievance reductions were

related to the program as determined bysupervisors (isolating the effects of training)

Steps Illustration

4. Determine anannual amountfor the change

Using the six month value, seven permonth (grievance reductions), yields anannual improvement of 84 (7 x 12 months)


39/49

39VRAJEEV


5. Calculate theannual value of

improvement

Annual value = 84 x Rs500 = Rs42,000

Steps Illustration


40/49

40VRAJEEV

Tabulatingcost of the

program

Cost components that should beincluded are :

The cost to design and developthe program

The cost of all program materialsprovided to each participant

The cost for facilitator

The cost of the facilities of thetraining program

Tabulating Cost of the Program


41/49

41VRAJEEV

Tabulatingcost of the

program

Cost components that should beincluded are :

Travel, lodging, and meal costs

for the participants Salaries, plus employee benefits

of the participants who attendthe training

Administrative and overheadcosts of the training function,allocated in some convenientway

Tabulating Cost of the Program


42/49

42VRAJEEV

C.Enhancing TrainingEffectiveness


43/49

43VRAJEEV

How Effective is Your Training Program?

Broad and Newstrom (1992) report studies have

shown less than 30% of what is actually taughttransfers to the job in a way that enhances

performance .Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packed strategies to ensure high payoff from training investments. Reading, MA: Addison- Wesley.


44/49

44VRAJEEV

Source of Barriers to Training Transfer

Lack of reinforcement on the job

Interference from immediate (work) environment

Nonsupportive organizational culture Trainees perception of impractical training programs

Separation from the inspiration or support of the trainer


45/49

45VRAJEEV

The Transfer Partnership

Trainee Trainee recognizes need fornew skills

Trainee Trainer Trainer designs and/ordelivers learning experiences

Trainee Trainer Manager supports learningand application on the job

Manager


46/49

46VRAJEEV

Manager

After Training

Plan trainees reentry Provide opportunities to practice new skills

Have trainees participate in transfer-relateddecisions

Reduce job pressures initially

Give positive reinforcement

Schedule trainee briefings for co-workers

Set mutual expectations for improvement

Arrange proactive (refresher) sessions

Provide and support the use of job aids


47/49

47VRAJEEV

Trainer

After Training

Provide follow-up support Conduct evaluation surveys and provide

feedback

Develop and administer recognition system

Provide refresher/problem-solving sessions


48/49

48VRAJEEV

Trainee

After Training

Practice self-management Review training content and learned skills

Develop a mentoring relationship

Maintain contact with training buddies


49/49

THANK YOU

Measuring Training

Documents

Transcript of Measuring Training