Training on Trial ASTD Mid-NJ May 21 09
-
Upload
creators-college -
Category
Documents
-
view
118 -
download
0
Transcript of Training on Trial ASTD Mid-NJ May 21 09
![Page 1: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/1.jpg)
Training on Trial
Mid New Jersey Chapter, ASTD
May 21, 2009
Presented by:
Jim Kirkpatrick , PhD
![Page 2: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/2.jpg)
![Page 3: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/3.jpg)
The future of training
“Training directors might be well advised to take the initiative and evaluate their programs before the day of reckoning arrives”
- From Techniques for Evaluating Training Programs, by Don Kirkpatrick - ASTD Journal, November, - 1959
![Page 4: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/4.jpg)
What are the charges?
“You, learning professionals, are charged with incurring training and development costs that exceed the value you bring to the business”
![Page 5: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/5.jpg)
Who exactly is the “we” who are being charged?
The training industry
Organizational Development
Learning functions
HRD
Training departments
Us
![Page 6: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/6.jpg)
Who is charging us?
Internal business partners
External clients
The business world
![Page 7: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/7.jpg)
How do we know we are being charged?
“We would like you to show us your ROI”
“We know you run a lot of programs and seem quite busy. Why then are so many good people leaving us?”
“We are thinking we need to put our efforts in revenue-producing activities.”
“We are going to put that on hold for awhile.”
“We have decided to make some cuts.”
![Page 8: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/8.jpg)
What evidence is being used brought against us?
![Page 9: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/9.jpg)
Brinkerhoff Study
- Josh Bersin and Associates, 2008
Training Application
Did not try new skills: 15%
Tried new skills and failed: 70%
Achieved sustained new behaviors: 15%
![Page 10: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/10.jpg)
Percentage of learning transfer
1975: Percentage of formal learning that is actually applied to the job:
15%
2005: Percentage of formal learning that is actually applied to the job
15%
From Dana Robinson, ASTD ICE, 2008
![Page 11: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/11.jpg)
Causes of “training failure”
Preparation and Readiness: 20%Learning Intervention: 10%
Application Environment: 70%
2006 ASTD Study
![Page 12: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/12.jpg)
Typical Learning Investment
Dr. Brent Peterson, Columbia University, 2004
P re-Work 10%
L earning E vent 85%
F ollow-Up 5%
![Page 13: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/13.jpg)
Are we guilty?
![Page 14: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/14.jpg)
What do we have to do to be found not guilty?
We need to extend our role beyond the traditional definitions of training and learning
![Page 15: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/15.jpg)
The increase in “informal learning”
Where Learning Takes Place
Prior to being "trained": 20%During "training": 10%On the job: 70%
- Josh Bersin and Associates, 2008
![Page 16: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/16.jpg)
Lets get very practical here
We need to redefine and expand our role through the entire “learning and performance” management spectrum:– Our expertise– Our involvement– Our influence– Our impact– Our value
![Page 17: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/17.jpg)
How does this relate to you
so far?
![Page 18: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/18.jpg)
Who will take our case?
![Page 19: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/19.jpg)
Measurement and Evaluation
Gathering data and information to answer specific questions regarding the value of learning and performance solutions; focusing on the impact of individual programs and creating overall measures of system effectiveness, leveraging findings to provide recommendations for change and to increase organizational effectiveness.
- taken from Mapping the Future, ASTD, 2004
![Page 20: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/20.jpg)
Why Evaluate?
1. Improve the program (or know when to retire it)
2. Maximize training effectiveness by reinforcing knowledge, skills, and attitudes
3. Demonstrate the value of training
p. 3
![Page 21: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/21.jpg)
Level 1: REACTION To what degree participants react
favorably to the training
Kirkpatrick Four Levels
p. 4
![Page 22: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/22.jpg)
Level 2: LEARNINGTo what degree participants acquire
intended knowledge, skills, and attitudes based on participation in
learning event
Kirkpatrick Four Levels
p. 4
![Page 23: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/23.jpg)
Level 3: BEHAVIOR To what degree participants apply what
they learned during training on the job
Kirkpatrick Four Levels
p. 4
![Page 24: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/24.jpg)
Level 4: RESULTS To what degree targeted outcomes occur,
as a result of the learning event(s) and subsequent reinforcement
Kirkpatrick Four Levels
p. 4
![Page 25: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/25.jpg)
Four Level Correlations
Enjoyment (L1) and learning (L2): no correlationsRelevance (L1) and immediate learning (L2): r = .26Relevance (L1) and transfer to job (L3): r = .18
Immediate knowledge (L2) and retention (L2): r = .34Immediate knowledge (L2) and skill demo (L2): r = .18Immediate knowledge (L2) and transfer to job (L3): r = .11Retained knowledge (L2) and transfer to job (L3): r = .08Skill demo (L2) and transfer to job (L3): r = .18
![Page 26: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/26.jpg)
Kirkpatrick Four Level Evaluation major principles
• Smile sheets, pre and post-tests, and hope for the best
• Strong connection between Levels 1 and 2
• Strong connection between Levels 3 and 4
• “The Great Divide”
• “The Missing Link”
![Page 27: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/27.jpg)
Let’s take some initial depositions . . .
![Page 28: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/28.jpg)
“What is your job here at the hotel?”
“I am a window washer.”
![Page 29: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/29.jpg)
“What is your job here at the resort?”
“I am part of a team that creates great experiences for our guests!”
![Page 30: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/30.jpg)
Group Activity
What is significant about these two window washers in relation to:
Group 1: individual employees?
Group 2: training and development professionals?
Group 3: the training and development industry?
![Page 31: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/31.jpg)
![Page 32: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/32.jpg)
Crossing over is not easy
![Page 33: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/33.jpg)
What does the word “charged” mean?
• Accused
• Challenged
![Page 34: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/34.jpg)
Common Myth
“Evaluation” only occurs at the end of the instruction design process, or at the end
of a program.
![Page 35: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/35.jpg)
Reality
An effective evaluation plan is considered at every step in the program and training development process, not JUST at the
end.
![Page 36: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/36.jpg)
Kirkpatrick Four Level Evaluation major principles
• The end is the beginning
• ROE is the ultimate indicator of value
• Business partnership is key
• Value must be created first
• Demonstrate value through a compelling chain of evidence
![Page 37: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/37.jpg)
Kirkpatrick Four Level Evaluation Model
Reaction
Behavior
Results
Learning
![Page 38: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/38.jpg)
ReactionLearningBehaviorResults
ROE
Business need
identified
Measure L1 ReactionMeasure L2 Learning
Measure L3 Behavior
Present L1 Reaction findings
Present L2 Learning findings
Present L3 Behavior findings
Present L4 Results findings
Gather final four level data / information, prepare for presentation
The Kirkpatrick Model
Identify critical behaviors and key
organizational drivers
Determine required KSAs,
Learning Objectives
Negotiate stakeholder success indicators,Business outcomes
Ana
lyze
find
ings
, adj
ust,
repe
at s
teps
as
nece
ssar
yConsider necessary learning environment
and conditions
Design and build learning program and evaluation tools
Initiate ongoing reinforcement and
monitoring
Measure L4 Results
Deliver learning program
p. 22
![Page 39: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/39.jpg)
DOT Case Example
Challenge – to increase highway work zone efficiency, effectiveness, and safety.
![Page 40: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/40.jpg)
The Kirkpatrick Business Partnership Model
1. Identify who is on your corporate jury2. Negotiate jury (stakeholder) expectations3. Convert expectations to Level 4 outcomes and
determine metrics4a. Identify the key new participant behaviors that will be
required to bring about the desired outcomes. If L3 behaviors are not relevant, choose L2 ongoing readiness factors
4b. Identify the few key organizational drivers that will be necessary to bring about the changes in behavior
5. (Determine and implement prerequisites)
![Page 41: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/41.jpg)
DOT Example
1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors.
2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning.
3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; increased compliment to complaint ratio.
4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures.
4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.
![Page 42: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/42.jpg)
Airborne Warning and Control System
![Page 43: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/43.jpg)
Activity #21. Break up into small groups as instructed.
2. Discuss and select one high impact training program from your group members to work with.
3. Develop your answers to steps 1-4. Use flip chart pages.
4. Be prepared to share your results.
![Page 44: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/44.jpg)
![Page 45: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/45.jpg)
Demonstrated SKA
Training-related Activities
Business Results
Did the critical behaviors contribute to a positive impact on business results?
Did the activities build sufficient capabilities to drive the critical behaviors?
Did the target audience react favorably to the activity?
Critical Behaviors
Did the target audience exhibit a sustained behavior change in the work environment?
Level 1 Level 2 Level 3 Level 4
Measurement Strategy
![Page 46: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/46.jpg)
ActivitiesBusiness Results
CriticalBehaviors
Business Processes
Strategy
Culture Structure
Talent
HR Processes
Business Partnership
![Page 47: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/47.jpg)
Preparing for and presenting our case
![Page 48: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/48.jpg)
Developing a Plan to Evaluate: Critical Criteria
![Page 49: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/49.jpg)
Evaluation methodsEvaluation Levels
Methods
1 Reaction
2 Learning
3 Behavior
4Results
Survey ● ● ● ●
Questionnaire/Interview ● ● ● ●
Focus Group ● ● ● ●
Knowledge Test/Check ●
Work Review ● ●
Skills Observation ● ●
Presentations / Teach Bk ● ●
Action Planning ● ● ●
Action Learning ●
Key Business HR Metrics ●
![Page 50: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/50.jpg)
1
Post Session L1
Level 1
Level 2
Level 3
Level 4
Was session delivered effectively?
Post Session Reinfcmt, Coaching & L3
Expected results will occur.Build your chain of evidence & showcase
Post Session L2
Did targeted learning occur?
Was there successful transfer of learning to behavior?
Improve Reinforcement
and/or Coaching
NO
YES
NO
NONOYES
YES
YES
Improve Session 2
3
4
5
6
7
8
9
10
11
Delivery of Learning EventL1 & L2 Formative Evaluation
Was failure due to session?
Kirkpatrick Four Level EvaluationFeedback Loop to Eliminate ‘Snags’
![Page 51: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/51.jpg)
Example: Valdez University strategic goal scorecard – goal #1
Up from last month
Same as last month
Down from last month
Help to move our
organization from
transaction-oriented to
‘trusted advisor’
Metric Actual Target Status
1. Level 1 satisfaction scores - aggregate 93% 90%
2. % courses learning objectives matched to new directive
85% 70%
3. Level 2 skills scores - aggregate 92% 90%
4. % leaders certified as trusted advisor coaches 66% 65%
5. Level 3 scores - aggregate 48% 75%
6. Gallup scores - aggregate 77% 90%
![Page 52: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/52.jpg)
Preparing for our closing arguments
![Page 53: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/53.jpg)
Chain of Evidence
Gather data at all four levels and show that your training delivers true value to your organization.
Level 1Reaction
Level 2Level 2LearningLearning
Level 4Results
Level 3Behavior
![Page 54: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/54.jpg)
Data is a critical part of our Chain of Evidence
![Page 55: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/55.jpg)
But do not underestimate the power of testimonials
![Page 56: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/56.jpg)
What will you do as a result of what you have learned today . . .
![Page 57: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/57.jpg)
that will lead you to the following tomorrow?
![Page 58: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/58.jpg)
that will hopefully lead to . . .
![Page 59: Training on Trial ASTD Mid-NJ May 21 09](https://reader033.fdocuments.us/reader033/viewer/2022061119/546a9c31b4af9f5d6f8b4724/html5/thumbnails/59.jpg)
References• Kirkpatrick, D.L. and J.D. Evaluating Training
Programs, 3rd Ed., Berrett-Koehler Publ., Inc. San Francisco, CA, 2006
• Kirkpatrick, D.L. and J.D. Transferring Learning to Behavior: Berrett-Koehler Publ., San Francisco, CA, 2005
• Kirkpatrick, D.L. and J.D. Implementing the Four Levels, Berrett-Koehler Publ., San Francisco, CA, 2007
• Kirkpatrick, J.K., and W.K. Training on Trial, AMACOM, New York, 2009
• Contact Jim at [email protected]
• www.smr-usa.com