Post on 14-Mar-2018
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
July 2016
The Blueprint for Radically Improved Training Evaluations
“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to
improve the value of training.” (p. 94‐95)
Tim Mooney and Rob BrinkerhoffCourageous Training:
Bold Actions for Business Results.
Demonstrate the Value
Improve theLearning
Three Reasons to Use Measurement in Learning
# 1
# 2
# 3
The Kirkpatrick 4-Level ModelLevel 1
LearnerReaction
Level 2
Tests of Learning
Level 3
On‐the‐Job Behavior
Level 4
Business Results
“Training Evaluations”
reaction sheetshappy sheets
level 1’s
Bridging Gap between Research and Practice
Research
Simplicity is the Ultimate Sophistication
Leonardo da Vinci
The Decisive Dozenfor Learning Design and Learning Measurement
http://is.gd/ddResearch
1. Content
2. ExposureBaseline
3. Guiding Attention
4. Creating Correct Conceptions
5. Repetition
6. Feedback
7. Variation
Engagement & Understanding
8. Retrieval Practice
9. Context Alignment
10. SpacingRemembering
11. Persuasion
12. PerseveranceApplication
http://Work-Learning.com/catalog.html
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Background
This year, improving your smile sheetswill likely be the MOST IMPORTANTthing you can do to improve learning!
Professional Development
Pro
1. Red‐flagging training programs that are not sufficiently effective.
2. Gathering ideas for ongoing updates and revision of a learning program.
3. Judging strengths and weaknesses of a pilot program to enable revision.
4. Providing instructors with feedback to aid their development.
5. Helping learners reflect on and reinforce what they learned.
6. Helping learners determine what (if anything) they plan to do with their learning.
7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.
8. Upholding the spirit of common courtesy by giving learners a chance for feedback.
9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.
Modified based on work by Robert Brinkerhoff
Reasons for Smile Sheets
# 1
We Work in aLearning-to-Performance
Ecosystem
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
Later on the job, learners remember
what they’ve learned.
The learners get a return on their efforts.
Performance can be prompted
through job aids, signage,
intuitive cues, performance
support, management, etc.
Learners can learn on-the-job through
trial & practice, insight learning, help from others,
social media, studying on their
own, etc.
© Copyright 2009-2016 Work-Learning Research, Inc.
YouTube: http://is.gd/LearningLandscape
# 2
We Should NOTExpect Perfection
20
Measurement doesn’t
illuminate everything.
Measurement
Measurement illuminates
some things.
There is no perfect measurement tool!
How would you rate your organization’s smile sheets?
Terrible
1
Bad
2
Okay
3
Good
4
Given, that there is NO perfect
measurement…
# 3
The Kirkpatrick ModelControls our Thinking!
The Kirkpatrick 4-Level ModelLevel 1
LearnerReaction
Level 2
Tests of Learning
Level 3
On‐the‐Job Behavior
Level 4
Business Results
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
© Copyright 2009-2016 Work-Learning Research, Inc.Regular
smile sheets
Immediate tests of
remembering & decision-
making
Does Job Performance
Improve?
Does Business
Performance Improve?
IR
L1-Delayed
L2-Delayed
L3-Supports
PD
RS JS PS
L1
L2Are Prompt Supports
Used? Effective?
Are Just-in-Time Learning
Supports Used?
Effective?
Are Retrieval Supports
Used? Effective?
Smile sheets
Retrieval &
Decision Making
Evaluation of Coaching & Learner Learning
Delayed smile
sheets
Delayed tests of remembering & decision-making
Do the learners actually benefit
from the learning in their work or
career?
Does the work environment
support performance?
Are Prompting Devices Used?
Effective?
On smile sheets, ask about
motivation to apply
L1
L2
L3
L4
Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation
• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.
• Training centric. Ignores prompting mechanisms & on-the-job learning.
• Ignores the role that management and the business side must play.
• Implies that higher levels are more important than lower levels.
• Ignores the causal chain from learning to remembering to performance to results.
• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.
• Pushes us to value learner ratings as predictive of learning and on-the-job performance.
“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…
[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.
[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)
http://is.gd/TrainingResearch2012
# 4
Most Smile Sheets Stink!
Level 1to
Level 3
r=.16
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Very Weak Relationship between Levels
Correlation between levels?
Level 1to
Level 2
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied
Psychology, 93, 280-295.
Correlation between levels?
No
Practical Significance
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
Level 1to
Level 2
r=.09
# 5
Likert-Like Scales &Numeric Responses
create Poor Data
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
4.1
Compared to:
• Previous• Standard• Others
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
We’d like to trust our learners…
Research shows that learnersdon’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Transmogrify
Strongly AgreeAgree
Neither Agree Nor DisagreeDisagree
Strongly Disagree
We Start with Fuzzy
Adjectives
54321
Magically We Turn Adjectives
Into Numbers
We Average
Responses, Losing
More Info
3.9
4.2
3.7
4.0
4.1
4.0
4.2
3.4
We Choose One
Question and Report Results
4.1
Garbage In -- Garbage Out!
When Our Smile Sheet Questions areBiased, Irrelevant, Unanswerable
Garbage In -- Garbage Out!
# 6
The Critical Importanceof Remembering!
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
The Learning Landscape
On-the-Job Performance
Remembering
Learning
© Copyright 2009-2016 Work-Learning Research, Inc., www.work-learning.com
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
Learners
Learners
If our learners start here.
But end up here.
Have we maximized the
learning benefits?
Less Remembering
More Remembering
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
There are many possible after-training results,
depending on:
(1) Design of the Learning(2) After-Learning Follow-up
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
YouTube: http://is.gd/LearningForgettingCurves
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
What does an end-of-course
assessment tell us?
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
If we measure
here?BiasedMetric!!
Understanding
Remembering
Application
DelayedTests
# 7
We must go beyondSmile Sheets!
Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts
Smile Sheets
Understanding
Remembering
Motivation to Apply
After Supports
Meeting Target Goals?
Job Performance
Organizational Results
Learner Expectations
Other Expectations
Supports:
Management Support?
Workplace Obstacles?
Reinforcement?
Reminders?
Learners able to:
Understand?
Remember?
Make Decisions?
Apply the Learning?
# 8
Smile Sheetscan be Improved!
My Journey in tryingTo create a better
Smile Sheet
My “New” Smile Sheet
Specific Concepts
How much Value?
ConceptNew?
HowWell
Taught?
Simple Overall Ratings
Helping Learners Calibrate
Best Feedback Comes from Comments
# 9
Performance-FocusedSmile Sheets
http://is.gd/TrainingResearch2012
http://is.gd/ddResearch
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding
numeric averages that discourage a standards-based decision on success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at
least try to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
Three Key Goals:
1. Are the Questions Well-Designed?
2. Are Learners Making Good Smile Sheet Decisions?
3. Are The Data we’re Getting Clear and Actionable?
Ensuring that we are focused on the science-
of-learning factors that matter!
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNING EFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding
numeric averages that discourage a standards-based decision on success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at
least try to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
What Most Smile Sheets
Provide
QUATERNARYGOALS
Traditional Smile Sheets
Learners rate instructors as credible and engaging?
Learners say that classroom environment was conducive to
learning?
Learners satisfied with experience?
Learners think course was well-organized?
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?
A. I’m NOT AT ALL ABLE to put the concepts into practice.
B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.
C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.
D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.
E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.
A Better Smile Sheet Question
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
A. I am NOT AT ALL ready to use the skills taught.
B. I need MORE GUIDANCE to be GOOD at using these skills.
C. I need MORE EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.
A Better Smile Sheet Question -- Shorter
© Copyright 2009 Work-Learning Research, Inc.
Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?
A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.
B. I have a BASIC FAMILIARITY with the concepts.
C. I have a SOLID UNDERSTANDING of the concepts.
D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.
E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.
Standards
Unacceptable
Unlikely
?
Acceptable
?
AcceptableUnacceptable
AcceptableSuperior
Traditional Smile Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
Performance‐Focused Smile
Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
OPEN TO
BIAS
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?
A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.
B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.
C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job.
D. I will make this a HIGH PRIORITY when I get back to my day-to-day job.
E. I will make this one of my HIGHEST PRIORITIES when I get back to my day-to-day job.
A Question About Motivation…
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.
A. I will have ENOUGH TIME to work on applying the learning.
B. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.
C. I will have someone available TO COACH OR MENTOR ME in applying the learning.
D. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.
E. I will have JOB AIDS to guide me in applying the learning to real job tasks.
F. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.
G. I will NOT get much direct support, but will rely on my own initiative.
A Question About Follow-Through…
# 10
Delayed Smile Sheets
Delayed Smile Sheet
Have you used what you learned in the workshop to make a significant improvement in your work?
• No, and I doubt that I will use what I learned.
• No, but I probably will use what I learned.
• No, but I have a plan to use what I learned.
• Yes, I have already used what I learned.
Are You Using What You Learned?
Yes!!
What Enabled You to Apply
What You Learned?
No
What is Holding You Back?
Feedback to:• Management• Learners• Us
Delayed Smile Sheet – Feedback for Improvement
What is holding you back from teaching others what you have learned?
• The learning content is not relevant to my colleagues’ work.
• I don’t think the learning content is valid.
• I don’t see how teaching others will benefit me.
• I can’t remember the learning content well enough.
• I’m not very good at teaching others.
• I haven’t had the time.
• I have had higher priorities.
• I have not had the resources.
• The risk of teaching this to others is too high.
• I’m still working to persuade management of the value.
• Other, please specify ____________________________
Delayed Smile Sheet
―――Feedback
toManagement,
Learners,and/or Us.
What are the top 3 to 4 reasons that have enabled you to put the workshop information into practice?
• I remember (or can look up) the workshop’s learning points.
• I believe strongly in the workshop’s learning points.
• I have the authority to make the necessary changes.
• I have decided to lead a change effort to make this happen.
• I have the time to put this into practice.
• I have the resources to put this into practice.
• I will be acknowledged or rewarded if I do this.
• I will be sanctioned or punished if I do NOT do this.
• My management has made this a high priority.
• My management is fully supportive of my efforts.
• My coworkers are fully supportive of my efforts.
• My Team (or our whole unit) is working together on this.
50%
50%75%
50%
Message toManagement
“More management-
driven than self-driven.”
This learning event will make a significant contribution to my work, in terms of the following: SELECT AS MANY AS YOU LIKE!
A. Improving my personal productivity.
B. Increasing my ability to innovate.
C. Enabling me to collaborate more effectively.
D. Improving my management performance.
E. Enabling me to generate more revenue.
F. Enabling me to lower costs.
# 11
Stealth Messages
Workplace Learning‐and‐Performance
Professionals
1. Training is not enough.
2. Training for awareness is a weak and insufficient approach.
3. Training should specifically minimize forgetting and support remembering.
4. It is better to train a few things really well, than many things perfunctorily.
5. Training should motivate subsequent on‐the‐job application.
6. Training content must be correct, validated, and relevant.
7. Traditional smile sheets are inadequate.
8. We must measure to provide ourselves with feedback to spur continuous improvement.
9. Prompting mechanisms (like job aids and performance support) should be used—as appropriate—as an integral part of training and as a replacement for training.
10. On‐the‐job learning should be leveraged in addition to formal training.
11. Learners’ managers and others within the learners’ chain of command should support after‐training application.
12. Supervisors play a critical role in enabling the success of training and in helping people be creative on the job.
Some Stealth MessagesWe Might Want to Send
conscious conscious
filters
ProactiveReactive
Etcetera
There are
Leverage Pointswithin our SOP’s
from where we can send
Stealth Messages
Recruiting of Work‐Learning Professionals
Formal Course Evaluations
TrainingRequests
(our response)
Instructor Evaluations
On‐the‐Job Performance
Data
Workplace Performance Assistance
Learning Needs Analysis
Annual Reports to Management
Course Reviews
Does the course use
methods that support
remembering?
Does the instructor use methods that support after‐
training follow‐through
?
Do we determine whether this is a training issue or a workplace
issue?
Do we ask trainers to show their consulting skills, not just their
platform skills?
Do we utilize Performance‐Focused smile
sheets & delayed smile
sheets?
What messages does a
traditional smile sheet
send?
What messages does a delayed performance‐focused smile sheet send?
What messages does a
performance‐focused smile sheet send?
Summary
1. Most smile sheets provide misleading feedback.
2. Smile sheets can be significantly improved.
3. Smile Sheets should NOT be our sole source of feedback.
4. Learners don’t always know their own learning.
5. Likert‐like scales and numeric scales create a garbage‐in garbage‐out problem.
6. We should set smile‐sheet standards before we get smile‐sheet results.
7. Delayed smile sheets offer potent information.
8. We are getting very poor feedback on the success of our learning interventions.
9. We should utilize the science of learning in our learning metrics.
10. To educate our stakeholders, we should consider using stealth messaging.
11. Performance‐Focused Smile Sheets align with science‐of‐learning findings and enable better decision‐making and reporting that enables urgency and clarity.
12. The Kirkpatrick Model is incomplete and sends some wrong messages about learning measurement.
Today’s Main Points:
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
www.SmileSheets.comuse Code: NGO16
• New Smile‐Sheet Questions – https://is.gd/NewSmiles1
• Avoid the Net Promoter Score — http://is.gd/NetPromoter
• Avoid Benchmarking — http://is.gd/DoNotBenchmark
• eLearning Manifesto 22 Principles — http://is.gd/manifesto22
• ATD Article on Smile Sheets — http://is.gd/atdsmile
• Subscription Learning — http://SubscriptionLearning.com
• Neuroscience & Learning: Not Yet! — http://is.gd/brainlearning
Subscribe to my Newsletter: — http://Work‐Learning.com/sign‐up.html
Slides available at: www.is.gd/willstuff
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
July 2016
The Blueprint for Radically Improved Training Evaluations
www.SmileSheets.comuse Code: NGO16
Slides available at: www.is.gd/willstuff
Thank you!!