Redefining Measurement for Continuous...
Transcript of Redefining Measurement for Continuous...
Redefining Measurement for Continuous Learning
Todd Tauber
Vice President, Learning Research
Bersin by Deloitte, Deloitte Consulting LLP
April 1, 2014
How to demonstrate impact when
learning goes beyond L&D
Copyright © 2014 Deloitte Development LLC. All rights reserved. 1 IMPACT 2014 THE BUSINESS OF TALENT®
The importance of measuring impact
Persistent challenges to measuring impact
New complexities in measuring impact
Redefining measurement and evaluation
Contents
Copyright © 2014 Deloitte Development LLC. All rights reserved. 4 IMPACT 2014 THE BUSINESS OF TALENT®
Measuring impact is about demonstrating – and communicating – the value of L&D to the organization
If you don’t tell people about your success,
they probably won’t know about it.
Donald Trump,
The Art of the Deal
Copyright © 2014 Deloitte Development LLC. All rights reserved. 5 IMPACT 2014 THE BUSINESS OF TALENT®
Learning is increasingly important
Source: Ocean Tomo LLC, Intangible Asset Value, 4/2011
Companies are increasingly relying on human capital
17%
32%
68%
80% 80%
83% 68% 32% 20% 20%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1970s 1980s 1990s 2000s 2010s
Sh
are
of
S&
P 5
00 V
alu
e Tangible assets
Intangible assets
Copyright © 2014 Deloitte Development LLC. All rights reserved. 6 IMPACT 2014 THE BUSINESS OF TALENT®
Organizations invest a lot in L&D
Organizations expect a lot from L&D
0%
1%
2%
3%
4%
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
% o
f p
ayro
ll
Source: ASTD, 2013 State of the Industry, 10/2013;
Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014
$1,169 per learner
Copyright © 2014 Deloitte Development LLC. All rights reserved. 7 IMPACT 2014 THE BUSINESS OF TALENT®
Three ingredients for “high-impact” learning
…and employee satisfaction isn’t one of them
Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012
Business and
organizational impact
Alignment
Efficiency
Effectiveness
Copyright © 2014 Deloitte Development LLC. All rights reserved. 8 IMPACT 2014 THE BUSINESS OF TALENT®
Learning isn’t working for many organizations
Source: Deloitte, Global Human Capital Trends, 3/2014;
Chief Learning Officer, Scrap Learning and Manager Engagement, 3/2011
of global HR executives
believe their organization
is “ready” with the right
people with the right skills
in the right places.
15%
of the knowledge and
skills people learn in
training are actually
applied on-the-job with
positive results
10%
of the money spent on
employee learning and
development is wasted –
irrelevant, redundant, not
absorbed, not applied.
50%
It’s not aligned It’s not effective It’s not efficient
Copyright © 2014 Deloitte Development LLC. All rights reserved. 9 IMPACT 2014 THE BUSINESS OF TALENT®
L&D is often one of the first things to get cut when times are tough
Learning is essential, but L&D is vulnerable
Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014
7.0 6.2
5.3 5.2
4.2
5.9
0
1
2
3
4
5
6
7
8
9
10
2008 2009 2010 2011 2012 2013
L&
D s
taff
/ 1
,000 l
earn
ers
Copyright © 2014 Deloitte Development LLC. All rights reserved. 10 IMPACT 2014 THE BUSINESS OF TALENT®
Would anyone miss you?
Source: Henley Business School, Big Data and HR, 7/2013
I told them not to send [the reports] to anyone…Guess what? At the
end of the month, hardly anyone had asked for their reports, so they
clearly weren’t getting any insights from them. [But] if we had asked
them if they wanted their reports, I’m sure they would have said yes.
The moral of the story is that overuse or misuse of data analytics…is
in danger of …diverting attention away from the real business issues.
former Chief Learning Officer,
manufacturing company
Copyright © 2014 Deloitte Development LLC. All rights reserved. 12 IMPACT 2014 THE BUSINESS OF TALENT®
Three stubborn hurdles
Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012
Where to start?
Who / what it’s for?
What methodologies to use?
People and relationships
Systems for capturing data
Tools for analyzing data
Skills
Funding
Leadership / management support
Culture
Frameworks
Infrastructure
What are the biggest challenges you have in measurement?
Copyright © 2014 Deloitte Development LLC. All rights reserved. 13 IMPACT 2014 THE BUSINESS OF TALENT®
How CEOs measure learning success
The customer is always right
Source: ASTD / ROI Institute, Measuring What Matters: How CEOs View Learning Success, 12/2010
96%
74%
61%
82%
28% 22%
8% 4%
11%
78%
32%
53%
Impact ROI Application Cost efficiency Learning Reactions
We should measure this
We do measure this
Copyright © 2014 Deloitte Development LLC. All rights reserved. 14 IMPACT 2014 THE BUSINESS OF TALENT®
Directional insights, not precise calculations
Source: CFO, Measured Response, 6/2011
Chief Learning Officer,
technology company
Where people get messed up is in thinking that they have to measure
individual training classes to make sure that participants absorbed
the information and are applying it to their jobs.
I'm not interested in that, and neither is our CFO. He wants to
measure whether managers are performing well and whether the
organization is productive.
Copyright © 2014 Deloitte Development LLC. All rights reserved. 15 IMPACT 2014 THE BUSINESS OF TALENT®
More mature L&D groups dig deeper into talent and business outcomes
Business metrics, not busy metrics
Source: Bersin by Deloitte, Practical and Proven Approaches to Learning Measurement, 4/2012
0%
50%
100%
0% 50% 100%
Frequency
Va
lue
Learner satisfaction (Level 1)
Enrollments
Utilization
Hours delivered
Job impact (Level 3)
Business impact (Level 4)
Impact on business metrics
ROI Manager
satisfaction
Learning
Scores
(Level 2)
Costs
Business leaders CLOs Instructional designers
Copyright © 2014 Deloitte Development LLC. All rights reserved. 17 IMPACT 2014 THE BUSINESS OF TALENT®
Learning isn’t just programs and courses anymore
Continuous learning happens with and without L&D
Time
Kn
ow
led
ge
/ s
kill
s
Tra
inin
g e
vent
Traditional training
Continuous learning
Reinforcement via… • Coaching & mentoring
• Peer feedback
• Job aids / tools
• Online / mobile courses
• On-demand content
• Conferences
• Professional networks
• Social network
• Communities of practice
• …etc.
Copyright © 2014 Deloitte Development LLC. All rights reserved. 18 IMPACT 2014 THE BUSINESS OF TALENT®
We learn
through
interaction,
everywhere,
all the time
Continuous learning activities can't be measured as simply as events, courses and programs
Mentoring
Books
Wikis
Conferences
e-learning
Video
Coaching Webinars
Websites
Podcasts
Job rotations
Manuals
Peers
Discussion boards
Lunch-and-learns
Chalk talks
Social networking
Whitepapers
Case studies
Following
Tagging
Micro-blogs
Virtual worlds
Stretch goals
Post-mortems
Performance support
Articles
Meetings
Onboarding
Communities-of-practice
Demonstrations
Debates
Role models
Job shadowing
Mistakes
Successes
Visual aids
Instructions
Consulting
Advice
Collaboration
Search
Lectures
After action reviews
Interviews
Feedback
Customer data
Observation Journaling
Role Playing Simulations
Goals Experiments
Labs
Play
Conversations Teaching
Presentations
Dialogue
Copyright © 2014 Deloitte Development LLC. All rights reserved. 19 IMPACT 2014 THE BUSINESS OF TALENT®
What % of learning hours are delivered via…
More digital learning means lots more stuff to track
2005 2006 2007 2008 2009 2010 2011 2012 2013
Source: Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014
Online self-study
Other things
Live instructor-led
Live virtual
Copyright © 2014 Deloitte Development LLC. All rights reserved. 20 IMPACT 2014 THE BUSINESS OF TALENT®
There’s a lot more to collect and connect
Data is fragmenting way beyond the LMS
Learning systems HR / HCM systems
• Multiple LMSs
• vILT systems
• Collaboration platforms
• Learning portals
• ATS / assessment
• Core HRIS
• Workforce analytics
• Talent management systems
Business systems Other systems
• ERP
• Finance
• Business intelligence
• CRM / marketing
• Social media
• Web analytics
• Mobile apps
• MOOCs
Copyright © 2014 Deloitte Development LLC. All rights reserved. 21 IMPACT 2014 THE BUSINESS OF TALENT®
2009
2014
Are your business leaders asking for quantitative measures of HR / L&D?
Growing demand for better insights
Source: KnowledgeAdvisors, The Business Drivers of Talent Analytics, 3/2014
In 2014
52% are saying yes
(and 29% expect
them to soon)
In 2009,
21% said yes
Copyright © 2014 Deloitte Development LLC. All rights reserved. 22 IMPACT 2014 THE BUSINESS OF TALENT®
Which functions have strong analytics capabilities?
Expectations outpacing capabilities
15%
56%
58%
77%
81%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
HR
Marketing
Sales
Operations
Finance
Bersin by Deloitte, High-Impact Talent Analytics, 10/2013
Copyright © 2014 Deloitte Development LLC. All rights reserved. 23 IMPACT 2014 THE BUSINESS OF TALENT®
Same old lack of know-how, systems and support
The more things change, the more they stay the same
For informal learning
For on-demand learning
For continuous, self-directed learning
New data
New data sources
New systems and tools
Changing demands
Increasing expectations
Different skills and capabilities
Culture
Frameworks
Infrastructure
Copyright © 2014 Deloitte Development LLC. All rights reserved. 25 IMPACT 2014 THE BUSINESS OF TALENT®
The basics still apply
Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012;
Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012
60% of L&D leaders we surveyed said their
companies don’t have a real plan for
measuring learning impact.
1.2x companies that regularly engage with
stakeholders are more likely to have
high-impact measurement capabilities.
2% 64% of L&D organizations spend 2% or
less of their budgets on measurement.
72% think they should spend more.
Dedicate resources
and manpower
Have a plan
Build relationships
with stakeholders
Copyright © 2014 Deloitte Development LLC. All rights reserved. 26 IMPACT 2014 THE BUSINESS OF TALENT®
Rethink yesterday’s tools and practices
Kirkpatrick and Phillips methods are only the beginning
Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012
Too simplistic…
Does liking a course or
passing a test really
translate directly into
better performance or
business results?
Too hard…
There are no clear,
consistent definitions
and it’s rarely simple or
easy to isolate
learning’s contribution
(…or L&D’s).
And too backward
Finding out after
training that people
were not satisfied – or
that results didn’t
improve – doesn’t tell
you what to change.
Copyright © 2014 Deloitte Development LLC. All rights reserved. 27 IMPACT 2014 THE BUSINESS OF TALENT®
Build (some) new capabilities
Utility / usefulness
Stakeholder satisfaction
Experience API (xAPI) Employee engagement
Learning records stores Readiness / promotions
Information technology Adaptive learning Employee retention
Finance and accounting Web analytics Time to proficiency
Marketing Social listening Activity / contributions
Business intelligence CRM systems Influence and feedback
New friends New tools New metrics
Copyright © 2014 Deloitte Development LLC. All rights reserved. 28 IMPACT 2014 THE BUSINESS OF TALENT®
Adopt some new approaches
Good enough is good enough
Predictive analytics
Continuous monitoring
Copyright © 2014 Deloitte Development LLC. All rights reserved. 29 IMPACT 2014 THE BUSINESS OF TALENT®
Marketers measure and optimize ad campaigns continuously
Find inspiration in finance and marketing
Source: Adapted from Harvard Business Review, A Better Way to Measure Your Ad Campaign, 3/2013
Market conditions • Season
• Consumer confidence
• Fuel prices
Marketing actions • TV advertisements
• Online ads
• Social media
Competitive activity • Product releases
• Pricing
• Promotions
• Search
• Online chatter
• Store visits
• Purchases
• Unit sales
• Revenues
• Margins
• Market share
• Share of voice
• Customer value
Response
Outcomes Analyze Optimize Allocate
Copyright © 2014 Deloitte Development LLC. All rights reserved. 30 IMPACT 2014 THE BUSINESS OF TALENT®
…which (often) makes it easier
to gain access to resources
The payoff
And more
efficient operations
Better
alignment
More
effective learning
…which (often) leads to better
relationships with business leaders
…which (often) leads to more
credibility and opportunities
This publication contains general information only and Deloitte is not, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional advice or
services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your business. Before making any decision
or taking any action that may affect your business, you should consult a qualified professional advisor.
Deloitte shall not be responsible for any loss sustained by any person who relies on this publication.
As used in this document, "Deloitte" means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of the legal structure of Deloitte LLP
and its subsidiaries. Certain services may not be available to attest clients under the rules and regulations of public accounting.
Copyright © 2014 Deloitte Development LLC. All rights reserved.
36 USC 220506
Member of Deloitte Touche Tohmatsu Limited