Learning's Big Data Problem: Measuring & Analyzing Impact Part 2

Learning’s Big Data Problem: Measuring & Analyzing ImpactPart 2

David Wentworth, Principal Analyst,

Brandon Hall Group@DavidMWentworth

Mike Rustici, CEO, Watershed

@mikerustici

Michael Rochelle, Chief Strategy

OfficerBrandon Hall Group@MichaelRochelle

HCM Research and Advisory Firm

© Brandon Hall Group 2017 2

FOUNDED IN 1993

Community of 300,000+

10,000 Global Clients

Leading Independent HCM Research & Analyst Firm

Publishing 2+ Pieces of Research Each Day

Sample of Our Clients

http://www.salesforce.com/

http://www.salesforce.com/

http://www.basinelectric.com/index.html

http://www.basinelectric.com/index.html

http://www.emerson.com/en-US/

http://www.emerson.com/en-US/

http://www.jabil.com/

http://www.jabil.com/

http://www.symantec.com/index.jsp

http://www.symantec.com/index.jsp

http://www.landolakes.com/

http://www.landolakes.com/

http://www.kohls.com/

http://www.kohls.com/

How We Help You


MEMBERSHIP

STRATEGICCONSULTING

EXCELLENCEAWARDS

PROFESSIONAL DEVELOPMENT

Learning & Development

Talent Management

LeadershipDevelopment

Talent Acquisition

WorkforceManagement

Open Surveys


Visit www.brandonhall.com click Open Surveys

If you would like to join a panel of survey takers, please contact us at [email protected]

• Extended Enterprise Learning 2017

• State of Talent Acquisition 2017

• HCM Technology 2017

http://www.brandonhall.com/

http://[email protected]

How To Ask Questions


• Submit your questionsor comments about the discussion to our presenters using the Questions tab on your control panel.

• Presentation slides and giveaways can be found in the Handouts tab on your control panel.

Recording & Slides will also be sent out after the webinar.

Agenda

The State of Learning Measurement

Moving and Measuring Beyond Formal

How Do We Get Better?

The Role of Learning Standards

© Brandon Hall Group 2017

What Got Us Here


8

Learning Measurement Impact Model


Drivers for Measurement

76%

55%

62%

53%

Organizational Performance Individual Performance

Levels 4 & 5 Levels 1-3


Measuring Outcomes


63%

54%

44%

41%

41%

31%

30%

33%

31%

24%

16%

39%

36%

14%

18%

13%

14%

15%

12%

9%

13%

10%

Individual performance

Employee engagement

Team effectiveness

Business process improvement

Voluntary turnover/retention

Ability to respond to market

Revenue growth

Rate of knowledge transfer

Leadership promotion rates

Profitability

Revenue per employee

Levels 1-3

Levels 4 & 5

Model Takeaways

• Organizations at all levels measure transactional learning outcomes: completion, grades, etc.

• Organizations at levels 4 and 5 are more focused on behavioral and organizational outcomes

• More than half (54.9%) of Level 1-3 companies say they don’t have the proper metrics to measure beyond Kirkpatrick Level 2, while only 32% of Levels 4 & 5 say the same.

11© Brandon Hall Group 2017

How Do We Measure Today?


Kirkpatrick

• Does not measure learning duration and uptake

• Learner sentiment does not correlate with change in behavior and performance

• Most organizations are at lower/lowest level but that doesn’t mean that something is wrong with learning measurement

• Requires additional level (level 5) which is enterprise return, hardest to quantify and attain


Kaufman

• More like Kirkpatrick plus

• Focuses on inputs and processes which should be broken out and evaluated separately

• Does shine a light on how the organization is doing with developing and delivering learning not just the learner reaction

• Does embody the idea of immediate yet small benefits for learners

• States training does not encompass all forms of learning

• Similar approaches to evaluating the application of training

• Similar in approach to measuring performance impact

• Societal output is a stretch but how your customers benefit is not


Brinkerhoff Success Case Method

• Does focus on connecting learning objectives to business outcomes

• Learner surveys can be effective if constructed properly

• Analyzing data is great if it is the right data

• Communicating what works and what doesn’t work is always good but it needs to be accompanied with a solution


Anderson’s Model for Learning Evaluation

• Does emphasize the need for a strong connection between learning and organizational performance

• More of an evaluation model for the overall learning strategy of the organization not an individual program

• Program does measure stakeholder returns compared to overall organizational returns for the long and short term

• Contemplates different metrics for measuring organizational and stakeholder requirements


Key Takeaways

• All four models fall short in taking into consideration the following:• Efficiency and effectiveness coefficients for learning• Multimodal learning approaches• Learning audience variability• Learning environments• Learning content impact vs. learning delivery impact • Require individual learning assessments• Link closely to performance management processes• Regular and frequent measurement intervals for behavioral

change and application of learning• Correlation of learning data to performance data• Manager vs. contributor learning• Team development learning• Unstructured learning or experiential learning• Quality of data being collected


18

A Broader Perspective


“Scalable learning is the new reason for large organizations to exist”

John HagelFounder

Deloitte Center for the Edge Innovation

A world of exponential change

“Without data you’re just another person with an opinion”

W. Edwards DemingManagement and Systems Legend

Use the data

So much data! Where do I start?

© Watershed 2017 21

categories + complexity

learning analytics =

Categories of data


Complexities of analytics


Bringing it all together


25

What Do I Need to

Get Started?


The role of learning standards


all data

Learning is so much bigger than the LMS. Track informal and

experiential activities.

xAPI and the LRS present a transformative technology for learning analytics. Seamlessly bring all your data together in one platform and perform real-time analytics tailor made for L&D.

business impact

Include data about behaviors and performance in your learning analytics real time

Create, view and distribute reports seamlessly in real-time.


A next generation learning ecosystembrings this all together.

This example from Visa shows that the LMS is just one small part of a modern learning ecosystem.

A new ecosystem

Getting started


01Gather your data

Start by collecting data in one place and in a common format. Make it low friction and ensure you have access.

04Explore your data

Starting to understand not just “what” is happening but “why”. Form questions and see if your data can answer them. Often the best answer leads to another another question.

02Get to know your data

Understand what you have. What is out there, what is reliable and what is missing? Do some simple evaluations and create baselines.

03Operationalize your data

Automate your way out of “Excel hell”. Define some interesting metrics and KPIs, start monitoring them routinely. What trends do you notice?

05Experiment with your data

Start a new learning program with analytics in mind. Set up well controlled experiments and A/B tests to validate a hypothesis. Create a culture of continuous improvement.

06Show off your data

We’ve long known that learning is vital, not we can probe it. Go forth and show the world!

Questions?

Submit your questions to the presenters via the

questions tab on your control panel

29

Type your questions here


Thank you for joining us today!

If you have any additional questions, please email us at [email protected].

Learn More About UsProducts and services that we offer


Membership Offers Tailored SupportRESEARCH ACCESS & EVENTS

• Reports • Case Studies, Frameworks & Tools• DataNow® & TotalTech®• Webinars and Research Spotlights• Annual HCM Conference

ADVISORY SUPPORT• Ask the Expert• 1 on 1 Consultations• Research Briefings• Benchmarking

CLIENT SUCCESS PLAN• Your Priorities• Executive Sponsor• Client Associate• Monthly Meetings


To learn more about becoming a member or to sign up for a free trial, please email us at [email protected]

Strategic Consulting Offers Expert Solution DevelopmentBENCHMARKING

• Competitive/Comparative

• Maturity Model

• Custom Research

STRATEGY• Business Case

• Planning

• Organization & Governance

TECHNOLOGY SELECTION• Vendor Selection

• Architecture Design

• Systems Evaluation

DEVELOPMENT & INTEGRATION• Program Design

• Assessment

• Survey

• Process Integration


To learn more about consulting, please email us at [email protected]

Why Brandon Hall Group


1

2

3

4

5

We offer flexible, high-value memberships that offer everything you need at a cost you can afford.

Our research is better – more targeted, easily digestible, focused on business results, predictive, prescriptive.

Our solutions gets better results – research-driven, focused on the business, simple, targeted, flexible approach.

Our case studies are unique for their diversity and focus on business results.

Our professional development is practical, tailoredto your needs, and equips you to take action.

We Do This Through…

Research and Analysis that Define New HCM Standards

Success Stories from Global Organizations that Show those Standards in Action

Professional Development that Assesses Current State and Enables Innovation

Flexible Services to Design and Implement Transformative Solutions

Recognition that Validates the Transformation


What Our Clients Have to Say

Brandon Hall Group’s Excellence Awards allows you to take a deeper dive and evaluate the programs that you’ve created at a level that you might not have the opportunity to do… They have always been a cornerstone for me.

- Katrina Williams, Director, Sales Capability

CDW

https://youtu.be/OgdA_tpM55U


https://youtu.be/OgdA_tpM55U

What Our Clients Have to Say

I describe the attention and that customer service that I’ve experienced as being excellent. They are very attentive and wherever I need to be connected to whomever, they are there to help me.

- Joanne Veech, Global Talent Leader

PwC

https://youtu.be/YkY-v8gfW5k


https://youtu.be/YkY-v8gfW5k

HCMx Radio


THE ONLY PODCAST IN THE HCM ARENA THAT WEAVES CURRENT MARKET RESEARCH, HR TECHNOLOGY AND INDUSTRY LEADERS INTO EACH EPISODE

To catch up on the latest episodes, visit www.brandonhall.com/hcmxradio or subscribe on iTunes

http://www.brandonhall.com/hcmxradio

Learning's Big Data Problem: Measuring & Analyzing Impact Part 2

Business

Transcript of Learning's Big Data Problem: Measuring & Analyzing Impact Part 2