Tracking ROI of Training

8/14/2019 Tracking ROI of Training

1/10

Bersin & Associates and LTI Magazine 1 August, 2003

Training AnalyticsMarket Analysis:

How Do Companies MeasureTraining and ROI?

Bersin & Associates

Bersin & AssociatesMarch, 2005


2/10


3/10


4/10

Bersin & Associates Market Research

Bersin & Associates 4 March, 2005

EffectivenessEffectiveness

EfficiencyEfficiency

ComplianceCompliance







1. Effectiveness. How well does this program actuallyimpact learning, job results, business results, or costsavings.

2. Efficiency. How cost-effectively did we reach theaudience? What was our cost per student or student-hourand how does it compare with other programs?

3. Compliance. Did we achieve our compliance targets? Who is in compliance andhow is not? What level of compliance did we achieve? What is our risk of falling out ofcompliance?

You can read more about our measurement approaches in our whitepaper, TrainingAnalytics, the Time is Now.

Learning Results: Scores and Certifications

A critical goal of training is to improve human performance; hence one would expect a

very high percentage of firms would measure scores or other learning-related metrics.This was not true. In our survey only about half of the respondents stated that theyroutinely track scores for training programs. Our qualitative surveys show that soresare used and tracked when the program has a mandatory certification nature.

Why is this? If you go to the trouble of training people, shouldnt you always measurewhether or not they learned anything? There are many reasons why programs are notscored: lack of infrastructure, the program did not have the budget, inability to decidewhat to measure, or just lack of time. These are not excuses but rather real businesslimitations. We would hope this number to go higher over time as more and moretechnology is applied to training, but as described above, many training departments donot have the budget or focus on this area.

There is also some confusion about whether assessments can adequately measure actuallearning and job performance. One participant commented, Pre-testing has come andgone because we are not clear that is best use of time and that it is accurate representation oflearning. We always have some degree of post-testing - as an effort to reinforce key messagesrather than to measure learning. This comment indicates that scoring is more of amotivational tool than a measurement tool.

Another aspect to scoring is the need to measure workers on-the-job. One participantdescribed his companys technique in this way: We regularly track observed performanceof skills. We are a high tech company and train technicians to fix and repair our products.Physical skills are observed prior to certification. As any experienced training professionalknows, demonstration of on-the-job skills is the most valuable way to truly test Level 2learning.

Compliance or Certification Driven Programs

Many training programs are compliance or certification driven. 52% of respondentstold us that they regularly run these types of programs. We define these as programsthat must be strictly monitored for completion, where completion may be defined assimply enrolling, actually completed, or passing a series of assessments. These programsare often driven by actual learning needs or may be driven by government regulationneeds.


5/10



In some cases learning is not the driver, hence scores are not used. A good exampleis sexual harassment training. If a company certifies that all its managers havecompleted sexual harassment training, they can no-longer be sued for certain types ofsexual harassment claims. The score or learning results of this program are notrelevant to the business need. The business driver is simply to make sure thateveryone has completed.

Measuring E-Learning vs. Traditional Approaches

With the surge in e-learning over recent years, a number of firms are evaluating their e-learning programs relative to their classroom-based programs. One participantcommented, We track why students elect to take web-based training when classroomtraining is available, the hours when students want to take web-based training, if the costs ofthe web-based training were reasonable, and what element (trainer, presentation, graphics, etc.)of the training was the most and least valuable.

Analysis: This is a very healthy trend. In the early days of e-learning companies were just

happy to get courses online. We have reached the maturity level now where organizations arestarting to try to benchmark the effectiveness of their e-learning vs. other approaches, and oftenfind that e-learning does fall short. As our Blended Learning: What Works study found,companies are quickly finding that for certain high-impact programs, blended delivery is needed.

What Job Impact do Companies Want to Measure?

Very few firms, less than 15%, are measuring learners behavioral changes or the jobimpact of training programs, and only one in ten measures the business impact of theirtraining programs on a routine basis. For those that do go to this level, what jobimpacts do they measure?

Analysis: This problem is one of the big reasons why we are very positive and working hard tohelp companies understand the value of Training Analytics systems, which make this problemmuch easier to solve.

When asked what types of business measures do you try to correlate to training, thelist was as follows:

Business MeasuresCompanies Want to Correlate to Training

Measure % of Organizations whowant to use It

Employee Turnover 11%

Employee Productivity 19%


6/10



Sales Revenue 16%

Sales Cycle Time (time to close a sale) 10%

Time to Market 13%

Product or Service Quality 29%

Operational Cost Savings 24%

Figure 2: What business measures do companies want to correlate?We were surprised by the number of comments from companies who want to measureother business impacts. Some other areas include:

Quality of service done the first time (rework)

Service call surveys (customer satisfaction)

Accidents (worker compensation incidents)

Increased sales of trained VARs vs. untrained VARs (value-added resellers)

Speed of answer (technical support).

Cost Savings Frequently Used as a Measure

As cited previously, costs are a primary concern and the most common way of assessingtrainings impact on the business. When asked about specific data being measured, costsavings was most often cited. One participant explained his companys measurementactivities:

I do an annual survey to determine Level 3 (Job impact of training). In the samesurvey, I ask managers/employees to tell me how the program has saved time/costand to be specific in terms of dollar amount.

The Trap of Cost Reduction (Commentary)

The most popular measure of business performance we found (and continuously hear) is

cost reduction. This particular program reduced costs by xyz. If these costreductions are actual cost reductions in business operations, these are excellentmeasures. However, if companies are trying to score their programs by the amount ofmoney saved on training, this becomes a trap.

Consider this. Most companies spend 1-3% of their payroll on training. If the trainingorganization saves 20 or 30% of this, the overall company savings will be a very smallfraction of overall company expenses. There is usually not that much budget in thetraining organization to save. Moreover, most e-learning savings are a one-time effect.The first year things look cheaper, but what do you do for the second year?

Ultimately if you really want to save money on training, you can eliminate it altogether.We think people should seriously consider whether this is a valid measure at all. The

business impact of training is 10-100X higher when viewed as a performance enhancer,so if you save $10 in training delivery but potentially lose $100-1000 in companyperformance, the savings are clearly nil.


7/10



Smaller Companies More Easily Measure Impact

Small to medium-sized organizations (those with less than 1,000 employees) weresomewhat more likely than large enterprises to measure the job or business impact oftraining. Seventeen percent (17%) of Small to medium-sized organizations reported thatthey routinely measure the job and business impact of their training programs. Justeleven percent (11%) of large enterprises measures the job impact and only 5% measurethe business impact of training. We believe the reason for this is that smaller companies

just have an easier time obtaining data and talking with managers about what impact atraining intervention had.

Large enterprises find it difficult to obtain good data due to the complexity of theirinternal structures and systems, as shown below. Some participants believe it is toodifficult to assess the business impact of training. As one individual commented:

The only relevant measures are employee-specific. More general business measuresare confounded by too many other factors; it is and would be impossible to identifythe training impact, per se, except in the broadest overall way.

A Clear Lack of Tools and Processes

We asked people what they wanted to measure more closely and what issues they facethat holds them back. The greatest need companies have is to understand how tobetter measure business impact.

Why do so few do it (less than 11%)? Companies clearly lack the tools and processesto make such measurement easy. 70% of respondents said that they would measurethe business impact of training programs more routinely if they had better tools, andthe same number said they would be helped by having more formal processes ormethodologies in place.

Needs for Measuring Business Impact of Training

61%

63%

63%

70%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Better reportingsystem

More resources or

budget

Higher priority by

mgmt.

Better measurement

tools

Better process or

methodology

All Organizations

71%

Figure 3: What do companies need to improve measurement processes and results?

Large enterprises were more likely than small to medium-sized organizations to cite alack of tools and processes in preventing them from measuring the business impact oftheir training programs. We believe this is because small companies can more easilytalk to managers and obtain information on their audiences.


8/10



LMS Systems Fall Short

We regularly study the Learning Management Systems (LMS) market, and recently

identified that centralizing information about all my training investments is now the #1reason companies give for purchasing an LMS. Despite this need, the promise appearsnot to be met.

While large enterprises are more likely to have an LMS in place, of the companies whohad an LMS, only 3% gave their systems top marks for being able to provide the neededmeasurements and reporting. In fact, 22% rated their LMS poor in providingreporting and measurement, and 39% rated their LMS fair. Our findings say that aftergoing to the expense of purchasing an LMS, only about a third of companies are satisfiedwith the reporting and measurement solution they have obtained.

There could be many reasons for this and our whitepaper Training Analytics: TheTime is Now describes some of these in detail. We believe that this is an important

new area for LMS systems, and strongly recommend that companies learn aboutTraining Analytics before selecting or purchasing an LMS system.

Rating of LMS's Ab ility to Provide Necessary

Measurements & Reporting

3%

39%

36%

22%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Excellent

Good

Fair

Poor

All organizations

Figure 4: How would you rate your LMS in its ability to provide you the measurement andreporting you need?


9/10



Demand for Greater Spending

Although measuring the impact of training is considered to be very valuable by mostfirms, very few have put significant resources behind it. The majority (51%) ofparticipants who were knowledgeable of their departmental spending said they spendless than 2% of their total annual training budget on tracking and measurement. Largeenterprises spend more than smaller organizations, but the spending is still extremelylimited.

Spending on Measurementas % of total training budget

13%

4%

18%

22%

57%

44%

9%

33%

0% 10% 20% 30% 40% 50% 60%

10% or more

5-9%

2-5%

less than 2%

Small/Medium-SizeOrganizations

Large enterprises

Figure 5: What percentage of your training budget do you spend on measurement and analyticstoday?

Most participants recognize that more needs to be spent on tracking and measurement.One in four said they should be spending much more on tracking and measurement activitiesand half said they should spend slightly more. We see a latent demand for better analyticssolutions.

Conclusions and Analysis: Opportunity for Impact

The subject of training measurement is one filled with research, opinions,methodologies, and case studies. Despite this body of knowledge, we find over andover that our clients and research participants struggle with the lack of tools andmethodologies to measure, benchmark, and score their training investments. Hence in

most organizations, there is a tremendous opportunity to more effectively measuretraining.

As you have probably read, If you cant measure it, you cant manage it. We are veryenthusiastic that companies understand the problems they face and are working hard toimprove this critical part of the training organization.


10/10



Additional Resources Available

We have invested a lot of our research into this market, and have many resources tohelp companies further understand the whys, hows, and whats of trainingmeasurements and analytics. These include our whitepaper Training Analytics, TheTime is Now and our recent 3-part webinar series on Training Analytics. Both areavailable on our website by going tohttp://www.bersin.com/research/analytics_measure.htm . If you become a researchsubscriber, you can receive additional methodologies and detailed case studies on bestpractices.

About Us

Bersin & Associates is a leading provider of corporate and vendor consulting services in

enterprise learning technology and implementation. With more than 20 years ofexperience in e-learning, training, and enterprise technology, Bersin & Associatesprovides a wide range of services including product development, product marketing,industry research, corporate workshops, corporate implementation plans, and sales andmarketing programs. Some of Bersin & Associates innovations include a completemethodology for LMS selection and application usage, an end-to-end architecture andsolution for e-learning analytics, and one of the industrys largest research studies onblended learning implementations. Bersin & Associates can be reached atwww.bersin.com or at (510) 654-8500.
http://www.bersin.com/research/analytics_measure.htmhttp://www.bersin.com/research/analytics_measure.htm

Tracking ROI of Training

Documents

Transcript of Tracking ROI of Training