Level 3 Evaluation Pilot - MUSSELs | The Ministry of...

Elizabeth Dalton

Assessments & Evaluations PM

Learning Products Group

Leve l 3 Eva lua tion P ilot

Elizabeth DaltonBurt Parce ls

Sun Proprie tary/Confidentia l: Interna l Use Only

Agendaÿ Overview of Kirkpatrick Evaluation Levelsÿ Data from levels 0-3ÿ Overview of Level 3 ÿ Specific Research Goalsÿ Method ÿ Summary of Findingsÿ Policy Implications of Findingsÿ Key Recommendations


Kirkpatrick Eva luation Leve lsLevel Purpose Methods Comments

0 Completion Survey

1 Reaction Survey

2 Learning Post-Test Requires Pre-Test for full analysis3 Application Expensive, can be intrusive

4 Business Impact Analysis of Metrics

5 ROI

Analysis of Assessments

Requires automated progress trackingEasy to implement, learners are used to format

Surveys, Po2, Observation, Interviews

Issues in defining and measuring Business Metrics

Analysis of Costs, Comparison to Level 4

Issues in tracking costs, issues with Level 4

Psychometric Quality

Requires statistical tools and skills


Hardware Product A Eva luation 0-2 OverviewComponent Completed CommentsP rew ork W LC 10164% Average Completion"P re test" (P o1A ) 214 175 PassedW ebcast (L& R ) ?M obile Lab 569*ILT ** 543?P ost-Test (P o1B ) 151 137 PassedLeve l 1 survey 240 94% Success Rate


Overview of Leve l 3ÿ Designated "Level 3" by Kirkpatrick ÿ Measures application of knowledge & skills ÿ Depends on thorough Level 2 analys isÿ Also benefits from thorough Level 1 ÿ May include interviews or direct observationÿ May be broad or narrow in scopeÿ Provides hard data to address policy issues


Logic Flow: Leve l IIIExit Score

OK ?

Yes

No Entry ScoreOK on

Pre Reqs

YesJob

Performanceis OK

Yes Possible Design Issue with Course

No Could be job factors or Design Issue w/CourseNo

Job Performance

is OK

Yes

No

Rare Case (individual factors)

Poss ible bad fit for the job.Entry Scoreindicates

MASTERY

Yes

No

Job Performance

is OK

YesNo Problem: Did NOT need Tra ining

NoPossible Attitude or Motivational Issues

Entry ScoreOK on

Pre Reqs

Yes

Job Performance

is OK

Yes No Problem!

No Possible transfer problem orposs ible issue with success crite ria o

poss ible des ign issuesNo

Job Performance

is OK

Yes

No

Possible entry/pre-req issues

Possible performance problem or transfer problem may be due tolack of pre req s /k


Specific Research Goalsÿ Level 2 Analys is :� Did Learning Occur?� Did Learning Vary By Learning Objective?� Rela tionship Between Learning and Topics '

Importance Rankings?� Rela tionship Between Learning and

Difficulty?


Specific Research Goalsÿ Level 3 Analys is :

� Was tra ining offered at the right time?� Did the right people a ttend tra ining?� Did we teach the right knowledges and skills?� Did we include the right tra ining features?� What kinds of re-learning took place a fter the

tra ining?� What kinds of improvements do our learners

suggest? � What kept potentia l learners from attending

tra ining?


ParticipantsActivity Invited RespondedHardware Product A Po2 501 16Hardware Product A Trained 42 22Hardware Product A Non-Traine 989 29Storage Product B Trained 178 56Storage Product B Non-Trained 1324 33


Summary of Findingsÿ Level 2 Overviewÿ Level 3

� Was tra ining offered a t the right time?� Did the right people a ttend tra ining?� Did we teach the right knowledges and skills?� Did we include the right tra ining features?� What kinds of re-learning took place after the tra ining,

and how could we he lp in that regard?� What kinds of improvements do our learners suggest?� What kept potentia l learners from attending tra ining?


Leve l 2 Findings: Hardware A

Su

n P

rop

rieta

ry/Co

nfid

en

tial: In

tern

al U

se

O

nly

Pre

pa

red

ne

ss

: Sto

rag

e B

Describe architecture

Install, configure, de-install

Describe volume emulation

Maintain and troubleshoot

Describe subsystem installation

Configure remote console

Install and use utilities

Connect Raid to SAN

Guide/advise customers

Attach Storage Subsystems to SAN

Describe SAN security issues

Install and use subsystem

Troubleshoot 'missing LUN'

ID differences between products

ID components, repair/exchange

Migrate data

GA & features of new products

0 5 15 25 35 45

Preparedness

Non-T

rained

Trained


Was tra ining offered a t the right time?Hardware Product A:

prior to Mar

Mar '02

Apr '02

May '02

Jun '02

Jul '02

Aug '02

Sep '02

Oct '02

no reply

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

6

Dates of Training and Application

Training

1st Encounter (Trained)

more than 90 days

60 to 90 days

45 to 60 days

30 to 45 days

14 to 30 days

Less than 14 days

0 1 2 3 4 5 6 7

Delay


Was tra ining offered a t the right time?Storage Product B:

Feb

Mar '0

Apr '0

May

Jun

Jul '0

Aug

Sep

Oct '0

Nov

Dec'

Jan

Feb

Mar '0

Apr '0

May

Jun

Jul '0

Aug

Sep'

Oct '0

no re

0

1

2

3

4

5

6

7

8

9

10

11

Storage B Training and Application Dates

Training

1st Encounter

more than 90 days

60 to 90 days

45 to 60 days

30 to 45 days

14 to 30 days

Less than 14 days

0 5 10 15 20

Delay


Skill Loss: Hardware Product A Comparison of Mean Scores by Objective (Po1-Po2)

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

Objective #1: TargetMkts/Industries

84.80% 87.50%

Objective #2: Idenify &discuss majorfeatures

89.60% 94.38%

Objective #3: Featuresof major components

82.30% 77.92%

Objective #7: Installand replace FRU's

89.70% 82.29%

Objective #8:Tro bleshoot vario s

93.38% 76.56%

Po1 Po2


Did the right people a ttend tra ining?

Storage B

Hardware A

0.00% 25.00% 50.00% 75.00% 100.00%

501501501501501501501501501

559559559559559559559559559

424242424242424242

178178178178178178178178178

989989989989989989989989989

132413241324132413241324132413241324

Trained v. Exposed

Product exposure: Overlap: Trained:


Did we teach the right topics? Hardware Product A:

Su

n P

rop

rieta

ry/Co

nfid

en

tial: In

tern

al U

se

O

nly

Did

we

tea

ch th

e rig

ht

top

ics?

Sto

rag

e P

rod

uct B

:

Describe architecture

Install, configure, de-install

Describe volume emulation

Maintain and troubleshoot

Describe subsystem installation

Configure remote console

Install and use utilities

Connect Raid to SAN

Guide/advise customers

Attach Storage Subsystems to SAN

Describe SAN security issues

Install and use subsystem

Troubleshoot 'missing LUN'

ID differences between products

ID components, repair/exchange

Migrate data

GA and features of new products

0 20 40 60 80 100

Importance and P

reparedness by topic

Importance

Preparedness


Did we include the right tra ining features?

Not primary offe ring

Han

ds-o

n La

b

Web

-bas

ed M

odul

es

Web

-bas

ed T

estin

g

Rec

orde

d W

ebca

st

Cla

ssro

om In

stru

ctio

n

0

10

20

30

40

50

60

70

80

Ratings of Aspects of Training

Hardware A

Storage B

Su

n P

rop

rieta

ry/Co

nfid

en

tial: In

tern

al U

se

O

nly

Did

re-le

arn

ing

occu

r? If s

o,

ho

w?

Read available doc's

Asked co-workers

Review web-based modules

Review recorded webcast

went to hands-on lab

Con-tacted identified resource

Used product team website

Other

No Re-learning Required (approx-imate)

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70

Relearning M

ethods


What kind of improvements do our learners suggest?

Mak

e it

ILT

Mak

e it

web

-bas

ed

Put

mat

eria

ls o

nlin

e

Ince

nt M

anag

ers

Tra

inin

g cl

oser

to u

se

Mat

eria

ls c

lose

r to

use

prov

ide

a le

arni

ng s

pace

Oth

er (

open

text

)0

10

20

30

40

50

60

70

80

Recommended Changes

Hardware AStorage B

Su

n P

rop

rieta

ry/Co

nfid

en

tial: In

tern

al U

se

O

nly

Wh

at ke

pt le

arn

ers

aw

ay?

Not offered in my geography

Arrived after the training

Not offered in my language

Found out too late

Not time available to attend

Other (open text)

0 10 20 30 40 50 60

Barriers to A

ttendance

Hardw

are AS

torage B


Obstacles to Analys isÿ Lack of individual test item dataÿ Lack of true pre-testing results from either groupÿ Extens ive manual data manipulation requiredÿ Weak/poorly designed test itemsÿ Inability of exis ting testing mechanisms to score tests

adequate lyÿ Lack of reporting and analys is capability in exis ting test

mechanismsÿ No level 2 data for Storage Product B, lack of access to

Level 1 data for Storage Product Bÿ Difficulty in accessing tra ining attendance and Case

Management data in useable formatÿ Training attendance data was highly inaccurate. Case

Management data a lso conta ined some inaccuracies.


Policy Implications of Findingsÿ Continue to emphasize high-order objectivesÿ Tra in closer to applica tion dateÿ Monitor third-party tra ining materia ls for qua lityÿ Include orienta tion to exis ting documentationÿ Continue to pursue Remote Labs and other

hands-on experiencesÿ Insert marketing names into tra ining materia ls as

early as poss ibleÿ Target tra ining more close ly to those who need it

firs tÿ Aid re-learning with adaptive re-learning toolsÿ Shorten learning time with adaptive /prescriptive

tools


Key Recommendationsÿ Work to target learners more close lyÿ Develop cost estimates of re learning, present to policy

makersÿ More links with Performance Support methods, e .g.

SunSolve, adaptive re-learning tools , J IT tra iningÿ Upgrade objective writing skillsÿ More focus to SLP or other means of tracking &

reportingÿ Upgrade testing deve lopment/de livery engineÿ Develop s tandard BRIO reportsÿ Consider survey issues (tracking, corre la tion with

tests)ÿ Require pretests , leve l 1, and leve l 2 data for any

Level 3 targeted LPD l id li f f L l 3 & i

Elizabe th Da lton

e lizabe th.da [email protected]

Level 3 Evaluation Pilot - MUSSELs | The Ministry of...

Documents

Transcript of Level 3 Evaluation Pilot - MUSSELs | The Ministry of...