Chap13 Evaluation

7/30/2019 Chap13 Evaluation

1/31

Human Resource Development: Principles and Practice By Brian Delahaye

13.1

CHAPTER 13: EVALUATION

MISCONCEPTIONS ABOUT EVALUATION

ASSESSMENT OF LEARNING

KIRKPATRICKS FOUR LEVELS

THE PRESAGE FACTORS

THE SCIENTIFIC MODELS

COST-BENEFIT ANALYSIS

THE EVALUATION PLAN

THE EVALUATION REPORT

THE LEGITIMATE SYSTEM

THE SHADOW SYSTEM

THE NEED FOR DIALECTIC THINKING


2/31


13.2





THE PRESAGE FACTORS



THE EVALUATION PLAN

THE EVALUATION REPORT THE LEGITIMATE SYSTEM

THE SHADOW SYSTEM



3/31


13.3

MISCONCEPTIONS ABOUT

EVALUATION

While evaluation is seen as professionally and theoreticallydesirable, some have doubts about its worth.

1. It is too costly. This misconception is based on a very short-

term view of costs. The cost of riskis also part of the equation.

These risks include not knowing if mistakes were made and notknowing how to avoid these mistakes in the future.

2. The measures are not exact. This misconception does not

recognise that any complex decision is based on less than perfect

information.

3. It is too difficult. True, most evaluation recommendations

apply more obviously to the legitimate system. However, the

developmental efforts of the shadow system can be evaluated.


4/31


13.4

THE ROLE OF EVALUATION

The role of the evaluation stage is four-fold:

1. to measure what change has occurred

2. to improve the other three stagesinvestigation, design and

implementation

3. to see if the change is attributable to the learning episode

4. to see if the amount of change is worthwhile.


5/31


13.5





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM



6/31


13.6


Assessment addresses the question What changes have occurred

in the mind of the learner as a result of the learning episode?. To

measure this change, the learning has to be converted into a

behaviour that can be observed and this has some weaknesses:

The behaviour is only a sample of the learning. The behaviourrepresents explicit knowledge only.

The learning generation processes may take some time.

The test has to tap into the appropriate knowledge.

The test has to tap into the same potential knowledge eachtime.

There are several issues that need to be considered when planning

assessment.


7/31Human Resource Development: Principles and Practice By Brian Delahaye

13.7

TYPES OF ASSESSMENT

Assessment can be divided into six types:

skill testsare used for procedural skillsobjective written testse.g. multiple choice

subjective written testse.g. essays

performance testsused to assess complex processes. Theexaminer may asses the process itself or the product often using a

pre-designed observation form (see Fig.13.1 on page 346).

learning diariesJournals written by the learner; good forassessing externalisation

portfolio assessmentcollections of the learners work; takesa long term view, focuses on quality and the learner reflects on

own work.



13.8

THE MEANING OF SCORES

When a test has been examined and given a quantitative score theresult is called a raw score.

This raw score can be interpreted in a number of ways:

criterion-referenced scoringthe learners score is comparedto a pass mark

norm-referenced scoringthe raw score is compared to theaverage of a nominated group

formative assessmentthe score is used for developmental

feedbacksummative assessmentthe score is used for evidence of

learning.



13.9

OVERLAP OF THE FOUR HRD

PROCESSES

The assessment of learning plays a dual rolebeing critical toboth the implementation stage and the evaluation stage of HRD.

In the implementation stage, assessment provides dynamic

feedback. Firstly, this feedback is often immediate. Secondly, it

provides irrefutable evidence that the learner finds difficult toignore.

In the evaluation stage, assessment provides initial evidence of

the success or otherwise of the learning experience. Thisfeedback may be taken into account immediately (e.g. the HR

developer changes the learning processes) or later (e.g.

recommendations in the Evaluation Report).



13.10





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM




13.11


Kirkpatricks model is based on four levels of ascending order:

1. reactionmeasures the reactions of the learners to the

learning episode

2. learningmeasures the knowledge gained by the learners.

Has been discussed under Assessment of Learning.3.behaviourexamines the change of behaviour of the learner

on the job

4. resultsmeasures the impact of the learning on the

organisation as a whole

Several comments can be made about Kirkpatricks model.



13.12

COMMENTS ON KIRKPATRICKS

MODEL

The strengths of Kirkpatricks model include:

simple and seemingly robust model

recognises the impact of a learning episode should extendbeyond the individual learner

provides an easily remembered checklisteach level provides a unique examination of HRD.

The criticisms of Kirkpatricks model include:

the levels are not co-dependent the model concentrates on processes and outputs, not inputs the initial level is easier but measurement of the deeper levelsare more susceptible to contamination.



13.13





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM




13.14

THE PRESAGE FACTORS

Brinkerhoffs model proposes six levels:

Stage I: Evaluate needs and goals ( this is similar to the HRDNI).

Stage II: Evaluate the HR design

Stage III: Evaluate implementation

Stage IV: Evaluate learning (similar to Kirkpatricks level 1)Stage V: Evaluate usage and endurance of learning (similar to

Kirkpatricks level 3)

Stage VI: Evaluate pay-off(similar to Kirkpatricks level 4).

Brinkerhoff, then, adds two that occur prior to the learning

episodeevaluate HR design and evaluate implementation.

These are called presage factors.



13.15

THE PRESAGE FACTORS (Continued)

Stage II, evaluating the HR design focuses on the quality of the

HR plan (i.e. the training or workshop program).

A stage II evaluation should occur when:

the design is unique or experimental

the costs of mounting the program are high the HRD issues are crucial

the participant groups are volatile, influential or demanding.

The design should be reviewed by a variety of stakeholders.



13.16

THE PRESAGE FACTORS (Continued)

Stage III, evaluating during the implementation provides data thatwill help shepherd the learning event to a successful conclusion.

This highlights the covert activity of the conducting HR

developer where the program is continually monitored to

compare reality with the program plan.

The HR developer can also be monitored, usually by another HR

developer. However, the big brother syndrome has to be

avoided.



13.17

TIME OUT

To this stage of the discussion, we have examined assessment of

learning, Kirkpatricks model and Brinkerhoffs model. This hasanswered two of the roles of evaluation:

1. to identify what change has occurred

2. to improve the other three stages.

The other two roles will now be addressed:

3. to see if the change is attributable to the learning episode. This

will be examined by looking at the SCIENTIFIC MODELS.4. to see if the amount of change was worthwhile. This will be

examined by looking at COST BENEFIT ANALYSIS.



13.18





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM



19/31


13.19

THE SCIENTIFIC MODELSThe scientific models are based on the experimental methods

used in research laboratories and are used to demonstrate

causality. They also assume that changes can be measured.

Post-test = learning experience/evaluation

Pre-test - Post-test = evaluation/learning experience/evaluationTime series evaluation = pre-test/pre-test/pre-test/learning

experience/post-test/post-test/post-test

Control group =Experimental grouppre-test/learning experience/post-test

Control group pre-test/usual duty/ post-test

The Solomon fouruses three control groups and theexperimental group

The HR developer has to compare the costs of the designs

compared to the expected benefits gained.


20/31


13.20





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM



21/31


13.21

COST-BENEFIT ANALYSISCost-benefit analysis (CBA) is used to see if the amount of

change was worthwhile. It is based on the premise:

identify the costs, in dollar terms, for the learning experience identify the thebenefits accruing from the learning

experience, in dollar terms

the ratiobetween the costs and benefits should be in favourof the benefits.

There are two problems often encountered:

converting thebenefits into monetary values deciding on the cut-off points.

There are two good reasons for conducting a CBA. Firstly, the

survival of the HR section may depend on proving the value of

development. Secondly, it helps in the decision of selecting the

learning programs that have the most impact.


22/31


13.22





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM


13 23


23/31


13.23

THE EVALUATION PLAN

Planning for evaluation commences during the design stage. This

ensures that the appropriate evaluation occurs at the appropriatetime. It also allows the coordination of developmental and

judgemental evaluation. Planning or evaluation should include:

develop the assessment fordevelopmental purposes first incorporate this into the evaluation plus any further

assessment required decide whatpresage variables will be evaluated and when incorporate appropriate HRDNI investigation instruments design daily and course/workshop reaction sheets

designpre-test and post-test instruments, if appropriate identify and plan methods to be used at Stage V and VI if CBA is to be used, arrange collection ofcosts and benefits

prepare abudget send evaluation plan to staff who are affected.

13 24


24/31


13.24





THE PRESAGE FACTORS



THE EVALUATION PLAN



THE SHADOW SYSTEM


13 25


25/31


13.25


Once the evaluation has been completed a report should be

completed for the key stakeholders. The evaluation plan has acommunication role, decision making role and becomes an

historical document. This report should include:

an executive summary a findings/recommendations section

a contents list the main body, which should include: reasons for the evaluation list of personnel involved

discussion of the various types of evaluation undertaken a discussion on the findings, the options and theconclusions

a list of, and discussion of, the recommendations appendices.

13 26


26/31


13.26





THE PRESAGE FACTORS

THE SCIENTIFIC MODELS COST-BENEFIT ANALYSIS

THE EVALUATION PLAN



THE SHADOW SYSTEM


13 27


27/31


13.27


Most of what has been discussed in this chapter applies directly

to the legitimate system. The methods mainly rely on negativefeedback systems. The methods also emphasise the legitimate

systems focus on the efficient use of resources and the immediate

survival of the organisation.

For the legitimate system, one of the advantages of competency-

based learning has been the recognition of the strong link

between learning in a planned program and the application of

that learning in the workplace.

It should also be recognised that, if the assessment process fails

to detect a lack of comprehension on the part of the learners, the

HR developer could be held liable.

13 28


28/31


13.28





THE PRESAGE FACTORS



THE EVALUATION PLAN



THE SHADOW SYSTEM


13 29


29/31


13.29

THE SHADOW SYSTEM

Evaluation in the shadow system is a very delicate affair. The

use, by a third party, of the methods described in this chapterresults in negative feedback loopsand this kills the creativity

needed in the shadow system.

The main evaluation method for the shadow system is the use of

values systems. The manger instills a particular value in thesystem and checks for evidence for this value.

The actors in the shadow system, however, should use the

evaluation methods on themselves and their own endeavours.

13 30


30/31


13.30





THE PRESAGE FACTORS



THE EVALUATION PLAN


THE SHADOW SYSTEM


13 31


31/31

13.31

THE NEED FOR DIALECTIC

THINKING

Some of the misperceptions about evaluation can be explainedby the concept of dialectic thinking (see Chapter 2).

Evaluation has a number of opposing characteristics:

evaluation can be both developmental and judgemental

the more objective the measure used, the less rich the insightsand vice versa

liberal education vs. vocational education.

Evaluation serves and supports a number of dual roles. Balancingthese conflicting roles is the new challenge as knowledge is a

valuable but delicate resource.

Chap13 Evaluation

Documents

Transcript of Chap13 Evaluation