7/30/2019 Chap13 Evaluation
1/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.1
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
2/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.2
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
3/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.3
MISCONCEPTIONS ABOUT
EVALUATION
While evaluation is seen as professionally and theoreticallydesirable, some have doubts about its worth.
1. It is too costly. This misconception is based on a very short-
term view of costs. The cost of riskis also part of the equation.
These risks include not knowing if mistakes were made and notknowing how to avoid these mistakes in the future.
2. The measures are not exact. This misconception does not
recognise that any complex decision is based on less than perfect
information.
3. It is too difficult. True, most evaluation recommendations
apply more obviously to the legitimate system. However, the
developmental efforts of the shadow system can be evaluated.
7/30/2019 Chap13 Evaluation
4/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.4
THE ROLE OF EVALUATION
The role of the evaluation stage is four-fold:
1. to measure what change has occurred
2. to improve the other three stagesinvestigation, design and
implementation
3. to see if the change is attributable to the learning episode
4. to see if the amount of change is worthwhile.
7/30/2019 Chap13 Evaluation
5/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.5
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
6/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.6
ASSESSMENT OF LEARNING
Assessment addresses the question What changes have occurred
in the mind of the learner as a result of the learning episode?. To
measure this change, the learning has to be converted into a
behaviour that can be observed and this has some weaknesses:
The behaviour is only a sample of the learning. The behaviourrepresents explicit knowledge only.
The learning generation processes may take some time.
The test has to tap into the appropriate knowledge.
The test has to tap into the same potential knowledge eachtime.
There are several issues that need to be considered when planning
assessment.
7/30/2019 Chap13 Evaluation
7/31Human Resource Development: Principles and Practice By Brian Delahaye
13.7
TYPES OF ASSESSMENT
Assessment can be divided into six types:
skill testsare used for procedural skillsobjective written testse.g. multiple choice
subjective written testse.g. essays
performance testsused to assess complex processes. Theexaminer may asses the process itself or the product often using a
pre-designed observation form (see Fig.13.1 on page 346).
learning diariesJournals written by the learner; good forassessing externalisation
portfolio assessmentcollections of the learners work; takesa long term view, focuses on quality and the learner reflects on
own work.
7/30/2019 Chap13 Evaluation
8/31Human Resource Development: Principles and Practice By Brian Delahaye
13.8
THE MEANING OF SCORES
When a test has been examined and given a quantitative score theresult is called a raw score.
This raw score can be interpreted in a number of ways:
criterion-referenced scoringthe learners score is comparedto a pass mark
norm-referenced scoringthe raw score is compared to theaverage of a nominated group
formative assessmentthe score is used for developmental
feedbacksummative assessmentthe score is used for evidence of
learning.
7/30/2019 Chap13 Evaluation
9/31Human Resource Development: Principles and Practice By Brian Delahaye
13.9
OVERLAP OF THE FOUR HRD
PROCESSES
The assessment of learning plays a dual rolebeing critical toboth the implementation stage and the evaluation stage of HRD.
In the implementation stage, assessment provides dynamic
feedback. Firstly, this feedback is often immediate. Secondly, it
provides irrefutable evidence that the learner finds difficult toignore.
In the evaluation stage, assessment provides initial evidence of
the success or otherwise of the learning experience. Thisfeedback may be taken into account immediately (e.g. the HR
developer changes the learning processes) or later (e.g.
recommendations in the Evaluation Report).
7/30/2019 Chap13 Evaluation
10/31Human Resource Development: Principles and Practice By Brian Delahaye
13.10
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
11/31Human Resource Development: Principles and Practice By Brian Delahaye
13.11
KIRKPATRICKS FOUR LEVELS
Kirkpatricks model is based on four levels of ascending order:
1. reactionmeasures the reactions of the learners to the
learning episode
2. learningmeasures the knowledge gained by the learners.
Has been discussed under Assessment of Learning.3.behaviourexamines the change of behaviour of the learner
on the job
4. resultsmeasures the impact of the learning on the
organisation as a whole
Several comments can be made about Kirkpatricks model.
7/30/2019 Chap13 Evaluation
12/31Human Resource Development: Principles and Practice By Brian Delahaye
13.12
COMMENTS ON KIRKPATRICKS
MODEL
The strengths of Kirkpatricks model include:
simple and seemingly robust model
recognises the impact of a learning episode should extendbeyond the individual learner
provides an easily remembered checklisteach level provides a unique examination of HRD.
The criticisms of Kirkpatricks model include:
the levels are not co-dependent the model concentrates on processes and outputs, not inputs the initial level is easier but measurement of the deeper levelsare more susceptible to contamination.
7/30/2019 Chap13 Evaluation
13/31Human Resource Development: Principles and Practice By Brian Delahaye
13.13
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
14/31Human Resource Development: Principles and Practice By Brian Delahaye
13.14
THE PRESAGE FACTORS
Brinkerhoffs model proposes six levels:
Stage I: Evaluate needs and goals ( this is similar to the HRDNI).
Stage II: Evaluate the HR design
Stage III: Evaluate implementation
Stage IV: Evaluate learning (similar to Kirkpatricks level 1)Stage V: Evaluate usage and endurance of learning (similar to
Kirkpatricks level 3)
Stage VI: Evaluate pay-off(similar to Kirkpatricks level 4).
Brinkerhoff, then, adds two that occur prior to the learning
episodeevaluate HR design and evaluate implementation.
These are called presage factors.
7/30/2019 Chap13 Evaluation
15/31Human Resource Development: Principles and Practice By Brian Delahaye
13.15
THE PRESAGE FACTORS (Continued)
Stage II, evaluating the HR design focuses on the quality of the
HR plan (i.e. the training or workshop program).
A stage II evaluation should occur when:
the design is unique or experimental
the costs of mounting the program are high the HRD issues are crucial
the participant groups are volatile, influential or demanding.
The design should be reviewed by a variety of stakeholders.
7/30/2019 Chap13 Evaluation
16/31Human Resource Development: Principles and Practice By Brian Delahaye
13.16
THE PRESAGE FACTORS (Continued)
Stage III, evaluating during the implementation provides data thatwill help shepherd the learning event to a successful conclusion.
This highlights the covert activity of the conducting HR
developer where the program is continually monitored to
compare reality with the program plan.
The HR developer can also be monitored, usually by another HR
developer. However, the big brother syndrome has to be
avoided.
7/30/2019 Chap13 Evaluation
17/31Human Resource Development: Principles and Practice By Brian Delahaye
13.17
TIME OUT
To this stage of the discussion, we have examined assessment of
learning, Kirkpatricks model and Brinkerhoffs model. This hasanswered two of the roles of evaluation:
1. to identify what change has occurred
2. to improve the other three stages.
The other two roles will now be addressed:
3. to see if the change is attributable to the learning episode. This
will be examined by looking at the SCIENTIFIC MODELS.4. to see if the amount of change was worthwhile. This will be
examined by looking at COST BENEFIT ANALYSIS.
7/30/2019 Chap13 Evaluation
18/31Human Resource Development: Principles and Practice By Brian Delahaye
13.18
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
19/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.19
THE SCIENTIFIC MODELSThe scientific models are based on the experimental methods
used in research laboratories and are used to demonstrate
causality. They also assume that changes can be measured.
Post-test = learning experience/evaluation
Pre-test - Post-test = evaluation/learning experience/evaluationTime series evaluation = pre-test/pre-test/pre-test/learning
experience/post-test/post-test/post-test
Control group =Experimental grouppre-test/learning experience/post-test
Control group pre-test/usual duty/ post-test
The Solomon fouruses three control groups and theexperimental group
The HR developer has to compare the costs of the designs
compared to the expected benefits gained.
7/30/2019 Chap13 Evaluation
20/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.20
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
7/30/2019 Chap13 Evaluation
21/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.21
COST-BENEFIT ANALYSISCost-benefit analysis (CBA) is used to see if the amount of
change was worthwhile. It is based on the premise:
identify the costs, in dollar terms, for the learning experience identify the thebenefits accruing from the learning
experience, in dollar terms
the ratiobetween the costs and benefits should be in favourof the benefits.
There are two problems often encountered:
converting thebenefits into monetary values deciding on the cut-off points.
There are two good reasons for conducting a CBA. Firstly, the
survival of the HR section may depend on proving the value of
development. Secondly, it helps in the decision of selecting the
learning programs that have the most impact.
7/30/2019 Chap13 Evaluation
22/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.22
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
13 23
7/30/2019 Chap13 Evaluation
23/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.23
THE EVALUATION PLAN
Planning for evaluation commences during the design stage. This
ensures that the appropriate evaluation occurs at the appropriatetime. It also allows the coordination of developmental and
judgemental evaluation. Planning or evaluation should include:
develop the assessment fordevelopmental purposes first incorporate this into the evaluation plus any further
assessment required decide whatpresage variables will be evaluated and when incorporate appropriate HRDNI investigation instruments design daily and course/workshop reaction sheets
designpre-test and post-test instruments, if appropriate identify and plan methods to be used at Stage V and VI if CBA is to be used, arrange collection ofcosts and benefits
prepare abudget send evaluation plan to staff who are affected.
13 24
7/30/2019 Chap13 Evaluation
24/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.24
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
13 25
7/30/2019 Chap13 Evaluation
25/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.25
THE EVALUATION REPORT
Once the evaluation has been completed a report should be
completed for the key stakeholders. The evaluation plan has acommunication role, decision making role and becomes an
historical document. This report should include:
an executive summary a findings/recommendations section
a contents list the main body, which should include: reasons for the evaluation list of personnel involved
discussion of the various types of evaluation undertaken a discussion on the findings, the options and theconclusions
a list of, and discussion of, the recommendations appendices.
13 26
7/30/2019 Chap13 Evaluation
26/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.26
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
13 27
7/30/2019 Chap13 Evaluation
27/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.27
THE LEGITIMATE SYSTEM
Most of what has been discussed in this chapter applies directly
to the legitimate system. The methods mainly rely on negativefeedback systems. The methods also emphasise the legitimate
systems focus on the efficient use of resources and the immediate
survival of the organisation.
For the legitimate system, one of the advantages of competency-
based learning has been the recognition of the strong link
between learning in a planned program and the application of
that learning in the workplace.
It should also be recognised that, if the assessment process fails
to detect a lack of comprehension on the part of the learners, the
HR developer could be held liable.
13 28
7/30/2019 Chap13 Evaluation
28/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.28
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
13 29
7/30/2019 Chap13 Evaluation
29/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.29
THE SHADOW SYSTEM
Evaluation in the shadow system is a very delicate affair. The
use, by a third party, of the methods described in this chapterresults in negative feedback loopsand this kills the creativity
needed in the shadow system.
The main evaluation method for the shadow system is the use of
values systems. The manger instills a particular value in thesystem and checks for evidence for this value.
The actors in the shadow system, however, should use the
evaluation methods on themselves and their own endeavours.
13 30
7/30/2019 Chap13 Evaluation
30/31
Human Resource Development: Principles and Practice By Brian Delahaye
13.30
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
13 31
7/30/2019 Chap13 Evaluation
31/31
13.31
THE NEED FOR DIALECTIC
THINKING
Some of the misperceptions about evaluation can be explainedby the concept of dialectic thinking (see Chapter 2).
Evaluation has a number of opposing characteristics:
evaluation can be both developmental and judgemental
the more objective the measure used, the less rich the insightsand vice versa
liberal education vs. vocational education.
Evaluation serves and supports a number of dual roles. Balancingthese conflicting roles is the new challenge as knowledge is a
valuable but delicate resource.