Exploring Divisional vs. Managerial Performance Evaluation ...
Outline – Part I 1. Definitions 2. Pseudo-evaluation vs. legitimate evaluation 3. Formative vs....
-
Upload
grace-shepherd -
Category
Documents
-
view
216 -
download
0
Transcript of Outline – Part I 1. Definitions 2. Pseudo-evaluation vs. legitimate evaluation 3. Formative vs....
Outline – Part I
1. Definitions
2. Pseudo-evaluation vs. legitimate evaluation
3. Formative vs. summative evaluation
4. Necessary skills
5. Planning an evaluationa. Planning a Formative evaluation
b. Planning a Summative evaluation
6. Cost benefit analysis
7. An experimenting society
8. Words of warning
Definition (1)
Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object.
-- William Trochim (Cornell University)
3
Pseudo-evaluation vs. legitimate evaluation
Pseudo-evaluations Evaluation usually occurs in a political context.
Legitimate evaluations
Be careful not to engage in the first type (Pseudo-evaluation)!
Be careful not to engage in pseudo-evaluations
Doing so may facilitate inappropriate decisions
It will also damage your professional reputation
Pseudo-evaluations – a taxonomy
Here are some procedures to watch out for (E.A. Suchman, 1967):
Eyewash – emphasis on surface appearances
Whitewash – attempts to cover up known failures
Pseudo-evaluations – a taxonomy
Submarine – political use of evaluation to destroy a programme
Posture – ritualistic evaluation to satisfy a funding requirement, without real interest in, or intention to use, its findings
Postponement – using the need for evaluation to delay action
Legitimate Evaluations – Four Criteria
Here are four criteria to help you recognize a legitimate evaluation:
1. Utility
2. Feasibility
3. Propriety
4. Technical adequacy
Four criteria for legitimate evaluations
1. Utility – will someone be able to use it?
As Robson says, “the purpose of an evaluation is not to prove, but to improve.” (2002, p. 209)
Four criteria for legitimate evaluations
2. Feasibility – will you have the resources, time, and co-operation you need? If not, don’t do the evaluation.
Won’t achieve anything useful May damage your professional reputation. Especially an issue in formative evaluation, where
results may be needed for program planning. Remember the engineer’s maxim:
“Good, fast, cheap. Pick any two.”
Four criteria for legitimate evaluations
3. Propriety – only do an evaluation if you can do it fairly and ethically.
No ‘submarines’ Acceptable outcome measures Say ‘no’ if you believe the course of action has
already been decided on, and a decision maker just wants ‘cover.’
Four criteria for legitimate evaluations
4. Technical adequacy – if you are satisfied on the first three issues, carry out the evaluation with technical skill and sensitivity.
How can you tell whether you have the technical skill? What do you have to think about in planning? What are the relevant skills?
We’ll consider these issues below…
What to think about in planning
Reasons for evaluating
Why is the evaluation being done? Who should have access to the information obtained?
What value will results have?
Will action be taken? Will someone not want results published?
What to think about in planning
Interpretation
Is the nature of the evaluation agreed upon by those involved?
Outcome measures
What type of change is good, or bad?
What to think about in planning
Subject
What kind of information do you need?
Evaluators
Who will gather the information?
Who will analyze the data and write the report?
What to think about in planning
Methods
What method is appropriate given the questions?
Can you develop your method in the time allowed?
Is your method acceptable to those involved? (Service providers and consumers.)
What to think about in planning
Time
What time is available? Is this sufficient?
Permissions and control
Necessary permissions obtained? Is participation voluntary? Who decides what goes into the report?
What to think about in planning
Use
Who decides how the evaluation will be used?
Will those involved (providers, consumers) see the report in a modifiable draft version?
Is the form of the report appropriate for the intended audience (style, length, stats)?
An evaluation culture
These ideas are based on Donald Campbell’s (1969) concept of an experimenting society, and Trochim’s related concept of an evaluation culture
To learn more about Trochim’s ideas, see: http://www.socialresearchmethods.net/kb/evalcult.php
An evaluation culture
An evaluation works and improves because the culture is:
Action-oriented Teaching-oriented Diverse, inclusive, participatory, responsive and
fundamentally non-hierarchical. Humble, self-critical
An evaluation culture
An evaluation works and improves because the culture is:
Interdisciplinary Truth-seeking, forward-Looking Ethical, and democratic
Words of warning
Keep it simple
Avoid complex designs and data analysis
Think defensively
Anything that can go wrong, will go wrong. Try to anticipate potential problems and plan how
you will deal with them.
Words of warning
Change will always have sponsors and critics. People’s lives may be radically changed On the basis of your findings. jobs may be on the linecareers may be advanced or sloweda program may be expanded or cut back
Words of warning
There will be many stakeholders – politicians, administrators, deliverers, targets, unions, taxpayers.
It is unlikely that the interests of all these groups will coincide.
Outline – Part II
1. Formative & Summative evaluation defined
2. Elements of a Formative evaluation
3. Elements of a Summative evaluation
4. Evaluation strategiesA.Scientific-Experimental Paradigms
B.Management-oriented systems models
C.Qualitative-Anthropological models
D.Participant-oriented models
5. Necessary Skills
Two Types of Evaluation
Formative evaluation
Helps in the development of a program or service.
Summative evaluation
Assesses the effects and effectiveness of the program
Covers all effects, not just those intended
Formative Evaluation - Elements
Questions about the process being evaluated:
1. Structured conceptualization
2. Logic model
3. Process evaluation
4. Implementation evaluation
Formative Evaluation – elements
1. Structured conceptualization – helps stakeholders define program, targets, and desired outcomes.
Stakeholders – who are they? Outcomes – how do you plan to measure them?
Formative Evaluation – elements 2. A logic model makes explicit the
steps that are expected to produce the desired change. It is often shown as a flow chart or map.
A good logic model may reveal hidden assumptions about how intervention will work.
Formative Evaluation – elements
3. Process evaluation – What alternative procedures are available for delivery of the program?
4. Implementation evaluation – Is program being delivered the way it is supposed to be? Are there unexpected consequences?
Summative Evaluation
Outcome evaluation Did program cause demonstrable effects on
predefined outcome measures?
Impact evaluation Broader – assesses overall effects, intended
and unintended, of a program
Summative Evaluation Cost-benefit analysis
Questions about efficiency Standardizes outcomes in terms of dollar
costs and dollar benefits Important when you have to choose how
to spend limited amounts of money
Cost-Benefit Analysis
To do cost-benefit analysis you need to know (in addition to program cost)
(a) magnitude of benefits a program produces and
(b) that the program produced these benefits.
These things can only be learned through an experimental design.
Cost-Benefit Analysis
Some issues to consider before you do CBA…
1. Opportunity cost
2. Present value of money
3. Fairness
4. Complexity
CBA and Opportunity Cost
CBA expresses values in dollars. This reveals opportunity cost – if you do X with
your money, you cannot do Y with the same money.
Some values are difficult to express in dollars. E.g., what is the value of having mail delivery in rural areas?
How do you express non-market values in dollars?
CBA & Present Value
CBA works with the Present Value (PV) of money.
Future outcomes are uncertain. Inflation alters value of money – e.g., PV of
$1m in 50 years at 5% inflation = $87,000 .
CBA & Present Value
$100 of benefit today is worth more in Present Value than $100 of benefit 5 years from now.
This makes sense, but biases program evaluation away from long-term outcomes
CBA & Fairness
CBA compares benefits and costs without regard to who benefits and who pays costs. Is that fair? Is it unavoidable?
For example, people who live in the city subsidize mail delivery to people who live in the country. Is that fair? CBA doesn’t answer that question.
CBA & Complexity
Most social problems, and many problems in the private sector are complex.
They have many interacting causes, so establishing cause may be difficult.
Any program is likely to make only a small difference.
But it still makes sense to quantify the value of a program, to see if we could spend our money to better effect.
The relevant skills? (Robson, 2002)
Writing a proposal Clarifying purposes of an evaluation Identifying, organizing and working with an
evaluation team Choosing design & data-collection techniques Interviewing Questionnaire construction and use
What are the relevant skills?
Observation Management of complex information systems Data analysis Report-writing Encouraging people to use the findings Sensitivity to political concerns