Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome...
-
Upload
sarah-andersen -
Category
Documents
-
view
218 -
download
2
Transcript of Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome...
Program Evaluation
Program Evaluation
Overview and Discussion of:• Objectives of evaluation• Process evaluation• Outcome evaluation• Indicators & Measures• Small group discussions
Program Evaluation: Here we go again!
WHY WHY WHY do evaluation?• Time• Resources• Validity and reliability• Action
Why Evaluate?
• Increase knowledge & understanding of the program being evaluated
• Identify specific problems/needs, as well as effective practices
• Develop capacity for ongoing assessment and improvement
• Evaluation is of little value unless it leads to further decisions and activities re: program improvement
Why Evaluate?In the past…
Most often, the focus of evaluation was on what agencies did – how many clients did they see, how many clinical hours were spent, how many sessions did they deliver
Now…There is an increasing focus on what happens when agencies deliver services – what changes, what improves for the client
Thorny Issues in Program Evaluation
• ?
• ?
• ?
• ?
• ?
• ?
Thorny Issues in Program Evaluation
• Cost and human resources• Buy-in at all levels• Logic models that defy logic (overly
ambitious)• Process versus outcomes/impacts• Choosing the right indicators/measures• Moving data to action
Capacity to do evaluation
Capacity to use evaluation
Human Resources•Staffing•Technical Skills•Communication Skills•Professional Development•Leadership
Organizational Resources•Budget•Ongoing Data Collection•Infrastructure
Evaluation Literacy•Results-Management Orientation•Involvement-Participation
Organizational Decision-Making•Management Processes•Decision Support
Learning Benefits•Instrumental-Conceptual Use•Process Use
{
{
Evaluation Planning and Activities•Evaluation Plan•Use of Consultants•Information Sharing•Organizational Linkages•External Supports
Framework of Evaluation Capacity, Isabelle Bourgeois. Do not cite without author’s permission.
Ensuring support & buy-in
• Efforts to ensure support & buy-in for evaluation from staff & stakeholders should happen from the onset of deciding to conduct an evaluation
• Hold preliminary meetings• Clarify roles & responsibilities over the evaluation• Identify why the evaluation is being done; clarify any
misconceptions; provide reassurance• Identify who the end-user will be• Develop agreement on what is to be evaluated• Assess readiness for evaluation
Logic Models
“And this is our new logic model!”
Keep it as simple and as focused as possible!
Process Evaluations
Key Issues:• Service Delivery – the extent to which
program components are being delivered as planned
• Coverage – extent of target population participation in program
• Bias – extent to which subgroups of the designated target population participate (or don’t participate)
Process Evaluations – Strengths & Weaknesses
Strengths:• Measures how well program meets client needs• Measures the extent to which the program was
implemented as intended• Ensures a consumer-driven program• Easy & inexpensive
Weaknesses:• Bias can be significant
o Few dissatisfied participants return surveyso Clients may report inflated levels of satisfactiono Very subjective
Outcome Evaluations
Focus: program results, program effectivenessFocus: program results, program effectiveness
Examples of OutcomesExamples of Outcomes: : • Change in circumstances• Change in understanding/knowledge• Change in attitude• Change in behaviour
What Are Some of Your Outcomes?How are you measuring them?
• ?• ?• ?• ?• ?• ?
Indicators and Measures – Measurement StrategyOutcome Indicator(s) Source of data
(records, clients, caregivers, etc.)
Method to Collect Data
& Frequency
Who collects
data
When collects
data
EXAMPLE:Skill acquisition – self management, adaptive
Increase in scores on administered psychometric test, caregiver survey
Caregivers,Psychometric measures
Pre and post administered tests, surveys
Evaluation Researcher
At the beginning and end of program, and at six months post program
Moving Results Into ActionUse results
• Develop action plans for improving program effectiveness
• Develop plans to disseminate and share findings – be innovative!
• Plans should be developed from the beginning and continue throughout
• Linkages & partnerships developed in the initial stages of the evaluation form the basis of plans for information sharing
Centre of Excellence Activities
• Consultation services– Development of evaluation framework, logic
model, indicators and measures– Training or workshops– Facilitating or developing networks/ communities
of practice (e.g., Triple P)– Review of grant applications
• Evaluation grants
Objectives – Evaluation Grants
• Build skills– Developing evaluation framework and logic model– Implementing evaluation plan and analyzing
results– Using evaluation findings
• Build support– Frequent communications between the Centre and
agencies– Engage internal and external stakeholders
• Build communities– Establishing communities of practice
Communities of Practice – Triple P
• Nine agencies – key contacts for Triple P within Ontario
• Developing a common approach to evaluation – measuring same set of outcomes with common measures
• Facilitating a provincial roll-out of Triple P data
Communities of Practice – Triple PWhat made it work?• Clearly identified purpose• Willingness to work together• Face-to-face meeting at strategic points• Communication portal to facilitate active
learning and sharing of ideas• Centre as facilitator
Small Group Discussion
1. Think of a program in your agency. List all the external and internal stakeholders for the evaluation.
• What are some challenges in meeting the interests of all stakeholders in your program?
• What are some ways your program has been successful in getting buy-in from key stakeholders?
Small Group Discussion
2. In your local area or region, what community of practice would you like to be involved in?
• What topics or issues would you like the community of practice to discuss and meet about?
• How can the Centre of Excellence help you in forming and maintaining this community of practice?
Program Evaluation
Contact InformationContact Information:
Susan Kasprzak
Research Associate(613) 737-7600 ext. 3320
Tanya Witteveen
Research Associate
(613) 737-7600 ext. 3483
Visit our website for more information:
www.onthepoint.ca