Evaluating Progress and...
Transcript of Evaluating Progress and...
6/3/2014
1
Evaluating Progress and Outcomes Through a Community Lens
Shauna MacKinnon (PhD)Assistant ProfessorUrban and Inner City StudiesUniversity of WinnipegCo-Investigator, Manitoba Research Alliance©Shauna T. MacKinnon University of Winnipeg Personal collection 2014
Presentation for University of Manitoba Summer Institute in Program evaluation
June 4, 2014
The basics… Decisions about what to measure and the
methods to be used must be made in partnership with funding agencies, researcher-evaluator, community being evaluated and members of the community as identified by the community in which the inquiry takes place
This should apply to summative as well as formative evaluation
6/3/2014
2
Questions to consider How to evaluate government
programs/investments/activities through a community lens;
How government evaluation processes can be more responsive to community needs;
How intergovernmental alignment of evaluation processes would benefit the community
What we hear from community organizations about evaluation
Who decides what and how to measure? What about unexpected outcomes? Community priorities don’t always align with
government/funder priorities So what we measure and how it does not
always align well
Who is accountable to the community?
6/3/2014
3
What to do? Take a step back and decide
What should the purpose of evaluation be? What do we want to learn? What will we do with what we learn?
What Good Evaluation IS… Includes formative-process evaluation (not only
sumative-outcome evaluation) Designed and implemented early in the process
part of the process of program implementation Centred within cultural norms
Responsive to the history, needs, and dreams of the people participating in and being affected by the program being evaluated (American Indian Higher Education Consortium 2009)
Engages ‘community’ in the design Focuses on strengths Aligns with the goals and objectives of the program
This means the goals and objectives identified by the community based organizations in collaboration with other stakeholders – not just the goals of the funding agency
6/3/2014
4
Flexible Includes both ‘countable’ and
‘uncountable’ things Outcome evaluation tends to focus on
‘countables’ Comprehensive evaluation is more than testing
whether our assumptions about an intervention were correct Did doing ‘A’ result in ‘B’? Focusing on ‘outcomes’ alone without an
indepth analysis of ‘how’ the outcome occurred can lead to many problems, including unsuccessful replication
Example – replication of Pathways to Education model
Properly resourced and not rushed Evaluation that is useful will be implemented
early so that it can inform further program development/refinement
Evaluation should be used to learn how to improve a program that was developed from basic assumptions
Evaluation can be empowering to program participants if they are included in the designand implementation
6/3/2014
5
Valuing unintended outcomes
“I always thought that I was stupid, but now I know that I am smart”
What good evaluation isn’t… An add on and‘backward’ focused
but RATHER in tandem with program design and implementation and ‘forward’ thinking.
Weakness focused Developed in a vacuum
(narrowly defined pre-determined objectives)
Focused only on ‘countables’
6/3/2014
6
How governments can better respond
Work in collaboration with communities to come to an agreement on: What we are evaluating Why we are evaluating How we are evaluating Who will evaluate How we intend to use the evaluation
Holistic Programs call for Holistic evaluation
We tend to want to evaluate ‘one thing’ or ‘one program’ but is this effective?
It is difficult to resist looking for specific information that aligns with our priorities but what do we miss when we do this? And what about the priorities of program participants?
What do they want?
6/3/2014
7
Examples: Assessing ‘school readiness’ may be useful,
but how will this resolve the complex issues that face many children after they begin school? (eg. Poverty, racism)
Funding of one program in an organization with multiple programs meeting multiple/interconnected community needs
CEDA Pathways to Education Pathways Canada focus is on evaluation ‘countables’ credit accumulation graduate rates attendance
We also tend to look at the countables through a single lens, comparing to ‘the norm’ Students should be attending full day
everyday Students should graduate within four years
6/3/2014
8
Answering the Questions How to evaluate government
programs/investments/activities through a community lens;
How government evaluation processes can be more responsive to community needs;
How intergovernmental alignment of evaluation processes would benefit the community?
Government evaluation processes can be more responsive to community needs if they are designed in collaboration (honest/real) with community All ‘parties’ have to be able to be ‘upfront’
about their goals, objectives, needs and interests so that evaluation can be effectively designed and implemented
6/3/2014
9
Intergovernmental alignment of evaluation processes would benefit the community – as well as governments It would ensure that we are gathering all of
the information we need to develop holistic programs that meet multiple goals
If we want to “evaluate government programs/investments/activities through a community lens” then we need to engage the community in determining whether the program/investment/activities are guided by the priorities that they see in the community
6/3/2014
10
Acknowledging the power imbalance We like to talk about ‘partnerships’ but
community organizations are well aware that these are not ‘equal’ partnerships
The agency funding the evaluation will have influence on the direction evaluation takes
Funding agencies need to be open to ‘redefining’ evaluation questions and approaches to reflect the interests of community organizations
Reassessing where evaluation will lead us Part of evaluation through a community lens is
looking at our goals, objectives, expectations differently. Should we be changing our expectations to better
align with realities rather than trying to force a different reality? (High School graduation)
Evaluation may show intervention has improved ‘school readiness’, but if parents remain poor, disengaged, disempowered, what will be the long term impact?
6/3/2014
11
Open to other priorities We become uncomfortable when it
comes to evaluating other priorities and/or recognizing the importance of other factors-other goals
For example – Indigenous goals and objectives
The Four QuestionsThe Indigenous community has established that “success” (however defined) requires individuals to have a strong sense of:Who am I?Where did I come from?What is my purpose?Where am I going?
If the Indigenous community has identified this as critical to all that follows, should we not be trying to evaluate progress here? And if so, how?
6/3/2014
12
If we agree that we want transformative change, then we need to measure progressive in a way that is transformative Forward looking Strength based Community driven Empowering