1 Chesapeake Bay Funders Network Program Evaluation Training Workshop OMG Center for Collaborative...
-
Upload
andrew-mccoy -
Category
Documents
-
view
217 -
download
0
Transcript of 1 Chesapeake Bay Funders Network Program Evaluation Training Workshop OMG Center for Collaborative...
1
Chesapeake Bay Funders Network
Program Evaluation
Training Workshop
OMG Center for Collaborative Learning
January 9-10, 2008
Evaluating Advocacy
2
Building an Evaluation Plan
I. Theory of Change
II. Outcomes
III. Indicators
IV. Methods
V. Putting Evaluation to Work for You
Same Building Blocks for Evaluating Advocacy
3
What is advocacy?
For our purposes, advocacy represents…
Strategies devised Actions taken Solutions proposed
…to inform or influence local, state, or federal decision making.
4
Evaluating Advocacy: How is it Different?
• Advocacy has always been considered hard to measure - less tangible
• Is evaluating advocacy different than evaluating traditional programs?
Yes and No....
5
Evaluating advocacy is different from evaluating traditional program work because of the nature of advocacy itself.
1. Advocacy strategies evolve over time; activities and outcomes can shift quickly
2. The policy process has many unpredicted and uncontrollable variables (windows of opportunity)
3. So many groups focus on an issue – hard to isolate attribution (Hence, contribution – is it plausible that your activities contributed to the end result?)
Evaluating Advocacy: How is it Different?
6
Evaluating Advocacy: How is it the same?
The same principles and approaches that we talked about earlier can be applied to evaluation of advocacy
1. You can develop a Theory of Change for advocacy
2. You need to determine outcomes and the indicators that will tell you whether the outcome has happened
3. You can use qualitative or quantitative approaches to collecting the data
7
Evaluating Advocacy: How is it the same?
4. Purpose is the same: Accountability – reporting on your results
Promotion – leveraging further interest in your work (other funders, volunteers, partners, etc..)
Inform Constituents/Stakeholders – results of their actions, learning
Decision-making – inform your work as you go, and reflection – what worked, what didn’t, what to do differently
8
What is the primary advocacy or policy change goal?
Public Awareness: Aims to increase public knowledge or salience of a particular problem or policy solution
Public Will: Aims to increase public willingness to act onbehalf of a particular policy issue
Political Will: Aims to increase policymaker willingness toact on behalf of a particular policy issue
*Slides 8-17 draw from “Using the Advocacy and Policy Change Composite Logic Model toArticulate an Advocacy Strategy or Theory of Change”, Julia Coffman, Harvard Family Research Project
Creating a Theory of Change for Advocacy
9
What contextual factors might affect your strategy’s success?
• Political Climate
• Economic Climate
• Social Climate
• Prior Experience with Advocacy
• Issue Competition
• Potential Partners/Competitors/Opponents
Creating a Theory of Change for Advocacy
10
Determining outcomes for advocacy-– TIMING MATTERS!
A Theory of Change for advocacy must have a realistic timeline– This type of work typically takes a long time
to yield outcomes– Short term, intermediate, long term
Creating a Theory of Change for Advocacy
11
Timing Matters: Short-term/Intermediate Outcomes
• Creating Partnerships or Alliances
• Collaboration and Alignment (include messaging)
• New Advocates
• New Champions (including policymakers)
• Media Coverage
Creating a Theory of Change for Advocacy
12
Timing Matters: Short-term/Intermediate Outcomes
• Issue Reframing
• Awareness
• Salience
• Attitudes or Beliefs
• Public Will
• Constituency or Support Base Growth
Creating a Theory of Change for Advocacy
13
Timing Matters: Longer Term Outcomes
• Policy Development
• Placement on the Policy Agenda
• Policy Adoption (passes through ordinance, ballot measure, legislation or legal agreement)
Creating a Theory of Change for Advocacy
14
Timing Matters: Longer Term Outcomes
• Policy Implementation (funding, resources, q.a.)
• Policy Monitoring and Evaluation
• Policy Maintenance
• Policy Blocking (opposition to a policy proposal that would go against your goals)
Creating a Theory of Change for Advocacy
15
Who is the audience?
• Most strategies will target multiple audiencesFor example: Elected officials, Voters, Media
Other potential audiences: Public administrators, Business, Community leaders, Courts, Specific constituencies
• Think about who needs to be doing the advocating and who needs to be hearing it to achieve the goal (levels of influence)
Creating a Theory of Change for Advocacy
16
What will it take to convince or move the audience?Activities needed to achieve the outcomes (Sequence)
Creating a Theory of Change for Advocacy
Policymaker/Candidate Education
New Champions Placement on the Policy Agenda
Demonstration Projects or Pilots
Awareness Policy Adoption
Constituency/Support Base Growth
17
I will know there is… When I see…
Advocacy: Outcomes and Indicators
New Champions High profile individuals speaking out on the issue
Placement on the Policy Agenda
Issue/idea appears on schedule for hearings
Awareness Target audience recognizes the issue/is familiar with the terms
Constituency/Support Growth
Increased number of people participate in advocacy or action on the issue
Policy Adoption Successful passing of the policy proposal through an ordinance, ballot measure, legislation or legal agreement
18
1. Focus Groups/Interviews/Surveys • Used to measure Awareness, Salience, Attitudes/Beliefs,
Public Will, Political Will, Constituency or Support Base Growth
• Specific techniques:1. “Bell weather Assessment”: Is the issue on people’s radar
screens? Semi-structured interviews with people who know what is going on in the relevant political realm
2. Rating Scales: Is there political salience/will for an issue? Interview top advocates working on an issue to a) Rate the legislators on receptivity/likelihood of voting in favor/against; b) Rate influence within the legislature; and c) Rate confidence in their own rating
Advocacy: Methods
19
2. Media Analysis – level of coverage, type of language used, increase in media hits following various actions (Lexis-Nexus on-line search engine for media mentions; services)
3. Growth in Partnerships/Alliances over time – see OMG Framework to assess partnership strength
4. Record Review to Assess Constituency/Support Base Growth; #s attending certain events, #s signing petitions, #s participating in list-serves, e-networks.
Advocacy: Methods
EXAMPLE: Fullojunk Watershed Alliance Advocacy
Strategic FocusImprove the environmental quality of the Fullojunk Watershed through local advocacy.
Long-term Outcomes
Passage of local ordinances to support comprehensive land use planning and reduction in storm water runoff
Short-term Outcomes
New champions and advocates (including Mayor)
Increased public awareness
Organizational visibility and recognition
Alliance Advocacy ActivitiesConduct issue/policy analysis research
Build relationships with decision makers and new administration staff
Lead grassroots organizing and mobilization
Assumptions
Assumptions
Assumptions
Contextual Analysis
There is a new mayor who is pro-environment
The Alliance has very little advocacy experience
Because there is a new mayor, a lot of advocacy organizations are clamoring to have their message heard, leading to issue competition