Why evaluate your advocacy? And how - School of Social … › sites › socwel.ku.edu ›...
Transcript of Why evaluate your advocacy? And how - School of Social … › sites › socwel.ku.edu ›...
Why evaluate your advocacy? And how?
Identifying Indicators, Implementing Measures, Assessing Impact
Why evaluate your advocacy—from the start
• Assess implementation of your strategies• Are you doing what you set out to do?
• Test your theory of change• Does changing X really change Y, like you thought it would?
• If not, what might be other routes to your objectives?
• Know when you’re on the right track• So that you can adjust—in real time—if not
• Communicate with funders• And those you wish would invest
PHASES OF ADVOCACY EVALUATION
1• FRAME the evaluation
• Why are you evaluating?
2• ENGAGE important partners
• Who else needs to think alongside you?
3• DEFINE what you are evaluating
• What is the intervention? How is it understood to work? How do you define ‘success’?
4• DESCRIBE what happened
• Activities, outputs, outcomes, context
5• Articulate CAUSATION for the observed outcomes, including accounting for
unknowns in the causal chain
6• ASSESS the overall success
• What lessons do you want to carry forward?
AUDIENCES
OU
TCO
MES AC
TIO
NW
ILL
AW
AR
ENES
S Voter Outreach
Public EducationPolicymaker Education
Influencer Education
Political Will Campaigns
Litigation
Media Advocacy
Regulatory Feedback
Public Forums
Champion Development
Model Legislation
Policy Analysis/Research
Demonstration Programs
PUBLIC INFLUENCERS
Using an Advocacy Framework to Guide your Evaluation
Community Mobilization
Coalition BuildingCommunity Organizing
Public Will CampaignsCommunications and Messaging
Leadership DevelopmentAdvocacy Capacity Building
Public Awareness Campaigns
Public Polling
Lobbying
DECISION MAKERS
Modified from Coffman & Beer, 2015: http://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework.pdf
AUDIENCES
DECISION MAKERS
OU
TCO
MES
AC
TIO
NW
ILL
AW
AR
ENES
S Voter Outreach
Public Education Policymaker EducationInfluencer Education
Political Will CampaignsMedia Advocacy
Regulatory Feedback
Public Forums
Champion Development
Model Legislation
Policy Analysis/Research
Demonstration Programs
PUBLIC INFLUENCERS
Community Mobilization
Coalition Building
Community Organizing
Public Will Campaigns
Communications and Messaging
Leadership Development
Advocacy Capacity Building
Public Awareness Campaigns
Public Polling
Awareness
SalienceIssue Reframing
Public Will Political WillMedia Coverage
Attitudes & Beliefs
New Advocates New Champions
Policy Change
New Donors
Collaboration & Alignment
More or Diversified Funding
Constituency Growth
Organizational Visibility and RecognitionAdvocacy Capacity
What outcomes could tell whether you’re hitting the mark?
LitigationLobbying
AUDIENCES
DECISION MAKERS
OU
TCO
MES
AC
TIO
NW
ILL
AW
AR
ENES
SRegulatory Feedback
Public polling
Policymaker ratings
PUBLIC INFLUENCERS
Event observation
Intense period debriefsPower mapping
Public pollingFocus groups, surveys
Polling, surveys, focus groups
Bellwether interviewsMedia tracking
Constituent action tracking
Policy tracking
Partnership Analysis
Network mapping
Donor tracking
Blog analysis, social media traffic
What tools measure these outcomes?
Champion tracking
DECISIONMAKERS
Framing penetration analysis
The Best Evaluation is the One you Use• Mobilization, Constituency
Growth, Public Will
• Political Will, Champion Development
• Attitudes, Beliefs, Awareness, Salience, Alignment
• Advocacy Capacity
• Policy Change
• Salience, Visibility, Media Coverage
• Media Coverage, Framing
• Organization Records (attendance, action response)
• Policymaker Ratings
• Surveys (public, partners)
• Validated assessments + observation/reflection on practices
• Intense Period Debriefs and Bellwether Interviews
• Tracking/Content analysis
• Media analysis
IF it really measures what you want to know
• If you want to assess implementation, measure inputs and outputs.
• If you want people to do something different, measure action.
• If you think that doing differently will require changes in thinking, measure awareness. Just don’t expect strategies aimed at awareness to automatically induce action.
Measure effects on the targets toward which you actually aimed; otherwise, you may undercount your impact, because you’re just
not looking at the right people.
The U of Good Advocacy Evaluation
• While we tend to spend the most time discussing instruments and data collection, the best advocacy evaluation is U-shaped, investing the most energy in the first and final phases:
1. Considering your questions and how to approach answering them
2. Making sense of your findings and figuring out how to incorporate what you learn into your advocacy, moving forward
THOUGHT EXERCISE
What questions might you develop to evaluate advocacy within these theories of change? What indicators could you develop?
Where could you get the data needed to answer those questions?
• How much positive coverage?
• How much public support? Among whom?
• How much political will? For what? Among whom?
• Do grantees and allies have shared definition?
• Have you increased organizational capacity?
• Is a diverse base represented?
• How many, who, and toward what end, have you mobilized a constituency?
• Have necessary volunteers and staff been recruited?
• What relationships have been built with targeted journalists?
• How aware are bicycle riders of helmet safety?
• What is rate of traumatic brain injury from bicycle riding (baseline and at post-intervention points)?
AND WHAT WOULD YOU DO WITH WHAT
YOU LEARNED?
A Learning Culture for Advocacy Evaluation
• What practices do you embrace to check progress toward your goals?
• What do you ask—and seek to answer—for your own purposes, without inducement from external stakeholders?
• Who is involved in evaluation for learning? Role models among senior leaders? Substantive engagement by frontline staff?
• Are you prepared to grapple with ‘failure’? Do people feel safe learning ‘out loud’?
Learning Organizations…
• Invest resources in evaluation capacity (human and technological; internal and external)
• Embed learning into organizational routines (e.g. staff meetings)
• Signal priority for learning AND real-time response
• Model transparency
• Interrogate and display data through multiple lenses, platforms, and formats
Derrick-Mills, T., Winkler, M.K., Healy, O.& Greenberg. E.(2015). A Resource Guide for Head Start Programs: Moving beyond a Culture of Compliance to a Culture of Continuous Improvement. Washington, DC: US Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation. http://www.acf.hhs.gov/opre/resource/a-resource-guide-for-head-start-programs-moving-beyond-a-culture-of-compliance-to-a-culture-of-continuous-improvement.
Shifting to Learning Culture
Winkler & Fyffe (2016). Strategies for Cultivating an Organizational Learning Culture. Urban Institute. Available at: https://www.urban.org/sites/default/files/publication/86191/strategies_for_cultivating_an_organizational_learning_culture_3.pdf.
EXAMPLES OF ADVOCACY
EVALUATION TOOLS
There is no ‘perfect’ tool—you want a good fit with what you’re actually trying to measure, alignment with your other operations, and unobtrusive enough that you’ll really use it
Policymaker Ratings
Uses of Policymaker Ratings
• Identifying champions• Who should be with you? How can they show it?
• Isolating effective neutralization and persuasion approaches• What is moving people?
• Testing sincerity and transparency of partner relationships• Who’s sharing what they know?
• Identifying issues ready to ‘tip’• Where are the stars aligning?
Intense Period Debrief
• Often, outside evaluators engage advocates in evaluative inquiry shortly after a policy window, special event, or other intense period of action occurs.
• With training and practice, advocates can facilitate intense period debriefs for themselves, although it can be difficult to get the ‘bandwidth’ needed until longer after the period has passed.
Framing an Intense Period Debrief
• What events triggered this intense period?
• How was the organization’s response determined? How was that decision communicated?
• Which elements of the response worked well? Which could have been improved?
• What was the outcome of the intense period? – Was the immediate aftermath positive or negative?
– What are the anticipated longer-term effects?
• What challenges did we face? Which were unanticipated? What do we wish we had known?
• What insights will we take away for next time?
Uses of Intense Period Debriefs• Reevaluating historic tactics/events
• Avoiding ‘autopilot’, born of scarcity of time
• Capturing reflections during legislative cycles or other predictably unmanageable epochs
• Systematizing informal reflections and making scheduled meetings more substantive• Increasing the likelihood that you learn what you should
and use what you learn
• Reducing the personalization of shared reflection; emphasis is on the intensity of the period, not any individual’s actions/inactions
Debrief of RDP Lobby Day
Bellwether Interviews
• Evaluators or other outsiders conduct structured interviews with influential people whose positions require that they track a broad range of policy issues.
• Ideally, the same is diverse, representing both the public and private sectors. At least part of the bellwether should lack direct connection to or expertise in the policy issue of interest.
• While formal evaluation experience is not necessarily imperative, bellwether interviews cannot be conducted effectively by an organization itself. Further, bellwethers’ consent to be interviewed should not highlight the policy issue of interest.
Questions for a Bellwether Interview1. What three issues do you think are at the top of the policy
agenda?
2. Considering the state's current educational, social, and political context, do you think the state should adopt [the policy] now or in the near future?
3. Looking ahead, how likely do you think it is that [the policy] will be established in the next 5 years?
4. Currently, what individuals, constituencies, or groups do you see as the main advocates for [the policy]? Who do you see as the main opponents?
5. If [the policy] is established, what issues do you think the state needs to be most concerned about related to its implementation?
Use of Bellwether Interviews• Testing new markets/geographies/issues
• What needs to change, for you to move into this space?• Assessing messages
• What works? How much have your message efforts ‘broken through’?
• Gauging relative salience among slate of issues• How much are we ‘on the radar’? How is the issue positioned on
the policy agenda?
• Assessing political will• What do people say about your issue when you’re not the one
asking?
• Forecasting the likelihood of future policy – What do people in the business of knowing think is coming next?
MBP Bellwether Interviews
Framework for Media Analysis
1. Is this earned, social, or opinion media? (Which do you value more?)
2. To what extent is our framing used in this coverage? (can rate 1-5)
3. How are our opponents’ arguments framed? Who represents them, and how well have we prepared for that opposition?
4. Whose voices are included? Whose perspectives are ignored?
5. To what extent are ‘unlikely’ stakeholders included?
6. Whose images are included? What does the unspoken coverage ‘say’?
7. Who initiated the coverage? (What did we have to do to get it?)
8. Has this coverage been ‘echoed’ by other outlets?
Tools for Social Media Analysis
• Hootsuite
• TweetReach
• Buzzsumo
• Twazzup
• Boardreader
• Mention
• Followerwonk
Uses of Media Analysis
• Assessing framing
• How much are your messages permeating?
• Identifying promising outlets/conduits
• Who is carrying your messages?
• Gauging prominence/salience
• How much are you on the agenda?
• Identifying allies and opponents
• Who is with you? And against you?
Tactical Dashboards• Align definitions of success across the organization
• Encourage dialogue about progress toward goals
• Facilitate timely identification of successes and challenges
• Ground decisions in concrete data, presented in user-friendly visual format
• Illuminate relationships between different activities
• Create a snapshot of current status as well as trends over time
• Highlight out-of-the-ordinary results
• Not just cool pictures, the indicators you display should:– Represent business model drivers
– Reflect progress toward intended outcomes
– Guide priorities and decisions (“what gets measured gets done”)
– Be limited to a number that can realistically be monitored and periodically reassessed
Use of Tactical Dashboards
• Engaging Board members
• “This is what you should pay attention to!”
• Facilitating supervision
• “This is what I’m looking for.”
• Focusing attention on priority areas
• “We are focused on moving this.”
• Cultivate learning orientation within organizational operations
Other Tools and How You Can Use Them
• Focus groups with advocates/donors
• What do they care about? Why do they support you? What roles do they want to play?
• Advocacy capacity assessments with coalitions, partners, members
• Where are you strong? Where do you need to grow?
• A/B testing of messages
• Which one moves people more? Where are different constituencies differently motivated?
• Social network analysis
• Who is connected, and to whom, and how?
• Cost-effectiveness
• How much did you spend to get outcome X? How do you assess whether it was ‘worth it’?
Advocacy Capacity Assessments
Free Advocacy Capacity Assessment from Bolder Advocacy
http://bolderadvocacy.org/tools-for-effective-advocacy/advocacy-capacity-tool
Getting Started
• Let your evaluation be guided by your questions: what do you need to understand in order to make the change you want to see?
• Figure out what you most want to evaluate; trying to do everything at once will overwhelm you.
• Budget for evaluation, but don’t assume that you need to hire an external evaluator.
• Reach out for help—utilize developed instruments, find thought partners, build on others’ learning.
Resources• Alliance for Justice Advocacy Capacity Tool (http://bolderadvocacy.org/tools-for-effective-
advocacy/advocacy-capacity-tool)
• Center for Evaluation Innovation (http://www.evaluationinnovation.org/)
• Aspen Institute Advocacy Progress Planner (http://planning.continuousprogress.org/)
• Innovation Network Learning Center (http://www.innonet.org/resources/)
• Stanford Social Innovation Review (“Elusive Craft of Evaluating Advocacy” http://www.ssireview.org/articles/entry/the_elusive_craft_of_evaluating_advocacy/)
• Continuous Progress: Better Advocacy Through Evaluation (http://dp.continuousprogress.org/)
• Users’ Guide to Advocacy Evaluation (http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-evaluation-planning)
• Monitoring and Evaluation of Policy Influence and Advocacy (https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8928.pdf)
• Readiness for Organizational Learning and Evaluation self-assessment (https://www.fsg.org/tools-and-resources/readiness-organizational-learning-and-evaluation-instrument-role#download-area)
• Organizational Learning Self-Assessment (https://theonn.ca/wp-content/uploads/2018/05/Self-Assessment-Tool-DRAFT.pdf)
• Unique Methods in Advocacy Evaluation (http://www.pointk.org/resources/files/Unique_Methods_Brief.pdf)
CENTER FOR COMMUNITY ENGAGEMENT AND
COLLABORATION
Melinda Lewis, LMSW, Associate Director, CCEC
University of Kansas School of Social Welfare
(785) 864-1047