Arizona advocacy

30
Evaluating Advocacy: Dilemmas, Tactics, and Methods Julia Coffman Center for Evaluation Innovation [email protected] October 7, 2011

description

Slides for Evaluating Advocacy Presentation

Transcript of Arizona advocacy

Page 1: Arizona advocacy

Evaluating Advocacy:Dilemmas, Tactics, and Methods

Julia CoffmanCenter for Evaluation Innovation

[email protected]

October 7, 2011

Page 2: Arizona advocacy

Three Questions

How is evaluating advocacy different?

What can we measure about advocacy?

How can we measure it?

Page 3: Arizona advocacy

How is evaluating advocacy different?

Page 4: Arizona advocacy

Challenge: The policy environment is complex, and that makes attribution hard.

Page 5: Arizona advocacy

What is my uniquecontribution?

Solution: If accountability is the purpose, demonstration of contribution

is expected, not attribution.

Page 6: Arizona advocacy

Shifting Politics Shifting Economics New Partners

Challenge: Advocacy strategies shift in response to the environment.

Page 7: Arizona advocacy

EV

AL

UA

TIO

N

ADVOCACY

Solution: Advocacy is a good opportunity to integrate or embed evaluation for learning

(not just accountability).

Page 8: Arizona advocacy

Challenge: Timeframes can be unpredictable.

Advocacy Timeframe

1 Year

Goal

Reporting Timeframe

1 Year 1 Year

10 Years5 Years1 Year

Page 9: Arizona advocacy

Goal

…progress…

Solution: Assess progress, not just the end result.

Page 10: Arizona advocacy

What can we measure about influencing?

Page 11: Arizona advocacy

Measure meaningful things. Don’t just count what is easy to quantify.

Page 12: Arizona advocacy

Have realistic expectations.

What are other

influencers doing?

What’s the opposition

doing? What are you doing and

who are you targeting?

What’s the political context?

Where is your issue in the policy

process?

Page 13: Arizona advocacy

Policy Influence

…progress…

Measure the changes made along the way, not just the end result.

INTERIM OUTCOMES

Page 14: Arizona advocacy

Interim outcomes are the changes you expect as you work toward your goal.

Think about the changes you will see in your audiences.

Page 15: Arizona advocacy

AUDIENCES

DECISION MAKERS

OU

TCO

MES

ACTI

ON

WIL

LAW

AREN

ESS

PUBLIC INFLUENCERS

WHO will change as a result of your work?

HOW will they change as a result of your work?

Use the framework to think about interim outcomes.

Page 16: Arizona advocacy

Where are your audiences?How far do you need to move them?

AUDIENCES

DECISION MAKERS

ACTI

ON

WIL

LAW

AREN

ESS

PUBLIC INFLUENCERS

Increase quality of child care

Child care providers

Parents of young

children

INTE

RIM

OU

TCO

MES

Legislators

Page 17: Arizona advocacy

Increased media coverage

Increased collaboration

among advocates

Increased willingness to act

Increased knowledge

Changed attitudes or beliefs

New and active advocatesIncreased salience

Reframing of the issue

New donors

Increased or diversified funding

Increased issue visibility or recognition

New and active high-profile champions

Activities and

Outputs

Policy Goals

Interim OutcomesAwareness

Will

Action

Increased capacity to act

Increased personal or collective

efficacy

Page 18: Arizona advocacy

How can we measure it?

Page 19: Arizona advocacy

Interviews

Surveys

Focus Groups

Polling

Traditional Evaluation Methods

Page 20: Arizona advocacy

System Mapping

Network Mapping

Advocacy Capacity Assessment

Critical Incident Timelines

Intense Period Debriefs

Media Scorecards

Research Panels

Crowdsourcing

Snapshot Surveys

Intercept Interviews

360-Degree Critical Incident Debriefs

Policy Maker Ratings

Media Tracking

ECCO Analysis

Policy Tracking

Non-Traditional Methods

Charting and Mapping

Survey and Interview

Debriefing and Rating

Media/Messaging Tracking

Bellwether Methodology

Champion Tracking

Page 21: Arizona advocacy

Bellwether Methodology

► Policymakers

► Administrators

► Other Advocates

► Funders

► Business

► Media

Bellwethers are:

“Influentials” in the public and private sectors whose positions require that they are politically informed and that they track a broad range of policy issues

Developed by Harvard Family Research Project

Page 22: Arizona advocacy

Bellwether Methodology

What are the top 3 Issues on the children’s (ages 0-18)

policy agenda?2005

(n=40)2008

(n=44)

1. K-12 education2. Health care3. Preschool

1. K-12 education2. Health care3. Preschool

Page 23: Arizona advocacy

Yes 84%

No 14%

Don't Know 2%

2008(n=44)

Can preschool help address the achievement gap?

How would you address the achievement gap?

2008 FULL SAMPLE (n=72) %

1. K-12 reform 67%

2. Education funding/resources 47%

3. Nonschool supports (health care, family involvement,after school)

32%

4. Supplemental school programs (ELL, literacy programs)

29%

5. Preschool/early care and education 21%

6. Other 13%

Bellwether Methodology

Page 24: Arizona advocacy

Very likely 9%

Somewhat likely 20%

Not very likely 50%

Not at all likely 20%

2008(n=44)

What is the likelihood that California will

increase preschool investments in the

next 3 years?

Bellwether Methodology

Page 25: Arizona advocacy

Policymaker Ratings

Support Influence Confidence

Ratings are completed for a whole governing body or defined group of policymakers

Developed by Harvard Family Research Project

Page 26: Arizona advocacy

Policymaker Ratings

35

71

15

3

35

55

23

7

0

10

20

30

40

50

60

70

80

Not at all supportive

Somewhat supportive

Supportive Extremely supportive

Year 1

Year 2

Developed by Harvard Family Research Project

Page 27: Arizona advocacy

Policymaker Ratings

Alameda

AlpineAmador

Butte

Calaveras

Colusa

Costa

Del Norte

El Dorado

Fresno

Glenn

Humboldt

Imperial

Inyo

Kern

Kings

Lake

Lassen

Los Angeles

Madera

Marin

Mariposa

Mendocino

Merced

Modoc

Mono

Monterey

Napa

Nevada

Orange

Placer

Plumas

Riverside

Sacramento

Benito

San Bernardino

San Diego

San FranciscoJoaquin

Obispo

Mateo

Barbara

Santa ClaraSanta

Shasta

Sierra

Siskiyou

Solano

Sonoma

Sutter

Tehama

Trinity

Tulare

Tuolumne

Ventura

Yolo

Yuba

Contra

San

San

SanLuis

San

Santa

Cruz

Stanislaus

Low Support

Medium Support

High Support

= Increase in Support

* Hypothetical Data

Developed by Harvard Family Research Project

Page 28: Arizona advocacy

Policymaker Ratings

Low Influence High Influence

Low Support

High Support

68 legislators 57% of the legislature

17 legislators 14% of the legislature

13 legislators 11% of the legislature

80% Democrat20% Republican

100% Assembly

85% Democrats15% Republican

75% Assembly25% Senate

50% Assembly50% Senate

67% Assembly33% Senate

49% Democrat51% Republican

64% Democrat36% Republican

22 legislators 18% of the legislatureLo Med Hi

Confidence Rating

70%5% 25%

Lo Med Hi

Confidence Rating

60% 40%

Lo Med Hi

Confidence Rating

100%

Lo Med Hi

Confidence Rating

40% 60%

Developed by Harvard Family Research Project

Page 29: Arizona advocacy

Wrap Up

Stay anchored in what you can reasonably change.

It’s okay to prioritize and focus on what is most important to assess.

Be creative—advocacy evaluation is an emerging field.

Page 30: Arizona advocacy

www.evaluationinnovation.org

Center for Evaluation Innovation