Champions for Change Workshop EVALUATING COLLECTIVE IMPACT · EVALUATING COLLECTIVE IMPACT VS....

Champions for Change Workshop

EVALUATING COLLECTIVE IMPACT | FEBRUARY 2014

2

© 2014 FSG

“My question is: Are we making an impact?”

Source: The New Yorker

3

© 2014 FSG

EVALUATING COLLECTIVE IMPACT VS. SHARED MEASUREMENT

Evaluation

Shared measurement systems (SMS)

- a common set of indicators to

determine if, where, and for whom the

initiative is making progress

SMS can be both an input to evaluation (by providing data and/or

shaping evaluation questions) and an object of evaluation

Shared

Measurement

Systems

(SMS)

Evaluation - a range of activities that

involve the planned, purposeful, and

systematic collection of information

about the activities, characteristics, and

outcomes of a CI initiative

Collective Impact Evaluation Framework

Context

Long-term social or

environmental

impact goals

Impact Changes in Systems

and Behavior

Intermediate outcomes,

related to changes in the

way people, organizations,

institutions, and systems

behave

Time and Impact

CI Design & Implementation

Five key elements; initiative’s capacity

and learning culture; implementation

of programs, activities, and

campaigns, as part of the initiative’s

strategy

Capacity

CI learning culture

Changes in individual

behavior and professional

practice

Changes in systems

(funding, norms, policy)

Changes in lives

Developmental Formative evaluation Summative

Planning for Effective CI Evaluation Begins With Choosing Specific Areas of Focus

Context 1 Key

Potential evaluation

focus Continuous learning

Elements of context

CI Process

Outcomes & Indicators


CI /Partner Activities


2 Intermediate

Outcomes

Behavior Change


Systems Change


Impact

CI Impact Goals


4

3

Time and Impact

SMS (in operation) Early performance indicators

6

© 2014 FSG

Within Each Selected Focus Area, CI Partners Should Identify the Key Learning Questions they Seek to Answer

Intermediate Outcomes Impact

• To what extent has the CI initiative

achieved its ultimate outcomes?

• What has contributed to or hindered the

achievement of the CI initiative’s goals?

Context

• What are the cultural, socioeconomic,

and political factors that are influencing

the design and implementation of the CI

initiative?

1

3 4

CI Design & Implementation 2

• Backbone Infrastructure: To what

extent and in what ways is the backbone

infrastructure providing the leadership,

support, and guidance partners need to

do their work as planned?

• To what extent / in what ways are the

flows of philanthropic and public funding

shifting to support the goals of the CI

initiative?

• To what extent / in what ways are social

and cultural norms evolving in ways that

support the goals of the CI initiative?

Changes in Systems

Sample Learning Questions


Example 1: Outcomes and Indicators for CI Process, Backbone Infrastructure

Context 1 Key



Elements of context

CI Process





2 Intermediate

Outcomes

Behavior Change


Systems Change


Impact

CI Impact Goals


4

3

Time and Impact


8

© 2014 FSG

Example Outcomes and Indicators – Backbone Infrastructure

Sample Outcomes Sample Indicators

The backbone infrastructure

(BBI) effectively guides the CI

initiative’s vision and strategy

BBI builds and maintains hope and motivation to

achieve the initiative’s goals

BBI celebrates and disseminates achievements of CI

partners internally and externally

Partners look to the BBI and SC for initiative support,

strategic guidance and leadership

The backbone infrastructure

ensures alignment of existing

activities, and pursuit of new

opportunities, toward the

initiative’s goal

BBI provides project management support, including

monitoring progress toward goals and connecting

partners to discuss opportunities, challenges, gaps,

and overlaps.

BBI convenes partners and key external stakeholders

to ensure alignment of activities and pursue new

opportunities

Learning Question: To what extent / in what ways is the backbone providing the

leadership, support, and guidance partners need to do their work as planned?


Example 2: Outcomes and Indicators for Systems Change, Funding Flows

Context 1 Key



Elements of context

CI Process





2 Intermediate

Outcomes

Behavior Change


Systems Change


Impact

CI Impact Goals


4

3

Time and Impact


10

© 2014 FSG

Example Outcomes and Indicators – Funding Flows

Sample Outcomes Sample Indicators

Philanthropic funding in the

targeted issue area/system is

increasingly aligned with the

goals of the CI initiative

Overall philanthropic funding for the targeted issue

area or system has increased

New / Existing philanthropic resources are aligned

towards evidence-based strategies in the targeted

issue area/system

Philanthropic funding is increasingly designed to

allow for program innovation and experimentation in

the targeted issue area/system

Philanthropic and public funders

leverage funding for the targeted

issue area/system through

partnerships and collaborative

funding efforts

Grantmakers participate in funding collaboratives to

leverage resources toward the targeted issue

area/system (as relevant)

Philanthropic and public funders engage in public-

private partnerships to leverage resources toward the

targeted issue area/system

Learning Question: To what extent and in what ways are the flows of philanthropic and

public funding shifting to support the goals of the CI initiative?

Shared Measurement

12

© 2014 FSG

Shared Measurement Is a Critical Piece of Pursuing a Collective Impact Approach

Identifying common metrics for tracking progress toward a common

agenda across organizations, and providing scalable platforms to share data, discuss learnings, and improve strategy and action

Definition

Benefits of Using Shared Measurement

Source: Breakthroughs in Shared Measurement and Social Impact, FSG, 2009

Learning and Course Correction

Catalyzing Action

Improved Data Quality

Tracking Progress Toward a

Shared Goal

Enabling Coordination and

Collaboration

13

© 2014 FSG

There Are Three Phases to Developing a Shared Measurement System

Design Develop Deploy

1 2 3

• Shared vision for the

system and its relation to

broader goals, theory of

change or roadmap

• View of current state of

knowledge and data

• Governance and

organization for

structured participation

• Identification of metrics,

data collection approach

(incl. confidentiality/

transparency

• Development of web-

based platform and

data collection tools

• Refinement and

testing of platform

and tools

• Staffing for data

management and

synthesis

• Learning forums and

continuous

improvement

• Ongoing infrastructure

support

• Improve system based

on ongoing evaluation

of usability and impact

Source: FSG Analysis

14

© 2014 FSG

Several Challenges Can Occur When Developing and Implementing Shared Measurement Systems

Coming to agreement on common outcomes and indicators

Concerns about relative performance / comparative measurement across

providers

Limited capacity (time and skill) for measurement and data analysis within

organizations

Alignment among funders to ask for the common measures as part of their

reporting requirements

Time and cost of developing and maintaining a system (human capital and

technology)

15

© 2014 FSG

There Are a Number of “Tips and Tricks” to Bear in Mind When Developing Shared Measures

Collecting and

Presenting Data

• Specific, time-bound goals; report progress relative to targets

• Data on whole populations, where possible

• Numbers as well as percentages make goals more tangible

Identifying

Indicators

• “Top-level” indicators limited to a manageable number (~15),

with additional contributing indicators if needed

• A set of criteria to guide the identification and prioritization of

potential indicators

Leveraging

Existing Efforts

and Expertise

• Voluntary team of data experts to advise on the design,

development, and deployment of a SMS

• Crosswalk of what partners are already measuring

• Leverage existing indicators adopted by relevant efforts at the

local, provincial, or federal level

16

© 2014 FSG

The Road Map Project Uses Indicators to Guide its Progress toward Doubling the Number of College Graduates in the Seattle Area

“The Road Map Project’s goal is to double the number of

students in South King County and South Seattle who are

on track to graduate from college or earn a career credential

by 2020. We are committed to nothing less than closing the

unacceptable achievement gaps for low income students

and children of color, and increasing achievement for all

students from cradle to college and career.”

17

© 2014 FSG

Work Groups Used a Multi-Step Process to Identify, Vet and Refine the Road Map Indicators

• Used criteria to identify and prioritize indicators

• Work group chairs calibrated indicators across continuum

• Short-list of top-level indicators for time-bound targets; also

supporting indicators

• Four work groups

• Identified indicators of student success for their part of the

continuum

• Baseline and annual reports to track progress on all

indicators

• Data experts advise on indicators over time so the list can

evolve

Source: FSG Analysis

Identifying

Indicators

Tracking

Progress and

Refining over

Time

Developing

Work Groups

18

© 2014 FSG

• Indicator must be valid measure linked to ultimate goal of postsecondary attainment and/or four sub-

goals

• Indicator must be easily understandable to local stakeholders

• Data must be produced by trusted source

• Priority given to including indicators comparable across school districts (or neighborhoods), and

having ability to be compared

• All or most indicators must be affordable to gather and report

• Data should be available consistently over time (preferably on annual basis or more frequently),

should be recent (2008 or later), and easily disaggregated by county, City / S. King County

community, and school (or neighborhood)

• Priority given to data that can be disaggregated by ethnicity, socio-economic status, ELL, and gender

• Trend data should be provided over at least 3-year period (beginning with or including 2007)

• Each indicator should be able to be influenced to significant degree by local action, and be useful in

daily work of this Working Group, other educators and organizations that are working to improve

education

Source: Based on criteria identified by Cincinnati’s Strive Partnership

Work Groups Were Given a Set of Criteria to Guide

Indicator Selection

19

© 2014 FSG

From this Process, Top Level Indicators Were Identified, Targets Were Set, and Progress Is Tracked Over Time

Healthy and

ready for

Kindergarten

Supported and

successful in

school

Graduate from

high school

college and

career-ready

Earn a college

degree or career

credential

• % of students proficient in:

- 3rd grade reading

- 4th grade math

- 5th grade science

- 6th grade reading

- 7th grade math

- 8th grade science

• % students triggering Early

Warning indicators

• % of students who

graduate high school

on-time

• % of graduating high

school students meeting

minimum requirements

to apply to a

Washington State 4-

year college

• % of students at

community and technical

colleges enrolling in pre-

college coursework

• % of students who

enroll in

postsecondary

education by age 24

• % of students

continuing past the

first year of

postsecondary

• % students who earn a

post-secondary

credential by age 24

• % of children ready to

succeed in school by

kindergarten

Readiness Attainment Achievement

Source: Road Map Project, http://www.roadmapproject.org/

22

© 2014 FSG

THANK YOU!

• Thank you for being part of the conversation today

• For additional guidance on this topic, see resources on the Collective

Impact Forum website, and look for FSG’s Guide to Evaluating

Collective Impact, Coming in Spring 2014

Goals of the Guide

Illustrate the general process by which CI initiatives address complex

problems 1

Explore the ways in which evaluation and learning support CI success 2

Answer common questions about planning for and implementing

evaluation activities 3

collectiveimpactforum.org

Champions for Change Workshop EVALUATING COLLECTIVE IMPACT · EVALUATING COLLECTIVE IMPACT VS....

Documents

Transcript of Champions for Change Workshop EVALUATING COLLECTIVE IMPACT · EVALUATING COLLECTIVE IMPACT VS....