Www.fsg-impact.org Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG...

www.fsg-impact.org

Boston Geneva San Francisco Seattle

Beginning the Evaluation Journey with FSG

KCIC Boot Camp

March 24, 2010

Prepared for:

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors2

Today’s Discussion

Today’s Agenda

Things You Need to Do to Evaluate Your Project’s Progress and Outcomes

Articulate your Theory of Change and develop a project Logic Model

Develop an evaluation plan

Execute your evaluation plan

Participate in Knight’s evaluation process


Evaluation Overview

The KCIC Evaluation Has Three Overarching Objectives

• Knight’s 2 Goals:

– Foundations are engaged in helping meet community information needs

– Communities are more informed and engaged

Evaluate Progress

Inform and Improve Program Strategy

Promote Learning

• Inform the Knight Foundation’s program strategy for the Community Information Challenge and identifying opportunities for refinements over time

• Create data that promotes shared learning within the KCIC grantee community and the field of place-based foundations to increase understanding of how community information needs can be addressed effectively


What is Evaluation?

The systematic collection of information based on questions of critical importance to the organization or community:What do you want to know?Why do you want to know it?Where will you find what you need to know?What will it look like?How will you use what you learn?Who are the intended users of the information?

To make judgments, inform decisions, and take action

Is an ongoing process; should not be an “event”

Is best when implemented as a collaborative learning activity that builds an organization’s capacity

Definitions of Evaluation


Process

The Evaluation Process: Planned, Purposeful, Systematic

Develop Recommenda-

tions and Action Plans

Articulate Theory of Change & Develop

Program Logic Model

Focus the Evaluation

Collect DataAnalyze Data and Interpret

Findings

Determine Design & Data

Collection Methods

(numbers, words, pictures)

Des

ign

Imp

lem

enta

tio

n

5 4

321

6

Ref

ine

Str

ateg

y B

ased

on

E

valu

atio

n F

ind

ing

s

Communicating and Reporting


Instrumental use - findings reflects changes that can be observed.

Conceptual use - findings may lead to stakeholders having different kinds of conversations, greater insights into future decisions, and/or greater commitment to the program or initiative.

Political or Symbolic use - legitimate uses of the evaluation findings may be applied to lobbying for the program or initiative, securing new or additional funding, or communicating that the evaluation has taken place.

Uses of Evaluation

How Can We Use Evaluation?

Well designed evaluations will serve multiple purposes


Internal Stakeholders• Board members/Trustees• Senior Leaders/Executives• Program staff

External Stakeholders• Legislators/Policymakers• Project partners• The general public• The media• Researchers• Other funders• Journalism programs

Stakeholders

Different Stakeholders Will Have Different Uses for Evaluation

Evaluation Stakeholders

Identifying stakeholders has implications for:

• which and how many questions are asked

• what indicators will be of interest

• how findings will be communicated and reported

• how results will be used


Example

This Theory of Change illustrates at a basic level why SVCF is doing what it’s doing

• The community needs a regional plan to address climate change and sustainability

• There is a low awareness and understanding among residents of the intricacies and implications of regional development decisions

• Residents are not engaged in the planning process today

• Creation of SVCF’s Venture Fund (with investors)

• Development of a web-based information platform for citizens to learn about potential scenarios

• Planned convenings on land use and sustainable cities

• Awareness campaign (KQED)

• Interactive kiosks

• Citizens are more informed and aware of key issues

• Citizens are more engaged in planning processes

• Regional plan is developed with community input

What we are doing to address the issue – how we plan to solve the problem or create

the change (Activities)

What the issue is and why we are taking action

(Assumptions)

The change we hope to see if we are successful

(Outcomes)

Source: FSG interview; used with grantee’s permission

Envision Bay Area – Theory of Change


As You Think About Your Own KCIC Project Outcomes, Consider the Possibilities

Outcomes are…the SO WHAT?

• The short and/or long term changes, results, and impacts from implementing a project, program, or initiative.

Changes may be:

• positive or negative, singular or multiple

• in knowledge, skills, and/or attitudes

• in organizational policies, practices, capacity

• at community level in terms of behavior, values, and attitudes

Outcomes

What will be different in your community if your project is successful?


Types of Outcomes

Short term Outcomes• Achieved during program timeframe• Within program control• “Expect to see”

For example:• New knowledge• Changed opinion/values• Increased skills• Changed motivation/intent• Changed attitudes• Changed aspirations• New relationships/networks

Short term Outcomes

Adapted from: Innovation Network, Inc. (2005)


Types of Outcomes

Intermediate Outcomes• Achieved at the end/beyond program timeframe• Follow shorter-term outcomes• “Want to see”

For example:• Modified behavior• Changed policies• Changed practices• Changed social action• Changed decisions• Modified structures

Adapted from: Innovation Network, Inc. (2005)

Intermediate Outcomes


Longer-term Outcomes

• Achieved after program timeframe

• Outside direct program control

• “Hope to see”

For example:

• Changed human condition

• Changed civic condition

• Changed economic condition

• Changed environmental condition

Longer-term Outcomes

Types of Outcomes

Source: Innovation Network, Inc. (2005)


Start Building Your Evaluation Plan by Developing Your Own Theory of Change and Identifying Your Project Outcomes

1. From our interviews with each of you, we have attempted to visually depict your project’s theory of change. Review our version of your ToC and change, add/or revise as needed.

2. If your project were hugely successful in three years, what would your target audience be doing, thinking? (long term outcomes)

3. After one year, what would your target audience be doing, thinking? (short term outcomes)

4. What are some indicators that your project is achieving what you set out to do? (outputs)

In Practice

See online assignment form: http://bit.ly/9nREQl


Next Steps

Developing a Project Logic Model and Key Evaluation Questions (Webinar #1)

Determining Indicators of Online and Offline Behavior (Webinar #1)

Choosing an Evaluation Design and Data Collection Methods (Webinar #2)

Analyzing Quantitative and Qualitative Data (Webinar #2)

Strategies for Communicating and Reporting Evaluation Findings to Multiple Audiences (Webinar #3)

Developing an Evaluation Budget (Webinar #3)

Tips for Hiring an Evaluator (Webinar #3)

Next Steps in Designing Your Project’s Evaluation Plan

Throughout 2010 – FSG will provide up to 2 hours of additional evaluation-related technical assistance to each grantee

Www.fsg-impact.org Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG...

Documents

Transcript of Www.fsg-impact.org Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG...