Goals, Outcomes and Program Evaluation

Community Memorial Foundation

March 5, 2014

Agenda

• Introductions

• Logic Models

• Goals, Outcomes, Baselines and Results

• Implementation and Outcomes Evaluation

• Q and A

Session Objectives

• To define terms used in CMF responsive grants application regarding program outcomes and evaluation

• To introduce tools and frameworks to facilitate the learning of program evaluation

Presenters

• Tom Fuechtmann

Program Officer, Community Memorial Foundation

tfuechtmann@cmfdn.org

• Jon Korfmacher, PhDAssociate Professor, Erikson Institute

jkorfmacher@Erikson.edu

Where Program Evaluation is helpful

• Understand, verify or increase the impact of services on clients

• Improve delivery mechanisms to be more efficient and less costly

• Verify that you're doing what you think you're doing

• Facilitate management really thinking about what their program is all about, including goals, how it meets goals and knowing if it has met its goals or not.

Where Program Evaluation is helpful

• Produce data or verify results that can be used for public relations and promoting services in the community.

• Produce valid comparisons between programs to decide which should be retained, e.g., in the face of pending budget cuts.

• Fully examine and describe effective programs for duplication elsewhere

Logic Models

A way to summarize how your program works: a “picture” of your program.

Note: There is no one “correct” way to develop a logic model

* Logic models must be submitted for requests over $25,000

How is a logic model used?

• Logic Models have multiple functions: – Program planning

– Program management

– Evaluation

– Communication• Within your agency• With people outside your agency

Logic Model Components

Outcomes:Short-term

Assumptions:What is necessary for this to happen?

External Factors:What else may impact outcomes?

Outcomes:Medium

Outcomes:Long-term

Outputs

Activities(What we do)

Participants(Who we serve)

Inputs(What we invest)

(Expect) (Want) (Hope)

• Resources dedicated to or consumed by the project– Staff or volunteer resources– Funding– Equipment– Materials or other resources

Inputs(What we invest)

Logic Model Components

Logic Model Components

Outputs

Activities(What we do)

Participants(Who we serve)

Inputs(What we invest)

Sometimesdescribed as activities acceptedby participantsEx: 10-week parent training session for teen parents

Ex: Teen parents attend at least 7/10 weeks of parent training

Logic Model Components

Outcomes:Short-term

Outcomes:Medium

Outcomes:Long-term

Outputs

Activities(What we do)

Participants(Who we serve)

Inputs(What we invest)

(Expect) (Want) (Hope)

• What changes do we expect to see right away?• What changes would we want to see after that? • What changes would we hope to see after that?

Ex: College HIV prevention

Short Term

College students have knowledge of how HIV/AIDS is spread

Intermediate

College students practice safe sex

Long Term

Lowered incidences of HIV cases on college campuses

Closer in Time

Easier to Measure

More attributable to Project

More distant in time

Harder to measure

Less attributable to project

Congratulations!You now have a logic model!

(Now what?)

Remember…

1. Logic models are working documents• Responsive programs need evolving models• They can and should be open to change

2. The process is as important as the product

3. Your logic model is only the first step of an evaluation

CMF Application

Application continued

Application continued

• Goal: What you hope to achieve with this program (ex. To Reduce Homelessness)

• Outcomes: Changes that will occur as a result of your work (short term, intermediate, long term). Ex: 20% of those receiving services will be in permanent housing 6 months after graduating from program.

• Baseline: What is the current level of care? Ex. Only 2% are in permanent housing after a period of 6 months.

Evaluating Outcomes: What changes have occurred?

• Outcomes: the changes you expect to see as a result of your work

• Indicators: the specific way you measure an outcome. In other words, “How will I know it?”

Outcome Examples

Short TermWhat do we Expect to

Occur?

IntermediateWhat do we Want to

occur?

Long TermWhat do we Hope will

occur?

Seniors increase knowledge of food contamination risks

Seniors practice safe food handling

Seniors experience lowered incidence of food borne illness

Participants learn job-seeking skills

Teen mothers report negative attitudes towards smoking while pregnant

Outcomes: Common types of Indicators

• New knowledge

• Increased skills

• Changed attitudes, opinions or values

• Changed motivation or aspirations

• Changed decisions

• Modified behavior

• Changed policies

• Changed conditions

Ex: HIV/AIDS Prevention

OutcomeIntermediate: Students

practice safe sex

Long Term: Number HIV/AIDS cases decrease on college campus

IndicatorPoor: Number of college

students in program over time.

Better: Percent students who talk about illness with friends

Strong: Percent students reporting regular condom use

Evaluating Outcomes: Indicator Statements

Elements of a Strong Statement:– How much?

– Who?

– What ?

– When?

Evaluating Outcomes: Indicator Statements

Example:

“ 75% of program participants find part-time employment within six months

of enrollment.”But…

Have a rationale for the indicator

(e.g., why 75%?)

Collecting Evaluation Information

Questions to Ask Yourself– What resources do you have?– Quantitative or Qualitative?– Standardized or non-standardized?

What makes the most sense for your program and the outcome in question?

CMF and Evaluation

Outcome Objectives and Program Review

– What is the impact and how will this move CMF’s mission forward?

– What are the chances those results will be achieved?

– Given the opportunities before us, is this the best use of the Foundation’s assets?

CMF and Evaluation (continued)

• Grantees as Partners

• Trust

• Open Dialogue

• A Learning Foundation

Questions and Answers

Additional Resources

• Innovation Network www.innonet.org

• Free Management Library www.managementhelp.org/evaluatn/fnl_eval.htm

• University of Wisconsin- Extension (Logic Model) http://www1.uwex.edu/ces/lmcourse/.