Best Practices in Nonprofit Impact Measurement , CNM

Best Practices inNP Impact Measurement

Presented by Charlotte Keany, Director of Consulting, and Mary Jones, Consultant

Workshop Objectives

Introduction to an outcomes based program evaluation process

Learn how to develop a logic model

Learn how logic models can be used in program planning and evaluation

Outcomes Based EvaluationWhat is it?

A systematic process to obtain

information on an organization’s activities, its

impacts and the effectiveness of its work, so that it can improve its activities and

describe its accomplishments.Paul W. Mattessich, The Manager’s Guide to Program Evaluation.

Outcomes Based EvaluationWhy is it important?

Determine whether program goals have been met

Refine programs to improve overall effectiveness

Enhance ability to communicate results

Enhance promotion and marketing of a program to the public

Allocate dollars more efficiently

“If you don’t know where you are going, how are you gonna

know when you get there?

Yogi Berra

Outcomes Based Evaluation Steps

Continuous Process

Program Improvement

What is a Logic Model?

A logic model is a framework that helps you design results-based programs.

It is a visual representation of how your program works.

It includes what you put into your program, what you do and what you plan to achieve.

Tells your story and the difference you are making in

the community

Strengthens your case for program investment

Demonstrates accountability to stakeholders

Builds understanding and promotes consensus about what the program is and how it works

Helps with planning, evaluation, implementation and

communications

Why use a Logic Model?

Logic Model Components

Inputs

Activities Outputs Outcomes

Your Planned Work Your Intended Results

INPUTS are materials and resources a program uses in its activities or processes to serve clients.

Elements of a Logic Model


Inputs

•Staff

•Volunteers

•Facilities

•Equipment

•Curricula

•Money

Activities Outputs Outcomes

Inputs are resources a program uses to achieve program objectives.

ACTIVITIES are what you do with your resources. The programs and services are designed to meet your clients’ needs and fulfill your mission and vision.



Inputs

•Staff

•Volunteers

•Facilities

•Equipment

•Curricula

•Money

Activities

•Feed & shelter homeless families

•Provide job training

•Mentor youth

Outputs Outcomes

Inputs are resources a programuses to achieve program objectives.

Activities are what you do with your resources. The programs and services are designed to meet your clients’ needs

and fulfill your mission and vision.

OUTPUTS are units of service regarding your program.



Inputs

•Staff

•Volunteers

•Facilities

•Equipment

•Curricula

•Money

Activities



•Mentor youth

Outputs

•Number of meals served

•Number of counseling sessions conducted

•Number of youth mentored

Outcomes

Inputs are resources a program uses to achieve program objectives.

Activities are what a program does with its inputs; the services it provides to fulfill its mission

Outputs are products of a program’s activities

OUTCOMES are the actual impact benefits for participants during or after their involvement with a program



Inputs

•Staff

•Volunteers

•Facilities

•Equipment

•Curricula

•Money

Activities

•Feed & sheltering homeless families


•Mentor youth

Outputs




Outcomes

•New Knowledge

•Increased skills

•Changed attitudes or values

•Altered status

Levels of Outcomes

Short-term outcomes – The most immediate benefits or changes participants experience

Intermediate outcomes – Links initial outcomes to the longer-term desired outcomes

Long-term Impact– The ultimate impact a participant will enjoy as a result of participating in a program or services of an organization.

Teen Mother Parenting Education Program

Teen Mother Parenting Education

Inputs

•Pregnant teens

•MSW Program Manager

•RN Instructor

•Education Manuals

•Videos

•Money

Activities

•Parenting classes; two times a week for one hour

•Content: infant nutrition, develop-ment, safety, caretaking

Outputs

•# pregnant teens served

•# high schools partici-pating

•# manuals distributed

Initial Outcomes

•Teen mothers are knowledge-able about prenatal nutrition and health guidelines

Intermediate

Outcomes

•Teen mother follow proper nutrition and health guidelines

•Teen mothers deliver healthy babies

Long-Term Outcomes

•Babies achieve 12 month milestones for physical, motor, verbal, and social development

Follow the Chain of Reasoning

To have a successful, sustainable program . . .

Certain resources/inputs are needed to operate your program.

If you have access to them, then you accomplish your planned activities.

If you accomplish your planned activities, then, you will deliver the products and/or services you intended (outputs).

If you accomplish your planned activities to the extent intended, then your participants will benefit in specific, measurable ways (outcomes).

If these benefits to participants are achieved, then changes in organizations, communities, or systems will occur with positive impact.

The Conceptual Chain

Inputs

•Staff

•Volunteers

•Facilities

•Equipment

•Curricula

•Money

Activities



•Mentor youth

Outputs




Initial Outcomes

Changes in:•New Knowledge

•Increased skills


Intermediate

Outcomes

Changes in behavior resulting from:•New Knowledge

•Increased skills


Long-Term Outcomes

Changes resulting in:•Altered status

Teen Smoking Reduction Program

Inputs

•Staff

•Instruction Modules

•Volunteers

•Training of Volunteers

Activities

•One Session Lecture

•Match with volunteer mentor for counseling

Outputs

•Number of youth in lecture

•Number of youth matched with mentors

Initial Outcomes

•Increased knowledge of risks about smoking

•Increased support from mentor to quit smoking

Intermediate

Outcomes

•Youth quit smoking

Long-Term Outcomes

•Youth quit smoking for more than one year

•Less illness

•Longer life expectancy

•Lowered health cost


Program Improvement

Continuous Process

Step 2: Develop Outcome Indicators

Indicators are specific, observable, measurable characteristics or changes that will represent achievement of the outcome

Indicators are specific statistics (e.g., number/percent) the program will calculate to summarize its level of achievement

Identify Outcome Indicators

Indicators should be: Direct Meaningful Useful Practical to collect Quantitative

Multiple Indicators

Teen Mother Parenting Education Outcome Indicators

Outcomes Indicators Initial Outcome: Teens are knowledgeable of prenatal nutrition and health guidelines

85% of teen mothers are able to identify food items that are a good source of major dietary requirements

Intermediate outcomes: Teens follow proper nutrition and health guidelines

70% of teen mothers are within the proper ranges for prenatal weight gain

95% of teen mothers do not smoke

95% of teen mother eat at least 4 calcium servings and one of each of the other nutritional groups each day

85% of teen mothers take a prenatal vitamin each day

Long Term Outcome: Teens deliver healthy babies

80% of newborns weigh at least 5.5 pounds and score 7 or above on the Apgar scale


Continuous Process

Program Improvement

Step 3: Collect Data

Identify data sources for your indicators

Design data collection methods

Pretest your data collection instruments and proceduresDocument your procedure

Data Collection Procedures

What data is collected?

How will they collect it? Who collects it? When will they collect it? What do they do with it?

Data Collection Methods:

Interviews Focus Groups Surveys Observation Document review Others (?)

Keep in Mind

20% of the effort

generates

80% of the results

Outcome Measurement Framework

Program Name:

Outcomes Indicators Data Source Data Collection Method

Teen Mother Parenting EducationMeasurement Framework

Program Name: Teen Mother Parenting Education Outcomes Indicators Data Source Data Collection Method

Teens are knowledgeable of prenatal nutrition and health guidelines

85% of teen mothers are able to identify food items that are a good source of major dietary requirements

Participants Self-administered survey after second week in the program

Teens follow proper nutrition and health guidelines

70% of teen mothers are within the proper range for prenatal weight gain

School scales Weekly weigh-ins

95% of teen mothers do not smoke

Participants

Self-report on daily checklist

Teachers Observation reported on weekly record

95% of teen mothers eat at least 4 calcium servings and one of each of the other nutritional groups each day

Participants "Healthy Baby" checklist for recording daily food intake

85% of teen mothers take a prenatal vitamin each day

Participants "Healthy Baby" checklist for recording daily food intake

Teens deliver healthy babies

80% of newborns weigh at least 5.5 pounds and score 7 or above on the Apgar scale

Hospital Records Contact hospital for birth records


Continuous Process

Program Improvement

Step 4: Analyze Results

Document and summarize data Tabulate results Analyze data

Outcome of Delivering Healthy Babiesby Age of Mother

Age All Participants

Newborn weighed above 5.5 pounds, scored 7 or above on Apgar scale <17 17+ Total Percent of

All

YesNumber 8 5

13 72%% of age group 67% 83%

NoNumber 4 1

5 28%% of age group 33% 17%

Totals 12 6 18 100


Continuous Process

Program Improvement

Step 5: Communicate Findings

Present data in clear and understandable format

Provide explanatory information related to your findings

Outcome Data Table:Teen Mother Parenting Education Program

Total AGE

12-14 15-16 17+ Number of babies born in reporting period

18 4 8 6

Newborns weighing above 5.5 pounds and scoring 7 or above on Apgar Scale

Number Percent

13 72%

2 50%

6 75%

5 83%

Data Presentation

Pie Chart: Bar Chart:

Trial-run Outcome Findings

Do the findings seem reasonable?

Is the information presented clearly? Are explanations of problem areas and

proposed remedies satisfactory?

Does anything seem to be missing?

What other charts or tables would be helpful?


Continuous Process

Program Improvement

Measure & Monitor Outcomesfor Continuous Improvement

Review your process and data; make any necessary adjustments to your program

Monitor and review your programs periodically


Continuous Process

Program Improvement

Myth Busters

It’s an event to get over with and then move on!Evaluation is a whole set of new activities –

“We don’t have the resources!”There’s a “right” way to do outcomes evaluation.

“What if I don’t get it right?”Funders will accept or reject my outcomes plan.I always know what my clients need – “I don’t need

outcomes evaluation to tell me if I’m really meeting the needs of my clients or not.”

Provide direction for Staff

Identify Staff and Volunteer training

Identify technical assistance needs

Identify program improvement needs

Support annual and long range planning

Guide budget and justify resource allocation

Focus Board members on programmatic issues

Use your Findings

Resulting in…… Meeting your Mission

Resources

Measuring Program Outcomes: A Practical Approach. United Way of America. To order, call 800-772-0008

Logic Model Development Guide. Clear and concise discussion of the use of logic models. W. K. Kellogg Foundation (www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf)

Information Gold Mine; Innovative uses of Evaluation. Paul W. Mattessich and Shelly Hendricks (Amherst H. Wilder Foundation 2007)

Measuring Program Outcomes: A Practical Approach. United Way of America 1996

www.managementhelp.org/evaluatn/lgc_mdl.htm The Manager’s Guide to Program Evaluation,

Paul W. Mattessich, (Fieldstone Alliance 2003)

http://www.managementhelp.org/evaluatn/lgc_mdl.htm

Questions

Best Practices in Nonprofit Impact Measurement , CNM

Business

Transcript of Best Practices in Nonprofit Impact Measurement , CNM