Using Data to Measure and Report Program ImpactAnne Basham, MFAMEMconsultants

Agenda Introductions Define Learning Objectives and Key

Concepts Apply Knowledge to Your Work Conclusions Questions?

Introductions About Anne

Program Evaluation Consultant, www.memconsultants.com Organizations serving youth and

communities Collect data to determine what is working

well about a program and to identify opportunities for improvement

Learning Objectives become familiar with the concepts of Performance

Measurement and Theory of Change; understand how data can be used to measure

performance at your specific site; clarify barriers to collecting quality data and

strategies to overcome these barriers; learn to interpret and report data once it has been

collected; share findings with multiple stakeholders; and identify next steps in the continuous quality

improvement cycle

Performance Measurement Performance measurement is the

process by which businesses, governments and other organizations establish criteria for determining the quality of their activities, based on organizational goals.

Performance measure: A value or characteristic that measures progress toward goals, and also used to improve progress, reduce risks, or improve cost-effectiveness.

Theory of Change Theory of Change defines all building

blocks required to bring about a given long-term goal.1

Includes assumptions, strategies and predicted outcomes

Customized to reflect the goals and needs of an organization

AKA: pathway of change, blueprint, logic model, outcome map, “so that”chain

1 as defined at www.theoryofchange.org

Logic Models

Resources/ Inputs Activities Outputs

Outcomes(short term, intermediate, long term)

Logic Model Example:WSC Academic Improvement Programs

Resources: AmeriCorps Members, Funding,

Program Site Structure

Activities: Recruit youth, Offer program

Output: 4,500

children complete

participation in K-12

education programs

Outcome: 70% of youth that complete

program demonstrate

improved academic

performance.

Logic Model Example:WSC Environmental Stewardship Programs

Resources: AmeriCorps Members, Funding,

Tools, Land Permit

Activities: Offer

training, Organize volunteer

opportunities

Output: 3,000

individuals participate in environmental

awareness and/or energy

efficiency education

Outcome: 70% of

individuals that

participate demonstrate

improved knowledge or awareness.

Logic Models

Resources/ Inputs Activities Outputs

Outcomes(short term, intermediate, long term)

“So That” chains…

Strategy An organization offers this opportunity or activity;

So that… This short term outcome or change can occur;

So that… This intermediate outcome or change can occur;

So that… This long term outcome or change can occur;

So that… This organizational goal is met.

“So That” Chain Example:WSC Healthy Futures Programs

Strategy Educate the public about healthy behaviors through outreach and activities

So that…2,500 individuals participate in health education activities led or supported by AmeriCorps Members

So that…

Individuals who participate in a health education opportunity indicate an• increase in knowledge; or • increased awareness; or • changes in healthy behavior.

End Goal• Improved access to health care; • Increased physical activity and improved nutrition for youth;• Seniors’ increased ability to remain in their homes.

“So That” Chain ExampleWSC Veterans & Military Families Programs

Strategy Offer volunteer opportunities and events for veterans and their families.

So that…500 veterans will be engaged in AmeriCorps volunteer projects, activities and events led or supported by AmeriCorps members

So that…Veterans and their families become more aware of opportunities to be involved in their communities.

So that…Veterans will indicate in increased interest and/or involvement in volunteer activities in the future.

End Goal• Improved quality of life for veterans• Strong military families

Program Outcome/Goal

Academic Achievement

Students demonstrate improved academic performance

Disaster Preparedness

Individuals are able to respond in the event of a disaster

Environmental Stewardship

Individuals increase their knowledge and awareness of energy efficient and environmentally conscious practices

Healthy Futures

Individuals increase their knowledge/awareness of and/or participation in healthy behaviors

School Readiness

Students demonstrate gains in school readiness in terms of social/emotional development

Veterans Veterans indicate increased interest and/or involvement in volunteer activities

Indicators

Measuring Performance - Indicators Indicators are the measurable evidence that

demonstrate progress towards your long term goal. To restate, they are measurable. Examples:

Number of people participating in a program, event or activity

Number of economically disadvantaged individuals Increased interest or prediction of involvement Increased knowledge and/or awareness Increased confidence to act Demonstrated gains in school readiness or

academic performance

Data Collection Methods:Tracking Indicators

Quantitative Qualitative

• Program Records (attendance)

• Structured Assessment

(test scores)

• Focus Groups• Document/

Literature Review

• Surveys• Interviews• Observations

Indicator Data Collection Method

Number of people participating in a program, event or activity Program Records/Tracking Log

Number of economically disadvantaged individuals

Program Records compared to census data, eligibility for benefits (FRL, TANF, Section 8, SCHIP, etc.)

Increased interest or prediction of involvement Post Survey

Increased knowledge and/or awareness Post Survey

Increased confidence to act Post Survey

Demonstrated gains in school readiness or academic performance

Standardized Assessment Tools

Pre and Post Surveys

Common Data Collection Challenges Time Working with others Measurement

mismatch Access Sample size

Comprehension “Bad” data Poorly designed

tools Measuring

prevention What else?

Data Analysis Quantitative

Totals Averages Percentages Pre/Post Comparison

Qualitative Emergent Top Down

Research-based strategies Don’t reinvent the wheel – learn from

past successes – “best practices” Acknowledge you aren’t doing research

or studies that are genearlizable You are collecting information about

your program Build off of the research of others

Data Reporting Other Staff or Volunteers Program Participants Board Members Funders Washington Service Corps Who else?

Continuous Quality Improvement

clarify

plan

develop

collectanalyze

communicate

use

Learning Objectives become familiar with the concepts of Performance

Measurement and Theory of Change; understand how data can be used to measure

performance at your specific site; clarify barriers to collecting quality data and

strategies to overcome these barriers; learn to interpret and report data once it has been

collected; share findings with multiple stakeholders; and identify next steps in the continuous quality

improvement cycle

Questions? Everyone will be unmuted so you can

ask your questions. You may also type questions into the

chat box.

Thank you! Slides will be available for your use Corporation for National and Community

Service (CNCS) Performance Measurement website: http://www.nationalserviceresources.org/pro

gram-financial-and-grant-management/performance-measurement

Contact Anne with questions: anne@memconsultants.com