Performance Measurement Series: Establishing the Basis for Program Performance Tracking April 17,...
-
Upload
steven-curtis -
Category
Documents
-
view
221 -
download
2
Transcript of Performance Measurement Series: Establishing the Basis for Program Performance Tracking April 17,...
Performance Measurement Series:Establishing the Basis for Program Performance Tracking
April 17, 2012, 9am CT
YWCA Great Lakes Alliance Region
Three Part Series:
1. Establishing the Basis for Program Performance Tracking and Measurement
April 17, 2012 9am CT
2. Defining Performance Indicators and Measures May 1, 2012 9am CT
3. Analyzing, Using and Reporting DataMay 15, 2012 9am CT
SESSION 1 TOPICSLaying the groundwork for
measurement….
SESSION 1 TOPICSLaying the groundwork for
measurement….
Organizational measurement systems and the role of program performance measurement
The Importance of program logic in establishing the basis for measurement
Outputs vs. outcomes Mission-driven chain of results Resources and examples
Organizational measurement systems and the role of program performance measurement
The Importance of program logic in establishing the basis for measurement
Outputs vs. outcomes Mission-driven chain of results Resources and examples
4
History of Nonprofit Performance Measurement
Financial Accountability Program products or outputs Adherence to standards of quality in
service delivery Participant related measures Client Satisfaction Program Outcomes Community Impact
5
Developments in Outcomes Measurement
Outcomes measurement processes developed
National organizations conduct research on outcomes
National organizations develop resources for local agencies
Agencies are measuring outcomes Funders are expecting outcomes
measurement New focus on community impact
Convergence of 2 Forces Driving Nonprofit Performance Measurement
1. Increased demand for accountability—boards, media, general public
2. Growing commitment of nonprofit managers to focus on results and strengthen performance
Why Measure Performance?…managing programs or agencies without performance measures has been likened to flying blind, with no instruments to indicate where the enterprise is heading.
Measures…can help managers reward success and take corrective action to avoid replicating failure.
Appropriately configured performance measures motivate managers and employees to work harder and smarter to accomplish goals and objectives.
Theodore H. Poister, 2003
Why Measure Performance?
Unless you are keeping score, it is difficult to know whether you are winning or losing.
Harry Hatry 1978
9
About Nonprofit Programs
Program – an integrated set of services conducted to meet specific, verified community needs by achieving certain specific outcomes among specific group(s) of clients in the community.Credible approachesUtilize community resourcesBased on assumptions or “theory of change”
10
Assess Mission Related Situation
Define Intended Impact and Outcomes to improve the situation
Define Programming/ Service Methods and Approaches to achieve intended impact
Determine Feasibility – Resources Needed and Available to carry out methods and approaches; Environmental Factors that could help or interfere
Establish Systems: Marketing, Performance Measurement, Improvement
Program Planning Steps
Mission
Vision
Strategic Goals & Objectives
Performance Measures
Performance Measures
Performance Measures
Program Management Measurement Systems
Resources consumed Activities conducted Transactions completed Clients Served Outputs produced Services Rendered Outcomes and Impact achieved
Program Theory
Reveals the critical assumptions and expectations inherent in a program’s design.
How the program services and practices are presumed to accomplish their purposes.
Provides the framework for performance measurement.
13
The “if-then” relationship
if a program provides prenatal counseling to pregnant teens, then the teens have increased knowledge of good prenatal care. If the teens have increased knowledge of good prenatal care, then this leads to changed behavior: the teens eat the proper foods, take a prenatal vitamin each day and avoid cigarettes, alcohol and other drugs. If the teens follow these practices, then the result is that the teens deliver healthy newborns.
14
Use of Logic Model for Framing Use of Logic Model for Framing Program TheoryProgram Theory
…a picture of how a program/services will work to solve identified problems
…the basis for a convincing story of a program’s/services’ expected performance
• Shows the major components of services• Establishes logical linkages among
components.• Describes the sequence of events
thought to bring about change.• Illustrates the theory and assumptions
underlying services (theory of change).
15
Logic Models help nonprofits… Construct a theory of change Intentionally address clearly defined situations, problems or
disparities Focus on achieving measurable outcomes that improve
these situations/ decrease these disparities Invest in activities that will lead to desired outcomes/
decreased disparity Assemble resources necessary to carry out activities
effectively Establish the basis for evaluation--determining, improving
and increasing success Drive program execution and development
16
Resources
Needed to effectively carry out strategies and actions
3. Strategies and Actions
that will lead to outcomes and, eventually contribute to impact
2. Intended Outcomes Impact
Reduction or elimination of problems, disparities
1. Situation/ Disparities
Mission-focused problems, disparities, circumstances
Program Theory
17
Situation: Mission/ Vision based: why we act
Problems, Disparities, Circumstances
Resources: our capacity, what we invest
StaffTimeVolunteersRevenuePartnersEquipmentFacilitiesTechnologyOther….
Activities/ Outputs:
What we doNumbers of: Workshops,
Counseling Sessions, Meals Served, Trainings, etc.
Who we reachNumbers of : Participants, Clients, Customers, Beneficiaries
Outcomes & Impact: what we hope to achieve
SHORTLearning Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation
MEDIUMActions
BehaviorPractice DecisionsPolicies Social action
LONG-TERMConditions HumanEconomicCivic Environment
Environment: factors that may help or impedePolitical, Economic, Values, Policies
Program Impact Theory
Describes the cause and effect sequence in which certain program activities, utilizing carefully chosen resources, eventually produce and contribute to social benefits.
Longer term/ultimate or “distal” outcomes are dependent on successful attainment of shorter term/intermediate or “proximal” outcomes.
Poister, 2003
Program Impact Theory
Programs rarely exercise direct control over the social conditions they are expected to improve
Programs tend to change critical but manageable aspects of a situation which is expected to lead to more far-reaching improvements.
Services affect some intermediate condition that, in turn, improves the social conditions of concern.
Poister 2003
20
Situation: Mission/ Vision based: why we act
Problems, Disparities, Circumstances
Resources: our capacity, what we invest
StaffTimeVolunteersRevenuePartnersEquipmentFacilitiesTechnologyOther….
Activities/ Outputs:
What we doNumbers of: Workshops,
Counseling Sessions, Meals Served, Trainings, etc.
Who we reachNumbers of : Participants, Clients, Customers, Beneficiaries
Outcomes & Impact: what we hope to achieve
SHORTLearning Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation
MEDIUMActions
BehaviorPractice DecisionsPolicies Social action
LONG-TERM IMPACTConditions HumanEconomicCivic Environment
Environment: factors that may help or impedePolitical, Economic, Values, Policies
21
Inputs/ Resources
Activities/ Outputs
Outcomes Impact
Program Logic Model
Money
Staff
Expertise
Facilities
Curriculum
Equipment
Partnerships
Volunteers
Strategies
Treatments
Advocacy
Services
Customer engagement
New knowledge
Increased skills
Changed attitudes
Modified behavior
Improved conditions
Altered status
Social Change
22
Steps for Measuring Outcomes
1. Identify desired outcomes and impact. Distinguish outcomes from inputs and activities/outputs
2. Determine outcomes sequence
3. Select outcomes to measure
4. Identify indicators (evidence) and performance standards (achievement levels)
5. Determine data collection approaches
6. Collect data
7. Use and learn from data
24
Determine Desired OutcomesGood program development begins with end result in mind.Whose situation/ status/ needs to change in order for the problem to be reduced?
Women? Women of color?
What impact would we hope to contribute to? What would it look like if the problems/disparities driving the program were eliminated? What changes in the situation driving the program need would we like to influence? What effects do we wish to contribute?
25
What are Program Outcomes?
Direct, intended beneficial effects customers/ participants experience during or after their involvement with a program or service
Problem related
Attainable
Measurable
26
Program Outcomes
Are Not….
Efforts or Activities
Participation in services
Numbers of customers served
Customer satisfaction
27
Program Outcomes
Are Not…. Are…
Efforts or Activities Results of efforts
Participation in services Results of participation in services
Numbers of customers served
Numbers of customers served who changed
Customer satisfaction Customer impact
28
What we Do The Desired ResultsEngage youth in racial
justice education programming.
Youth who complete the program understand racism and white privilege. They learn how to interrupt racist comments and behaviors.
Engage women in political leadership development.
Women learn what it takes to run for office; some decide to run.
Provide employment preparation and training.
Individuals develop resumes, engage in interviews and become employed.
Efforts vs. Outcomes
Find the Outcomes 30 older women participated in the Social Security
workshop. Provide multicultural books in the childcare locations. 50 girls learned how to manage a checkbook. 66 participants attended the racial justice conference. Program coordinator hired to manage the racial justice
project. Women are satisfied with the money management
series. Women who provide in-home day care learned cash-
flow management techniques. 20 women entered the minority apprenticeship program.
Find the Outcomes 30 older women participated in the Social Security
workshop. Provide multicultural books in the childcare locations. 50 girls learned how to manage a checkbook. 66 participants attended the racial justice conference. Program coordinator hired to manage the racial justice
project. Women are satisfied with the money management
series. Women who provide in-home day care learned cash-
flow management techniques. 20 women entered the minority apprenticeship program.
31
Managers who Measure Outcomes Report A clear definition of the program's intended outcomes, in itself,
provides focus for the program's work. Understanding their current level of outcome achievement provides
a barometer to assess progress and direct future activities. Outcome measurement provides invaluable information to improve
programs and then see if the improvements make the intended difference.
Outcome information is a powerful motivator of staff, who now can observe the progress they are making with participants in a consistent, tangible manner.
It becomes a powerful recruitment tool for volunteers who have many other choices for how they spend their time.
It helps position the agency in the community as a successful organization, which in turn leads to increased promotion and financial support
(Hatry, van Houten, Plantz, and Greenway, 1996).
32
Why Focus on Outcomes? What gets measured gets
done. If you don’t measure
results, you can’t tell success from failure.
If you can’t see success, you can’t reward it.
If you can’t reward success, you’re probably rewarding failure.
If you can’t recognize failure, you can’t correct it.
If you can’t see success, you can’t learn from it.
If you can demonstrate results, you can win public support.
Excerpt from Reinventing Government by David Osborne and Ted Gaebler
33
Clarify problem(s) as basis for outcomes
Why does this program exist? What problem(s) does the program try to reduce
or eliminate? Who is affected by the problem?
What changes do we want those affected to experience?
34
Examples of Problems
Racial Justice Women of color are more
likely to live in poverty than white women.
The median earnings for women of color is 28% less than for white women.
The mortality rate from lung cancer (48.7 per 100,000) among African American women is higher than national rates and that for white women.
Women’s Economic EmpowermentSingle-mother families have the highest poverty rates of any household type—1 of every 2 families.Women earn 69 cents for every dollar earned by white men.Low-income women are uninsured at a rate of 19.5%.
From Status of Women in Wisconsin Reports 2004, 2008
35
From Problem to Impact
Think about the problem reduced or eliminated
PROBLEM LONG TERM OUTCOME/ IMPACT
High incidence of women in poverty; low wages
Increased economic security among women.
Homelessness among single mothers.
Women and their children in safe, affordable housing.
Community geographically segregated by race.
Fully integrated community.
Outcomes or Impact?How is impact Different?
Long term effects of our outcomes on those we serve
Eventual outcomes—things that happen because of the outcomes we produce
Less predictable—often dependent upon many other variables
Impacts are what we hope for but outcomes are what we work for.
36
Outcomes to Impact
SHORT
Learning
Awareness
Knowledge
Attitudes
Skills
Aspiration
Motivation
MEDIUM
Action
Behavior
Practice
Decisions
LONG TERM IMPACT
Conditions
Human
Economic
Civic
Environment
Outcomes to Impact
SHORT
Learning
Awareness
Knowledge
Attitudes
Skills
Aspiration
Motivation
MEDIUM
Action
Behavior
Practice
Decisions
LONG TERM IMPACT
Conditions
Human
Economic
Civic
Environment
Program Outcomes
Are the links between our actions and our mission
The tangible evidence that our work is leading to mission achievement
39
Defining Outcomes for a Program What should lead to the desired long term
impact? Outcomes should represent meaningful change
Make a marked difference in the situation, create a new situation
Replace something negative with something positive Outcomes should address “root causes”--
produce lasting, rather than temporary effects—beyond the absence of the problem.
40
41
Defining Outcomes:Start with where you are… Existing program New program idea Problem we wish to address
But be sure to clarify the problem and/or program purpose
Avoid initial preoccupation with measurementUse logic model
42
Existing Programs
1. Review Program purpose and activities—why do we do this? What would success for customers look like?
2. Clarify problem/ situation and who is affected—what would it look like if the problems were resolved?
3. Clarify outcomes based on purpose and problem
1. Clarify problem/ situation and who is affected first
2. Define desired impact and outcomes
3. Determine program purpose, activities that most likely to result in outcomes
New Program Development
43
Sources of Outcomes Information
Examine outcomes of similar programs/ services.
Talk to program staff. Talk to current and former customers. Find research about the problem and proven
solutions. Use situation/problem as the basis.
Program Outcomes
The tangible evidence that our work is leading to mission achievement
Are the links between our actions and our mission
44
ACTIONS….…OUTCOMES……….IMPACT……..MISSION
Shifting from what we do to changing lives
EFFORT: Provide
financial literacy programming to 40 women
OUTCOMES 35 of the 40 women
who completed the financial literacy program developed household budgets and savings plans; of those, 30 opened savings accounts.
IMPACT: Economic
security among women
MISSION: Empowered
women
Changing lives
EFFORT Provide
100 girls with technology training.
OUTCOMES Of the 100
girls who attended technology training, 75% demonstrated software, hardware proficiency
IMPACT Girls enter
technology-focused careers
MISSION Empowering
women
Changing lives
EFFORT Racial
justice education series provided to 150 individuals.
OUTCOME 125 of those
who attended the racial justice education series increased their understanding of racism and how to work against it.
IMPACT Anti-racist
community Reduced
incidence of racism.
MISSION Eliminating
racism
Changing lives through Advocacy
EFFORT Engaged
50 individuals in campaign against hate crimes in the state.
OUTCOME 50 individuals
wrote letters to their elected officials encouraging them to introduce state legislation against hate crimes. 10 held meetings with elected officials.
IMPACT Hate crime
legislation passed.
Reduced incidence of hate crimes.
MISSION Eliminating
Racism Empowering
Women
50
Most Programs are Designed to Achieve Multiple Outcomes
Outcomes Sequence Most programs involve a series of outcomes. Hierarchy of logically related changes or
benefits. “Chain of outcomes”.
52
Outcomes Sequence
SHORT
What changes do we expect
to occur within the
short term?
MEDIUM
What changes do we want to see occur after that?
LONG TERM
What changes do we hope to see over time?
53
Outcomes Sequence
SHORTLearning
AwarenessKnowledgeAttitudesSkillsAspirationMotivation
MEDIUMAction
BehaviorPracticeDecisions
LONG TERM IMPACT
Conditions
HumanEconomicCivicEnvironment
Racial Justice Program Example
Short Termknowledge
Intermediatebehavior
Long Term/ Impactcondition
Increased understanding about racism, privilege and their impact
Desire to work against racism
Increased interaction among people of different racial heritage groups
Commitment to action and action on commitment
Increased relationships between whites, people of color
More integrated community
Decreased disparities based on race
55
SITUATION INPUTS ACTIVITIES and OUTPUTS
OUTCOMES
Short Term Interm.
Long Term
Women represent 15% of elected leadership positions in the community
Recent report indicates few women choose to run
Policy climate not favorable to women; back burner, erosion of choice
Women with experience in political leadership
Funds to support program coordination
Curriculum, trainers
Series of political education workshops reach 100 women
Women in leadership group mentors 25
Women increase their understanding of political leadership process, opportunities
Women believe they can run and win
More women run for office
Women elected to 50% of all positions
Policies more favorable to women
56
Staff
Money
Partners
Design parent edu curriculum
Provide 6 training sessions
Targeted
parents
attend
Parents increase knowledge of child develop
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS ACTIVITIES/OUTPUTS OUTCOMES/ IMPACT
Family Strengthening Program
Approaches to Mapping the Outcomes Sequence
Review outcomes identified for the program.
Arrange the outcomes in a logical sequence that suggest a “theory of change”
Outcomes may be arranged in more than one “chain”
Determine if outcomes are missing from a chain and make adjustments.
57
Situation: why we actSingle women in the community with families do not progress economically at the same rate as men; incomes of women of color consistently lower than white women.•Some lack higher education, training needed for higher paid positions.•Few own assets, including a home.•Many women are paid less than men for same work.
Resources: our capacity, what we investStaff with expertise in sexism, racism, economic literacy.Scholarship funds.Partnerships with community college, university.Partnerships with bank.Partnership with 9 to 5.3 year investment.
Activities: What we do, who we reach
Services for single women with families that especially target women of color:
•College assistance with scholarships;
•Financial literacy with savings match;
•Self-advocacy training.
Community education about pay equity.
Outcomes & Impact: what we hope to achieve
Women enter higher education programs, increase financial literacy and build savings; understand and advocate for higher earnings.
Women graduate from college. Move into higher paid positions.Build savings, own homes. Community advocates for pay equity.
Decreased income disparities based on race, gender decrease.
Environment: factors that may help or impedeEconomy, state budget, federal pay equity laws, national or local incidents
Should we Measure all Outcomes?
Criteria for Choosing Outcomes to Measure
AchievableFits program scope – what we can reasonably
influenceData is available, can be developedThe outcome is compelling
Which Outcomes to Measure
Review the “chain of outcomes” Focus measurement on the levels where you
are accountable; where you have reasonable control.
Decide how far out the chain of outcomes you can go.
Keep an eye on the ultimate long-term outcome or impact, although it may not be easily measured.
62
Issues to Consider
The longer term the outcome, the less direct influence the program has over its achievement and the more likely other extraneous forces are to intervene.
Because other forces affect an outcome doesn't mean it shouldn't be included.
Long-term outcomes should not go beyond the program purpose or target audience.
Staff
Money
Partners
Design parent edu curriculum
Provide 6 training sessions
Targeted
parents
attend
Parents increase knowledge of child develop
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS ACTIVITIES/OUTPUTS OUTCOMES
Family Strengthening Program
Staff
Money
Partners
Design parent edu curriculum
Provide 6 training sessions
Targeted
parents
attend
Parents increase knowledge of child develop
Parents learn new ways to discipline
Parents use improved parenting skills
Reduced rates of child abuse & neglect
INPUTS OUTPUTS OUTCOMES
Family Strengthening Program
Which Outcomes to Measure?
Example: Racial Justice Education Program
Short Termknowledge
Intermediatebehavior
Long Termcondition
Increased understanding about racism, privilege and their impact
Desire to work against racism
Increased interaction among people of different racial heritage groups
Commitment to action and action on commitment
Increased relationships between whites, people of color
More integrated community
Decreased disparities based on race
Example: Racial Justice Education Program
Short Termknowledge
Intermediatebehavior
Long Termcondition
Increased understanding about racism, privilege and their impact
Desire to work against racism
Increased interaction among people of different racial heritage groups
Commitment to action and action on commitment
Increased relationships between whites, people of color
More integrated community
Decreased disparities based on race
SITUATION INPUTS ACTIVITIES and OUTPUTS
OUTCOMES
Short Term Interm
Long Term
Women represent 15% of elected leadership positions in the community
Recent report indicates few women choose to run
Policy climate not favorable to women; back burner, erosion of choice
Women with experience in political leadership
Funds to support program coordination
Curriculum, trainers
Series of political education workshops reach 100 women
Women in leadership mentor those who complete the series of workshops
Women increase their understanding of political leadership process, opportunities
Women believe they can run and win
More women run for office
Women elected to 50% of all positions
Policies more favorable to women
SITUATION INPUTS ACTIVITIES and OUTPUTS
OUTCOMES
Short Term Interm
Long Term
Women represent 15% of elected leadership positions in the community
Recent report indicates few women choose to run
Policy climate not favorable to women; back burner, erosion of choice
Women with experience in political leadership
Funds to support program coordination
Curriculum, trainers
Series of political education workshops reach 100 women
Women in leadership mentor those who complete the series of workshops
Women increase their understan-ding of political leadership process, opportuni-ties
Women believe they can run and win
More women run for office
Women elected to 50% of all positions
Policies more favorable to women
Situation: why we actSingle women in the community with families do not progress economically at the same rate as men; incomes of women of color consistently lower than white women.•Some lack higher education, training needed for higher paid positions.•Few own assets, including a home.•Many women are paid less than men for same work.
Resources: our capacity, what we investStaff with expertise in sexism, racism, economic literacy.Scholarship funds.Partnerships with community college, university.Partnerships with bank.Partnership with 9 to 5.3 year investment.
Activities: What we do, who we reach
Services for single women with families that especially target women of color:
•College assistance with scholarships;
•Financial literacy with savings match;
•Self-advocacy training.
•Community educ. and engagem. in pay equity. advocacy
Outcomes & Impact: what we hope to achieve
Women enter higher education programs, increase financial literacy and build savings; understand and advocate for higher earnings.
Women graduate from college.
Move into higher paid positions.
Build savings, own homes.
Decreased income disparities based on race, gender decrease.
Environment: factors that may help or impedeEconomy, state budget, federal pay equity laws, national or local incidents
Preparing for Measurement Clarify the outcomes statement as a target:
Specifies the meaningful change Identifies who will experience the change Defines the period of time for achievement May reference program effort.
Single women, especially women of color, in the community with families who are low income, who participate in and complete the 6 month financial literacy program, will increase their financial literacy and develop savings by the end of the program.
The best outcomes are those that are narrowly focused and specific.
70
Preparing for Measurement Develop outcomes statements for the outcomes
you will measure: Specifies the meaningful change Identifies who will experience the
change Defines the period of time for
achievement May reference program effort.
71
Example Outcomes Statements Girls who complete the techgyrls program will
become technology savvy. Women who engage in transitional housing
will secure safe, affordable housing within 18 months.
Children who complete the pre-school program will be ready for first grade.
72
Challenges to Measuring Outcomes
Aren’t there some things that we just can’t measure?
Pre-and post tests are too cumbersome. What can we really influence? Where do we begin? How can racial justice be measured?
Why Measure Outcomes?
Determine EffectivenessAre we making progress on eliminating racism?Do our efforts help women advance
economically?
Increase EffectivenessCould our impact be greater?What should we do more of, less of?
Communicate Value What value do we add? Why is our mission
worth supporting?
Next Session:Defining Performance Indicators and
Measures Framing Indicators Sources of Data Validity, Reliability and other Criteria Measurement Problems and Challenges Resources and Examples
Resources to Assist with Measuring Outcomes
Urban Institute www.urban.org Publishes studies on discrimination issues, including race and
gender; measuring discrimination; community indicators Proposed national report card on racial and ethnic
discrimination
Institute for Women’s Policy Research www.iwpr.org Publishes reports on status of women—US, states, specific
issues Attention to race, ethnicity
Resources for Measuring Outcomes
Poverty and Race Research Action Council www.prrac.org Links research to advocacy on issues of race and gender
The Grantsmanship Center www.tgci.com Offers guides to program evaluation
United Way www.unitedway.org/outcomes Offers outcome measurement resources, guides