What Works, and Why:Issues and Methods in Evaluation in the Non-Profit Sector

Edward T. JacksonCarleton University, Ottawa

Presented to the Future Fund Evaluation Workshop, Ontario Trillium Foundation, Toronto

April 28, 2010

2

Overview

• Context• Issues in Evaluation• Results-Based Management• Participatory Strategies• Social Return on Investment• Next Steps• Resources

3

Context I: Ontario Economy – Challenges

• Restructuring driven by globalization (BRICs), high dollar

• Shrinking of the manufacturing sector• A return to a branch-plant, staples-

driven economy• Reduced number of Ontario multi-

nationals• Decrease in corporate and individual tax

revenues to governments

4

Context II: Ontario Economy – Opportunities • Enabling New Ontarians, Aboriginal Ontarians and young

people to fill upcoming labour shortages in all sectors• Accelerating the development of good jobs in the green

economy (i.e. energy, water, etc)• Building new, large-scale Ontario-based anchor

companies in key sectors with a focus on value-added production, global markets and local suppliers

• Levering social innovation-mobilizing technology, ingenuity and entrepreneurship to solve pressing social problems

• Exporting Ontario’s expertise in higher education management, education and research

• Optimizing the role of and linkages with non-profit organizations and social enterprises in these initiatives

5

Context III: OTF’s Response

• Funding Priority: “Enhanced employment and economic potential”

• Future Fund: “To support initiatives that create significant and sustainable opportunities for communities, networks and individuals to participate fully in Ontario’s labour market… to break down barriers to employment and self-employment”

• Implication: To rapidly convert innovation into large-scale implementation.

6

Context IV: OTF Evaluation

• Main Purpose: “To measure the success of a project or program by the end of the grant”

• Learning: What works, what doesn’t and why?• Meeting Needs: Are the needs of the target

population being met? If not, how can they be better met?

• Replication: What are the key components of the intervention worth replicating?

• Accountability: To what extent is the project or program efficient and effective?

7

Context: OTF Approach to Measuring Results• Results should be specific, concrete and

measurable, of short-term or long-term duration, and should be based on qualitative or quantitative data, or both.

• Results should be measured at the following levels:

– Individuals– Organizations– Communities

8

Key Issues in Evaluation

• Who will use the evaluation, and how will they use it?• How feasible is it to actually achieve the proposed

results (are they realistic or overstated/hyped)?• What are the different expectations among

stakeholders as to what constitutes a credible evidence base for assessing effectiveness?

• What are the costs – in terms of money and time – of conducting the evaluation?

• To what extent are there asymmetries in stakeholder participation in the design and implementation of the evaluation?

• What are the real prospects and mechanisms for downstream replication and scaling-up?

9

Results-Based Management

• Results-Based Management• Results Chain• Impact Value Chain• Results Grid• Case Study – Alterna Micro-Loan

Fund Evaluation

10

Results-Based Management

• “A life-cycle approach to management that integrates strategy, people, resources, processes and measurement to improve decision-making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning and changing, and reporting performance” (CIDA)

11

Results Chain

• “A depiction of the causal or logical relationships between inputs, activities, outputs and outcomes of a given policy, program, or initiative” (CIDA, OECD)

12

Results Chain

Inputs Activities

Outputs

Immediate Outcomes

(Short-Term)

Intermediate Outcomes (Medium-

Term)

Ultimate Outcomes

(Long-Term)

13

Impact Value Chain

Inputs Activities Outputs Outcomes Goal Alignment

Impact = Measure of Change

What you put in

What you do

Results that are measured

Collection of all results intended and/or unintended

—MINUS—What would have happened anyway, or next best alternative (results “but for” the intervention)

= IMPACT

How well outcomes align with intended goals• Funds

• Time• Expertis

e• Network

s• Influenc

e

• Grantmaking

• Programs• Information• Sharing• Investing

Activity results (e.g. number of people served)

(Source: Goedeke and Pomares, 2009)

14

Results Grid

Level Immediate Outcomes

Intermediate Outcomes

Ultimate Outcomes(Impacts)

Macro-Level(Policy)

     

Meso-Level(Institution)

     

Micro-Level(Local Organizations, Households, Individuals)

     

(Source: Jackson, 2010)

15

Case Study: Logical Framework for the Alterna Savings Community Micro-Loan Program

 

 

Mission Statement: To provide a compelling financial alternative to Canadians through a unique banking philosophy based on cooperative values and focused on professional financial counsel

Program Components Inputs Activities Outputs Intermediate

OutcomesFinal

Outcomes Impacts

Small Business

Development loans

Alterna’s Capital (Credit Union members’ shares)1 Full-Time Community Micro-Loan Manager + 1 Full-Time CSR Manager

Provision of small loans (maximum $15,000 at prime rate + 6%) upon approval of loan requests submitted by interested applicants

Loans disbursed to qualified applicants in a timely fashion (e.g. within two weeks from the applicants’ initial request)

Start-up or expansion of small businesses (including purchase of productive assets, workspace rental,

e-commerce development)

  Micro-Level (Borrowers)

   

Meso-Level (Corporate)       

Macro-Level (Policy)

Free financial literacy

programs and professional development opportunities

Volunteer work by both Alterna Staff and recognized local business professionals/Instructors

-

Business financial literacy programs offered to both members and prospective applicants -

Member Display Booth and Information Boards at Alterna branches

-

Networking Café conducted as planned-

Presentations made by Alterna CSR Officers at local business development centres and training colleges

-Members’ enhanced understanding and adoption of effective business practices (e.g. business plan development, accounting, marketing)-Development of partnerships among borrowers in different fields(Source: Tarsilla, 2010)

16

Case Study: Results Grid for the Alterna Micro-Loan Program: Outcome

Evaluation Layers Final Outcomes ImpactsMicro-Level

(Borrowers)

Increase in Borrowers’ productivity and personal assets Increased borrowers’ payment of federal and provincial income taxes

Encouragement of borrowers’ self-sufficiency Reduced number of borrowers relying on government assistance

Increase in borrowers’ annual income Enhanced well-being of borrowers and their households (including housing, nutrition, health)

Meso-Level

(Corporate)

Increased free media-coverage of corporate CSR as well as of their services/products at large; boost corporate membership

Increased corporate reputation within the community

Development of new products and services specifically catering to the most marginalized and vulnerable population groups (e.g. new Canadians, individuals with no or low credit rating)

Enhanced corporate brand differentiation

Consolidation of Alterna’s membership base (e.g. through referral) and enhance borrowers’ opening of additional accounts other than business related (e.g. personal lines of credit, mortgages).

Increased customer loyalty

Macro-Level

(Policy)

Contributions to the health and success of the local economy; save government a large amount of resources otherwise allocated to assistance programs; contribute to federal and provincial revenues due to the borrowers paying higher income taxes over time.

Promotion of self-employment and entrepreneurship amongvulnerable population groups in Toronto

Fostering the creation of new jobs (e.g. micro-loan borrowers hiring employees to run their business).

Contribution to the reduction in the local unemployment rate

(Source: Tarsilla, 2010)

17

Participatory Strategies

• Participatory monitoring and evaluation• Why is PME important?• Core Principles• Front-end costs• Multi-Stakeholder engagement

strategies• Stakeholder network• The PME cycle• Framework for Organizational Self-

Assessment

18

Participatory Monitoring and Evaluation: Definition – PME• Participatory monitoring and evaluation

is an approach to performance review in which stakeholders in an intervention (local citizens, policy makers, funding agencies, and non-governmental organizations) work together to decide how to assess progress, conduct data collection and analysis, and take action on their findings

(Source: Jackson, 2005)

19

Why is PME Important?

• PME is geared towards not only measuring the effectiveness of a project, but also towards building ownership and empowering beneficiaries; building accountability and transparency; and taking corrective actions to improve performance and outcomes

(Source: World Bank, 2002)

20

Core Principles

• Primary stakeholders are active participants – not just sources of information

• Building capacity of local people to analyze, reflect and take action

• Catalyzes commitment to taking corrective measures

(Source: World Bank, 2002)

21

Front-End Costs

• Time-consuming and expensive at the front end

• Authentic stakeholder engagement takes time

• Additional professional involvement as facilitators/intermediaries also adds costs

(Source: Jackson, 2000)

22

Multi-Stakeholder Engagement Strategies

• Evaluation Coordination: Evaluation committee, working group; evaluation team

• Methods:– Qualitative: Workshops, community meetings,

participant observation, online collaboration, interviews, questionnaires, video/film/theatre, other visualization tools

– Quantitative: Large-scale surveys, cost-benefit analysis, statistical analysis

– Other: Mapping, GPS, corporate/legal research, scientific monitoring (e.g. water quality)

• Role of Evaluators: Catalyst and facilitators; ensure voice, choice; do no harm

23

Stakeholder Network

*The most results-oriented stakeholders(Source: Jackson, 1998)

Taxpayer*/ Donor* Individual*

GovernmentAgency/

Foundation Implementing

AgencyGroup/

Enterprise

Non-ProfitsNon-Profits

Other Family Members

Children

Spouse

Consultants Consultants

24

The PME Cycle10

Taking Action on the Findings

9Sharing the Findings

8Preparing the Report

7Analyzing the Findings

6Gathering Relevant Data

5Developing Results Indicators

4Identifying Key

Performance Issues

3Establishing Goals for

the PME Exercise

2Forming a Multi-Stakeholder Vehicle to Lead the Process

1Deciding Who Participates

25

Framework for Organizational Self-Assessment

(Sources: International Development Research Centre/Universalia)

26

Social Return on Investment

• Social Return on Investment: Definition

• Case Study: Turnaround Couriers

• Expanded Value-Added Statement

• Case Study: PARO Women’s Centre

27

Social Return on Investment: Definition

• Discounted, monetized valuation of the social value that has been created, relative to the value of the investment

• Key leaders in SROI include the Roberts Enterprise Development Fund, Jed Emerson, New Economics Foundation, Social Capital Partners, and the Social Economy Centre at OISE/UToronto

• Receiving greater attention, but SROI can be resource-intensive and difficult to replicate

28

SROI Snapshot: Turnaround Couriers - Theory of Change

Goals Methods Success Metrics• Hire couriers and office

administrative staff from disadvantaged youth population

• Provide transitional work experience to enable youth to develop employability skills, a resume and a support network

• Enable youth to access the mainstream job market

• Enable youth to stabilize life situation, begin a career path and leave the shelter system

• Recruit youth from youth shelters and youth serving agencies across Toronto

• Provide a real job, not a job training experience

• Establish a supportive management environment

• Assist youth with planning and making next steps regarding housing and employment

• Youth are able to get out of shelter system and into independent housing

• Youth meet or exceed job expectations

• TurnAround helps youth secure next job and establish a career path

• Youth are able to get off and stay off government financial assistance

(Source: Social Capital Partners, 2009)

29

SROI Snapshot: Turn Around Couriers (Extract)Overview of Target Population (sample)

•38% recruited directly from shelters•23% female•Average age: 21•100% unemployed at time of hire•54% receiving social assistance at hire

•54% been involved with justice system

•54% did not complete high schoolSustainable Livelihood Outcomes (sample)

•89 youth in total have been hired over 5 years

•100% target population recruited from shelters able to get out of shelter system and secure independent housing within 6 months of employment at TAC

•85% who relied on income support through social assistance at time of hire able to get off and stay off

Avg Change in Societal Contribution (Target Employees): $9,391

Average Number of Target Employees: 10

Current Year Cost Savings to Society: $93,910

Cumulative Cost Savings (prior to Y5): $191,170

Total Cost Savings to Date: $285,080

Cumulative Societal Payback Period: 1.8 years

Cumulative SROI: 285%

Note: initial SCP investment = $100,000

Employment Outcomes (sample)•Increased target/non-target staff ratio to 83%

•69% continue to work at TAC (9)•15% moved onto mainstream employment in window cleaning industry (2)

•8% went on to post secondary education (1)

(Source: Social Capital Partners, 2009)

30

Expanded Value-Added Statement

• An integrated social accounting method that employs carefully constructed assumptions to quantify, in dollar terms, the social value that a non-profit or co-operative produces (e.g. volunteer hours contributions, unpaid contributions to other stakeholders, etc.)

• Usually applied to a single fiscal year, but can be used to assess multiple indicators over 10-20 years

• The EVAS integrates the calculations of different types of social value into the overall financial statements of the organization being assessed.

31

Case Study: PARO Women’s Centre

• Provides training, mentoring and small loans to women in northern Ontario, and manages a store that sells artisnal products

• EVAS, calculated for 2005-2006, found that PARO created at least $132K from volunteer contributions, the value of mentorship and consulting by the Executive Director

• For every dollar of purchases on goods and services, PARO created more than $2 worth of value added, including about $.70 worth of social value

(Source: Babcock, 2007)

32

Next Steps

1. Reflect on key issues relating to the evaluation of your Future Fund grant

2. Learn more about evaluation strategies and methods

3. Work with OTF and other stakeholders to design an evaluation that is results-oriented, participatory and measures social return

4. Build a community of practice on FF evaluation

5. Document your experience for others, and learn from them as you proceed forward.

33

Resources

• Blended Value (blendedvalue.org)• Canadian Evaluation Society (evaluationcanada.ca)• Canadian International Development Agency (acdi-cida.gc.ca)• Carleton Centre for Community Innovation (carleton.ca/3ci)• International Development Research Centre (idrc.ca)• International Program for Development Evaluation Training

(ipdet.org)• Institute of Development Studies, University of Sussex (ids.ac.uk)• New Economics Foundation (neweconomics.org)• Organisation for Economic Co-operation and Development

(oecd.org)• Roberts Enterprise Development Foundation (redf.org)• Social Capital Partners (socialcapitalpartners.ca)• Social Economy Centre, OISE/UT (sec.oise.utoronto.ca)• Social Finance (socialfinance.ca)• Treasury Board of Canada Secretariat (tbs-sct.gc.ca)• Universalia (universalia.com)• World Bank (worldbank.org)