Rotary Foundation Cadre Training: Monitoring and Evaluation

Monitoring and EvaluationTRF Cadre Training 12 January 2015

CADRE TRAINING | 2

PRESENTERS

Philip Silvers•Cadre Chairman 2014-15•Past RI Director and board liaison to the Trustees•44-year career as a program evaluation and survey research professional•10 years professor of graduate-level Research Methods

James Robinson•Director, Strategy Innovation and Evaluation•Former Division Manager Humanitarian Grants and Future Vision Pilot•RI-USAID Steering Committee

Korhan Atilla•Cadre Technical Coordinator 2008-11•Civil Engineer•Conducted sustainability check site visits•District Governor Elect 2014-15, D-2430

CADRE TRAINING | 3

Cadre’s Efforts to Support Sustainability Phil Silvers

CADRE TRAINING | 4

The New Grants Model Is Working Fairly Well– TRF Manuals Are Helpful– Clubs and Districts are working together– The Cadre Is an Important Resource

• Technical assistance

• Stewardship (accountability)

Proposal Writers and Sponsors are struggling– Online application system is a challenge– Effective needs assessments are not being done– Sustainability—still an elusive concept– Monitoring and Evaluation (M&E) Who will do it?– Average time to grant approval: 3 months

YEAR ONE – WHAT DID WE LEARN?

CADRE TRAINING | 5

Core Effects:Sustainable

Change/Lasting Results

ProcessResults

(‘Doing Good’)

Training

Cultural Experiences

Adventure

Travel

Friendships

HumanitarianSupplies

Equipment

Externalities(The ‘Sizzle’)

Challenges

Rotary“Credits”

CADRE TRAINING | 6

• Real sustainability begins with the Community Needs Assessment—by talking to the grass-roots people before you come up with the solution.– Need to comprehend their view of the world—their

fears, their challenges, their dreams for their kids• Needs Assessment, Sustainability, and M&E are not

independent concepts—they must be integrated into one piece by one team.

• Depending on what you are trying to accomplish, the following slide gives you some options on measuring Sustainability (or ‘red flags’ if you omit them)

BASIC PRINCIPLES OF SUSTAINABILITY

CADRE TRAINING | 7

Security—•People

•Premises•Equipment

Security—•People

•Premises•Equipment

Dimensionsof

Sustainability

Ecological

Human Capital OrganizationalSystems

Behavioral Change MechanicalSystems

New Technologies(Tools) Cultural Ethos

Legacy

DIMENSIONS OF SUSTAINABILITY

Economic New Policy/Legislation

Cultural Ethos/Legacy

CADRE TRAINING | 8

Security refers to the protection against harm to people, to premises, and to equipment. Other than a good needs assessment, this is the most fundamental issue in Sustainability.

Are the participants secure from coercion, threats to their reputation, privacy, and physical/mental well-being?Are the premises secure from unwanted intruders?Are the equipment and materials safe from theft and vandalism?

CADRE TRAINING | 9

Economic refers to the financial viability of the project over the long haul.

Are revenue streams planned to support the program after the grant funding ends?

Operating costs? Equipment maintenance and replacement?

Who will be responsible for managing the revenue streams?Are contingency plans in place if the planned funding streams fail to deliver?

Economic

CADRE TRAINING | 10

Cultural ethos refers to what is acceptable or not acceptable behavior within society, e.g., smoking in common areas, young girls’ access to education, women’s role in government, getting tested for HIV/AIDS, tribal enmities, xenophobia.

Have project planners done a ‘force-field’ analysis of the factors inhibiting and fostering change?What rewards, incentives, modeling, or strategies might help shift prevailing attitudes?What various measures will reveal whether a change has happened?

Cultural Ethos/Legacy

CADRE TRAINING | 11

SUSTAINABILITY

How Else Can You Say It?

Game changer Systemic change Lasting results A permanent difference Self-perpetuating Continuity Can continue without outside help Can exist on its own indefinitely The ‘buck’ does not stop here!

CADRE TRAINING | 12

Evaluation Plan OverviewJames Robinson

CADRE TRAINING | 13

EVALUATION PLAN OVERVIEW

M & E Activities13-14

14-15

15-16

16-17

17-18

18-19

19-20

20-21

21-22

Areas of Focus X X X

Strategic Partners X X X

Grant Model X X X

Sustainability Assessments X X X

Multi-year evaluation Site visits for 5-10% of Global Grants Study common project types in each area of focus Replicate sustainability check methodology

CADRE TRAINING | 14

M & E Framework

CADRE TRAINING | 15

TERMINOLOGY

Monitoring typically refers to the routine and ongoing process of collecting data as agreed upon before project implementation.

Evaluation generally involves a post-project assessment of the degree to which the project goals were met and the impact the project activities had on beneficiaries.

A Measure is a quantitative or qualitative description of a project’s activities or impact.

Slide 15

CADRE TRAINING | 16 Slide 16

ONLINE REPORTING


ONLINE REPORTING


Is monitoring and evaluation data required to close a global grant?

Yes. Sponsors must provide the minimum standard (at least one measure per AoF) for M & E data in the final report. This information is required to close the grant.

If Sponsors promised more M & E data than they can provide at the time of reporting, can close the grant if the minimum standard has been met.

How should M & E funds budgeted in a global grant be processed as part of grant closure?

Sponsors can utilize up to 10% of the grant budget for M & E.

Provide accounting of expenditures at time of final report. If applicable, sponsors may provide a plan to expend remaining funds post closure and report M & E data into online system

GRANT REPORTING REQUIREMENTS

CADRE TRAINING | 20

CHARACTERISTICS OF GOOD M & E PLANS

1. Less is More

Rotarians should be modest in the number of measures they will track…if it’s in the application, we’ll expect to see it in the report.

Advise Rotarians to limit the number of areas of focus they select in the application as this will multiply the number of measures required.

Slide 20

CADRE TRAINING | 21

2. Measures match core project components

Rotarians do NOT need to measure every project component. Instead, they should select the most important components and focus on measuring their success in those.

Measures should be relevant to the main goals of the project and should indicate success in reaching the goals.

Slide 21


CADRE TRAINING | 22

3. Mixture of output & outcome measures

Counting activities and participants is a good place to start when developing an M/E Plan, but falls short of demonstrating quality.

Strong M/E plans include measures that demonstrate the quality of the project in terms of changes in the behavior or knowledge of the beneficiaries.

Slide 22


CADRE TRAINING | 23

4. Reasonable attribution

– Measures should be closely and obviously connected to activities conducted and the project participants.

– Grant officers can advise Rotarians on what is reasonable attribution of project outputs to outcomes.

Slide 23

CHARACTERISTICS OF GOOD M & E PLAN

CADRE TRAINING | 24

5. Resources are dedicated to M/E activities Eligible items include:

– Contracted labor of a local org or paying stipends to community members to collect data.

– Consulting costs to technical experts to review and validate data.

– To purchase data sets and other information necessary to establish a baseline.

– Purchasing equipment such as phones, GPS equipment, and laptops to monitor activities.

– Local travel costs, such as car rental, fuel, and train fare to conduct site visits.

– Cost of electronic surveys; meeting costs for focus groups for evaluation purposes.

– Printing costs for manuals and other materials needed for data collection.

Slide 24


CADRE TRAINING | 25

5. Resources are dedicated to M/E activities

Ineligible items include:

– International travel to visit the project site.

– Accommodations, meals and per diems.

– Producing project promotional or recognition materials.

– Any measurement expenses that are incurred prior to the grant approval date, such as the costs of conducting a community assessment.

Slide 25


CADRE TRAINING | 26

Sustainability

CADRE TRAINING | 27

SUSTAINABILITY

For The Rotary Foundation, sustainability means:

ensuring that grants provide long-term solutions to community needs that the benefiting community can maintain after grant funding ends.

CADRE TRAINING | 28

Legacy Grants; Giving A Man A Fish

SUSTAINABILITY

https://www.google.com/imgres?imgurl&imgrefurl=http%3A%2F%2Fetc.usf.edu%2Fclipart%2F70300%2F70377%2F70377_262_rg-090_o.htm&h=0&w=0&tbnid=gvozojwibdzhpM&zoom=1&tbnh=225&tbnw=225&docid=7yoZB4Zh43KSPM&tbm=isch&ei=D21tU8COJIWxyAT4pIGQCw&ved=0CAIQsCUoAA

CADRE TRAINING | 29

Rotary Foundation

ensuresSustainable

impact ofgoods or services.

Clubs and Districts ensure sustainable

impact of goods and services

Individuals & organizations benefit

from sustainable impact of goods and

services

Community or organization ensures sustainable impact of goods and services

TRF ensures sustainable impact of goods and services

SUSTAINABILITY

Global Grants; Teaching A Man to Fish

CADRE TRAINING | 30

Sustainability Framework

CADRE TRAINING | 31

SUSTAINABILITY FRAMEWORK

CommunityNeeds/

Strengths

Materials/Technology

Funding

• Community Needs Assessment

• Meeting beneficiary needs

• Involving community partners

• Confirm local funding for long-term support

• Compensate project participants to ensure continuity of work

• Purchase equipment locally

• Involve community in selecting technology

• Train community to operate and maintain equipment

CADRE TRAINING | 32


Knowledge MotivationMonitoring/Evaluation

• Provide training to beneficiaries

• Create plan to transfer knowledge to new beneficiaries

• Collaborate with local organizations with expertise

• Give incentives to beneficiaries to provide ongoing support

• Identify leaders committed to outcomes

• Prepare the community to own the project when funds expended

• Develop measurable objectives

• Identify ways to collect data

• Establish baseline data to track the impact of the project

CADRE TRAINING | 33


CommunityNeeds/

Strengths

Materials/Technology

Funding

Knowledge

Motivation

Monitoring/Evaluation

High incidence of waterborne illness among children under 5 in a village

CADRE TRAINING | 34

Abt Sustainability Check

CADRE TRAINING | 35

Average Lifecycle Scores For MGs & GGs

1.001.502.002.503.003.504.00

Application &Proposal

Contract Point InTime/Final

Lifecycle Stage

Ave

rage

Sco

re (1

-4 P

ossi

ble)

Average Of All Grants

Average Of All GGs

Average of All MGs

Mid-Point

ABT SUSTAINABILITY CHECK

CADRE TRAINING | 36

Lifecycle Sustainability Scores; All Grants

1.00

1.50

2.00

2.50

3.00

3.50

4.00

1 2 3

Lifecycle Stage

Sco

re (

1-4

Pos

sibl

e)

India GG 25034

Kenya GG 25081

India GG 25027

Honduras GG 25065

Honduras GG 25363

India MG 74944

India MG 74942

Kenya MG 70520

Honduras MG 71320

Honduras MG 71446

Honduras MG 71940

Mid-Point

Cooperating Organizations

ABT SUSTAINABILITY CHECK

CADRE TRAINING | 37

Sustainability Matrix

CADRE TRAINING | 38

SUSTAINABILITY MATRIX

Eight Factors of Sustainability Evaluated

Grant Management

Outcomes

Financial

Community

Beneficiary Organizations

Cooperating Organizations

Training

Software/Equipment

CADRE TRAINING | 39


Methodology

Assessment is comprised of eight sections

Each section has 15-30 indicator questions

Each grant is assigned a score for each indicator within the applicable sections

Score is between 1-4

Each grant is evaluated independently

CADRE TRAINING | 40


Grant Management

The Grant Management section focuses on overall project management

Key indicators include:

Does the project have precise and finite objectives?

Is there an implementation workplan including milestones and benchmarks?

Were sustainable activities incorporated into the original project design?

Were the required progress reports submitted on time?

Was a sustainability plan included in the final report?

CADRE TRAINING | 41


Measuring Outcomes

The Measuring Outcomes section focuses on the measurability of a project, and whether or not appropriate M&E was conducted during and after the project


Does the project have clear and measurable outcomes?

Does the monitoring system include community involvement?

Are the project outcomes shared with the community and other stakeholders in ways meaningful for sustained project support?

CADRE TRAINING | 42


Financial

The Financial section focuses on budget planning and the financial role of the cooperating organization


Does the projected budget indicate sufficient revenue to fund the workplan?

Does the project Spending Plan (Budget) align with the implementation workplan line items (Activity Plan)?

Does the project budget indicate sufficient revenue to fund the sustainability activities beyond the end of the grant?

CADRE TRAINING | 43


Community

The Community section focuses on community involvement and buy-in for the project


Was a community needs assessment completed? Were recommendations made?

Does the workplan include activities to create or promote local community involvement and ownership?

Do grant reports describe community relationships developed which support the continuation of project outcomes beyond the life of the grant?

Does the grant identify community champions willing to assume leadership roles for sustaining project outcomes after the life of the grant?

CADRE TRAINING | 44


Beneficiary Organization

The Beneficiary Organization section focuses on the project’s relationship with the beneficiary organization and the organization’s capacity to take ownership of the project


Do written agreements define the terms of cooperation, roles and responsibilities of the beneficiary organizations?

Does the documentation identify personnel willing to assume leadership roles for sustaining project outcomes?

Does the local Club/District demonstrate strong partnerships with beneficiary organizations as appropriate to objectives?

CADRE TRAINING | 45


Cooperating Organization

The Cooperating Organization section focuses on the project management plan and structure of the relationship between the project sponsors and cooperating organization


Does the documentation include analysis and/or justification of cooperating organization(s) selection based on demonstrated expertise or experience?

Does the design integrate into the cooperating organization(s) regular systems or operations?

Does the cooperating organization agree in writing to participate in program financial reviews?

Does the documentation include a succession plan to hand over the project to the partner organization(s) at the end of the grant lifecycle?

CADRE TRAINING | 46


Training

The Training section focuses on the training and follow up associated with the project


Does the training plan address gaps identified in the assessment of beneficiaries targeted for training (doctors, teachers, unskilled labor)?

Are the training course design (delivery and materials) locally and culturally relevant (e.g., language, practical examples)?

Is there a measurement process to assess learner outcomes after training?

Is there a plan for keeping skills up-to-date?

Has the project resulted in demand for training outside the targeted beneficiary group (organization, individuals)?

CADRE TRAINING | 47

SUSTAINABILITY MATRIX SECTIONS

Software/Equipment

The Software/Equipment section assesses whether culturally appropriate technology was used during the project


Does the documentation include a justification of why equipment is needed?

Does the proposed equipment respect local social and cultural norms?

Was the availability of equipment on the local market considered?

Is there an inventory management system?

Do user guidelines exist for equipment use?

Are local spare parts and replacement equipment available?

Is equipment being used as intended?

CADRE TRAINING | 48

Questions?

CADRE TRAINING | 49

Areas of FocusJames Robinson

CADRE TRAINING | 50

AREA OF FOCUS EVALUATIONS – FUTURE VISION PILOT

Global Grants:

Sample of all Global Grants approved during pilot, 35 site visits across 5 areas of focus Nov 2013- Feb 2014

Nine countries on four continents

Focused on VTT and Humanitarian Projects, not Scholarships

10 Cadre and 4 Area of Focus Managers

CADRE TRAINING | 51

Pilot Evaluation - Area of Focus Site VisitsKorhan Atilla

CADRE TRAINING | 52

WHAT IS SUSTAINABILITY?

(is it QUALİTY?)

AND WHAT IT IS NOT?

(is it only QUALİTY?)

CADRE TRAINING | 53

MATTERS OF SUSTAINABILITY

1- Financial Sustainability

2- Maintenance

3- Proper Operation

4- Precautions during installation

5- Rotarian Involvement

6- Benefitting Community Involvement

CADRE TRAINING | 54

CADRE TRAINING | 55

CADRE TRAINING | 56

CADRE TRAINING | 57

CADRE TRAINING | 58

CADRE TRAINING | 59

CADRE TRAINING | 60

CADRE TRAINING | 61

CADRE TRAINING | 62

CADRE TRAINING | 63

CADRE TRAINING | 64

CADRE TRAINING | 65

CADRE TRAINING | 66

CADRE TRAINING | 67

CADRE TRAINING | 68

CADRE TRAINING | 69

CADRE TRAINING | 70

CADRE TRAINING | 71

CADRE TRAINING | 72

CADRE TRAINING | 73

CADRE TRAINING | 74

CADRE TRAINING | 75

CADRE TRAINING | 76

CADRE TRAINING | 77

CADRE TRAINING | 78

Questions?

Rotary Foundation Cadre Training: Monitoring and Evaluation

Government & Nonprofit

Transcript of Rotary Foundation Cadre Training: Monitoring and Evaluation