CARE NEPAL

PROJECT DESIGN,

MONITORING & EVALUATION STRATEGY:

August 18, 1998 Contact Persons: Marcy Vigoda Assistant Director Balaram Thapa Program Development Coordinator

F:\ARCHIVE\STRATEGY\DM&E\STRAT002.DOC

Table of Contents

1. Introduction ............................................................................................................................. 1 2. Our terminology ...................................................................................................................... 1

2.1 Project Hierarchy Terminology ................................................................................. 1 2.2 Indicators ................................................................................................................... 2 2.3 Monitoring and Evaluation Terminology .................................................................. 3

3. Overall approach to design, monitoring and evaluation.......................................................... 4 4 What we do ............................................................................................................................. 5 5. Strengths and weaknesses ....................................................................................................... 7 6. Goals and future plans........................................................................................................... 12

ANNEXES: Annex 1: Hierarchy of Objectives Annex 2: Key Steps in Project Design Annex 3: Guidelines for Monitoring and Evaluation Plan Annex 4: Current Project Monitoring and Evaluation Systems

Abbreviations

AIP 1.2 ..................................Annual Implementation Plan (activity plan) AOP .......................................Annual Operating Plan (prepared by each country office) API .........................................Annual Portfolio Information, basic data required by CARE

USA and CARE Canada ARI.........................................Acute Respiratory Infection CO..........................................Country Office DM&E....................................Design, Monitoring and Evaluation FCHV.....................................Female Community Health Volunteer HLS(A) ..................................Household Livelihood Security Assessment M&E ......................................Monitoring and Evaluation MER.......................................Monitoring, Evaluation, Reporting (a software program

being piloted by CARE) NGO.......................................Non-governmental organization PIMS ......................................Project Information Management System (output

monitoring formats for CARE Nepal) PIR .........................................Project Implementation Report (six-monthly reports) SNV........................................Netherlands Development Organization

1. Introduction For several years, CARE Nepal has emphasized the improvement of design, monitoring and evaluation systems (DM&E), in order to better demonstrate the results and impact of our work. CARE Nepals annual plans have explicitly and extensively addressed monitoring and evaluation issues since FY94. Accordingly, a number of steps have been undertaken in the past several years to improve DM&E in CARE Nepal. These derived from the recognition that while we felt we were doing very good work, it was not well documented. One of the major and unique challenges for CARE Nepal is to establish DM&E systems that capture the diversity of CARE Nepals multi-sectoral approach, while being manageable in size and scope. There have certainly been improvements in the last several years. Project designs are high quality team efforts, understanding and skills in M&E and systems are stronger, and CARE Nepal has developed a number of innovative monitoring and evaluation tools. There is still however scope for further improvements, approaches for which are suggested in this paper. This paper begins with a section on terminology, and then describes developments over the past few years before moving into the strategy itself. This is important so that CARE Nepal staff and other interested readers can be sure that they are talking about the same thing, thus promoting a useful and constructive dialogue among colleagues. In preparing this paper it often seemed that project design was something of a poor cousin to monitoring and evaluation, not receiving the same attention. This is not surprising, insofar as project design is a periodic activity while monitoring and evaluation are far more regular and require sustained attention. A lesson learned, however, is that improved project design is intimately related to better M&E. 2. Our terminology This section on terminology borrows heavily from the Discussion paper on monitoring, evaluation and impact assessment in CARE Nepal, April 1996 and from CARE USA Program Measurement Task Force documents. It is repeated here so that users of this strategy are all working from a common understanding. Any discussion of improving CAREs design, monitoring and evaluation needs to begin with a clarification of terms. 2.1 Project Hierarchy Terminology Below, the key terms input, output, effect and impact are distinguished. Then, working definitions for monitoring and evaluation are given. After each of these, there is a description of what is currently done in CARE Nepal.

Output The direct result of process; products of project activities

Impact Changes in human conditions or well-being

Effect Improvements in access to, or quality of, resources or systems, and changes in practices

Process Interventions or activities done by project utilizing the inputs

Input Resources need by project (ie funds, staff, commodities, in-kind)

This terminology corresponds directly to what has been used in the past in CARE:

Table 1: New and old CARE terminology Old terminology New terminology

Final goal Impact Intermediate goal Effect

Output Output Activity Process

Input Input The new terminology more precisely and accurately describes what is being addressed by each goal level.1 Normally, inputs, processes and outputs are measured through routine monitoring. Effects can sometimes be tracked through monitoring, but are more usually measured during an evaluation, and require surveys, samples, and comparisons with baseline status. Impacts are also measured during evaluations, but may require post-project evaluations before they are evident. Annex 1 provides more detailed information on each level of objective. 2.2 Indicators An indicator is a measure or criterion used to assist in verifying whether a proposed change has occurred. Indicators are quantitative or qualitative criteria for success that enable one to measure or assess the achievement of project goals. There are five general types of indicators:

Box 1: Different Types of Indicators Input indicators: Describe what goes into the project, such as the number of staff, the

amount of money spent. Process indicators: Describe the activities undertaken by the project, such as the

number of management training sessions for womens groups, the number of trainings on integrated pest management, etc.

Output indicators: Describe results of project activities such as the number of Leader Farmers trained, the number of womens groups strengthened, or the number of bridges constructed.

Effect indicators: Describe the changes in systems or behaviours/practices, such as the percentage of farmers utilizing integrated pest management, percentage of communities with access to contraceptives.

Impact indicators: Measure actual changes in conditions of project participants, related to the basic problem the project is addressing. This might include changes in livelihood status of the target population, health and nutritional status, wealth, etc. Some impacts, such as fertility, cannot be measured either within the life of the project or even at the end of the project: such measures take more time.

1 Sometimes we use different terminology where that is very much preferred by donors. For example, in the Family Health Project we have a goal (impact), purpose (effects) and then outputs and activities (processes).

Input and output indicators are easier to measure than impact indicators, but they provide only an indirect measure of the success of the project. In the past, we have taken it for granted that if we achieve our intended effects (intermediate goals), and, if the assumptions we have identified hold true, then we can assume that our intended impact (final goal) is achieved. But now we are aiming to try to actually measure the impact. We often use indirect, or proxy, measures for those things which are not easily measured directly. For example, because fertility is difficult to measure, we instead measure the contraceptive prevalence rate. Similarly, because income is difficult to measure, we may instead measure household consumption patterns (assuming that if income rises, households will spend the additional money on certain items). Note that because most CARE Nepal projects are multi-sectoral, impact goals tend to be broader than in other missions. For example, a typical impact goal is about achieving livelihood security for a certain target group rather than focusing on an individual aspect of livelihood security, such as improved educational or health status. 2.3 Monitoring and Evaluation Terminology The diagram to the left shows that monitoring happens far more frequently than evaluation.

EvaluationMonitoring

Monitoring Evaluation

Monitoring

Monitoring Monitoring

Monitoring is, the systematic and on-going collection and analysis of information for the management of project implementation. With the information generated by monitoring, project staff can track implementation of activities and compare them to project plans, and make informed management decisions (CARE USA Program Manual). As noted above, we can also monitor progress towards intended effects, and sometimes even project impacts. Staff and/or participants normally monitor. Monitoring should always serve as a management tool for field staff, supervisors and managers, providing them with information that can be used to adjust activities or strategies. Evaluation can take place during a project, at the end of a project, and sometimes years after project completion. It is, the broader investigation of the design, implementation and results of a project. Evaluation (may) identify strong and weak points of project design and implementation; determine progress toward effect and impact achievement; identify project results, both positive and negative, both intended and unplanned; identify lessons learned; and assess if a project has, or probably will have, sustainable, substantial impact. (Adapted from CARE USA Program Manual) The focus on lessons learned is particularly important, especially as a contribution to improving existing projects and designing better ones.

Although in reality all of us evaluate projects all the time for example during field visits here we refer to periodic formal assessments of projects. There are different kinds of evaluations, such as formative evaluations which take place during the life of the project and are intended to improve the design of the project; end-of-project evaluations which assess goal achievement and document lessons learned; and post-project evaluations which typically are done two to three years after project completion, and assess the impacts the project has had. 3. Overall approach to design, monitoring and evaluation CARE Nepal is utilizing an overall approach to project design, monitoring and evaluation, which is outlined below. This table is consistent with recent thinking in CARE USA (as outlined in documents at the 1998 Asia Region Household Livelihood Security Conference). Table 2: Design, monitoring and evaluation activities Activity Outputs 1. LRSP CO plan including vision and future directions, comparative advantage, targeting, broad

choice of partners, donor prospects: provides framework for future project designs 2. Situational Analysis to identify new programming areas

Multi-sectoral study in intended project area, involving both secondary and primary data collection. Primary data collection largely qualitative, intended to result in stakeholder analysis, problem analysis and development of project interventions, and thereby facilitate project design process

3. Project Design Project proposal document, including logical framework (Where appropriate, indicators may be drawn from CARE USA sectoral logframes or HLS indicators.) Ideally, a project monitoring and evaluation plan, which guides all data collection activities in the project, should be included in the design, but this more typically occurs at project start-up. (See Annex 2 for a summary of the design process.)

4. Baseline study Data needed to measure goal achievement; data to assist in more specifically designing interventions.

5. ONGOING: Project monitoring

The monitoring that is undertaken is driven by the M&E plan. (See Annex 3 for guidelines on developing M&E plans.) Monitoring provides regular data to measure progress against annual plans and project goals (specifically, activities, outputs and to a more limited extent, effects). Monitoring outputs include short reports, progress shown towards indicators; community-generated reports for community monitoring.

6. PERIODIC: Small-scale activity evaluations/case studies

Documents process, outputs, effects and impacts of specific interventions or communities. Represents a form of monitoring and evaluation.

7. Mid-term evaluation

Assess implementation approach and progress towards goals. May result in revised implementation strategy. Typically includes project self-evaluation, community evaluation and external evaluation.

8. Final evaluation Assesses goal achievement, and lessons learned which can be used for future project designs. Again, normally includes project self-evaluation, community evaluation and external evaluation. Baseline needs to be followed up.

9. End of project report

This may be covered off by the follow-up to the baseline. If there is not a proper baseline, this follow-up may be replaced by this report which provides a snapshot of the status of interventions and groups. (How many groups? What is their status? How many water systems? What is their coverage? What is their status? Are water tariffs collected?) This is necessary to enable post-project assessments.

10. Post-project evaluation

Assessment of the sustainability of institutions, practices and/or activities facilitated by the project.

4 What we do Below is a summary table that summarizes the major activities and formats used to monitor and evaluate projects. It separates out what is done at the project, country office, and global levels. Following the table is a more extensive description of monitoring and evaluation activities. Table 3: Hierarchy of monitoring and evaluation activities: Monitoring/ evaluation of:

Project formats used: Mission formats used:

CARE global formats used:

Process: Sectoral monitoring formats/PIMS

AIP 1.2/PIRs MER (on a pilot basis)

Global Achievement Formats/PIMS

Mission AOP API

Outputs Sectoral monitoring formats/PIMS

AIP 1.2/PIRs Assessment of progress against

logframe (this is included in PIRs)

MER (on a pilot basis)

Global Achievement Format/PIMS

Mission AOP API

Effects Sectoral monitoring formats/PIMS (to a very limited extent)

Case studies Project evaluations Spider Model Organizational capacity

assessment tool (for NGOs, VDCs, DDCs)

MER (on a pilot basis) Assessment of progress against

logframe (this is included in PIRs)

Small scale evaluations Mission AOP

API (to a limited extent)

Impacts Project final/post-project evaluations

Case studies MER (on a pilot basis)

Small scale studies

(Note that different levels of monitoring may be reported in the same format, e.g. the API reports mostly on outputs, but also on effects.) Note also that this matrix excludes the monitoring of inputs such as finances, personnel and procurement. Monitoring Activities Sectoral monitoring formats, part of an overall Project Information Management

System (PIMS), which includes standard formats that projects use for interventions that are applicable. Information generated in the PIMS provides the input for most other output monitoring reports that are required by CARE at different levels. The PIMS collects information on most activities/interventions supported by CARE Nepal projects. Each project then decides which formats are appropriate for them to use.

The PIMS includes recording and reporting formats to be completed at the site, project and country office levels, consolidating information as it moves upwards. The squares

below show each of the different levels of information involved in the PIMS. The circles below the squares show the other formats to which the PIMS data directly contribute.

Project Office recording formats (consolidation of all siteoffice formats)

Central Office Formats (roll up of all projects)

Project Office ->Central Office reporting formats (information needed at central office to complete global achievement formats, etc.) Sectoral

databases Global

Achievement roll-up

Global Achievement

AIP 1.2 API

Site -> Project office reporting

ion

Office)

formats (roll up of informatsent to Project

Site office recording formats

Regular six-monthly reporting of AIP 1.2 activities by staff, results of which are reported

and analyzed in the six-monthly PIR. (Information generated largely from PIMS.) The PIR also includes reporting of progress against logframe indicators at the effect level.

Annual reporting of key, selected outputs in the Global Achievement Formats. (Much if not total overlap with PIMS data.)

Monitoring of organizational capacity through innovative tools like the Spider Model for community-based organizations, and organizational capacity assessment tools for NGOs and Village Development Committees (local government institutions).

Annual reporting of outputs and to a limited extent effects in the CARE USA Annual Portfolio Information. (Some overlap with PIMS.)

Small-scale evaluations of interventions, which have included the improved cookstove program and kitchen gardening program in FY95; non-formal education and latrine program in FY96/97; income-generation activities and partnership in FY97/98. FY99 evaluations to be planned.

Case studies, which aim to document the process, outcomes and lessons learned from discrete interventions or programs in a single community, have been completed.

Participation in MER Pilot Project: MER or Monitoring, Evaluation, Reporting is an automated system originally developed in CARE Honduras, and now being piloted globally. The MER system is an automated one that can facilitate the collection and analysis of monitoring and evaluation data. However, it depends on strong, logical project designs, and perhaps this is its greatest strength, forcing projects to have such designs for the system to work well.

Evaluation Baselines: Baselines are done for all new projects. These typically include a structured household

questionnaire and participatory methods. Health/family planning baselines are done

separately. For household questionnaires, a cluster sampling approach (meaning that clusters in a randomized sampling of wards are surveyed) is used.

A follow-up to the original 1995 health/family planning baseline was done in mid-1997, yielding useful and important data. This report has been summarized in a four-page document, suitable for wide dissemination.

Project/Small Scale Evaluations These are done in accordance with contract requirements, typically involving a mid-term

and final evaluation, used to strengthen implementation of current projects and contribute to the design of new projects. Mid-term evaluations are usually process focused; while they do look at results, they are more interested in the approaches undertaken by the project and whether these are likely to lead to the stated objectives of the project. (A recent design exercise, for the follow-on phase of three CARE Danmark supported projects, involved a workshop which began with a one-day exercise compiling and discussing lessons learned from recent evaluations.)

The small-scale evaluation series has continued (see above, under monitoring). Special studies, such as the Field Staff Workload Study, aimed at better understanding

current processes, implementation modalities, and how they could be improved to more effectively achieve project goals.

All project evaluations include both a project self-evaluation (usually facilitated by a staff person from central office), as well as the formal external evaluation. A number of evaluations also include an element of community participation.

Post-project evaluations are planned for several projects. This includes the Begnas Tal Rupa Tal Watershed Management Project (for which unrestricted funds will be required), Jajarkot Poverty Alleviation Project (for which funds are allocated in the project design).

5. Strengths and weaknesses The 1996 discussion paper outlined strengths and weaknesses of the system in place at the time. At that time, identified strengths included: A dynamic, evolving approach. Gender-disaggregated monitoring data. The use of evaluations and studies to improve programs. Community involvement in evaluation activities. The use of PIRs to reflect on progress and problems Staff interest in case studies and small scale studies. Major problems included the following (note that this information is a summary of the discussion in the 1996 paper): 1. Weak monitoring systems: We were regularly requiring additional data collection, and

often the data collected through slightly different formats yielded different numbers! There have been too many formats, often developed separately within each project, with quite a bit of repetition between the formats. When staff transferred, there was no way to ensure that those who replaced them had access to information about their working area, including what had been achieved, problems encountered, etc.

2. Weak indicators: Indicators were often plucked from thin air, and were not related to the existing baseline status or appropriate and realistic goals.

3. Inadequate information: There was an absence of tools to measure important interventions (e.g. group capacity). Baseline information related to effects was not available. Qualitative information on interventions was lacking, for example to understand the differences in men and womens perspectives on new crop varieties.

4. Weak links between project designs and monitoring and evaluation activities: indicators expressed in percentage terms (e.g. 40% of farmers adopt) were never translated into numbers, making it difficult for a field worker to know what the goal really meant.

5. Late and irrelevant information: Projects were busy collecting data for village profiles. While information is interesting, it is collected too late to be of use in planning interventions, or is not relevant (e.g. number of people in the district who have achieved different levels of schooling).

6. Quality of information collection: Information quality was compromised by the use of English-language formats, the involvement of staff in collecting impact-related data about interventions they had supported; inadequate support to staff in doing case studies; and methodological weaknesses of data collection methods (e.g. for crop yields). Moreover, the absence of clear definitions (what is a kitchen garden?) further compromised data quality and consistency. This is in large part related to inadequate staff skills in monitoring and evaluation.

7. How we use information: We had lots of numbers, but little analysis of their meaning, or their context. Data, once rolled up, is again in English, and cannot be easily used by field staff. Information was not well communicated in any language to projects. Often information was not used at all to improve designs, interventions and strategies.

8. Who is responsible: Responsibility for data collection, roll-up, analysis and dissemination was not clear. Case studies had the problem of not adequately involving central office staff, resulting in little discussion about design, methodology, etc, and were usually of poor quality.

Since then, there have been significant improvements. The following table outlines recent improvements, and also suggests areas to strengthen. Activities in bold are those which can conceivably be addressed in FY99: 2

2 Some of this is drawn from a session in the August 1997 CARE Nepal M&E Workshop.

Positive Changes Areas to Strengthen Design All but one staff member involved in the original project design

training has actively participated in a subsequent project design A situational analysis, using structured guidelines, is done for new

project designs. (This is similar to a Household Livelihood Security Assessment, though there are important differences between the two.)

Project designs are getting better; staff more involved in the design and are well oriented with new project designs.

Logframes, in particular, are much stronger. Goals and indicators are properly placed at appropriate levels (impact, effect, output, and activity).

All field staff are familiar with their project logframes, which are more actively and extensively used as a framework for project planning.

A proposal review mechanism is in place, to ensure design quality. A core team of four senior staff (in addition to others called upon periodically) reviews all proposals, in addition to which two or three external consultants are hired to review each proposal.

Projects have or are about to have monitoring and evaluation frameworks, and are able to measure impact. (See Annex 4 for a summary of this information by project.)

Proposal review criteria might be a useful tool for both internal and external consultants to utilize during the review process. (These could be either draft CARE USA criteria, or something developed in-house.)

Situational analysis guidelines are to be revised, drawing on the most positive elements of the Household Livelihood Security Assessment approach.

Consultation with government counterparts at early stages of design, either through participation in the design or regular consultation might avoid later delays in obtaining project approvals.

Positive Changes Areas to Strengthen Capacity Building

A core of senior staff (managerial and technical) has participated in project design training, and subsequently put it to use.

Four staff attended the MER workshop in India, and are able to support implementation of MER in CARE Nepals pilot project.

A further group of staff has participated in management training (especially in Denmark) where they have received logframe training.

All senior program staff participated in the M&E workshop in August 1997. The material has been revised, so as to develop a curriculum for project level workshops. As of the end of FY98, projects will have participated in M&E workshops, a product of which is a detailed project M&E plan.

Staff responsibilities for DM&E are clearly outlined in updated job descriptions.

Guidelines have been developed for: a) situational analysis for project design b) designing terms of reference for studies c) implementing case studies d) project self-evaluation e) community evaluation

The project M&E workshops have been well received. Staff are clearer on M&E concepts. However, follow-up training should be planned for so that staff consolidate new knowledge and skills and further expand these skills.

Project design training for new senior staff would be appropriate. Training for project counterparts, especially those who would likely be

involved in future project designs, should be considered and planned for, perhaps as part of training for new senior staff.

There is very limited capacity in community monitoring and this needs attention.

Baselines Baseline studies are done for all new projects. These studies typically use a mix of a structured household questionnaire and participatory methods, and are focused on gathering information related to the logframe that will enable an assessment of goal achievement.

Follow-up studies for health/family planning baselines have taken place, enabling CARE to measure progress and goal achievement.

In-house capacity should be further developed. Staff are good at collecting information, less so at analyzing it.

A guideline for designing baseline studies is under development and will be finalized by the end of FY98.

Positive Changes Areas to Strengthen Monitoring A Project Information Management System

(PIMS), aimed at improving the collection of output indicators and ensuring some consistency across projects, is being put into place in FY99. Some projects have been field-testing it in FY98, so that improvements made reflect actual field experience using it.

The MER system is being piloted in one project and, if successful, will be expanded to other new projects. The MER will be compatible with the PIMS, and other pre-existing data collection formats, such as the CARE USA/CARE Canada API.

PIRs now report against logframe indicators, which is good, as there is more orientation towards assessing progress made at the effect level

Projects are actively using internal studies (which straddle the boundary between monitoring and evaluation) to make adjustments to program implementation, though there are opportunities to further improve staff learning from studies.

There is more analysis of health data going on. The development of tools to assess group capacity

is an important and useful change. Most projects will have M&E systems in place.

MER has yet to be fully implemented in the pilot project: this will require a significant level of effort and commitment.

There is a lot of duplication in different databases (e.g. sectoral databases, global achievement format), with duplicate reporting of the same information. We are still collecting too much data!

There is still insufficient clarity about who is responsible for data collection and analysis. Not enough analysis happens. There is still not enough use of data to improve project implementation, despite improvements!

Terminology remains unclear (what is a kitchen garden, an improved kitchen garden).

Current systems as yet do not permit us to adequately measure coverage and adoption rates: what proportion of a population is benefiting or adopting project promoted activities or systems. CARE Nepal must carefully assess whether the PIMS addresses this important need.

We need to put more emphasis on effect level monitoring: if we are promoting agricultural technologies, we need to know during the life of the project whether farmers are adopting them. This does not mean a large survey. Participatory and qualitative methods can be used to assess this. For example, in addition to our own observations, a well-done set of focus groups, or a workshop with leader farmers, could be used to assess whether farmers are using new technologies and why or why not. Existing social maps could be used to actually track the number of farm households adopting improved technologies.

PIRs could focus more on analysis of monitoring data, major issues facing the project, the ways in which problems have been addressed, etc.

Strengthen two-way communication of monitoring data (sharing it with those who provide it).

Support implementation of monitoring and evaluation systems.

6. Goals and future plans The overall goal of this strategy is: To improve the impact of CARE Nepals projects through strong design, monitoring and evaluation. To do this, CARE Nepal wants to strengthen the capacity of CARE Nepal and its partners to: Do high quality project design, monitoring and evaluation, Move even further towards measuring impact, and Actively use monitoring and evaluation results to further improve project interventions.

To achieve this, the following areas will receive particular attention: 1. Strengthen in-house capacity in design, monitoring and evaluation. CARE Nepal has already taken steps to strengthen staff skills in these areas, and will put further effort into this. CARE will continue to use external assistance as needed, but does not want to be largely dependent on external inputs. 2. Development of project M&E plans to measure outputs, effects and impact, and include community-based monitoring. Projects already collect a lot of data, but certain things are still missing. Effect monitoring, discussed in the table above, can be further strengthened. Community-based monitoring, where we involve communities not only to measure outputs but effects and implementation modalities, provide a different perspective on the project which can be very valuable to improve interventions. 3. Learning from monitoring and evaluation. The small scale study and case study series are examples of activities that CARE Nepal uses to promote learning. More can be done to ensure that information is disseminated to all staff, and used in project planning. 4. Developing partners capacity in M&E. As CARE works more with local partners, they need to develop basic M&E skills, so as to report on their activities, and learn from successes and mistakes. The following table outlines M&E related activities in CAREs FY99 AOP. These represent key activities planned. These plans focus on internal capacity building as well as skills development for partners. In addition, the key area of data analysis is addressed. Another element is the development of mechanisms for community-based monitoring. Note that responsible persons and timelines, included in the AOP, are not described here.

Strategic Direction: Developing and applying impact-oriented monitoring and evaluation systems for institutional learning. Objective 1: Refine and implement M&E plans

Activities: Complete M&E workshops in all projects Apply improved approaches for project and community evaluation Continue MER piloting Strengthen community monitoring as a part of project monitoring process

in Jajarkot

Objective 2: Increase analysis and use of monitoring data

Activities: Establish and implement PIMS in the mission, including analysis,

synthesis and sharing of information Conduct one small-scale evaluation Conduct one case study Produce summaries of two studies or reports to share widely with outside

audiences

Objective 3: Strengthen M&E capacity of selected partners

Activities: Identify key partners in all projects requiring M&E capacity

strengthening Arrange training/workshop on M&E for key partners Facilitate M&E plan preparation for key partners

Structures CARE Nepal has made the conscious decision NOT to establish a separate M&E unit. Rather, responsibilities for developing and supporting monitoring and evaluation systems are shared among senior project staff and specifically included in annual Individual Operating Plans. For close to three years, through the end of FY98, CARE Nepal had the services of an SNV Development Associate who served as an Evaluation and Documentation Officer, and significantly contributed to improvements in this period. Outputs include all the baseline studies, a guide to conducting baseline studies, and three small-scale activity evaluations/studies (on literacy programming, income generating activities and community organization). During each of his assignments, he worked with CARE staff, thereby developing their capacity to conduct studies. CARE Nepal anticipates that these services will be replaced by: a) the use of local consultants b) senior CARE Nepal staff taking a greater role in these activities. CARE may hire a research officer, to support studies, strengthen project-level documentation, and work closely with external consultants on studies.

As well, the NGO support manager to be hired at the start of FY99 will have responsibility for monitoring and evaluation related to partnerships. This includes both refining and supporting use of tools to assess and monitor partner capacity (such as the organizational assessment tools), and supporting partners to obtain the training and advice they need on monitoring and evaluation. All levels of staff have important roles to play in implementing and supporting design, monitoring and evaluation systems. These include:

Staff Responsibilities Field-based staff Collect field-level data; aggregate it at the site level; analyze and use data to

improve implementation; support community-based monitoring initiatives; participate as needed in baseline studies.

Project sector heads

Aggregate data, analyze it at project level; ensure field staff have systems in place to collect accurate data; support their initiatives to analyze site-level data; take lead on community-based monitoring initiatives; participate as needed in baseline studies; conduct case studies with support from consultants/CO staff; contribute to project completion reports; assist in planning project self-evaluations.

Project Managers Ensure M&E plan in place; guide data analysis at project level, ensure that this is followed through sectorally and by site; guide project self-evaluations and community evaluation; oversee case studies on behalf of project; participate in project design teams as requested; prepare regular monitoring reports (PIRs).

Senior Technical Advisers/ Senior Training Officers

Establish/refine sectoral monitoring formats; aggregate and analyze mission data and ensure it is communicated with projects; support case studies; participate, as requested, in project self-evaluations; support external studies as required; participate in project design teams as requested. Identify partner M&E needs. Ensure that partners adequately involved in new project designs.

Project Coordinators

Ensure that adequate training provided to project staff to support monitoring and evaluation requirements. Ensure M&E systems in place; participate in project designs; guide project case studies; facilitate project self-evaluations; contribute to TORs for studies and support them as required; edit and help finalize PIRs and other donor reports.

Program Coordinator, Program Development Coordinator, Assistant Country Director

Ensure that mission M&E strategy is in place, revise strategy as needed, ensure that training resources are available to projects and CO staff; review project M&E plans to ensure quality control; take lead on new project design; review project designs; secure outside reviewers; plan and organize annual M&E activities (evaluations, etc); initiate community-based monitoring activities; support MER pilot project; ensure donors receive high quality reports.

Partnering Partnership with local organizations, as well as government, presents challenges with respect to monitoring and evaluation. Local organizations often have extremely limited capacity in M&E, while government partners often have their own systems that can be difficult to influence.

To date, CARE has worked with partners to jointly develop monitoring and evaluation indicators. In some cases (such as for health/family planning in Syangja), CARE has taken the lead on the baseline survey, involving local NGO staff in data collection and initial data analysis, with CARE taking responsibility for report completion. In other cases, this has been left to local organizations, but without fully addressing the issue of their capacity. It is critical that in agreements with partners we do the following: a) identify who is responsible for what kinds of monitoring b) with small emerging NGOs, ensure that we support the organization (directly or through

specialized organizations) to develop monitoring plans c) build into our agreements plans for CARE to do regular monitoring of NGO activities. With limited capacity partners, we should initially expect that they monitor up to the output level, and put initial emphasis on that, while at the same time doing some supportive (rather than policing) independent monitoring. CARE can take the lead on effect level monitoring. At the same time, efforts should be made to orient local groups to the importance of assessing the effectiveness of their interventions. Because many of our partners are based in the community, they are well placed to draw upon the observations and perceptions of local people about their interventions. Eager to demonstrate how well they carry out planned activities, our partners may be reluctant to openly share problems or difficulties being faced. We need to emphasize the importance of critical reflection. (This can perhaps be done by demonstrating it ourselves!) For baselines, the role of partners and CARE will vary. If we are doing a baseline for a new project area, indicators related to partners activities could easily be included, thereby avoiding extra work or duplication of effort. Partners can be trained to participate in the baseline exercise. In cases where an overall project baseline is not being done, or specific information is needed for new interventions supported by partners, CARE should ensure that partners have developed adequate plans for baselines, and support them as necessary. Efforts should be made to limit the collection of data to what is really essential. In FY99, CARE will begin to address partner capacity in M&E, but also recognizes that this is an incremental process, and that if our commitment to developing more equitable partnerships is high, we must also accept that M&E cannot therefore be under our control, and that quality will, at least initially, suffer.

Annex 2: Project Design Steps and Activities Necessary to Achieve Them Project Design Step Activities

1. Problem Statement Conduct problem analysis Conduct needs assessment Conduct institutional environment scanning (In CARE Nepal, this would normally be done through a review of secondary data, a field-based situational analysis involving a multi-sectoral team, and analysis of the data in a project design workshop)

2. Strategy Development Develop goals and indicators Formulate project strategy Develop outputs and activities (This would result in the development of a logframe, and short write-ups on the planned project strategies.)

3. Evaluation Plan Develop a monitoring and evaluation plan (this may or may not include key questions to be addressed in the evaluation)

(Elements of the monitoring and evaluation plan are implicit in the logframe. Where it is not feasible to develop a full monitoring and evaluation plan for the logframe, and this is done instead at start-up, there should be a description of major monitoring and evaluation activities.)

4. Operational Plan Develop implementation schedule 5. Resource Requirements Determine staffing requirements

Develop organization chart Write job descriptions Determine other inputs necessary to develop budget

6. Financial Plan Develop project budget This is adapted from a handout used in the CARE Asia Project Design Workshop developed by Rich Caldwell.

Annex 1: Hierarchy of objectives New CARE termi-nology

Old CARE termi-nology

What it means Who does it Who takes credit for it

When it occurs

Examples of objective by level

Examples of associated indicators

Input Input Resources neededby the project

Project staff use them (and are accountable)

100% attributable to the project

Within the life of project (continuously)

Process Activity Interventions oractivities done by the project

Project staff do it (and are accountable)

100% attributable to the project

Within the life of project (continuously)

Eight training courses conducted for FCHVs in ARI management and referral

# of courses conducted

Output Output Products directlyproduced by the project

Project staff produce it (and are accountable)

100% attributable to the project

Within the life of project (when processes bear fruit)

100 FCHVs trained in ARI management and referral

# of FCHVs participating in course # of FCHVs successfully completing

course

Effect Inter-mediate goal

Improvements in access to or quality of resources (ie changes in systems), and changes in practices

Beneficiaries/ participants do it, systems reflect it

Should be largely attributable to the project, with other influences relatively minor

Within the life of project (may require special study to measure)

Improved referral, treatment and care of children with ARI symptoms

# of children with reported ARI symptoms in one month prior to survey

% of children with ARI symptoms appropriately treated

Impact Final goal Changes in human conditions or well-being of the target population at household level

Beneficiaries/participants experience it

Attribution is difficult, with other influences substantial and inevitable

Sometimes measurable within life of project, but more likely requires post-project evaluation

Improved health status of children 0-5 years

ARI incidence rate

(This is adapted from work done in CARE Uganda.)

Annex 3: Guidelines for Monitoring and Evaluation Plans A monitoring and evaluation plan outlines what information needs to be collected through (and after!) the life of the project, in order to assess the completion of activities and outputs, and achievement of effect and impact goals. The plan should be developed during or immediately after project design. The plan is what will guide all data collection during the life of the project, and it should reflect the need for both quantitative and qualitative data. The following matrix can be used to develop the M&E plan. Note that depending on the data collection methods used, not all columns will be needed. For example, if the data source is a household questionnaire, then the data collection method will be the very same. However, if the data source is community PRA, then in the data collection method the particular tools to be used can be specified. Narrative Summary

Indicators Data Needed

Data Source

Data Method

Frequency of Collection

Responsible Person(s)

Data Analysis

Dissemination and Utilization

Impact Goal

Effect Goals

Outputs Activities There is no column for assumptions in this matrix. However, if there are important assumptions in the project design that need to be periodically and systematically monitored, they should be included. Indicators: measures that determine whether change has occurred Data Needed: the data necessasry to characterize the indicator. In some cases, a number of different

types of data will be needed for one indicator; in other cases it may only be one. For example, if the indicator reads, % of households using latrines, the data needed will include:

total # of households # of households with latrines # of households where latrines reported/observed to be used (and a complication

here is that not all members of the household will necessarily use the latrine) Data Source: where to find the data, from whom to collect the data. Possibilities might include a

household survey; district-level secondary data; community PRAs. Data Methods: specific data-collection methods that will be used to obtain the data (e.g. Venn diagrams,

social mapping) Frequency of Collection: how often will the data be collected? (e.g. at baseline and end of project?

Monthly? Quarterly?) Person(s) Responsible: Who is responsible for collection and/or analyzing data? Analysis: How will data be analyzed? Will simple trends be developed? Averages? Will

statistical tests be done? Dissemination and Utilization: What reports will be generated from this information?

Annex 4: Current Project Monitoring and Evaluation Systems Project M&E

Framework? Monitoring

System? Baseline? Scheduled

Evaluations Impact Measurement?

PN-15 Solu In process Yes Yes See Family Health Yes PN-16 Mustang In process Yes No May 1998 No PN-17 Mahottari In process Yes Yes mid-term: 1/98 For health/family planning

final: late 99 PN-19 Bajura Yes Yes Yes mid-term: 2000 Yes

final: 2002 PN-23 Gorkha Yes Yes Yes Final: 5/99 YesPN-24 Syangja Yes Yes Yes mid-term: 2000 Yes

final: 2002 PN-29 Bardia Yes Yes Yes Final: 99/00 NoPN-30 FPP Yes Yes Yes Mid-term: 99 No (Another donor-supported

project tasked with impact measurement)

PN-31 Jajarkot In process Yes Yes mid-term, final and post-project

Yes

Family Health Yes Yes Yes mid-term, final(phase one), baseline, mid-term, final (phase 2)

Yes

August 18, 19981. Introduction2. Our terminology2.1 Project Hierarchy TerminologyTable 1: New and old CARE terminology

2.2 IndicatorsBox 1: Different Types of Indicators

2.3 Monitoring and Evaluation Terminology

3.Overall approach to design, monitoring and evaluationTable 2: Design, monitoring and evaluation activities

4What we doProject/Small Scale Evaluations

5. Strengths and weaknessesDesign

6.Goals and future plansStructures

Partnering

ActivitiesThis is adapted from a handout used in the CARE Asia Project Design Workshop developed by Rich Caldwell.Indicators