THE LIMITATIONS OF PROJECT DESK SCREENING AS AN INSTRUMENT OF DONOR AID MANAGEMENT* ABSTRACT: Donor agencies rely to varying degrees on desk reviews by headquarter staff for the selection, monitoring and evaluation of projects introduced by non governmental organisations (NGOs). If information provided in the various documents submitted by NGOs is inadequate or incomplete then the reliability of desk reviews as a management tool for allocation of project funding becomes questionable. A study of European Union funding of NGO projects compared findings from a desk evaluation of a structured sample of 30 projects with those from a field assessment of the same projects. The findings show that a field assessment on the basis of more complete information from a wider range of stakeholders leads to a markedly lower scoring on most of the standard criteria of project evaluation. An important policy implication is that ex-ante screening, monitoring and evaluation on the basis of project documents is an ineffective management tool for donor agencies if not backed up by substantive follow-up in the field. AUTHORS Nathalie Holvoet Robrecht Renard Institute of Development Policy and Management (IOB) of the University of Antwerp, Belgium 1. INTRODUCTION There is an on-going discussion within many donor agencies about the appropriateness of different forms of aid delivery. Whereas development projects have long been the major category of aid, a notable shift is occurring towards activities that are less circumscribed in time and space such as sectoral programme, debt relief and budget support (Howard, 1996). The latter forms of aid delivery allow a quick disbursement of financial resources with reduced transaction costs to both donor agencies and recipient countries. There is also general recognition that compared to project financing such “higher level” types of financing facilitate the integration of external funding in the administrative procedures the recipient country, thus increasing sustainability. Objectives formulated at the level of sectoral budget support for instance can more easily be integrated in the overall strategy designed by the recipient. By contrast, linkages between specific objectives of individual projects and the sector and macro priorities of the recipient country are often diffuse or at least indirect (Stokke, 1991). The delivery of non-project aid also tends to be less donor-driven and gives fuller responsibility to recipients in deciding how resources will be spent. This so-called recipient “ownership” is generally felt to constitute an important contributing factor to success. This is linked with a second evolution consisting of an enhanced role for decentralised public authorities and non-governmental organisations (NGOs) and other civil society actors. For the donor countries this leads to a considerable increase in the number of partners that have to be dealt with. In addition to the central government and a number of line ministries, an important public donor in an aid-dependent country may now be funding several dozens of local authorities and northern (developed country) and southern (developing country) NGOs. This has only been

* Paper presented at the European Evaluation Society Conference, Sevilla, October 10th – 12th 2002

1

possible because donors at the same time have been reducing considerable the administrative workload, especially in their dealings with northern NGOs. The changing relationship between official aid agencies and northern NGOs is in particular reflected in changes in funding arrangements that have been taking place over time. Not only are larger volumes of official aid channelled through northern NGOs1, as just indicated, agreements about what activities are financed, how progress and results are reported and accounted for are also gradually changing. Whereas project financing was in the beginning the major basis for NGO funding, since the 1970s many aid agencies have gradually shifted towards ‘envelope’ funding schemes where each NGO is awarded a financial sum allowing a series of often unrelated projects and institutional support activities to be carried out in a particular period of time. Sometimes this is called ‘programme funding’, referring to the fact that the NGO is appraised on the total annual or, more typically, multi-year programme it presents rather than on all the individual project components2. A 1993 OECD study on the evolving relationships between donors and NGOs and its 1999 update3 showcase examples from the Swedish, the Dutch, the Norwegian, and more recently the Belgian public aid agencies, who made strong strides in the direction of multi-year programme funding. Others like the US, Denmark and the United Kingdom followed the same pattern by extending programme funding to the larger, more experienced NGOs, while continuing to offer traditional project financing for smaller and shorter term projects to smaller organisations. Criteria for allowing admission to the more attractive programme funding include the managerial and institutional capabilities of NGOs, their overall past performance and the consistency of their goals with the general development policy of the official aid agency. Under programme grants donor agencies no longer request detailed ex ante information about individual projects but grant funding on the basis of an assessment of the institutional capabilities of NGOs with regard to identification, implementation, monitoring and evaluation (Samset, Forss and Cracknell, 1994). Simultaneously, and intimately related to the foregoing, the core of the public donor’s management of NGO financing gradually moves from ex-ante project screening towards monitoring and evaluation of general performance. Notwithstanding the shift away from project towards programme funding, there remain significant pockets of project funding in donor agencies’ aid portfolios. Some OECD countries continue to use project financing for the smaller and less experienced NGOs while it was until recently the only basis for collaboration between official aid agencies and NGOs in some member states like Spain and Italy (Smillie and Helmich, 1993, 1999). In order to facilitate the administratively demanding task of management on a project-by-project basis and to improve their appraisal system donor agencies have invested in methodologies and tools that standardise project identification, appraisal, implementation, monitoring and evaluation. USAID took the lead and has implemented the logical framework method since the beginning of the 1970s. Most other Western donor agencies have introduced the same or a slightly adapted methodology4 during the 1980s (Berlage and Stokke, 1992). Despite the fact that the logical framework method and its application have over time generated a fair amount of criticism5, the method is generally acknowledged to offer a comprehensive framework for project identification, appraisal, implementation, monitoring and evaluation. The format required for the documents in each of the phases of the project cycle also derives from the framework and is the basis for project appraisal (selection), project monitoring and project evaluation. For donor agencies involved in project financing of Northern NGOs, it remains extremely difficult, even under ideal standardised reporting procedures, to keep track of what is really going on in so many different countries, sectors, and local settings. There are two reasons for

2

this. First, donors lack the means to monitor in the field such a wide array of activities. Secondly, project review at headquarters is administratively extremely demanding and thus tends to be performed incompletely. One may thus question whether headquarter-based procedures permit aid bureaucracies to acquire a credible insight in project quality. Do project documents entered at different stages of the project cycle allow understanding a project and assessing its success? Is the quality and quantity of the information provided satisfactory so as to enable the desk evaluator to make a sound judgement about project quality? And more specifically, how close does a desk study come to the more complete and thus presumably better field assessment? In what follows we report on an experiment that was set up to shed some light on those questions. 2. CASE STUDY The experiment was fully integrated in a larger ex post evaluation study of the B7-6000 budget line, the European Union’s (EU) major co-financing scheme for European NGOs (ENGOs) 6. The scheme is a case of project-by-project financing in the above sense, meaning that an ENGO has to introduce a separate demand for every individual project it wants funded. A structured sample of 30 projects introduced to B7-6000 was studied for the period under review, 1994-1999. This was done independently by a desk evaluator on the basis of documents available in Brussels, without the benefit of any field visits, and by a different group of evaluators who had all the information from Brussels but in addition visited the projects in the field. We then compared the findings of the desk review with those of the field reports and tried to interpret the differences between them. In the period under study projects on the budget-line B7-6000 had to be submitted by ENGOs to the offices of the European Community in Brussels. All project documents and reports had to be structured according to the same ‘Basic Format’, based on the EC’s Project Cycle Management method (PCM)7. The criteria making up the ‘Basic Format’ may be regrouped under the following headings (Eggers, 1998):

• background: the identification of the underlying policy, the beneficiaries and problems to be solved

• intervention: the objectives of the project (project purpose) and the activities identified to reach the objectives

• assumptions: the risks and external factors which are thought to be essential for the project’s success but which are not controlled by the project

• implementation: the concrete project description (means, costs, procedures, time-table) • sustainability factors: the policy, technological, environmental, socio-cultural,

management and economic factors which are included in the project design as to increase sustainability

• monitoring and evaluation: the instruments that will be used to monitor and evaluate the project

The standardised project documents were the major source of information for project officers at the EC-headquarters in Brussels8 and constituted the basis for decision-making on project financing, continuation and/or renewal. In our experiment, the desk evaluator, one of the authors of the present paper, simulated the Brussels decision-making process by using the same information at disposal of the typical EU project officer. More specifically she used project identification reports submitted by the ENGOs, notes from field offices and thematic desks and, depending on the particular stage of the project cycle, reports on implementation and evaluation. Interim and end-of-project reports were available for 19 and 11 projects out of 30 respectively. Evaluation reports were included in one

3

third of the interim reports and in four fifth of the end-of-project reports. A team of field evaluators then repeated the same evaluation exercise. The team always consisted of a senior northern evaluation expert and a local southern consultant who was familiar with the country, the local culture and the language. By design, the field evaluator team had more and better information than the expert who did the desk study. They went through the same project files as the desk evaluator but obtained additional material from the offices of the ENGOs and in particular through the field visits. During the field assessment, they spoke at length with the different stakeholders of the projects including the beneficiaries, a number of local non-participants, the project staff and the southern implementing NGOs. 3. SAMPLING The study covered the period 1994-1999. Over this period, 2582 projects9 were accepted for EU funding on the B7-6000 budget-line. Out of the 2582 projects, 30 projects were selected. For reasons of time and resource constraints, the 30 projects were selected from 6 countries, located in 5 different regions: India (more especially the states of Karnataka, Tamil Nadu and Andhra Pradesh) in South Asia, Senegal in West Africa, Kenya in East Africa, Cuba in the Caribbean and Central America, and Bolivia and Brazil in South America. These countries were selected after interviewing EC-staff and were judged to be representative enough for the problems and challenges met in the overall population of projects. About one fourth of all EU-co-funded projects in the period 1994-1999 took place in these 6 countries. The 30 projects were further selected in such a way as to cover four broad areas of EC-funding: socio-economic, human resources, political/institutional and multisectoral10. A number of important sectors for the first three broad areas of intervention were identified: agriculture, investment and environment within the ‘socio-economic’ area, health and education within ‘human resources’, ‘women and development’ and capacity building within the ‘political/institutional’ area of intervention. Despite the small sample size, an effort was made to also take into consideration the size of the ENGO, the country of the ENGO, the size of the project, and its administrative status, all of which might possibly influence project performance. Table 1, which gives an overview of sample characteristics, shows that the sample is fairly representative of the overall population. Large ENGOs were unintentionally privileged, and to a lesser extent large projects. The latter bias is a result of the former as larger NGOs tend to identify larger projects. The overrepresentation of large ENGOs and large projects needs to be taken into account when extrapolating from our sample findings to the overall population. Larger NGOs tend to enjoy economies of scale: they have better-equipped departments, they hire more specialised personnel, develop more in-house knowledge and institutional memory, and tend to have better analytical capacity. Importantly, they also develop more elaborate networks of local collaborators, and tend to select the stronger local partners, both of which should favourably influence project screening. They can also more easily mobilise financial and human resources to address contingencies. One can thus expect them to identify the better projects, to write better project proposals, to implement and monitor better and to write better monitoring and evaluation reports. This is of course not to say that there is a perfect correlation between the size of the ENGO and the size of the project or between the latter and the quality of project documentation. Neither the size of the NGO or the project guarantee project quality: large NGOs can become bureaucratic and ineffective.

4

Table 1: Overview of Sample characteristics Sample Characteristics Number

(sample) Percentage (population of 2582 projects)

Recipient Country India (Tamil Nadu, Andhra Pradesh, Karnataka) Senegal Kenya Cuba Bolivia Brazil

6 5 5 5 3 6

6.5 2.2 2.9 1 3.2 9.3

Area/sector of interventiona

Local socio-economic development Agriculture Industry Environment

Development of human resources Education Health

Political/Institutional Women and development Capacity building

Multisectoral

8 6 1 1 11 8 3 5 2 3 6

30.1 15.9 1.7 6.6 43.2 31 10 13.1 3.1 7 13.6

Size of project Small (up to 200 000 ECU EC contributionb) Medium (from 200 000 ECU EC contribution to 400,000 ECU EC contribution) Large (more than 400 000 ECU EC contribution)

11 10 9

37.2 33.9 28.9

Administrative status of project Engaged Closed

19 11

62.5 37.5

Size of European NGO Small (up to 1 000 000 ECU total turnover in 1999) Medium (from 1 000 000 ECU total turnover to 3 000 000 total turnover in 1999) Large (more than 3 000 000 total turnover in 1999)

6 8 16

59.2 22.8 18

Country of the European NGOc

Germany Italy United Kingdom France Belgium The Netherlands Spain Austria Ireland Luxembourg Portugal Denmark Sweden Finland Greece

5 5 4 4 4 4 2 1 0 0 0 1 0 0 0

20.5 17.5 17.3 15 7.3 6.3 5.4 2.7 2.7 2.6 1.9 0.7 0.5 0.5 0.4

Notes: a. The EU did not use a sector classification system before 1998. It was consequently impossible to obtain the frequency distribution over the different sectors for the overall population of 2582 projects. We ourselves classified projects into sectors on the basis of the project titles for all projects in the 6 selected countries (a sub-population of 648 projects, 25% of the overall population). The figures that are given in the second column apply to the sub-population of 648 projects. b. The EU financial contribution for projects was limited to 50% of the total project cost and to a maximum of 500 000 ECU. c. Countries are put in descending order according to the share of total EU-subsidies on budget-line B7-6000 in the period 1994-1999.

5

4. METHODOLOGICAL ISSUES On the basis of the specific information at their disposal the desk evaluator and the field evaluators independently assessed the 30 projects according to a number of pre-determined criteria. Table 2 gives an overview of the criteria, the corresponding answering categories and the rating scale. For each of the 30 projects, the performance criteria were awarded a rating based on a four-point scale ranging from weak (=1) to excellent (=4). When evaluators deemed it impossible to make a sound judgment on the basis of the information available or when they thought that particular performance criteria were not applicable to a project under study they indicated ‘impossible to determine’ or ‘not applicable’. A common objection to the use of a rating system for subjective criteria is that cardinal scores cannot be assigned on the basis of subjective data. Scores which are based on judgements and grading criteria are thought not be valid as they might be influenced by interpersonal differences in value judgements or severity of judgement. In our case it means that disagreements among scores attributed by desk and field evaluators might not stem from asymmetry in information only. Other influences are indeed difficult to rule out. One way to test for this would be to ask several evaluators to assign scores independently from each other but on the basis of the same information. In their desk study about the impact of participation on project performance, Isham et al (1995) asked two different coders to assign each of the 121 development projects studied a cardinal score on a number of indicators measuring participation and project performance. The coders did this on the basis of the qualitative information included in the project documents at their disposal. The degree of correlation between the two independent scoring sets was remarkably high, suggesting that translating subjective data into cardinal scores might be acceptable when the criteria used are straightforward and familiar. One obvious way of increasing inter-subjective consensus among evaluators is by training. In the case study reviewed here all evaluators were brought together in a one-week seminar in Brussels where definitions of performance criteria and rating scales were discussed and agreed upon. As will be commented later on in section 5, the analysis of our data lends support to the idea that we have been successful in reducing the influence of the subjective measurement error. Table 2: Overview of criteria for project assessment Variables Answering categories and rating system • initiative-taker of the project (Q1)

• ENGO • southern partner • local beneficiaries

• characteristics of intended beneficiaries (Q2)

• belong to deprived groups in society • do not belong to deprived groups in

society BENEFICIARY PARTICIPATION • beneficiary participation in project identification (Q3) • beneficiary participation in project monitoring (Q4) • beneficiary participation in project evaluation (Q5) • criteria for selecting project participants among potential

beneficiaries (Q6) • addressing of gender issues during project formulation

and implementation (Q7)

For Q.3. to Q.27 the same answering and scoring categories were used:

• weak (=1) • satisfactory (=2) • good (=3) • excellent (=4)

6

PROJECT RELEVANCE • relevance of intervention strategy (as set against the

major constraints of the region) (Q8) • relevance of the activities chosen (to obtain the project

objective) (Q9) • fit of action into those undertaken by other development

actors (Q10) • relevance of the measures taken to ensure project

sustainability (Q11) • project effectiveness (Q20) • project efficiency (Q21) • project impact (Q22) • project sustainability (Q23) • effect of the project on gender equity (Q24) • organisational capacity of the ENGO (Q25) • organisational capacity of the southern partner (Q26) • overall project success (Q27)

• impossible to determine/not applicable

5. ANALYSIS AND FINDINGS The first two questions listed in table 2 were treated separately as the format was different from the others. In the answers to these two questions, a large degree of consensus between the desk and field evaluation was revealed. To the first question, ‘who took the initiative for the project’, the desk evaluator and the field evaluators gave the same answer for 23 out of 30 projects. For four out of the seven remaining projects, the desk evaluator was of the opinion that the initiative had been taken by the local beneficiaries, whereas the field experts felt that the southern NGO (SNGO) had done so. In one other project it was the opposite: the desk evaluator thought the SNGO had taken the initiative, whereas the field evaluator thought it were the local beneficiaries. In only two projects was there a more serious disagreement, the desk evaluator estimating in each case that the SNGO had taken the initiative, the field evaluators estimating that the ENGO had done so. If one assumes, as we do, that the judgements made by the field evaluators are the more correct ones, as they were based on more complete information, the picture that emerges is that of ENGOs slightly exaggerating in their written submissions to the EC the degree of local participation. Regarding the second question, ‘do the intended beneficiaries belong to deprived groups within society’, desk evaluator and field evaluators gave the same answer in 22 cases out of 30. In two cases only did they disagree, the field evaluators being each time more convinced that the intended beneficiaries belonged to deprived groups within society than the desk evaluator. For the remaining six projects one of the experts (in all but one case the desk evaluator) but not the other felt that the available information did not allow to give a clear-cut answer (‘impossible to determine’). All in all, the answers to these two first questions reveal a close fit between the two assessments, suggesting that screening on the basis of submitted project identification reports can be performed with a reasonable degree of accuracy. However, the answers to the remaining 17 questions show a much wider divergence between the two assessments, and lead to quite different conclusions. Table 3 contains the basic information for 510 observations (17 questions for 30 projects) and highlights the difference in score between the desk and field evaluators. Numbers indicate rank differences obtained by subtracting the numerical scores by the desk evaluator from the numerical scores by the field evaluators. A

7

positive value thus indicates that the field evaluator gave a higher score than the desk evaluator, a zero that they gave the same score, and a negative value that the field evaluators gave a lower score than the desk evaluator. The possible range of the score differences is –3 to +3. When one team marked the ‘impossible to determine’ or the ‘not relevant’ boxes, whereas the other assigned a score, this was marked with ‘ID’ in the table11. An ‘ID’ cell in table 1 which is shaded grey signals that the field evaluators did not give a number score. The non-shaded ‘ID’ cells, 95 percent of the cases, signify that it was the desk evaluator who felt unable to provide a clear answer on the basis of the available information. When both ticked ‘impossible to determine’ or ‘not relevant’, we inserted a zero in the table, indicating the agreement among them.

8

Table 3: Comparison of desk and field evaluation of 30 projects

Variable n° 3 4 5 6 7 8 9 10 11 20 21 22 23 24 25 26 27 N° Beneficiary

country

CS IS PS NS TG

1 Senegal ID ID ID 1 0 -1 1 0 -1 ID ID ID ID ID ID -1 0 8 3 2 -3 52 Senegal ID ID ID 0 -1 -1 -2 ID 1 -1 ID ID ID ID 0 0 0 9 4 1 -5 63 Senegal 1 -1 ID ID 0 0 0 ID -1 -1 ID ID ID 0 -1 -1 ID 10 4 1 -5 64 Senegal 0 0 0 2 2 -1 0 0 0 0 ID ID ID 2 1 1 1 14 7 9 -1 105 Senegal ID ID ID ID ID ID ID ID ID ID ID ID ID 1 ID 1 2 3 0 4 0 46 Senegal 0 ID ID 0 0 0 0 ID ID -1 0 ID -2 0 ID -3 -2 11 7 0 -8 87 Kenya 0 ID ID ID -1 0 0 0 -2 -1 ID ID ID ID ID ID -1 8 4 0 -5 58 Kenya -2 -1 0 0 ID -1 0 1 -1 -2 ID 0 0 ID 0 ID -1 13 6 1 -8 99 Kenya ID ID ID ID -1 0 0 0 -1 0 ID ID ID 0 ID ID -1 8 5 0 -3 3

10 Kenya ID ID ID ID ID ID ID ID ID 2 ID ID ID ID ID ID ID 1 0 2 0 211 Kenya ID ID ID ID 1 -1 -1 0 -1 0 ID ID ID 1 ID ID -3 8 2 2 -6 812 India ID 0 ID 1 -1 -2 -1 -1 ID ID ID ID ID 0 ID -2 -2 9 2 1 -9 1013 India ID ID ID 1 0 0 -1 1 1 ID ID ID ID ID ID ID 1 7 2 4 -1 514 India ID ID ID 0 0 2 2 2 2 ID ID ID 0 1 ID 2 2 10 3 13 0 1315 India -2 -1 ID ID 0 ID 1 0 -1 0 ID ID ID -1 ID 0 0 10 5 1 -5 616 India ID ID ID ID 1 0 -1 -2 0 0 ID ID ID 1 ID 0 -1 9 4 2 -4 617 India 0 ID ID 1 ID ID 2 ID ID ID ID ID ID ID ID ID 1 4 1 4 0 418 Cuba ID ID ID ID -1 -1 -1 -2 0 ID ID 0 ID ID -2 0 -1 9 3 0 -8 819 Cuba ID ID 0 0 ID -1 -1 ID 0 ID ID ID ID 0 ID ID -1 7 4 0 -3 320 Cuba ID ID 0 0 ID -1 -2 ID -1 0 ID 0 0 ID ID 0 -1 10 6 0 -5 521 Cuba ID 0 0 ID 0 ID ID ID -1 0 ID ID -1 ID ID ID 0 7 5 0 -2 222 Cuba ID ID 0 0 ID -2 ID ID -2 0 ID ID 0 ID 0 0 -1 9 6 0 -5 523 Brazil 0 ID ID ID 0 0 1 ID 0 ID ID ID ID ID ID 0 0 7 6 1 0 124 Brazil ID ID ID 1 2 -1 -1 ID 0 ID ID ID ID ID 0 ID -1 7 2 3 -3 625 Brazil 0 -1 0 1 -1 0 -1 ID -1 ID ID ID ID ID ID 1 0 10 4 2 -4 626 Brazil ID ID ID ID ID 0 -1 ID -1 1 ID ID -3 ID -1 0 0 8 3 1 -6 727 Brazil ID ID ID ID ID 0 -1 ID 0 -1 -1 ID 0 0 ID ID -1 8 4 0 -4 428 Bolivia ID ID ID ID ID 0 0 -1 ID 1 ID 0 ID 0 0 ID 0 8 6 1 -1 229 Bolivia ID ID ID ID ID 0 -1 0 -1 -1 ID 0 0 ID 0 -1 ID 9 5 0 -4 430 Bolivia -2 ID ID -1 ID 0 0 0 -1 -1 ID ID -1 ID 0 ID 0 10 5 0 -6 6

CS 10 7 7 15 18 25 26 15 24 19 2 5 10 13 11 17 27 251 118 55 -114 169 IS 6 3 7 7 8 13 8 8 7 8 1 5 6 7 7 8 9 118 total total total total PS 1 0 0 8 6 2 7 4 4 4 0 0 0 6 1 5 7 55 total NS -6 -4 0 -1 -6 -13 -15 -6 -16 -9 -1 0 -7 -1 -4 -8 -17 -114 total TG 7 4 0 9 12 15 22 10 20 13 1 0 7 7 5 13 24 169 total

notes: CS : complete scores IS : identical scores PS : positive sum NS: negative sum TG : total gap Desk and field studies produced identical answers, indicated by a zero in table 3, in 118 cases, or 23 percent of the observations. This number includes the cases where both considered that it was impossible to assign a score. In 133 cases, or 26 percent of the observations, scores differed, indicated by nonzero numbers in table 3. In the other 259 cases, or 51 percent of the cases, one - but not the other - team ticked the ‘impossible to determine’ or ‘not relevant’ box, indicated by ‘ID’ in table 3.

9

When teams disagreed, the field evaluators tended to give lower scores than the desk evaluator. The sum of positive rank differences over all the cases where the field evaluator gave a better score than the desk evaluator is 55. The sum of negative rank differences is -114, almost twice as high in absolute value. This pattern is consistent over five of the six teams of field evaluators12 and strengthens the conclusion that differences in assessment between the desk and field evaluators are not due to subjective differences in judgement among evaluators. It is highly unlikely that differences in scoring between field and desk evaluators are primarily due to the fact that the desk evaluator was more lenient in her judgements than all but one team of field evaluators. The findings lend support to the thesis that the documents available in donor headquarters paint a picture of the projects that is too rosy, a result that does not come as a surprise. Some of the more detailed information contained in table 3 is brought out in the following graphs. Figure 1 plots the number of ‘complete evaluation outcomes’, out of a total of 30 per question. A complete evaluation outcome occurs when both evaluators assign a score from weak=1 to excellent=4, or when both indicate that no answer is possible or relevant. Huge differences are observed. Analysis of the data makes clear that the low scores are mainly due to

the desk evaluator not feeling able to assign a score on the basis of the information available to her. Identification, monitoring and evaluation reports submitted to the EC headquarters in Brussels did, for instance, not allow assessing whether beneficiaries had participated in monitoring and evaluation, or whether the project was efficient. The field evaluator, on the other hand, gathered enough information during the interviews with the project staff and beneficiaries to assess a project on these aspects.

Figure 1: Number of complete evaluation outcomes

0

5

10

15

20

25

30

Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27

Figure 2 focuses on the complete evaluation outcomes. The percentage of non-identical outcomes ranges from 0 tot 71 percent, with an average of 54 percent. Note in particular the hefty 67 percent disagreement regarding the summary assessment in question 27 (‘how do you rate the project’s overall success’). The apparent agreement on question 5 (‘did beneficiaries participate in project evaluation’) and question 22 (‘how do you assess project impact’) is mainly due to the fact that both desk and field evaluators found it impossible to answer the question, or judged the question not relevant, rather than that they agreed on a score13.

10

Figure 2: Non-identical complete evaluation outcomes

0102030405060708090

100

Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27

%

Figure 3 sets out for every question the positive and negative rank differences. Whereas the negative gap is more than twice as large in absolute size as the positive gap for to the summary assessment in question 27 (‘how do you rate the project’s overall success’), as it is for all questions combined, there are some exceptions where the field evaluators attributed higher scores than the desk evaluator had done. This is the case for question 6 (‘how do you rate the criteria used for selecting project participants among potential beneficiaries’) and question 24 (‘how do you assess the effect of the project on gender equity’)14. Figure 4 sets out the absolute gap between the scores of desk and field evaluators, without distinguishing between the negative and positive score differences. This basically gives an alternative presentation of the information already contained in figure 3.

Figure 3: Positive and negative gap between the desk and the field studies

-20

-15

-10

-5

0

5

10Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27

11

Figure 4: Total gap between the desk and the field studies

0

5

10

15

20

25

30

Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27

Figure 5 provides information on the average difference in score between desk and field evaluators, irrespective of who gave the higher score. The differences range from 0 to 0.9. The units are (absolute) rank differences between scores. For all the questions combined, the difference is two-thirds of one point. Figure 5 may be contrasted with figure 2. Whereas figure 2 measures the frequency of disagreements, figure 5 indicates their intensity. The two figures differ especially for questions 3 (‘did beneficiaries participate in project identification’) and 23 (‘how do you assess project sustainability’). In both cases figure 2 reveals that there were not so many instances of disagreement compared to other questions. Figure 5 indicates however that where evaluators disagreed, they did so more strongly than on other questions.

Figure 5: Average score differences

0.00.10.20.30.40.50.60.70.80.91.0

Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27

6. CONCLUSION AND POLICY IMPLICATIONS Our overall conclusion is that in the case of EU budget-line B7-6000, a Brussels-based assessment of projects was hazardous, whether before, during, or after implementation. If we abstract from subjective differences in the assessment by evaluators, the data in table 3, derived from a sample of 30 projects, can be summarised in three points. First, a large number of important questions relating to on-going and completed projects funded by the budget-line B 7-6000 (246 out of 510, or 48 percent) could not be answered on the basis of the information available in Brussels whereas a field visit enabled the evaluator to make an assessment in all these cases. Secondly, when the desk study did pick one of the four categories - weak,

12

satisfactory, good or excellent - this judgement is confirmed in less than half of the cases by the field visit (86 out of 216, or 40 percent). Thirdly, documents in Brussels overstate more than they understate the quality and achievements of projects by a factor of two to one (score-difference ratio of 114:55). These discrepancies between the desk and field studies are striking, and testify to the limited reliability of the former. This may be partly due to the fact that the reporting requirements imposed on the NGOs were not adequate, or that NGOs were allowed to comply very incompletely with these requirements in the period under study. We came across examples of both in the desk study, and there is no doubt that the system could be improved in this regard. There is however a more structural problem. It will always be extremely difficult, even under ideal reporting procedures, to keep track of what is going on in so many different countries, sectors, and local settings, without direct observation of the situation in the field, and with written information coming almost exclusively from an involved party. The findings of the experiment therefore by and large reinforce the arguments in favour of programme funding of NGOs. This modality reduces transaction costs for public donors and NGOs alike and makes room for more substantive monitoring and evaluation15.

13

References Berlage, Lodewijk and Olav Stokke (1992) ‘Evaluating Development Assistance: State of the Art and Main Challenges Ahead’, in L. Berlage and Olav Stokke (eds) Evaluating Development Assistance. Approaches and Methods. EADI Book Series 14, pp. 1-32. London: Frank Cass. Berlage, Lodewijk and Olav Stokke, eds (1992) Evaluating Development Assistance. Approaches and Methods. EADI Book Series 14. London: Frank Cass. Clements, P. (1995) ‘A poverty-oriented cost-benefit approach to the analysis of development projects’, World Development 23(4): 577-592. Cracknell, Basil E. (1994) ‘Monitoring and evaluation of public sector investment in the UK’, Project Appraisal 9(4): 222-230. Cummings, Harry F. (1997) ‘Logic Models, Logical Frameworks and Results-Based Management: Contrasts and Comparisons’, Canadian Journal of Development Studies 18: 587-596. Eggers, Hellmut (1994) ‘Integrated project cycle management: roots and perspectives’, Project Appraisal 9(1): 59-65. Eggers, Hellmut (1998) ‘Project cycle management revisited’, The Courier 169: 69-72. Commission of the European Communities (1993) Manual on Project Cycle Management (Integrated Approach and Logical Framework). Brussels: Commission of the European Communities, Directorate General for Development, Evaluation Unit. European Commission (2001) Manual Project Cycle Management. Brussels: European Commission, EuropeAid Office, General Affairs, Evaluation. Gasper, Des (2000) ‘Evaluating the ‘Logical Framework Approach’ towards Learning-Oriented Development Learning’, Public Administration and Development 20: 17-28. Isham Jonathan, Deepa Narayan and Lant Pritchett (1995) ‘Does participation improve performance? Establishing causality with subjective data’, The World Bank Economic Review 9: 175-200. Musto, Stefan A. (1991) ‘Evaluation Policy and Performance in the Federal Republic of Germany’, in O. Stokke (ed) Evaluating Development Assistance: Policies and Performance, pp. 127-148. London: Frank Cass. Development Assistance Committee (DAC) (1991) Principles for Programme Assistance. Paris: OECD. Overseas Development Institute (1995) ‘NGOs and Official Donors’, ODI Briefing Paper 4: 1-4. Samset, Knut, Kim Forss and Basil Cracknell (1994) ‘Evaluating systems when aid policies are changing’, Project Appraisal 9(1): 29-36. Smillie, Ian and Henny Helmich, eds (1993) Non-governmental organisations and governments: Stakeholders for Development. Paris: Organisation for Economic Co-operation and Development (OECD)

14

15

Smillie, Ian and Henny Helmich, eds (1999) Stakeholders. Government-NGO Partnerships for International Development. Paris: OECD and London: Earthscan Publications Ltd. Stokke, Olav, ed (1991) Evaluating Development Assistance: Policies and Performance. EADI Book Series 12. London: Frank Cass. Stokke, Olav (1991) ‘Policies, Performance, Trends and Challenges in Aid Evaluation’, in O. Stokke (ed) Evaluating Development Assistance: Policies and Performance, pp. 1-59. London: Frank Cass. White, Howard (1996) ‘Evaluating Programme Aid. Introduction and Synthesis’, IDS Bulletin 27(4): 1-13.

Notes

1 As indicated in the Overseas Development Institute (1995) it is extremely difficult to obtain reliable estimates of

the total amount of official development assistance that is channelled through NGOs. Official reports such as those

of the DAC only give very partial estimates, focusing exclusively on the amount of bilateral development assistance

that is contributed to NGOs, leaving aside for instance the proportion of multilateral official development assistance

that is channelled through NGOs.

2 The Development Assistance Committee of the OECD (DAC, 1991) classifies aid as programme aid if it supports

the country’s overall development strategy, or one or more sectors rather than specific projects. This definition of

programme aid centres on the relation between northern donor and southern recipient. In this paper we will be

dealing with another relationship, that between northern public donors and northern NGOs. We will talk about

programme funding when the public donor awards a sum of money to the NGO that allows the latter to finance in

turn a series of projects or other activities in a developing country.

3 The 1993 OECD study (Smillie and Helmich, 1993) and 1999 study (Smillie and Helmich, 1999) include case

studies for 14 and 22 OECD bilateral and multilateral donors respectively.

4 After USAID, the Canadian International Development Agency (CIDA) was the first to introduce in 1977 the

Logical Framework Approach. Being influenced by the Results Based Management environment, CIDA has, since

the mid 1990s, modified the Logical Framework Approach to a ‘Results-Oriented Logical Framework’ (Cummings,

1997). Other slightly adapted versions of the Logical Framework are, for instance, the Project Framework, which

was introduced by the UK’s Overseas Development Administration (ODA) in 1985 (Cracknell, 1994), the ZOPP

(Zielorientierte Projektplanung, target-group-oriented project-planning) method used by the Deutsche

Geschellschaft für Technische Zusammenarbeit (GTZ) (Musto, 1991), the Project Cycle Management (PCM)

16

method introduced by the Directorate General for Development (DG VIII) of the European Commission in 1992

(Eggers, 1994).

5 The Logical Framework Approach (or logframe) is considered a poor instrument for the analysis of efficiency and

cost-effectiveness because it does not provide insights into the comparative merits of other alternative investments

to reach the same objectives (Clements, 1995). Acknowledging the value of the Logical Framework Approach as an

analysis tool, Des Gasper (2000) points at a number of recurrent failings: the logframe may easily turn into a ‘lock-

frame’ as it tends to limit flexibility in implementation and encourages project managers to take too static a view of

projects; it may become a ‘lack-frame’ as it is often too simple to capture vital aspects of a project design and it

often degenerates into a ‘logic-less frame’ only creating an illusion of logic. The latter often occurs when the logical

framework approach is reduced to the presentation of an earlier identified project into a matrix format in order to

comply with the requirements of funding donor agencies (see also Eggers, 1998).

6 The study was commissioned by the EU and executed by a consortium of evaluators (including the consultancy

firms South Research, INTRAC, Particip, Prospect and the authors) during the period January-July 2000. The main

aim of the study was to contribute to the improvement of mechanisms, practices and procedures of the budget-line

B7-6000. The full evaluation report is accessible at the website of the EC. The results discussed more fully in the

present paper are contained in annex 6 (http://europa.eu.int/comm/europeaid/evaluation/reports/other/).

7 ‘Project Cycle Management (PCM)’ is a tool that combines the Logical Framework Approach and the Integrated

Approach. The latter manages the various phases of a project cycle in an integrated way by applying the same

criteria for cohesion and sustainability throughout the entire project cycle. It specifies the basic format for the

documents to be produced in each of the phases (Commission of the European Communities, 1993) (European

Commission, 2001).

8 At the moment of project appraisal, project officers at the headquarters also ask an appraisal from EC country

delegations, geographic and thematic desks. While EC country delegations mostly provided their comments, less

traces were found of reactions from geographic and thematic desks.

9 On the B7-6000 budget-line, three types of projects were financed: i) ordinary projects, ii) chapter XII projects

(supporting the activities of grassroots organisations in developing countries) and iii) block grants. The study was

limited to ordinary and chapter XII projects. Block grants are composed of small projects made available to

European NGOs that are judged to have shown their professionalism in the use of the EU funding for a number of

years. They are approved according to a more simplified procedure. As a result, information contained in project

documents was minimal and did not permit a desk evaluation of any depth.

17

10 We classified projects as multisectoral actions when they covered more than two sectors and when there was no

clear indication that one of the sectors was predominant. For instance, an education project for women and children

was classified as an education project whereas an integrated development project dealing with crop production,

education and health was classified as a multisectoral action.

11 For simplicity we lumped the few cases where the answer was “not relevant’ together with the “impossible to

determine” category.

12 Projects located in India (states of Karnataka, Tamil Nadu and Andhra Pradesh) are the only exception. A possible

explanation for the fact that the desk evaluator was relatively more negative in her assessment regarding these

projects might be her thorough knowledge of development projects in this region. Unintentionally, she had more

information at her disposal than was included in the project identification documents and this might have influenced

her assessment.

13 For question 5 desk and field evaluators only twice picked the same score - weak, satisfactory, good or excellent -

for question 22 they never did.

14 In this last case we believe that differences are mainly due to the use of different benchmarks. Whereas the desk

evaluator took ‘changing of strategic gender needs’ as the benchmark, the field evaluators also attributed an

excellent score to cases where only changes in practical gender needs were realised. In her personal notes, the desk

evaluator explicitly indicated that in the project documents there was indeed often some reference to the way women

could profit from the project, but this information did not go beyond blunt and unhelpful generalities to the effect

that women are bound to profit somehow as workers, mothers or daughters. In her opinion, there was hardly any

indication about the way projects could contribute to change the underlying gendered relations.

15 The New General Conditions for EU-ENGO co-financing issued in January 2000 move in this direction. They

include a shift in focus to institutional dialogue, more emphasis on broader objectives and on capacity building of

local partners rather than on appraisal of projects.