Report No 19630

Poverty AssessmentsA Follow-up Review

August 23, 1999

Operations Evaluation Department

Document of the World Bank

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Acronyms

CAS Country Assistance StrategyCODE Committee on Development EffectivenessDEC Development Economics Vice PresidencyESW Economic and Sector WorkLAC Latin America and Caribbean (Regional Office)MNA Middle East and North Africa (Regional Office)NGO Nongovemmental organizationOD Operational DirectiveOED Operations Evaluation DepartmentPPA Participatory Poverty AssessmentPREM Poverty Reduction and Economic Management (Network)QAG Quality Assurance GroupWBI World Bank Institute

Director-General, Operations Evaluation :Mr. Robert PicciottoDirector, Operations Evaluation Departrnent :Ms. Elizabeth McAllisterManager, Sector and Thematic Evaluations :Mr. Gregory IngrarnTask Manager :Ms. Soniya Carvalho

The World BankWashington, D.C. 20433

U.S.A.

ROBERT PICCIO11ODirector-GeneralOperations Evaluation

June 9, 1999

MEMORANDUM TO TIIE EXECUTIVE DIRECTORS AND THE PRESIDENT

SUBJECT: Poverty Assessments: A Follow-up Review

The attached OED report, a follow-up to its 1996 study on poverty assessments, reviewsthe Bank's success in fulfilling its commitment to completing poverty assessments for allborrowers and to improving relevance, efficacy and quality, local partnership and consultation,and efficiency of poverty assessments relative to the standard set forth in OD 4.15. In response tointerest expressed by the Committee on Development Effectiveness, this report also analyzes thecountry-level impact of poverty assessments measured by their influence on within-countrypoverty debates, policy design and implementation, and institutional development, and presentsthe results of a survey of government officials, NGO staff, and technical experts undertaken todetermine client views of the poverty assessments.

The report finds that there has been a modest improvement in the proportion of povertyassessments rated satisfactory for their economic quality, but that substantial improvements canbe made, particularly in their policy analysis and their focus on specific goals. The survey ofstakeholders indicated that six out of seven were satisfied with poverty assessments, but that theyviewed local partnership and consultation -- as well as knowledge transfer -- as the leastsatisfactory aspects of poverty assessments. These two aspects are strongly correlated withpoverty assessment impact at the country level. The report also finds significant slippage in thecompletion of poverty assessments; on average about half of the assessments planned over fiscal1997-99 were completed through May 10, 1999. A concerted effort to address some of therecommendations of the 1996 OED report has begun only in the past year, and it is too early toassess whether these efforts are having their intended effect.

The recommendations of this report are that the Bank should improve poverty assessmentquality and process, focus on goal and priority-setting, and set realistic targets for the completionof poverty assessments and meet them. Bank management has endorsed the report's findings andrecommendations.

This document has a restricted distribution and may be used by recipients only in the performance of their officialduties. Its contents may not otherwise be disclosed without World Bank authorization.

Contents

Acknowledgements ............................................. iii

Executive Summary .............................................. v

1. Context .............................................. 1

Previous Reviews ............................................ 1I

Approach and Methodology ............................................. 2

2. Follow-up to the 1996 OED Report .............................................. 3

Poverty Assessment Timetable ............................................. 3

Poverty Assessment Quality ............................................. 4

3. Relevance .............................................. 4

Assessment Scope ............................................. 5

Strategic Timeliness ............................................. 5

Government Reactions to Assessments ............................................. 6

Adaptation to Country Context ............................................. 6

4. Efficacy and Quality .............................................. 8

Specification of Goals ............................................. 8

Poverty Profile ............................................ 10

Data Use ............................................ 10

Analysis of the 1990 WDR Poverty Strategy Elements ............................................ 11

Policy Analysis .................................................................................................................. 12

Policy Recommendations ................ 14

5. Local Partnership and Consultation .................................... 15

Time Adequacy .................................. 16

Developing Ownership .................................. 16

Dissemination and Follow-up .................................. 19

Institutional Development .................................. 20

6. Poverty Assessment Efficiency and Impact .................................... 21

7. Recommendations .................................... 23

Bibliography ..................................... 25

ii

Annexes

A. Evaluation Framework and Criteria ................................................................. 27

B. List of Poverty Assessments in the Review Sample ................................................................. 37

C. Stakeholder Survey Results ................................................................. 39

D. Approach Paper ................................................................. 47

E. Ledger of OED Recommendations ................................................................. 53

F. Management Response ................................................................. 55

G. Report from CODE ................................................................. 59

Tables

1. Completed Poverty Assessments and Updates .................................................................. 3

2. Correlation between Overall Impact and Four Major Attributes of Poverty Assessments ........ 213. Correlation between Inputs and Outcomes ................................................................. 22

Figures

1. Inclusion of Relevant Issues and Groups .................................................................. 52. Quality Ratings: Economic Perspective .................................................................. 83. Satisfaction with Poverty Profile Dimensions .................................................................. 0

4. More Clients Are Dissatisfied with Recommendations than Staff ........................................... 145. Bank-Government Cooperation ................................................................. 176. Influence of Stakeholders on Assessments ................................ ................................. 17

7. More Clients Are Dissatisfied with Follow-up, More Staff with Dissemination ...................... 198. Local Capacity Building Due to Assessments ................................................................. 20

9. Overall Satisfaction ................................................................. 22

Boxes

1. Bangladesh and Thailand: Adapting the poverty assessment to country conditionsimproves receptivity and influence .................................................................. 6

2. Criteria used in assessing the quality of poverty assessments .....................................................93. Bolivia: Good quality analysis fosters ownership ............................................................... .. 124. Weakness of process information complicates analysis ............................................................ 155. C6te d'Ivoire: Adapting the timeline to government needs increases the in-country

influence of the poverty assessment ................................................................. 16

6. Gabon: Broad-based government participation improves the chances that povertyassessment recommendations will be owned and implemented ............................................... 18

iii

Acknowledgements

This report was prepared under the direction of Gregory Ingram and was edited by WilliamHurlbut. Guidance from Alain Barbu and Roger Slade is gratefully acknowledged. The AdvisoryGroup for the report comprised Alison Evans (internal advisor), and Joan Nelson, RaviSrivastava, and Howard White (external advisors). Background papers and/or synthesis reportswere prepared by Jo Beall, David Greene, Jeremy Holland, Steve Mugerwa, Joan Nelson,Marcelo Neri, Pauline Peters, David Sahn, Christopher Scott, and Ravi Srivastava. ShamitChakravarty and Sohail Malik analyzed key data. Ajay Bhardwaj managed the stakeholder surveyfor The Gallup Organization. Research assistance was provided by Anna Amato, AlexanderArenas, Laura Fried, Ginger Reich, and Veronica Marghescu. Pilar Barquero processed thedocument.

The cooperation received from the 81 country clients who were surveyed by The GallupOrganization and from other country-clients who were interviewed during the course of the fieldwork is much appreciated. Useful discussions with or comments from Zhanar Abdildina, SarahAdam, Zoubida Allaoua, Christine Allison, Rasul Aliaga Bagirov, Ana Maria Ariagada, IanBannon, Deborah Bateman, Jeanine Braithwaite, Gianni Brizzi, Christopher Chamberlin, AllainColliou, Monica Das Gupta, Lionel Demery, Nora Dudwick, Nichola Dyer Cisse, Laurie Effron,Carlos Elbirt, John English, John Eriksson, Boniface Essama-Nssah, Kene Ezemenari, YelenaFadeeva, Osvaldo Feinstein, Fransisco Ferreira, Luisa Ferreira, Richard Florescu, VincenteFretes-Cibils, Ashraf Ghani, Christopher Gibbs, Roy Gilbert, Gita Gopal, Martin Greeley,Christiaan Grootaert, Isabel Guerrero, Betty Hanan, James Hanson, John Heath, Jesko Hentschel,Norman Hicks, Thomas Hoopengardner, Wendy Jarvie, Jeni Klugman, Shigeo Katsu, Roger Key,Qaiser Khan, Lida Kita, Valerie Kozel, Bernardo Kugler, Praveen Kumar, Alexandre Marc,Timothy Marchant, Serge Michailof, Linda Morra, Daniel Morrow, Caroline Moser, JosetteMurphy, Mamta Murthi, Deepa Narayan, Syed Nizamuddin, Phillip O'Keefe, Albert Osei,Azedine Ouerghi, Patti Petesch, Tanaporn Poshyananda, Nadine Poupart, Giovanna Prennushi,Menahem Prywes, Rachidi Radji, Mansoora Rashid, Martin Ravallion, Caroline Robb, LawrenceSalmen, Ana Maria Sandi, Sarosh Sattar, Norbert Schady, Michael Sarris, Edilberto Segura,Anwar Shah, Parmesh Shah, Shekhar Shah, Lynne Sherburne-Benz, Sudhir Shetty, JayashankarShivakumar, Antoine Simonpietri, Shatrughna Sinha, Susan Stout, Kalanidhi Subbarao, ParitaSuebsaeng, Judith Tendler, Hasan Tuluy, Willem van Eeghen, Warren Van Wicklin, MichaelWoolcock, Michael Walton, Tevfik Mahmet Yaprak, and Kadir Yurukoglu enriched the analysis.

v

Executive Summary

A country poverty assessment is a quantitative and qualitative analysis of poverty intended toinform World Bank policies toward a country and to inform country-level policy design andaction. The first such assessment was completed in 1989, and by the end of fiscal 1998 the Bankhad carried out 101 poverty assessments or updates. In 1996, OED conducted its first review ofpoverty assessments, examining those completed through December 1994. It found that 54percent of the 46 assessments completed met the standards set by Operational Directive 4.15-which provides guidance to Bank staff for addressing poverty issues and carrying out povertyassessments.

This report is a follow-up to OED's 1996 study on poverty assessments and reviews the Bank'ssuccess in fulfilling its commitment to completing poverty assessments for all borrowers and toimproving the relevance, efficacy and quality, local partnership and consultation, and efficiencyof poverty assessments relative to the standard set forth in OD 4.15. In response to the 1996 OEDstudy, the Committee on Development Effectiveness (CODE)' suggested that an impactassessment and a survey of borrowing countries' views on poverty assessments be undertaken.Accordingly, this follow-up review also examines the country-level impact of 19 povertyassessments and updates (all those done in fiscal 1997 and 1998, plus a sample from fiscal 1996).Rather than focus on the influence of poverty assessments within the Bank, which is at best aninput measure, this review focuses on their impact at the country level measured by theirinfluence on the poverty debate, on policy design and implementation, and on institutionaldevelopment. In addition, it reports on a survey of government officials, NGO staff, and technicalexperts to determine their views of the poverty assessments that the Bank has carried out in theircountries.

The main findings of this review are:

* There has been modest improvement in the proportion of poverty assessments ratedsatisfactory for their economic quality, from 54 percent of those done through 1994, to 63percent of those done in fiscal 1996 through 1998.

* Substantial improvements can be made in the quality of poverty assessments, particularly intheir policy analysis and focus on specific goals.

* The overall impact of poverty assessments at the country level is strongly correlated with thequality of the poverty assessments, the degree of partnership and consultation at the countrylevel in carrying out the poverty assessments, information sharing and knowledge transfer,the timeliness of the results, and the adequacy of time available to prepare the assessments.

* The survey of stakeholders indicated that six out of seven were satisfied overall with thepoverty assessment for their country, but that they viewed knowledge transfer and localpartnership and consultation as the least satisfactory aspects of poverty assessments. Thepresent review found that these two aspects are strongly correlated with the overall impact ofpoverty assessments at the country level, they deserve special attention and priority in effortsto improve poverty assessments. The survey shows that in most cases a higher proportion ofBank staff are positive about poverty assessments than country clients. In addition, Bank staff

1. A committee of the World Bank's Board of Executive Directors.

vi

are also more intense about their feelings-32 percent say that they are "very satisfied" with thepoverty assessment for their country compared with only 10 percent of country clients.

* A review of activities taken and changes made in response to the 1996 OEDrecommendations indicates that poverty assessments are still subject to significant slippage(the number of poverty assessments completed each year over the period from fiscal 1997through May 10, 1999, averages about half of the plan). Moreover, a concerted effort toaddress some of the recommendations of the 1996 OED report has only begun in the pastyear, and it is too early to assess whether these efforts are having their intended effect.

The main recommendations of this report are:

* Improve poverty assessment quality and process. The country management and regionalChief Economist's office should increase management oversight of the analytical quality ofpoverty assessments as well as their partnership and consultation and knowledge transferaspects. The Bank should undertake self-evaluations of poverty assessments. The PovertyReduction Board and Thematic Groups should support the Regions with training and advice,and through providing tools and best practice examples on poverty analysis, engagingpartners, supporting collaboration and local participation, and developing local capacitythrough knowledge transfer. The Poverty Reduction Board and Thematic Groups shouldensure that the Regions have adequate and appropriate guidance, including on best practices,for the preparation of poverty assessments. The Poverty Reduction Board should report thefindings of the Bank's self-evaluations in the Annual Progress Report on Poverty.

* Focus on goal and priority setting. Country management should ensure that povertyassessments - from the concept paper on - define clear goals and priorities, provide adequatejustification for prioritization of issues and for any selectivity in scope and coverage,adequately reflect country conditions, and link work programs and analysis to goals. Theregional Chief Economist's office should monitor and provide quality control for this. ThePoverty Reduction Board and Thematic Groups should support the Regions with advice andbest practice examples.

* Meet targets for poverty assessment completion. The Regions, in collaboration with thePoverty Reduction Board, should set realistic targets for completing poverty assessments. Tounderscore the importance of the poverty reduction agenda, senior management shouldprovide the institutional inputs and incentives to produce quality poverty assessmentseffectively and on time.

1. Context

A country poverty assessment is a quantitative and qualitative analysis of poverty that measureslevels and trends in poverty, assesses the determinants of poverty, and examines what policies arelikely to be successful in reducing poverty in the country. Poverty assessments are intended toinform World Bank policies toward a country and to inforrn country-level policy design and action.The bulk of the poverty assessments have been carried out by the Bank since 1990, following the1990 World Development Report (WDR), which codified the Bank's approach to povertyreduction. Through fiscal 1998, the Bank has completed 101 poverty assessments or updates, and a1991 Operational Directive (OD 4.15) and a 1993 Poverty Reduction Handbook provide guidanceto Bank staff for addressing poverty issues and carrying out poverty assessments.

A poverty assessment, the OD states, "provides the basis for a collaborative approach to povertyreduction by country officials and the Bank. It helps to establish the agenda of issues for the policydialogue" (para. 15). The Bank's 1991 policy paper on poverty reduction, which first proposed theidea of systematic poverty assessments, articulated the rationale for them: "through careful analysis,the Bank, in collaboration and dialogue with the relevant governments, will be well placed to designa program of assistance that most effectively incorporates the goals of poverty reduction into thecountry's own efforts" (p. 19). By conceiving of the poverty assessments as providing a basis for acollaborative approach to poverty reduction by country officials and the Bank, the OD placespartnership and process at the heart of poverty assessments.

Previous Reviews

In 1996, OED conducted its first review of World Bank poverty assessments,2 and evaluatedassessments completed through December 1994, with respect to three issues: (i) the progress madein fulfilling the Bank's commitment to completing poverty assessments; (ii) their content andresponsiveness to the OD; and (iii) their influence on country assistance strategies and on Banklending designated as poverty-targeted under the Program of Targeted Lending. It assessed thecontent and responsiveness of the poverty assessments to OD 4.15 based on: inclusion of a set ofrelevant poverty indicators, scope and depth of the diagnosis of poverty, breadth and focus of theprescriptions for poverty reduction, and strategic content of those prescriptions. It found that theadequacy of poverty assessments was highly variable, and that 54 percent of the 46 assessmentscompleted before December 1994 met the benchmarks set by OD 4.15 with few shortcomings.

The 1998 Quality Assurance Group (QAG) assessment of the quality of a sample of economic andsector work carried out in fiscal 1998 found that poverty analysis was the weakest type of ESW-with only 43 percent rated satisfactory. The uneven quality of poverty assessments has also beenpointed out in "Taking Action to Reduce Poverty in Sub-Saharan Africa" (World Bank 1997b) andin "Poverty Reduction and the World Bank: Progress in Fiscal 1998" (World Bank 1999).

This review is a follow-up to the 1996 OED study. It follows-up on the Bank's success inaddressing the first two issues examined by the previous study, using somewhat broader criteria toassess the second issue. The third issue is being examined by OED's forthcoming poverty study.The aim of this review is to gauge the Bank's success in fulfilling its commitment to completingpoverty assessments for all borrowers and to improving the relevance, efficacy and quality, localpartnership and consultation, and the efficiency of poverty assessments relative to the standard setforth in OD 4.15.3 In addition, as a response to the view of the Committee on Development

2. "Poverty Assessments: A Progress Review," Report Number 15881, World Bank, 1996.

3. The design approach of this study is presented in Annex B.

2

Effectiveness (CODE) that "impact assessments and a survey of borrowing countries' views onPoverty Assessments might be useful,"4 this follow-up review also examines the country-levelinfluence or impact of a sample of poverty assessments, and it surveys government officials, NGOstaff, and technical experts to determine their views of the poverty assessments carried out by theBank in their countries. The country level impact of poverty assessments is measured qualitativelyin this study by their influence on the poverty debate, on policy design and implementation, and oninstitutional development. The focus is on results at the country level rather than within the Bank.

Approach and Methodology

This report uses the ESW evaluation framework developed by OED as well as the QAGframework for reviewing ESW (World Bank 1998, 1997a). Modifications have been made toreflect the specifics of the poverty assessment instrument. Accordingly, the review assesses therelevance, efficacy and quality, partnership and consultation, and efficiency of the povertyassessments. The standard against which the assessments are evaluated is based on OD 4.15. Thisreview does not attempt to measure the extent to which assessments have reduced poverty in thecountries concerned.

The review uses four tools:

* Country case studies - in-depth fieldwork conducted in four countries and involving semi-structured interviews of a range of country clients to gain a detailed understanding of theirperspectives.5

* Stakeholder survey - an extensive telephonic survey of country clients (government officials,NGO staff, technical experts), assessment task managers, and resident mission staff'

* Quality review - analysis of assessments and updates aimed at establishing their quality fromboth an economic perspective and in terms of their analysis of social, political, and institutionalissues.

* Process review - file research and examination of institutional databases supplemented by asurvey of task managers to gauge the soundness of the processes supported in preparing,disseminating, and following-up on the assessments.

A description of the tools is presented in Annex A. The staff of the Poverty Division of the PovertyReduction and Economic Management (PREM) Network, poverty assessment task managers, andkey staff working on poverty reduction issues across the Bank reviewed and commented on thedesign of each of the tools used in this review. Relevant comments were incorporated in the designof the tools. The 19 poverty assessments and updates reviewed include all 15 assessments/updatescompleted in fiscal 1997/98 as well as four assessments/updates completed in fiscal 1996 (see list inAnnex B).7 Three considerations underlie this selection. First, information may be distorted if

4. CODE 96-64, World Bank, 1996.

5. The four countries are: Bangladesh, Bolivia, CMte d'Ivoire, and Thailand. They were chosen for regional balance.Armenia and Yemen had to be dropped for unforeseen operational reasons.

6. The results of the stakeholder survey are reported in Annex C. The survey covered 81 country clients in 17 countriesidentified by assessment task managers, resident mission chiefs, and/or country directors. It also covered 25 assessmenttask managers and resident mission staff. The survey was administered in three languages: English; French; andSpanish.

7. The fiscal 1998 Rwanda Poverty Note and the fiscal 1998 Chile Poverty Assessment Update are not included as theywere classified as poverty assessments after the sample for this review was finalized.

3

clients are asked to recall events more than three years past, or if staff turnover has occurred inclient countries. Second, current practice may not be reflected in the older cohort of assessments.Third, poverty assessments have their main influence within a year or two of their production. Thefour assessments completed in fiscal 1996 have been included specifically to account for thepossibility that country circumstances might prevent an immediate adoption of therecommendations of the assessment and result in a delayed response. All six Bank Regions arerepresented in the sample.

2. Follow-up to the 1996 OED Report

The 1996 OED report included specific recommendations concerning the completion of povertyassessments, the quality of poverty assessments, and the linkage of country assistance strategies andlending with poverty assessments. The current report follows up on the first two issues. The thirdissue-the influence of poverty assessments on country assistance strategies and on Bank lending-is being addressed by OED's forthcoming Poverty Study.

Poverty Assessment Timetable

By the end of fiscal 1998, the Bank had completed 101 poverty assessments (including 15 updates).Over fiscal 1997-99, the Bank committed itself to completing 50 poverty assessments and updates.As of May 10, 1999, it had completed only 21 of those (on average about half of the plan). Table 1shows the number of poverty assessments so far completed and those planned for fiscal 2000-01.The Poverty Reduction Board is monitoring delivery of the poverty assessments to ensure that thedelays in completion which severely curtailed last year's plans (resulting in only 7 of the planned 20poverty assessments being completed) do not recur. The Bank clearly needs to adopt realistic targetsand provide the necessary management support and budgets early on to ensure the timely andeffective completion of quality poverty assessments.

Table 1. Completed Poverty Assessments and Updates

Fiscal Year Assessments Updates Actual Total Planned Total(includes updates)

1989 1 0 1 -1990 5 0 5 -1991 9 0 9 -1992 3 0 3 -1993 9 1 10 -1994 9 2 11 -1995 22 1 23 -1996 17 5 22 -1997 8 2 10 181998 3 4 7 201999 2 2 4- 122000 - - - 332001 - - - 10

Grand Total 88 17 101Source: PRMPO Database, World Bank, 1999.

* As of May 10, 1999. Several additional assessments are expected to becompleted in fiscal 1999.

4

Poverty Assessment Quality

The 1996 OED report had expressed concern about the quality of the assessments. In response,the Bank made some attempts to improve quality between 1996 and 1997, including a variety oftraining programs. However, owing to internal reorganization in the Bank, concerted action toimprove the quality of poverty assessments is only now taking place. The Bank has recentlyinitiated a number of actions. The particular actions taken and their dates are as follows.

* The peer review system has been strengthened through the introduction of a list of "certified"peer reviewers (March 1999). The list of reviewers approved by the Poverty Reduction Boardis now available on the PREM intranet site and was distributed to the members of the PREMNetwork.

* Regions have been asked to develop mechanisms to ensure the timely delivery of qualitypoverty assessments (March 1999). The number of completed poverty assessments ismonitored in the Bank-wide Monthly Operations System (MORE). In the Regions,responsibility rests with the Chief Economists.

- PREM, in collaboration with other Networks, DEC, and WBI, delivered 10 internal trainingcourses on poverty-related topics for Bank staff. WBI sponsored three external courses forpolicymakers, some of which were carried out in partnership with local research institutes(July 1998-May 1999).

- A census of household surveys has been conducted (December 1998). Results are available inthe fiscal 1998 Progress Report on Poverty, on the intranet, and on the Poverty Net externalwebsite.

- An initiative to improve the quality of survey data for poverty monitoring was started in theLatin American Region in cooperation with the Inter-American Development Bank(December 1998). Several initiatives were taken as part of the Program for the Improvementof Living Conditions project (MECOVI), including a three-week training course in Mexico.

* Training in analysis of poverty data was carried out in the Middle East and North AfricaRegion (September 1998).

As is evident, these Bank actions are very recent. There is, therefore, little to report concerningthe impact of these initiatives. Furthermore an assessment of the success the PREM Network hashad in enhancing analytical work on poverty would also be premature, as would an analysis of theextent to which behavior among Bank staff has changed as a result of the Bank's neworganizational and incentive structure. This report includes recommendations to help ensure thatthe actions taken so far by the Bank have their intended effect (see p. 22).

3. Relevance

The relevance of a poverty assessment is judged by the extent to which it adapts its scope andrecommendations to country conditions, addresses key poverty issues for the country, identifiesthe relevant groups as poor, and tailors its content and process to ground realities. The analysis inthis section is based on the quality review, process review, and country case studies.

5

Assessment Scope

A thorough understanding of the country is a prerequisite to asking the correct questions,focusing on the right issues, addressing the salient policy dilemmas, and makingrecommendations that reflect country realities. It also helps to factor into the recommendationsthe political and institutional constraints on the capacity to implement desirable policy. There iswide variation in the extent to which the assessments reviewed in this study systematicallyappraise the social, political, and institutional conditions prevailing in the country.

Some assessments (for example, Bangladesh and CMte d'Ivoire) are clearly motivated by a senseof the challenges facing the countries, and in these cases, the design and content of theassessments, including their recommendations, reflect well the economic, social, political, andinstitutional realities in the country. A few assessments are far removed from an intimateunderstanding of the country context and circumstances.

The broad and standard policy recommendations (for example, improve cost recovery, invest ingirls' education, encourage community participation) made in over one-third of the assessmentsare devoid of country context and indicate a lack of appreciation of the country circumstances.The stakeholder survey indicated that significant proportions of country clients feel that therecommendations were unrealistic given the institutional capacity of the country (31 percent) andthe political situation facing it (22 percent).

A failure to understand the country context results in inadequate analysis of the relevant groups ofpoor as well as inadequate analysis of relevant poverty-related issues facing the country. Thestakeholder survey found that about half of both country clients and Bank staff (52 percent and 44percent, respectively) thought that some relevant population groups in the country were excludedfrom the assessment or needed moreattention. Furthermore, about 44 percent Figure 1. Inclusion of Relevant Issues and Groupsof country clients and a similarpercentage of Bank staff thought that not 100 ,

all relevant poverty-related issues were . 80

adequately addressed (Figure 1). NGOs 800

were more likely than other groups to 60 -

respond negatively to both points. 40 . n2

Strategic Timeliness 0

All groups included All issues includedTimeliness is an important dimension ofrelevance. The extent to which an a Bank staff EMCountry clients

assessment is strategically timed to Source: Poverty Assessment Stakeholder Survey

coincide with a favorable climate forpolicy reform is vital to its acceptance by country clients and to its ultimate impact. The degree towhich the reviewed assessments explicitly considered the timing in relation to their potential tohave an impact is unclear. (Also see "Time Adequacy," p. 16).

In one case, a poverty assessment was launched under obviously unpromising circumstances. Thecountry was experiencing widespread political and social disintegration, and government interest inpoverty issues was extremely weak. This severely interfered with the production of the assessment.Furthermore, owing to the dramatically changed country circumstances at the end of the assessmentprocess, the Bank concluded that the assessment was outdated soon after it was released.

6

Government Reactions to Assessments

The reaction of a government to a poverty assessment is difficult to predict, and its attitude towardthe assessment may change over time. Upcoming elections in some countries caused incumbentgovernments to oppose or delay the assessment. Yet, the incumbent government in another countrydid not object, despite an upcoming election (which it lost).

The receptivity or opposition of governments to assessments also can be based on unpredictableperceptions. For example, while the government in one country did not wish to be embarrassed bythe possibility of an unfavorable assessment, the government of another country reportedly wasconcerned lest the assessment conclude that poverty was less than the very high level it was using inits discussions with donors. In yet other countries, the poverty assessments were perceived asopportunities to identify poverty policies with some popular support for the election campaigns.

The reaction to a poverty assessment is unrelated to the type of political system a country has.More open, democratic political systems are sometimes assumed to be more responsive to theinterests of the poor, especially when the poor are a large proportion of the population. Whileseveral of the most responsive governments have established or emerging competitive politicalsystems, two finnly institutionalized democracies in the sample were only mildly enthusiasticabout the poverty assessment. Conversely, Azerbaijan and Kazakhstan were receptive, althoughneither has an open, competitive political system.

Adaptation to Country Context

Adapting the content and process of a poverty assessment to country circumstances enhances itscountry-level influence. Effective adaptation requires that the country circumstances be carefullyunderstood in the first place.8 In two of the country case studies undertaken for this review, theBank was particularly successful in tailoring the scope and content of the assessments to thepriorities and key issues facing the countries (Box I).

Box 1. Bangladesh and Thailand: Adapting the poverty assessment to country conditionsimproves receptivity and influence

The Bangladesh Poverty Assessment aligned its goals to the country context and supported ongoingefforts within the country to build capacity for poverty measurement and analysis. By doing so, the overallreceptivity to the assessment in the country was much enhanced. The assessment team worked closelywith the Bangladesh Bureau of Statistics (BBS) not only helping to collect poverty data, but alsosupporting training programs to bring about changes in basic analytical methods employed by governmentstatisticians. By all accounts, the assessment contributed to the strengthening of capacity at BBS.

The Thailand Poverty Assessment also adapted itself well to the prevailing country context. During thepre-crisis period when the assessment was being undertaken, the Bank had little leverage in Thailand.The Thai economy was considered "mature" and the government had access to capital markets witheasier disbursement procedures, cheaper terms, and an absence of conditionalities. Furthermore, withfew signs of impending doom, poverty was not a major issue for Thai policymakers. Under thecircumstances, the poverty assessment focused narrowly on tackling persistent and increasingly skeweddistributional income inequalities while also improving the targeting and content of existinggovernment anti-poverty programs. The decision to limit the focus of the poverty assessment led, atleast in part, to clear and actionable recommendations that were well-received by the government: whileit is not possible to make strict causal linkages, some of the recommendations of the assessment areechoed in subsequent policy changes.

Source: Country case-studies undertaken for this revieew, February-March, 1999.

7

The particular combination of conditions a country exhibits affects the levels and nature ofcountry-level influence. Based on the process review undertaken for this study, there are threescenarios of country conditions and corresponding levels and natures of assessment influence:

Government non-support. In one of the sampled countries, the government was hostile to theassessment effort, partly because of its denial that poverty had worsened and partly because ofmethodological differences relating to poverty measurement. Yet the Bank persevered. Theassessment team made vigorous and repeated efforts to involve government agencies but wasconsistently rebuffed. Even an explicit suggestion that the assessment might feed into a large loanfailed to soften government opposition. So the assessment team worked with agencies outside thegovernment, and trained people in those agencies in data collection and analysis approaches.Despite the lack of direct and immediate influence, the assessment encouraged and supportednongovernmental research and analysis and contributed new information and ideas to theembryonic public debate.

In such situations, poverty assessments may produce limited immediate impact but haveimportant delayed and indirect impacts. In this context, the function of the assessment will be tobegin to create interest and contribute information, methodologies, and ideas to those groupsinside and outside of government who have some interest in poverty reduction efforts. Such casesmay be hard to predict. If some collaboration turns out to be feasible, it should certainly beencouraged. The operational implication is a modest rather than full-fledged analytic effort andcareful phasing.

Government Skepticism. In another country, a poverty assessment was undertaken in a context ofmany decades of intensive debate, analysis, and experimentation with poverty reduction policies.The government believed that the poverty assessment could contribute little new insight given thevast existing analysis of poverty issues in the country. Institutional capacity and governmentownership of, and commitment to, poverty issues were not in question, though the assessmentwas expected to enter an arena of intense technical and methodological debate. In this setting, theassessment was deliberately given a fairly narrow focus and a short timetable. While widespreadlocal consultation and collaboration were not undertaken, dissemination was extensive. Theassessment intensified the debate about factors explaining the levels of poverty and about povertytrends following liberalization. Its findings also were taken into account in the thinking on policydesign, especially with respect to the human resource and safety net analysis.

In such situations, immediate but more specific influences on government thinking and action andon dialogue can be expected because the assessment is adding to an already extensive store ofknowledge. It may also be viewed as intruding on the intellectual and methodological territory ofcountry specialists. A more narrow focus and collaboration with particular country clients islikely to be sensible under the circumstances. The assessment may make important specificcontributions. The operational implication is somewhat smaller budgets and shorter timetables.

Government support. The governments of Azerbaijan and Kazakhstan were sensitive to povertyreduction issues. The assessment teams took advantage of this favorable environment and workedwith the govemments to adjust the scope and content of the assessments to address the interests ofthe governments. In both countries, the assessments provided analysis of a kind the governmenthad never before had. The governments welcomed the policy analysis and advice of the Bank. InKazakhstan, as the assessment demonstrated the value of Kazakhstan Living Standards Surveydata, the relevant unit in the Ministry of Labor and Social Protection made a strong request fortraining in its use (incorporated later in a World Bank Technical Assistance loan). Furthermore,the government acted on many of the recommendations of the assessment, such as the need fortargeting child allowances and the level of minimum pension. Similarly, in Azerbaijan the

8

government used the survey data generated by the assessment, incorporated a number ofrecommendations in its policies, and initiated several follow-up steps arising out of therecommendations, such as the establishment of a special agency to deal with refugeerehabilitation issues.

In such situations, the most immediate, visible, and wide impact on govemrnment action can beexpected. In effect, the assessment is providing data, analysis and ideas in an intellectual and politicalmarket where there is considerable demand for, and limited supply of, these products. Extensiveconsultation and collaboration with government and non-government clients is important. Theoperational implication is to adjust budgets and schedules accordingly.

4. Efficacy and Quality

The efficacy of a poverty assessment is judged by the extent to which it achieves its goals. Thequality of those goals is therefore important to the country-level impact of an assessment. Otherdeterminants of quality are also important. The discussion below draws heavily upon the evidencefrom the quality review of the 19 poverty assessments, supplemented by the findings of thestakeholder survey. 9 The review rated each assessment on several dimensions of quality (see Box2 and Annex A). The results of the ratings for economic quality are summarized in Figure 2.

Specification of Goals

About two-third of the reviewed poverty assessments state their general goal, which is to assessthe extent and causes of poverty and propose a strategy to assist the poor. Even when they statethe general goal, however, few assessments explicitly articulate the underlying questions thatmotivate and frame the analysis. Furthermore, 63 percent do not state specific goals. It is

Figure 2. Quality Ratings: Economic Perspective

0 20

0.

60

0 40 ~ ~ C,..04

20 ~ ~ ~ ~ ~ 0<

00

o0

Attribute

Source: Poverty assessment quality review

9. Poverty assessments for the following countries were reviewed for economic quality: Albania, Armenia, Azerbaijan,Bangladesh, Bolivia, Chad, Congo, Costa Rica, C6te d'Ivoire, Djibouti, Gabon, Guinea, India (two assessments),Kazakhstan, Romania, Thailand, Ukraine, and Yemen. The poverty assessments for the following countries (roughlyhalf of the sample) were reviewed for the quality of their analysis of social, political, and institutional issues: Albania,Armenia, Bangladesh, Bolivia, Cote d'lvoire, India (two assessments), Thailand, and Yemen.

9

therefore unclear which of the possiblegoals (implicit in the OD) the Box 2. Criteria used in assessing the quality ofassessments aim to achieve, for example, poverty assessmentsto establish a collaborative approach to The sampled poverty assessments were assessed usingpoverty reduction between the Bank and specific criteria of economic quality as well as ofgovernment, to establish the agenda for social, political, and institutional quality. The criteriathe policy dialogue, to influence the reflect broad dimensions; the mechanical application ofgovernment's poverty reduction policies a predetermined checklist was deliberately avoided.and programs, to help build localcapacity to address poverty, or to help The criteria for economic quality, which derive fromthe Bank refine its assistance strategies. 4., are:

Futeroe th quait reie fid 7 Clarity of general and specific goals andconsistency with the OD (general goals: what the

that when multiple goals are specified, poverty assessment aims to assess; specific goals:they are seldom prioritized. what change the poverty assessment aims to

achieve).The resuits of the stakeholder survey . Thoroughness and quality of the poverty profile,show that respondents are generally including poverty line(s) and characteristics of thesatisfied with the stated goals of the poor.poverty assessments. Large majorities of . Quality of coverage, with prioritization andcountry clients and Bank staff said that selectivity adequately justified, of the three aspectseach of the following was a goal for the of poverty reduction strategy (promoting efficientassessment in their country: to establish a labor-intensive growth, human resourcecooperative effort between Bank and development, and targeted interventions and safetygovernment in trying to reduce poverty nets).ientheountry; toyn allow redue Bantohelp *Thoroughness and quality of the policy analysisin the country; to allow the Bank to help and whether it is based on a sound analytical andinfluence the government's poverty empirical methodology and is rooted in thereduction policies and programs; to help analysis of the causes of poverty at the macro andbuild country-level capacity to analyze micro levels.and monitor poverty in the country; and . Thoroughness and quality of the policyto help the Bank to refine the poverty recommendations and whether they follow fromfocus of its own operations. If all these the policy analysis and are prioritized based on anwere the goals, however, the results of analytical foundation with tradeoffs beingthe process review suggest thatfew explicitly recognized or explored.

assessments were designed to respond to The criteria for social, political, and institutionaleach of them. quality, which derive from OD 4.15 in conjunction

The failure to identify and prioritize the with other Bank documents and guidelines, are:. Extent to which a multi-dimensional approach to

goals of an assessment is a serious definitions and profiles of poverty is adopted.omission because the goals cannot be . Extent to which the qualitative/participatory workconsidered self-evident even to those is based on a sound methodology.conversant with the OD, much less to . Extent to which the qualitative/participatory workcountry clients. Also, specific goals help and the quantitative household survey work areto focus the assessment on the particular coordinated (sequenced or combined) to enrich thecircumstances of the country in question. analysis, to explain issues, and/or confirm/refuteIn addition, clear goals help to specify hypotheses.the technical design of assessments. They . Extent to which the policy analysis hasalso have implications for the process internalized the qualitative/participatory work andthrough which the assessments are reflects an understanding of the social, cultural,undertaken. For tnexassessmple,ne onlygoal political, and institutional context of the country.undertaken. For example, if the only goal * Extent to which recommendations are informed byof the poverty assessment is to create a the qualitative/participatory policy analysis andknowledge base within the Bank, limited are consistent with the social, cultural, andclient consultation and participation will institutional context of the country.

10

be appropriate. If, on the other hand, the goal of the assessment is to influence the government'spoverty reduction policies, client partnerships are critical.

Poverty Profile

The reviewed poverty assessments generally follow the OD in terms of defining poverty andpreparing a poverty profile. In 63 percent of the cases, the poverty assessments provide a detailedprofile ofpoverty along numerous dimensions (among them, gender, age, employment, location,and ethnicity). Some assessments have exploited quantitative household data to reveal a richersocial texture of poverty through, for example: gender disaggregation in the Bangladesh PovertyAssessment; or gender and ethnic disaggregation in the Bolivia Poverty Assessment; or age,gender, and size-of-family disaggregation in the Azerbaijan and Kazakhstan PovertyAssessments. About 60 percent of the assessments reviewed for the analysis of social, political,and institutional issues use qualitative or participatory work to define poverty in non-incometerms. The new qualitative work and participatory poverty assessments (PPAs) are generally ofsound quality and make it possible to capture the impoverishment of groups such as displacedpersons (Armenia, Azerbaijan), low castes (India), and HIV/AIDS cases and child employees(C6te d'Ivoire). It also allows a greater focus on intra-household and social dynamics both ascorrelates and causes of poverty. The overall framing of the assessments and the poverty analysis,however, continue to be based on poverty defined in monetary terms.

The qualitative research undertaken for the 1998 India Poverty Assessment highlights dimensionsof well-being that are normally absent from consumption-based poverty research, such as thevalue of social capital which improves access to the formal economy, and the implications ofviolence and insecurity for well-being. Similarly, the CMte d'Ivoire Poverty Assessment draws ona qualitative study to identify dimensions of poverty not captured in the statistical data, such asincreased competition among uses ofland and other natural resources and Figure 3. Satisfaction with Poverty Profiledeclining visits to medical centers by Dimensionsfamily members in the context ofdeclining real incomes. 100 .

The results of the stakeholder survey r 80 -show that while large percentages of 60-both country clients and Bank staff LO

are satisfied with the poverty profile, 40-a far higher proportion of Bank staff 20 -than country clients are satisfied withspecific aspects of it, in particular, 0°,the: treatment of rural poverty, Coverage of Rural Poverty Methodologicalmethodological rigor, and coverage relevant groups Rigorof the appropriate population groups O Bank staff EUCountry clients

(Figure 3). Ssu,rcc Poverty Assersment Stakeholder Survey

Data Use

The OD allows the depth of analysis in poverty assessments to vary with data availability. Thiedata requirements are demanding to prepare a good assessment. Adequate household survey dataare a prerequisite, albeit not sufficient, for sound poverty analysis.

11

Several of the reviewed assessments have good data and are based on one or more representativehousehold surveys.'° For countries where reasonably comparable data sets over time are available(for example, India, Thailand, Bangladesh, and CMe d'Ivoire), the scope for analysis is greatlyenhanced. The review found that opportunities for analysis have been exploited to varying extentsin these countries. Other countries with more limited data did not fully use what they had. Forexample, one assessment made no attempt to use the available integrated household survey toanalyze issues like the determinants of school enrollments or demand for health care, both keyissues in poverty analysis.

The wide variation in the quality and quantity of data obviously influences the analysis in thepoverty assessments. However, there is not a one-to-one correspondence between data qualityand assessment quality. Furthermore, existing qualitative data have also been used to varyingdegrees in different countries. Two of the reviewed assessments almost completely ignorednumerous, high-quality contextual studies in their focus countries.

Overall, about half of the poverty assessments under-exploit existing quantitative and qualitativedata, as well as research from the countries concerned and even from within the Bank.'" The useof secondary sources depends on serendipity, resource or time constraints in the povertyassessments, or the commitment of individual team members to seek them out, rather than beinganalyzed as a matter of routine.

Analysis of the 1990 WDR Poverty Strategy Elements

Every poverty assessment need not give equal weight to all three elements of the Bank's povertyreduction strategy (labor-intensive growth, human resource development, and social safety nets)or address them in equal depth. But the coverage and balance should be sufficiently justified. Thequality review found that nearly half of the assessments do not address individual elements wellor do not sufficiently justify the balance between the three elements!2 A main failure is that theydo not address macro linkages to poverty (such as trade and exchange rate policy), or discusssectoral issues, particularly the prominent role of food policy, agriculture, and rural developmentin contributing to poverty reduction. Regional dimensions of a poverty reduction strategy,including investment decisions, also receive limited attention.

The importance of labor-intensive growth to poverty reduction is recognized in all assessments, butoften is not sufficiently analyzed. The Bolivia Poverty Assessment is a best practice example in thisregard: its discussion of promoting labor-intensive growth is superior to that in most of theassessments reviewed, and the relative weight accorded to each of the three aspects of the povertyreduction strategy is superior.'3 The good quality of the Bolivia Poverty Assessment contributed toits broad use within the country by both government officials and the opposition party (Box 3).

10. A census of the availability of country-level household data across borrower countries is provided in "PovertyReduction and the World Bank: Progress in Fiscal 1998" (R99-46, World Bank 1999, pp. 21-22). It indicates that 110of 124 Part II countries had data from at least one consumption/income survey in 1998 and 60 countries hadcomparable data for at least two points in time. In addition to traditional household surveys, innovative non-income/consumption household surveys such as the Core Welfare Indicators Survey (CWIK), food security surveys, orservice satisfaction surveys can also provide useful data for poverty assessments.

I1. Based on a review of eight poverty assessments.

12. The justification was not provided in the poverty assessments or concept papers in these cases.

13. The Bolivia Poverty Assessment was an update. The fact that it was an update did not make a difference to qualityin this case or in the other updates in the sample.

12

The overall failure in most poverty Box 3. Bolivia: Good quality analysis fostersassessments to give the broadermacroeconomic and sectoral policy ownershipframework prominence in developing a The Bolivia Poverty Assessment made good use ofpoverty reduction strategy is inconsistent the limited poverty data in the country. The soundwith the OD. One explanation for this is analysis of the three elements of the poverty reductionthat the OD states that a more narrowly strategy led to clear and coherent recommendationsconcentrated assessment is appropriate which in tum helped to inform the politicalwhere poverty is also narrowly discussions of the time. During the 1997 democraticconcentrated. In some cases, particularly elections for the Presidency, the new Vice President,in Eastern Europe, an emphasis on safety then in the opposition, recalls extensively consulting

nes my bthe assessment for campaign debates. The assessmentnets may be justified. But even here, it ISunrealistic . address the causesofwas used both by central government and theunrealistic to address the causes of opposition to trace their strategies. The newpoverty without a sustainable growth government (which was in the opposition when thestrategy that includes unemployed and assessment was being prepared) has since madelow-paid workers, poor households, and poverty reduction a key objective and included twobackward regions or that addresses the expressions from the poverty assessment's title,economic stagnation and decline that "equity" and "opportunity," in its own four-pillarresult from fundamental failures of the poverty reduction strategy. This is seen in Bolivia aseconomic policy framework. Another reflecting, at least partially, the influence of thepossible explanation is that such analysis poverty assessment on Bolivian policy. Furthermore,is contained in other, complementary the assessment recommendations contributed directlyto the preparation of proposals for the annual meetingdocuments, or IS considered the of the Consultative Group.responsibility of those dealing withmacroeconomic, structural, and sector Source: Country case-study undertaken for this review, February 1999.

issues. However, even if other Bank workdoes address such issues from a poverty reduction perspective, it remains for the povertyassessment to integrate the work, make the necessary linkages, and draw out the implications forthe overall poverty reduction strategy in a country.

Policy Analysis

About 58 percent of the reviewed assessments make an inadequate effort to conduct rigorous ormeaningful policy analysis. The policy analysis in these assessments is very limited and generallynot based on sound and rigorous methodological approaches. The assessments rarely incorporatea clear conceptualization of the link between macro, structural, and sectoral reforms and povertyor distributional outcomes, and generally fail to conduct sound inferential analysis.

Constructing complex computable general equilibrium and multi-market models may be beyondthe reach of the assessments, and such models have well-known limitations. However, evensimple modeling techniques and partial equilibrium tools can be illuminating, and in manyinstances existing models could have been enlisted in support of the analysis. For example, in oneof the reviewed assessments a number of macroeconometric models existed for the country andcould have been exploited to address issues raised in the assessment, but they were not. In a fewother cases, various econometric models are used, particularly in the analysis of labor marketsand issues related to health and education demand and educational attainment. The quality ofthese models varies, however, and even when estimated, these models are often not wellexploited in terms of simulations and policy analysis. Rather, the assessments do not presentsimple comparative statistics or address issues of how poverty would be affected by alternativepolicy choices. Perhaps most important, many of the assessments fail to distinguish betweendescription (e.g., how different income groups benefit from a certain level of spending on, say,primary health care) and policy analysis (how the benefits of a specified increase in primary

13

health care spending would be distributed across income groups and the consequent impact onhealth outcomes). The lack of sound policy analysis is most obvious for, although certainly notisolated to, the benefit incidence analysis in the reviewed assessments. Conducting sound policyanalysis is predicated upon a higlh degree of techlnical competence and skill in economic policyanalysis; the expertise necessary to do this cannot be underestimated. This underscores the needfor using all available expertise, analysis, and research from inside and outside the Bank.

Despite the advantages that qualitative work has in highlighting the perspectives and preferences ofpoor groups, and in explaining the mechanisms by which poverty-related interventions actthemselves out, the impact of such work on policy analysis has been relatively small. About 43percent of the assessments reviewedfor the analysis of social, political, and institutional issueswere found to be unsatisfactory for their use of qualitative/participatory work in policy analysis andthe incorporation of the social, political, and institutional context in policy analysis. For example,the qualitative work in one assessment indicates the heavy agricultural work done by women inrural areas and the fact that they were largely responsible for agricultural work because young menmigrated for work. It also provides case material to support the poverty assessment's statistical dataon the much larger loss of jobs among women in recent political and economic changes and theconsequent adverse effect on women's earnings. Yet, although this finding warranted morecomplete analysis of the gender dimensions, this was not undertaken. The assessment used thegender-disaggregated data to develop a poverty profile, but failed to use it to inform the policyanalysis. Thus, qualitative work is used only to "give life" to the statistical skeleton rather than toinform its policy analysis and recommendations.

Even when assessments incorporate social, political, and institutional perspectives and commissionqualitative and participatory poverty assessments (PPAs), they tend to use the findings asillustrations, rather than for sharpening analytical focus and argument. Furthermore, they oftenslhow little recognition of the need to place economic data and analysis within particular social,political, and cultural contexts. For example, the assessment for one country does not mention thatan economically backward region was not included in its surveys, nor does it acknowledge themany economically and socially vulnerable groups in the country. The impact of natural disasters(to which the country is prone) is not addressed, and the existence of vested interests, whilementioned, is not incorporated adequately into policy analysis. On the other hand, good policyanalysis on specific issues is contained in the Bolivia Poverty Assessment, which analyzes genderand ethnicity issues well and is also strong in its analysis of the way in which a complex web offormal statutes intersects with informal customary institutions regarding land tenure. This is a rareexample of undertaking analysis of the informal institutional context.

Combining quantitative and qualitative methods is consistent with the commitment in OD 4.15 tointer-disciplinary approaches to poverty reduction and is supported in the Bank's povertyassessments. However, it remains to be mainstreamed into the poverty assessments. About 57percent of the assessments were found to be satisfactory on combining quantitative and qualitativemethods. A best practice example is the assessment for C6te d'Ivoire. The assessment makesespecially good use of quantitative and qualitative methods to facilitate more robust discussion of anumber of issues. For example, the qualitative work highlighted the increasing and dynamicparticipation of women in informal sector activities and linked this to gender disparities in incomeand assets emerging from the quantitative data. It also indicated that women seemed able to take onnew activities in the informal sector more readily than men who had been made redundant. Thepoverty assessment analyzed the problems of this increased burden of work and responsibility forwomen and the possibility that women might gain more influence and authority in family contextsas a result of their increased responsibility. Furthermore, the higher statistical incidence of povertyamong migrants than Ivorians was linked to the qualitative report's finding of increasing tension

14

and conflict among these groups over access to land as cultivation was being expanded andintensified by more people pushed out of other sources of employment and income.

In addition to demonstrating the value of using combined methods in assessing poverty, thisreview shows that it is also important to coordinate the process by which methods are combined.4

For example, the authors of the qualitative study for one of the poverty assessments concludedthat their study might have achieved more specific focus had it been conducted after somepreliminary analysis of the quantitative survey data rather than immediately after the survey.This, they believed, would have helped identify issues that needed in-depth probing. In fact, apartfrom post-validation, the extent to which the qualitative and quantitative methods inform eachother at the conception stage is not clear for most poverty assessments. Even with respect to post-validation, issues for which the findings of the qualitative analysis do not square with the findingsof the quantitative analysis or vice versa are often glossed over.

Policy Recommendations

Given weaknesses of the policy analysis it is not surprising that over a third of the povertyassessments reviewed (37 percent) have policy recommendations that are broad and based on aweak foundation. In these cases, recommendations are based on current conceptual and generalthinking without any analysis of whether they will lead to desired outcomes (e.g., better health,more literacy, increased income for poorfarmers). Policy recommendations are Figure 4. More Clients Are Dissatisfied withgenerally made without a sense of priority, Recommendations than Staffespecially relative to some notion of cost-effectiveness.

100 .The results of the stakeholder survey show 80that, although most respondents were 6ssatisfied with assessment recommendations, ..there was a wide disparity between the views . 40 -

of Bank staff and country clients (Figure 4). 20Furthermore, a significant proportion of the L .country clients were dissatisfied with the 0prioritizing of recommendations (20 percent) Prioritization Actionabilityand with how actionable the O Bankstaff ECountry clients

recommendations were (27 percent). Within so,r-v Poverty Assessment Stakebolder Survey

the country client group, the mostdissatisfied clients were central government officials (34 percent dissatisfied).

The Bangladesh Poverty Assessment contains policy recommendations that are informed by thepreceding policy analysis. It provides detailed analysis of the cost-effectiveness of the safety netoptions and the distributional impact of public expenditures on health and education. This isintertwined with various institutional delivery options to suggest cost-effective deliverymechanisms. Portions of some other assessments (such as Thailand's analysis of child labor,

14. This confirms the analysis in "Combining Quantitative and Qualitative Approaches to Poverty Measurement andAnalysis: The Practice and the Potential," Soniya Carvalho and Howard White, World Bank, 1997. Excellentdiscussions of how the two methods can be combined are contained in: "Participation and Combined Methods inAfrican Poverty Assessment: Renewing the Agenda," David Booth, Jeremy Holland, Jesko Hentschel, Peter Lanjouw,and Alicia Herbert, DFID, 1998; "Contextuality and Data Collection Methods: Framework and Application to HealthService Utilization," Jesko Hentschel, Journal of Development Studies, vol. 35, No. 4 (forthcoming); "ImpactEvaluation: A Note on Concepts and Methods," Kene Ezemenari, Anders Rudquist, and Kalanidhi Subbarao, PRMPO,World Bank,1999; and "Poverty Reduction and the World Bank: Progress in Fiscal 1998," R99-46, World Bank, 1999.

15

Bolivia's analysis of rural credit market, Ukraine's analysis of the labor market, Kazakhstan'sanalysis of social protection measures, and India's analysis of human resources and safety nets)also show that tangible and well-founded recommendations result from good analysis.

The extent to which qualitative or participatory work informs the recommendations variesconsiderably. About 57 percent of the assessments reviewedfor the analysis of social, political,and institutional issues were found to be satisfactory in the degree to which theirrecommendations are informed by qualitative/participatory analysis and are consistent with thesocial, cultural, political, and institutional context. In the Albania Poverty Assessment, insightsand conclusions from the qualitative study inform recommendations, for example, on the type ofcredit provision in rural and urban areas. Another assessment also makes effective connectionsbetween qualitative findings and recommendations, although somewhat selectively. For example,findings from the qualitative analysis about why children were not going to school are used inconjunction with the statistical data to deepen understanding, but the improved understandingdoes not, for the most part, feed into the recommendations. The recommendations point tounhelpful parental attitudes, an issue that is contradicted by the discussion of qualitative findingsabout the distress parents felt about having to force their children into income earning.

In another assessment, the community-level research module does not inform therecommendations at all. For example, the statistical analysis finds that many women engage inpaid work for less than 35 hours a week. The accompanying recommendation is that the under-employment of women needs to be addressed. This recommendation overlooks other claims onwomen's time, such as childcare and domestic or community-level responsibilities. Furthermore,it ignores the community research, which found fairly widespread resistance to women'seconomic participation. Although the assessment recognizes the need to address resistance towomen's participation in the labor force, this is not followed up by recommendations on how todeal with this resistance or how to relate it to the work NGOs and other agencies are doing withwomen who suffer problems related to health, domestic violence, or community conflict as aresult of being over-worked or socially challenged. As with many other poverty assessments, therecommendations are not based entirely on the evidence.

5. Local Partnership and Consultation

The extent of client involvement andcollaboration in the preparation of a povertyassessment affects the level of commitment, Box 4. Weakness of process informationownership and assumption of responsibility complicates analysisfor implementing its recommendations. The For many of the assessments, information aboutkey questions to answer in assessing what they were trying to achieve is scarce. Theparticipation and consultation are: How process questionnaire sent to task managers of themuch have the various stakeholders 19 assessments and updates examined for this studyparticipated in the assessment process? Does yielded only 15 responses and almost all of thosethe govemment take "ownership" of the had gaps-some unanswered questions or evenrecommendations? And is there adequate whole sections for which no information wasfollow-up and dissemination? The available. This reflects both poor knowledgepreviously noted lack of clarity about goals management and high staff turnover, which often

means that task managers, after moving to otherhas implications for the ability to analyze jobs, are especially unaware of the disseminationpartnership. There is, furthermore, a lack of and follow-up to their assessments. The lack ofgood process information (Box 4), which good information makes it difficult to determinemakes it difficult to generalize across the whether the process by which the assessments wereassessments reviewed for this study. The undertaken was appropriate.

16

discussion in this section draws largely upon the results of the process review and country casestudies. The process review consisted of a survey of task managers, supplemented by file researchon 15 poverty assessments from the complete set of 19 assessments. '5

Time Adequacy

Poverty assessments are mandated for all Bank borrowers. At least five of the reviewedassessments were motivated largely by the need to meet this management requirement. Twoothers were motivated mainly by the need to provide an input for an anticipated part of theBank's or donor's lending program in a country. Poor planning on the Bank's part meant thatunder both these circumstances there tended to be pressure on the timing of the assessment,thereby affecting the character of the process, including the extent and depth of country-levelconsultation and collaboration. For instance, the assessment in one country was rushed in order tosupport the first adjustment operation and to set the stage for a social fund, even though the timepressure was inconsistent with a process that would increase understanding, ownership, andcommitment within the government. Where the Bank adapted the timeline to government needs,sometimes extending it beyond the Bank's original schedule, the country-level influence of theassessment was much enhanced. This is illustrated by the C6te d'Ivoire assessment (Box 5).

Box 5. C6te d'Ivoire: Adapting the timeline to government needs increases the in-country influence of the poverty assessment

At inception, there was reluctance in C6te d'Ivoire about doing a poverty assessment. The Bank's PovertyAssessment team proceeded steadily as a number of forces coalesced toward a heightened interest inpoverty issues within the government. Following the death of President Houphouet-Boigny, there began aslow and steady move toward increased political pluralism and more open political debate about povertyissues. The donors' insistence that aid programs be tied more closely to performance in the social sectorsand to the welfare of rural populations also contributed to the government's increasing appreciation ofpoverty issues. In this context, the most notable program was the HIPC initiative: the country could nothave qualified for debt relief unless the government put poverty issues on its agenda. The assessmentteam, through its well-structured dialogue, nurtured the government's increasing sensitivity to povertyissues, but to its credit, did so without rushing the process and allowing the various forces to run theircourse. Even though the assessment was ready for Gray Cover, the Bank allowed a whole year to passduring which the government prepared its national poverty reduction policy. The Bank formally releasedthe assessment only after the government announced its own poverty reduction plan, which incorporated alarge number of the recommendations of the poverty assessment and added further targets of its own.Donors also began using the government's targets to establish performance criteria and benchmarks fortheir programs as in the case of the HIPC program. Donors and the government seemed to have found arare case of policy 'lock-in.'

Source: Country case-study undertaken for this review, March 1999.

Developing Ownership

The fact that the Bank usually initiates a poverty assessment has implications for the preparationof the assessment, particularly at the beginning: efforts to reach out to governments and activelyseek their support and involvement are crucial. In Cote d'Ivoire, the assessment team's initiativein starting an early dialogue with a reluctant government paid off by building mutual trust andfostering greater government ownership.

15. The 15 poverty assessments are those for Bolivia, Costa Rica, Djibouti, Gabon, Guinea, Cote d'lvoire, Yemen,Albania, Azerbaijan, Kazakhstan, Ukraine, Bangladesh, India (two assessments), and Thailand.

17

In the reviewed assessments, the Bank Figure 5. Bank-Government Cooperationgenerally informed the governmentabout the initiation of the assessmentbefore the concept paper stage, but data o 100-limitations make it difficult to ascertain , 80-whether more than information-sharing _took place at this stage. In at least one 60 .case, however, the government was c 40first informed about the assessment 20after the concept paper had been 0

0written. The request for comments a

0 Concept Data Data Overallprompted initial refusal by the a. paper collection Analysisgovernment.

n Bank staff E Country clients

Regarding the degree to which Source: Poverty Assessment Stakeholder Survey

government participation is sought inplanning the content and process of the assessment, the stakeholder survey indicates room forimprovement in Bank-government cooperation, especially during the concept paper stage-only57 percent of the country clients and 36 percent of Bank staff say the effort was joint or mostlyjoint (Figure 5).

"Who" participates is just as important as "how much" they participate: both need to bedetermined in light of poverty assessment goals and the country context. Evidence from theprocess review suggests that the bulk of partnership and consultation with governments whilepreparing poverty assessments has been with cross-sectoral central ministries (Finance, Planning)and specialized policy and analytic units (National branch of the Regional Unit for TechnicalAssistance in Costa Rica or Unidad de Analisis de Politicas Sociales in Bolivia), but thatoperating ministries (line agencies) and local governments have been much less involved. Often,their involvement has been limited to providing information.

Two of the four country case studies validate the finding from the process review. Theassessments in both these countries focused on top policy-making agencies, but they failed toreach lower levels of government both at central and local levels in any significant way. Program-level staff in these countriesremained, for the most part,untouched by the assessments.

Figure 6. Influence of Stakeholders on AssessmentsThe stakeholder surveyfurther validates the finding c 100 v

CDabout local governments.When asked how much = 80 -influence central governmentofficials, local government X 60

0officials, NGO staff, and E 40technical experts had on the Ifinal assessment report, only 20 -

small minorities of bothcountry clients and Bank staff 2 0

thought that the local Central Govt. Local Govt. NGOs Technicalgovernment officials had expertsstrong or moderate influence l Bank staff M Country clients

(Figure 6). Since operating S .. rce: Poverty Assessment Stakeholder Survey

18

agencies and local governments are responsible for implementing the policies of the government,the lack of their involvement in the assessment preparation process hampers subsequentimplementation of its recommendations. On the other hand, reaching and engaging the interest ofa broad range of government officials in cross-sectoral and operational agencies throughout theassessment process enhances the country-level influence of the assessment. This is illustrated bythe case of Gabon (Box 6).

Box 6. Gabon: Broad-based government participation improves the chances that povertyassessment recommendations will be owned and implemented

Taking advantage of a receptive government, the poverty assessment in Gabon was carried out in closepartnership with the government. While the Ministry of Planning was the Bank's main counterpart for theassessment, it set-up an Inter-Ministerial Technical Committee, including representatives both from cross-sectoral as well as line agencies, to oversee the assessment work. The committee was given theopportunity to comment and actively participate in each step of the assessment's elaboration (Issues Paper,White, Yellow, and Green Covers). The successive drafts were also sent to a wide array of cross-sectoraland line ministries, they were kept notified of upcoming missions, and they were consulted duringmissions beyond simnply for information-gathering purposes. The govermnent was preparing its ownpoverty reduction strategy concurrently with the poverty assessment and there was a great deal of iterationand interaction between the two efforts. The result was a government poverty reduction strategy that drewon the poverty assessment, and an assessment whose approach followed the broad lines of thegovernment's poverty reduction strategy, as well as quick action on a number of recommendations.

Source: Process review undertaken for this review, 1999

A majority of the assessments involved NGOs and civil society in some capacity. However, thestakeholder survey indicates that roughly half ofthe country clients thought that NGOs andtechnical experts had exerted "only a slight influence or none at all" on the final assessmentreport. Furthermore, the process review indicates that in a majority of the cases internationalrather than national NGOs were involved in the assessments. While this reflects the low capacityof many national NGOs in policy matters, it also has implications for the local ownership of theassessment.

However, the appropriateness of these features of the process can only be gauged in the contextof the specific goals of the assessment. If influencing the country poverty debate and thegovernment's poverty reduction policies was the goal, then these features of the process would beinappropriate. According to a recent Bank paper "... influencing policy requires more than writinga policy document. A broad dialogue is necessary in order to foster a greater understanding ofpolicy alternatives, present a good analysis of these alternatives, increase support of the evolvingpolicy, raise awareness of the problem thereby shiftingpolicymakers' approaches, and createpolicy space to enable previously sensitive issues... to be placed on the political agenda. Thisbroader dialogue with different people in society including the poor and those who will beresponsible for implementing the policy, creates a higher chance of the policy beingimplemented.""le If, on the other hand, improving the data collection capacity of the statisticalagency in a country was the only goal, these features would be of less concern.

16. Caroline Robb, Ibid.

19

Dissemination and Follow-up

Getting the maximum impact from a poverty assessment depends on the level of dissemination. Thewide transmission of the main messages of an assessment through a carefully managed campaignimproves the chances of a healthy debate and subsequent internalization of the messages and theirimplementation. All the reviewed assessments adopted some form of dissemination, especiallythrough newspapers, but the extent of dissemination varied widely. The Azerbaijan and Bangladeshassessments had significant broadcast exposure as well as print. In Azerbaijan, two majorworkshops (which were both locally televised and one of which was multi-audience) and severalseminars were held for government officials using the translated Green Cover report. Missions heldmeetings with NGOs and donors. Logistical help from UNICEF facilitated wide coverage in localnewspapers. A website of the survey data is under preparation. Formal distribution of the reportincluded the Finance Ministry, PlanningMinistry, Sector Ministries, CabinetMembetrs, donors andsnumes nGs Figure 7. More Clients Are Dissatisfied with

members donor, an ueou Gs Follow-up, More Staff with DisseminationThis dissemination effort increased generalawareness of the report and receptivity to its 100findings. In some other countriesdissemination was limited to a single 80 -workshop for a small group of government 60 -officials held in the capital city.

., 40

While local translations are a relatively low- a 20 -cost way of improving receptivity and 0 oreadership, it is striking that about half of Dissemination Follow-upthe assessments were not translated into thelocal language. If the goal of the assessment EOBank staff ECountry clients

is to foster a dialogue and debate and S.rce: Poverty Assessment Stakeholder Survey

influence policy-making at the countrylevel, wide accessibility to it is critical. In a number of cases, task managers pointed out the lack ofbudgets for dissemination: in one country this meant that the mission to the country to present thefinal draft to the government had to be cancelled, preventing a fornal discussion of the keymessages with the government. In another country a number of dissemination activities weredropped due to budget constraints. The stakeholder survey shows that significant proportions ofcountry clients and Bank staff (about one-third of each group) are dissatisfied with the overalldissemination effortfor assessments in their countries (Figure 7). More than half of theNGOs aredissatisfied.

In only 6 of the 15 assessments in the process review did the assessment teams identify the agenciesresponsiblefor undertakingfollow-up. Country-level follow-up to a poverty assessment improvesthe chances that its recommendations will be implemented. Identifying the agencies responsible forimplementing the recommendations-and addressing implementation constraints and challengesthey face-improves the chances that the recommendations will be implemented. In fact, the soonerthese agencies are identified and involved in key decisions about the assessment, the betterinformed their response is when it comes to the actual implementation of the recommendations. InGabon, delegating responsibility for the dissemination and follow-up of the poverty assessment tothe same Inter-Ministerial Technical Committee that had overseen preparation of the assessmentcontributed to continuity, ownership, and more responsiveness in implementing therecommendations.

In most cases, there was little systematic discussion in the country of the next steps to the povertyassessment. Consequently, it is difficult to envisage strong follow-up except in cases where

20

proactive governments quickly internalized and fully owned the recommendations of theassessment. Furthermore, only about half of both the country clients and Bank staff interviewed inthe stakeholder survey expressed satisfaction with the follow-up to poverty assessments. Amongcountry clients and Bank staff, central government officials were most satisfied with the follow-up,but even here a third were dissatisfied (Figure 7). Furthermore, the stakeholder survey indicates theneed for deciding the responsibility for follow-up more clearly: a majority of the centralgovernment officials said that the central governrment had in fact assumed responsibility for thefollow-up to the assessment, but only 27 percent of technical experts, 17 percent of NGOs, and 4percent of Bank staff agreed with that view.

Institutional Development

The production of a poverty assessment, like most other economic and sector work, offers anopportunity to increase client capacity in policy analysis and strengthen institutions. In this case,achieving institutional development will require that the country client actively participate in theplanning and execution of the poverty assessment. Of course, where participation and consultationare weak, the ability to affect long-term change is similarly weak.

Capacity building is explicitly identified as a benefit of participatory economic and sector work bythe World Bank Participation Sourcebook (p. 207): "Participatory country economic and sectorwork (CESW) increases in-country capacity for subsequent analysis in the same or new areas ofconcern. It treats CESW notjust as preparation for Figure 8. Local Capacity Building Through Assessmentspolicyformulation andinvestment decisions but asa development activity 100

itself improving the 80capacity within member .s

a 60 countries to take over theproduction of technically 40

informedpoliciesfor 20themselves." Xl

0

The appropriate level of Data Data analysis Policy Povertyinstitutional development in collection formulation monitoring

a specific case depends on DBank staff SlCountry clients

the goals of the povertyassessment and country Source: Poverty Assessment Stakeholder Survey

conditions. The key questions to judge institutional development in poverty assessments are theextent to which the assessment builds local capacity to measure and analyze poverty, formulatepoverty reduction policies, and implement the recommended policies.

Using local (rather than international) agencies, institutes, and experts through the various stages ofthe assessment helps to transfer learning to the country. The Bangladesh Poverty Assessmentexplicitly aimed at building the capacity of the country's statistical agency in data collection andanalysis, and succeeded. But in about half of the reviewed assessments, local consultants wereeither not involved at all or involved only in a minor way. In one of the four case study countries,the effort to involve local technical experts in undertaking background analysis paid off by buildinglocal capacity but also pointed to the fact that internal Bank deadlines are inimical to capacity-building efforts. Building capacity takes time.

21

The results of the stakeholder survey found that a substantial proportion of clients and Bank staff(up to 42 percent) felt that the assessment had not increased local capacity even moderately (Figure8).

6. Poverty Assessment Efficiency and Impact

This study evaluated and scored several attributes of each assessment using consistent criteria.The attributes scored (and the basis for the scoring) included assessment quality (based on thequality review), and the extent of partnership and consultation, the amount and effectiveness ofinformation sharing and knowledge transfer, and the timeliness and time adequacy of theassessment (based on the process review and country case studies). In addition, the overall impactof each poverty assessment was evaluated in terms of its contribution to or influence on thepoverty debate, the design and implementation of policy, and institutional development (based onthe process review and country case studies). Each poverty assessment was assigned a rating foreach of the attributes. Overall, slightly over a third of the poverty assessments achieved highcountry-level impact, and slightly under a third of the assessments achieved only low impact.

The evaluated attributes are strongly associated with the overall measure of impact as measuredby the simple correlations shown in Table 2, and all of the correlations are statistically significant.The extent of partnership and consultation and information sharing and knowledge transfer areclearly strong indicators of overall impact. The magnitude of the correlations suggest thattimeliness and time adequacy, though important, are less strongly associated with impact than theother three attributes.

Table 2. Correlation between Overall Impact and FourMajor Attributes of Poverty Assessments

Affribute CorrelatdonQuality 0.82Partnership and consultation 0.85Information sharing and knowledge transfer 0.78Timeliness and time adequacy 0.53

What other evidence can be used to help decide which attributes of the poverty assessments needto be strengthened? The stakeholder survey responses can be aggregated into categories thatvirtually match the attributes reported in Table 2, and Figure 9 shows the proportion ofstakeholders that are satisfied or dissatisfied with each attribute. Stakeholders are clearly leastsatisfied with the degree of partnership and consultation and the extent of knowledge transfer-attributes that are strongly associated with the overall influence of poverty assessments. Thisevidence strongly suggests that strengthening the partnership and consultation and knowledgetransfer activities of poverty assessments should have priority as a means of increasing theiroverall impact. Of course, this would make the most sense when the poverty assessments havebroad objectives. A narrow and more focused poverty assessment may not benefit as much frompartnership or knowledge transfer.

22

The review found that poverty assessment direct costs and staff years are not significantlycorrelated with country-level influence, quality, partnership and consultation, information sharingand knowledge transfer, and timeliness and time adequacy. In other words, spending more timeand money is not strongly associated with better poverty assessment outcomes. The correlationbetween inputs (direct costs and staff years) and outcomes (the five major attributes) arepresented in Table 3.

Table 3. Correlation between Inputs and Outcomes

Attribute CorrelationDirect Cost

Country-level influence 0.09Quality 0.02Partnership and consultation 0.20Information sharing and knowledge transfer 0.05Timeliness and time adequacy -0.15

Staff YearsCountry-level influence 0.40Quality 0.10Partnership and consultation 0.30Information sharing and knowledge transfer 0.20Timeliness and time adequacy 0.20

While different types of economic and sector work vary in scope and quality and cannot bestrictly compared, the average cost and time for poverty assessments and updates ($213,000 and1.7 staff years) are roughly similar to those for Public Expenditure Reviews and CountryEconomic Memoranda and updates ($240,000 and 1.8 staff years) in the countries in this review,with substantial overlap in their ranges.

Figure 9. Overall Satisfaction

100% -

80%

60%ied

40%

20%

0%/Relevance Efficacy, Partnership, Knowledge Overall

quality consultation transfe r

Source: Poverty Assessment StakehoJder Survey

23

7. Recommendations

Improve poverty assessment quality and process. The country management and regional ChiefEconomist's office should increase management oversight of the analytical quality of povertyassessments as well as their partnership and consultation and knowledge transfer aspects. TheBank should undertake self-evaluations of poverty assessments. The Poverty Reduction Boardand Thematic Groups should support the Regions with training and advice, and through providingtools and best practice examples on poverty analysis, engaging partners, supporting collaborationand local participation, and developing local capacity through knowledge transfer. The PovertyReduction Board and Thematic Groups should ensure that the Regions have adequate andappropriate guidance, including on best practices, for the preparation of poverty assessments. ThePoverty Reduction Board should report the findings of the Bank's self-evaluations in the AnnualProgress Report on Poverty.

Focus on goal and priority-setting. Country management should ensure that povertyassessments - from the concept paper on - define clear goals and priorities, provide adequatejustification for prioritization of issues and for any selectivity in scope and coverage, adequatelyreflect country conditions, and link work programs and analysis to goals. The regional ChiefEconomist's office should monitor and provide quality control for this. The Poverty ReductionBoard and Thematic Groups should support the Regions with advice and best practice examples.

Meet targets for poverty assessment completion. The Regions, in collaboration with thePoverty Reduction Board, should set realistic targets for completing poverty assessments. Tounderscore the importance of the poverty reduction agenda, senior management should providethe institutional inputs and incentives to produce quality poverty assessments effectively and ontime.

25

Bibliography

Aronson, Dan. 1994. "Participation in Country Economic and Sector Work." World Bank,Washington, D.C.

Booth, David, Jeremy Holland, Jesko Hentschel, Peter Lanjouw, and Alicia Hebert. 1998."Participation and Combined Methods in African Poverty Assessment: Renewing theAgenda." United Kingdom Department for International Development, London.

Carvalho, Soniya, and Howard White. 1997. "Combining Quantitative and QualitativeApproaches to Poverty Measurement and Analysis: The Practice and the Potential."World Bank, Washington, D.C.

Clark, John, Lawrence Salmen, and Andrew Norton. 1993. "Participatory Poverty Assessment."World Bank, Washington, D.C.

Dudwick, Nora and Catherine Wanner. 1996. "Ethnographic Study of Poverty in Ukraine:October 1995-March 1996." World Bank, Washington, D.C.

Dudwick, Nora. 1995. "A Qualitative Assessment of the Living Standards of the ArmenianPopulation: October 1994-March 1995." World Bank, Washington, D.C.

. 1996. "Albania Poverty Assessment Part I: An Ethnographic Study of the RuralMigrants in Peri-Urban Tirana and Shkodra." World Bank, Washington, D.C.

. 1996. "Albania Poverty Assessment Part II: An Ethnographic Study of the Small-Scale Credit Program in Tirana and Elbasan Districts." World Bank, Washington, D.C.

Ezemenari, Kene, Anders Rudquist, and Kalanidhi Subbarao. 1999. "Impact Evaluation: A Noteon Concepts and Methods." Poverty Anchor, World Bank, Washington, D.C.

Hanmer, Lucia, Graham Pyatt, and Howard White. 1997. "Poverty Assessment in Sub-SaharanAfrica: What Can be Learned from World Bank Poverty Assessments? ISS, The Hague.

Hentschel, Jesko. Forthcoming. "Contextuality and Data Collection Methods: Framework andApplication to Health Service Utilization." Journal of Development Studies, vol. 35, No.4.

Norton, Andrew and Thomas Stephens. 1995. "Participation in Poverty Assessments."Participation Series, World Bank, Washington, D.C.

Robb, Caroline. 1999. Can the Poor Influence Policy? Directions in Development Series, WorldBank, Washington, D.C.

Salmen, Lawrence F. 1995. "Participatory Poverty Assessment: Incorporating Poor People'sPerspective into Poverty Assessment Work." Social Development Papers, No. 1 1, WorldBank, Washington, D.C.

Sauma, Pablo, Carmen Camacho and Manuel Barahona. 1997. Percepciones Sobre la Pobreza enComunidades Pobres de Costa Rica. World Bank, Washington, D.C.

26

World Bank. 1999. "Poverty Reduction and the World Bank: Progress in Fiscal 1998." ReportNo. 99-46. Washington, D.C.

1998. "Quality of ESW in FY98: A QAG Assessment," Quality Assurance Group,Washington, D.C.

1997a. "Improving the ESW Quality: An Approach Paper," memo from Prem Garg,August 11.

1997b. Taking Action to Reduce Poverty in Sub-Saharan Africa. Development inPractice Series. Washington, D.C.

1996. "Poverty Assessment: A Progress Review." Operations Evaluation Department,Report No. 15881. Washington, D.C.

1996. The World Bank Participation Sourcebook. Washington, D.C.

1995. Social Assessment Guidelines. ESSD Network, Washington, D.C.

1993. Poverty Reduction Handbook. Washington, D.C.

1991. Assistance Strategies to Reduce Poverty. Policy Paper. Washington, D.C.

1991. Operational Directive on Poverty Reduction, OD 4.15.

1990. World Development Report 1990: Poverty. Oxford University Press, New York.

"Evaluation of Economic and Sector Work-A Review." 97-55, September 15, 1997,Committee on Development Effectiveness.

Discussion regarding Poverty Assessment: A Progress Review. Minutes 96-64,September 9, 1996, Committee on Development Effectiveness.

(A list of reviewed Poverty Assessments is attached as Annex B.)

27 Annex A

Evaluation Framework and Criteria

This annex describes the framework for each of the four distinct parts of the poverty assessmentevaluation: country case studies, stakeholder survey, quality review, and process review. Thesections that follow outline the criteria applied in each part of the evaluation.

COUNTRY CASE STUDIES

The country case studies assessed the within-country impact and influence of their respectivepoverty assessments. The main methods for the studies were semi-structured in-depth interviewsand focus group discussions with key informants. The studies sought the views of manystakeholders: resident mission staff, government policymakers in sectoral and cross-sectoraldepartments, key local government officials, policy observers from academia and privateinstitutions, policy-oriented/advocacy-based NGOs, NGO service providers, representatives ofcommunity-based organizations, particularly those directly involved in the poverty assessment,and private sector organizations. The semi-structured interviews were also used to identifyadditional respondents. Secondary sources, including within-country policy reports produced bythe government, policy-focused NGOs, and university departments; Bank reports and evaluations(memos, working papers, gray cover material); and newspaper articles were reviewed to theextent possible. The methods and sources were triangulated as appropriate.

The organization of the case study reports is flexible to accommodate variability in countrycontext and experience. The reports share a common structure:

I. Background and BaselinesII. Poverty Assessment ProcessIII. Post-Assessment Changes in Poverty Debate, Public Policies and Programs, and

Institutional CapabilitiesIV. Explaining ChangesV. Conclusions

The remainder of this section describes the content and characteristics of these five mainheadings.

Background and Baselines

* A brief description of the character of the poverty debate within the country before thepoverty assessment. (Cases probably range from official denial that poverty is a severeproblem to decades of serious analysis and anti-poverty effort.) What were the major themes?(Sectoral, regional, long-run vs. immediate relief, etc.) Areas of consensus and of sharpdisagreement? Were non-official groups (NGOs, church-affiliated groups, etc.) andindividuals (academics, journalists) active participants in debate?

* The general character of the policy formulation process and institutions with respect topoverty reduction. Was any central agency responsible? Was there good or poor co-ordination among key ministries/agencies? Did the legislature play a role? Councils ofgoverning political parties?

* A brief description of the major categories of government anti-poverty action before theassessment.

28 Annex A

The Poverty Assessment Process

The "Process " review provided much of the information for this section (see "Process Review"section below). Interviews with country participants on the process provided differentperspectives and additionalfacts. See also "The Poverty Assessment Process: A Checklist,"below, which guided this part of the review.

* The analytic process: Who was involved? At what stages (e.g., initial discussions,conceptualizing framework, conducting primary analysis, compiling and analyzing data,policy analysis, dissemination?). What were the substantive and methodological debates?What were the major constraints? (Lack of data? Poor cooperation from key agencies due tolack of capacity, interest, or resources? Pressure from the Bank to complete the assessmentrapidly? Other constraints?)

* Partnership and consultation, in and outside of government: Who? How? With what kinds ofresponses? (For Armenia and Cote d'Ivoire: a brief discussion of the participatory povertyassessment.) A stakeholder participation matrix might be constructed by stakeholder typewith columns referring to the level of participation (inform, consult, partnership, and control)and rows referring to assessment stages (identification, planning, implementation, andmonitoring and evaluation).

* Dissemination: To what groups? Was the assessment simply distributed, or was thereaccompanying discussion and explanation?

* Follow-up action.

The Poverty Assessment Process: Key Questions

Background factors affecting the process* Did the country/government have prior experience with related Bank studies or activities that facilitated (or

complicated) the assessment process: e.g., relevant Sector Studies, established Living StandardsMeasurement Survey, etc.?

* Was the timing of the assessment an issue (viewed by the government or major agencies as particularlyinconvenient or conducted at a time when key politicians or officials were distracted by other matters)?Were there any important disruptive events (changes or splits in government, changes in key officials,natural catastrophes, financial crises, etc.) during the period covered by the report?

* Did members of the assessment team have substantial prior experience and contacts in the country? Howlong did they spend in the country? Did they travel outside the capital much?

Poverty assessment initiation* Who initiated the assessment process (e.g., government-requested, donor-driven?). Which stakeholders

were identified during the early stages of the assessment? (e.g., ministry officials at different levels, localgovernment officials, other policy actors and analysts, beneficiary groups, private sector organizations orNGOs)? Who, on reflection, should have been identified at this stage?

Who was involved in gathering information, analyzing issues, and formulating recommendations for theassessment?* Within the government: which were the key counterpart agencies? What were their capabilities, and their

attitudes toward the task? Were there significant rivalries among agencies that affected the process? Wereagencies left out that later raised criticisms? Were there major clashes of views among agencies? Did theassessment process improve understanding (or perhaps even generate new areas of consensus) amongconflicting agencies?

* What stakeholders outside of the central government were involved in the process? Consider interestgroups (labor or farmers' unions, business associations, etc.), NGOs and church-affiliated groups,

29 Annex A

provincial or local government officials, commnunities (Participatory Poverty Assessment: C8te d'lvoireonly), other bilateral and multilateral donor agencies?

* To the extent that any such groups were involved: at what stage? One time only, or on an ongoing basis?How were they asked to participate? (provide information, suggest issues, comment on proposals, etc.?)Through what channels? What was the general tone and temper of participation? (analytic, polemic,consensual, conflictual?)

* Where there were strong differences of opinion or judgment within the government or among otherparticipants, were these differences noted and discussed in the assessment?

* Were practical problems of implementation discussed? Did those discussions include those actuallyresponsible for later implementation?

How was dissemination carried out?* Was the assessment promptly translated?* To what agencies and groups were copies provided?* Were publicity events (release announcements, seminars, workshops, etc.) organized:

- within key government agencies?- for the legislature, or legislative commissions?- for the media?- for nongovernmental groups (NGOs, universities, private institutions, interest groups)?- for state or local government officials or communities?- other?

* What was the general tone of the reception, by any of the above groups?* Were any groups that took part in preparing the assessment not included in the dissemination process?

Implementation and follow-up* Did the assessment specify indicators of progress? Did it specify what agencies or groups were responsible

for collecting those indicators? Are they doing it? If not, why not? Has the assessment helped to generatenew types or sources of data?

• Poverty assessments often identify important areas of inadequate knowledge and data. Have there beenfollow-on efforts (governmental, nongovernmental, World Bank, other donors) to address some of thesegaps?

* If there have been major lags in implementation, where have the major obstacles appeared? For instance:- failure at high levels to adopt new policies or alter old ones- legislative resistance to approving necessary legal measures- failure to allocate adequate budgetary resources- lack of interest and/or capacity in key implementing agencies (at national or sub-national levels)- conflicts among agencies or levels of government- resistance from outside the government- inadequate external assistance.

* Do there seem to be any links between the main sources of resistance or lack of capacity to implement, andthe earlier process of preparing the assessment?

Post-Poverty Assessment Changes in Poverty Debate, Public Policies and Programs, andInstitutional Capabilities

This section described the recent evolution of debate and dialogue, programs/expenditures, andinstitutions. The section also noted to what extent and on what points these trends were in accordwith or divergedfrom the recommendations in the poverty assessment.

* Evolution of poverty debate(s) within the country. Also, any evolution in thinking; it ispossible that particular ministries or other actors disagreed with the assessment on particularissues, but nonetheless altered their thinking as a result of the debate. How have concepts ofpoverty changed among officials and policymakers? How has the policy discourse changed?

30 Annex A

* Shifts and trends in government policies and programs.

- The process of formulating policies (what changes have occurred in the policy priorities?Has there been "institutionalization of the poverty agenda" in the country?

- Design- Implementation (what policy changes have been there in implementation - policy in

principle versus policy in practice?) Shifts in allocation of public funds (budgeted and,where data are available, expended). Additional information on implementation.

* Institution-building (inside or outside the government)

- Strengthened capacities, restructuring, other reforms or improvements in important anti-poverty analysis and action agencies?

- Improved coordination among agencies, improved strategic planning?

* Evolution of dialogue between the Bank and the government. (The "Process" questionnairewill provide some information on this point from the Bank's perspective.)

Explaining Changes: What Was the Impact of the Poverty Assessment? What FactorsExplain its Impact or Lack of Impact?

This is the analytic heart of the country studies. It moved beyond before-after description to with-without explanation. The section also tried to untangle or disaggregate the concept of "theimpact of the poverty assessment" into more specific components: aspects ofprocess and content.While the structure and details vary, each case considered:

* Broad post-assessment economic trends or events. Might some of the (favorable orunfavorable) changes described in the preceding section have occurred without theassessment, because of changes in the country's economic situation?

* Broad (or particularly salient) post-assessment political events. Might some of the changesdescribed in the preceding section have occurred without the assessment, because of politicalfactors? More specific aspects of the politics of poverty reduction efforts may also beconsidered. For instance, if post-assessment policies have been quite different from thoserecommended in the assessment, part or most of the explanation may be substantivedisagreements (perhaps ideological, perhaps with other sources) among key groups or actorswithin the country regarding appropriate policies; or rivalries among implementing agencies;or adamant opposition from vested interests (through many possible channels, including thelegislature).

* The content and quality of the poverty assessment, including:

- The extent to which it either adopts stakeholders' opinions or addresses those viewsexplicitly and explains why it reaches different conclusionsThe extent to which it realistically assesses problems of political commitment andcapacity

- The extent to which it realistically assesses administrative, technical, and organizationalproblems of implementation

- Any other substantive features (of commission or omission) that help explain impact orlack thereof.

31 Annex A

It would be useful to summarize reactions to the assessment content for each major set ofactors or stakeholders for which information is available (e.g., political leaders, governmentofficials, NGOs, interest groups, academics, etc.) as a way to explore reasons for impact orlack thereof. There may be major differences in responses within each category: Theunanimity or divisions of opinion are useful to analyze. The stakeholder survey will provideinformation on perceptions.

* Aspects of the process used to produce the poverty assessment, as described in the precedingsection. Again, the varying reactions of stakeholders will be important to assess. Someguiding questions are in "Eliciting Stakeholder Reactions: Questions to Ask," below.

* Other factors that help explain impact or lack thereof.

Eliciting Stakeholder Reactions: Questions to Ask

* What were the stakeholder reactions to the poverty assessment? (Did they feel ownership of the assessmentprocess or aspects of that process? Did institutions start to lobby more effectively for policy change,resources etc.? Did government become more accountable, efficient, responsive, transparent?)

* What effects did stakeholder involvement in the assessment or in its dissemination have on otherstakeholders? (Did stakeholders subsequently draw counterpart institutions into the policy debate? Wasthere a sustained and more systematic inclusion of marginal groups/institutions in policy dialogue?)

. What impacts did the assessment have on relations between stakeholders or networks of stakeholders?(Did the assessment process help to build trust or new norms of interaction between stakeholders? Did itencourage attitudinal change among stakeholders toward each other - including between nationalgovernment and donors, within and between line ministries or other govermment departments, betweennational and local government, between government and nongovernmental policy analysts or other civilsociety groups? Did the process give rise to conflict? Was this conflict resolved? If so, how?)

. What did these changes in relationships, mean for post-assessment policy delivery framework? (A moreacceptable climate, or "policy space" for dialogue between government and Bank? A more coordinatedapproach between some or all the stakeholders? More effective implementation of policy directives?

Conclusions

* A short statement of those features of the assessment process and content that seem to havebeen most effective in increasing the sophistication of content and scope of participation inthe poverty debate within the country, and dialogue with the Bank; and in improving thefocus and efficacy of anti-poverty programs, policies, and institutions.

* A summary of weaknesses of the assessment process or content that probably detracted fromits impact.

STAKEHOLDER SURVEY

The stakeholder survey obtained country client and Bank staff perceptions about the usefulnessand impact of the sampled poverty assessments. The survey sought feedback on overallsatisfaction with the assessments as well as satisfaction with the following specific attributes:

Relevance - the appropriateness of the assessment's goals, coverage of key populationgroups, coverage of key issues, and realism of the recommendations

32 Annex A

Efficacy - achievement of the objectives (as identified by the respondents), and impacton the Bank's Country Assistance Strategy

Internal Quality - the clarity of the objectives, soundness of the analysis, rigor of themethodology, treatment of rural poverty, quality of the recommendations (prioritized,alternatives offered, actionable), and specific skills of the assessment team

Partnership and Consultation - how well the Bank encouraged participation, whatimpact participation had on the final product, dissemination of findings, and views onfollow-up actions and responsibilities

Institutional Development - stakeholder perceptions of improvement in local capacityto collect and analyze data, formulate policies, and monitor poverty reduction.

Identification of country clients. Task managers of poverty assessments were asked to provide alist of stakeholders (in central government, local government, NGOs, and technical experts) whowere involved in the preparation of the assessment or engaged in implementing itsrecommendations. The task managers were also asked to include the names of stakeholders whowere not consulted due to time or resource constraints but who, in their view, should have beeninvolved. Resident mission chiefs (or country directors in countries without resident missions)were also asked to provide a list of names of stakeholders who should have been involved in thepreparation of, or follow-up to, the assessment. The overlap between the two lists was eliminatedand a random sample, stratified by stakeholder type, was drawn.

Identification of Bank staff Each poverty assessment task manager was interviewed, as was eachresident mission contact suggested by the task managers.

The survey covered 106 interviews (81 country clients, 25 Bank staff) in three languages(English, French, and Spanish) in 15 countries.

QUALITY REVIEW

International experts examined poverty assessments for their quality both from an economicperspective as well as with respect to social, political, and institutional issues. The criteria foreconomic quality derive from OD 4.15. The criteria for social, political, and institutional qualityderive from 0D4. 15 in conjunction with other Bank documents and guidelines. The importanceof an interdisciplinary approach to poverty reduction which takes into account "cultural,sociological, and political factors" is emphasized in OD 4.15 (page 293). Moreover, it also statesthat poverty assessments should identify the poor "by gender, age, and ethnic characteristics,where they live, and how they fit into the consumption and production activities of the economy"(page 295). In addition, a number of other Bank initiatives have highlighted the importance ofsocial, political, and institutional issues in poverty reduction, for example, the Participation andLearning Group (1990) which led to Clark, Salmen, and Norton (1993), Aronson (1994), Salmen(1995), and Norton and Stephens (1995) and culminated in the Participation Sourcebook (1996),as well as the Social Assessment Guidelines (1995).

Criteria for Economic Quality

* Clarity of goals and consistency with OD 4.15* Thoroughness and quality of the poverty profile, including poverty lines and characteristics of

the poor

33 Annex A

* Thoroughness and quality of the policy analysis and whether it is based on a sound analyticaland empirical methodology and is rooted in the analysis of the causes of poverty at the macroand micro levels

* Quality of coverage, with prioritization and selectivity adequately justified, of the threeaspects of a poverty reduction strategy (promoting efficient, labor-intensive growth; humanresource development; and targeted interventions and safety nets)

* Thoroughness and quality of the policy recommendations and whether they follow from thepolicy analysis and are prioritized based on an analytical foundation with tradeoffs beingexplicitly recognized or explored

Criteria for Social, Political, and Institutional Quality

* Extent to which a multi-dimensional approach to definitions and profiles of poverty isadopted

* Extent to which the qualitative/participatory work is based on a sound methodology* Extent to which the qualitative/participatory work and the quantitative household survey

work are coordinated (sequenced or combined) to enrich the analysis, to explain issues,and/or confirm/refute hypotheses

* Extent to which the policy analysis has internalized the qualitative/participatory work andreflects an understanding of the social, cultural, political, and institutional context of thecountry

* Extent to which recommendations are informed by the qualitative/participatory policyanalysis and are consistent with the social, cultural, political, and institutional context of thecountry

PROCESS REVIEW

Institutional files were examined to compile information about poverty assessment preparation, aswell as dissemination and follow-up. Information missing from the files was gathered from thetask managers.

Preparation

Time frame . What was the proposed date for the gray cover?* What was the actual date for the gray cover?. Was the initial time frame realistic?* What were the causes of delay, if any?. Did time constraints affect the content/technical quality?* Did time constraints compromise consultation with/participation of country clients?

Initiation . Was the assessment undertaken at the request of the government, aid groupmeeting, or some other entity?

. Was the assessment the first sector work done for the client, so that it had noprevious studies/resources to build upon?

* Was the issues/concept paper drafted by Bank staff alone, Bank staff withgovernment input, or government with Bank input?

* If the government did not initiate the assessment, was the government first formallyinformed of the assessment before the issues/concept paper, during its preparation,after most data collection was completed, after completion of the analysis, aftercompletion of the report, or never?

Within-Bank Collaboration . Was the poverty assessment task manager from PREM, HD, or some otherand Discussion Network?

* What was the skill mix of the assessment team?. What were the Bank departments from which comments were requested on the

assessment?

34 Annex A

. What were the Bank departments from which comments were received on theassessment?

. Did the peer review process make a significant impact on the final draft?* Did the management review process add value?* Did the preparation of the assessment coincide with the preparation of related Bank

economic and sector work and, if so, was there interaction between the variousreports?

. Did any of the following internal Bank processes have a significant negativeinfluence on the assessment?

- management aKtention/action - during execution, after completion- management changes

- internal organizational changes- task manager changes

- team member changes

- inappropriate skill mix- complications from donor coordination

* Did any of the following internal Bank processes have a significant positive influenceon the assessment?- management aKtention/action - during execution, after completion- management changes

- internal organizational changes- task manager changes

- skill mix

- budget constraints

- reliance on non-Bank sources for core team skills* Did the Bank have a resident mission in the country? If so, did mission staff

contribute to carrying out the assessment? What was their role?Within-country Collaboration . Which government agency has primary responsibility for designing poverty reductionand Discussion policies in the country?

* Which government agency has primary responsibility for implementing povertyreduction policies in the country?

* Did the assessment involve both of the above agencies in the preparation process?* Did the assessment involve high-level central government civil servants? If so, in

what aspect of the assessment were they involved and at what stages?. Did the assessment involve low-level central government civil servants? If so, in

what aspect of the assessment were they involved and at what stages?. Which line ministries were consulted? If so, in what aspect of the assessment were

they involved and at what stages?* Did the assessment involve high-level local government civil servants? If so, in what

aspect of the assessment were they involved and at what stages?. Did the assessment involve low-level local govemment civil servants? If so, in what

aspect of the assessment were they involved and at what stages?* How many NGOs/international agencies were consulted?* Was collaboration with NGOs/intemational agencies hampered for any reason (e.g.,

govemment objections, issues of foreign relations)?. Were they the key NGOs/intemational agencies in the country?. Were other civil society members, including academics, consulted?* Were they the key ones?* Were some key civil society players left-out?* Did within-country discussions affect the assessment's scope/focus, range of

organizations consulted, terms of reference, implementation?* Were related poverty reports being prepared by the government during the

preparation of the assessment? If so, were all these documents prepared interactively?* Is there an officially announced government poverty reduction policy in the country?

When was it announced?. If not, what are the most important poverty reduction policies and programs of the

government?* If the government was developing a poverty policy during the preparation of the

assessment or after its completion, did the assessment influence the government's

35 Annex A

policy?* Were local workshops held to identify existing worklongoing research?* Were local workshops held to build a consensus view on an analytical approach, or

to adjust the assessment to the local discourse on poverty?. Were workshops held during the preparation of the assessment to foster a sense of

ownership? At what stage? Who attended? What topics were discussed/presented?Who were the speakers and their organizational affiliation?

* Were workshops held only in the capital city or in other places as well? If so, werethese rural or urban?

* Did the assessment involve the poor in the consultations? If so, was it throughsystematic qualitative work?

* What was the proportion of local to international consultants?* Were the local consultants that were hired the usual Bank consultants or first-time

consultants?* Who chose the local consultants - Bank, other donors, government, NGOs, others?* Was there an adverse change in the commitment to the assessment on the by key

counterparts during the preparation stages? If so, was it due to political changes,within-Bank changes, other?

* Did the assessment build capacity? If so, whose and for what?Budget . What was the total budget for this assessment?

* Did the budget enable/disable certain activities?* Did the use of trust funds enable studies that would not have otherwise been

possible?. If trust funds or other non-Bank sources of financing were used, what percentage of

the total budget did they comprise?

Dissemination & Follow-Up

Time frame * At what stage was dissemination of the assessment clearly identified?. At what stage was follow-up to the assessment clearly identified?* Were audiences for the assessment clearly identified, if so, at what stage and who

were they?* What was the first time the dissemination of the assessment was discussed with the

government - issues/concept paper stage, middle of preparation, after completion,never?

* What was the first time the follow-up to the assessment was discussed with thegovernment?

Initiation * Who initiated dissemination of the assessment - Bank, government, other?* Who initiated follow-up to the assessment - Bank, government, other?

Within-Bank Dissemination . How was the assessment disseminated within the Bank?and Follow-up * How was follow-up undertaken by the Bank?

- Bank lending?

- policy dialogue?

- high-level government commitment sought?- monitoring of key recommendations, and indicators thereof, agreed with

government?* Was there an agreement on performance indicators to monitor follow-up?* Was there a review of the poverty focus of the existing Country Assistance Strategy,

or if a new CAS was being prepared, did the assessment influence it?Within-Country * How was the assessment disseminated within the country?Dissemination and Follow- - workshops for central government?up - workshops for local government?

- workshops for NGOs/community organizations?- workshops held in capitol cities, towns, and/or rural areas?- debriefing population of the study area or the intermediary organization that

assisted with the study?- translation of report into local languages?- short summary of findings for widespread dissemination?

36 Annex A

- media coverage?. Was the completed assessment formally presented to the government?. Who were the main agencies to whom the final report was formally distributed

(finance ministry, sector ministries, planning ministries, local government, cabinetmembers, opposition leaders, local governments, donors, academics and privateinstitutions, NGOs: UNICEF, IFRC, labor unions, business and industrial associations,media/journalists, other)?

. At what stage was follow-up of the assessment first discussed with the government?* Was responsibility delegated to a within-country individual or group for ensuring

follow-up and implementation of recommendations from the assessment?. Was the resident mission involved in dissemination or follow-up? If so, what was its

role?. Was there collaboration/consultation with the various levels of government

responsible for implementing the assessment recommendations?* Was there collaboration/consultation between the government and NGOs and other

donors involved in implementing the assessment recommendations?. How was the assessment used by the government? As a source of data? As a tool

to guide the design of programs and policies?. Did the assessment include explicit measures to increase capacity within the country

institutions for dealing with poverty?. Did the assessment serve as a tool for other donors?. Was there any improved coordination between data producers and users?. Were practical issues and problems of implementing the recommendations of the

assessment discussed at the completion of the assessment - with central governmentofficials, with local government officials, with non-governmental stakeholders, withothers?

* Was there an adverse change in the commitment to the assessment by keycounterpart during the dissemination and follow-up stages? If so, was it due to politicalchanges, within-Bank changes, other?

Budget * Was the budget adequate for proper dissemination and follow-up?Role of the Poverty * Did you approach the Central Poverty Group for assistance?Anchor/Central Poverty . Did the Central Poverty Group provide adequate assistance?Group in the Bank

* Was the quality of the assistance satisfactory?. What specific kinds of assistance would you like from the Central Poverty Group in

the future in preparing poverty assessments?

37 Annex B

List of Poverty Assessments in the Review Sample

Country Poverty Assessment Title Date Of Completion Report No.(Fiscal Year)

Albania Growing Out Of Poverty May 30, 1997 (1997) 15698-ALB

Armenia Poverty Assessment June 10, 1996(1996) 15693-AM

Azerbaijan Poverty Assessment February 24, 1997 (1997) 15601-AZBangladesh From Counting The Poor To Making The Poor Count March 29,1998 (1998) 17534-BDBolivia Poverty, Equity And Income: Selected Policies For February 22, 1996 (1996) 15272-BO

Expanding Earning Opportunities For The Poor (Update)Chad Poverty Assessment: Constraints To Rural Development October 21, 1997 (1998) 16567-CDCongo Poverty Assessment June 16,1997 (1997) 16043-COBCosta Rica Identifying The Social Needs Of The Poor: An Update May 12, 1997 (1997) 15449-CR

(Update)

C6te d'lvoire Poverty In C6te d'lvoire: A Framework For Action June 14, 1997 (1997) 15640-IVCDjibouti Crossroads Of The Horn Of Africa: Poverty Assessment October 7, 1997 (1998) 16543-DJIGabon Poverty In A Rent-Based Economy June 27, 1997 (1997) 16333-GAGuinea A SocioeconomicAssessment Of Well-Being And Poverty March 31, 1997 (1997) 16465-GUIIndia Achievements And Challenges In Reducing Poverty (Update) May 27, 1997 (1997) 16483-INIndia Reducing Poverty In India: Options For More Effective Public June 29, 1998 (1998) 17881-IN

Services (Update)Kazakhstan Living Standards During The Transition March 23, 1998 (1998) 17520-KZRomania Poverty And Social Policy April 1997 (1997) 16462-RO

Thailand Growth, Poverty, And Income Distribution: An Economic December 31, 1996 (1997) 15689-THReport

Ukraine Poverty In Ukraine June 27, 1996 (1996) 15602-UAYemen Poverty Assessment June 26, 1996 (1996) 15158-YEMThe fiscal 1998 Rwanda Poverty Note and the fiscal 1998 Chile Poverty Assessment Update are not included, as theywere classified as poverty assessments after the poverty assessment sample for this review was finalized.

Stakeholder Survey Results

Was it a goal to establish a cooperative effort between the World Bank and the government of the country in trying toreduce poverty in the country?

Yes No Don't Know RefusedCountry client 77.8 18.5 2.5 1.2

WB staff 84.0 12.0 4.0 0.0

Was it a goal to allow the World Bank to help influence the government's poverty reduction policies and programs?Yes No Don't Know Refused

Country client 65.4 30.9 2.5 1.2WB staff 92.0 4.0 4.0 0.0

Was it a goal to help build in-country capacity to analyze and monitor poverty in the countryYes No Don't Know Refused

Country client 84.0 12.3 2.5 1.2WB staff 76.0 20.0 4.0 0.0

Was it a goal to help the World Bank refine the poverty focus of its own operationsYes No Don't Know Refused "0

Country client 76.5 17.3 6.2 0.0WB staff 92.0 4.0 4.0 0.0

Do you think the Poverty Assessment included all relevant population groups in the country, or do you think there weresome groups that either were not included in the Assessment or needed more attention

All included Some not included Don't know RefusedCountry client 45.7 42 2.5 0

WB staff 52 11 0 0

Do you think the Poverty Assessment included all relevant poverty -related issues for the country, or do you think therewere some issues that either were not included or should have been given more attention

All included Some not included Don't know RefusedCountry client 51.9 44.4 3.7 0

WB staff 56 44 0 0

Very realistic Realistic Unrealistic Very Unrealistic Neutral Don't Know Refused

Country client 13.6 76.5 4.9 1.2 1.2 2.5

WB staff 32.0 64.0 0.0 0.0 4.0

_-_ _ 0 _ Very realistic Realistic Unrealistic Very Unrealistic Neutral Don't Know Refused

Country client 13.6 71.6 9.9 0 3.7 1.2

WB staff 32.0 60.0 4.0 0 0.0 4.0

Very realistic Realistic Unrealistic Very Unrealistic Neutral Don't Know Refused

Country client 3.7 58.0 25.9 0 4.9 7.4 0.0

WB staff 36.0 32.0 20 0 8.0 4.0 4.0

- a * a ,Ba . . 0111 ISM 0 4 0-0Very realistic Realistic Unrealistic Very Unrealistic Neutral Don't Know Refused

Country client 3.7 54.3 19.8 2.5 2.5 11.1 6.2

WB staff 20.0 64.0 4.0 0.0 4.0 0.0 8.0

Very realistic Realistic Unrealistic Very Unrealistic Neutral Don't Know Refused

Country client 1.2 61.7 22.2 0 2.5 11.1 1.2

WB staff 16.0 76.0 4.0 0.0 0.0 0.0 4.0

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 8.6 80.2 8.6 0.0 2.5 0.0 0.0

WB staff 20.0 64.0 4.0 0.0 0.0 1.0 0.0

=_=== - -IE "". *. a . 0 00Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 8.6 59.3 27.2 1.2 0.0 3.7 0.0

WB staff 12.0 76.0 12.0 0.0 0.0 0.0 0.0*~~~~~~ ~~ ..0~a-aa -- s *~a a -

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused ;0

Country client 13.6 67.9 16.0 1.2 1.2 1.2 0.0

WB staff 28.0 60.0 8.0 0.0 4.0 0.0 0.0 X(QI

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 11.1 65.4 21.0 0.0 0.0 2.5 0.0

WB staff 24.0 64.0 4.0 4.0 0.0 0.0 4.0

To what extent has the PA helped to establish a cooperative effort between the WB and the government of the country in.i thecountry? - .- _ .

Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 27.2 51.9 11.1 3.7 4.9 1.2

WB staff 32.0 44.0 8.0 16.0 0.0 0.0. . _ _ _ . . . .. . _ .. ........ _ ..., ................. q q q ... ........ .. ... _ ..... ._ ... ...........

To what extent has the PA influenced the government's poverty reduction policies and programs?Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 23.5 49.4 17.3 4.9 3.7 1.2

WB staff 24.0 44.0 20.0 12.0 0.0 0.0

To what extent has the PA built or strengthened in-country capacity to analyze and monitor poverty in the country?Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 11.0 58.0 23.5 4.9 2.5 0.0

WB staff 32.0 32.0 16.0 20.0 0.0 0.0

To what extent has the PA helped the WB to refine the .,iie focus of its own operations?Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 32.1 39.5 12.3 1.2 14.8 0.0

WB staff 32.0 52.0 16.0 0.0 0.0 0.0_ ,,. . _,, , , _ ^ .. ... ..................................................................................................

Overall, to what extent have the other critical goals identified by you been achieved?Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 12.3 54.3 25.9 1.2 4.9 1.2

WB staff 24.0 48.0 12.0 4.0 8.0 4.0_ __ .. _ \M ....................................... ........... ,, ,,,,,__ ........... ... ............................

.

-To what extent do you think the PA has influenced the WB's Country Assistance strategy for the country?Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 28.4 44.4 17.3 2.5 6.2 1.2

WB staff 52.0 28.0 12.0 0.0 8.0 0.0

CD

* **27-MMIMPI Mg* MINIM *-. S - v in i

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 27.2 64.2 6.2 0.0 0.0 2.5 0.0

WB staff 52.0 36.0 0.0 0.0 0.0 12.0 0.0

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 21.0 63.0 11.1 1.2 1.2 2.5 0.0

WB staff 68.0 32.0 0.0 0.0 0.0 0.0 0.0

* -.R..-M.IRM-M. MM Sn .ym MITS ~ -

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 16.0 45.7 25.9 4.9 1.2 4.9 1.2

WB staff 40.0 48.0 12.0 0.0 0.0 0.0 0.0

* .- .M 111.. .R,L-,LS A 1-2 .iT.M ,, * , 5 _ MVery satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 17.3 61.7 14.8 1.2 1.2 1.2 2.5

WB staff 72,0 28.0 0.0 0.0 0.0 0.0 0.0

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 4.9 70.4 17.3 0.0 1.2 6.2 1.2

WB staff 20.0 56.0 12.0 0.0 0.0 4.0 8.0

11111_11 __-S - .Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 32.0 60.0 4.0 4.0 0.0 0.0 1.2

WB staff 12.3 68.9 15.1 1.9 0.0 0.9 0.0

S ....

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 3.7 63.0 24.7 2.5 0.0 3.7 2.5

WB staff 48.0 40.0 12.0 0.0 0.0 0.0 0.0

How satisfied are you with the PA team's expertise in technical matters?Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Countryclient 16.0 63.0 13.6 1.2 3.7 0.0 2.5

WB staff 52.0 44.0 0.0 4.0 0.0 0.0 0.0... .. .. .. __ . __ .. ...... ...... ...

How - l;1iaTri1j.with the PA team's ability toy interact and work with people?.... ... ..... ....... . ... ....... .Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 17.3 55.6 17.3 1.2 6.2 0.0 2.5

WB staff 60.0 36.0 0.0 4.0 0.0 0.0 0.0

How satisfied are you with the PA team's availability or accessibility?Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 16.0 56.8 21.0 1.2 2.5 0.0 2.5

WB staff 60.0 32.0 0.0 4.0 4.0 0.0 0.0

I . Fi. = . ' . _ .. 0 0 .. _ _

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know Refused

Country client 8.6 60.5 24.7 1.2 3.7 0.0 1.2

WB staff 32.0 52.0 4.0 0.0 8.0 0.0 4.0

W much did the PovertY Assessment Team encourage participation from relevant groups or agencies within central

r k . a...I..11 ... ... .... . _ .....

Very strong effort Moderate effort Only slight effort No effort Don't Know Refused

Country client 32.1 44.4 14.8 2.5 6.2 0

WB staff 60.0 28.0 8.0 0.0 0.0 4.0

How much did the Poverty Assessment Team encourage participation from relevant groups or agencies within localgovernment

Very strong effort Moderate effort Only slight effort No effort Don't know Refused

Country client 17.3 44.4 18.5 13.6 6.2 0

WB staff 8.0 40.0 40.0 4.0 4.0 4.0

How much did the Poverty Assessment Team encourage participation from relevant non-governmental organizationsVery strong effort Moderate effort Only slight effort No effort Don't know Refused

Country client 19.8 40.7 23.5 8.6 7.4 0

WB staff 36.0 48.0 16.0 0.0 0.0 0.0

* a * . .1 0f1 , I. i M.-I *1.9.SVery strong effort Moderate effort Only slight effort No effort Don't know Refused

Country client 18.5 46.9 19.8 11.1 3.7 0WB staff 44.0 40.0 12.0 4.0 0.0 0

* .NM IM -S. *I.. Central Very strong Moderate Slight No influence Don't know RefusedGovernment:

Country client 23.5 33.3 21.0 18.5 3.7 0WB staff 36.0 40.0 16.0 8.0 0.0 0

Local Very strong Moderate Slight No influence Don't know RefusedGovernment:

Country client 7.4 23.5 30.9 34.6 3.7 0.0WB staff 4.0 20.0 48.0 20.0 4.0 4.0

Non- Very strong Moderate Slight No influence Don't know Refusedgovernmentalorganizations

Country client 7.4 37.0 21.0 28.4 6.2 0.0WB staff 12.0 60.0 20.0 8.0 0.0 0.0

Academic or Very strong Moderate Slight No influence Don't know Refusedresearchorganizations

Country client 13.6 37.0 24.7 21.0 3.7 0WB staff 40.0 2800 20.0 12.0 2.8 0

Joint effort Joint, but more WB Joint, but WB Not a joint effort at Don't know Refusedinfluence dominated all

'Country client 28.4 28.4 34.6 2.5 6.2 0WB staff 12.0 24 .0 44.0020.0 0.0 0.0

Joint effort Joint, but more WB Joint, but WB Not a joint effort at Don't know Refusedinfluence dominated all

Country client 42.0 22.2 21.0 7.4 2.5 0.0WB staff 56.0 8.0 0.0 32.0 4.0 0.0

How would you describe the degree of cooperation between the government and the World Bank in conducting the PovertyAssessment's analysis of the data?

Joint effort Joint, but more WB Joint, but WB Not a joint effort at Don't know Refusedinfluence dominated all

Country client 32.1 19.8 37.0 9.9 1.2 0.0WB staff 24.0 24.0 28.0 24.0 0.0 0.0

How would you describe the degree of cooperation between the government and the World Bank in conducting the overallPoverty Assessment?

Joint effort Joint, but more WB Joint, but WB Not a joint effort at Don't know Refusedinfluence dominated all

Country client 39.5 27.2 29.6 2.5 1.2 0.0WB staff 16.0 32.0 36.0 16.0 0.0 0.0

Was there any adverse or favorable change in the commitment of the government to the Poverty Assessment, either duringor after the process, such as a change in personnel, or some other event that would have changed the relationship betweenthe government and the World Bank?

Yes, during PA Yes, after PA Yes, both during and No Don't know Refusedafter

Country client 6.2 19.8 8.6 56.8 7.4 1.2WB staff 20.0 24.0 16.0 28.0 8.0 4.0

How satisfied are you with efforts to disseminate the findings of the PA?Very satisfied Satisfied Dissatisfied Very Dissatisfied Don't Know Refused

Country client 7.4 63.0 24.7 3.7 1.2 0.0WB staff 32.0 32.0 24.0 8.0 4.0 0.0

u-.usuisuit~u are you with the follow-up to the PA?Very satisfied Satisfied Dissatisfied Very Dissatisfied Don't Know Refused

Country client 7.4 45.7 34.6 4.9 7.4 0.0WB staff 16.0 32.0 24.0 8.0 8.0 12.0

_____O__0i0 0 0Pi 0m m 0* 0 iMostly gov't Mostly WB Gov. w/WB help Both equally Neither Don't know Refused

Country client 22.0 4.9 69.1 0.0 1.2 2.5 0.0WB staff 20.0 8.0 72.0 0.0 0.0 0.0 0.0

3>

4 . T4* * 44 S 4 - 4 4 4- *. . -Mostly gov't Mostly WB Gov. wNVB help Both equally Neither Don't know Refused

Country client 44.4 23.5 16.0 4.9 6.2 4.9 0.0

WB staff 4.0 48.0 12.0 8.0 16.0 12.0 0.0

* . . . S. S * W . .5. 5..F 6 IMP-1.011 . - --Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 18.5 40.7 30.9 7.4 2.5 0.0WB staff 40.0 32.0 20.0 8.0 0.0 0.0

===_ - . - S. S I I .== I.M. - . *s. . S M3-TVery large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 12.3 50.6 28.4 6.2 2.5 0.0WB staff 36.0 36.0 20.0 8.0 0.0 0.0

* . *4f i r_..4 .4*- .Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 7.4 49.4 35.8 3.7 1.2 2.5WB staff 20.0 40.0 16.0 16.0 4.0 4.0

* . 64 0 4 . .4. 46 4 S 4 - *. . MET. IT9 -Very large extent Moderate extent Small extent Not at all Don't Know Refused

Country client 6.2 48.1 35.8 6.2 2.5 1.2WB staff 16.0 44.0 20.0 16.0 4.0 0.0

0 _ ..- _*.

Very satisfied Satisfied Dissatisfied Very Dissatisfied Neutral Don't Know RefusedCountry client 9.9 75.3 12.3 1.2 0.0 1.2 0.0

WB staff 32.0 56.0 4.0 4.0 4.0 0.0 0.0

CD

47 Annex D

Approach Paper

Background

1. In 1995-96, OED conducted a first review of World Bank Poverty Assessments."7 At thediscussion by the Committee on Development Effectiveness (CODE), a view was expressed that"impact assessments and a survey of borrowing countries' views on Poverty Assessments mightbe useful ."'" The present report responds to this interest. It aims at systematically analyzing theusefulness and impact of World Bank Poverty Assessments by: (a) polling the views of borrowersand other stakeholders about the in-country impact of Poverty Assessments, complemented by theviews of Poverty Assessment Task Managers and Resident Mission staff;,9 (b) checking thequality of the Poverty Assessments; (c) examining the processes supported by the Bank inpreparing the Poverty Assessments, as well as in following-up on them both within the Bank andin-country; and (d) determining the extent to which the recommendations of OED's previousPoverty Assessment review have been addressed by the Bank20 Findings relating to each of theseissues will be triangulated to arrive at an "independent" OED view of the usefulness and impactof World Bank Poverty Assessments in light of the guidelines contained in OD 4.15 on PovertyReduction.

Scope

2. The sample of Poverty Assessments for this review comprises a total of eighteen PovertyAssessments including all fifteen Poverty Assessments completed in fiscal 1997-98 as well asthree Poverty Assessments completed in fiscal 1996. This timeframe has been chosen based onthree assumptions. First, information may be distorted if stakeholder memories are pushed backbeyond three years, or if staff-turnover has occurred in client countries. Second, the current state-

'7 / "Poverty Assessment: A Progress Review," Report Number 15881, World Bank, 1996.

s / CODE 96-64, World Bank, 1996.

'9 / The Participation Sourcebook (World Bank, 1996) defines stakeholders as "those who are affected by the outcomeor those who can affect the outcome of a proposed intervention." Based on this definition, the term "stakeholders" hereincludes representatives of cross-sectoral ministries (e.g., Finance, Planning), sectoral ministries (e.g., Education,Industry), local governments, NGOs, and technical experts, as identified by the Task Managers and Resident Missionstaff. The poor themselves are not included among the "stakeholders" except in the case-study countries which alsohave a Participatory Poverty Assessment.

20 / The recommendations of the 1995-96 Poverty Assessment review as summarized in the "Active LedgerRecommendations as of end-FY97" (CODE97-62, October 9, 1997) are: (i) The Bank should commit to a new andrealistic timetable for the completion of poverty assessments, and establish a policy and a timetable for updatingcompleted poverty assessments as part of a reinvigorated poverty reduction strategy, involving enhanced ownership ofthe poverty reduction agenda by management and staff. (ii) Bank managers and staff in the regions and PSP mustintensify their efforts to raise the quality of information about the extent, incidence, and causes of poverty, and theoperational content of the relevant actions recommended to reduce poverty in Poverty Assessments and CountryAssistance Strategies (CASs) by more consistently applying the guidance contained in the existing OperationalDirective on Poverty Reduction (particularly paragraphs 16-20, and 47-49 of OD 4.15) and the Bank Procedures onCountry Assistance Strategies (particularly paragraphs 3 (a), 8, and 13 of BP 2.11). (iii) PSP should strengthen itsannual report on poverty reduction by systematically monitoring and evaluating the content of poverty assessmentspaying particular attention to: (a) improvements in the quality of poverty profiles and analysis of the causes of poverty;(b) the quality of country poverty reduction strategies; and (c) the recommended actions for poverty reduction,including the priorities accorded to different relevant programs and policies. (iv) Where reliable information on povertyis already available, it should be fully exploited by country departments to raise the quality of poverty assessments andcountry assistance strategies. Where data are deficient, country departments with support from HCD and DEC, shouldconsider ways to improve their availability. (v) CASs should indicate how the findings and recommendations ofpoverty assessments have been incorporated as called for in paragraph 20 of OD 4.15 and paragraphs 3(a), 8, and 13 ofBP 2.11.

48 Annex D

of-the-art may not be reflected in the older cohort of Poverty Assessments. Third, the limitedshelf-life of Poverty Assessments may mean that either they are influential in the immediate-to-medium term, or are largely forgotten. The three Poverty Assessments completed in fiscal 1996have specifically been included to account for the possibility that particular countrycircumstances might prevent an immediate adoption of the recommendations of the PovertyAssessment and result in a more lagged response. All six Bank regions will be represented amongthe eighteen Poverty Assessments that comprise the sample for this study.

3. This report focuses exclusively on Poverty Assessments -- which are only one element ofthe World Bank's poverty reduction effort. An assessment of the impact and effectiveness of theBank's overall poverty reduction effort will not be attempted here. This is the topic of OED'sforthcoming "Poverty Study." The present report will comprise one input into establishing theoverall picture.

4. This report draws upon the ESW evaluation framework developed by OED2 as well asthe QAG framework for reviewing ESW.22 Modifications have been made to reflect the specificsof the Poverty Assessment instrument. The evaluation criteria, as defined for purposes of thisreport, are presented below.

(i) Relevance refers to the extent to which the Poverty Assessment addresses key povertyissues for the country and provides an appropriate diagnosis and prescription for povertyreduction. The main questions to gauge relevance will include the extent to which, for example,the groups identified as poor in the Poverty Assessment are the right ones, the causes of povertyidentified in the Poverty Assessment are accurate, and the implementation of the PovertyAssessment recommendations is likely to help reduce poverty in the country.

(ii) Efficacy refers to the extent to which the Poverty Assessment achieves its goals, in termsof identifying the policies, public expenditures, and institutional issues that constrain effectivepoverty reduction, as well as in terms of 'forming the basis for a collaborative approach topoverty reduction by country officials and the Bank, and helping to establish the agenda of issuesfor the policy dialogue. " 23 The likely influence of Poverty Assessments on budgetary processesand public spending, as well as on other in-country poverty reduction policies, will also beassessed since these are the higher order impacts that may be expected of Poverty Assessments"4

Accordingly, the key questions to determine efficacy will include the extent to which, forexample, the Poverty Assessment provides the basis for a collaborative approach to povertyreduction between the govemment and the Bank, the Poverty Assessment helps to guide thepolicy dialogue, the Poverty Assessment is likely to influence budgetary processes and the leveland composition of public spending on pro-poor expenditure categories, and the PovertyAssessment is likely to have an impact on in-country policy formulation.

(iii) Efficiency refers to the cost and time taken to prepare the Poverty Assessment in relationto its impact. Since Poverty Assessment impacts are difficult to quantify, benchmarks such as thecost of the Poverty Assessment in relation to other similar ESW, its cost as a percentage of total

21 / See "Evaluation of Economic and Sector Work -- A Review," CODE 97-55. World Bank, September 15, 1997.

22 / See "Improving the ESW Quality: An Approach Paper," Memo from Prem Garg to Operational Managers andAdvisors, World Bank, August 11, 1997.

23 / This is the role identified for Poverty Assessments in the Poverty Reduction Operational Directive 4.15, WorldBank, 1992.

24 / A point emphasized with respect to ESW in "Evaluation of Economic and Sector Work -- A Review," Op. cit.

49 Annex D

lending to the country, or its cost compared with the cost of other donor Poverty Assessments willbe established (controlling for quality in each case).

(iv) Timeliness refers to the extent to which, for example, the Poverty Assessment isstrategically timed in terms of the climate for policy reform in the country and the availability ofpoverty data.

(v) Internal Quality refers to the extent to which, for example, the Poverty Assessmentincorporates a sound analysis of poverty issues in the country, the Poverty Assessment draws onrelevant experiences and state-of-the-art analysis from elsewhere in the Bank or internationally,the conclusions and recommendations are substantiated by empirical evidence, and the PovertyAssessment recommendations take account of the political, cultural, social and institutionalrealities of the country.

(vi) Participation and Ownership refer to the extent of borrower involvement andcollaboration in the Poverty Assessment preparation process as well as the commitment,ownership, and assumption of responsibility to implement its recommendations. Key questions toassess participation and ownership will include the extent to which, for example, the differentstakeholders contribute to the preparation of the Poverty Assessment, the government "owns" therecommendations of the Poverty Assessment, and the government takes responsibility forensuring Poverty Assessment follow-up and implementation of recommendations.

(vii) Institutional Development refers to the extent to which the Poverty Assessment improvesan agency's or country's capacity to assess poverty and/or take appropriate actions to address it.25

Key questions to assess institutional development will include the extent to which, for example,the Poverty Assessment builds local capacity to measure poverty, analyze poverty, formulatepoverty reduction policies, implement the recommended poverty reduction policies, as well as theextent to which the Poverty Assessment provides guidance on how to build local capacity toimplement the recommendations.

Methodology

5. Definition of "Process. " This report will adopt a broad definition of "process." In otherwords, the "process" assessed here will refer not only to the preparation of the PovertyAssessment documentper se, but rather to all the activities around the Poverty Assessment (e.g.,workshops, seminars) that lead-up to the analysis and recommendations contained in thedocument. Details of the "process" followed in preparing each Poverty Assessment will bedocumented based on file research and Task Manager interviews, and stakeholders will beinformed, as warranted, prior to interviewing.

6. Definition of "Stakeholders. " The Participation Sourcebook define$ stakeholders as"those who are affected by the outcome or those who can affect the outcome of a proposedintervention."2 6 Based on this definition, the term "stakeholders" here includes representatives ofcross-sectoral ministries (e.g., Finance, Planning), sectoral ministries (e.g., Education, Industry),local governments, NGOs, and technical experts, as identified by the Task Managers and

25 / While OD 4.15 does not require the Poverty Assessment to have an institutional development goal, capacitybuilding is one of the goals identified for ESW in "Evaluation of Economic and Sector Work -- A Review" Op. cit. Inany case, this review will look at institutional development as a "good practice," rather than a "deficiency" if it islacking.

26 / World Bank, 1996.

50 Annex D

Resident Mission staff. The analyses of views will be conducted separately for "borrowers" and"other stakeholders." The poor themselves will be classified among the "stakeholders" and willbe directly interviewed in the case-study countries which have a Participatory PovertyAssessment.

7. Sample of Stakeholders. For the eighteen Poverty Assessments to be covered in thisreview, the Poverty Assessment Task Managers will be asked to provide a list of all borrowersand other stakeholders, in government and non-government, who were involved in thepreparation of the Poverty Assessment and/or are currently engaged in implementing itsrecommendations. They will also be asked to include relevant names of those not consulted dueto time or resource constraints; but who, in their view, should have been involved. In addition,Resident Mission staff will be asked to provide a list of names of relevant stakeholders whoshould have been involved in the preparation of, or follow-up to, the Poverty Assessment. Theoverlap between the Resident Mission staff list and the Task Manager list will be eliminated and asample of stakeholders will be drawn for interviewing.

8. Tools. The key tools for this review will comprise: (a) detailed case studies in selectedcountries involving field visits and semi-structured interviewing of stakeholders, Task Managers,and Resident Mission staff for an in-depth understanding of their different perspectives; (b) anextensive Gallup telephonic survey of stakeholders, Task Managers, and Resident Mission staff;(c) analysis of the Poverty Assessment reports and related poverty literature aimed at establishingthe quality of the Poverty Assessments; (d) file research and examination of institutionaldatabases to assess the soundness of the processes supported both in preparing the PovertyAssessments and in their follow-up, supplemented by Task Manager interviews as necessary; and(e) discussions with Bank Anchor and Regional Departments aimed at assessing the adequacyand appropriateness of the actions taken in response to the recommendations of OED's previousPoverty Assessment review.

9. In-depth Case-Studies. The in-country usefulness and impact of Poverty Assessmentswill be probed through field visits in six countries. The selection of the six countries has beendone purposively to achieve regional balance: Bangladesh, Bolivia, Cote d'Ivoire, Thailand,Ukraine, and Yemen -- of which Bangladesh, Bolivia, and Ukraine will be common with OED'sPoverty Study. The field visits will include semi-structured, open interviewing of stakeholders, aswell as focus group discussions involving a mix of stakeholders. The stakeholders will beidentified by Task Managers and Resident Mission staff, but will also include individuals chosenthrough "snowballing" as a result of the semi-structured interviews. Task Managers and ResidentMission staff themselves will also be interviewed in a semi-structured, open interview format,using GroupWare as appropriate.

10. Gallup Telephonic Survey. Following pilot testing, the Gallup organization willadminister a questionnaire on 100 randomly chosen stakeholders in the eighteen countries out ofthe full list of stakeholders which is assumed to total approximately 200 stakeholdersF7 Thesample will be stratified by institutional level of borrower (e.g., central/local government), andtype of stakeholder (i.e., govemment, NGOs, technical experts). Gallup will also administer thequestionnaire on the eighteen Poverty Assessment Task Managers, as well as on eighteenResident Mission staff involved in either the preparation or the follow-up to the PovertyAssessment.

11. In order to facilitate analysis of the survey results, a profile of each individual selected forinterviewing (respondent) will be constructed up front using a special questionnaire. The depth of

2 7/ Gallup has applied this technique successfully in OED's "Appraisal Process Review," 1997.

51 Annex D

the subsequent questioning will be guided by the respondent's profile. Sample questions in therespondent profile questionnaire will relate to, for example, the respondent's organizationalaffiliation, the respondent's overall familiarity with and knowledge of the Poverty Assessment,and the extent of the respondent's involvement in preparing the Poverty Assessment or in itsfollow-up.

12. Analysis of Poverty Assessment Reports and Related Literature. A panel of threeinternational experts -- one macroeconomist, one microeconomist, and one social scientist -- willexamine each of the eighteen Poverty Assessments for their quality. They will also draw uponany relevant country or sector literature from Bank and non-Bank sources. Task Managers will beinterviewed, as necessary.

13. Review of Institutional Files and Databases. The files for each of the eighteen PovertyAssessment reports will be examined in order to document the processes followed in preparingthe Poverty Assessments and in undertaking dissemination and follow-up once the report isfinalized. The institutional databases will be mined for administrative information, such asfrequency of missions, composition of Poverty Assessment teams, consultation process,dissemination strategy, and staff time and cost in preparing them.

14. Discussions with Bank Departments. Discussions with the PREM Anchor will be held toidentify the initiatives launched in response to the recommendations of the previous OED report.In addition, discussions with selected Country Directors and other relevant staff will beundertaken to determine how the initiatives of the Anchor affect them and the extent to whichthey have resulted in improvements in the way in which Poverty Assessments actually get doneand are followed-up.

15. Analysis of Survey Results. Findings on each of the above issues will be triangulated inorder to arrive at an "independent" OED view of the usefulness and impact of World BankPoverty Assessments. The guidelines contained in OD 4.15 will be used as the point of reference.Any divergences between the stakeholder, Task Manager, Resident Mission staff, and"independent" views will be analyzed and key lessons for enhancing the usefulness and impact offuture Poverty Assessments will be distilled.

Dissemination

16. Since the findings of this report will be of interest to a vast cross-section of policymakersand practitioners, the study will be disseminated widely upon completion. This will enabledebriefing of stakeholders about the study findings as well as the provision of lessons to othercountries. Within the Bank, the relevant Country Desks, Network Anchors, and Thematic Groupswill be continually apprised of the progress of the study and will be provided feedback uponstudy completion.

Ledger of OED Recommendations

OED Recommendations Managenment Response

Improve poverty assessment quality and process. The country Management agrees with the general thrust of this recommendation tomanagement and regional Chief Economist's office should increase improve the quality of poverty assessments. Following the discussion with themanagement oversight of the analytical quality of poverty assessments as Board on the Poverty Progress Report, management is taking the followingwell as their partnership and consultation and knowledge transfer aspects. The actions:Bank should undertake self-evaluations of poverty assessments. The Poverty (a) each of the regions has set into place a mechanism for ensuring qualityReduction Board and Thematic Groups should support the Regions with poverty assessments and timely completion;training and advice, and through providing tools and best practice examples (b) the Poverty Reduction Board and thematic groups are supporting povertyon poverty analysis, engaging partners, supporting collaboration and local assessments in the regions through facilitating consultants, peerparticipation, and developing local capacity through knowledge transfer. The reviewers, good practice examples, training and clinics.Poverty Reduction Board and Thematic Groups should ensure that theRegions have adequate and appropriate guidance, including on best practices, As with other ESW/AAA, quality control rests with Country and Sector

for he repaatin ofpovrty ssesmens. he Pvery Reucton Bard Directors (in the case of Poverty Assessments, usually the PREM Director).for the preparation of poverty assessments. The Poverty Reduction Board Che Ecnmit wil reie an giv gudnet' eetdsuisshould report the findings of the Bank's self-evaluations in the Annual Chief Economists will review and give guidance to selected studies,Progress Report on Poverty. Management will continue to monitor the quality of poverty assessments

under the work of QAG, and the Poverty Reduction Board will report on theresults of such evaluations in the Annual Progress Report on Poverty.

A more extensive program of action is under preparation, to be discussed withthe Board in the fall. Although the content of this action program is not yetfully defined, management expects it to include:(a) New guidelines on the content of poverty updates, including measures to

ensure that concept papers for poverty assessments include a strategy(and budgeting) for dissemination, local capacity building andpartnership development.

(b) The development of poverty notes to accompany country strategies thatcontain core information on a country's poverty profile, povertydynamics and the links between these and country-wide developments.Beyond this, selected deeper analysis would be carried out on the salientpoverty issues, country strategy issues and activities of other >development partners. This may take the form of a poverty assessment,focused economic and sector work, or support to the work of localresearch partners.

OED Recommendations Management Response

(c) Ensuring, as a country strategy issue, that all countries have an adequatedatabase (both quantitative and qualitative) to make good and deeppoverty analyses possible, or a strategy to develop one, where this islacking.

(d) Action, where necessary, to improve staffing within the regions,supported by the development of a pipeline of candidates to be preparedby the Poverty Reduction Board.

Focus on goal and priority-setting. Country management should ensure that Management agrees that a clear focus on goals and priorities is essential for

poverty assessments -- from the concept paper on - define clear goals and quality poverty analysis, and that Country Directors should assume a stronger

priorities, provide adequate justification for prioritization of issues and for role in this respect. As with other ESW/AAA, quality control rests with

any selectivity in scope and coverage, adequately reflect country conditions, Country and Sector Directors; Chief Economists will review and give

and link work programs and analysis to goals. The regional Chief guidance to selected studies and the PREM Network will work with the Chief

Economist's office should monitor and provide quality control for this. The Economists to ensure the quality of poverty assessments. As mentioned

Poverty Reduction Board and Thematic Groups should support the Regions above, the Poverty Reduction Board and thematic groups are supporting

with advice and best practice examples. poverty assessments in the regions through facilitating consultants, peerreviewers, good practice examples, training and clinics.

Meet targets for poverty assessment completion. The Regions, in For first-round poverty assessments, management has recommitted to their -n

collaboration with the Poverty Reduction Board, should set realistic targets timely completion and has put into place a tracking system for poverty

for completing poverty assessments. To underscore the importance of the assessments and poverty updates which the Poverty Board monitors. It should

poverty reduction agenda, senior management should provide the institutional be noted that the timing of many of the remaining mandatory assessments

inputs and incentives to produce quality poverty assessments effectively and will be a function of country conditions-especially for societies in conflict.

on time.For poverty updates, the program of action to be presented to the Board in thefall will outlinie the response to this recommendation, along the linesdiscussed above for recommendation number 1.

55 Annex F

Management ResponsePoverty Assessments: A Follow-Up Review

I. Introduction

1. This report is a follow-up on OED's 1996 study on poverty assessments, and it reviewsthe Bank's record in fulfilling its commitment to completing poverty assessments for allborrowers, and to improving their quality and impact, as well as local participation in theirpreparation. The current study includes a survey of client country views on poverty assessment,in response to a request made by CODE during the discussion of the 1996 report. This follow-upstudy examines 19 poverty assessments and updates (a sample of FY96 PAs, plus all thosecompleted in FY97 and FY98).

2. Management is in the process of preparing a program of action on how to increase theimpact of Bank activities on poverty, following up on the Board discussion of the FY98 ProgressReport on Poverty Reduction (April 5, 1999). This will be discussed with the Board after thesummer recess. The current OED study provides useful input into the development of suchprogram of action.

TI. Summary of OED Findings and Recommendations

3. Findings of the Report. OED finds that there has been modest improvement in theproportion of poverty assessments rated as satisfactory, from 54 percent of those done through1994, to 63 percent of those done in fiscal 1996 through 1998. Therefore, substantialimprovements can be made in the quality of poverty assessments, particularly in their policyanalysis and in their focus on specific goals. Improving quality is paramount since the overallimpact of poverty assessments is strongly associated with the quality of the poverty assessments.Further, impact is also related to the degree of partnership and consultation at the country level incarrying out the poverty assessments, infonnation sharing and knowledge transfer, the timelinessof the results, and the adequacy of time available to prepare the assessments.

4. OED conducted a client country survey as an input into drafting the report. The surveyindicated that a large majority of the clients surveyed were satisfied with the overall results ofpoverty assessments, but that they viewed local partnership and consultation, and knowledgetransfer, as the least satisfactory aspects of poverty assessments. The survey showed that in mostcases a higher proportion of Bank staff than country clients were positive about povertyassessments.

5. OED also reviewed the activities taken and changes made in response to the 1996 OEDrecommendations. Poverty assessments still suffer significant slippage. A concerted effort toaddress some of the recommendations of the 1996 OED report has begun in the past year, so it istoo early to assess whether these efforts are having their intended effect.

6. Recommendations. OED makes three recommendations in its report, regarding: (a)improving the quality as well as process aspects of poverty assessments; (b) focusing on goal andpriority-setting; and (c) meeting targets for poverty assessment completion (see attached Ledger).

56 Annex F

III. Management's Views

7. This report contains much useful information, especially on the feedback and impact ofpoverty assessments, which was obtained through client surveys. As such, it provides usefulinput into the development of the program of action on how to increase the impact of Bankactivities on poverty, which management is preparing following the CODE and Board discussionsof the FY98 Progress Report on Poverty Reduction.

8. With respect to the recommendations outlined above, management agrees with the mainthrust of enhancing quality and ensuring timeliness. Management would like to emphasize,however, that it has already taken a number of steps in this direction since the discussion of thePoverty Progress Report with the Board. Specifically, these include:

(a) each of the regions has set into place a mechanism for ensuring quality povertyassessments and timely completion;

(b) the Poverty Reduction Board and thematic groups are supporting poverty assessments inthe regions through facilitating consultants, peer reviewers, good practice examples,training and clinics.

9. Additionally, a more extensive program of action is under preparation. Although thecontent of this action program is not yet fully defined, management expects it to include:

(a) New guidelines on the content of poverty updates, including measures to ensure thatconcept papers for poverty assessments include a strategy (and budgeting) fordissemination, local capacity building and partnership development.

(b) The development of poverty notes to accompany country strategies that contain coreinformation on a country's poverty profile, poverty dynamics and the links between theseand country-wide developments.

(c) Action, where necessary, to improve staffing within the regions, supported by thedevelopment of a pipeline of candidates to be prepared by the Poverty Reduction Board.

10. As to the specific recommendations for quality enhancements included in the Ledger, it isworth noting that quality control for all ESW/AAA rests with Country and Sector Directors (inthe case of Poverty Assessments, usually the PREM Director). Chief Economists generallyreview and give guidance to selected studies only. The PREM Network will work closely withthe Chief Economists to support the quality of Poverty Assessments. In addition, Managementwill continue to monitor the quality of PAs under the work of QAG, and the Poverty ReductionBoard will report on the results of such evaluations in the Annual Progress Report on Poverty.

57 Annex F

IV. Summary and Conclusions

11. In summary, the report provides useful insights on the process and quality of povertyassessments, and its recommendations will feed into the program of action being prepared bymanagement. Management looks forward to working with OED in the coming months as theprogram of action is being prepared, so as to benefit from the insights of the upcoming OEDstudy on the Bank's poverty reduction strategy.

59 Annex G

August 11, 1999 CODE99-69

consultation - as the least satisfactory aspect ofReport from poverty assessment. The report also foundsignificant slippage in the completion of PAswith only half of PAs scheduled between 1997-

CODE 99 having been completed.

Committee on Development The Committee welcomed the report andmanagement's draft response and indicated that

]Effectiveness it saw PAs as an important instrument to be usedin sharpening the poverty focus of countryassistance strategies. The Committee broadly

Poverty Assessments: A Follow-up Review endorsed the report and the recommendations forimproving the quality, as well as process aspects,of PAs, focusing on goal and priority setting,

CODE met on June 23, 1999 to discuss the GED meeting targets for PA completion andreport Poverty Assessments: A Follow-up encouraging more local participation and transferReview (CODE99-5 0), together with the draft of knowledge.Management Response (CODE99-5 1). Thereport is a follow-up to the GED 1996 progress The Committee raised the following issues:report on poverty assessments (PAs). The reportevaluates how well the Bank has done in meeting Poverty Assessments Suffer from Genericits commitment to undertake PAs for all Problems of Economic Sector Work. Theborrower countries. At CODE's request, the Committee noted that PAs, like other economicfollow-up report also examines the impact of and sector work, often suffer from lack of clearPAs at the country level. Members had also objectives and specificity of goals for thespecifically asked for the inclusion of a survey analytic work, weak links betweenon the views of borrowers and key stakeholders recommendations and desired outcomes and tooregarding the usefulness and effectiveness of little interaction with clients beyond centralPAs. ministries (e.g. local governments, operational

ministries, beneficiaries). The Committee thusThe reports s main findings included: welcomed the ESW reform program that ismodest improvement in the proportion of PAs getting underway. The Committee was alsorated satisfactory for their economic quality; and concerned with the variability of the quaity ofhigh stakeholder satisfaction with PAs overall PAs, as well as the significant slippages inbut significant room for improvements in the meeting targets and wondered what follow-upareas of policy analysis and focus on goals. The actions could be taken to improve those PAs thatimpact of PAS overall was strongly correlated had been not been rated satisfactory. Thewith the quality of the poverty assessment, extent Committee also requested that the indicators ofof local participation and involvement in the success and best practice be disseminated aspoverty assessment, timeliness of the results, as widely as possible.well as the transfer of knowledge anddissemination of the findings. Stakeholders Implementation of Poverty Assessments is Slow.viewed two of these aspects - knowledge There was general agreement thattransfer and local partnership and

60 Annex G

August 11, 1999 CODE99-69

improvements in PAs since the last review have also concerned that about half of the PAs werebeen only modest and some members felt more not successful in linking qualitative andneeded to be done to identify the institutional quantitative methods and wondered if PREMand organizational culture constraints to had the kind of skills required to address theexecuting good PAs. qualitative aspects of poverty. It was c

onsidered essential to associate oth erPoverty Assessments Suffer from Poor networks, and especially the expertise ofOversight Members noted that the matrix qualitative social scientists, to the design andstructure of the Bank does not identify clear lines review of PAs.of accountability/answerability between NetworkBoards and Regional management for delivering Poverty Assessments and the Macro Economicquality and timely PAs and they emphasized the Framework. Committee members stressed theimportance of providing appropriate need stronger linkages between the PAs and theorganizational incentives in this regard. The macro economic policy framework, includingCommittee said that the Poverty Reduction the role that PAs play in developing the CAS.Board had an imiportant role to play in One member noted the increasing involvementmonitoring poverty assessment completions, of the IMF in undertaking its own PAs andboth for quality and to ensure the meeting of wondered if the BankflMF were applying atargets. common approach.

Methlods and Sampling. The Committee noted Poverty Assessments Need to be Betterthe importance of including the views of key ResourceS The Committee noted that the drop instakeholders (the poor, local governments and the number of PAs in recent years reflected theoperational ministries) in the assessment process completion of the first-round of assessments inand was concerned about the differing views of most countries. Members noted there is a needthe Bank versus its clients on the usefulness of for medium-term budget support for povertythe PAs. They feared that poverty assessments analysis if it is to form the basis for developingmay still be too donor driven. Members were CASs.

Jan Piercy, Chairperson

Distribution:

Executive Directors and AlternatesPresidentBank Group Senior ManagementVice Presidents, Bank, IFC and MIGADirectors and Department Heads, Bank, IFC, and MIGA