QUALITY AT ENTRY IN FY03...

QQUUAALLIITTYY AATT EENNTTRRYY IINN FFYY0033 ((QQEEAA66))

AANNNNEEXXEESS TTOO TTHHEE QQAAGG AASSSSEESSSSMMEENNTT

January 14, 2004

46142v 2

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Acknowledgement A team managed by P. Hari Prasad, which included Arun Banerjee and Robert Drysdale, prepared this report. Contributions were also provided by specialized review teams led by Chona Cruz (Poverty and Social Development), Jean Roger Mercier (Environment), Mimi Ladipo (Financial Management), Velayutham Vijayaverl (Procurement). Data processing and analysis was done by Melvin Vaz. Assessment logistics were managed by Brónagh Murphy. Documentation was managed by Salli Cudjoe. Leila Cruz and Susan Crisostomo also provided administrative and logistical support. QAG wishes to thank the panelists and task teams who participated in the QEA6 exercise, as well as the quality assurance staff in the regions for their support and valuable contributions. Mr. Prem Garg, Director, Quality Assurance Group, guided the overall effort.

ACRONYMS AND ABBREVIATIONS

ACS Administrative and Client Service MDG Millennium Development Goals AFR Africa Region M&E Monitoring and Evaluation APL Adaptable Program Loan MNA Middle East and North Africa Region CAS Country Assistance Strategy MP Montreal Protocol CDD Community-Driven Development OPCS Operations Policy and Country Services CMU Country Management Unit OVP Operational Vice President CPAR Country Procurement Assessment Review PAD Project Appraisal Document CPIA Country Policy & Institutional Assessment PCD Project Concept Document CPAR Country Procurement Performance Review PHRD Policy & Human Resource Development DO Development Objectives PIP Project Implementation Plan EA Environmental Assessment PREM Poverty Reduction & Economic Management Network EAP East Asia and Pacific Region PRSP Poverty Reduction Strategy Paper ECA Europe and Central Asia Region PSAL Programmatic Structural Adjustment Loan EMP Environmental Management Plan PSD Private Sector Development ERL Emergency Recovery Loan PSI Private Sector Development & Infrastructure Network ESSD Environmentally and Socially Sustainable

Development Network PSR Project Status Report

FIL Financial Intermediary Loan QAG Quality Assurance Group FM Financial Mangement QEA Quality-at-Entry Assessment FMS Financial Management Specialist QER Quality Enhancement Review FSE Financial Sector Network QSA Quality of Supervision Assessment GEF Global Environment Facility SAL Structural Adjustment Loan HDN Human Development Network SAP Systems, Applications, and Products in Data

Processing IAD Internal Auditing Department SAR South Asia Region IP Implementation Progress SMU Sector Management Unit IRIS Integrated Records and Information System SIL Specific Investment Loan LCR Latin America and the Caribbean Region TAL Technical Assistance Loan LIL Learning and Innovation Loan TF Trust Funds TTL Task Team Leader

TABLE OF CONTENTS ANNEXES 1. List of QEA6 Operations ....................................................................................................1

2. Approach Paper ..................................................................................................................2

3. Main Contributors ..............................................................................................................7

4. Statistical Tables ................................................................................................................9

5. QEA6 Guidance Questionnaire – Quality Assessment Results (QEA1 - 6)........................ 21

6. Poverty and Social Aspects............................................................................................... 31

7. Environmental Aspects..................................................................................................... 35

8. Financial Management Aspects ........................................................................................ 41

9. Procurement Aspects ........................................................................................................ 46

10. Summary Results of Task Team Leaders, Panelists and Observers’ Survey ...................... 50

11. Methodology for Sampling, Stratification and Weighting ................................................. 54

1

ANNEX 1

LIST OF QEA6 OPERATIONS

Africa BEAC Regional Payment System Albania Second Community WorksAngola Demobilization and Reintegration Bosnia-Herzegovina Forest Devlpt/Cnsrv TAAngola Economic Management TA Croatia Pension Systems InvBurkina Faso Competitiveness & Enterprise Development Georgia Municipal Development 2Burkina Faso PRSC 2 Georgia Primary Health Care DevtBurundi ERC Kazakhstan Nura River CleanupChad Critical Elec. & Water Services Kosovo Energy Sector TAL IIChad Fifth Structural Adjustment Credit Kyrgyz Republic Govt TACongo, Democratic Republic of Emergency Multisector Rehab.& Recon. Romania Electric MarketCongo, Republic of Emergency Recovery/Community Project Romania PSAL 2Ethiopia Energy Access Project Russian Federation Customs DevelopmentEthiopia Pastoral Community Dev Russian Federation TB/AIDS ControlEthiopia Road Sector Development Program II Serbia and Montenegro PFSAC 2Ghana Health Sector Program Support Project II Slovak Republic Public Fin MgmtKenya Arid Lands II Turkey Basic Ed 2 (APL #2)Malawi Emergency Drought Recovery Credit Ukraine Rural Land Titling & CadastreMozambique HIV/AIDS Response Project Ukraine TB/AIDS ControlNigeria Lagos Urban Transport ProjectRwanda Institutional Reform Credit Argentina Economic and Social TransitionSierra Leone Basic Educ Rehab Argentina Jefes de Hogar ProgramSierra Leone ERRC III Bolivia Decentr. PSAC IISierra Leone National Social Action Project Brazil HD Prgm. Sector Reform LoanTanzania Dar Water Supply & Sanitation Brazil Program Fiscal Reform IIUganda Northern Uganda Social Action Fund Colombia Program. Fiscal and Institutional AdjUganda PRSC II Colombia Social Sector AdjustmentUganda Second Local Government Development Dominican Republic Health Reform Support (APL)

El Salvador Judicial Modernization ProjectCambodia Health Sector Support Project Guyana Public Sector Tech. Assistance CreditCambodia Rural Investment and Local Governance Jamaica JM- Reform of Secondary Ed. (ROSE II)China 3rd Xinjiang Hwy Project Nicaragua PSACChina Shanghai Urban Envirn (APL1) Peru National Rural Water Supply & SanitationEast Asia and Pacific Second Timor-Leste Petroleum Project Peru Rural Education and Teacher DevelopmIndonesia Water Resources & Irr.Sector Mgt Program Peru Trade Facil. and Prod. Improv. T. A.Lao People's Democratic Republic Sustainable Forestry for Rural Development Uruguay SSALPhilippines ARMM Social FundPhilippines KALAHI-CIDSS Project Algeria Urban Natural Hazard VulnerabilityPhilippines 2nd Agrarian Reform Communities Devel Djibouti HIV/AIDS, Malaria and TB ControlVietnam Vietnam PRSC II Iran, Islamic Republic of Earthquake Emergency Response

Morocco Adult Literacy (Alpha Maroc)West Bank and Gaza Emergency Municipal Services RehabYemen, Republic of Port Cities Development Program

Europe and Central Asia (ECA)

Middle East and North America (MNA)

Latin America and the Caribbean (LCR)

East Asia and Pacific (EAP)

Africa (AFR)

Afghanistan Emergency Transport Rehab.Afghanistan Health Sector Emergency RehaBangladesh Central Bank Strengthening ProjectBangladesh Development Support CreditIndia Chatt DRPPIndia Food & Drugs Capacity Building ProjectNepal Financial Sector Technical AssistancePakistan Sindh Structural Adjustment CreditSri Lanka Second Community Water

South Asia (SAR)

2

ANNEX 2

SIXTH QUALITY-AT-ENTRY ASSESSMENT (QEA6) APPROACH PAPER

Introduction and Objectives 1. The Quality Assurance Group (QAG) announces commencement of the Quality-at-Entry Assessment (QEA6) for projects approved by the Board in FY03. No major procedural changes are planned from QEA5 in this year’s assessment. TTLs and line managers for the operations will be notified in mid-April and mid-June, as the two parts of the QEA6 sample are drawn. Stage I interviews will be conducted beginning in the last week of May, with the Stage II reviews ending in October, 2003. 2. Based on the results for QEA5, which showed a departure from the previous trend of continuously improving quality at entry, senior management has decided that QEA6 should comprise a sample large enough to permit analysis of Bank-wide, Regional, and Networks’ results at approximately the 90% confidence level , +/- 10%. As a result, the sample will cover 80 operations to permit disaggregated analysis at the Regional and Network levels. 3. Overall objectives of QEA-6 are similar to those of the previous five assessments. These objectives are as follows:

• Maintaining accountability for quality by providing information to management and staff, including indicators of quality of operations at-entry at the Bank-wide, Regional, and Network levels and compliance with Bank safeguard and fiduciary policies.

• Improving targeted learning in selected areas of operational quality and

disseminating assessment findings to appropriate units in the Bank.

Targeting the QEA Process 4. Sampling. Aggregate results from QEA3-5 show that operations funded by IDA are rated ten percentage points lower than those funded by IBRD. In order to arrive at a better understanding of factors contributing to these results, the IDA cohort will be over- sampled by selecting approximately one out of every three operations, while limiting the IBRD sample to one out of approximately every four operations. A further stratification of the sample will be made based on loan/credit size. This would entail including in the sample half of all Loans above $150m and credit above $50m and a lower proportion of smaller loans/credits. It is expected that this sample will provide robust results for most Regions and Networks, even without stratification on a Regional/Network basis. However, in case the actual Board approvals differ significantly from present forecasts or if the wave one sample is unusually skewed, the sampling approach will be revisited to maximize robustness of results.

3

5. As in previous years, the sample will exclude all operations where the Task Team Leader was assessed in a QAG review under ESW-FY02, QSA-5 or QEA-5. Team Leaders who were subject to a review by the Inspection Panel during FY02-03 will also be exempted. 6. Timing of the interviews. Task teams will be asked to select an interview date that suits their schedule within the stipulated assessment window. Task teams are expected to provide relevant project documentation three weeks ahead of the interview. To simplify the interview process, every effort will be made to resolve outstanding issues ahead of the interview so that the attendance at the interview can be kept to a minimum and demands on the Task Team’s time can be reduced. 7. Questionnaires. The assessments will continue to use two questionnaires, one for investment projects and the other for adjustment operations. In addition, the specialized reviewers covering poverty and social aspects, environment, financial management and procurement will use supplementary questionnaires. Both the main and supplementary questionnaires (copy of main questionnaires for both investment and adjustment operations attached) will remain largely unchanged from last year’s assessment. 8. Aspects for special attention. Special efforts will be made in QEA6 to obtain a better understanding of:

• Results Focus . Panelists will be asked to comment on whether the operations show adequate focus on results (i.e. outcomes, rather than outputs or process) in their design. As in QEA5, task teams will be requested to identify the operation’s central development objectives, to specify interim benchmark indicators. They will also be asked to identify links between project components and key objectives. Since M&E has been identified as a continuing area of weakness, panelists will be asked to identify examples of good practice, and also to specify the binding constraint where M&E arrangements are not of satisfactory quality (i.e. whether this is because of Bank staff capacity constraints, lack of commitment of counterparts, etc.).

• IDA operations. Panelists will be requested to identify factors affecting the

quality of IDA funded operations (e.g., whether the shortcomings are country related or reflect deficiencies in the Bank’s performance). QEA6 will aim to frame conclusions/recommendations to resolve the quality shortfalls of IDA operations.

• Sector Managers’ contribution to quality enhancement. Panelists will be asked to interview managers of all operations where two or more quality dimensions are rated less than satisfactory. In multi-sectoral operations, panelists will be asked to consider separately inputs from the different sector managers.

• Implementation Arrangements . The assessment questionnaire was modified

last year to examine this aspect in greater depth. Following the same questionnaire, panelists will be asked to carefully examine the PIP; to highlight inconsistencies between the PAD’s formal sign-off on the readiness of the first

4

year’s program and actual conditions on the ground, and the appropriateness of the M&E arrangements for implementation.

Assessment Process 9. Stage I Assessments. The main focus of the assessments would continue to be the appropriateness of the operation’s approach and the likelihood of it meeting its stated development objectives. Using the Guidance Questionnaires as the basic framework, six assessors (generalist, sector specialist, poverty and social development specialist, environment specialist, procurement specialist, and financial management specialist) would assess each operation through a review of documentation and interviews with the task team (and management, as needed). All ratings would continue to be based on a four point scale : highly satisfactory, satisfactory, marginal, and unsatisfactory. Operations receiving an unsatisfactory rating on any of the Bank’s fiduciary or safeguard policies would normally be rated as less than satisfactory overall. Each panel will first send to the task team a set of questions in advance of the interview. Following the interview, the panel would complete the assessment questionnaire and share its ratings, conclusions, and detailed comments in draft with the task team and concerned line managers. This would be the point for task teams to provide supplementary information/clarifications, if any, to the panel’s consideration. Operations rated less than satisfactory at the end of the Stage I assessment would go through a Stage II review if so requested by the Region. QAG may also ask for a Stage II review for projects with contentious issues or for purposes of institutional learning. 10. Stage II Assessments. About 10% of operations in the QEA6 sample may require a Stage II review. The Stage II panel would normally comprise three to five professionals (internal and external) who are experienced, credible, and respected, with a mix of complementary skills and country knowledge. 11. Stage II panels are expected to spend about 3 days in an intensive review of the operation, and supplementary documentation would be compiled and a wider range of interviews than in Stage I would be arranged as needed by QAG in consultation with the TTL. The panel’s report would provide the rationale for the final rating, identify strengths and weaknesses, highlight lessons for other Bank operations, and comment on the quality of the Bank’s decision-making processes. The draft report would be shared with the task team and line managers, providing an opportunity for further comments or clarifications before finalization. The final report would be provided to the team, the concerned regional managers, and the sector board. Whenever a Region disagrees with the panel’s findings, it may attach a rejoinder stating the reasons, and/or proposing follow-up actions to address the shortcomings identified. 12. Documentation. Every effort will be made to minimize the burden on task teams associated with assembling the supporting documentation. QAG will provide binders with tabs for each operation in the sample, and attach an indicative Table of Contents along with every electronic notification. QAG will also provide orientations to TTLs and ACS staff for the operations sampled. Core documents would normally include, for investment projects, the PCD, PAD, loan and project agreements, PIP, CAS, decision memoranda, and managers’ and peer reviewers’ comments. Task teams will be expected to provide key supporting documents such as missions’ aide-memoires, correspondence with the Borrower and co-financiers, and documents pertaining to poverty, environment, financial management and

5

procurement assessments and action plans. For adjustment operations, documents corresponding to the respective processing steps would be included, as well as relevant country/sector analyses. TTLs are responsible for providing the documentation they consider relevant, and should exercise selectivity, since an overwhelming mass of documentation tends to slow down and complicate the process. Where TTLs request QAG to download items from the IRIS, they should provide specific details of the documents (document number, date, brief description of contents). 13. Panelists, Observers, and Moderators . As in the previous QEA assessments, panelists would be selected from senior Bank staff (grade H+), high pote ntial staff at grade G, retired Bank staff, and from external sources. A greater effort will be made this year to include current staff members as panelists, including Country Office staff traveling to HQ during the review period. With a view to bringing in perspectives from experts outside the Bank, wherever possible, panels would include non-Bank specialists from NGOs, bilateral donors, the private sector, and academia. An effort will be made to have non-Bank panelists on at least half the operations in QEA6. In light of comments received from task teams taking part in QEA-5, greater effort will be made this year to select panelists who have knowledge of key international languages (French and Spanish) and who are familiar with country conditions or at least with relevant recent operational experience in the region. In addition, Sector Boards and the Regions will again be invited to nominate staff to participate as observers in the assessments 14. All prospective panelists would have an orientation program, and would be expected to conduct the interviews in a spirit of cooperation and knowledge – enhancement. They would also be expected to provide their assessment reports in time to permit sharing with the task teams and line managers within two working weeks of the interview. 15. Panelists would have access to a pool of specialized resource persons (e.g. OPCS, Legal, CTR, and the Networks) for consultations as needed. To avoid conflict of interest, panelists would not review operations with which they had a previous involvement. A QAG moderator would participate in each assessment to ensure that assessments: (i) are as rigorous, objective, and professional as possible; (ii)utilize a collegial and constructive approach; (iii)incorporate stakeholders’ inputs; (iv) are consistent in approach across sectors and Regions. Moderators would also ensure that assessment reports are timely, provide insights into the areas of special attention in QEA6, and contribute to learning (e.g. by providing examples of good practice in Peer Reviews, PADs, or other elements of process and design). 16. Inputs from Stakeholders. Last year’s assessment included for the first time interviews with selected Borrowers’ officials, other donors and stakeholders in the country as listed by TTLs and/or Country Managers. We propose to continue the practice and will expand it to request inputs from stakeholders for all operations in QEA6. Panelists will supplement the electronic inputs with telephone interviews as needed in their judgment, while minimizing demands on stakeholders’ time and attention. 17. Preparation of Synthesis Report. A summary report describing the overall trends and patterns in quality at entry for FY03 would be issued at the end of the review process. The report would include analyses of the areas for special attention in QEA6. Results would be presented at the Bank-wide level, and also disaggregated by Region and Network.

6

18. All issues requiring re-examination of policy or a broad operational response would be forwarded for action to the Managing Director, Operations. Where necessary, follow-up action would be taken by the appropriate unit (OPCS, for example) with eventual consideration by the OPCS or OVPs. Other issues would be referred direct to the appropriate Bank units (e.g. Legal, CTR, Regions, and/or Sector Boards). QAG would continue collaborating with WBI to disseminate through Bank training programs the findings and lessons from QEA6. A the end of the exercise, staff, managers, and panelists would be invited to submit comments, which would be considered by QAG and incorporated in any future evaluations. Inputs and Costs 19. The average direct cost of each Stage I assessment is estimated at about $8,600. This estimate includes two main panelists for five person-days, an external panelist, when feasible (expected for about half of the operations) for three person days, a moderator for 2.5 person-days, and four specialist reviewers for 2.5 person-days, for a total of about $690,000 for 80 operations. The cost of each Stage II assessment would be about $9,000 including main panelists and specialized reviewers for 9 person-days and a moderator for three person-days. Assuming that 10 QEA6 operations will undergo Stage II reviews, this amounts to $90,000. QAG’s administrative support expenses are estimated at about $104,000, and management of the process, synthesis report writing, and contingencies at about $216,000. The overall cost of QEA-6 is estimated at about $1.1 million, of which the sector boards are expected to meet $100,000 (about 50% of the costs for the four specialized review areas). 20. Based on past experience, it is estimated that a typical task team spends about $3000 to prepare and participate in a QEA review. The overall estimated cost incurred by task teams for the 90 Stage I and Stage II reviews under QEA6 would therefore be about $270,000, which is expected to be borne by the Regions. Schedule 21. The assessment will commence with the sampling and notifications by mid-April, 2003 and the synthesis report is expected to be ready for the OVPs’ review by early February, 2004. Detailed steps in the assessment process will be as follows:

First sampling (40-50 operations) April 11, 2003 Second Sampling (30-40 operations) Mid-June, 2003 Submission of documents by task teams (first sampling) Late April, 2003 Submission of documents by task teams (second sampling) Early July, 2003 Interviews (Stage 1 and most of Stage II May-September, 2003 Completion of Stage II interview October, 2003 Assessment Reports Finalized Mid-November , 2003 Data Analysis Completed End-November, 2003 Draft Synthesis Report to Panelists December, 2003 Draft to Quality Directors Mid-January, 2004 OVP review February, 2004 CODE review March, 2004

7

ANNEX 3

MAIN CONTRIBUTORS

Staff Director

ModeratorsLogistics

Data Analysis

Adams, James W. Gausch, Jose Lovei, Laslo Rose, Thomas Agueh, Florent Gil-Diaz, Jose (IAD) Lowry, Heath (Princeton) Rosenblatt, David

Ahmad, Masood Gill, Jit Ludwig, Peter Schiavo-Campo, Salvatore

Aiyer, Sri-Ram Griffin, Dan (CRS) Mahmood, Khalid Schneider, Robert

Alaerts, Guy Guadamillas, Mario Manning, Nick Sesay, Baika (CRS)Alikhan, Mohsin Hakim, Peter (IAD) Marsden, David Shand, David

Alvarado, Oscar Halperin, Ricardo Matavel, Argentina (World Vision) Sharkey, Katrina

Ansu, Yao Hanan, Betty McCarthy, Nancy (IFPRI) Shifter, Michael (IAD)

Arnold, Margaret Harris, Bruce McKechnie, Alastair Shimizu, Masaharu Babu, Suresh (IFPRI) Harrison, Jim McLaughlin, Julie Singh, Ramesh Inder

Bajpai, Jit Hassan, Pervez Mejia, Abel Skolnik, Richard

Banerjee, Arun Hayward, John Mikeska, Gretchen (Urban Institute) Slabyj, Stefania (FHIl)

Barghouti, Shawki Holsen, John Mitchell, Sheila (FHIl) Socknat, Jim Barnes, Douglas French Hume, Ian Mohadjer, Gishu Sokol, Jose

Belli, Pedro Irigoyen, Jose Montes-Negret, Fernando Somel, Kutlu

Benbow, Jane (CARE) Iyer, Raj Moock, Peter Sood, Dave Berryman, Sue Iyer, Subramaniam V. Morrow, Daniel Steeds, David

Bottelier, Pieter Jafarli, Buba (CARE) Moser, Patricia (ADB) Stout, Susan

Carruthers, Robin Jimenez, Emmanuel Mott, Jessica Struben, Willem

Cartwright, Kimberly (Urban Institute) Jme'An, Suhail J. S. Muller, Helga Sutch, Helen Chang, Mae Chu Kabongo, Makasa (CRS) Mulusa, Mary Swartzendruber, Fred

Chen, Guang Kafka, Barbara Musalem, Alberto Taylor, Robert

Chopra, Ram Kagia, Ruth Nash, Petula (CRS) Thompson, Anthony

Churchill, Anthony Kalantzopoulos, Orsalia Nawaz, Tawhid Tuck, LauraCliffe, Sarah Kariuki, R. Mukami Nidel, Rick (CRS) Upadhyay, J.

Dani, Anis Kavalsky, Basil O'Brien, Steve Valdivieso, Cecilia

de Geyndt, Willy (Inter Health) Khan, Qaiser Otoo, Samuel van der Bijl, Rudolf

de Merode, Louis Koeberle, Stefan Owen, Daniel Varma, Keshav De St. Antoine, Jean Kohli, Harrinder Pantelic, Jelena Von Amsberg, Joachim

Donovan, Paula Kostner, Markus Pringle, Robert (Harvard) Webb, Richard (Peru)

Ecevit, Zafer Kress, Daniel Pulgar-Vidal, Max (IADB) Westin, Richard

Faiz, Asif Krishnamurthy, NC Rajagopalan, V.N. Wilson, Mike Fardoust, Shahrokh La Cava, Gloria Randol, Steven (U.S. Army) Wolfensohn, James

Faruqee, Rashid Lane, Michael (Global) Raven, John (Belgium) Woo, Robin (FDA)

Favaro, Edgar Lavadenz Paccieri, Isabel G. Razavi, Hossein Worrell, DeLisle (IMF)

Feder, Gershon Lewis, Maureen Redwood, John Yepes, Galli Fernandez, Jesus M. Ljumezi, Nicole (CRS) Rigo, Andres (Judge) Yepes, Guillermo

Fischer, Fritz Lochery, Peter (CARE) Ringskog, Klas Zacaria, Fernando (PAHO)

Fitzcharles, Kevin (CARE) Loos, Jane Ritchie, Daniel Zagha, Roberto

Funck, Bernard Lopes, Ana Paula Rosales, Alfonso (CRS) Zampaglione, Giuseppe

Baeumler, Axel E. N. Clert, Carine MacKay, Diana S. Rwechungura, Vedasto

Hakim, Roxanne Devnosadze, Sophie Mayfield, Jason S. Sosale

Akala, Francisca Ayodeji Dione, Ousmane Michel J. Welmond Salenna Prince

Armaly, Maha J. Djachechi, Yvette Laure Ndegwa, Stephen Sandi, Ana Maria Badjo, Yao Dropped/CD took VC Nobakht, H. Sieminska, Elzbieta

Baingana, Florence Frere, Jean-Jacques Pape-Christiansen, Andrea Tewari, D.

Bazarova, Saodat Hilary Kiell Perrot, Francoise Verardo, Barbara

Boskovski, Denis Horton, Brendan Peter Okwero Wazny, Allen Briceno-Garmendia, Cecilia Johansen, Anne Prevost, Yves Winters, Eliza

Byamugisha, F. Kaoues, Nora Randa El-Rashidi

Christensen, Olav Kristensen, P. Randa El-Rashidi

Christoph Prevost Linden, Toby Rondot, Pierre

Environment Financial Management Procurement Poverty and SocialBaratz, Bernard Agossou, Hugues Ginnsz, Jean Louis Cruz, Maria C.J.

Ijjasz-Vasquez, Ede Jorge Ambrose, Andrina Kiell, Hilary K. Dani, Anis

Mercier, Jean Roger Bruneau, Hilarion Larrieu, Jorge Gacitua-Mario, Estanislao

Rees, Colin Falconer, Brian Vick, Randall Harris, BruceGraham, Douglas Vijayaverl, Velayutham Marsden, David

Ladipo, Omowunmi Viswanathan, Narayanaswami Rennie, John Keith

OBSERVERS

THEMATIC SPECIALISTS

PANELISTS

QAG TEAMHari Prasad, P.

Prasad, P. Hari, Agueh, Florent; Banerjee, Arun; Chopra, Ram; Drydale, Robert; Golan, Amnon; Rana, Saeed; Wyss, Hans; Yurukoglu, Kadir.

Murphy, Brónagh; Cudjoe, Salli.Vaz, Melvin

ANNEX 4

8

TABLE 1. RESULTS BY REGION A. OVERALL ASSESSMENT

B. SUBRATINGS (% SATISFACTORY OR BETTER)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6AFR 57 26 85 88 92 89 79 100 79 95 74 95 94 82 87 92EAP 37 11 88 91 86 100 91 91 87 100 97 82 98 91 83 100ECA 57 17 89 90 91 97 90 97 83 82 98 93 93 100 82 97LCR 35 16 100 83 100 89 94 87 87 83 91 80 91 94 96 94MNA 22 6 96 48 96 87 84 61 82 84 78 79 87 100 92 100SAR 22 9 87 100 82 100 93 100 82 100 96 100 100 100 85 100

Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93 87 96

No. of Projects OA R1Region

R3R2 R6R5R4

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6AFR 57 26 85 88 68 80 68 72 78 88EAP 37 11 88 91 79 91 77 71 83 91ECA 57 17 89 90 70 100 81 90 82 90LCR 35 16 100 83 91 92 84 76 91 83MNA 22 6 96 48 87 61 88 48 80 87SAR 22 9 87 100 81 91 80 86 83 86

Bank-wide 230 85 90 85 78 88 78 76 83 88

RegionNo. of Projects OA R7 R8 R9

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6AFR 57 26 100 100 3 2 6 6 45 21 79 81 9 3 16 12EAP 37 11 100 100 6 1 15 10 27 9 73 82 4 1 11 9ECA 57 17 100 100 9 7 16 49 42 8 73 41 6 2 11 10LCR 35 16 100 100 8 2 21 11 27 11 79 72 0 3 0 17MNA 22 6 100 100 1 0 5 0 20 4 91 48 1 2 4 52SAR 22 9 100 100 6 0 23 0 14 9 64 100 2 0 13 0

Bank-wide 230 85 100 100 33 12 14 16 175 62 76 69 22 11 10 15

SatisfactoryNo. of Projects %

Marginal/UnsatisfactoryNo. of Projects %

RegionNo. of Projects %

Total Highly SatisfactoryNo. of Projects %

OA = Overall Assessment R5 = Fiduciary AspectsR1 = Strategic Relevance and Approach R6 = Policy and Institutional AspectsR2 = Technical, Financial and Economic Aspects R7 = Implementation ArrangementsR3 = Poverty and Social Aspects R8 = Risk Assessment

R4 = Environmental Aspects R9 = Bank Inputs and Processes

ANNEX 4

9

TABLE 2. RESULTS BY NETWORK/SECTOR (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6ESSD 45 12 93 92 95 100 93 100 92 100 89 91

Environment 18 1 94 100 94 100 94 100 100 100 87 100Rural Sector 25 9 91 88 96 100 91 100 86 100 90 87Social Development 2 2 100 100 100 100 100 100 100 100 100 100

FSE 15 5 90 100 90 100 82 100 94 100 91 50Financial Sector 15 5 90 100 90 100 82 100 94 100 91 50

HDN 60 23 92 80 90 97 91 84 82 92 89 85Education 25 6 86 51 82 91 91 51 79 91 90 67Health, Nutrition and Population 15 10 95 89 95 100 87 100 91 86 84 86Social Protection 20 7 96 100 96 100 94 100 79 100 92 100

INF 60 19 87 86 90 90 86 100 82 92 92 93Energy and Mining 17 5 81 100 91 100 78 100 77 100 94 100Global Information/Communications 3 0 100 0 100 0 53 0 100 0 100 0Transport 21 5 86 100 86 80 91 100 86 100 91 100Urban Development 8 5 100 53 100 79 100 100 73 75 84 79Water Supply and Sanitation 11 4 84 100 84 100 89 100 85 100 94 100

PREM 41 24 92 82 94 86 85 87 71 84 87 90Economic Policy 19 10 81 69 86 69 81 85 68 92 72 100Public Sector Governance 21 13 100 94 100 100 88 88 71 84 100 75LAJ/Poverty Reduction 0 1 0 0 0 0 0 100 0 0 0 100

PSDN 9 2 91 100 91 100 82 100 100 42 73 100Private Sector Development 9 2 91 100 91 100 82 100 100 42 73 100

Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6ESSD 92 92 98 100 84 92 84 76 87 92

Environment 100 100 96 100 100 100 90 100 88 100Rural Sector 84 88 100 100 75 88 82 65 88 88Social Development 100 100 100 100 54 100 54 100 54 100

FSE 87 74 84 100 84 100 82 80 77 100Financial Sector 87 74 84 100 84 100 82 80 77 100

HDN 96 96 92 97 76 78 82 71 81 93Education 97 100 88 100 77 60 83 34 76 91Health, Nutrition and Population 95 89 100 92 72 89 68 89 80 89Social Protection 96 100 91 100 78 85 91 90 88 100

INF 91 100 79 100 77 88 75 74 84 80Energy and Mining 92 100 63 100 65 86 68 73 81 100Global Information/Communications 100 0 100 0 77 0 30 0 77 0Transport 90 100 86 100 86 60 81 80 86 100Urban Development 91 100 100 100 78 100 72 53 100 53Water Supply and Sanitation 89 100 76 100 84 100 89 100 77 71

PREM 98 90 80 90 67 96 68 78 86 82Economic Policy 96 81 77 92 66 100 58 58 81 69Public Sector Governance 100 94 85 94 70 100 76 88 92 94LAJ/Poverty Reduction 0 100 0 0 0 0 0 100 0 0

PSDN 100 100 91 100 91 42 91 100 68 100Private Sector Development 100 100 91 100 91 42 91 100 68 100

Bank-wide 94 93 87 96 78 88 78 76 83 88

R8 R9

Network Sector

Network Sector

R4

R5 R6 R7

No. of Projects OA R1 R2 R3

OA = Overall Assessment R5 = Fiduciary Aspects

R1 = Strategic Relevance and Approach R6 = Policy and Institutional Aspects

R2 = Technical, Financial and Economic Aspects R7 = Implementation ArrangementsR3 = Poverty and Social Aspects R8 = Risk Assessment


ANNEX 4

10

TABLE 3. QEA 3-5 VS QEA6 RATINGS BY SECTOR

(% HIGHLY SATISFACTORY/SATISFACTORY )

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Agriculture 18.76 7.33 93 86 94 96 92 94 90 100 92 88 94 90Law & Public Admin 61.96 21.69 94 88 96 95 88 91 82 84 91 94 97 92ICT 3.93 0.77 97 100 100 100 65 100 64 100 90 100 100 95Education 21.39 8.04 87 59 83 86 89 63 80 90 96 73 97 99Finance 16.31 5.53 92 97 92 100 84 99 93 99 86 87 91 77Health & Social 35.35 19.3 90 83 91 90 90 99 87 89 85 88 96 93Industry & Trade 13.93 3.6 89 83 92 90 80 92 91 80 76 94 95 94Energy 18.85 5.61 84 99 92 98 85 99 76 99 89 94 92 95Transportation 22.39 6.21 88 95 88 95 91 97 81 88 90 100 92 100Water & Sanitation 17.13 6.92 87 86 88 88 91 98 81 90 91 88 80 100Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Agriculture 18.76 7.33 93 86 95 99 81 90 83 54 89 86Law & Public Admin 61.96 21.69 94 88 90 95 79 94 80 85 88 90ICT 3.93 0.77 97 100 97 100 73 100 70 100 82 100Education 21.39 8.04 87 59 89 97 78 70 83 44 84 86Finance 16.31 5.53 92 97 88 99 81 98 77 77 83 94Health & Social 35.35 19.3 90 83 92 93 72 86 72 80 75 82Industry & Trade 13.93 3.6 89 83 88 98 83 87 81 83 78 83Energy 18.85 5.61 84 99 69 99 64 89 70 80 79 99Transportation 22.39 6.21 88 95 83 100 81 69 76 79 87 95Water & Sanitation 17.13 6.92 87 86 83 94 86 94 87 85 80 75Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R3Sector

Total Projects OA R1 R2

Sector

R4 R5

Total Projects OA R6 R7 R8 R9



ANNEX 4

11

TABLE 4. QEA 3-5 VS QEA6 RATINGS BY THEME (% HIGHLY SATISFACTORY/SATISFACTORY )

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Econ. Mgmt. 7.35 3.96 88 89 90 89 88 96 80 83 94 100 94 100Public Sector/Gov. 25.55 12.29 93 94 97 100 91 94 71 89 93 87 96 93Law 11.21 2.37 97 100 97 100 89 100 94 95 91 100 99 94Finance & PSD 40.96 10.06 89 100 92 100 79 100 83 94 90 95 94 90Trade 5.88 3.23 87 100 91 100 95 100 88 63 88 100 83 82Social Protection 17.72 10.52 92 79 92 82 91 93 77 98 95 87 98 91Social Development 27.7 8.46 87 73 88 87 90 85 87 80 85 100 98 95Human Dev. 24.22 14.79 89 72 91 91 88 81 80 92 84 84 96 92Urban 20.09 6.42 87 70 85 88 89 92 77 77 91 86 88 94Rural 22.7 7.27 87 94 89 95 88 90 89 98 88 81 89 98Envir & Nat. Res. 26.62 5.63 96 88 97 94 90 97 92 100 90 85 92 94Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Econ. Mgmt. 7.35 3.96 88 89 88 100 73 100 89 91 88 91Public Sector/Gov. 25.55 12.29 93 94 88 95 72 100 69 82 86 93Law 11.21 2.37 97 100 83 93 79 100 89 92 80 100Finance & PSD 40.96 10.06 89 100 80 100 76 92 72 83 82 98Trade 5.88 3.23 87 100 96 100 86 61 72 97 87 100Social Protection 17.72 10.52 92 79 90 98 78 90 82 68 81 78Social Development 27.7 8.46 87 73 91 86 69 71 83 77 81 83Human Dev. 24.22 14.79 89 72 92 96 73 77 74 70 77 87Urban 20.09 6.42 87 70 76 94 79 89 77 55 81 70Rural 22.7 7.27 87 94 89 99 82 94 80 74 84 89Envir & Nat. Res. 26.62 5.63 96 88 94 100 91 90 88 64 90 83Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R4 R5Total Projects OA R1 R2 R3Sector

R7 R8 R9Sector

Total Projects OA R6



ANNEX 4

12

TABLE 5A. RESULTS BY LENDING INSTRUMENT WEIGHTED (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Adjustment 39 19 94 75 96 82 87 85 78 89 90 80 96 82

SAD 9 2 100 100 100 100 82 100 74 100 90 100 100 100SAL 24 8 89 78 93 78 85 100 74 89 88 69 93 72

SECAL/SSL 1 1 100 100 100 100 100 100 100 100 100 0 100 100PSL 3 5 100 48 100 71 100 48 100 77 0 0 100 77PRC 2 3 100 100 100 100 100 100 100 100 100 100 100 100

Investment 191 66 90 87 91 95 88 93 84 90 89 91 93 95APL 31 8 89 78 96 91 83 91 85 91 80 86 100 87ERL 9 8 84 70 84 60 91 100 66 85 73 84 93 100FIL 1 0 100 0 100 0 100 0 100 0 100 0 100 0LIL 16 1 100 0 100 100 92 0 90 100 85 0 100 100SIL 120 36 88 96 88 100 90 97 86 89 92 95 92 100SIM 2 1 100 0 100 100 100 100 100 100 100 0 30 0TAL 12 12 93 100 93 100 73 100 71 90 100 100 100 93

Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Adjustment 39 19 94 75 87 89 74 100 80 69 90 75

SAD 9 2 100 100 100 100 100 100 100 100 92 100SAL 24 8 89 78 82 89 65 100 66 89 86 89

SECAL/SSL 1 1 100 100 100 100 100 100 100 0 100 0PSL 3 5 100 48 70 77 34 100 100 48 100 48PRC 2 3 100 100 100 100 100 100 100 58 100 100

Investment 191 66 90 87 87 98 78 85 78 77 82 90APL 31 8 89 78 96 100 69 87 81 69 80 78ERL 9 8 84 70 75 85 52 71 75 84 58 70FIL 1 0 100 0 100 0 100 0 100 0 100 0LIL 16 1 100 0 91 100 84 0 90 0 87 100SIL 120 36 88 96 85 98 81 94 77 75 81 93SIM 2 1 100 0 100 100 100 0 100 0 100 0TAL 12 12 93 100 87 100 84 93 52 100 93 100

Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R8

R2

R9Instrument

No. of Projects OA

No. of Projects OA R1

R6 R7

InstrumentR4 R5R3





ANNEX 4

13

TABLE 5B. RESULTS BY LENDING INSTRUMENT - NOT WEIGHTED (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Adjustment 39 19 92 79 95 84 85 89 77 89 88 82 95 84 85 89SAD 9 2 100 100 100 100 78 100 78 100 86 100 100 100 100 100SAL 24 8 88 75 92 75 83 100 71 88 87 71 92 75 79 88SECAL/SSL 1 1 100 100 100 100 100 100 100 100 100 0 100 100 100 100PSL 3 5 100 60 100 80 100 60 100 80 0 0 100 80 67 80PRC 2 3 100 100 100 100 100 100 100 100 100 100 100 100 100 100Investment 191 66 90 89 91 94 88 95 84 90 90 92 94 95 87 97APL 31 8 90 75 97 88 84 88 84 88 82 88 100 88 97 100ERL 9 8 78 75 78 63 89 100 67 88 83 88 89 100 78 88FIL 1 0 100 0 100 0 100 0 100 0 100 0 100 0 100 0LIL 16 1 100 0 100 100 94 0 88 100 83 0 100 100 88 100SIL 120 36 89 97 89 100 90 97 86 92 92 97 92 100 85 97SIM 2 1 100 0 100 100 100 100 100 100 100 0 50 0 100 100TAL 12 12 92 100 92 100 75 100 73 89 100 100 100 92 83 100Bank-wide 230 85 90 87 92 92 88 94 83 90 89 90 94 93 87 95

InstrumentTotal Projects OA R1 R4 R5 R6R2 R3

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Adjustment 39 19 94 79 72 100 77 74 87 79SAD 9 2 100 100 100 100 100 100 89 100SAL 24 8 89 75 63 100 63 88 83 88SECAL/SSL 1 1 100 100 100 100 100 0 100 0PSL 3 5 100 60 33 100 100 60 100 60PRC 2 3 100 100 100 100 100 67 100 100Investment 191 66 90 89 79 86 78 77 82 89APL 31 8 89 75 71 88 81 63 81 75ERL 9 8 84 75 44 75 78 88 56 75FIL 1 0 100 0 100 0 100 0 100 0LIL 16 1 100 0 81 0 88 0 88 100SIL 120 36 88 97 82 92 78 75 83 94SIM 2 1 100 0 100 0 100 0 100 0TAL 12 12 93 100 83 92 58 100 92 100Bank-wide 230 85 90 87 77 89 78 76 83 87

R7 R8 R9Instrument

Total Projects OA



ANNEX 4

14

TABLE 6. RESULTS BY LOAN SIZE

(% Satisfactory or Better)

TABLE 7. RESULTS BY ELAPSED TIME (FROM CONCEPT TO BOARD) (% Satisfactory or Better)

Loan SizeQEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6

a. <= 25 99 23 92 80 92 96 88 81 86 82 88 100 94 93b. > 25 & <= 50 55 19 88 86 90 95 86 100 81 95 88 82 94 86C. > 50 & <= 100 31 20 94 89 94 89 88 100 86 100 93 86 97 96C. > 100 45 23 88 91 92 87 90 96 79 85 90 93 91 100Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

Loan SizeQEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6

a. <= 25 99 23 92 80 88 97 76 86 81 73 81 89b. > 25 & <= 50 55 19 88 86 83 95 70 81 73 74 83 81C. > 50 & <= 100 31 20 94 89 95 96 86 96 84 76 86 89C. > 100 45 23 88 91 87 96 84 92 75 83 84 91Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R6

R1 R2 R3 R4 R5QANo. of Projects

No. of Projects QA R9R8R7

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6a. 0 - 6 51 21 95 71 95 78 89 93 74 82 86 92 97 94b. 6 - 12 51 29 88 85 91 96 90 87 86 89 92 92 93 95c. 12 - 24 85 24 92 88 93 97 88 93 87 91 90 82 91 86d. > 24 43 11 85 100 85 100 84 100 82 100 88 100 96 100Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6a. 0 - 6 51 21 95 71 85 94 71 91 75 65 83 71b. 6 - 12 51 29 88 85 92 95 77 80 83 79 90 95c. 12 - 24 85 24 92 88 89 100 86 88 80 69 85 88d. > 24 43 11 85 100 80 95 68 100 75 93 70 93Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R7Elapsed Time

No. of Projects QA

R1Elapsed Time

No. of Projects QA

R6

R5R4

R8 R9

R2 R3



R2 = Technical, Financial and Economic Aspects R7 = Implementation Arrangements

R3 = Poverty and Social Aspects R8 = Risk Assessment


ANNEX 4

15

TABLE 8. RESULTS BY QUARTER OF APPROVAL (% Satisfactory or Better)

TABLE 9. RESULTS BY SOURCE OF FUNDS (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Q1 29 21 87 73 84 89 80 83 90 91 86 86 97 94Q2 36 12 86 86 93 94 89 100 89 100 87 72 98 86Q3 43 13 91 89 92 89 97 89 74 100 74 87 93 100Q4 122 39 93 91 93 96 86 95 83 82 98 96 92 92

Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Q1 29 21 87 73 83 97 74 83 89 60 81 84Q2 36 12 86 86 90 100 73 86 83 69 68 86Q3 43 13 91 89 93 100 82 86 73 89 83 89Q4 122 39 93 91 85 93 78 91 76 83 88 90

Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

Approval Quarters

No. of Projects QA

No. of Projects QA R1

R6 R7 R8 R9

R2 R3 R4 R5Approval Quarters

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6IBRD 85 38 96 84 96 94 91 88 82 88 95 80 92 98IDA/SF/SPF 130 47 86 86 89 92 85 95 83 91 86 94 95 89GEF/MP 15 0 92 0 92 0 92 0 100 0 83 0 100 0Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6IBRD 85 38 96 84 90 96 82 89 86 77 87 92IDA/SF/SPF 130 47 86 86 85 96 72 86 72 74 79 84GEF/MP 15 0 92 0 95 0 100 0 87 0 85 0Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

Source of FundNo. of Projects QA R1 R2 R3 R4 R5

R6 R8 R9Source of Fund






ANNEX 4

16

TABLE 10. RESULTS BY PREPARATION COSTS ($’000) (% Satisfactory or Better)

TABLE 11. RESULTS BY LOCATION OF PROJECT TASK TEAM LEADER (TTL) (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6a. $0 - 200K 70 15 95 68 95 89 89 73 81 80 85 100 94 90b. $200 - 400K 53 29 94 91 96 100 90 100 78 95 92 95 95 91c. $400 - 600K 43 14 88 83 90 83 86 100 93 86 87 76 95 95d. > $600K 64 27 85 94 86 95 87 94 82 93 92 86 93 96Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6a. $0 - 200K 70 15 95 68 88 90 83 80 80 67 86 81b. $200 - 400K 53 29 94 91 88 97 75 89 80 82 86 91c. $400 - 600K 43 14 88 83 82 100 76 90 73 69 84 88d. > $600K 64 27 85 94 89 97 76 91 79 81 76 90Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

Source of FundNo. of Projects R9R8R7QA R6

R2 R3 R4 R5Source of Fund


QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6FieldBased 33 11 95 93 92 93 88 93 98 93 90 100 98 100Headquarters 197 74 90 84 92 93 88 91 81 89 89 87 93 92Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6FieldBased 33 11 95 93 87 100 75 100 82 65 90 84Headquarters 197 74 90 84 87 96 78 86 78 77 82 88Bank-wide 230 85 90 85 87 96 78 88 78 76 83 88

R3 R4 R5No. of Projects OA R1 R2

R8 R9Location of TL

No. of Projects OA R7R6

Location of TL





ANNEX 4

17

TABLE 12. RESULTS BY CPIA GROUP (% Satisfactory or Better)

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Low CPIA 101 30 86 79 89 88 83 96 77 85 90 87 93 89 85 94High CPIA 113 49 94 87 94 95 91 88 88 91 89 90 95 97 88 97Bank-wide 230 85 90 85 92 93 88 92 83 90 89 89 94 93 87 96

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Low CPIA 101 30 86 79 68 81 69 67 81 81High CPIA 113 49 94 87 85 91 85 78 85 90Bank-wide 230 85 90 85 78 88 78 76 83 88

R7

Total Projects OA R1 R2 R3

Those countries with CPIA ratings below 3.5 are labelled as Low CPIA group, the other countries are labelled as High CPIA group. Those countries (6 projects in QEA6) without CPIA ratings are excluded from the computations of the results for low and high CPIA group

CPIA Group

CPIA GroupR8 R9Total Projects OA

R4 R5 R6





ANNEX 4

18

TABLE 13. AVERAGE ELAPSED TIME AND PREPARATION COST BY INVESTMENT

QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6 QEA3-5 QEA6Adjustment 39 19 3.8 2.6 4.3 2.9 8.3 5.6 442 379

SAD 9 2 6.1 2.6 5.0 3.1 11.1 5.7 578 541SAL 24 8 3.1 2.9 4.4 3.1 7.8 6.0 410 447

SECAL/SSL 1 1 3.8 2.6 2.2 1.0 6.0 3.6 354 20PSL 3 5 0.9 1.4 0.9 2.8 3.4 4.1 284 235PRC 2 3 6.6 4.1 3.6 3.3 10.3 7.5 511 447

Investment 191 66 10.2 9.8 7.1 5.5 17.3 15.3 476 540APL 31 8 9.9 10.7 6.5 5.7 16.4 16.4 564 678ERL 9 8 1.2 2.5 2.1 3.9 3.5 6.3 187 430FIL 1 2.2 2.8 5.0 88LIL 16 1 7.3 2.8 6.6 6.9 14.0 9.7 250 112SIL 120 36 11.8 11.3 8.0 6.1 19.8 17.4 531 583SIM 2 1 14.2 14.1 7.5 2.5 21.7 16.6 674 1,049TAL 12 12 4.5 9.9 4.6 4.9 9.1 14.8 213 383

Bank-wide 230 85 9.1 8.2 6.6 5.0 15.7 13.2 470 504

From Concept to Board

Average Preparation Costs ($'000)

InstrumentNo. of Projects

From Concept to Appraisal

From Appraisal to Board

ANNEX 4

19

Table 14. Project Objectives (% of the total projects in a category)

ADJ INV IBRDIDA/SF/

SPF GEF/MP Bank-wide% of H or M among total projectsPoverty Reduction 95 74 76 80 0 78Structural and Sector Policy Reform 100 73 78 80 0 79Private Sector Development 77 65 73 63 0 67Institutional Development/Capacity Building 100 96 100 93 0 96Human Development 57 40 48 38 0 82Environmental Sustainability 61 80 81 75 0 77Infrastructure Development 87 85 84 85 0 85 Macroeconomic Management 83 82 95 67 0 83Other 61 97 90 89 0 90

% of projects rated H or M Likely to be achievedPoverty Reduction 40 56 58 47 0 52Structural and Sector Policy Reform 77 77 94 62 0 77Private Sector Development 64 62 73 53 0 62Institutional Development/Capacity Building 50 78 78 70 0 74Human Development 76 87 78 92 0 84Environmental Sustainability 100 88 86 92 0 90Infrastructure Development 71 100 100 95 0 97Macroeconomic Management 61 88 93 38 0 74Other 55 90 87 84 0 86

Sustainability 41 63 73 47 0 60

20

ANNEX 5

1 = Highly Satisfactory

2 = Satisfactory

3 = Marginally Satisfactory

4 = Unsatisfactory

NA = Not Applicable

QEA1 QEA2 QEA3 QEA4 QEA5 QEA6

R1 = Strategic Relevance and Approach 85 90 91 93 90 93

R2 = Technical, Financial and Economic Aspects 81 85 87 87 91 92

R3 = Poverty and Social Aspects 70 68 80 87 80 90

R4 = Environmental Aspects 95 93 90 90 87 89

R5 = Fiduciary Aspects 0 77 89 95 98 93

R6 = Policy and Institutional Aspects 74 68 83 90 87 96

R7 = Implementation Arrangements 76 81 82 83 64 88

R8 = Risk Assessment 72 72 73 78 85 76

Over All Assessment 82 86 89 94 86 85

R9 = Bank Inputs and Processes 75 85 81 91 72 88

QEA6 GUIDANCE QUESTIONNAIRE

Quality Assessment Results

Assessment Rating

21

R1 Strategic Relevance and ApproachQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

1.1 Quality and coherence of development rationale underpinning the project? 80 78 83 91 93 97

1.2 Consistency of the projects' objectives NA NA NA NA NA 92

1.2(a) Are the DOs adequately focused on outcomes? NA NA NA NA NA 93

1.2(b) Are the DOs sufficiently but not overly ambitious? NA NA NA NA NA 93

1.3 Appropriateness of project approach? 83 84 84 86 82 89

1.4 Extent to which lessons of experience are adequately reflected in project approach? 78 79 83 88 76 85

1.5 Adequacy of country and sector knowledge underpinning the project? NA 85 85 96 99 98

1.6 Level of borrower ownership 80 88 92 97 90 99

1.6(a) Degree of borrower participation in project design and implementation? NA NA NA 93 90 97

1.6(b) Credible evidence of borrower ownership and commitment based on track record and pre-Board actions? NA NA NA 94 88 92

1.6(c) Presence of strong champions? NA NA NA 96 90 98

1.7 Appropriate partnership arrangements with other donors? NA 93 90 89 85 96


22

QEA1 QEA2 QEA3 QEA4 QEA5 QEA62.1 Technical soundness of the project? 89 86 92 86 92 98

2.1* Adequacy of external financing case for the operation? NA NA NA 84 94 86

2.2 Appropriateness of alternative designs considered? 70 89 84 85 82 89

2.2(a)* Adequacy of total amount of financing relative to projected financing requirements? NA NA NA 82 100 93

2.2(b)* Is the Bank's financial contribution consistent with its role of providing medium-term support for structural reform? NA NA NA 82 94 91

2.3 Quality and coherence of economic rationale and analysis underpinning the project? 79 72 82 73 93 85

2.3* Extent and appropriateness of medium-term framework? 79 72 82 69 84 752.4 Appropriateness and realism of project conditionality? 77 72 76 86 59 85

2.5 For revenue-earning projects NA NA NA 83 63 100

2.5(a) Quality of financial analysis over the useful life of the investment? NA NA 92 83 71 66

2.5(b) Appropriateness of financial covenants and performance indicators? NA 52 88 87 71 100

2.5* Appropriateness of tranching arrangements? 70 89 84 100 92 100

2.5(a)* How well does the phasing of disbursements match the financing requirements? NA NA NA 100 100 100

2.5(b)* How well does the phasing of reforms match the phasing of disbursements? NA NA NA 100 84 82

2.6 Realism of the project's financing plan? 86 87 90 85 87 88

2.7 Projects in Disputed Areas (OP 7.60) NA 100 100 100 NA NA

2.8 Projects on International Waterways (OP 7.50) NA 100 83 100 100 100

R2 Technical, Financial and Economic Aspects


23

R3 Poverty and Social AspectsQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

3.1 Project's focus on poverty issues NA NA 90 91 80 88

3.1(a) Quality and depth of poverty diagnosis? NA NA NA NA 70 89

3.1(b) Appropriateness and comprehensiveness of poverty design? NA NA NA NA 72 86

3.2 Extent to which gender issues were considered during project design? NA NA NA 69 66 81

3.3 Extent to which relevant social development issues were considered in project design and institutional arrangements? 77 70 78 81 69 82

3.4 Quality of stakeholder analysis and consultation NA NA NA 78 68 92

3.4(a) Adequacy of stakeholder analysis? NA NA NA 70 68 89

3.4(b) Degree of beneficiaries and other key stakeholders, participation? NA NA NA 69 64 91

3.5 Appropriateness and realism of arrangements to manage and mitigate any adverse social impacts arising out of the project? 65 74 63 77 66 84

3.6 Indigenous Peoples (OD 4.20) NA 100 100 100 100 94

3.7 Involuntary Resettlement (OD 4.30) NA 90 78 87 90 100

3.8 Cultural Property (OPN 11.03) NA 100 67 100 100 100


24

R4 Environmental AspectsQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

4.1 Assessment of environmental impacts and risks 91 91 88 85 86 93

4.1(a) Extent to which significant environmental impacts (positive and negative) and risks were identified and assessed? NA NA NA NA 88 93

4.1(b) Were alternatives examined and compared? NA NA NA NA 87 97

4.1(c) Extent to which consultation with affected groups conducted in a satisfactory manner? NA NA NA NA 91 94

4.1* Environmental classification for SECAL operations NA NA NA 80 100 100

4.1(a)* Was an environment classification carried out? NA NA NA NA 100 100

4.2 Adequacy of arrangements for mitigating and managing any adverse environmental impacts during (and following) project implementation? NA 86 79 84 84 89

4.2* Environmental Assessment for SECAL operations classified A or B: NA NA NA 100 NA 84

4.3 Adequacy of assessment of in-country environmental policies and institutional capacity for carrying out the EA and ensuring subsequent compliance? NA NA NA 81 90 96

4.3* Extent to which the adjustment operation adheres to environmental Good practice? NA NA NA NA 100 68

4.4 Appropriateness and quality of the Environmental Assessment NA NA NA 88 90 90

4.4(a) Appropriateness of the selected environment category ? NA NA NA NA 93 94

4.4(b) Choice of the EA instrument (SEA/REA/EIA/EMP/Audit) ? NA NA NA NA 87 98

4.4(c) Adequacy of public disclosure procedures (BP 17.50)? NA NA NA NA 93 97

4.4(d) Quality of EA report? NA NA NA 100 89 92

4.4(e) Quality of the EMP? NA NA NA NA NA 87

4.4(f) EA results properly reflected in project documentation (e.g., Log Frame)? NA NA NA NA 88 81

4.5 Environmental Assessment (OP 4.01) NA 94 92 91 93 93

4.6 Natural Habitats (OP 4.04) NA 93 89 75 84 100

4.7 Forestry (OP 4.36) NA 100 100 89 100 100

4.8 Pest Management (OP 4.09) NA 100 86 95 100 100

4.9 Safety of Dams (OP 4.37) NA 100 100 100 100 100


25

R5 Fiduciary Aspects QEA1 QEA2 QEA3 QEA4 QEA5 QEA6

5.1 Financial Management NA NA NA 94 88 92

5.1(a) Adequacy of financial management systems and personnel for project accounting, internal controls, auditing and reporting as described in the Project Implementation Plan? NA 74 89 84 94 96

5.1(b) Adequacy of financial management assessment as recorded in the Project Appraisal Document? Were appropriate and time bound measures introduced to correct any shortcomings? NA 72 68 89 86 87

5.1(a)* Compliance with disbursement and auditing requirements? NA NA NA 100 100 93

5.1(b)* Extent to which the fiduciary concerns regarding the overall use of borrower resources were addressed? NA NA NA 93 75 77

5.2 Procurement NA 96 92 91 95 96

5.2(a) Quality and adequacy of capacity assessment of the implementing agency(ies)?)? NA NA NA 94 NA 94

5.2(b) Quality and adequacy of arrangements for managing procurement? NA NA NA 90 94 99


26

R6 Policy and Institutional AspectsQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

6.1 Extent to which the prevailing economic policy environment was considered in project design? 89 86 92 94 94 98

6.2 Appropriate identification of agencies and institutions (e.g. government, regulatory, judicial, private sector, NGOs) responsible for project execution and clarity of their respective roles? 87 73 81 95 86 95

6.2* Consistency of the policy and institutional reform actions supported by the operation with the stated reform/development objectives? NA NA NA 84 83 79

6.3 Extent to which the implementing agencies have 69 63 77 87 79 90

6.3(a) Capacity to implement the project NA NA NA 75 68 73

6.3(b) Commitment to vigorously implement the project and promote the stated objectives? NA NA NA 92 84 96

6.3(c) Political capacity to carry out reforms and institutional changes included in the project? NA NA NA 79 81 84

6.4 Where the institutional capacity is deficient, appropriateness and realism of the capacity building measures NA NA 81 78 81 91

6.4(a) Institutional capacity assessment? NA NA NA 79 82 90

6.4(b) Technical assistance arrangements? NA NA NA 81 80 89

6.4(c) Staff and management training? NA NA NA 75 84 89


27

R7 Implementation ArrangementsQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

7.1 Prospects for completing the project within the prescribed time-frame? (taking into account the realism of the project implementation schedule, and the extent to which experience from similar projects in the country/region were considered)? 78 81 82 69 59 83

7.1(a)* Realism of time allotted for completing reforms supported by future tranches, relative to their ambitiousness and institutional capacity? NA NA NA NA 62 80

7.1(b)* Extent to which experience from similar operations in the country/region are reflected in the implementation plane? NA NA NA NA 80 90

7.2 Readiness of the first year's program for implementation? 77 75 84 82 74 88

7.2(a) Adequacy and timeliness of counterpart funding for first year's? NA NA NA NA 94 96

7.2(b) Bid documents for first year's procurement reviewed and approved? NA NA NA NA 65 88

7.2(c) Arrangements for disbursements in place? NA NA NA NA 98 93

7.2(d) On-lending arrangements in place? NA NA NA NA 73 96

7.2(e) Project manager appointed prior to Board approval? NA NA NA NA 86 90

7.2(f) TA, staffing and management arrangements agreed and in place? NA NA NA NA 55 79

7.2(a)* Unless specifically justified by special circumstances, absence of effectiveness conditions or substantive reform actions? NA NA NA NA 92 83

7.2(b)* Arrangements for disbursements in place? NA NA NA NA 100 100

7.3 Quality of Project Implementation Plan? NA NA NA NA 70 89

7.4 Appropriateness of arrangements to monitor implementation and to review progress with the borrower? Specificity of indicators and benchmarks? Clear assignments of monitoring responsibilities to implementing agencies? NA 74 78 91 75 87

7.4(a) Tracking of implementation progress? NA NA NA 90 81 85

7.4(b) Poverty and social aspects? NA 46 51 75 53 80

7.4(c) Adverse environmental impacts? NA NA NA 79 91 89

7.4(d) Procurement monitoring? NA NA NA 99 95 99

7.5 Appropriateness of arrangements (e.g. baseline survey, outcome indicators) for evaluating impact and measuring outcomes of 72 65 64 73 62 85

7.5(a) Institutional and policy reform aspects? NA NA NA 73 58 89

7.5(b) Project's development objectives? NA NA NA 74 61 81

7.5(c) Poverty and social aspects? NA NA NA 73 52 76

7.5(d) Adverse environmental impacts? NA NA NA 73 83 85


28

R8 Risk AssessmentQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

8.1 Quality of risk assessment (With particular reference to candor, rigor, comprehensiveness and realism) and appropriateness of strategies for dealing with risk during program implementation (overall rating) NA NA NA NA 83 82

8.1(a) Country capacity (Non supportive macroeconomic and political environment, governance problems and borrower commitment) (Overall Rating) NA NA NA NA 78 84

8.1(b) Institutional effectiveness to implement the project (Overall Rating) NA NA NA NA 68 88

8.1(c) Technical soundness (Design complexity (multiple or untested components), first time operation in the country/sector)(Overall Rating) NA NA NA NA 67 87

8.1(d) Economic viability (Economic soundness based on cost-benefit or cost-effectiveness analysis)(Overall Rating) NA NA NA NA 81 89

8.1(e) Financial viability (For revenue-earning projects, financial soundness based on financial rate of return or NPV, appropriateness of tariffs in relation to marginal cost and O&M cost; reliability of financial reporting; for all other projects, fisc NA NA NA NA 74 74

8.1(f) Social risks (Risk of adverse social impact like exclusion of key groups, socio-political support) (Overall Rating) NA NA NA NA 54 85

8.1(g) Environmental risks (Including natural resource management and indigenous peoples or resettlement issues)(Overall Rating) NA NA NA NA 82 86

8.1(h) Financial management capacity (Financial management risks in the country that may impact the project)(Overall Rating) NA NA NA NA 95 89

8.1(i) Procurement capacity (Procurement risks faced by the country that may impact the project)(Overall Rating) NA NA NA NA 91 95

8.1(j) Resilience to exogenous factors (Reaction of local and external civil society advocates, support from relevant stakeholders)(Overall Rating) NA NA NA NA 72 89

8.2 Realism of the operation's overall risk rating and its relation to rewards NA NA NA 73 78 70


29

R9 Bank Inputs and ProcessesQEA1 QEA2 QEA3 QEA4 QEA5 QEA6

9.1 Elapsed time (months) NA NA NA NA NA NA

9.1(a)(i) Elapsed time : Appraisal-Board NA NA 6.3 6.7 7.2 5

9.1(a)(ii) Elapsed time : Appraisal-Board : Was the service standard of 5 months met? NA NA NA 50 63 70

9.1(b)(i) Elapsed time : Board Effectiveness NA NA 4 3.8 2.8 3.8

9.1(b)(ii) Elapsed time : Board Effectiveness : Was the service standard of 4 months met? NA NA NA 46 54 42

9.4 Task team's composition in relation to operation's complexity? NA NA NA 91 95 97

9.5 Appropriateness of processing time considering operation's complexity and urgency? NA 89 90 85 91 85

9.6 Overall efficiency of resources used? 85 86 86 86 75 84

9.7 Value Added from management NA NA NA 86 73 81

9.7(a) Country Management NA NA NA 87 78 85

9.7(b) Sector Management NA NA NA 80 76 81

9.7(c) Regional Quality Assurance Team NA NA NA 88 76 88

9.8 Quality Enhancement 100 77 83 85 59 91

9.8A Peer Reviews NA NA NA NA 63 91

9.8A(a) Quality Enhancement : A. Peer Reviews a) Appropriateness of reviewers' selection? NA NA NA 92 92 94

9.8A(b) Quality Enhancement : A. Peer Reviews b) Quality of guidance given? NA NA NA 93 82 92

9.8A(c) Quality Enhancement : A. Peer Reviews c) Appropriate use of advice provided? NA NA NA 86 72 85

9.8B QER NA NA NA NA 64 88

9.8B(a) Quality Enhancement : B. QER a) Appropriateness of reviewers' selection? NA NA NA 95 100 87

9.8B(b) Quality Enhancement : B. QER b) Quality of guidance given? NA NA NA 86 81 92

9.8B(c) Quality Enhancement : B. QER c) Appropriate use of advice provided? NA NA NA 86 64 88

9.9 Quality of Support Provided NA 98 94 94 95 98

9.9(a) Sector Board and Anchor NA NA NA 93 89 96

9.9(b) Legal NA NA NA 92 89 97

9.9(c) LOA (Disbursement) NA NA NA 94 94 99

9.10 Quality of Bank's Documents (PAD, Legal documents) 85 78 80 75 59 79


30

ANNEX 6

POVERTY AND SOCIAL ASPECTS

OVERVIEW This year’s QEA sample consisted of 19 adjustment and 66 investment operations. Of these, 3 were considered not applicable for poverty and social aspects due to the technical nature of the operation. Furthermore, projects were classified into three categories of focused , inclusive, or enabling, and different benchmarks were used to assess projects in each category. 1

KEY FINDINGS The satisfactory rating for poverty and social aspects in QEA6 of 90% shows a return to the long-term trend of improving ratings despite the dip to 80% in QEA5 (see graph), which involved a smaller sample of 50 projects.

• Improvements from last year were registered in all regions except for MNA, and in three networks (HDN, PREM and INF).

• Poverty diagnos is received a satisfactory rating of

90%. However, there is a lag between poverty diagnosis and design, which was rated at 82% satisfactory (51 out of 62 investment projects), based on the extent to which the operation’s design was derived from poverty analysis.

• Quality of monitoring arrangements for poverty and

social impacts improved slightly over earlier years (80% satisfactory rating this year compared to 53% in QEA5, and 73% in QEA4), although this remains one of the weakest aspects of project quality. Arrangements for evaluating impacts and measuring poverty and social outcomes was 76% satisfactory or better.

• Social safeguard ratings continued to improve, with 100% satisfactory ratings for

involuntary resettlement (90% in QEA5) and cultural property. The satisfactory rating for indigenous peoples dropped to 93% from the previous 100% ratings for

1 As defined in the benchmarks for the specialized review for QEA6, focused operations contain actions targeted to bring benefits predominantly to poor people; they address specific and practical needs and strategic interest of poor and marginalized people. Inclusive operations benefit broad based population groups, including the poor; these operations contain components that improve opportunities and services generally while also addressing issues of equity and barriers to participation of the poor. Enabling operations include policies that promote sound economic growth, governance/security, and social and institutional development that support a country’s poverty alleviation strategy. For this review, there were 8 focused; 36 inclusive; and 38 enabling, while poverty was deemed to be not applicable to three operations.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

QEA2 QEA3 QEA4 QEA5 QEA6

Poverty and Social Aspects (% Satisfactory)

QEA2-QEA6

31

QEA2-5, but this was due to one project out of 15 in the QEA6 cohort that involved indigenous people. However, assessment of poverty and social risks received less attention than safeguards, being rated 85% satisfactory or better.

• There was a significant improvement in consideration of gender issues in project

design, but monitoring of gender impacts needs to be improved. Out of 68 projects, 55 received satisfactory ratings on gender issues (80%), which was an improvement over the last 2 years (69% in QEA4 and 66% in QEA5). However, only 58% of these projects included monitoring of gender impacts.

• This year, 19 investment projects used CDD approaches. These projects were found

to be better in designing mechanisms for M&E of project impacts, with a satisfactory rating of 95%, compared to 74% for non-CDD projects.

• 20 projects in this year’s cohort are emergency operations. The satisfactory rating on

ana lysis of social risks in these emergency operations is lower at 79%, compared to the 86% average rating for the 51 non-emergency types of projects.

. DISCUSSION

Strong poverty diagnosis but weak in design. There is a gap between poverty diagnosis and design. The satisfactory rating on poverty diagnosis of 90% can be attributed largely to references and more explicit linkages to poverty analyses conducted by other Bank supported activities (e.g., PRSPs, Poverty Assessments and poverty analysis in other AAA and links to the poverty reduction in the CAS, as well as other parallel poverty reducing operations).2 However, the findings on poverty analyses were not always utilized well in project design.

Timeliness vs. comprehensiveness. This year’s cohort included 12 emergency operations which required urgent responses from the Bank. These included 8 ERLs and 4 projects designed to address pressing health issues. Social safeguard requirements for ERLs are less stringent than those for normal projects.3 Panel Members felt that, although given the necessity of providing immediate assistance in emergency operations, comprehensive poverty diagnosis is not expected during preparation, task teams should consider the poverty implications in the design of the project.4 This is especially true for operations in countries which have had previous poverty assessments or PRSPs. Review of adjustment operations. This year’s cohort contained a larger number of adjustment operations (19 compared to 10 in QEA5). Some of these operations were for balance of payments and support to deal with financial crises. Four of the 8 SALs were in conflict-

2 One-half of projects (41) made use of findings from completed poverty assessments (15 HD, 6 SP, 2 SD, 2 energy, 9 rural, 3 transport, and 4 water projects), and contained output indicators related to the MDGs. Eight of the 12 countries that completed their PRSPs by end of FY02, including 3 PRSC projects (Burkina, Uganda, Vietnam) were included in this year’s cohort. 3 OP8.50 (Emergency Recovery Assistance) states that “Each ERL project is adapted in form and scope to the emergency’s particular circumstances and retains flexibility. ERL projects use disaster-resilient reconstruction standards.” 4 The average preparation time, up to Board approval, was 3.75 months for the 8 ERLs, and 5 months for the 3 special health intervention projects (except for Ukraine, which was prepared in 11 months).

32

affected countries. Good practices included those operations which designed programs by drawing on completed poverty analyses and PRSPs, taking account of the distributional impacts of these programs, and providing strategies for mitigating social risks (see Box). NEXT STEPS Integrate lessons from poverty diagnosis more systematically into design. Task teams should make more effective use of findings from poverty diagnosis in project design. PREM and SDV should help task teams incorporate, more systematically, lessons from upstream analytical work (e.g., ESWs, Poverty and Social Impact Analysis), and consider the non-income dimensions of poverty in the design of project activities and programs. Be flexible in emergency operations while including actions to address social risks. Flexibility is needed in assessing projects which are prepared under difficult conflict conditions. However, as suggested by Panel Members in an emergency TAL: “Some of the reforms and changes being promoted by this project will require a strong participatory and consultation process and a good outreach to various segments of civil society. Greater emphasis should be placed on these aspects during implementation, particularly in conjunction with the project’s support of the PRSP process.”5 In such cases, project preparation and social appraisal tools need to be adapted to country circumstances to inform project design. Integrate gender issues more systematically into Bank operations . There continues to be a problem with translating gender issues into operational design. Task teams need to draw on the findings from Country Gender Assessments in project design. Projects also need to expand programs for monitoring gender impacts. Use improved M&E of CDD projects for scaling up activities. The increasing number of projects with CDD components indicate an expansion of CDD in the Bank’s portfolio. Task teams should apply the lessons and experiences in CDD projects, as documented and evaluated through its improved M&E systems, to inform the process of scaling up CDD within countries. Make poverty and social review benchmarks more transparent. This year, different sets of benchmarks were applied depending on whether the projects were poverty focused, inclusive, or enabling. PREM/SDV should disseminate these benchmarks more widely to the regions and networks so that they are systematically applied by task teams during project design and preparation.

5 Angola, Economic Management and Technical Assistance, p. 3.

33

QEA6 Poverty and Social Aspects: Two Examples of Good Practices

The Vietnam Poverty Reduction Strategy Credit (PRSC) II demonstrates good practice in translating findings from poverty diagnosis and analyses into practical, country-based poverty reduction programs. Panel Members noted the strong country ownership and the consultative and inclusive process throughout project preparation and design of activities. These were despite difficulties in the “negotiation environment” in the country, including “multiple donors with differing agendas.” The project’s design was drawn from upstream poverty diagnosis, which used a highly participatory approach, and closely aligned with the country’s PRSP (CPRGS). Programs were designed based on results of poverty diagnosis and on analyses of poverty and social impacts (PSIA) of policy reforms in the agricultural and rural sectors. In the future, Panel Members recommend building on these programs and expanding work on corruption, public expenditure reforms, and increasing the pace of SOE reforms. The Kenya Arid Lands II project contains elements of good practice in implementation of the CDD approach within a region of chronic drought, and a project design that reflects sensitivity and responsiveness to gender issues through inclusion of activities that empower women and monitor gender impacts. Panel Members “welcomed (the CDD project) as it gives more attention to implementation arrangements than is sometimes seen in operations emphasizing community empowerment and decentralization alone. The project components clearly address issues related to income generation and improvement of community assets and public good services.” Design of activities was also based on the Country Gender Assessment for Kenya, which showed the growing importance of female -headed households. Gender mainstreaming is a priority program which the task team translated into specific activities like recruitment of agricultural extension staff and gender sensitivity training programs, which would feed into mechanisms for monitoring gender impacts. In order to do such monitoring, the project convinced the government to appoint a gender focal point in the project’s implementing unit.

34

ANNEX 7

ENVIRONMENTAL ASPECTS Overview A team of Environment Specialists assisted QAE 6 panels assess the quality of the Bank’s integration of environmental management and planning requirements in the design of 85 sampled projects. Environmental safeguard and other environmental protection issues were found to be relevant in 63 projects – 52 investment and 11 adjustments. This annex summarizes key findings, analyzes the adequacy of the treatment of environmental issues and offers recommendations to improve perfor mance. reas in which environmental performance were assessed included:

Ø Assessment of environmental impacts and risks; Ø Adequacy of arrangements for mitigating, managing and monitoring adverse

environmental impacts during project implementation and operation; Ø Adequacy of assessment of environmental policies and institutional capacity for

carrying out the EA and ensuring subsequent compliance; Ø Appropriateness and quality of the EA; Ø Compliance with the Bank’s environmental safeguard policies; and, Ø Implementa tion arrangements and risk assessment and management.

Sample Profile Table 1 shows the distribution of the environmental impact categories of the QAE 6 sample. Some 14 category C projects and 8 uncategorized projects ranging from public sector governance to technical assistance, posed no significant environmental impacts relevant to project preparation and implementation. The remaining 63 projects were reviewed and the quality of the Bank’s treatment of environmental issues was rated on the basis of compliance with the Bank’s environmental safeguard policies and in effective arrangements for mitigation and other protective measures. These projects covered energy, infrastructure, rural development and health and education sectors. In contrast to previous years, there was a preponderance of community-driven development and adjustment operations.

Table 1: QAE 6 Sample

Environmental Impact Category No. of Projects

Environmental Aspects No. rated % HS/S

A 7 7 100

B 38 38 89

C* 22 7 100 FI 4 4 100

U* 14 7 87 Total 85 63 *23 structural adjustment operations were distributed between

Category C and Unrated projects.

35

Key Findings Overall, projects rated 89% highly satisfactory or satisfactory (HS/S) on environmental aspects compared to 87% in QAE 5 and 90% or above in other QAE exercises. Such ratings were relatively lower for projects in EAP, LCR, and MNA; the remaining Regions rated 90% or above (AFR 96%, ECA 90%, and SAR 100%). HS/S ratings by network were highest in PSI (95%), the others ranging from 50 % (FSE) to 92% (PREM). Investment Projects Areas rated highly satisfactory or satisfactory above the 90% level include: assessment of environmental impacts and level of risk (93%), adequacy of assessment of in-country environmental policies and institutional capacity for conducting the EA (96%), quality of the EA report (92%) and compliance with the Bank’s Safeguard Policy on EA (93%). Areas lapsing include: arrangements for mitigating and monitoring adverse environmental impacts (89%), ade quate reflection of EA results in project documentation (81%), appropriateness of implementation arrangements for monitoring (88%) and for evaluating impacts and measuring outcomes (83%) and the adequacy of strategies for dealing with risk during project implementation (86%). Environmental safeguard policies were triggered a total of 63 times, with 51 projects triggering OP 4.01 Environment Assessment and seven triggering one or more other policy (mostly OP 4.04 Natural Habitats and OP 4.36 Forestry). Three instances of marginal compliance with environmental safeguard policies were recorded in the case of OP 4.01 Environmental Assessment (two in EAP and one in LCR). All other policies were considered to be applied satisfactorily. Failure to involve Environmental Specialists or Safeguards Units during preparation occurred three times. There were two instances of marginal mis-categorization. It was felt that a B categorization of two FI projects would have been more appropriate to tackle the treatment of sub-projects unknown at the time of appraisal. As in previous QAG reviews, a significant number of Task Teams reported that they had insufficient financial resources to use Environment Specialists during supervision; in some other cases, Bank Environment Specialists were unavailable to join missions because of conflicting demands.

Adjustment Projects

Of such projects, 80% were rated highly satisfactory or satisfactory in the implementation of OD 8.60 Structural Adjustment. Ratings were lowest for evaluating impacts and measuring outcomes (62%), risk assessment (74%) and monitoring (74%), including use of indicators and benchmarks. Arrangements for mitigation measures were rated 86% satisfactory. Detailed Analysis Investment Projects Identification and assessment of environmental impacts saw an improvement over QEA 5 results; this is mainly with respect to a comparison of alternatives and consultation -- perhaps a reflection the increasing role played by the Regional Safeguards Units in project processing

36

along with the current preponderance of CDDs, SFs, etc., stipulating clear guidelines/procedures to review alternatives and guide appropriate implementation. The high level of assessment of in-country policies and capacity (96%) and consultation with affected groups (94%) may also find similar explanation. The same cannot be said of arrangements for the management of adverse environmental impacts (mitigation and monitoring) during project implementation, with a particular lapse in the quality of EMPs (87%) -- a continuing cause for concern. Consequently, it is not surprising that implementation arrangements were rated 89% satisfactory with the evaluation of impacts and outcomes dropping to 83%. A more encouraging finding is that assessment of environmental policies and institutional capacity to undertake EA attained a level of 96%, up 4% from QAE 5. The poor quality of some EMPs invariably resulted in inadequate attention to the implementation of environmental protection requirements in PIPs or Operational Manuals. These provide on-the-ground guidance and are critical to effective monitoring and supervision. The lack of rigor of many EMPs may also explain why the application of environmental indicators/benchmarks was rated 86% and arrangements for evaluating impacts and measuring outcomes rated 83%. Assessment and the handling of risk was rated 86% satisfactory. In some instances, geographic proximity to other development projects and possible cumulative impacts was not assessed. This was particularly surprising in the case of some water resources management and rural development projects involving extensive watersheds. The result was lost opportunities for integration/synergy to attain environmentally sustainable development; in some cases, there is concern about the lack of attention to adverse impacts being generated by the cumulative impacts. There were proportionately fewer cases of marginal compliance with the environmental safeguard policies than under QEA 5 and they were confined to OP 4.01 Environmental Assessment. A project supporting water resources management and irrigation left no record of consultations and disclosure and failed to provide relevant documents before appraisal. Other projects, involving health and education, did not produce an adequate EMP and sufficiently detailed arrangements for implementation of environmental protection measures. Adjustment Projects The OD 8.60 on Structural Adjustment and an Operational Memorandum (dated June 5, 2000) define approaches to addressing environmental issues. However, guidance is very general and much is left to the discretion of the Task Team may explain the relatively poor treatment of environmental issues in the QAG sample. In some instances, despite satisfactory EAs or audits, no arrangements were developed to implement agreed recommendations, monitor/supervise actions to be taken or support linkage with Bank-funded projects in the same sector or sphere of influence. LCR is aware of lack of guidance and has provided the clearest and most constructive instruction to Task Teams and follow up support.

37

Issues Requiring attention and Proposed Actions Quality of EMPs The variable quality of EMPs continues to pose risk to the effective implementation of environmental planning and management requirements. In some instances, the projects in the QAG sample benefited little from diligent environmental assessment of impacts and the provision of mitigation measures. In particular, there continues to be poor scheduling of the mitigation measures and their monitoring and clear delineation of institutional responsibilities. This was exacerbated by the often-vague guidance provided in PIPs and Operational Manuals -- frequently the only instrument available at the field level giving direction on the implementation of environmental protection measures, e.g. contract clauses for consultants and contractors. Nonetheless, projects in the road sector can claim progress with incorporating more effective provisions in EMPs and PIPs and Operational Manuals backed by pragmatic arrangements for institutional strengthening and training. In some cases, sectoral reforms were achieved the application of specially developed environmental guidelines and a rigorous training program. Recommended Actions

Ø Regions and the Environment Department should continue to train Bank and Borrower operations staff in the preparation and implementation of EMPs, including use of good practice examples.

Ø The Environment Department should complete the draft EA Update on EMPs and the Regions should pool examples of good practice.

Ø Regional Environment Specialists should be more diligent in ensuring that environmental management requirements are satisfactorily reflected in PIPs and Operational Manuals. To support Task Teams, they might also develop guidance on the inclusion of environmental protection measures in the preparation and implementation of such Manuals.

Implementation Arrangements Arrangements to monitor implementation and review progress with the Borrower were addressed insufficiently in many EAs and EMPs. This left projects without means or indeed need to report anything at supervision. Similarly, means to coordinate or link environmental requirements under the project with other sectoral agencies and the environmental agency were at to the discretion of the government, usually the Project Implementation Unit – a situation likely to produce mixed results, especially where mitigation measures have to be carefully phased with civil works and where cumulative impacts are anticipated. Training was sometimes introduced in an almost cavalier fashion with the expectation that it would bring capability up to the required standards, at least for the life of the project.

38

Recommended Actions

Ø M&E mechanisms and institutional organization should be better developed during project preparation to monitor and evaluate the effectiveness of environmental mitigation and other protective measures.

Ø Task Teams and their Environment Specialists should consider the value of independent monitoring and spot checks to assure due diligence at the field level.

Ø Coordination or linkage with other projects should be better assessed and means to optimize integration specified clearly in project documents.

Ø Training programs should be more carefully designed and targeted to maximize gains beyond the life of the project.

Evaluation Impacts and Measuring Outcomes

Use of indicators and benchmarks to monitor progress remain very general in project documents leaving supervision missions with no effective mechanism to quantify progress or identify trends (positive and negative) during implementation. In some instances, this posed risk to successful implementation of environmental protection measures, including measuring the effectiveness of institutional arrangements. Recommended Actions

Ø Task Teams would benefit from assistance in the design and use of environmental performance indicators to assess and evaluate the effectiveness of mitigation and other protective measures in the performance of Bank-funded projects. Notes produced by the Environment Department are a valuable first step and should be elaborated, especially regarding changes in environmental quality and service delivery and efficiency of sustainable resources use.

Cumulative/Indirect Impacts A number of projects gave inadequate regard to indirect environmental impacts, especially in rural investments. This will require rectification during implementation, perhaps at additional expense. In both investment and structural adjustment projects, development activities often crowded into an area thickly populated with Bank or other donor-funded projects. Some Task Teams recognized the value of synergy and introduced mechanisms to assure complementarity, if only for succeeding projects. Other Teams, however, missed such valuable opportunities, potentially eroding the sustainability of the project or neighboring projects. Recommended Actions

Ø Environment Specialists should be more diligent in helping Task Teams assess the need to coordinate/link other geographically proximal Bank or donor -assisted projects having similar objectives to gain efficiencies in dealing with systemic

39

issues and enhance sustainable development. The possibility of generating indirect impacts should be similarly carefully assessed.

Ø Training sessions on safeguards and other environmental management issues might include treatment of cumulative/indirect impacts.

Adjustment Projects Attention to these projects is a clear priority given that they were the least satisfactorily rated. The new OP/BP Development Policy Lending requires determination of whether specific policies supported by an operation pose significant environmental impacts and an assessment of the government’s approach to adequately mitigate any adverse impacts. Such impacts may be addressed within the operation or in parallel. However, the OP/BP gives no additional guidance to Task Teams. Recommended Action Regions and Environment Department should assist Task Teams apply the new OP/BP effectively in dealing with environmental issues by developing a set of pragmatic guidelines on opportunities and mechanisms to better integrate environmental planning and management needs, e.g., building upon national assessments to define systemic opportunities for policy/regulatory reform and linkage with currently funded projects to gain synergy and enhance a country’s environmental management capacity. Follow-up training sessions should be considered.

ANNEX 8

FINANCIAL MANAGEMENT ASPECTS

OVERVIEW The FM reviewers evaluated the quality of project financial management arrangements and the extent to which project documents provide information sufficient to reach conclusions on the appropriateness of the proposed FM arrangements. Standard questionnaires were employed for the purposes of the reviews, largely unchanged from QEA5. As in QEA5, 92% of the 85 projects in the sample were rated Satisfactory. No project was rated Unsatisfactory. The results reflect the fact that FM is now mainstreamed within Bank operations, and the relevant FM Guidelines which have being issued are widely understood and used by FM staff. Better guidelines and increased awareness of FM standards, however, means that the bar on what constitutes “good practice” has been raised – only 15% of operations were rated as Highly Satisfactory compared with 32% in QEA5.

TABLE 1. OVERALL RATINGS

% Satisfactory QEA2 QEA3 QEA4 QEA5 QEA6

77 89 97 92 92 QEA6 sample consisted of 66 investment and 19 adjustment operations and separate questionnaires were used for each type of operation. Reflecting the fact that clear guidance is an important issue for investment operations, the results for QEA6 show that FM performance was satisfactory in 93% of investment operations compared to 89% for adjustment operations, where absence of clear guidelines may have led to less consistency in presentation.

TABLE 2. FM RATINGS BY TYPE OF OPERATION

Number of Projects Type of Operation

Highly Satisfactory Satisfactory

Marginally Satisfactory Unsatisfactory Total

Investment 12 49 5 0 66 Adjustment 1 16 2 0 19 One region, ECA, reported 100% satisfactory performance. This reflects the region’s highly structured approach to quality control including a review of all assignments by designated senior FM staff, and the use of mandatory risk analysis procedures. While this sets a baseline of acceptable standards, experienced staff are able to leverage this baseline in arriving at creative and good practice approaches to innovative project design.

TABLE 3. RATINGS BY REGION

% Satisfactory or Better AFR EAP ECA LCR MNA SAR 92 82 100 94 83 89

41

Overall, while issues raised in QEA5 such as early involvement of FMSs in project preparation and better supervision by FMSs of consultants have been largely resolved, other issues raised, such as the appropriate exercise of professional judgement particularly relating to issues of fiduciary risk and incorporation of pertinent knowledge obtained from fiduciary ESW into adjustment lending operations, continue to require attention. Quality At Entry – Investment Operations The FM reviewers addressed the specific issues laid out in Table 4 in arriving at conclusions in respect of investment operations. Highly Satisfactory ratings were characterized by the presence of one or more ingredients the combination of which are more likely to result in FM arrangements which are responsive to project design, country environment and assessed risks. These ingredients are a robust risk analysis; an advanced state of preparedness as measured by few, if any, FM effectiveness conditions; evidence of incorporation of lessons learned from previous projects in the portfolio; FM capacity strengthening components built into the overall attainment of project development objectives; close and early collaboration between the FMS and the task team; and incorporation of the findings of relevant ESW. Marginally Satisfactory ratings reflect the fact that the proposed arrangements were either not likely to be in place by effectiveness or were likely to re quire extensive interventions during implementation. Marginally satisfactory ratings thus arose in connection with limited involvement by FMSs; weak risk analysis and therefore uncertain risk mitigating measures; and extensive conditions of effectiveness. Operations which adhered to the FMA Guidelines and that featured adequate disclosure in the project documents were rated Satisfactory.

TABLE 4 . FM SUPPLEMENTARY QUESTIO NS

Investment Operations

Question Issue Yes No NA Total 1) Adequate description of FM arrangements in the PIP 57 9 - 66 5.1 (a)

2) Adequacy of the arrangements 58 7 1* 66 1) FM Assessment by FMS 64 2 - 66 2) Conclusion on FM adequacy 62 4 - 66 3) Time-bound Action plan 59 7 - 66 4) Action Plan carried out 55 11 - 66 5) FM Risk Analysis in the PAD 54 12 - 66 6) Appropriate Risk mitigation strategy 57 9 - 66 7) Appropriate FM analysis and description in PAD 56 10 - 66 8) FM Assessment Guidelines followed 63 3 - 66

5.2 (b)

* The FM reviewer was unable to arrive at a conclusion on the adequacy of the proposed FM arrangements.

Project Implementation Plans (PIP). Recognizing that the PIP is no longer the main vehicle for documenting FM arrangements, QEA6 projects were evaluated on the adequacy of FM arrangements as set out in the PIP, Operations Manual (OM), or the Financial Management Manual (FMM) that described the implementing agency’s FM arrangements and their role in project management and implementation. The PIP/OM/FMM for 9 projects did not adequately describe the project financial

42

arrangements and consequently for 6 of these projects the arrangements were considered inadequate. FM Assessment. OP/BP 10.02 requires the FM arrangements of a project and the implementing agencies be assessed no later than at appraisal. The lack of consistency in the assessment process and documentation of FM arrangements prior to effectiveness that were observed in previous years has largely been addressed by the Guidelines on Assessment of the Financial Management Arrangements in Bank-financed Projects, issued by the FM board in July 2001. The Guidelines were followed in 95% of the cases reviewed, and good FM Assessment based on structured risk analysis, together with an appreciation and creative use of the capacity building po tential of FM systems, were the main features observed in those projects rated Highly Satisfactory overall. Disclosure in Project Documents. An important part of the review concerned the extent to which project documents provide information to arrive at conclusions on the appropriateness of the proposed FM arrangements. The principal document summarizing the project’s FM arrangements is the PAD. The FM arrangements may also figure in the project risk analysis and in the main loan conditions. Inadequate risk analysis in the PAD was the largest single issue in the cases reviewed (12 out of 66). Inadequate FM analysis and description was found in 10 out of 66 projects. In some cases this was a failure to reflect an otherwise satisfactory FM assessment in the PAD; in others, FM issues appeared to be crowded out by other considerations; and in a few cases the inadequate PAD description reflected marginally satisfactory FM analysis. Readiness (Action Plans, Conditions of Effectiveness). For projects with shortcomings in their FM arrangements, Bank policies require remedial time -bound actions plans to be prepared and agreed with the Borrower. Out of the 66 investment projects, 59 had action plans in place by negotiations, 55 of which were carried out in the appropriate time -frame. The review therefore shows that those action plans that were contemplated were mostly implemented in an adequate timeframe and suggests better realism in the approach to what constitutes an adequate FM system at entry. Risk Analysis. The issuance of the FM Assessment Guidelines substantially improved the quality of FM risk analysis as recorded in the PAD. Eighty-two percent of the projects identified the particular FM issues and risks and the measures proposed to mitigate them. The review shows the correlation between a good risk assessment and the overall quality of FM aspects in the rest of the areas evaluated; all projects that were considered to have an appropriate risk analysis were rated Satisfactory or better. The converse is also true. Where projects were rated Marginally Satisfactory, there tended to be a lack of specificity defining fiduciary risks and mitigating measures. Quality At Entry – Adjustment Operations The FM reviewers addressed the specific issues laid out in Table 5 in arriving at conclusions in respect of adjustment operations. In conducting these reviews, a higher degree of subjectivity had to be exercised as, while there is emerging consensus on the fiduciary approach to adjustment operations, clear guidance is yet to be issued to staff in this respect. Accordingly, in the context of the specific circumstances where a reasonable approach had been adopted, operations were rated Satisfactory. One operation was rated Highly Satisfactory (Uganda: PRSC 2) and was characterized by an underpinning of key fiduciary issues identified in relevant ESW; upstream involvement of the FMS evidenced by FM input into the policy matrix; specific discussion of risks attaching to the use of

43

Bank funds; and good overall disclosure of fiduciary issues in the program documents. Marginally Satisfactory ratings on the other hand were characterized by late involvement of the FMS and limited FM input on conditionality; the lack of a clear linkage between ESW and the fiduciary disclosures in the program documents; little or no evidence of risk assessment; and a “cookie cutter” approach to the disclosure of proposed FM arrangements.

TABLE 5 . FM SUPPLEMENTARY QUESTIO NS

Adjustment Operations

Question Issue Yes No NA Total 1) Funds flow and fiduciary risks identified 16 3 - 19 2) Rationale for the decision to undertake or not an audit

12 7 - 19

5.1 (a)

3) Right to audit included in documentation 19 - - 19 5.2 (b) 1) CFAA/ ROSC and /or IMF Safeguards analysis

factored into the operation 13 6 - 19

Audit and Disbursement Requirements. These are standardized in the model legal language and compliance was achieved in all cases. However, in terms of proactively evaluating the basic requirements against the specific context of the operation, there was less success. Approximately 40% of the cases did not provide a rationale for the arrangements proposed. In cases where the reviewer thought that the rationale for the decision about the performance of an audit was not appropr iate, disclosure in the project documents was limited to a statement not requiring the audit of the deposit account but reserving the right to do so at any time. This result is consistent with the fact that in only approximately the same number of cases were there considerations of ESW findings in either a fiduciary risk assessment or in the policy development aspects of the program. Bank’s Fiduciary Concerns with the Use of Overall Borrower Resources. The reviewers evaluated whether the strengths and weaknesses identified in the CFAA, ROSC and IMF Safeguards Assessments had been taken into consideration when assessing the fiduciary issues under SALs. There were 6 projects in which the findings from the ESW reports were not included in the analysis, although in 4 of those cases the CFAA was underway or scheduled for FY04. The requirement that a CFAA be undertaken prior to embarking on a PRSC was observed. The extent to which the fiduciary concerns regarding the overall use of borrower resources was addressed was satisfactory in 84% of the projects. Risk Analysis. The quality of the risk analysis was largely dependent upon the extent of ESW undertaken. Where a CFAA had been completed with country FM risk analysis, it was incorporated into the operation. During the period covered by QEA6, many CFAAs were still in progress, or not yet started, in which case, the country risk analysis was less detailed. While the overall picture regarding risk analysis was satisfactory, in the three cases where fiducia ry risk analysis was rated less than satisfactory, completed or ongoing ESW was not incorporated in the document. Conclusions and Recommendations Risk. Though the quality of FM risk assessments has improved since the issuance of the FM Assessment Guidelines, considerable variations in the quality of risk analysis was observed. There is scope for further improvement, particularly in:

44

• the linkage of the mitigating action plan to the risk assessment; actions are

sometimes not related to the risks identified, there being a tendency to recommend standardised action plans regardless of type and levels of risk; and

• the realism of the capacity assessments of implementing institutions; there is a tendency to pile on TA and training programs in the hope that these will overcome what are sometimes more fundamental underlying defects.

ESW. While the quality and volume of ESW activities and products has multiplied, particularly in support of adjustment operations, there is a need for better integration of ESW into the FM assessment for both investment and adjustment operations. There is little evidence of a structured approach to country level solutions to FM issues in any of the documentation, particularly where FM capacity building is an important aspect of the loan. FMA Guidelines. The Guidelines issued by the Sector Board have contributed to standardizing the FM assessments and improving the overall quality and presentation of FMS work. For investment projects, they were followed in 95% of the cases. In Adjustment operations, the lack of clear guidance probably resulted in a less consistent approach. Role of the FMS in adjustment operations is less clearly perceived by TTLs than in investment projects and the incorporation of financial management ESW is still not universal. Documentation. The need to improve FM documentation and the standard of disclosure in official documentation is still an issue. The review continues to show uneven PAD presentation and mixed performance on FM manuals and PIPs. FM involvement at an early stage. QEA6 confirmed that FMSs add more value when they are involved at an early stage of project preparation. The QEA6 review shows that FMS involvement has been firmly established, particularly in investment projects, and that in only 2 cases was there late involvement. In both of these adjustment credits the lack of FMS involvement resulted in a less than satisfactory FM rating. Early involvement of FMS needs continuing emphasis. Emergency Operations. The FM assessments for emergency operations remain an issue, with perhaps the absence of guidance being a factor affecting quality. In general, in order to compensate for capacity constraints, FM assessments tended to propose an unrealistic remedial action plan to be implemented by effectiveness or soon thereafter. In other words, despite the emergency nature of the operation, staff are not calibrating the acceptable level of FM risk. Instead, where it is obvious that there is limited capacity on the ground, attention should ha ve focused on agreeing short term arrangements, such as buying in capacity while providing a rationale for adopting this approach. Financial Monitoring Reports. While there was a high level of compliance with the requirement to agree on the format of FMRs at negotiations, the detailed formats and instructions on how to prepare them, were not always included in the FM manuals.

45

ANNEX 9

PROCUREMENT ASPECTS

The assessment was carried out by specialized procurement reviewers. As in previous QAG assessments, the main objective was to evaluate the quality of compliance with the Bank's procurement policies, which are designed to ensure that acceptable capacity exists and appropriate procedures and thresholds are in place to implement procurement under Bank-financed projects. The quality of the procurement input was also examined to determine if it supported the overall objectives of the project and readiness for implementation. The reviewers conducted in-depth assessments of various aspects of preparation related to procurement including quality and adequacy of: (i) the capacity of implementing agency(ies); (ii) the procurement risks and measures taken to mitigate them; and (iii) quality and status of the procurement pla n for the first year implementation for Investment Projects. For Adjustment Operations, the assessment was whether the documentation and conditionality adequately addressed the procurement concerns noted in the CPAR or other information available on the performance of the public procurement system. Out of the 85 projects in the sample, 66 were Investment Projects and 19 were Adjustment Operations. Results for the QEA6 sample indicate that 94% of the projects benefited from satisfactory preparation of procurement aspects. These results are similar to QEA5.

TABLE 1: RATINGS BY REGION FOR PROCUREMENT QUALITY AT ENTRY

Note: Out of the sample of 85 projects, there was 1 project without any procurement.

Region No. of

Projects HS S HS or S M U

AFR 26 Projects

19% 5 Projects

73% 19 Projects

92% 24 Projects

8% 2 Projects

0%

EAP 11

Projects 27%

3 Projects 73%

8 Projects 100%

11 Projects 0%

0%

ECA 16

Projects 19%

3 Project 75%

12 Projects 94%

15 Projects 6%

1 Projects 0%

LCR 16

Projects 0%

100%

16 Projects 100%

16 Projects 0%

0%

MNA

6 Projects 0%

83%

5 Projects 83%

5 Projects 17%

1 Project 0%

SAR

9 Projects 11%

1 Project 78%

7 Projects 89%

8 Projects 11%

1 Project 0%

Bank-wide 84

Projects 14%

12 Project 81%

67 Projects 94%

79 Projects 6%

5Projects 0%

46

TABLE 2: RATINGS BY INVESTMENT/ADJUSTMENT PROCUREMENT ASPECTS

Note: Out of the sample of 85 projects, there was 1 project without any procurement. Key Findings

• 94% of the projects, (95% for investment and 89% for adjustment operations) had

satisfactory procurement preparation (compared with 95% in QEA5).

• 14% of the projects were rated highly satisfactory, an improvement from 8% in QEA5.

• 94% of the projects had satisfactory rating for assessment of procurement risks that may impact the project.

• 94% of the projects had satisfactory rating for of quality of documents, including Annex 6 to the PAD.

• 89% of the projects had satisfactory rating for quality and status of the procurement plan for the first year implementation6.

• Results indicated that there is no visible relationship between sectors and the quality of procurement preparation and therefore the results are not provided by sectors. However, as procurement is managed regionally in the Bank, regional breakdowns are given.

Strengths

The increase (14% of all projects i.e. 12 projects, including one adjustment operation) in highly satisfactory projects are due to the quality improvement in Capacity Assessment, Capacity Building plan, Procurement Plan, and Project Operational/Implementation Manual and involvement of procurement staff early in project preparation by the Borrower. The following were observed in some of the well prepared projects which were rated highly satisfactory.

• Capacity Assessment: Very comprehensive and detailed capacity assessment which covers all aspects associated to procurement capacity, organization, process and legal aspects. Proposed action plan is comprehensive and satisfactory. The discrepancies between the local procurement law and Bank’s guidelines are duly addressed through a side letter, included as part of the Loan Agreement.

6 This was calculated for 62 projects, as out of the 66 Investment Projects there were 4 projects with CDD for which procurement plans cannot be prepared in advance and therefore excluded.

Instrument No. of

Projects HS S HS or S M U

Investment 66 Projects

17% 11 Projects

79% 52 Projects

95% 63 Projects

5% 3 Projects

0%

Adjustment

18 Projects 6%

1 Project 83%

15 Projects 89%

16 Projects 11%

2 Project 0%

Bank-wide

84 Projects 14%

12Project 81%

67 Projects 94%

79 Projects 6%

5Projects 0%

47

• Capacity Building : Many projects include a community-based participation component, if not full community based projects. Community Partic ipation is usually well integrated in the national institutional framework (central, provincial and local) based on decentralization and focused on community participation. An effective capacity building tool for the community is the comprehensive project implementation training program, including procurement, for training of trainers for the Provincial technical staff early in project implementation. Procurement capacity building action plan includes TOR and draft contract for procurement specialist to be hired.

• Procurement Plan: The procurement plan for the project is quite comprehensive.

Procurement Plan is well integrated with Project Implementation schedule and Project monitoring system established for quarterly procurement reports which requires update of procurement plan every quarter and procurement audits. In projects with demand-driven community sub-projects for which procurement plan cannot be provided, the documents provide adequate information on the procurement environment, institutional arrangements and capacity building, detailed Procurement Procedures Manual. Bank staff are increasingly aware of the availability of the Bank Manual on Community Participation, and of the simplified procedures that can be developed for communitie s.

• Project Operational/Implementation Manual: Procurement Manual is

comprehensive and of high quality with complete, detailed, clear and adequate information with handbooks for Community-Driven Program. This facilitates implementation.

• Early involvement of procurement staff on the Borrower’s side : In one

infrastructure project the procurement engineer was hired during project preparation and the procurement documents were complete and ready at the time of Board approval. In another project given the emergency nature of the project in a post conflict country, the project team prudently assessed the risk as high and requested the Borrower to engage specialized procurement consultants at an early stage, and the progress achieved was remarkable with about 65% of contracts awarded by credit effectiveness. Hiring of procurement staff is increasingly a condition of either negotiations or Board approval.

Weaknesses Less than satisfactory Investment Projects (5% of projects i.e. 3 projects): Generally these projects had poor and inadequate capacity assessment; did not provide any action plan to address capacity and risk issues; and had poorly prepared and incomplete implementation manual. Less than satisfactory Adjustment projects (11% of projects i.e. 2 projects): One of the two projects did not address procurement at all. This is particula rly important as Bank funds provided under adjustment operations are subject to the financial management, procurement and other control systems that exist in the country. If these systems are not functioning, Bank funds are at risk and appropriate steps need to be taken to address the risks.

48

Issues Requiring Further Attention. (a) Procurement Risk: Even though the assessment of procurement risks that may impact the project had an overall highly satisfactory rating, the current risk analysis categorizes projects into high, medium and low risk categories, and mitigation of the risk is addressed by (i) shifting the balance between prior and post reviews, (ii) increased focus on post reviews and procurement audits, (iii) higher frequency of Bank supervision, (iv) monitoring through quarterly FMRs that include a Status of Procurement, and (v) in some cases, hiring international procurement consultants. These risk mitigation arrangements have resource implications for the Bank and the Borrower and could pose problems during implementation. Therefore, the method of assessing of procurement risks and the mitigation measures needs to be addressed in a comprehensive manner to optimize the resource allocation to match the risks and to achieve maximum benefit in project implementation. (b) Capacity Building : Provide incentives for Country Directors to support procurement specialists, decentralized in Country offices to help the Government to create a cadre of trained (by our staff in those Country offices) trainers to increase the capacity of procurement staff in the Borrower's agencies.

49

ANNEX 10

SUMMARY RESULTS OF TASK TEAM LEADERS, PANELISTS

AND OBSERVERS’ SURVEYS

1. Following completion of the QEA6 panel interviews, Task Team Leaders (TTLs), Panelists as well as Observers were asked to participate in a survey to provide feedback to QAG on QEA6. Of the 85 TTLs, 182 Panelists and 43 observers who participated, 57 TTLs (67%), 145 panelists (80%) and 33 observers (77%) responded to the survey.

A. TTL Feedback on Project Preparation

Additional Project Preparation Budget 2. The TTLs were asked to assume the availability of an additional $25,000 to prepare their projects and to distribute these resources among a number of priority items listed in the survey questionnaire. TTLs gave highest priority to the hiring of additional technical inputs (44% of the additional budget was allocated for this purpose) and for more frequent missions and/or longer field visits (28%). Under QEA5, more frequent missions and/or longer field visits was the first priority and the hiring of additional technical inputs was a second priority. In QEA4, however, the priorities were the same as in QEA6. The QEA6, QEA5 and QEA4 results concerning additional amounts allocated to ensuring compliance with Bank’s fiduciary and safeguard policies are very similar.

3. The feedback indicates that TTLs consider funding for doing the Bank’s fiduciary and safeguard compliance related work to be generally adequate. Should additional resources be made available, TTLs would use them primarily for traditional technical analysis work and spend more time in the field to improve sector dialog and client relations.

Staff Time Input for QEA6

4. The TTLs report that, on average, they and their task team members have spent 40 hours preparing for and participating in the QEA6 process. This average represents a slight decrease reported under the QEA5 TTL survey (42 hours).

5. Staff time input for QAG assessments remains a non-funded mandate for which TTLs are not being provided with additional budgets. The issue of non-funded QAG assessments remains perhaps the single most important irritant vis-à-vis TTLs’ attitude towards QAG reviews.

Additional Feedback

6. The TTL’s satisfactory or better responses to all other QEA6 survey questions are presented in the table below and compared to QEA5 responses.

50

N = Number of respondents

7. The above table shows that there is a generally improving trend in almost all the areas where TTL opinion was sought. Specific findings are that:

• TTL satisfaction with the quality and timeliness of management support, from both country department and sector, is high and improving since QEA5.

• TTL satisfaction with different aspects of Bank IT systems is growing since

QEA4, with two out of three TTLs now satisfied with SAP and PDS;

• Satisfaction with the support received in project preparation from Country Field Offices, Legal, FM, Social Development and Procurement remains high

QEA5 QEA6 % N % N Management Support

• Sector Manager’s guidance & support 83 36 93 55

• Timely decisions by Sector Manager 90 29 96 50

• Sufficient attention given by the Country Department 89 36 96 57

Support from Bank IT

• SAP 59 34 70 54

• PDS 56 32 63 52

• IRIS 90 33 86 51

• BW 77 13 83 24

• Other 80 5 100 1

Other Support

• Usefulness of PDS as a documentation tool 53 34 58 53

• Support by the country field office 89 36 98 54

• Support from Legal 97 31 92 52

• Support from Procurement 69 32 88 52

• Support from Environment 61 31 67 52

• Support from Social Development 71 31 81 52

• Support from Financial Management 87 31 96 45

• Impact of decentralization on project preparation quality 61 31 73 51

Factors with negative influence on quality

• Management changes 16 31 6 47

• Internal organizational changes 26 31 4 47

• TM changes 10 30 7 46

• Team Member changes 13 30 11 46

• Inappropriate skill mix 10 30 11 46

• Budget constraints 26 31 17 46

• Reliance on non-Bank sources for core team skills 23 30 9 47

• Complications arising from donor coordination 27 30 32 47

51

and improved from QEA5; support received from Environment, while improving, remained low.

• While budget constraints are decreasing since QEA4, the most important factor negatively influencing project preparation was with complications arising from donor coordination (32%), the same as in QEA5. Changes in team members and inappropriate skill-mix also had some negative influence.

B. TTL, Observer and Panelist Feedback on QAG’s Performance 8. QAG’s performance was assessed by TTLs, Panelists and Observers in five main aspects: i) Assessment panel; ii) Assessment approach; iii) QEA process; iv) Learning; and v) Overall performance. The first three main aspects include several sub-questions. The table below presents satisfactory or better ratings given by TTLs, Panelists and Observers.

TTL Panelists Observer % N % N % N 1. Assessment Panel

• Panel skill mix • Panel Familiarity with country specific issues • Panel fairness and professionalism

NR 75 55 77

NR 57 55 57

97 99 90 NR

90 145 145 NA

100 100 90 NR

26 33 32 NA

2. Assessment Approach

• Panel focused on relevant issues • Were there major omissions • Panel’s preparation • Appropriateness of the time allotted for the interview • Individual panelists views taken into account when rating the

project

NR 80 70 79 95 NR

NR 56 46 57 57 NA

96 NR NR 94 NR 97

89 NA NA 145 NA 145

100 97 91 97 94 NR

27 33 31 33 33 NA

3. Assessment Process

• Fairness and transparency • Panel listening to the Task Team • Task team listening to the panel • Role of the moderator • Quality of QAG administrative support • QAG burden on the TT • Value added by the Interview

NR NR NR NR NR 98 70 68

NA NA NA NA NA 56 24 56

99 98 98 90 99 99 NR NR

79 145 145 142 110 145 NA NA

100 97 90 100 100 NR NR NR

24 31 30 30 31 NA NA NA

4. Learning 75 55 94 84 96 26

5. Overall QAG Performance 75 55 99 141 93 30

NR = Not Rated; NA = Not Applicable; N = Number of respondents

52

9. The responses provide valuable insights into the perceptions concerning QAG’s performance. Overall, the following observations can be made:

• QAG’s overall performance received an overall 75% satisfactory rating from

TTLs and a 99% and 93% satisfactory rating from panelists and observers, respectively, which is about the same as in QEA5;

• Satisfactory ratings given by the TTLs are almost all significantly lower than ratings from the panelists and the observers;

• TTLs, on the whole, have a somewhat more critical opinion of QAG panels in terms of their skills-mix and, fairness and professionalism.

• TTLs also continue to have reservations about Panels’ familiarity with country specific issues and the burden imposed by QAG reviews. However, TTL perception regarding country knowledge of panels is not corroborated by checking actual country experience of panels members of each of the assessments.

53

ANNEX 11

METHODOLOGY FOR SAMPLING, STRATIFICATION AND WEIGHTING The QEA6 universe consisted of the 260 IDA, IBRD, Special Funds, and GEF operations approved during FY03 as follows: IDA tasks over US$50 million: 47 tasks IDA tasks between US$20 and US$50 million: 51 tasks IDA tasks under US$20 million: 50 tasks IBRD tasks over US$150 million: 29 tasks IBRD tasks between US$50 and US$150 million: 29 tasks IBRD tasks between US$20 and US$50 million: 25 tasks IBRD tasks under US$20 million: 29 tasks The QEA6 sample was stratified by IBRD/IDA and loan size. There were three strata in the case of IDA and four in the case of IBRD, resulting in 7 strata for sampling. The sample size of 85 total tasks allows for statistically robust results at the Bank-wide, IBRD/IDA, and large region level. However, the results are not robust for most instruments, small regions, sectors, and themes. As a check, key variables from the sample were compared against those for the universe as a whole and, in fact, the sample has proved to be robust. The sample was drawn in 5 waves as follows: IDA tasks over US$50 million: 23 tasks IDA tasks between US$20 and US$50 million: 17 tasks IDA tasks under US$20 million: 13 tasks IBRD tasks over US$150 million: 14 tasks IBRD tasks between US$50 and US$150 million: 9 tasks IBRD tasks between US$20 and US$50 million: 6 tasks IBRD tasks under US$20 million: 3 tasks The sample size for each strata was determined for each strata individually, in order to ensure robust results for large tasks and IDA. The approximate sampling fractions were: IDA tasks over US$50 million: 50% (weight 2.043) IDA tasks between US$20 and US$50 million: 33% (weight 3.000) IDA tasks under US$20 million: 25% (weight 3.846) IBRD tasks over US$150 million: 50% (weight 2.071) IBRD tasks between US$50 and US$150 million: 33% (weight 3.222) IBRD tasks between US$20 and US$50 million: 25% (weight 4.167) IBRD tasks under US$20 million: 10% (weight 9.667) In order to correct for these different sampling fractions, the aggregate results are weighted. The weights are computed as a ratio of total projects in the strata divided by the number of projects sampled from that strata. This ensures that each task sampled is weighted as a proportion of the universe it was taken from.

QUALITY AT ENTRY IN FY03...

Documents

Transcript of QUALITY AT ENTRY IN FY03...