North-East Asia Development Cooperation Forum 2018• Session3: Project Level Evaluation• Venue: Aloft Hotel, Seoul• Date: September 15, 2018 9:30-11:45

A Critical Review of the Practices of Development Evaluation in South Korea:

Focusing on KOICA’s ex-post Evaluation (1998-2016)

Hyuk-Sang Sohn

• Dean, Graduate School of Public Policy & Civic Engagement• Executive Director, Center for International Development

Cooperation (CIDEC), Kyung Hee University

Contents

I. History

1. EDCF

2. KOICA

3. ODA IntegratedEvaluation System

II. Analysis of KOICA’s Evaluation Reports

1. Data

2. Analysis

IV. Conclusion

1. Challenges Ahead

2. Lessons

III. Integrated Evaluation System

1. Key Issues

2. Improvement Measures

3. Analysis Results

Korea’s ODA Governance Structure

3

Committee for International Development

- Chair: Prime Minister- Member: Related Office Ministers, Civilian Experts

Working Committee

- Chair: Vice Minister, Prime Minster’s Office- Member: Related Office Directors, Civilian Experts

Ministry of Foreign Affairs

- Supervise grant aid- Multilateral and to the UN and

other international organizations

Ministry of Strategy and Finance

- Supervise concessional loans- Multilateral aid to development banks

KOICA

- Execute grant aid- Execute multilateral aid

EDCF or Korean EXIM Bank

- Operate loan activities

Working levelconsultation

Working relationship

Other Ministries

Working levelconsultation

Sub-Committee forEvaluation

Development Evaluation Experiences as EPM

4

Year Titles Countries for field research Contractor Ordering

Institutions

2010 Comprehensive Evaluation on the NGO-support Program Tanzania & Mongolia KHU CIDEC KOICA(Civil

Society Team)

2014 PPP: Evaluation on Global CSR Program Indonesia KHU CIDEC KOICA

2016 Ex-post Evaluation on Agriculture Business Philippines & Laos

KHU CIDEC KOICA

2016 Ex-post Evaluation on the Project for the Construction of Water Supply System

Vietnam & Senegal KHU CIDEC KOICA

2017-8 IO-NGO Partnership projects: NGOs as Implementing Partner(Good Neighbors)

Bangladesh & Tanzania KHU CIDEC Good Neighbors

Int’l & KHU CIDEC

2017 Humanitarian Assistance(DRR) Projects Vietnam & Myanmar KHU CIDEC Ministry of

Foreign Affairs

2017 Income Generating Project of Good Neighbors in Malawi Malawi KHU CIDEC Good Neighbors Int’l

2018 Comprehensive Evaluation for Community-based MCH Capacity Building Ghana KHU CIDEC KOICA

I. History: EDCF

Founded in July, 1987

• EDCF established

August,1993

• Published “Manual for Ex-post Evaluation on Projects funded by EDCF”

2002

• Implemented Ex-post Evaluation among projects completed over 3 years ago

April, 2007

• Mandatory indicator set-up for performance evaluation

• Introduction of performance based evaluation system

March, 1990: First Amendment

• Suggested specific evaluation periods

1996

• Amended project evaluation period: After completion of the projects’ → After submitting the Project Completion(Final) Report

February, 2005

Standards on Ex-post Evaluation and its mandatory procedure

Reference: The EXIM Bank (2007)

3

History: EDCF

• Projects within 1 year that were evaluated after the project was completed 3 years ago

• Within 3~4 years after the project completion evalution

• Projects over 20 million USD of approved loan

• Independent Planning Division (exception: request outside experts when the loan is over 35 million USD)

Terms

Objects

Subject

4

Reference: The EXIM Bank (2007) reinterpretation

I.

History: KOICA

평가대상

Founded in 1991 1999• ‘Guideline of Project

Evaluation’: standard, procedures, feedback etc.

• OECD DAC criteria

2008• Detailed definition of evaluation

standards• Include cross-cutting issues

evaluation stands

2011• ‘Independent’ Evaluation Office• Strengthen Evaluation System

1996

• Project evaluation team established

2006

• 2nd Amendment of ‘Guideline of Project Evaluation’

• Department for project evaluation

2009

• Implemented Feedback System• Joint Evaluation

5

I.

Reference: KOICA (2011), “Korea International Cooperation Agency 1991-2010,”

History: KOICA

평가대상

Types Selection Standard Evaluation Cycle

Policy/Strategy/

Thematic Evaluation

1. IF, special demands requests from internal (KOICA) departments or external entities

2. IF, an analysis of development-related issues and major pending issues is required by

the Evaluation Office’s own judgment;

(*Evaluation of beneficiary satisfaction is conducted annually for submission of KOICA evaluation)

when demand

Country Evaluation

1. List of KOICA Key Country Partners

2. Select not to be bias by continent or region

3. Select a country with no Comprehensive Evaluation in the last 3 years and consider the cycle of

Country Partnership Strategies (CPS)

Every 5 years

Sectorial Evaluation1. KOICA 5 Key Sectors

2. Select subject in detail based on programme classification by sector concerned Every 3 years

Porgramme Evaluation 1. Subject to regular and comprehensive inspection of KOICA in the Evaluation Office Every 3 years

Ex post Project

Evaluation

1. First round of selection among the projects with completion evaluation within 2~4 years

2. Second round of selection to projects that are 2 million USD or more

3. Final selection via feasibility studies and evaluation environment among the second round of

selections

Every year

Reference : KOICA (2016), p. 7. 6

I.

History: ODA Integrated Evaluation

평가대상

Year Contents

May, 2009 ⦁ Implement ODA Integrated Evaluation System

December, 2009⦁ Composition of Evaluation sub-committee ⦁ Enactment of evaluation guidelines and integrated evaluation manuals

2010 ⦁ Conducted an ODA Integrated Evaluation

July, 2010 ⦁ Enforcement of the Framework Act on International Development Cooperation

2011 ⦁ Full implementation of ODA integrated evaluation

July, 2013 ⦁Revised the Framework Act on International Development Cooperation

January, 2014 ⦁ Enactment and revision of regulations related to ODA integrated evaluation

References: :International Development Cooperation Evaluation Committee of the Cabinet Office (2015; 2016), Office of Development Cooperation Policy Management (2018) reinterpretation

7

I.

History:Self-evaluation : Number of evaluation and agencies increased after implementing ‘Mandatory self-evaluation by project implementing agencies(2014)’

평가대상

*Voluntarily self-evaluation: 2010~2013

12

4 5 7

2529

32 33 34

16

21

30

39

80

58

78 7875

2010 2011 2012 2013 2014 2015 2016 2017 2018

기관수 건수

Reference: :International Development Cooperation Evaluation Committee of the Cabinet Office (2018), “ODA Integrated Evaluation Improvement Measures” reinterpretation

8

I.

Number ofOrgs.

Number ofProjects

Analysis of KOICA’s Evaluation Reports: Data

평가대상

Reference: www.koica.go.kr ‘Project Evaluation’ reinterpretation (Last assessed on August 24, 2018)

KOICA Project Evaluation Reports (1998-2017)

(N=178)

Ex-post Evaluation 59%

Impact Evaluation 3%

Completion Evaluation

2%

Theme· Sector13%

Programme 7%

Country 4%

Policy·Strategy4%

Beneficiary 4%Joint Evaluation

2%Research

Evaluation2%

9

II.

12

(N=134)

KOICA Evaluation Reports by Sector (1998-2017)

(N=134)

KOICA Evaluation Reports by Region (1998-2017)

Reference: www.koica.go.kr ‘Project Evaluation 'reinterpretation (Last assessed on August 24, 2018)

II.

19

33

21

36

14

11

PublicAdministration

Education Agriculture andFishery

Development

Public Healthand Medical

IndustrialEnergy

InformationTechnology

77

17

27

7 6

Asia Africa Latin America Middle East Central Asia

Analysis of KOICA’s Evaluation Reports: Data

13

KOICA Evaluation via Quantitative Evaluation Method(N=27)

*Value Engineering, Difference in Difference, Data Envelopment Analysis (DEA), AHP(Analytic Hierarchy Process)

Reference: www.koica.go.kr ‘Project Evaluation '(Last assessed on January 6, 2018)

II.

17

5

1

1

1

1

1

0 2 4 6 8 10 12 14 16 18

Triangulation Method

Cost-benefit Analysis

Economic Analysis

VE Analysis

DID

DEA

AHP

Analysis of KOICA’s Evaluation Reports: Data

14

III.

Indicators

Sustainability

Enhancement of Nation’s position

Cross-cutting Issue

Results

PublicAdministration

Capacity building

Harmonization

Incentive System

Partnership

Ownership

Knowledge sharing

KOICA’s ex-post Evaluation Network Analysis_Findings (1998-2010)

Minority Issue

Network building

Gender Issue

Environment Issue

Local supply

Satisfaction rate

Feasibility study

Baseline survey

Effectiveness Impact

Efficiency

RelevanceManual

Incentive System

Volunteer activity

Economic growthPublic relations

Risk analysis

Ex-post management

Expert issueEconomic growth

Communication

BenchmarkingMonitoringAlignment

SSC

Follow-up project

Retraining

Analysis of KOICA’s Evaluation Reports: Results

*Content Analysis Software: Atlas.ti (7.5.17), UCINET, KrKwic

15

III.

Capacity building

Capacity building

Income growth

Economic growth

PartnershipCommunication

Logical FrameworkCross-cutting Issue

Feedback Process

Economic growth Enhancement of Nation’s position

Alignment

Network buildingBaseline survey

Change of perception

Risk analysis

PublicAdministration

Knowledge sharing

Monitoring

Ownership

Exit Strategy

Efficiency

Relevance

Follow-up project

Environment Issue

Gender Issue

Manual

Indicators

Satisfaction rate

Effectiveness

SustainabilityImpact

Results

Experts

Ex-post management

Triangular cooperation

Volunteer issue

Minority issue

Feasibility

Incentive system

Human resource building

Differentiated strategy

Strategy of enterprise

Public relations

Analysis of KOICA’s Evaluation Reports: Results KOICA’s ex-post Evaluation Network Analysis_Findings (2011-2016)

*Content Analysis Software: Atlas.ti (7.5.17), UCINET, KrKwic

16

III.

Strategy of enterpriseSatisfaction rate

Benchmarking

Environment Issue

Economic growth

Effectiveness Local supply

Change of perception Public relations

Feasibility studyKnowledge sharing

Sustainability

Effectiveness

Impact

Efficiency

Relevance

Human resource building

Income growth

SSC

Risk analysis

Follow-up project

Responsibility

Network building

Incentive system

Ownership Indicators

Manual

Communication

Volunteer issueEnhancement of Nation’s position

Retraining

Partnership

PublicAdministration

Capacity building Expert issue

Ex-post managementResults

Monitoring

Feedback Process

Feasibility

Analysis of KOICA’s Evaluation Reports: Results KOICA’s ex-post Evaluation Network Analysis_

Lessons & Recommendations (1998-2010)

*Content Analysis Software: Atlas.ti (7.5.17), UCINET, KrKwic

17

III.

Strategy of enterprise Public relations

Benchmarking

Alignment

Differentiated strategy

Networking building Incentive system

Harmonization

Sustainability

Effectiveness

Impact

Efficiency

Relevance

Partnership

Human resource building

Relevance

CommunicationRetraining

Triangular cooperation

Local supply

Knowledge sharingPublic administration

Experts

Capacity building

Results

Ex-post management

Feasibility study

MonitoringManual

Indicators

Minority issue

Ownership

Baseline survey

Change of perception

Satisfaction rate

Follow-up project

Economic growth

Feedback ProcessExit strategy

Income growthResponsibility

Environment Issue

Gender Issue

Volunteer issue

Enhancement of Nation’s position

Linkage of Loan & Grant

Logical framework

Cross-cutting Issue

Risk analysis

Analysis of KOICA’s Evaluation Reports: Results KOICA’s ex-post Evaluation Network Analysis_

Lessons & Recommendations (2011-2016)

*Content Analysis Software: Atlas.ti (7.5.17), UCINET, KrKwic

18

Integrated Evaluation System: Key Issues

01

02

03

04

• Systematic Limitations of Integrated Evaluation Systems① Poor consistency of evaluation methodology and criteria ② Problem of procedure application③ Poor function of evaluation sub-committee

• Self-evaluation quality deviation① Lack of specific criteria for selection ② Reliability problem of evaluation③ Poor evaluation infrastructure

• Poor Exchange of Integrated Evaluation Results & Public System① Separation of external evaluation and self-evaluation② Limitations in extracting evaluation results and

implementation tasks ③ Poor public disclosure and sharing of evaluation results

Reference: :International Development Cooperation Evaluation Committee of the Cabinet Office (2018), “ODA Integrated Evaluation System Improvement Measures”

III.

19

01

02

03

04

• Improvement of integrated evaluation system① Prepare a systematic evaluation base ② Improvement of evaluation guidelines③ Improvement of evaluation procedure④ Strengthening the functions of the evaluation

subcommittee

• Improve quality of self-evaluation ① Rationalize & simplify with selection for evaluation ② Secure objectivity and reliability of evaluation process ③ Expand evaluation infrastructure

• Expanding the practical use of evaluation results① Strengthening the linkage between internal

evaluation and self-evaluation ② Reforming the reflux system③ Expanding the disclosure of information to the public &

partner countries

Integrated Evaluation System: Improvement Measures

III.

Reference: :International Development Cooperation Evaluation Committee of the Cabinet Office (2018), “ODA Integrated Evaluation System Improvement Measures” reinterpretation

20

Conclusion : Challenges Ahead

Reference : Kwon et. Al. (2016), 『Meta Evaluation Research on Self-Evaluation of ODA implementing agencies』 Korea Institute for International Economic Policy

Quality Improvement Plan for Self-Evaluation: Strengthen Performance Measurementto increase objectivity of evaluation

The selection criteria for evaluation must be strictly enforcedThe role of the assessment

cooperative agency and assessment subcommittee should be strengthened

The internal guidelines need to be changed so that certain portions of the project budget can be organized into performance management and evaluation budgets, and the budget authority needs institutional support.

IV.

21

Conclusion : Topics for further discussion

IV.

▶ Independence of evaluation unit within agencies

▶ Centralized vs Decentralized Evaluation System

▶ Selection criteria and procedure for evaluation

▶ Evaluation knowledge management system for better results

▶ Evaluating evaluators and partners

▶ Validation of project completion report

……….

22

References

강경재. 2011. “원조사업 성과관리의 개요 및 동향: AFD의 사례를 중심으로.” 『국제개발협력』 제4권.___. 2012. 『KOICA 기관 성과관리 개선방안 연구: 개발협력 사업 성과관리를 중심으로』 한국국제협력단.국무조정실. 2015. “제18차 국제개발협력 평가소위원회 보고안건(제18-1-2호).” 국제개발협력 평가소위원회.___. 2016. “2017년 국제개발협력 통합평가 계획.” 국제개발협력위원회 평가소위원회.___. 2018. “ODA 통합평가체계 개선방안.” 국무조정실 개발협력정책관실.권율 외. 2016. 『ODA 시행기관의 자체평가에 대한 메타평가 연구』 대외경제정책연구원.권율·정지선. 2012. 『OECD DAC의 ODA 평가체제와 한국의 개선과제』 대외정책연구원.김현경 외 7인. 2016. 『ODA 평가체계 연구: 보건복지분야 사업평가의 시사점』 한국보건사회연구원.민지홍·김철우. 2012. 『주요 수원국의 ODA 사업 관리체계 비교분석』 한국행정연구원.박숙현. 2012. “DAC Peer Review의 소개와 원조평가.” 『국제개발협력』 제3권, 185-196.박한우·Loet Leydesdorff. 2004. “한국어의 내용분석을 위한 KrKwic 프로그램의 이해와 적용: Daum.net에서 제공된 지역혁신에 관한 뉴스를 대상으로.”

Journal of the Korean Data Analysis Society 6(5), www.hanpark.net.윤수재. 2013. 『ODA사업 평가 및 모니터링 시스템 개선에 관한 연구』 한국행정연구원.___. 2014. 『한국 ODA사업 메타평가 및 개선방안에 관한 연구』 한국행정연구원.윤수재·조태준. 2017. “공적개발원조(ODA) 사업 자체평가에 대한 메타평가 연구.” 『한국거버넌스학회』 제24권 2호, 27-60.이지영. 2013. “평가의 등급화: 무상원조 프로젝트 등급화를 중심으로.” 『국제개발협력』 제3권.장효진·김우림·권혁주. 2015. “개발효과성 관점에서 개발협력사업 평가체계의 분석: 독일·영국·한국의 개발협력사업 평가에 대한 비교를 중심으로.”

『한국정책학회보』 제24권 3호, 61-94.정지선·오태현. 2013. “ODA 국별 성과관리체제 및 평가방법에 관한 연구.” 대외경제정책연구원(KIEP).한국국제협력단. 2011. 『한국국제협력단 20년: 1991-2010』 한국국제협력단.___. 2016. 『2015 KOICA 평가연보』 한국국제협력단.한국수출입은행. 2007. 『대외경제협력기금 20년사』 한국수출입은행.한승수·박광국. 2015. “AHP를 이용한 한국국제협력단(KOICA) 사업 성과관리 개선방안.” 『새마을운동과 지역사회개발연구』 제11권, 1-30.홍재환 외. 2012. 『ODA 정책사업의 평가체계 연구』 한국행정연구원.차은주. 2010. “국제개발협력 평가 동향과 협력단의 과제.” 『국제개발협력』 제2권, 64-81.Sasaki, Ryo. 2012. “An In-Depth International Comparison of Major Donor Agencies: How do They Systematically Conduct Country

Program Evaluation?” Journal of MultiDisciplinary Evaluation 8, No.18, 29-45.

Thank you hsohn@khu.ac.krhttp://cidec.khu.ac.kr