Seminar on Performance Budgeting and Fiscal Transparency, Tangier, Morocco, April 21-23, 2009

Session 8. Monitoring and Evaluation:

Challenges and Issues

Nowook Park (npark@kipf.re.kr)Center for Performance Evaluation and Management Korea

Institute of Public Finance

2

Contents

Impact of performance budgeting in Korea1

Roles of the MOF, line ministries, parliament, and audit office

2

3

4 How to motivate performance? How to engage politicians?

5

Performance Information : Indicators and Evaluations

1. Impact of Performance Budgeting in Korea

4

Evaluation results Quality of performance information has not improved much Programs are showing better results

Link between evaluation results and budget Evaluation results are utilized at every stage of budget

process

Moving away from incremental budgeting Evaluated programs are subject to bigger budget change

compared to other programs

Observations from Program Review Process

5

Evaluation Results by Total Score

0

50

100

150

200

250

300

0 10 20 30 40 50 60 70 80 90 100

Total

Frequency

2005 2006 2007

6

  Total EffectiveModerately

EffectiveAdequate Ineffective

2005Number 555 28 100 340 87

(%) (100.0) (5.0) (18.0) (61.3) (15.7)

2006Number 577 30 94 388 65

(%) (100.0) (5.2) (16.3) (67.2) (11.3)

2007Number 584 66 139 348 31

(%) (100.0) (11.3) (23.8) (59.6) (5.3)

Evaluation Results by Ratings

7

TotalScore(100)

Planning(30)

Management(20)

Results(50)

 Sub total(30)

Design(15)

Performance Planning

(15)

2005 60.1 23.1 13.8 9.3 15.1 21.9

2006 59.9 22.9 14.3 8.6 14.7 22.3

2007 66.0 23.4 14.2 9.2 15.5 27.1

Evaluation Results by Section

8

MOSF encouraged ministries/agencies to use the results in reshuffling budget allocation

MOSF announced at least 10% budget-cut would be done to “ineffective” programs, in principle

MOSF submitted evaluation results to the National Assembly upon their request

Evaluation results are open to public since 2006

Utilization of Evaluation Results

9

-20

24

695%

CI/F

itte

d v

alu

es/a

gencyuse

0 20 40 60 80 100perf

95% CI Fitted values

agencyuse

Use of Performance Information by Agencies (2005)

10

-20

24

695%

CI/F

itte

d v

alu

es/m

pbuse

0 20 40 60 80 100perf

95% CI Fitted values

mpbuse

Use of Performance Information by MOSF (2005)

11

Use of Performance Information by Legislature (2005)

12

Rating

‘05Budget

(A)

‘06 Budget Difference Ratio

Agency(B)

MPB(C)

Final(D)

B-A C-A D-A Ratio_1 Ratio_2 Ratio_3

Effective 15,600 24,948 18,955 22,489 9,348 3,355 6,889 0.52 0.38 0.50

Mod.Effective

92,994 104,335 107,055 105,762 11,342 14,062 12,768 0.32 0.33 0.28

Adequate 208,066 204,473 195,625 201,214 -3,593 -12,441 -6,852 0.10 0.05 0.05

Ineffective 33,081 28,644 29,007 28,505 -4,437 -4,074 -4,576 -0.15 -0.25 -0.19

Total 349,740 362,400 350,642 357,970 　 　 　 　 　 　

( Unit : 100,000won, % )

Link between Evaluation and Budgeting (2005)

13

Rating 　‘06

Budget(A)

‘07 Budget Difference Ratio

Agency(B)

MPB(C)

Final(D)

B-A C-A D-A Ratio_1 Ratio_2 Ratio_3

Effective 8,891 9,467 9,337 8,872 575 446 -19 0.11 0.10 0.06

Mod.Effective

33,156 35,701 35,364 35,654 2,545 2,208 2,498 0.12 0.08 0.09

Adequate 297,180 296,769 290,481 289,969 -411 -6,699 -7,211 0.10 0.04 0.03

Ineffective 11,431 6,039 5,400 5,380 -5,392 -6,031 -6,051 -0.15 -0.24 -0.25

Total 350,658 347,975 340,582 339,875 　 　 　 　 　 　

( Unit : 100,000won, % )

Link between Evaluation and Budgeting (2006)

14

Rating 　‘07

Budget(A)

‘08 Budget Difference Ratio

Agency(B)

MPB(C)

B-A C-A Ratio_1 Ratio_2

Effective 17,112 18,211 17,503 1,099 391 0.18 0.12

Mod.Effective

266,051 295,121 291,319 29,070 25,268 0.20 0.13

Adequate 146,034 153,026 142,044 6,992 -3,990 0.28 0.15

Ineffective 3,870 3,457 3,066 -414 -805 -0.001 -0.15

Total 433,067 469,815 453,932

( Unit : 100,000won, % )

Link between Evaluation and Budgeting (2007)

15

Programs have been subject to larger budget change after evaluation

Coefficient of Variation in Funding Change (Excluding Programs of which funding change is greater than 200%)

CVYear

(B04-B03)/B03 (B05-B04)/B04 (B06-B05)/B05 (B07-B06)/B06 (B08-B07)/B07

2005 3.1 2.7 9.2

2006 2.7 2.7 -14.3

2007 2.5 3.1 3.9

Moving Away from Incremental Budgeting

16

Budget allocation based on performance is spreading among line ministries Utilization of SABP results in budget requests Changing practices of program management

Evaluation activities become active among line ministries Need for program evaluations are recognized and

accepted among line ministries

Cultural Changes among Line Ministries

2. Roles of the MOF, line ministries, parliament, and audit

office

18

Ministry of Strategy and Finance (MOSF)

Evaluate line ministries’ program Issuing guidelines on performance-based budgeting to line ministries/agencies Evaluating their program’s performance

Use performance information in budget formulation Encouraging line ministries/agencies to use performance evaluation results in preparing their budget requests Incorporating the performance information into its decisions during budget formulation

19

Line ministries/agencies

Producing performance information in compliance with central budget authority’s initiative Submitting strategic plans, annual performance plans

and performance reports Evaluate their programs based on SABP checklist

Conduct program evaluation Use performance information in budget request

20

The National Assembly

Before the enactment of National Finance Act, there had not been official role of the National Assembly They had requested evaluation results for budget deliberation on an ad hoc basis

The National Assembly receives annual performance plan (from 2008) and report (from 2009) with budget documents The National Assembly Budget Office intends to analyze budget in connection with performance information

21

The National Audit Office

Before the enactment of National Finance Act, there had not been official role of the National Audit Office As an independent audit office within the

Administration, it has not played any official role However, they examined the operation of

performance budgeting system and produced report

The National Audit Office is assigned with the role of verifying annual performance report from 2009

3. Performance Information : Indicators and Evaluations

23

Status

Slight increase in outcome measures but there are big room for further improvement Outcome performance measures increase by 3.2 percentage

points, while output performance measures decrease by 4.1 percentage points

More than 40% of programs do not have relevant performance indicators in SABP

Performance IndicatorsPerformance Indicators

TotalType of indicators

Input Process Output Outcome

FY 2007 2,01673

(3.6)140(7.0)

837(41.5)

966(47.9)

FY 2008 2,037111(5.4)

125(6.1)

761(37.4)

1,040(51.1)

24

ExamplesPerformance IndicatorsPerformance Indicators

Good Bad Ugly

Support the IT business

to go abroad

- Customer satisfaction

- Increase the amount of

export($)

Student Loans Program

- The amount of loans- The number of loans

Percentage of attainment- rate of planned goal achievement about employment (x)-rate of employment (o)

Recommendation :Using ‘actual value of attainment’ instead of ‘percentage of achievement’. (FY 2008 manual of KPART)

Recommendation : Using outcome measure ex) Enrollment rates: The postsecondary enrollment gap between low- and high-income high school graduates will decrease each year.

25

Problems

Difficulties to develop outcome measures in some programs Program producing results after long period

• Using milestone indicators may help

• However, using different evaluation cycle needs to be considered

Successful management of program may not change outcome measures

• Reduce scope of outcome measures

• Particular program may not worth evaluating

Data do not exist

• Investment in data is necessary

• Compare cost and benefit of producing new data

Performance IndicatorsPerformance Indicators

26

Lessons

It is desirable to develop outcome measures, but it is not possible for every program

Be aware of some exceptions and apply it

flexibly according to program type

Performance IndicatorsPerformance Indicators

27

Status

Self-Assessment of Budgetary Program Review based on the checklist Among the checklist, there is a question whether program

evaluation is conducted or not It encourage line ministries to conduct program evaluation at

least once in three years

In-depth Program Evaluation In-depth evaluation for selected programs by the central budget

authority

Program EvaluationsProgram Evaluations

28

Status - KPARTProgram EvaluationsProgram Evaluations

• Has your program been evaluated in depth using verifiable data ?• Has your program evaluation been conducted by independent organization ? ex) external institution, audit office, internal evaluation expert, etc.• Does evaluation cover important issues of program ?

Note) ‘yes’ in case that evaluation is in progress or conducted within 3 years latest

YES !YES !

Q. 3-1. Did you conduct comprehensive program evaluation objectively?

29

Problems

Little experience with program evaluation among program managers

Lack of data to prohibits meaningful program evaluation

Lack of fund to conduct program evaluation

Difficulties in maintaining independence of evaluators Trade off between independence and expertise in the

society where social ties are strong and dense

Program EvaluationsProgram Evaluations

30

Example : BadProgram EvaluationsProgram Evaluations

Promotion of the intellectual property (IP)

Obstacle

Purpose

• Measuring the impact of program performance

• Realign the way of investment

• Monitoring the performance management system

IssueDoes the number of IP center affect the number of industrial property ?

Methodology Estimation based on the fixed effect model

• Insufficient Data - absence of data which is needed to conduct the evaluation - using customer satisfaction (or number of counseling) instead of the number of intellectual property in each localities• Lack of understanding on program evaluation among program managers

Result : Couldn’t find the evidence for the regional impact of the IP center !

31

Example : GoodProgram EvaluationsProgram Evaluations

Rural development program

Purpose• Evaluate the impact of the program

• Monitoring the program in progress

IssueDoes the amount of fund improve the condition of the undeveloped-area ?

Methodology Data analysis and the survey

Result of EffectivenessFind the evidence for the regional impact! - There was positive relation between the program’s spending and the infrastructure - In addition, the infrastructure created value-added business

Result of ManagementFind the problems of the management system! - The management system is not developed and the each manager’s autonomy is limited

32

Lessons

Plan ahead for data collection Data should go hand in hand with program management Data management is a part of program management

Be creative in securing fund for evaluation Evaluation should be part of program management In budget negotiation, evaluation plan should be presented

Program EvaluationsProgram Evaluations

4. How to motivate performance? How to engage politicians?

Institutionalized Incentives for line ministries

How performance information will be used in budget preparation is stipulated in guidelines for budget preparation In principle, 10% budget cut for ineffective program is

encouraged to line ministries It draws attention from program managers but not from high

ranking decision makers

Results from SABP is used as a part of evaluation information for the whole department Now high ranking decision makers pay attention But their attention is focused more on getting good scores than

on improving programs’ performance

Forms of Information does Matter

Performance information (performance indicator) at monitoring level has not been able to draw attention from decision makers They are internally useful information, but not utilized by central

budget authority• Remains to be seen how they will be utilized by involved parties

Performance indicators are good starting point for communication rather than decision making

Performance information (program ratings) from program reviews are actively used by central budget authority It summarizes various information into simple ratings Power of Simplicity!

Politician’s Engagement

Although there is no formal mechanism for politician’s engagement, politicians show interest in performance information They requests performance information on an ad hoc basis There has not been much suspicion on the evaluation results

from the Administration so far It remains to be seen whether politicians will use

performance information more systematically with the submission of annual performance plan and report to the National Assembly,

Publicizing performance information may help to force politicians to pay attention to PI