Program evaluation Program evaluation working groupworking group

Information reportInformation report

5th Knowledge and sharing Committee (KSC)

Annual Meeting

New Delhi 18-19 september 2013

ISC Evaluation : the way forwardISC Evaluation : the way forward

Where we come from

Where we are

Where we could get to

PROGRAM EVALUATION WG 2

Where we come fromWhere we come from

Evaluation working group set in 1992 (XIVth INCOSAI Washington)

The Evaluation Working Group, is composed of 19 SAIs and chaired by France

1995 XVth INCOSAI Cairo : demand for methodological guide

1998 XVIth INCOSAI Seoul : draft report on methods and practices

PROGRAM EVALUATION WG 3

2004 XVIIIth INCOSAI Budapest : introductory report (fundamental concepts and examples)

2008 XIXth INCOSAI Mexico : Evaluation related to Knowledge sharing more than Standards

2010 XXth INCOSAI Johannesburg : endorsement of the introductory report “Program evaluation for SAIs- A primer”, proposed by EWG under the auspices of KSC

PROGRAM EVALUATION WG 4

Where we come fromWhere we come from

A website dedicated to the WG activity has been open since November 2010.

A questionnaire to collect case studies and practical information on Evaluation methodology and experiences (elaborated by a sub-group composed of 5 SAIs) was approved by the Group in Paris in May 2011 and sent to 191 SAIs. Its results (44 SAIs have answered) were presented to last KSC in 2012.

PROGRAM EVALUATION WG 5

PROGRAM EVALUATION WG 6

A large fraction of SAI are practicing A large fraction of SAI are practicing different aspects of evaluation different aspects of evaluation

43%

27%

21%

9%

SAIs carrying out noevaluations

SAIs carrying out no formalevaluations, but in practiceperforming some aspectsSAIs carrying out evaluations

Not answered

PROGRAM EVALUATION WG 7

There is a growing use of participative There is a growing use of participative processprocess

41%

14%11%

34%

Involvement of auditees in the evaluation process

Managers andbeneficiaries regularlyconsulted

Managers consulted

No consultation withauditees

Not answered

PROGRAM EVALUATION WG 8

But still, a minor regular use of But still, a minor regular use of external expertiseexternal expertise

PROGRAM EVALUATION WG 9

Survey general conclusionsSurvey general conclusions

Practices seem diverse, show clear signs of converging but also persistent differences due to administrative and political culture

However, the answers confirm that, as long as the mandate of the SAIs is not limited to auditing compliance but extends to analysing the efficiency and effectiveness of actions taken by government administrations, the SAI is capable of assuming a broader scope of analysis, should they be ready to make the necessary changes in their method of work and culture.

What we have done since 2012What we have done since 2012

Paris June 2012 : the WG must go on and prepare methodological guideline for evaluation.

March 2013 : a draft summary of the purpose and the content of the methodological guidelines was elaborated by France and sent to participants.

Written comments were sent by Switzerland, USA, Gabon and Poland.

PROGRAM EVALUATION WG 10

What we have done since 2012What we have done since 2012

Draft document setting main items for the methodological guide for evaluation• Definition(s)• Objectives• Actors and institutional environment for SAIs• Methodology• Dissemination and use of evaluation results

PROGRAM EVALUATION WG 11

What we have done since 2012What we have done since 2012 Draft document setting main items for the

methodological guide (follow)• Methodology

• Choice of subject to be evaluated• Evaluation Planning• Setting the researchable questions• Organizing the evaluation• Process building with stake holders• Scientific tools• Ressources identification• Reviewing results• Counter argument phase• Finalising results

PROGRAM EVALUATION WG 12

PROGRAM EVALUATION WG 13

Last PAS meeting Ottawa May 2013Last PAS meeting Ottawa May 2013

• According to resolution taken in 2012, the EWG French representative attended the 6th PAS meeting in Ottawa, May 28-29 2013

• From now on, this representative is also the French member of PAS

• The purpose was :• To give information to PAS about work progress of EWG• To discuss common points and differences between

Performance audit and Evaluation• To clarify the position of Evaluation methodological guidance

regarding professionnal standards (ISSAIs)

• It was decided to go on cooperation between EWG and PAS and to exchange information and documents.

PROGRAM EVALUATION WG 14

Last meeting Paris June 2013Last meeting Paris June 2013

10 of the EWG members participated to the Paris meeting on June 25, 2013 chaired by France• Belgium, Finland, France, Gabon, Republic of Korea,

Morocco, Poland, Switzerland,• the United States via Webex • Brazil was invited as a guest SAI as the Chair of the INTOSAI

Professional Standards Committee• Chile, Costa Rica, El Salvador, Finland, Georgia, Germany,

Libyan Arab Jamahiriya, Lithuania, Mexico, Pakistan, Papua New Guinea did not participate but were asked to validate the resolution

PROGRAM EVALUATION WG 15

Last meeting Paris June 2013Last meeting Paris June 2013

The EWG discussed the links between PAS and Evaluation Guidelines

The EWG discussed and amended the draft document proposed in April

The EWG concluded that :• the need for guidelines was still there• the content proposed by the draft document was clear • the elaboration of the document itself would be shared within

sub-groups for specific issues • there was no clear reason at this stage to erect an ISSAI for

evaluation

Where we could get toWhere we could get to

First proposal for KSC september 2013 and INCOSAI Beijing october 2013

Elaborating the Pratical guide within subgroups in 2013-2014

Giving this guide the status of INTOSAI Guideline (cf Communicating and promoting the Value and Benefits of SAIs, presented to KSC by WGVBS)

Endorsment by Governing Board 2015 Presentation to INCOSAI 2016

PROGRAM EVALUATION WG 16