MICHELE TARSILLAEVALUATION SPECIALIST

AFCOP ANNUAL MEETING(MAY 24, 2011)

Evaluation of the African Community of Practice on Managing for

Development Result (AfCoP-MfDR)

OVERVIEW

① Introduction ② Theory of Change and

Key Hypotheses③ Key Evaluation Questions④ Methodology⑤ Key Findings⑥ Group Discussion

1. INTRODUCTION

AFCOP EVOLUTION

1.1. AfCoP Key Objectives (2007-2011)To provide an interactive online

platform for African MfDR practitioners, policymakers and researchers;

To enhance individual MfDR capacities;

To establish a community to build support across Africa for MfDR;

To exchange and disseminate MfDR knowledge and good practices in Africa and beyond.

AFCOP EVOLUTION

1.2. Key Objectives (as of 2011)To establish Eleven new

National Community of Practices (NCoP);

To create partnerships with other regional institutions;

To bring a new dynamic to knowledge-sharing activities

2. Theory of Change and 3 Key Hypotheses

2.1 Key Hypothesis 1

AfCoP contributes to MfDR Individual learning and practice enhancement

-Through online exchanges occurring among a variety of people with different backgrounds, experiences and skills, the content quality of virtual discussions is improved, learning among participants is greatly enhanced and the adoption of the newly acquired knowledge accrues (Wenger, McDermott, & Snyder, 2002; Hinds & Kiesler, 2002; Medley, 1999) (Galegher, Sproull, & Kiesler, 1998).

2.2 Key Hypothesis 2

AfCoP contributes to MfDR-related Community learning and community building both online and offline

-Electronic discussions and blog posts over time contribute to strengthening the ties among the platform individual users (Scott and Thomson, 2005), thus leading to the development of a community of practitioners (Martin & Bearden, 1998; Jensen, Danziger, & Venkatesh, 2007).

2.3 Key Hypothesis 3

AfCoP contributes to institutional building and regional partnerships

- Exchanges occurring over the internet make learning opportunities accessible to a much larger number of people than face-to-face training programs or thematic conferences could ever do (Scott and Johnson, 2005; Johnson, 2001; Jamieson, 2000).

2.4 AfCoP Theory of Change

3. Key Evaluation Questions

3. 1 Key Evaluation Questions (I)

① To what extent has AfCoP contributed to increasing the knowledge of MfDR good practices among its members?

② To what extent has AfCop contributed to theadoption of MfDR good practices among its members?

③ To what extent has AfCop contributed to community building among its members?

3.2. Key Evaluation Questions (II)

④ To what extent has AfCoP contributed to strengthening MfDR country processes?

⑤ To what extent has AfCoP contributed to generating country ownership of MfDR?

⑥ To what extent has AFCoP contributed to strengthening South-South cooperation among its members, as well as to connecting its member countries to other country processes?

4. Evaluation Methodology

4.1 Methodology

The evaluation design is non-comparative retrospective with multiple lines of evidence Firmly grounded on a mixed-method approach, the evaluation has been

relying on the use of both quantitative and qualitative data collection methods

Qualitative data collection tools include: Literature review; Documents review; Thematic analyses of online resources and open-ended items included in the

Member Survey; Online structured discussions; Key Informant Interviews with AfCoP Secretariat staff (past and present) Case studies

Quantitative data collection tools include: AfCaa aoP Members survey (via Survey Monkey); AfCoP CMT survey; Trend and browsing pattern analysis

5.1 Key Evaluation Question 1

To what extent has AfCoP contributed to increasing the knowledge of MfDR good

practices among its members?

Increase in MfDR Knowledge

On average, all survey respondents claim that they have improved their understanding of all the 5 MfDR pillars as a result of their participation in AfCoP

MfDR Pillar Change in the number of respondents with good or very good knowledge of MfDR pillar

before and after joining AfCoP (%)

Leadership +59%

M&E +89%

Accountability and Partnerships

+84%

Planning and Budgeting

+60%

Statistical Capacity

+51%

Key learning through the AfCoP website

Key learning on MfDR enhanced by the AfCoP website include the following:

How to use M&E for decision-making (n=142);How to strengthen MfDR national capacity (n=113);How to perform results-based budgeting (n=66);How to implement results-based development

initiatives (n=59);How to formulate results-based action plans (n=57);How to improve Development Effectiveness (n=54)

Quality rating of AfCoP resources

The majority of respondents (n=159) find the AfCoP website either very easily accessible (n=46) or easily accessible (n=113);

Eighty-five percent of respondents (n=208) find the quality of the online resources either very good (n= 62) or good (n=146), while a smaller group (n=38) finds it average;

As to the clarity of the AfCoP website content, 84% of respondents (n=200) rate it either very good (n=38) or good (n=162). A smaller number of respondents rate it average (n=38);

Seventy-two percent of respondents (n=172) claim that AfCoP online resources are relevant to their needs. The remaining 28% of respondents (n=68) point out that the online resources made available by the AfCoP website do not always meet their needs due to a variety of reasons including:

-The lack of practical or pertinent MfDR examples -Limited variety of MfDR topics presented and/or discussed online

5.2 Key Evaluation Question 2

To what extent has AfCop contributed to the

adoption of MfDR good practices among its members?

MfDR Adoption Rate

Fifty-two percent of respondents (n=242) claim that they have adopted MfDR concepts and tools learned through the AfCoP website, mainly in six areas: MfDR Capacity Building Systematic use of logical frameworks Adoption of innovative practices Contribution to PRSP formulation and

implementation Enhancement of business success Alignment of budget with Work Plans

At what level and in which sector is MfDR adopted?

Eighty-four percent of respondents (n=268) work in planning or management capacity, mostly in 4 key sectors: donor aid, finance, agriculture and education

Environmental factors hindering AfCoP members’ adoption of MfDR

According to the survey, environmental factors hindering the AfCoP members’ application of MfDR at work include the following:

Low awareness of MfDR at the workplace (e.g. among colleagues) (n= 117);

Lack of an adequate Management Information Systems (N=88);

Lack of director’s support (n=68); Lack of incentives or awards for the adoption of MfDR (n=82); Lack of champions in the professional sector of interest

(n=69); Lack of communication between planners and managers

(n=37); Job turnover (n=19)

5.3 Key Evaluation Question 3

To what extent has AfCop contributed to community building

among its members?

Contacts developed within AfCoP (I)

Sixty percent of respondents (n=146) claim to have developed some contacts with some other AfCoP members over time. The remaining 40% (n=97) claim to have developed no contact.

The majority of those who developed contacts (n=57) linked quite closely with 1-2 other members, while a slightly smaller contingent of members (n=54) have developed 3-7 contacts. Furthermore, 19 members have developed 8-15 contacts and 16 members have linked closely with more than 15 members.

Among those respondents who have developed contacts with other AfCoP members, a large number of them (n=92) discuss with their contacts offline (e.g., via personal e-mail, msn, Skye or face-to-face meetings). On the contrary, the remaining 126 members do not discuss with each other offline.

Contacts developed within AfCoP (II)

Fifty-three percent of respondents (n=96) have established contacts with other members in the same professional capacity while the remaining 47% (n=84) have developed contacts with members in a different professional capacity;

Furthermore, 60% of respondents (n=110) developed contacts with members working in a different sector than theirs while the remaining 40% (n=70) have liaised with other AfCoP members working in the same field (this is likely to foster great opportunities for cross-fertilization).

5.4 Key Evaluation Question 4

To what extent has AfCoP contributed to strengthening MfDR

country processes?

Improvements in MfDR national processes

Fifty-three percent of respondents (n=125) claim to have seen some improvements in their country planning processes over the last three years while the remaining 47%(n=111) claim to have witnessed none.

When asked about the type of improvements that occurred in their national planning processes over the last three years, respondents identified 4 main types:

Institutionalization of MfDR (n= 39) Enhanced accountability (n=19) Introduction of performance-based budgeting (n=16) Enhancement public sector performance (n=13)

AfCoP Contribution to MfDR Improvements (I)

Among those who claimed that some improvement took place, 47% of respondents (n=75) claim that AfCoP contribution was substantial: in their opinion, AfCoP contributed to national MfDR improvements either very much (n=21) or much (n=54). The remaining 53%(n=86) declared that AfCoP contribution was more modest and that it influenced national strategic planning processes either little (n=31) or very little (n= 14). Key AfCoP contributions include the following:

Enhanced Focus on Results at all levels (n=25); Strengthened in-country MfDR Capacity Building programs (n=15); Accrued accountability and inclusiveness in national planning processes

(n=12); Increased number of National CoPs (n=12); Introduction of MfDR good practices in key national sectors (n=9); Increased use of MfDR tools and methodologies (n=6); Enhanced exchanges among national key players in the area of MfDR (n=5); Increased focus on results during PRSP formulation processes (n=4); Increased use of Performance-based budgeting (n=4)

AfCoP Contribution to MfDR improvements

More precisely, 38% of respondents (n=91) indicated that they have contributed to their country MfDR national process by doing the following:

Participating in the formulation of National or regional MfDR policies (n=27);

Delivering MfDR Capacity Building programs (n=16); Enhancing the effectiveness of public programs (n=12); Informing the development of their country’s PRSP

(n=11); Promoting the creation of national CoP (n=7); Promoting the use of performance-based Budgeting (n=6); Aligning Donor’s funding to national strategies (n=3);

Key Evaluation Question 5

To what extent has AfCoP contributed to generating country ownership of MfDR?

National Recognition of MfDR

When asked in which stage of the MfDR capacity development they would place their country, the majority of respondents (n=74) indicated experimentation, followed by transition (n=55), awareness (n=48), and sustainable implementation (n=20);

When asked whether their country recognize MfDR, 89% of respondents (n=191) answered yes while the remaining 11% (n=23) answered no. Furthermore, 76% of respondents (n=146) added that their country’s governments are generally aware of AfCoP

Incentives and National MfDR Events

With respect to the availability of incentives to individuals adopting MfDR practices, 78% of respondents (n=133) acknowledged that none is available;

Fifty-three percent of respondents (n=116) also indicated that, to their knowledge, their government had not sponsored any MfDR-related event over the past 12 months. The remaining 37% (n=80) said that some events were organized and an additional 10% simply did not know about it (n=22);

When asked which ministry was the most active supporter of MfDR events in their country, respondents mentioned a plurality of governmental entities (see table in the next slide)

Sponsoring Ministry 

Number Respondents

Sponsoring Ministry 

Number Respondents

Economics & Finance 17 Economics 2

National Planning 16 Education 2

Health 9 Environment 2

Offiee of the Prime Minister

7 Community Development

1

Agriculture 6 Development 1

Finance 5 Economic Planning 1

Economic Development

4 Energy 1

Office of the President

4 Infrastructure 1

Planning and Development

4 Innovation 1

Public Service 4 National Development 1

State Reform 4 Prospective Development

1

AfCoP 3 Public Policy Evaluation 1

All Ministries 3 Rural Development 1

Most ministries 3 State Secretariat 1

Budget and Finance 2 Youth and Employment 1

National Ministries sponsoring MfDR Events

The MfDR-MDG Link

In addition, when asked whether MfDR practices are being used to attain MDG in their country, 43% of respondents (n=102) said that it is done often, while the remaining 57% (n=135) said either always (n= 19), very often (n=49), rarely (n=46), or never (n=10).

5.6 Key Evaluation Question 6

To what extent has AFCoP contributed to strengthening South-South cooperation

among its members, as well as to connecting its member countries to other

country processes?

Informant Interviews are still being conducted and a Document Review is

underway

6. Group Discussion