Post on 31-Dec-2015
description
COMING OF AGE
OF JOINT EVALUATIONS?
ALNAP Presentation at OECD-DAC
21st January 2008
2
Agenda
BackgroundFindings Future Agenda for Humanitarian JEsOther relevant ALNAP work
3
Background
ALNAP is a network of all the major humanitarian actors Now in its tenth year
Works to improve humanitarian performance through learning and accountability
Range of products and outputs, including a meta evaluation of humanitarian evaluations periodic review a sample of evaluation reports against a Quality
Proforma which has been developed according to accepted evaluation good practice.
Systematic use of the Proforma over a number of years has made it possible to identify trends in evaluation quality over time.
4
The ALNAP meta-evaluation series
Meta-evaluation defined as “meta-analysis and other forms of systematic synthesis of evaluations providing the
information resources for a continuous improvement of evaluation practice”.
Overall aim of ALNAP meta evaluation is “...to improve evaluation practice, by identifying areas of weakness that deserve
attention, and examples of good practice that can be built upon...”
There is qualitative evidence that this aim is being met E.g. the use of DAC criteria in EHA has gradually strengthened in the last few years;
consultation with primary stakeholders has improved in evaluation methodology (despite still being poorly rated overall)
In other areas there has been little or no improvement, for example attention to cross-cutting issues such as gender equality, protection and advocacy
Where improvement in quality has been noted, it has usually happened quite gradually
.
5
Humanitarian Joint Evaluations
In the humanitarian sector, the first significant joint evaluation was the seminal multi-agency Rwanda evaluation in 1996 shorter history than in the development sector
Subsequent JEs have usually been collaborative efforts of donor governments, but involvement is broadening to include UN and NGOs
The 6th ALNAP Meta Evaluation focuses on growing number of Joint Evaluations Currently being finalised, findings presented here for discussion
and debate
6
6th ALNAP Meta evaluation: specific objectives
To review the quality of joint evaluation exercises, where possible comparing these with the quality of past single agency evaluations reviewed in previous ALNAP meta-evaluations;
To document in an accessible way the lessons from the growing experience of humanitarian JEs – especially examples of good practice – to feed into future joint endeavours
Thus, to make a significant contribution to the emerging body of knowledge about humanitarian JEs
7
Meta Evaluation Methodology
The methodology used was comparable to previous meta-evaluations, based on a sample of 18 evaluation reports
The quality of the evaluation reports has been assessed against the ALNAP Quality Proforma (slightly adapted to be appropriate to JEs) http://www.alnap.org/resources/quality_proforma.htm
The evaluation processes have been reviewed through individual and group interviews with those involved in the JEs, iterating between these two methods Interviews have been carried out with representatives from 15 different organisations as
well as lead / central evaluators
The data from the assessment against the Proforma was analysed and compared with the results from previous ALNAP meta-evaluations, which have covered a total of 138 evaluations.
8
Typology of JEs (adapted from DAC, 2005) ‘Like-minded agencies’ (or qualified): agencies with similar characteristics
coming together WFP/ UNHCR pilot food distribution (UN agencies operating to an MOU) All ECB evaluations (group of NGOs); DEC evaluations (group of NGOs); IASC RTEs (UN agencies)
Most common
‘Partnership’: donor recipient agencies evaluate together as equal partners ECHO/ WHO/ DFID JE: WHO emergency response, Pakistan
‘Hybrid multi-partner’: disparate actors coming together playing variable roles (eg active/ passive) IHE evaluations (comprising UN agencies, NGOs academics, recipient government etc)
‘System-wide’: open to all actors in the system TEC
9
Agenda
BackgroundFindingsA Future Agenda for Humanitarian JEsOther relevant ALNAP work
10
Ten Hypotheses to be tested
Humanitarian Joint Evaluations...1. help to build trust and social capital within the sector 2. tend to be driven from the centre (ie headquarters)
than the field3. do not involve sufficiently the government of the area
affected by the humanitarian crisis4. offer greater opportunity for beneficiaries to be
consulted/ surveyed than in single agency evaluations5. have more rigorous methodologies than single
agency evaluations
11
Ten Hypotheses to be tested
Humanitarian Joint Evaluations...6. pay more attention to international standards and
guidelines than single agency evaluations 7. are stronger on cross-cutting issues such as gender and
protection than single agency evaluations8. overall quality tends to be higher than single agency
evaluations9. are more likely to address both policy issues and
programme performance than single agency evaluations10. pay attention to wider debates within the humanitarian
sector, and situate their findings accordingly
12
Findings in relation to hypotheses (1)
JEs are no longer solely the domain of donor governments, the early champions of JEs UN agencies and some NGOs are now fully engaged still early days, and some efforts to promote and institutionalise a JE approach have
come and gone, despite evidence that JEs help to build trust and social capital amongst the participating organisations (hypothesis 1)
JEs have so far been northern and headquarters-driven (hypothesis 2) Reflecting the set-up of international humanitarian agencies
Real progress is needed in in fully involving stakeholders in-country - national NGOs, other organisations and governments Generally poorly represented (hypothesis 3). Involving latter will be easier in natural
disasters than in conflict-related humanitarian crises, especially if government is an active party in the conflict
There may be important lessons for the humanitarian sector from JEs on the development side, from work done by DAC to strengthen developing country participation and from the Paris Declaration on Aid Effectiveness
13
Findings in relation to hypotheses (2) Conclusive evidence that JEs are overall of higher quality. than single
agency (hypothesis 8) their terms of reference are generally clearer and more useable consultation with local populations and beneficiaries is stronger (hypothesis 4) more attention is paid to international standards (hypothesis 6); and the OECD-DAC EHA
criteria are more rigorously used
JEs have more rigorous methodologies than single agency evaluations is proven, but not across the board (hypothesis 5) There are striking gaps and weaknesses in the JEs reviewed, especially their attention to
cross-cutting issues such as gender equality, protection and advocacy (hypothesis 7).
Hypotheses that JEs are more likely to address policy issues and locate their findings within wider debates within the sector met with a mixed response (hypotheses 9 and 10) There is some evidence of this (e.g. UN-led humanitarian reform processes), but also
missed opportunities, a number of JEs in our sample had not fulfilled this potential, despite the generally high quality
14
Broader findings: most JEs of humanitarian action are multisectoral, focus on a particular humanitarian crisis
FOCUS OR SCOPE OF EVALUATION
HOW
ACTORS
WORK TOGETHER
Program focus
Institutional Sectoral or thematic focus
Multi-sectoral focus, related to a particular humanitarian crisis (usually bounded geographically)
Global eg global Policy
‘Partnership’: donor & recipient agencies evaluate together as equal partners
ECHO/ WHO/ DFIDJE: WHO emergencyresponse, Pakistan
‘Like-minded agencies’ (or qualified): agencies with similar characteristics coming together
WFP/ UNHCR pilot food distribution (UN agencies operating
toan MOU)
All ECB evaluations (group of NGOs); DEC evaluations (group of NGOs); IASC RTEs (UN agencies)
‘Hybrid multi-partner’: disparate actors coming together playing variable roles (eg active/ passive)
IHE evaluations (comprising UN agencies, NGOs, academics, recipient government etc)
‘System-wide’: open to all actors in the system
TEC evaluation
15
Broader findings: Purpose of JEs
The ToR for most JEs in the sample emphasise both accountability and learning as their purpose in practice learning has dominated ranging from learning about partners’ approaches, to sharing good practice, to
learning about the programme or initiative being evaluated Both the TEC and ECB clearly identified another learning purpose: to learn from the
process of implementing a joint evaluation, and buyild evaluation capacity
Accountability is partially met by JE reports ending up in the public domain. Peer accountability is also strong in JEs which usually demand a higher level of transparency than single agency evaluations
Other purposes of JEs include: building evaluation capacity, learning about the process of doing JEs, and relationship-building between participating agencies
16
Broader Findings: JE skillset
JEs require a different skill set than single agency evaluationstechnical, political and inter-personal skills
Getting the right evaluation team in place is keythe pool of sufficiently skilled evaluators for JEs is
small compared with demand, implying a need to invest in evaluator capacity
For policy-focussed evaluations, there are benefits of hiring policy analysts to lead or be part of the team.
17
Broader Findings: Follow-up & utilisation Generally more accessible than single agency evaluation reports
Possibly because of the higher skill set of evaluation team leaders Use of professional report editors may play a role
Conclusions are slightly stronger than evaluations in previous years, but there is little difference in the quality of recommendations strongest sets of recommendations were those targeted to individual agencies or where
responsibility was clearly indicated the weakest where there were too many recommendations and/ or they were inadequately
focussed
Utilisation-focus is more challenging for JEs because of the range of stakeholders involved with different needs weak link in the chain there are examples of good practice in terms of how the process is designed at the outset and how
the evaluation team engages with stakeholders, especially in-country.
When JEs are part of a wider institutional framework/ relationship, there tend to be better-established mechanisms for discussion and follow-up to recommendations
18
Broader findings: coherence with single agency evaluations “Should JEs replace or reduce the need for single agency evaluations?”
Active but distracting debate “they are very different animals” with different purposes JEs can fulfil accountability purposes, but at a different level needs of a single agency.
Accountability to peers, and to some extent to beneficiaries through stronger consultation, are features of a number of the JEs in our set.
But if individual agencies need to be accountable to their funders in any detail, a JE may not fulfil this need.
JEs clearly complement single agency evaluations by placing the response in the bigger picture/ wider context, exploring how agencies work together, and addressing wider policy issues. when a “club” group of like-minded agencies come together to evaluate their work in a
particular area (e.g. ECB / UN) reducing the number of evaluation teams on the ground asking very similar questions of local communities, government officers and others, is clearly a good thing.
Fewer but more considered JEs of this type may facilitate follow-up by reducing the overload on the humanitarian sector
19
Agenda
BackgroundFindings A future agendaFollow up ALNAP work
20
Where next for humanitarian JEs?
Project that analyses and describes the pros and cons of different collaborative management structures, to guide decision-makers in their future choices
Action research project exploring different and creative ways of consulting beneficiaries
JEs in some thematic and policy areas in the humanitarian sector that are relatively new and/ or challenging to the international system e.g. protection as part of humanitarian action, or livelihood support in the midst of a
humanitarian crisis
JEs should play a more active role in agency evaluation policies, based on clear understanding of the relative costs and benefits of different types of JEs
21
Where next for humanitarian JEs? A planned system-wide JE
A 3rd system-wide humanitarian evaluation should be considered in the next 18 months, focussed on a significant but relatively forgotten/ under-evaluated humanitarian crisis, for example in eastern DRC The sector has much to learn from such an exercise, yet there
would be less pressure to act fast at the expense of process, and it would provide an opportunity to apply the learnings from the TEC whilst they are still fresh. This proposal should be discussed by the ALNAP membership and the wider aid community.
22
Agenda
BackgroundFindings Future Agenda for Humanitarian JEsFollow up ALNAP work
23
ALNAP follow up work of relevance to the OECD-DAC Evaluation Network
Workshop on Humanitarian JEs (2nd half 2008) ALNAP Guide to Real Time Evaluation
end of March 2008
Humanitarian Evaluation Trends and Issuesinitiating case study based research of humanitarian
evaluation, background study nearing completion
7th ALNAP Meta Evaluation (2009) on RTEs
24
Thank you!
6th ALNAP Meta Evaluation Available in March / April
Please get in touch for copies
Ben Ramalingamb.ramalingam@alnap.org www.alnap.org