Executive Summary EN - European...

Evaluation of ECHO’s Interventions in Pakistan (2010-‐2014)

Executive Summary Objective and Scope

This independent and summative evaluation focuses on the actions in Pakistan of the European Commission’s Directorate-‐General for Humanitarian Aid and Civil Protection (ECHO) during the period of 2010-‐2014, and assesses the relevance, appropriateness, effectiveness, coverage, efficiency, connectedness, coherence, coordination and EU Added Value of these actions.

The ECHO interventions were funded through 8 funding decisions (6 Humanitarian Implementation Plans – HIP – of a total of 137 funded contracts of a total of more than 406 mio Euro, and 2 DIPECHO1 decisions amounting to a total of 3 mio Euro).

The scope of the evaluation covers the four pillars of DG ECHO’s humanitarian response, which are:

1. Interventions in response to armed conflict;

2. Interventions supporting the protection of Afghan refugees in the protracted refugee-‐crisis;

3. The nutrition crisis response: a multi-‐sectoral intervention focussing on the districts most affected by acute child malnutrition; and

4. The response to disasters caused by natural hazards aiming – besides emergency relief – at increasing the preparedness for future strikes of natural hazards and thus the reduction of risks (DRR) and at strengthening the overall resilience of the population, enabling them to better cope with future threats.

In addition, the evaluation offers perspectives on the extent to which DG ECHO was able to • operate within the reduced humanitarian space of Pakistan and manage to adhere to

the humanitarian principles;

• ensure that the principle of do-‐no-‐harm and protection was respected throughout the supported implemented actions;

• successfully mainstream the issues of Linking Relief Rehabilitation and Development (LRRD), Disaster Risk Reduction (DRR) and resilience issues into the actions; and

• take into account and address gender inequalities in its interventions supported.

Finally, the evaluation provides a map of the factors that hinder or limit the success of ECHO-‐supported interventions, with the purpose of providing inputs for further focusing ECHO’s monitoring.

The evaluation covers interventions under the six financial decisions made by DG ECHO between 2010 and 2014 in Pakistan under the respective HIPs, and under the two allocations

1 Disaster Preparednes ECHO programme


for Disaster Preparedness/Disaster Risk Reduction implemented under the South Asia DIPECHO Action Plans in 2011 and 2013. In addition, the evaluation considers the funding decisions for Afghanistan that included actions related to the situation of Afghan refugees in Pakistan.

While the evaluation assesses individual projects funded by DG ECHO and implemented though ECHO’s partner organisations (international Non-‐Governmental Organisations (NGOs), United Nations (UN) Agencies and the Red Cross and Red Crescent Movement), the conclusions and recommendations address the overall strategic evaluation of DG ECHO’s interventions in Pakistan between 2010 and 2014, based on the overall information collected during the evaluation process.

Methodology

17 Evaluation Questions guided the evaluation and the methodology was structured in 5 phases and in 4 methodological steps (STEP 1 – Evaluation of the project portfolio; STEP 2 – Case study of a representative sample of the portfolio; STEP 3 – Assessment of the overall ECHO strategy in Pakistan; STEP 4 -‐ Documentation and presentation of the evidence). An Evaluation Matrix representing the analytical framework guided the data collection and analysis. In addition, an intervention logic of ECHO’s actions in Pakistan was developed.

The Evidence Matrix was used by the team to capture all primary evidence collected and then used as a basis for the production of the Data and Analysis Report – a proprietary data aggregation tool. This methodology allowed the team to base its analysis on a total of 1216 organised data points, derived from 42 structured interviews, and focus group discussions, with a total of 150 interlocutors representing the relevant constituencies. The on-‐site data collection was conducted through a 3-‐weeks field visit to Pakistan.

The evaluation faced some limitations, i.e.:

1. The timing of the evaluation a full year after the end of the evaluation period and five year after its beginning, which represented a constraint in terms of access to the memory of informants;

2. The coincide of the field phase with the 2015 flooding, which resulted in a certain unavailability of humanitarian partners for interviews, and

3. The light staffing of DG ECHO’s representation in Islamabad during the field phase, which limited the related data-‐generation efforts.

These limitations were mitigated to an extent that the final set of primary evidence provided a robust enough basis for the consolidation of findings and conclusions.

The country context

Post-‐independence Pakistan is characterised at the political level by periods of military rule, political instability, and conflicts with neighbouring India. Through the internal conflicts, particularly with Taliban factions in the Federally Administered Tribal Areas (FATA), various


separatist groups in the south-‐western province of Balochistan, and military operations in the North-‐West, mostly in the Khyber Pakhtunkhwa province (KPK) and Waziristan, the number of Internally Displaced Persons (IDP) amount to approximately 1,5 million people. ECHO has classified the protracted IDP crisis in the areas bordering Afghanistan in the northwest as a Forgotten Crisis2. In addition, the country faces numerous other challenges such as poverty, overpopulation, low literacy rates, terrorism and endemic corruption. Added to an already complex picture are a range of recurring natural disasters such as earthquakes, as well as droughts and floods.

In addition to the above, Pakistan has hosted refugees from neighbouring war-‐torn Afghanistan3 (currently 1 500 000) since 1979, of which close to 40 % are living in so-‐called refugee villages and close to 60 % in urban and rural host communities throughout Pakistan, exacerbating a situation defined by decades of weak governance, and a wide range of socio-‐economic deficits, first and foremost severe poverty. The population’s vulnerability is aggravated by uneven land ownership and gender inequality. Floods affected an estimated 22 million people during the evaluation period. The provinces most affected by the floods were Balochistan, Punjab, Sindh, Khyber-‐Pakhtunkhwa and the Federally Administered Tribal Areas, where also most of the ECHO-‐funded projects were situated. In addition, droughts affected the province of Sindh. A nutrition crisis manifested as an endemic and systemic issue has resulted from the combination of the above factors.

Overall Pakistan represents a challenging context for providers of humanitarian aid, which is rendered even more complex by government restrictions placed on international staff and humanitarian workers regarding access to disaster or conflict-‐affected areas, and by severe security threats. At the same time, the most important partner in delivering humanitarian assistance in Pakistan is the government. The National Disaster Management Authority, which is a central body, coordinates aid at the national, provincial, and district levels.

Main conclusions

The conclusions presented below focus on standard evaluation criteria and provide a summary of the discussion in chapter Error! Reference source not found. of the main document, which also contains further specific conclusions for each pillar of intervention.

Relevance

ECHO’s portfolio is clearly relevant both to humanitarian needs and priorities, and to its mandate. Among the many elements that contributed to relevance were:

• The conduct of independent assessments, and the consultation with target

2 Defined as severe, protracted humanitarian crisis situations where affected populations are receiving no or insufficient international aid and where there is no political commitment to solve the crisis, due in part to a lack of media interest.

3 http://www.unhcr.org/pages/49e487016.html


beneficiaries and different communities to inform the definition of the intervention;

• The great flexibility in execution, allowing to adapt the interventions to changing conditions while keeping focus on their overall objectives;

• The respect of the humanitarian principles as well as the principles of mainstreaming protection and ‘do no harm’ during design.

At times, implementing partners faced obstacles in targeting; this was due to the extremely challenging local context and to two main factors:

• Moving targets in fluid environments like large-‐scale sudden onset disasters, with large parts of the affected populations living in urban settings;

• Restrictions to access: Government restrictions to freedom of movement, and security problems resulted in gaps in the targeting of the population in need.

However, given the overwhelming magnitude of needs, all ECHO-‐financed actions still addressed relevant humanitarian needs at all times.

Appropriateness

Projects funded by ECHO in Pakistan were largely appropriate with respect to the particular needs of the affected population, especially in rural areas: they considered special needs of particularly vulnerable members of the communities; beneficiaries and wider communities were appropriately involved in project design (as mentioned under relevance) and delivery. Appropriateness of interventions in urban areas, however, was lower, due to an absence of guidance from ECHO and the scarce previous experience of implementing partners in urban settings. This represents a considerable gap in consideration of the fact that 98% of the IDPs reside in urban areas.

Connectedness

The overall intervention logic4 of ECHO in Pakistan is solid in terms of traditional humanitarian assistance. However, aspects of sustainability, LRRD and resilience building are not sufficiently captured or planned for, starting from the country strategy formulation (HIP).

As a result, there is limited evidence of linkages between ECHO funded actions in Pakistan and longer-‐term development processes, or of a specific focus on sustainability. One notable exception was the nutrition intervention in the Sindh region, where ECHO and DEVCO5 developed and are implementing clear strategies of cooperation. Effective LRRD would imply the ‘handing over’ of results of ECHO’s interventions to DEVCO or the national authorities. While the need for implementing LRRD policies is generally acknowledged, the conditions for allowing a ‘LRRD programming continuum’ between ECHO and DEVCO do not seem to be in place as each of the two DGs is bound to internal programming cycles and processes that are not necessarily consistent with those of the other DG. The ‘handing over’ of ECHO interventions to national authorities is generally not planned for;

4 A model outlining the logical relationships between the resources, activities, outputs and outcomes of a program.

5 The European Commission’s Directorate-‐General for International Cooperation and Development


however, the case of the nutrition intervention in the Sindh region is to be mentioned again as a positive exception. The difficulties in achieving a higher level of cooperation in order to strengthen LRRD is much wider than the relation between the two parent Directorates-‐General of the European Commission and touch a large number of humanitarian and development agencies. Also, the DIPECHO Programme in Pakistan was not found to provide any added value over either ECHO or DEVCO funded interventions, and its objectives do not seem to be well aligned with other ECHO initiatives. While DIPECHO was designed for providing countries like Pakistan with specific actions focussing on preparedness to better cope with future disaster, the need for preparedness is so omnipresent in Pakistan that it would be better addressed by mainstreaming it in all relevant actions rather than by addressing it with a separate and dedicated programme.

Coherence ECHO’s strategy and interventions in Pakistan are coherent with the mandate and policies of the DG, and with the wider humanitarian policy framework.

While respective theories of change and objectives of the different EU-‐funded interventions in Pakistan do not conflict with each other, their alignment and coordination is not actively pursued. ECHO’s strategy and interventions are not necessarily coherent – and in most parts not actively coordinated – with national policies and priorities in Pakistan. There are obstacles to this coordination as the national government may be guided by policies that are based on priorities other than those of principled humanitarian assistance; and may also be one of the parties of conflict. However, the national government is a key stakeholder that (by international law) is the primary responsible party in providing relief to disaster-‐affected populations on the territory of the country. In consideration of this, the assistance provided by ECHO was not fully aligned with its guiding principles of Coherence, Subsidiary and Solidarity with the national stakeholders, as priority was given to the humanitarian principles, which prescribe independence. ECHO’s coherence with the priorities and lines of actions of other donors and key humanitarian agencies is positively assessed; however, inter-‐donor coordination could be improved, especially on DRR and resilience. A higher level of operational coordination would allow better exploitation of synergies and would further improve the coherence of the overall international humanitarian approach in Pakistan.

Coverage

No significant gaps in terms of coverage were identified, apart from those relating to cases where the condition for access would involve abandoning humanitarian or other ECHO principles. The objective for a single donor (even of the importance of ECHO) to intervene on all the identifiable needs with a comprehensive coverage would be unrealistic in Pakistan. Two main aspects limited ECHO’s coverage in Pakistan:

• Funding level: Humanitarian needs in Pakistan are greater than the available funds, and appeals for funds (not only in Pakistan) are never answered with sufficient funding. ECHO contributed, substantially to the appeals for funds in


Pakistan; however, funds made available by the international community to face the humanitarian crises have been insufficient and priorities of intervention had to be put in place. This prioritisation allowed identifying the most vulnerable people; however, significant needs remained uncovered.

• Limited access: ECHO access to some areas was limited due to two factors: o Limitation to the free movement of humanitarian operators imposed by

the obligation to be accompanied by military or para-‐military escorts. This rule touched some disaster-‐affected regions such as FATA and KPK. In these cases ECHO prioritised independence of humanitarian action over the humanitarian imperative and limited the scope of its assistance.

o Limitation to access for international operators for security reasons: Interventions in such areas cannot be monitored by senior and international staff, thus creating a situation of potential conflict with the monitoring and evaluation (M&E) policy of ECHO. While there is a provision for remote or third-‐party monitoring in ECHO’s instruction note for Remote Management, this is to be used as a last resort option. Pakistan, during the evaluation period, was not considered a context that calls for this last resort. As a result, some parts of the country were not covered by ECHO’s assistance.

Efficiency

The evidence shows that ECHO’s interventions in Pakistan were efficient in terms quantity and quality of outputs. Two important factors contributed to efficiency in delivering outputs:

• The relative freedom given by ECHO to implementing partners in the selection of operational strategies and approaches to deliver. As a consequence, various transfer modalities – ranging from food or cash for work and cash & vouchers to direct delivery systems, involving the beneficiaries and very often the district authorities – were used by implementing partners, according to their consolidated experience and the suitability to the specific local context.

• The consortium approach that ECHO promoted and funded via two consortia, which overall helped 1) to improve delivery, 2) to use economies of scale in terms of a common use of assets, 3) to ensure adherence to standards, and 4) to reduce the level of overheads. Consortia, however, have like single partners also shown certain limitations in terms of being able to implement quality actions in the Pakistan context.

Two areas for improvement have been identified, which are not specific to the interventions in Pakistan but are more widely connected to the relation between ECHO and its implementing partners, i.e.:

• Timeliness of funds disbursement (particularly in the immediate onset of a crisis) is a critical factor to implementing partners, impacting on their financial capacity and – if delays occur – could potentially exclude participation of some performing organisations of a modest financial standing.

• Some implementing partners underline difficulties in complying with the application and management rules of ECHO, which are considered heavy and more demanding


than those of other donors. While it is well acknowledged that ECHO has to comply with the absolute imperative to properly manage (and account for) the use of public funds, this aspect requires a careful and organisation-‐wide analysis in terms of balancing priorities.

Effectiveness

ECHO’s interventions were largely effective; in the vast majority of cases, ECHO was able to achieve its purpose and enabled partners to deliver what they would have been unable to provide otherwise: An effective contribution to reducing the suffering of millions of beneficiaries in Pakistan. This positive assessment shall not hide the bitter reality of a country regularly affected by recurrent natural and made-‐made disasters. A consistent number of the beneficiaries that were affected by the floods in 2010 and 2012 were also displaced by conflicts, and report that they are worse off today than in 2009. Shortcomings in the sustainability of the interventions and insufficient focus on making populations more resilient to hazards are responsibilities that the international community should share.

The remarks developed under efficiency about timeliness of funds disbursement are also relevant to effectiveness: timely funding is a key prerequisite for an on-‐time and meaningful response, and thus a precondition for effectiveness. Likewise, the setting-‐up of large NGO consortia, which was praised under efficiency, had a positive effect on enhancing effectiveness as well.

Coordination Assessment of coordination is double-‐faceted.

• On the positive side ECHO has been actively promoting initiatives having a strong and positive impact on coordination. Noteworthy are the major funding of UNOCHA that allowed its scaling-‐up during the floods and the coordination of the extensive countrywide needs assessments on which the response plan was based, as well as the consortium approach in the food and nutrition sector, which further improved the coordination of the nutrition response.

• On the negative side, there was limited coordination with other EU Commission services and the National Disaster Management Authority, which limits ECHO’s ability to effectively promote resilience and long-‐term perspectives of humanitarian aid and LRRD in Pakistan. This is a limitation relevant also to ECHO’s ability to coordinate a possible and future exit strategy.

EU added value

The actions financed by ECHO in Pakistan have a clear added value in relation to actions financed by other donors. The most prominent aspects of ECHO’s added value are the direct consequences of its identity as an EU body, in comparison with initiatives undertaken by bilateral donors, i.e.:

• The principled stance that ECHO defends; ECHO has operated as an exceptionally principled donor and – by leveraging on its EU, super-‐national identity – adhered to principles of independence and neutrality more than other donors. ECHO manages to enforce the respect of the same principles by its implementing partners as well; in the long-‐term this is expected to strengthen the ability of


implementing partners to deliver in the context of complex conflicts.

• ECHO’s independence — the strict absence of a further-‐reaching political agenda to its humanitarian aid.

• ECHO’s financial influence, which allows not only to meet needs of a highly significant scale, but also to some extent to enforce a more coordinated, coherent and quality-‐driven approach on implementing partners.

• Finally, its contribution (especially in the areas of health and water as ECHO’s response strengthened local capacity and built on existing systems) to strengthen quality and sustainability of the overall humanitarian response.

Recommendations The recommendations presented in this Executive Summary are a summary of what is presented in chapter Error! Reference source not found. of the main report, which also contains a short rationale to each recommendation and develops further the text of the recommended measures. The report issues six recommendations of a strategic nature, and four of an operational nature. Recommendations of a strategic nature

S1) To strengthen LRRD in Pakistan. In order to achieve this crucial objective the following measures are suggested:

• To ECHO in cooperation with DEVCO o To reconsider their respective working practices in Pakistan in order to

enhance cooperation in disaster management6. To adopt an approach to disaster management that is consistent with key-‐sector policy frameworks spanning from preparedness to risk reduction, response, recovery and finally to development (including both enhancing of preparedness and risk reduction).

o To develop a common ECHO-‐DEVCO framework per sector to envisage actions to increase resilience to future hazards, envisaging synergies and complementarities of interventions of the two parent DGs.

o Besides the ongoing collaboration in nutrition, to step up collaboration in education; agriculture/ rural development; and displacement and in support to Afghani returnees and IDPs.

o To cooperate for achieving the objective to include planning for a continuum approach (i.e. linking ECHO and DEVCO actions) into the next-‐coming Multiannual Financial Framework (MFF).

o To work with DEVCO on exit strategies, primarily in the nutrition sector in Sindh and in the Afghan refugee response (see S4 below).

• To ECHO in cooperation with the Instrument contributing to Peace and Stability (IcPS)

6 Disaster Management can be defined as the organization and management of resources and responsibilities for dealing with all humanitarian aspects of emergencies, in particular preparedness, response and recovery in order to lessen the impact of disasters.


and the EU Delegation (EUD) o To maintain an active and regular dialogue with the External Action Service

(EEAS) on country programming, involving both the Desk in Brussels and the Delegation.

S2) To develop concrete scenarios and strategies on how to apply the humanitarian principles in complex crises. The following measures are suggested:

• To ECHO HQ in cooperation with other humanitarian donors

o To develop concrete scenarios and strategies of collaboration for complex contexts where humanitarian access is restricted in some parts of the Country but humanitarian needs are so immense that the strict application of the principles of neutrality and independence can have major impacts on the needs-‐orientation and effectiveness of the response.

• To ECHO HQ in cooperation with other EU bodies o To develop, together with DEVCO and the EEAS a better understanding of the

possibilities and limitations of inter-‐service collaboration at country level to leverage the combined possibility to influencing parties restricting humanitarian space.

• To ECHO HQ o To consider, subject to conditions, Third-‐Party Monitoring as a viable

substitute modality for conducting monitoring in areas of no access for international humanitarian operators or for direct implementing partners.

S3) To develop an explicit strategy on advocacy and engagement with Government, together with partners.

• To ECHO HQ o To consider relationship management with government stakeholders –

explicit and refine the strategy to collaborate with civilian authorities and at technical level – as part and parcel of the humanitarian response in Pakistan, and develop an explicit strategy on whether and how to engage with the political representation of the Government itself on questions of humanitarian space. This can take form of direct or indirect (through partners) advocacy.

o To invest into an active and strong relationship with its direct counterpart, the National Disaster Management Authority, searching for common grounds and launching systematic advocacy efforts for widening up the humanitarian space. To this end, to consider the delivery of specific capacity building initiatives.

• To ECHO HQ in cooperation with implementing partners o To cooperate with NGO and UN partners to develop an advocacy strategy,

both on the issue of humanitarian space in Pakistan and on issues relating to the root causes of vulnerabilities in Pakistan.

• To ECHO HQ in cooperation with other donors and the EUD o To explore the leverage that could be deployed in furthering issues of prime


humanitarian concern with the Government, which ECHO considers itself unable or inappropriate to tackle.

S4) To hand over the Afghan refugees portfolio to DEVCO.

• To ECHO in cooperation with DEVCO

o To phase out the humanitarian assistance to Afghan refugees and hand it over to DEVCO. DEVCO should take on the portfolio and programme an intervention with a long-‐term perspective and an initial priority on education and livelihoods strategies.

S5) To discontinue DIPECHO funding for Pakistan

• To ECHO HQ o To discontinue DIPECHO funding, at least in the format in which it was

delivered during the evaluation period. To assess all projects funded by ECHO against their potential to deliver towards national capacity strengthening for disaster preparedness and risk reduction; and to include such a component in the relevant actions.

o Should ECHO decide not to discontinue DIPECHO in Pakistan, to define a long-‐term strategy detailing the expectations from the programme in terms of strengthening capacities and capabilities for disaster preparedness and risk reduction.

S6) To develop a clear policy for the delivery of humanitarian assistance in urban settings

• To ECHO in cooperation with implementing partners

o This policy needs to be clearly communicated to implementing partners in terms of practical consequences in terms of available skills and expertise.

o Selection of partners to implement interventions in urban settings should value previous experience in comparable crises.

Recommendations of an operational nature

O1) To capitalise more on partnership

• To ECHO in cooperation with DEVCO

o To jointly call for annual consultations at country level with partners to discuss their interventions in the protracted and recurrent crisis context in Pakistan at a more strategic level. This would allow envisaging opportunities to bridge the gap between humanitarian and development funding.

• To ECHO o To involve partners in consultation at the drafting stage of the country HIP to

capitalise on combined insights for a more detailed conflict analysis and a review of protection challenges.

O2) To maintain a presence in Pakistan

• To ECHO o To maintain an office in Islamabad and not to decrease the funding level for


meeting humanitarian needs in Pakistan. Should humanitarian space not widen up, ECHO should consider to accept applying the full remote management approach to its funded portfolio, including M&E via Third Party Monitoring (see S2).

o Should ECHO, however, decide to close down its presence in Pakistan due to lacking humanitarian space, it should do so publicly, drawing wide attention to its decision.

O3) To ensure timeliness of funding

• To ECHO o Although the need for accountability is undebated at all levels of funding,

some funding decisions should be agreed using a fast-‐track process to ensure timely mobilisation of funds. This could go along with conditionalities to be fulfilled by implementing partners in order to get the full amount of funding. However, conditionalities shall not place on partners the financial burden to pre-‐finance actions.

o In order to address the issue of funding gaps between different project cycles ECHO is advised to assess the possibility of a ‘funding-‐bridge’ for interventions funded from year-‐to-‐year.

o Finally, ECHO is recommended to consider possibilities for multi-‐annual funding of programmes that require longer-‐term interventions in relation to protracted crises.

O4) To design the evaluation period closer to the end of implementation phase

• To ECHO o For a country with such a large number of events and humanitarian

interventions, the evaluation period should not exceed three years. Ideally, the evaluation should coincide with the last 3 months of the up-‐to three-‐year period to ensure that the informants are still available and their memory still vivid on the topics to be covered by the evaluation.

Executive Summary EN - European...

Documents

Transcript of Executive Summary EN - European...