Humanitarian aid evaluation at Medecins sans Frontieres

Sabine Kampmüller, MIHMSF Vienna Evaluation Unit

http://evaluation.msf.at

Dimensions of evaluation:

Source: SDC (2002)

Evaluation post-disaster: Haiti earthquake response

Evaluation conducted 5 – 10/2010

Evaluation process

1st time (!) all section Review 6 specific reviews: Global/operational,

medical/surg, log/supply, Comms, FR, HR

Quantitative & qualitative methods data, field visits, interviews, qualitative

research - beneficiaries, web survey -staff

Limitations: Incomplete and non-uniform data Recall problems

Patients / Community’s Perception Very positive perception of MSF´s

interventions and services Fear/experience of stigmatisation,

increase of violence in the camps  Lack of space for occupational and

educational activities.   More attention to socio-cultural and

spiritual needs

Haiti evaluation findings: MSF was one of the biggest emergency

health actors Operational choices in line with emergency

needs; less coherent over time Common data collection difficult to

impossible International platforms / working groups

are underused / undervalued Successful adocacy on some issues,

complicated decision making prevented more

Main recommendations:

1. Organize inter-section capacity for assessment and monitoring of evolving needs and assistance

2. Define strategy on mass casualty 3. Revise emergency supply strategies4. Ensure uniform data collection5. Strengthen technical working groups6. Focus on organisational learning7. Engage with national and international actors

Thematic Evaluation: Response to displacement

Evaluation conducted2009/2010

Evaluation process

Comparison of 6 case studies (urban/rural, low/middle income, etc)

Quantitative & qualitative methods Literature research, document/tools & data

review, field visits, interviews,

Limitations: Availability of data and key people Poor documentation

Particularity of open settings

Lack of clear boundaries : Geographical spread, invisibility, needs difficult to

identify and measure, protection issues

Displaced settle in environment with available resources and existing health system Better survival capacities, but deterioration likely

Protracted, chronic or intermittent character: Mortality near-normal levels (pre-emergency), might rise

slowly over time

Findings on assessment

Complexity of open settings requires more attention to assessment

Critical aspects of information missing Quantitative information difficult to obtain Health system issues, access barriers little addressed

Concerns on the use of surveys Views of displaced and host often omitted Capacities and vulnerabilities not

assessed

Recommendations on assessment

Develop innovative assessment approaches for inaccessible areas

Distance assessment“ Promote systematic use of qualitative

methods Adopt concept of “continual” assessment Develop a frame to assess vulnerabilities,

capacities and coping Provide better support and guidance

assessment toolbox, experts, training

Findings on intervention

Engagement with the health system a main challenge

Outreach workers invaluable, set ups improvable

Non-medical assistance is marginal Overambitious targets of coverage Strategy adapted to the level of

emergency

Recommendations on intervention

Need for new intervention frameworks Adopt existing models

Generalize the practice to cover both displaced and host where appropriate

Define the criteria / scope of “light support”

Develop community based strategies

Evaluation criteria (donors)

Relevance Effectiveness Efficiency Impact Sustainability

Appropriateness (Connectedness) (Coherence)

Timeliness Coordination Coverage

Adopted from: Hallam, A. Good practice review (ODI), 1998