External Evaluation of Christian Aid’s-ICCO Funded Ebola Emergency Response Project

Final Draft Report

December 2015

Submitted By

Social and Demographic Research Solutions

i

Table of Contents KEY ACRONNYMS ..................................................................................................................................... ii

EXECUTIVE SUMMARY ............................................................................................................................ iii

1.0 Introduction .................................................................................................................................. 1

1.1 Evaluation and Project Background .............................................................................................. 1

1.3 Objectives of the Evaluation ......................................................................................................... 2

2.0 Methodology ................................................................................................................................. 2

2.1 Research Methodological Approach ............................................................................................. 2

2.2 Data Collection Tools Design......................................................................................................... 2

2.2.1 Evaluation Tools .................................................................................................................... 2

2.3 Scope of the Evaluating and Sampling .......................................................................................... 3

2.3.1 Sample size ............................................................................................................................ 3

2.4 Field Work ..................................................................................................................................... 4

2.5 Data Quality, Management and Analysis ...................................................................................... 4

2.5.1 Quantitative Data .................................................................................................................. 4

2.5.2 Qualitative Data .................................................................................................................... 4

3.0 Evaluation Findings ....................................................................................................................... 5

3.1 Introduction .................................................................................................................................. 5

3.2 Demographic Characteristics of Targeted Beneficiaries ............................................................... 5

3.3 Project Management and Distribution Methods .......................................................................... 5

3.4 Target for Food Items ................................................................................................................... 6

3.5 Food Items Component ................................................................................................................ 7

3.5.1 The Efficiency of Food Items Distribution ............................................................................. 7

3.5.2 Food Items Distribution Effectiveness .................................................................................. 9

3.5.3 The appropriateness of Food items distributed ................................................................. 11

3.6 Beneficiary Target for Non Food Items (NFIs)............................................................................. 12

3.6.1 Appropriateness of Non Food Items ................................................................................... 13

3.6.2 Effectiveness of NFIs Distribution ....................................................................................... 14

3.6.3 The Efficiency of NFIs Distribution ...................................................................................... 15

3.7 Cash Transfer Component .......................................................................................................... 16

3.7.1 Target for Cash Transfer...................................................................................................... 16

3.7.2 Cash Transfers Effectiveness ............................................................................................... 17

3.7.3 Cash Transfer Component’s Relevance and Appropriateness ............................................ 18

3.7.4 Justifying the Cash Transfers in the Ebola Emergency Response ....................................... 18

3.8 Observing Humanitarian Standards ............................................................................................ 19

3.8.1 Payment of cash or in-kind for the FIs received ................................................................. 19

3.8.2 General respect accorded to beneficiaries ......................................................................... 20

3.8.3 Knowledge of Complaint Procedures.................................................................................. 20

4.0 Limitations of the Evaluation ...................................................................................................... 21

5.0 Conclusion and Recommendations............................................................................................. 22

5.1 Conclusion ................................................................................................................................... 22

5.2 Recommendations ...................................................................................................................... 23

ii

KEY ACRONNYMS

CA Christian Aid

CAHSec Community Action for Human Security

CSB Corn Soya Blend

CsPro Census and Surveys Programme

DERC District Emergency Response Committee

EAs Enumeration Areas

EVD Ebola Virus Disease

FGDs Focus Group Discussions

FIs Food Items

HH House Hold

ICCO International Christian Community Organization

IDI In-Depth Interview

IPC Infection Prevention and Control

KII Key Informant Interview

MCSL Methodist Church Sierra Leone

MSWGCA Ministry of Social Welfare, Gender and Children's Affairs

NETHIPS Network of HIV Positives in Sierra Leone

NFIs Non Food Items

NGO Non-Governmental Organization

OVC Other Vulnerable Children

RADA Rehabilitation And Development Agency

REWAP Real Women in Action Programme

SADRS Social and Demographic Research Solutions

SEND Social Enterprise Development Foundation

SLSAV Sierra Leone Social Aid Volunteers

SPSS Statistical Package for Social Scientists WHO World Health Organisation

iii

EXECUTIVE SUMMARY

Introduction

Christian Aid Sierra Leone (CA), with funding from ICCO, was able to respond to the Ebola Virus Disease (EVD) epidemic through implementing partners in nine districts1 in Sierra Leone. The ICCO-funded EVD response project provided emergency relief assistance and also carried out various activities in support of Ebola survivors, people in quarantined homes, elderly caregivers, orphans and other vulnerable children (OVCs). The response aimed to provide water, food and Non-Food Items (NFIs) to quarantined homes to avoid the need to break the quarantine in search of essential supplies thereby reducing the risk of spreading the infection. The response also aimed to provide food, cash and other basic needs to EVD survivors and victim families aimed at reducing the stigma surrounding those who had contracted the disease and supported their reintegration back into their families and communities. Additionally, CA’s Ebola response project mobilised and trained traditional healers in infection prevention and control (IPC), as traditional healers were perceived to be a major source of transmission. Recognising that some children could be orphaned or their families impoverished as a result of the accompanying restrictions, CA also provided teaching and learning materials to these children, especially in the Tonkolili district. Methodology The evaluation used both qualitative and quantitative methods to collect data. For the qualitative methods, three focus group discussions were conducted in each of the six districts: adult male, adult female and mixed youth groups. In addition to the focus group discussions, three in-depth interviews were conducted in each of the six project evaluation districts, with implementing partner (IP) staff, community leaders, District Ebola Response Centre (DERC) or a member of the Social Welfare, Gender and Children’s Affairs Ministry. In total, eighteen focus group discussions and eighteen in-depth interviews were conducted across the six evaluation districts. Household questionnaires were also designed to collect quantitative data from beneficiaries of the intervention. In total, 313 questionnaires were administered with Port Loko having the least number of questionnaires (35) and Kailahun the highest (91). The sampling was simply based on proportion to size of the intervention in the district. Quantitative data was entered using CsPro and exported into SPSS v.20 for analysis. For the qualitative data, interviews and discussions were listened and analysed using simple framework analysis. Key Findings About two-thirds of the respondents (67.1%) were survivors while about the same number (63.1%) were household heads of their respective households. Project Management: The study found that there were limitations with coordination and monitoring of the overall implementation by CA was not consistently done. The IPs sent periodic reports to CA but response to challenges in the field were either sent late or sometimes not sent at all. Although internal records were maintained by the IPs for reference, yet the delays made it difficult to share lessons learnt in real time. Food Received: About 9 out of 10 (87.9%) respondents reported having received food supply from a Christian Aid partner while at least 1 out of 10 reportedly never received any food supply from any CA partner. Glaringly, just about 6 out of 10 respondents (60.4%) in the Bo district reported to have received food. For the districts in the north, all the respondents reportedly received food supply.

1 Bo, Bombali, Kailahun, Kambia, Port Loko, Pujehun, Tonkolili, Western Area (Rural) & Western Area (Urban)

iv

Timeliness of Support: Nearly half of the respondents (47.9%) felt that the support was not timely particularly when most people received their items at least few days into quarantine. Just over half of the respondents (52.1%) perceived the support to be timely. There were examples of whole districts where food supply was done after quarantine which could be seen as untimely, although some of the items were not strictly for quarantined homes. This however also raises questions of efficiency of the intervention. Availability and Adequacy of items: Generally, respondents reported that most needed items were available and adequate in the package. Similarly, about 90% of respondents reported that the items supplied were appropriate and rated their quality highly. This positively responds to the question of effectiveness of the response. Cash Transfers: Apparently, one laudable intervention was the cash transfers. This was so because it gave the beneficiaries the option to buy what they wanted and to also use it to generate further income. It slightly deviates from the usual prescriptive approach of aid agencies. Although the money was meant for income generation, giving out money in crisis where survival was key led people to convert the money for personal upkeep, as was originally thought to be the purpose by the beneficiaries. Respecting Humanitarian standards: the three parameters used to assess humanitarian standards were: respect, payment/bribery and complaint procedures. Respondents rated the first two very highly; that means the workers and process were very respectful and that they hardly paid anything in kind or cash to receive the services. However, nearly 9 out of 10 respondents reported that they had no knowledge of complaint procedures, in case they were violated during the process. Key Recommendations

More robust monitoring: Although some monitoring was reportedly done, the study’s findings indicate that CA could improve on their monitoring of IPs during implementation to share lessons learnt in real time.

IPs to be carefully selected and trained: Although the project was implemented during a crisis period, a more careful selection process could prevent changing IPs midway through the project. Where capacity is questioned, some training to enhance capacity could help.

More IP involvement in training: Usually, in emergency situations a few people plan and take decisions, as procrastination is an enemy of progress; but more involvement of IPs in planning would help CA understand local contexts better and IPs feel as true ‘partners’.

Procurement to be decentralised and packages more need-specific: Where possible CA would want to consider, in future, decentralising procurement of items to suit local needs and packages should not be one-size-fits-all.

Amount of cash increased and purpose made known: Although half a loaf is better than none, CA would want to consider increasing the amount of money given to beneficiaries for income generation if the purpose is indeed to start a business. Also, the purpose of the cash should be made unequivocally clear from the onset so that potential beneficiaries do not misunderstand it to be pocket-money.

Complaint procedures to be made known to beneficiaries: Because of the duty of care, where feasible, CA should consider increasing awareness of their prospective relief aid beneficiaries of complaint procedures in case they suspect abuse of the process.

1

1.0 Introduction

1.1 Evaluation and Project Background This report discusses the evaluation findings of Christian Aid’s ICCO-funded Ebola response project, implemented through various partners across the four regions of the country. Christian Aid Sierra Leone (CA), with funding from ICCO, was able to respond to the Ebola Virus Disease (EVD) epidemic through implementing partners in nine districts2 in Sierra Leone. The EVD response provided emergency relief assistance and also carried out various activities in support of Ebola survivors, people in quarantined homes, elderly caregivers, orphans and other vulnerable children (OVC). As the Ebola Virus continued to spread, the government and its partners focused mainly on curtailing further spread and their main area was to ensure safe burials and discouraging crowds. For some time during the response, psychosocial care did not receive the required attention. As people continued to bury their relatives in secret—accusing the government’s response team of undignified burials, thereby disrespecting ingrained cultural practices, other players including NGOs and civil society organisations saw the need for a strengthened and organised community engagement in the process. Although the Ministry of Social Welfare, Gender and Children’s Affairs (MSWGCA) officially headed the psychosocial aspect of the EVD response, they were not adequately equipped—both with logistics and capacity—to sufficiently handle the complex psychosocial issues that accompanied the Ebola outbreak. CA then saw the need to bolster the Ministry in contributing to support vulnerable populations such as orphaned children, survivors, elderly caregivers, widows and quarantined households. The response aimed to provide water, food and Non-Food Items (NFI’s) to quarantined homes to avoid the need to break the quarantine in search of essential supplies thereby reducing the risk of spreading the infection. The response also aimed to provide food, cash and other basic needs to EVD survivors and victim families aimed at reducing the stigma surrounding those who had contracted the disease and supported their reintegration back into their families and communities. The Ebola response got to a point where traditional healers were identified as a significant source of transmission, as they treated people with almost any disease. In some parts of the country, traditional healers, who are also community opinion leaders, are more regarded than their orthodox scientific medical counterparts; hence it was believed that some people avoided accessing health facilities but instead consulted traditional healers for their health solutions. The situation became worse because, at the initial stage of the outbreak, responders to the disease claimed there were little or no chances of surviving Ebola. Traditional healers, on the hand, always welcomed sick people, even when they were apparently convinced they could not cure them. Recognising this, Christian Aid mobilised traditional healers and trained them in Infection Prevention and Control (IPC) to help break the chain of transmission. This was an awareness-raising campaign by Christian Aid—if the traditional healers know of the dangers of their interventions, they could stop treating sick people and would refer them to the appropriate healthcare facility, hence breaking the chain of transmission. This report would therefore assess the effect of Christian Aid’s interventions on the people, the communities and the response generally.

2 Bo, Bombali, Kailahun, Kambia, Port Loko, Pujehun, Tonkolili, Western Area (Rural) & Western Area (Urban)

2

1.3 Objectives of the Evaluation The evaluation objectives of the EVD response project among others included:

a. To assess and validate the outcomes and impact of the response from the standpoint of beneficiaries and partners

b. To capture and synthesise Christian Aid (CA) and partners’ learning from the response

2.0 Methodology

2.1 Research Methodological Approach This research was a mixed-sectional evaluation conducted using participatory methods involving both quantitative and qualitative approaches. Desk review of relevant project documents was done for background information to add to the knowledge base of the evaluation team.

2.2 Data Collection Tools Design The evaluation data collection tools were developed through participatory approach; the consulting firm

Social and Demographic Research Solutions (SADRS) prepared a draft of the evaluation tools and Christian

Aid’s project management team consulted before the final tools for data collection were adopted. The tools

were further reviewed during the training and translated by the enumerators to have a common

understanding.

2.2.1 Evaluation Tools

Household (HH) questionnaires: A single designed Household questionnaire was used to collect data

at household level of beneficiaries of Food Items (FIs), Non Food Items (NFIs), Teaching and Learning

Materials, and Cash Transfers.

Key Informant or In-Depth Interview (KII) guides: One KII question guide (with separate and clearly

defined themes/sections for each interviewee/institution) was used for the three components of the

project. The questions were tailored to collect nuanced information that could not be obtained by

the HH closed-ended questions.

Focus Group Discussion (FGD) guide: A single FGD guide was used to obtain information from direct

and indirect project beneficiaries to determine their perception of the support received through the

project.

Selected participants for the FGDs were grouped into three demographic categories: youths

comprising both sexes– so that they are allowed to express their views amongst their peers, adult

female and adult male groups. The reason for separating the adult men from the women was to

avoid male dominance in the discussions. Local opinion leaders/chiefs were not recruited into the

focus group discussions as they could influence the views of the other group participants.

3

2.3 Scope of the Evaluating and Sampling The scope of the evaluation exercise was limited to selected project beneficiary communities in six districts.

The targeted beneficiary communities/ towns served as Enumeration Areas (EAs) for the administration of

household questionnaires and the FGDs.

The selection of districts was purposive, as some project outcomes were only applicable in particular

districts. In this case, districts and research participants were selected for what they are, where they are and

what was their planned outcome. The six selected districts out of the nine project districts are: Tonkolili, Port

Loko, Kambia, Kailahun, Bo and Western area (Rural).

2.3.1 Sample size

As mentioned in the technical proposal to Christian Aid, it was agreed that 300 household questionnaires

would be administered across the six sampled districts. However, while in the field, the research team were

able to administer 313 household questionnaires and efforts were made by the team to ensure that all

categories of beneficiaries (including survivors and elderly caregivers) were included without complicating

the sampling methodology. The (quantitative survey) sample size was then proportionately allocated as per

number of beneficiaries in each district—Bo: 56, Kailahun: 91, Kambia: 45, Port Loko: 35, Tonkolili: 46, and

Western Area (rural): 40.

Table 1: Determining the Sample size

District Number of Beneficiaries (n)

Proportion (p=n/N)

Sample size

(P*300)

Actual administered HH questionnaires3

Bo 67 0.16 47 56

Kailahun 150 0.35 105 91

Kambia 62 0.14 43 45

Port Loko 50 0.12 35 35

Tonkolili 50 0.12 35 46

Western Area 49 0.11 34 40

Total(N) 428 1 300 313

3 While in the field especially Kailahun District, it was discovered that there were internal movement of beneficiaries between

localities/towns as well movement of beneficiaries to other district/regions making it difficult to trace the beneficiaries. This necessitated to add to the sampling size to cater for those initially sampled but were unavailable

4

2.4 Field Work Data collection was conducted for an average of 8 days for both household questionnaires and FGDs by

enumerators4. The field work was organized in such a manner that household questionnaires were

administered first in the project communities. Key Informants were selected based on their perceived in-

depth knowledge and experience of the project. People with in-depth knowledge of the project were

interviewed from the IPs, district field staff of the Ministry of Social Welfare, Gender and Children’s Affair

(MSWGCA), a member of the District Emergency Response Committee (DERC), and local authorities were

each interviewed as key informants. The key informant interviews were conducted by SADRS’ research

coordinators. Three FGDs and KIIS, each, were conducted in each district. All KIIs and FGDs were audio-

recorded.

2.5 Data Quality, Management and Analysis The data was managed using standard systemised techniques.

2.5.1 Quantitative Data

To ensure data quality, all administered household questionnaires (by enumerators) were checked by

research coordinators in the field for review and coding, where applicable. This ensured the correct

questions were asked and issues are addressed by the relevant enumerator while still in the field.

All completed and verified questionnaires were then sent to Freetown to the data inputting clerks for data

entry. Quantitative data entry was done using Census and Surveys Programme (CsPRO) version 7.0 statistical

software. Upon completion of the data entry, data was exported to Microsoft Excel for further cleaning to

ensure that there were no additional errors generated at data entry stage. Double entry of data was done to

check for data entry errors and where disparities were identified, the original completed questionnaire

would be used to correct the anomaly.

The Excel cleaned data is then exported to SPSS version 20.0—a statistical software for data analysis and

generation of tables and charts.

2.5.2 Qualitative Data

Qualitative data was analysed manually; that is, the analysis of qualitative data from KIIs and FGDs included

listening to the audio-recorded conversation files and writing the information verbatim, arranging the

information by key evaluation categories (impact, efficiency, appropriateness, and effectiveness etc.), and

direct quoting. These were finally organised as per project main interventions and used to complement the

quantitative data in the report.

4In Bo and Port Loko districts, data was collected for 9 days each

5

3.0 Evaluation Findings

3.1 Introduction The findings of this evaluation are presented by project intervention components and based on the criteria of

evaluation. The demographic characteristics such as family size and gender are first presented. The section

will consider the findings of each of the components, that is, the Food Items (FIs), Non Food Items (NFIs),

Cash Transfers, and Teaching and Learning Materials (TLMs). The data is presented in graphs, tables and

figures, as deemed appropriate.

3.2 Demographic Characteristics of Targeted Beneficiaries A total of 313 respondents were interviewed, where 150 (47.9 %) were female and 163 (52.1 %) were male.

Majority (67.1 %) of the respondents were EVD survivors, 40 (12.8%) of the respondents were elderly

caregivers, and 63 (20.1 %) of the respondents were other categories of beneficiaries. In addition, 200

(63.9%) of targeted respondents were household heads, where 118 (37.7%) of the household heads were

males and 82 (26.2 %) of the household heads were females, highlighting the vulnerability and thus likely

dependency of humanitarian assistance for food and other basic needs during the EVD outbreak.

When asked about the size of their households during the quarantine period, 170 (55.0 %) of the

respondents reported having between 6 – 10 members in their households during the quarantine period.

The evaluation also sought to know the number of household members categorized by age groups that

benefited from CA Ebola emergency response project. The result shows that 23.3 percent of the project

beneficiaries were within the age cohort 0 -5 years, 21.9 percent of the beneficiaries were within the cohort

of 6 – 17 years, 17.6 percent were in the age group 51-56, and 16.0 percent of the beneficiaries were 65

years and above.

Figure 1:Percentage of beneficiaries by age groups

3.3 Project Management and Distribution Methods For Christian Aid to achieve their objectives, there were six key areas of intervention

Distribution of food items

Distribution of non-food items

6

Textbox 1(3.4) ‘’…most people received food support from SLSAV most

especially the survivors and elderly caregivers of vulnerable children in Kambia. The supply reached the people, especially the orphans’’

FGD Adult Female, Kambia

Direct cash transfers

Distribution of teaching and learning materials

Mobilisation and training of traditional healers

Engaging vulnerable groups in income-generation activities For analysis purposes, some of these areas will be merged. The evaluation assessed Christian Aid’s general management of the intervention and found that although CA monitored their Implementing Partners (IPs), it was however weak and they could improve on it. For example, up to the time of the evaluation, CA did not have a comprehensive list of beneficiaries for all districts. They had to rely on the individual IPs for a list which was eventually never comprehensive. Expectedly, CA should be in possession of such list as it is an audit issue. Secondly, there seemed to be limited coordination between and amongst IPs. As a result, there was hardly any sharing of lessons-learnt by the IPs during programme implementation. Although this could be done after the evaluation, it would have been more useful if done in real time. One IP could not learn from the mistake of another when lessons learnt are shared after programme implementation. It should however be noted that for the most part internal records were kept by individual IPs. Another management concern was the selection of the items in the package. One would have expected CA to review the items as the outbreak unfolded and new issues arose but been that the procurement process was centralized, CA stuck with the same package thereby limiting variety and choice. IPs were hardly involved in the planning phase of the project thereby including activities and items that were not context-sensitive. If IPs were involved in the procurement process, for example, appropriate items and potential delivery delays would have been addressed. Lastly, there were capacity issues with IPs. Although they mostly were able to cope, they were however not very prepared, capacity-wise, to handle a complex emergency situation as that of the Ebola response. CA would want to consider providing some training for their IPs in similar circumstances in future.

3.4 Target for Food Items From the quantitative data, interviews and focus group discussions, it was evident that the project was

successful in targeting quarantined homes, survivors, elderly caregivers and other categories of beneficiaries.

86.9 per cent of the respondents confirmed that they received support from one of the Christian Aid Ebola

emergency response project implementing partners. 13.1 per cent of the respondents reported that they did

not get any support from the CA project IPs.

Further analysis showed that 60.4 per cent of the respondents interviewed in Kailahun District acknowledged

receipt of food support from a project IP. In Bo

District, 94.6 per cent of the respondents

confirmed support from the project IP whilst in

Port Loko, Kambia and Tonkolili districts all the

respondents interviewed acknowledged receipt

of FIs from the project IPs. For respondents

interviewed in the Western Area Rural, 95 per

cent confirmed receipt of food support from the project IP. Overall, the proportion of respondents that

acknowledged receipt of food support is over 80 per cent and this shows that the project met its objectives.

7

This point is illustrated by a sentiment expressed by an adult female participant in a focus group discussion in

Kambia exemplified in textbox 1 above.

As a possible case for the low percentage of respondents in Kailahun District, it was gathered from the IDIs

conducted with the IPs from that district (MCSL and SEND) that support services were initially given by MCSL

but later transferred to SEND because the former could not submit to CA some required financial documents

in time. This change, as was acknowledged by the IPs, might have affected the smooth running of the project.

In that vein, after Social Welfare and the relevant IPs had initially agreed on a list of beneficiaries (which was

forwarded to CA), the IPs found out in the field that some these beneficiaries were already on other

agencies’ (such as World Health Organisation: WHO) distribution lists. To avoid duplication and waste of

resources, these prospective beneficiaries were not given the CA supplies. Other survivors and OVCs were

therefore given the items. The list of the additional/replaced beneficiaries were not supplied to CA—one

reason why some beneficiaries denied receiving food items as those were the ones dropped for appearing on

other IP lists for food and non-food distribution.

3.5 Food Items Component

3.5.1 The Efficiency of Food Items Distribution

The efficiency of FIs distribution component of CA’s Ebola emergency response project was assessed against

timeliness of the support and how long quarantined homes had to wait to receive FIs from project IPs.

Timeliness of the support: The evaluation found that 52.1 per cent of the respondents stated that the food

assistance they received from the project implementing partners was timely and 47.9 per cent of the

respondents reported that the support was untimely. Figure 2 below shows the respondents’ opinion about

the timeliness of the food support.

Figure 2: Timeliness of food support by district

Respondents were further asked how long they had to wait during quarantine period to receive food items

from IPs. Overall, the result shows that 14.1 per cent of the respondents stated that they received food

supply from IPs the same day of quarantine whilst 15.0 per cent of the respondents reported that they

received the FIs one day after the quarantine. 12.5 per cent of the respondents interviewed confirmed that

they received FIs two days after their houses were quarantined whilst 58.1 per cent of the respondents

8

acknowledged that they received FIs after two weeks into the quarantine period. 0.3 per cent of the

respondents reported that they received their FIs before their homes were placed under quarantine. The

chart below shows how long quarantined homes had to wait to receive FIs.

Figure3: How long quarantine homes had to wait to receive food supply

In-depth analysis of the evaluation data indicates that in Kambia District 2.2 per cent of the respondents

reported that they received FIs before their houses were placed under quarantine. 82.9 per cent of the

respondents in Port Loko District confirmed that they had to wait more than one week into the quarantine

time before receiving FIs from project IPs; and similarly, 62.5 per cent of the respondents in Bo District stated

that they had to wait for more than a week before receiving FIs from RADA. A good proportion of

respondents in Tonkolili and Western Rural confirmed receipt of FIs the same day their homes were placed

under quarantine, although this negates the IP interviews. The table below shows the percentage of

respondents and the time they had to wait to receive food supply form project IPs.

Table2: How long quarantine homes have to wait to receive food supply

Source: Evaluation household questionnaire data

Textbox 2 below is a statement made by an adult male in a Focus Group Discussion (FGD) in Port Loko District

expressing sentiments about when quarantined homes received food supply.

District Before quarantine

Same day of quarantine

One day after quarantine

Two days after quarantine

More than One week into quarantine time

Percentage of respondents confirming food supply

Kailahun 1.1% 2.2% 57.1% 60.4%

Bo 14.3% 8.9% 8.9% 62.5% 94.6%

Port Loko 8.6% 8.6% 82.9% 100.0%

Kambia 2.2% 11.1% 40.0% 37.8% 8.9% 100.0%

Tonkolili 37.0% 34.8% 13.0% 15.2% 100.0%

Western Rural

35.0% 10.0% 15.0% 35.0% 95.0%

Textbox 2 (3.5.1) ‘’The quarantined people received food after the quarantine period. When the first houses were quarantined the homes received food from local authorities most especially the Paramount Chief. When the second phase of quarantine happened, the SLSAV started supplying food items. SLSAV was the first NGO who intervened during the second phase of the quarantine. They did a lot in this community.” FGD Adult Male, Port Loko District

9

Textbox 4 (3.5.2) ‘’The CA project implementing partner SLSAV did a

very good job in the distribution of food items. Even though we were under quarantine and our neighbours were afraid to come close to give us help the SLSAV field staff took the food items to us and talk to nicely’’’ FGD Adult Female, Port Loko

Textbox 3 (3.5.2) ‘’The food supplied to us was good and enough, most

especially the rice and sardine. I regained my normal body status after three weeks when I was discharged from the treatment canter. I was eating enough and fine’’. FGD Mixed Youth, Bo District

3.5.2 Food Items Distribution Effectiveness

From the inception of the CA Ebola emergency response project, one of the desired goals of the FIs distribution component of the project was to address immediate food needs for quarantined homes, EVD survivors, orphans and other vulnerable people. It was also geared towards improving the nutritional status of beneficiaries, most especially orphans. This was mostly achieved as demonstrated in Textbox 3. The overarching5 goal of the FIs distribution by CA was to ensure that Ebola transmission is reduced by ensuring that quarantined homes had access to adequate and appropriate food and to ensure that quarantined homes didn’t have to break quarantine restrictions in search of food. The evaluation assessed the effectiveness of the FIs distribution by looking at the quality of the FIs distributed, the method used by implementing partners in the distribution FIs, the availability and adequacy of food items distributed and how organized were the field staff in the distribution process. The foregoing factors were examined in the paragraphs below. Quality of Food items distributed: Nine6 food items that were in the FIs distribution package were assessed to know the quality of FIs distributed to beneficiaries. Respondents were asked about the quality of the FIs they received. From the evaluation result, majority (92.6%) of the respondents stated that the quality of the FIs distributed by CA project IPs was good. 7.0 per cent of the respondents stated that the quality of the FIs distributed was reasonable, and 0.4 per cent of the respondents reported that the quality of the FIs distributed was poor. The textbox (3) above is a statement made by an FGD participant which reflects the level of satisfaction with the quality of FIs supplied by IPs. Assessing the distribution method: Beneficiaries were asked to judge the method used by IPs in the distribution of FIs to quarantined homes and other project beneficiaries. From the household data 24.6 per cent of the respondents interviewed stated that the method used by IPs in the distribution of FIs to project beneficiaries was very good whereas 61.7 per cent of respondents confirmed that the method used by IPs was good. 13.1 per cent of the respondents admitted that the distribution method of FIs by IPs was reasonable and 0.7 reported that the distribution method was poor. Out of the six evaluation districts, only in Kailahun about 2.2 per cent of those who received food items in the district reported that the distribution method used by IPs was poor. Majority of the food aid recipients in Tonkolili and Bo (78.3 and 73.2% respectively) confirmed that the food distribution method by IPs was good. Textbox 4 presented above is a statement by one of the female FGD participants in Port Loko District summarising the sentiments expressed by respondents.

5 Christian Aid –Crisis Response outcome: ICCO 1 Act formal proposal narrative.

6 Maggi, Tea leaf, Sardine, Vegetable oil, Powdered Milk, Salt, Sugar, Rice and Tomato Paste.

10

Figure 4: Percentage respondents judging the food distribution methods by project IPs

Availability and adequacy of most needed FIs in the package: The evaluation investigated to know if there

were any additional FIs that were mostly needed by beneficiaries at the time of the distribution and how

adequate were these FIs in the distribution package. The availability of the FIs was assessed at the household

levels while the adequacy of the needed food items was investigated during FGDs, IDIs and KIIs.

At household level, respondents were asked if most needed FIs were not available in food distribution

package. Analysis of the result indicates that, 43.1 per cent of the respondents admitted that there were

additional items that were mostly needed by them at the time of the distribution but were not included in

the food package, whilst 56.9 per cent stated that distribution package contained their most needed FIs. This

point is illustrated by statements expressed by participants during FGDs and KIIs as exemplified by the quotes

in textboxes 5 and 6 below.

How organised were the field staff in the distribution of FIs: During the Ebola crisis many people, including

humanitarian workers, had serious security and safety concerns to go close to communities or homes that

had been quarantined. This created a lot of food shortage for quarantined homes and other beneficiaries.

With this background, the evaluation sought to know how organised the IP field staff during the distribution

of FIs. When asked to describe the IP field staff and distribution process as either ‘very organised’,

‘organised’, or ‘not organised’, 36.4 per cent of the respondents reported that the IP field staff were very

organised

Textbox 5(3.5.2) ‘’The package was very rich and it contained about 19 items, mostly food stuff. At some point in time we complained about standardisation from government and other NGOs. But I often tell them that’s how the project was designed by CA and we cannot

alter it in the field’’- IDI, Bo District.

t alter it in the field’’- a statement by a project field staff in Bo District.

Textbox 6(3.5.2) ‘’Yes there was enough food for us, so we don’t need to go out in search

of food’’ FGD Adult Female, Western Area

Textbox 7(3.5.2) ‘’The distributors conducted the process in a fair and organized manner. They talk to people nicely

and they were also sympathetic to survivors and those that lost their love ones in the community’’. FGD Youth Participant in Bo Districts

11

Textbox 8(3.5.3) “I will prefer RADA because the one supply given to us was more than all the other supplies we received from other NGOs”.

FGD Adult Female, Bo District

while 62.3 per cent of interviewed beneficiaries affirmed that the field staff were organised. However, 1.3

per cent reported that they were disorganised. A similar question was posed to participants in FGDs to gauge

a wider view of direct and indirect beneficiaries of the project. One of the FGD participants’ statements is

illustrated by a quote in the textbox (7) below.

3.5.3 The appropriateness of Food items distributed

One of the objectives of the FIs distribution component of the CA Ebola emergency response project was to cover the immediate basic food needs of the quarantined homes, survivors and vulnerable groups in Ebola affected communities. The evaluation sought to know if the food items distribution was the right thing to do in the right way and at the right time. On this note, the appropriateness of the CA Ebola emergency response project was assessed based on: if beneficiaries’ expectation was in conformity with the FIs received from project IPs. At household levels, beneficiaries were asked if FIs they received from CA project IPs were in line with their expectation and also if urgently needed FIs were not included in the food package.

Food items received were in line with beneficiaries’ expectation: one of the objectives of the food items distribution component was to provide the immediate basic food needs of survivors and quarantined homes so as to eliminate the desires of quarantined people to break quarantine lines in search of food. The result shows that 63.6 per cent of the respondents indicated the FIs they received were in line with what they were expecting according to the prior information they received, whereas 36.4 per cent of the respondents stated that the food package received was not in conformity with their food expectation. One food item that many respondents thought was missing is fish. During FGDs similar questions were posed to participants to have a wider view of beneficiaries’ perception about the appropriateness of the food distribution. Several statements were made by participants. The text box above is a statement made by one female FGD participant in Bo District. However other statements were made by other participants disagreeing with the appropriateness of the food supply. Below is a statement made by a local authority (Town Chief) in Port Loko District expressing his disapproval of the manner in which a field worker treated them. Urgently needed FIs included in the food package: During the Ebola outbreak there were many food aid interventions by NGOs but people continued to question the type of food items distributed to quarantined homes, survivors and OVCs. The evaluation attempted to investigate the content of the food package distributed by the CA project IPs. In this regard, respondents were asked if there were any additional FIs that they urgently needed at the time of the distribution that were not included in the food package. The result shows that 56.9 per cent of the respondents stated no and 43. 1 per cent confirmed that there were additional FIs that they urgently needed but were not included in the food package. Further analysis shows

Textbox 9(3.5.3) “One of the field workers first visited us and informed us that he wants to come with food supply

for survivors and quarantined homes. Later he came back in a hurry and took the survivors’ cards and he collected the supply on behalf of this town but took the supply to Mile 36. He took the remaining food items to waterloo”. IDI, Port Loko District

12

Textbox 10 (3.5.3) ‘’When the quarantined homes received the food items from SEND, they stopped going out to hustle for food. Before the SEND food distribution, the virus was spreading faster in this community but when people received food items the spread of the virus stopped quickly in our community.’’ IDI, Kailahun District

that 14.1 percent of the respondents reported that they mostly needed corn soya blend (CSB) but was not included in the food package, 11.2 percent of the respondents claimed that they needed vegetable oil urgently but was not included in the package whilst 8.0 percent of them acknowledged that they urgently needed fish but was not included in the food package. The table below shows the percentage of respondents reporting the urgently needed food items but were not included in the food package.

Table3: percentage respondents reporting the food items the urgently needed A similar question was posed in FGDs to gauge a wider view of beneficiaries and the textbox above is an excerpt of the sentiments expressed by a community leader in a key informant interview. This further illustrates how the response was able to slow the spread of the disease, hence contributing to combatting the disease.

3.6 Beneficiary Target for Non Food Items (NFIs) The intervention was designed to target closed contacts of people infected by the disease especially in quarantined homes, survivors and care givers providing assistance/care to orphans who lost their parents to the disease. The evaluation found that overall, 83.1 per cent of the respondents acknowledged receipt of NFIs from CA implementing partners. Of this, 55.1 % were survivors while 10.2% were elderly care givers. About 18 per cent out of the 83.1 per cent were other beneficiaries.

Food Items Percent of respondents

Corn Flour 14.1 Cooking Utencils 1.9 Fish 8.0 Ground Nut 1.9 Onions 0.3 Palm Oil 2.6 Pepper 0.3 Rice 0.3 Sardines 1.6 Soft Drinks 0.6 Tea Leaf 0.3 Vegetable Oil 11.2 TOTAL 43.1

13

Table 4: Proportion of recipients of Non Food Items

3.6.1 Appropriateness of Non Food Items

As a result of the contagious nature of the Ebola scourge and the efforts by the government and its partners in containing and breaking the chain of transmission, all property formerly owned by Ebola survivors would be destroyed in order to decontaminate the environment and prevent further transmission. In that case, survivors usually have nothing left, as their property, including personal effects and kitchen paraphernalia, would have been destroyed by concentrated chlorine solution or burnt down. The expectation is that survivors would be provided new items to replace the lost ones. In many cases though, survivors were provided with a standard package across the board. The evaluation result shows that a one-size fits all package was given to all beneficiaries within the same category, hence not catering for special needs and situations. The evaluation asked if there were items they urgently needed but were not included in their packages. The study found that about six out of ten people (60.4%) reported that the items were appropriate while four out of ten (39.6%) said there were missing items they urgently needed. This indicates that a stakeholder consultation (including potential beneficiaries) prior to distribution of items is advisable.

Figure 5: Percentage of respondents reporting the availability of needed items in the package

39.6

60.4

Yes No

Of these, about six in ten respondents (57.1%) in Kailahun reported urgently needing items that were missing in the package. However, in Bo and Kambia districts at least eight in ten respondents reported that their urgent needs were met by the package.

Category of Beneficiaries Proportion of Respondents

Survivors 55.1%

Elderly Care Givers 10.2% Other Beneficiaries 17.8%

Overall 83.1%

14

Table 5: Percentage of Respondents reporting the unavailability of urgently needed NFIs in the package

District Yes No

Kailahun 57.1% 42.9%

Bo 19.6% 80.4%

Port Loko 51.4% 48.6%

Kambia 15.6% 84.4%

Tonkolili 47.8% 52.2%

Western Rural 35.0% 65.0%

The evaluation further asked respondents if the items received were in line with what they expected, according to the information they received before the distribution. Overall, about 64% reported that the items they received were in tandem with their expectations, while about 36% reportedly received less than they expected. In Kambia (93.3%) and Western Rural (92.5%) at least 9 out of 10 respondents said they received what they expected, whilst in Tonkolili (60.9%) and Kailahun (56.0%) about six in ten respondents said that they did not receive what they expected. Some respondents expected the package to include bedsheets/bedcovers.

Figure 6: Percentage of respondents reporting that the items were in line with their expectation

3.6.2 Effectiveness of NFIs Distribution

The effectiveness of the project intervention provides an indication of the quality of its implementation measures. As a result of urgency of the response, many partners provided different supports to ensure that the needs of the affected beneficiaries are addressed in a timely and effective manner. To this end, the evaluation assessed the timely provision of the food and non-food items during the emergency response period of the outbreak. Quality of NFIs received by beneficiaries The evaluation noted that 21.4% of the respondents reported that the quality of the NFIs distributed was

excellent while 68.7% rated it as good. Only about one per cent of the respondents rated the quality as poor.

15

Figure 7: Percentage of respondents reporting on the quality of NFIs

Proportion of respondents rating the quality of NFIs received by District When disaggregated by districts, 71% of the respondents in Kambia District rated the items as excellent while

only 3.3% of the respondents in Kailahun rated the items as excellent. It is worthy to note that in the Western

Area (Rural) 5% of the respondents felt that the quality of the NFIs was poor. Those respondents who

thought the quality of the NFIs was poor said that the items broke easily, shortly after they had been

distributed.

Table 6: Percentage of respondents by districts reporting the quality of NFIs District Excellent Good Reasonable Poor

Kailahun 3.3% 81.3% 15.4% -

Bo 10.7% 89.3% -

Port Loko 37.1% 54.3% 8.6% -

Kambia 71.1% 24.4% 4.4% -

Tonkolili 4.3% 91.3% 4.3% -

Western Rural 27.5% 47.5% 20.0% 5.0%

3.6.3 The Efficiency of NFIs Distribution

Efficiency is the extent to which the programme converted or was expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results in order to achieve the maximum possible outputs, outcomes, and impacts with the minimum possible inputs. Timeliness of the NFIs

Respondents were asked about how timely they thought the distribution of the NFIs was in relation to their

needs. Over half of them (52.1%) reported that the receipt of the items was timely. However, a statistically

significant percentage of people (47.9) reported that the receipt of the items was untimely. This is

particularly important because people’s perception of humanitarian assistance goes a long way to

determining their appreciation of it. This indicates that efficiency of the project could be improved. Below is

an indication of how timely the respondents thought of the NFIs.

16

Figure 8: Percentage of respondents reporting the timeliness of receipt of NFIs

Disaggregated by districts, while 73.3% of respondents in Kambia reported that it was timely, 94.3% of the

Port Loko district respondents felt that the time the non-food items were distributed was untimely.

Table 8: Percentage of respondents reporting the timeliness of receipt of NFIs

District Timely Untimely

Kailahun 69.2% 30.8%

Bo 33.9% 66.1%

Port Loko 5.7% 94.3%

Kambia 73.3% 26.7%

Tonkolili 58.7% 41.3%

Western Rural 47.5% 52.5%

3.7 Cash Transfer Component

3.7.1 Target for Cash Transfer

The cash transfer component of the CA Ebola emergency response project was designed to benefit 250 families (2,100 individuals) that have lost bread winners, 200 EVD survivors and 150 Other Vulnerable Children in 6 districts. The evaluation investigated to know if the beneficiaries received cash support from CA project IPs. 67.1 per cent of the respondents that received financial support from CA project IPs were survivors, 12.8 per cent were elderly caregivers and 20.1 per cent were other7 categories of beneficiaries interviewed.

Figure 9: Percentage of respondents acknowledging receipt of cash from project IPs

Categories of beneficiary

Did you receive money from project IPs during the Ebola outbreak as a support to

your family

Total

Yes No

Survivor 34.5% 32.6% 67.1%

Elderly Caregiver 9.6% 3.2% 12.8%

Other 18.5% 1.6% 20.1%

Source: Evaluation household questionnaire

7 They include widows, those that have lost breadwinners and women in agriculture.

17

Textbox 12 (3.7.2) ‘’Late last year when things were really difficult for me and not expecting any financial assistance

from anybody and as a widow and a caregiver for an Ebola orphan, one SEND worker informed me that my name was on their list to receive cash’’. FGD Adult Female, Kailahun District.

3.7.2 Cash Transfers Effectiveness

Ideally, to effectively implement and achieve the objectives of a project, the project should have an approved

policy to be followed. The project policy stipulates: who should be a beneficiary, the selection and targeting

of beneficiaries and beneficiaries to have pre-knowledge of their entitlement to increase the effectiveness of

the project. The evaluation of the CA Ebola emergency response project is based on the above mentioned

factors and these factors are examined below.

Availability of Stipulated project policy on beneficiaries: During the review of the project documents,

information sourced from project documents indicates that CA’s Ebola emergency response project cash

transfer component was designed to provide cash to families that had lost breadwinners, EVD survivors and

elderly caregivers of vulnerable children. This seems to indicate that the project had a specific policy guide in

place regarding intended target beneficiaries.

To find out how the cash transfer effectively responded and was appreciated by the beneficiaries, FGD

participants were asked to express their opinions on the extent to which the cash transfer benefits target

groups in the community. The text box below is the statement made by one elderly caregiver as a participant

in an FGD in Bo District.

In this regard, the project has a policy/guideline for implementing the cash transfer component and the availability of this policy helps to ensure project effectiveness and also its efficiency. Selection and Targeting of Beneficiaries: Considering the lucrative nature of the cash transfer component of the project coupled with limited livelihood strategies and opportunities in most EVD affected communities at the time of the project implementation, everyone would want to be included as beneficiary for the cash transfer. To safeguard this, the evaluation found that the IPs collaborated with the MSWGCA and District Ebola Response Committee (DERC) to identify beneficiaries. The text box below is a sentiment expressed one FGD participant showing how the technique was effective.

In addition to the above sentiment expressed by the FGD participant, household respondents were asked if they gave anything to anybody in return for the cash they received and 94.4 per cent of the beneficiaries interviewed indicated that they did not give anything to anybody in return of the cash they received. Beneficiaries having pre-knowledge of cash entitlement: In any project implementation, it is ideal for beneficiaries to have foreknowledge of their entitlement, as this will help to ensure effectiveness and efficiency of the project. The evaluation sought to know at household level if beneficiaries had pre-

Textbox 11 (3.7.2) “I am aware of the cash transaction given by RADA to orphans and survivors last year after the quarantine period. I was part of it; we were given Le 500, 000 each as widows and Le200, 000 for orphans. That was the highest money I handled at a time last year for myself and my two surviving orphans”. FGD Adult Female, Bo District

18

knowledge of the amount of money they were supposed to receive from CA project IPs. 75.6 per cent (149) of the respondents interviewed who received money from the IPs stated they had no pre-knowledge of the amount of money they were entitled to and 24.4 per cent (48) confirmed that they had pre-knowledge of the amount of cash they were supposed to receive. Respondents were further asked about who informed them about the amount they were supposed to receive. The figure below illustrates the percentage of respondents indicating the source from which they had the information about the amount of money they are supposed to receive from project IPs.

Figure 10: Who informed the beneficiaries about the amount money they are supposed to

receive.

3.7.3 Cash Transfer Component’s Relevance and Appropriateness

In recent times many critics have often questioned the cost-effectiveness of the distribution of FIs during

emergencies and also the inability to reach the intended beneficiaries. This idea is getting widely accepted by

many relief agencies to resort to cash transfers in emergency situations.

3.7.4 Justifying the Cash Transfers in the Ebola Emergency Response

Various stakeholder discussions were held at district level, to gauge the perception of the respondents on the justification and appropriateness of the CA Ebola emergency response project cash transfer component. Most of the stakeholders interviewed believed that the rationale on the part of CA to have chosen cash transfers as a humanitarian response for EVD victims was justifiable because of the following reasons:

The scale of suffering caused by the EVD: Many stakeholders highlighted that the EVD caused a widespread of human suffering and many people suffered severe consequences as a result of the EVD outbreak in country and the need to provide humanitarian assistance in the affected communities for survivors, elderly caregivers, widows, orphans and people that have lost breadwinners to meet their basic needs and enable these victims to re-establish their livelihood is unquestionable. Cash was suggested as a way to help alleviate that suffering, especially when they could choose what to do with it.

Many interventions provided in the form of food and non-food items by other projects: During the EVD outbreak, there was huge humanitarian response by many NGOs to the EVD victims in many communities. It was highlighted by many stakeholders during evaluation field visits that CA’s cash transfer initiative to EVD victims is unique and as such, the cash transfer component was considered to be highly complementary with other humanitarian relief interventions undertaken by many other NGOs such as the distribution of FIs and NFIs. The text box below indicates the view of a key informant in Port Loko District.

19

3.8 Observing Humanitarian Standards

Humanitarian assistance rendered during emergencies has certain standards to respect which are essentially

to ensure that humanity was respected and basic standards were met during implementation. The evaluation

examined three key issues in this respect:

i. Payment of cash or kind to receive items

ii. General respect accorded to beneficiaries

iii. Knowledge of complaint procedures

3.8.1 Payment of cash or in-kind for the FIs received

If project beneficiaries did give any cash or in-kind to anybody for the food items they received was gauged by the evaluation. Respondents were asked if any cash or in-kind was given to anybody by beneficiaries for the items they received. From the analysis, the result revealed that 97.8 per cent of the beneficiaries interviewed stated that they did not give any cash or in-kind to anybody for the food items they received whilst 2.2 per cent of the respondents interviewed confirmed that they did give cash or in-kind to someone for the food items they received. Further analysis of the result shows that 5.7 per cent of beneficiaries who received FIs in Port Loko District alleged that they paid cash or kind to someone for the food items they received and such allegation was not reported by beneficiaries in the Western Rural. 2.2 per cent of beneficiaries that received FIs from CA project IPs in Kailahun, Kambia and Tonkolili stated that payment of cash or in-kind was given to someone for the food items they received. The table below shows a percentage summary of beneficiaries that reported paying cash or in-kind to someone for the FIs they received.

Table 9: Percentage of respondents that reported giving cash or in-kind for items received

District Did you give any cash or in-kind to anybody for the food items you received during the distribution?

Percentage of respondents receiving FIs

Yes No

Kailahun 2.2% 58.2% 60.4%

Bo 1.8% 92.9% 94.6%

Port Loko 5.7% 94.3% 100.0%

Kambia 2.2% 97.8% 100.0%

Tonkolili 2.2% 97.8% 100.0%

Western Rural 0.0% 95.0% 95.0%

Textbox 13 (3.7.4): ‘’I have witnessed in many situation the sales of Ebola emergency relief items most especial the food items and also many cases the relief items provided by NGOs do not match the beneficiary needs and one of the main advantage of cash transfers intervention is that the beneficiaries have the power to decide what they want to buy with the money’’. IDI, Kailahun District

20

3.8.2 General respect accorded to beneficiaries

Asked how respected beneficiaries felt with the way the IP treated them during distribution, 99.4% of the respondents reported that IPs treated them respectfully. It is worthy of note that Tonkolili (2.2%) and Western Rural (2.5%) respondents reported being treated disrespectfully by IPs. Details of ‘respect’ and ‘disrespect’ were not probed.

Figure 11: Percentage of respondents reporting how respectful the IPs treated them

3.8.3 Knowledge of Complaint Procedures

The evaluation sought to know if the beneficiaries were aware of any complaint procedure if they felt their

rights were violated. Overall, only 8.9% of the respondents acknowledged that they knew where to lodge

complaints if they wanted to. This is a little worrying as it points to management and monitoring lapses.

Ideally, this should have been embedded as a key sensitisation component of the project, as instances

abound across the world where aid workers violate their beneficiaries’ rights. It is worthy to note that 100%

of respondents in Kailahun and Western Rural districts reported that they knew nowhere to complain, in case

they had any issues with the process.

Figure 12: Percentage of respondents having knowledge of complaint procedure

21

4.0 Limitations of the Evaluation The evaluation was carried with a few challenges. Below are the key ones and how we mitigated them:

Rainy Season: Considering that data was collected in November (where it still rained a little), data collectors

were provided with plastic folders to protect the documents.

Bad Road Network: Many of the roads in the provinces are very bad. In that regard, we discouraged night

travels and over-speeding to prevent accidents.

Transportation: Although SADRS had a pre-contract agreement for Christian Aid to provide vehicles to drop

off and pick up enumerators, in one chiefdom, for each district, Christian Aid expressed at last minute that

they experienced constraints and could only provide one vehicle. SADRS then resorted to hiring commercial

vehicles which incurred extra loss to the firm.

Beneficiary List: Up to the eve of the training Christian Aid/IPs could not provide a comprehensive list of

beneficiaries. This was a very serious challenge as it prevented the consultancy from employing a complete

randomised sampling, as planned. One of the beneficiary lists provided was completely illegible and

therefore not usable. In many instances, the closest identifying information against names is chiefdoms, after

we had stressed the need to have specific addresses, localities and phone numbers, where possible. CA

suggested that IP representatives would help us identify these beneficiaries (which was less than ideal) but

eventually did in some cases. The team pointed out that this could possibly influence the response and a

likely cause for social desirability8.

CA Rep at Training: In spite of calls for a CA representative to be at the training to give an overview of the

project and its activities, especially those changes/amendments that may have occurred post-proposal

period, including method of distribution, CA responded that it’s against their policy as it could cause bias,

although the team thought otherwise. SADRS’ technical team leads played this role.

Other Logistics: Also, in our pre-contract agreement, CA promised to provide a projector for the training,

however, and on the eve of the training the agreement was reneged. This resulted to the Consultancy hiring

a projector throughout the period, at a financial cost to the firm.

8 Simply put, social desirability can be described in research as a situation where the research participant tells the

researcher what the participant thinks the researcher would like to hear, even if it’s not the truth.

22

5.0 Conclusion and Recommendations

5.1 Conclusion Christian Aid’s ICCO-funded Ebola response project as implemented through various partners across the four

regions of the country was considered by many to have been well-conceived by the project designers,

especially within the context of an emergency situation. It was evident that the project was successful in

targeting quarantined homes, survivors, elderly caregivers and other categories of beneficiaries which

injected a feeling of care and hope in a situation of fear and despondency. This position is further buttressed

in that majority of the respondents confirmed that they received support from one of Christian Aid’s Ebola

implementing partners and huge percentage (92.6%) of the respondents were also satisfied with the quality

of the FIs distributed; with 52.1 percent of the respondents stating that the food assistance they received

from the project implementing partners were timely. SADR observes that although 47.9 percent of the

respondents confirmed the support was untimely, yet this could not have largely affected the project’s

implementation and objectives.

The findings showed that the EVD response project indeed provided emergency relief assistance and through which, succeeded in contributing to reducing the stigma surrounding those who had contracted the disease and enhanced their reintegration back into their families and communities. The latter was especially true for the children, whose non-food items like toys and clothes easily attracted the company of others of their age group. However, SADRS observed that despite that CA’s Ebola response packages were (in comparison to other partners’) bigger and holistic (comprising many items), the supply could not march the demand and as a result, many “would-have-been beneficiaries” went without. Like that of the food and non-food items, majority of the beneficiaries received the cash assigned. However, CA would want to consider increasing the amount as to be able to kick-start any enterprise and the notion of giving cash to beneficiaries to start agriculture at the time when planting season was almost over was a huge challenge. With the exception of few9, most of the beneficiaries saw the cash as an extension of the food items as monies were mostly used to provide food in the homes. On project management, the selection technique of beneficiaries of all of the IPs contacted was to a large

extent effective and transparent. All of the IPs maintained internal reporting system, especially between the

field and office staff of the IPs and periodic reports were sent to CA. However, consistent field monitoring of

IPs’ activities by CA was weak, and being an emergency situation, it was sometimes difficult to sit and wait for

feedback before actions were taken.

It was further observed that most IPs depended strongly on social mobilisation (or partnership with other donors) to implement project; despite the fact that the figure heads of most of the IPs had had experience in some kind of relief work, yet there were limited staff strength for most of the IPs to cope with the immediacy of the job they had to do. Although such partnership made implementation more transparent and less stressful for partners, working with multi-social institutions sometimes caused delays; IPs were requested to attend several meetings in a week which sometimes affected timely distribution10. Maybe as a result of the emergency situation, SADRS observed that there was weak and inadequate assessment of IPs and this further

9 There were however one or two isolated stories of successful impact ; for instance like the story of a female survivor at Baomahun

at Bo District – she was reported to have used the cash to buy one jerry can of palm oil (5 gallons) and is now trading that by pint at the local market. 10

It should be noted that in an emergency situation of that magnitude, it would have been difficult (at least for information sharing purposes) for any one partner to work without attending these meetings.

23

affected planning, implementation and communication. For instance there was a case of an IP’s contract being terminated and services transferred to another that was not even shortlisted by CA for the same contract. During the implementation CA was more concerned with strategic level activities (e.g. procurement) and less involved in the monitoring process of field activities. Similarly, procurement was centralised in Freetown and as a result, a one size-fits-all package was delivered to beneficiaries. For instance SADRS observed that men were supplied sanitary pads, pots supplied without covers and some children had to exchange their sandals because their sizes were never supplied. Though few, some IPs reported that some of the foods were spoiled before getting to the beneficiaries.

5.2 Recommendations General Management

I. CA to Monitor IPs during implementation: The Ebola response project was widely acknowledged and considered appropriate by both IPs and majority of the beneficiaries. However for future programmes, project monitoring should be more robust on the part of CA. CA should not just depend on the field reports of IPs but their M&E Department should assign staff to periodically monitor projects implementation. This will make it easier to address field-related challenges, share lessons learnt in real time, and improve on coordination and monitoring of the overall implementation.

II. IPs to be carefully selected and trained: The selection of IPs for future projects should be more thorough to avoid termination of contracts midway project implementation. Being an emergency situation (which may have the tendency of affecting thoroughness), focus should be on IPs that are resident in the localities rather than contracting IPs at central level. In a case where an IP at local level has the requisite experience but has limited staff capacity to absorb services required, project should support such IPs with training packages or to embark on short term employment contracts to scale up capacity. Although cost might be slightly higher with such arrangement, yet CA would have helped improved the local partner and CA or other NGOs needing IPs might benefit from the services of such IPs in the future.

III. IPs’ involvement in planning could be improved: Once contracts are signed, IPs should be granted some leverage to make input into the planning process at the strategic level, especially on issues relating to challenges encountered in the field. For instance the procurement and distribution of food and non-food items would have been less challenging if the input of the IPs was constantly sought. Future needs assessments of beneficiaries would be better done with more involvement of IPs as ‘partners’.

IV. More post-Ebola follow- up programmes for sustainability: To sustain impact, more post-Ebola follow-up programmes should be considered, especially for the women and children who stand to suffer most when the income of the household drops or fails. There were glaring stories of survivors, orphans and OVCs whose livelihoods have been threatened since they returned to live with extended family members. CA may wish to follow-up to enhance the re-integration of these children or liaise with child focused NGOs to achieve such goal.

V. Complaint procedures to be made known to beneficiaries: Because of the duty of care, where feasible, CA should consider increasing awareness of their prospective relief aid beneficiaries of complaint procedures in case they suspect abuse of the process.

24

Food and Non-Food Items

VI. Procurement to be decentralised: In line with recommendation (III) above, where possible, the procurement of food and non-food items in the future should not only involve the IPs but done at regional or district levels to minimise damage to non-food items, spoilage of food items and delays in supplies to the final beneficiaries. Or where procurement is done at central level, suppliers should be informed that goods are considered acceptable when delivered at regional or district levels. This could improve on project’s effectiveness and efficiency.

VII. Packages to be need-specific: Although most of the respondents generally reported that most needed items were available and adequate in their packages, yet CA should in the future disaggregate items and packages by sex and age categories to enhance beneficiaries’ usage and satisfaction with their packages. A one size fits all package support should be avoided; better inventory of the needs of the beneficiaries will be required.

Cash Transfers

VIII. Volume of cash and knowledge of IP’s purpose for cash transfer be increased: One very laudable intervention of the Ebola response project was the cash transfers. However, to deepen the impact of such in the future, the amount of cash should be increased to an amount that can feasibly start a business. The amount of cash transferred was reportedly small and could not start any “meaningful business”; most ended up using it to buy food. CA in the future may want to encourage beneficiaries to form corporative groups around particular business proposals from beneficiaries, and a basket or pool fund made available to the groups rather than to individuals. This could be under the supervision of the IPs or directly supervised by CA M&E department. Similarly, cash was handed to beneficiaries (most of whom had limited or no business acumen) without any clue as to what could be done. Future cash transfers could be done with business orientation discussions or talks that would motivate, re-orientate and help beneficiaries better understand the purpose of the cash transfers.

25

6.0 ANNEX ANNEX 1

ICCO-funded Christian Aid Ebola Response Programme, Sierra Leone

1 Nov 2014 to 30 November 2015

End of Programme External Evaluation

Terms of Reference

I. Background

Christian Aid in Sierra Leone with funding from ICCO (1 November 2014 to 31 August 2015) was able to

respond to the Ebola Virus Disease (EVD) epidemic through supporting partners in 9 districts to provide

emergency relief assistance and to carry out various activities in support of Ebola survivors, people in

quarantined households, and orphans and other vulnerable children (OVCs).

By providing water, food and Non-Food Items (NFI’s) to quarantined homes the need to break the quarantine

to seek essential supplies was removed thereby reducing the risk of the infection spreading further. Providing

survivors with food and other basic needs aimed to reduce the stigma surrounding those who had contracted

the disease and supported their reintegration into their families and communities.

ICCO Funding

Phase 1 100,000

Phase 2 200,000

Addendum 101,882

Total in Euro 401,882

ICCO-Funded Activities

Phase 1:

Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners

Phase 2:

Outcome 1: The basic needs of 250 families (2,100 individuals) that have lost bread winners, 200

survivors, and 150 OVCs are provided in 6 districts with cash and food packages;

Outcome 2: 150 survivors and Ebola widows have established income generation activities (IGAs),

started agricultural activities, and successfully reintegrated in their communities.

Outcome 3: The spread of EVD is mitigated and stigma reduced by providing food and NFIs to 150

quarantined households and mobilising 50 traditional healers as community health workers for

infection prevention and control (IPC)

26

Additional Activities funded under the Addendum:

Outcome 4: Support to elderly caregivers (provision of micro grants with related training for

livelihoods inputs such small animals, seeds and vegetables to allow elderly caregivers to sustainably

support vulnerable children)

Outcome 5: 285 orphans and vulnerable children in Ebola-affected communities in Tonkolili District

provided with primary and junior school learning materials and supplementary food to enable them to

restart their schooling

RMA

Research: ‘Keeping the Faith’ research on the role of faith leaders in the Ebola response (in

collaboration with Tear Fund, CAFOD and Islamic Relief)

Disaster Risk Reduction (DRR): 1-2 day workshop for partners and CASL, DCI & RADA staff

27

Partner Activities

District Partner Activities Quarantined & Survivor Households,

Families that have lost Breadwinners, Widows

Orphans & Vulnerable Children

(OVC)

Elderly Caregivers Traditional Healers

Planned Achieved Planned Achieved Planned Achieved Planned Achieved

PHASE 1 1 Nov 2014- 31 Jan 2015

Kailahun Methodist Council Sierra Leone (MCSL)

Food & NFIs 210 180 0 30

Bo & Pujehun

Rehabilitation and Development Agency (RADA)

Food & NFIs 200 100 0 100

Port Loko

Sierra Leone Social Action Volunteers (SLSAV)

Food & NFIs 100 100 0 0

PHASE 2 +Addendum 1 Feb 2015- mid-Sept 2015

Bombali Community Action for Human Security (CAHSec)

Outcome 3: Food & NFIs to quarantined households; Mobilising 50 traditional healers for IPC (Infection prevention & control)

50

50

50 50

Port Loko

SLSAV Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners

100 100 50 50

Kambia

SLSAV Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners

100 100 50 50

Tonkolili Real Women in Action Programme (REWAP)

Outcome 4: Support to elderly caregivers (extension to 15 Sept) Outcome 5: Support to 285 orphans and vulnerable children (180 primary; 105 junior/secondary) with primary and junior/secondary school learning materials and supplementary food to enable them to restart their schooling

285

285

50

50

Western Area Rural & Western Area Urban

Network of People Living with HIV (NETHIPS)

Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools

50

50

50

50

28

Kailahun SEND-SL

Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools

100

50

100

50

100 100

Bo

RADA Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools

100

50

100

50

50 50

29

II. Purpose of the Evaluation

This final over-all programme evaluation will have the following two-fold purpose:

1. To assess and validate the outcomes and impact of the response from the standpoint of beneficiaries

and partners, and;

2. To capture and synthesize Christian Aid (CA) and partner’s learning from the response.

III. Key questions the evaluation must address

1. On outcomes and impact :

a. What has been achieved by the response, where and for whom?

b. What intended and unintended changes did the programme bring to target communities and

partners?

c. How has the response addressed local needs and priorities in combatting the spread of Ebola?

2. On learning :

a. What were the learnings of CA and partner? How have these been documented, shared and used to

improve the response? What can be replicated for future humanitarian responses?

b. What were the downward accountability mechanisms established within the programme delivery

process? What and how can they be improved?

c. What strengths, weaknesses, benefits and constraints resulted from engaging a consortium of

partners? What value has been added by working with new partners?

IV. Evaluation Outputs

1. An inception report by the external consultant setting out an evaluation plan, methodology/design,

and timetable

2. Draft and final report

3. Management response to the evaluation on the draft report

4. Learning event to present the key findings of the evaluation

The final evaluation report should be no longer than 10 pages, excluding any appendices, and include an

executive summary. This should be addressed to the evaluation steering group and will be shared with the

Donor and/or publicly on Christian Aid’s website with the full synthesis and country reports available upon

request.

V. Methodology

The final detailed methodology will be developed by the external evaluator and reviewed by and agreed with

the evaluation steering group. It will include but not limited to the following:

A desk review of existing documents (partner proposals and reports, and other related documentation)

Quantitative and qualitative analysis using participatory methods with a sample of direct and indirect

beneficiaries

30

Interviews, discussions and debriefing sessions with partners and CA staff in Sierra Leone and UK via

Skype

VI. Evaluation Team

The evaluation team will be made up of:

1. A team leader/external lead evaluator with a strong background in humanitarian evaluation

practice specifically using participatory approaches and with an excellent understanding of the Sierra

Leone Ebola context. S/he will work with an external assistant evaluator with strong understanding of

emergency response and application of humanitarian standards to assist in country with data

gathering / interviews

VII. Management

The evaluation team will report to Christian Aid’s Regional Emergency Manager (REM) for West Africa.

The evaluation steering group will oversee the management of the evaluation and ensure it fulfils its

purpose. This group will sign off the ToR and then meet as required. This is likely to be at least thrice – to

agree and sign off the ToR and inception note and to review and collate the feedback on the initial findings

from the evaluation. They will also assist in communicating the findings of the evaluation.

Suggested membership: REM, ICCO representative, and CASL Country Manager/Policy & Programme

Manager and Emergency Response Programme Manager.

VIII. Timeframe

Recruitment of evaluation team – 1 week

Desk review – 1 week

Inception report drafting and finalisation

Logistics and admin preparation for field work

Field work – 1 week

Draft report submission – 1 week

Steering group review of report & drafting of management response

Final report submission – 20th November 2015