External Evaluation of Christian Aid’s-ICCO Funded Ebola Emergency Response Project
Final Draft Report
December 2015
Submitted By
Social and Demographic Research Solutions
i
Table of Contents KEY ACRONNYMS ..................................................................................................................................... ii
EXECUTIVE SUMMARY ............................................................................................................................ iii
1.0 Introduction .................................................................................................................................. 1
1.1 Evaluation and Project Background .............................................................................................. 1
1.3 Objectives of the Evaluation ......................................................................................................... 2
2.0 Methodology ................................................................................................................................. 2
2.1 Research Methodological Approach ............................................................................................. 2
2.2 Data Collection Tools Design......................................................................................................... 2
2.2.1 Evaluation Tools .................................................................................................................... 2
2.3 Scope of the Evaluating and Sampling .......................................................................................... 3
2.3.1 Sample size ............................................................................................................................ 3
2.4 Field Work ..................................................................................................................................... 4
2.5 Data Quality, Management and Analysis ...................................................................................... 4
2.5.1 Quantitative Data .................................................................................................................. 4
2.5.2 Qualitative Data .................................................................................................................... 4
3.0 Evaluation Findings ....................................................................................................................... 5
3.1 Introduction .................................................................................................................................. 5
3.2 Demographic Characteristics of Targeted Beneficiaries ............................................................... 5
3.3 Project Management and Distribution Methods .......................................................................... 5
3.4 Target for Food Items ................................................................................................................... 6
3.5 Food Items Component ................................................................................................................ 7
3.5.1 The Efficiency of Food Items Distribution ............................................................................. 7
3.5.2 Food Items Distribution Effectiveness .................................................................................. 9
3.5.3 The appropriateness of Food items distributed ................................................................. 11
3.6 Beneficiary Target for Non Food Items (NFIs)............................................................................. 12
3.6.1 Appropriateness of Non Food Items ................................................................................... 13
3.6.2 Effectiveness of NFIs Distribution ....................................................................................... 14
3.6.3 The Efficiency of NFIs Distribution ...................................................................................... 15
3.7 Cash Transfer Component .......................................................................................................... 16
3.7.1 Target for Cash Transfer...................................................................................................... 16
3.7.2 Cash Transfers Effectiveness ............................................................................................... 17
3.7.3 Cash Transfer Component’s Relevance and Appropriateness ............................................ 18
3.7.4 Justifying the Cash Transfers in the Ebola Emergency Response ....................................... 18
3.8 Observing Humanitarian Standards ............................................................................................ 19
3.8.1 Payment of cash or in-kind for the FIs received ................................................................. 19
3.8.2 General respect accorded to beneficiaries ......................................................................... 20
3.8.3 Knowledge of Complaint Procedures.................................................................................. 20
4.0 Limitations of the Evaluation ...................................................................................................... 21
5.0 Conclusion and Recommendations............................................................................................. 22
5.1 Conclusion ................................................................................................................................... 22
5.2 Recommendations ...................................................................................................................... 23
ii
KEY ACRONNYMS
CA Christian Aid
CAHSec Community Action for Human Security
CSB Corn Soya Blend
CsPro Census and Surveys Programme
DERC District Emergency Response Committee
EAs Enumeration Areas
EVD Ebola Virus Disease
FGDs Focus Group Discussions
FIs Food Items
HH House Hold
ICCO International Christian Community Organization
IDI In-Depth Interview
IPC Infection Prevention and Control
KII Key Informant Interview
MCSL Methodist Church Sierra Leone
MSWGCA Ministry of Social Welfare, Gender and Children's Affairs
NETHIPS Network of HIV Positives in Sierra Leone
NFIs Non Food Items
NGO Non-Governmental Organization
OVC Other Vulnerable Children
RADA Rehabilitation And Development Agency
REWAP Real Women in Action Programme
SADRS Social and Demographic Research Solutions
SEND Social Enterprise Development Foundation
SLSAV Sierra Leone Social Aid Volunteers
SPSS Statistical Package for Social Scientists WHO World Health Organisation
iii
EXECUTIVE SUMMARY
Introduction
Christian Aid Sierra Leone (CA), with funding from ICCO, was able to respond to the Ebola Virus Disease (EVD) epidemic through implementing partners in nine districts1 in Sierra Leone. The ICCO-funded EVD response project provided emergency relief assistance and also carried out various activities in support of Ebola survivors, people in quarantined homes, elderly caregivers, orphans and other vulnerable children (OVCs). The response aimed to provide water, food and Non-Food Items (NFIs) to quarantined homes to avoid the need to break the quarantine in search of essential supplies thereby reducing the risk of spreading the infection. The response also aimed to provide food, cash and other basic needs to EVD survivors and victim families aimed at reducing the stigma surrounding those who had contracted the disease and supported their reintegration back into their families and communities. Additionally, CA’s Ebola response project mobilised and trained traditional healers in infection prevention and control (IPC), as traditional healers were perceived to be a major source of transmission. Recognising that some children could be orphaned or their families impoverished as a result of the accompanying restrictions, CA also provided teaching and learning materials to these children, especially in the Tonkolili district. Methodology The evaluation used both qualitative and quantitative methods to collect data. For the qualitative methods, three focus group discussions were conducted in each of the six districts: adult male, adult female and mixed youth groups. In addition to the focus group discussions, three in-depth interviews were conducted in each of the six project evaluation districts, with implementing partner (IP) staff, community leaders, District Ebola Response Centre (DERC) or a member of the Social Welfare, Gender and Children’s Affairs Ministry. In total, eighteen focus group discussions and eighteen in-depth interviews were conducted across the six evaluation districts. Household questionnaires were also designed to collect quantitative data from beneficiaries of the intervention. In total, 313 questionnaires were administered with Port Loko having the least number of questionnaires (35) and Kailahun the highest (91). The sampling was simply based on proportion to size of the intervention in the district. Quantitative data was entered using CsPro and exported into SPSS v.20 for analysis. For the qualitative data, interviews and discussions were listened and analysed using simple framework analysis. Key Findings About two-thirds of the respondents (67.1%) were survivors while about the same number (63.1%) were household heads of their respective households. Project Management: The study found that there were limitations with coordination and monitoring of the overall implementation by CA was not consistently done. The IPs sent periodic reports to CA but response to challenges in the field were either sent late or sometimes not sent at all. Although internal records were maintained by the IPs for reference, yet the delays made it difficult to share lessons learnt in real time. Food Received: About 9 out of 10 (87.9%) respondents reported having received food supply from a Christian Aid partner while at least 1 out of 10 reportedly never received any food supply from any CA partner. Glaringly, just about 6 out of 10 respondents (60.4%) in the Bo district reported to have received food. For the districts in the north, all the respondents reportedly received food supply.
1 Bo, Bombali, Kailahun, Kambia, Port Loko, Pujehun, Tonkolili, Western Area (Rural) & Western Area (Urban)
iv
Timeliness of Support: Nearly half of the respondents (47.9%) felt that the support was not timely particularly when most people received their items at least few days into quarantine. Just over half of the respondents (52.1%) perceived the support to be timely. There were examples of whole districts where food supply was done after quarantine which could be seen as untimely, although some of the items were not strictly for quarantined homes. This however also raises questions of efficiency of the intervention. Availability and Adequacy of items: Generally, respondents reported that most needed items were available and adequate in the package. Similarly, about 90% of respondents reported that the items supplied were appropriate and rated their quality highly. This positively responds to the question of effectiveness of the response. Cash Transfers: Apparently, one laudable intervention was the cash transfers. This was so because it gave the beneficiaries the option to buy what they wanted and to also use it to generate further income. It slightly deviates from the usual prescriptive approach of aid agencies. Although the money was meant for income generation, giving out money in crisis where survival was key led people to convert the money for personal upkeep, as was originally thought to be the purpose by the beneficiaries. Respecting Humanitarian standards: the three parameters used to assess humanitarian standards were: respect, payment/bribery and complaint procedures. Respondents rated the first two very highly; that means the workers and process were very respectful and that they hardly paid anything in kind or cash to receive the services. However, nearly 9 out of 10 respondents reported that they had no knowledge of complaint procedures, in case they were violated during the process. Key Recommendations
More robust monitoring: Although some monitoring was reportedly done, the study’s findings indicate that CA could improve on their monitoring of IPs during implementation to share lessons learnt in real time.
IPs to be carefully selected and trained: Although the project was implemented during a crisis period, a more careful selection process could prevent changing IPs midway through the project. Where capacity is questioned, some training to enhance capacity could help.
More IP involvement in training: Usually, in emergency situations a few people plan and take decisions, as procrastination is an enemy of progress; but more involvement of IPs in planning would help CA understand local contexts better and IPs feel as true ‘partners’.
Procurement to be decentralised and packages more need-specific: Where possible CA would want to consider, in future, decentralising procurement of items to suit local needs and packages should not be one-size-fits-all.
Amount of cash increased and purpose made known: Although half a loaf is better than none, CA would want to consider increasing the amount of money given to beneficiaries for income generation if the purpose is indeed to start a business. Also, the purpose of the cash should be made unequivocally clear from the onset so that potential beneficiaries do not misunderstand it to be pocket-money.
Complaint procedures to be made known to beneficiaries: Because of the duty of care, where feasible, CA should consider increasing awareness of their prospective relief aid beneficiaries of complaint procedures in case they suspect abuse of the process.
1
1.0 Introduction
1.1 Evaluation and Project Background This report discusses the evaluation findings of Christian Aid’s ICCO-funded Ebola response project, implemented through various partners across the four regions of the country. Christian Aid Sierra Leone (CA), with funding from ICCO, was able to respond to the Ebola Virus Disease (EVD) epidemic through implementing partners in nine districts2 in Sierra Leone. The EVD response provided emergency relief assistance and also carried out various activities in support of Ebola survivors, people in quarantined homes, elderly caregivers, orphans and other vulnerable children (OVC). As the Ebola Virus continued to spread, the government and its partners focused mainly on curtailing further spread and their main area was to ensure safe burials and discouraging crowds. For some time during the response, psychosocial care did not receive the required attention. As people continued to bury their relatives in secret—accusing the government’s response team of undignified burials, thereby disrespecting ingrained cultural practices, other players including NGOs and civil society organisations saw the need for a strengthened and organised community engagement in the process. Although the Ministry of Social Welfare, Gender and Children’s Affairs (MSWGCA) officially headed the psychosocial aspect of the EVD response, they were not adequately equipped—both with logistics and capacity—to sufficiently handle the complex psychosocial issues that accompanied the Ebola outbreak. CA then saw the need to bolster the Ministry in contributing to support vulnerable populations such as orphaned children, survivors, elderly caregivers, widows and quarantined households. The response aimed to provide water, food and Non-Food Items (NFI’s) to quarantined homes to avoid the need to break the quarantine in search of essential supplies thereby reducing the risk of spreading the infection. The response also aimed to provide food, cash and other basic needs to EVD survivors and victim families aimed at reducing the stigma surrounding those who had contracted the disease and supported their reintegration back into their families and communities. The Ebola response got to a point where traditional healers were identified as a significant source of transmission, as they treated people with almost any disease. In some parts of the country, traditional healers, who are also community opinion leaders, are more regarded than their orthodox scientific medical counterparts; hence it was believed that some people avoided accessing health facilities but instead consulted traditional healers for their health solutions. The situation became worse because, at the initial stage of the outbreak, responders to the disease claimed there were little or no chances of surviving Ebola. Traditional healers, on the hand, always welcomed sick people, even when they were apparently convinced they could not cure them. Recognising this, Christian Aid mobilised traditional healers and trained them in Infection Prevention and Control (IPC) to help break the chain of transmission. This was an awareness-raising campaign by Christian Aid—if the traditional healers know of the dangers of their interventions, they could stop treating sick people and would refer them to the appropriate healthcare facility, hence breaking the chain of transmission. This report would therefore assess the effect of Christian Aid’s interventions on the people, the communities and the response generally.
2 Bo, Bombali, Kailahun, Kambia, Port Loko, Pujehun, Tonkolili, Western Area (Rural) & Western Area (Urban)
2
1.3 Objectives of the Evaluation The evaluation objectives of the EVD response project among others included:
a. To assess and validate the outcomes and impact of the response from the standpoint of beneficiaries and partners
b. To capture and synthesise Christian Aid (CA) and partners’ learning from the response
2.0 Methodology
2.1 Research Methodological Approach This research was a mixed-sectional evaluation conducted using participatory methods involving both quantitative and qualitative approaches. Desk review of relevant project documents was done for background information to add to the knowledge base of the evaluation team.
2.2 Data Collection Tools Design The evaluation data collection tools were developed through participatory approach; the consulting firm
Social and Demographic Research Solutions (SADRS) prepared a draft of the evaluation tools and Christian
Aid’s project management team consulted before the final tools for data collection were adopted. The tools
were further reviewed during the training and translated by the enumerators to have a common
understanding.
2.2.1 Evaluation Tools
Household (HH) questionnaires: A single designed Household questionnaire was used to collect data
at household level of beneficiaries of Food Items (FIs), Non Food Items (NFIs), Teaching and Learning
Materials, and Cash Transfers.
Key Informant or In-Depth Interview (KII) guides: One KII question guide (with separate and clearly
defined themes/sections for each interviewee/institution) was used for the three components of the
project. The questions were tailored to collect nuanced information that could not be obtained by
the HH closed-ended questions.
Focus Group Discussion (FGD) guide: A single FGD guide was used to obtain information from direct
and indirect project beneficiaries to determine their perception of the support received through the
project.
Selected participants for the FGDs were grouped into three demographic categories: youths
comprising both sexes– so that they are allowed to express their views amongst their peers, adult
female and adult male groups. The reason for separating the adult men from the women was to
avoid male dominance in the discussions. Local opinion leaders/chiefs were not recruited into the
focus group discussions as they could influence the views of the other group participants.
3
2.3 Scope of the Evaluating and Sampling The scope of the evaluation exercise was limited to selected project beneficiary communities in six districts.
The targeted beneficiary communities/ towns served as Enumeration Areas (EAs) for the administration of
household questionnaires and the FGDs.
The selection of districts was purposive, as some project outcomes were only applicable in particular
districts. In this case, districts and research participants were selected for what they are, where they are and
what was their planned outcome. The six selected districts out of the nine project districts are: Tonkolili, Port
Loko, Kambia, Kailahun, Bo and Western area (Rural).
2.3.1 Sample size
As mentioned in the technical proposal to Christian Aid, it was agreed that 300 household questionnaires
would be administered across the six sampled districts. However, while in the field, the research team were
able to administer 313 household questionnaires and efforts were made by the team to ensure that all
categories of beneficiaries (including survivors and elderly caregivers) were included without complicating
the sampling methodology. The (quantitative survey) sample size was then proportionately allocated as per
number of beneficiaries in each district—Bo: 56, Kailahun: 91, Kambia: 45, Port Loko: 35, Tonkolili: 46, and
Western Area (rural): 40.
Table 1: Determining the Sample size
District Number of Beneficiaries (n)
Proportion (p=n/N)
Sample size
(P*300)
Actual administered HH questionnaires3
Bo 67 0.16 47 56
Kailahun 150 0.35 105 91
Kambia 62 0.14 43 45
Port Loko 50 0.12 35 35
Tonkolili 50 0.12 35 46
Western Area 49 0.11 34 40
Total(N) 428 1 300 313
3 While in the field especially Kailahun District, it was discovered that there were internal movement of beneficiaries between
localities/towns as well movement of beneficiaries to other district/regions making it difficult to trace the beneficiaries. This necessitated to add to the sampling size to cater for those initially sampled but were unavailable
4
2.4 Field Work Data collection was conducted for an average of 8 days for both household questionnaires and FGDs by
enumerators4. The field work was organized in such a manner that household questionnaires were
administered first in the project communities. Key Informants were selected based on their perceived in-
depth knowledge and experience of the project. People with in-depth knowledge of the project were
interviewed from the IPs, district field staff of the Ministry of Social Welfare, Gender and Children’s Affair
(MSWGCA), a member of the District Emergency Response Committee (DERC), and local authorities were
each interviewed as key informants. The key informant interviews were conducted by SADRS’ research
coordinators. Three FGDs and KIIS, each, were conducted in each district. All KIIs and FGDs were audio-
recorded.
2.5 Data Quality, Management and Analysis The data was managed using standard systemised techniques.
2.5.1 Quantitative Data
To ensure data quality, all administered household questionnaires (by enumerators) were checked by
research coordinators in the field for review and coding, where applicable. This ensured the correct
questions were asked and issues are addressed by the relevant enumerator while still in the field.
All completed and verified questionnaires were then sent to Freetown to the data inputting clerks for data
entry. Quantitative data entry was done using Census and Surveys Programme (CsPRO) version 7.0 statistical
software. Upon completion of the data entry, data was exported to Microsoft Excel for further cleaning to
ensure that there were no additional errors generated at data entry stage. Double entry of data was done to
check for data entry errors and where disparities were identified, the original completed questionnaire
would be used to correct the anomaly.
The Excel cleaned data is then exported to SPSS version 20.0—a statistical software for data analysis and
generation of tables and charts.
2.5.2 Qualitative Data
Qualitative data was analysed manually; that is, the analysis of qualitative data from KIIs and FGDs included
listening to the audio-recorded conversation files and writing the information verbatim, arranging the
information by key evaluation categories (impact, efficiency, appropriateness, and effectiveness etc.), and
direct quoting. These were finally organised as per project main interventions and used to complement the
quantitative data in the report.
4In Bo and Port Loko districts, data was collected for 9 days each
5
3.0 Evaluation Findings
3.1 Introduction The findings of this evaluation are presented by project intervention components and based on the criteria of
evaluation. The demographic characteristics such as family size and gender are first presented. The section
will consider the findings of each of the components, that is, the Food Items (FIs), Non Food Items (NFIs),
Cash Transfers, and Teaching and Learning Materials (TLMs). The data is presented in graphs, tables and
figures, as deemed appropriate.
3.2 Demographic Characteristics of Targeted Beneficiaries A total of 313 respondents were interviewed, where 150 (47.9 %) were female and 163 (52.1 %) were male.
Majority (67.1 %) of the respondents were EVD survivors, 40 (12.8%) of the respondents were elderly
caregivers, and 63 (20.1 %) of the respondents were other categories of beneficiaries. In addition, 200
(63.9%) of targeted respondents were household heads, where 118 (37.7%) of the household heads were
males and 82 (26.2 %) of the household heads were females, highlighting the vulnerability and thus likely
dependency of humanitarian assistance for food and other basic needs during the EVD outbreak.
When asked about the size of their households during the quarantine period, 170 (55.0 %) of the
respondents reported having between 6 – 10 members in their households during the quarantine period.
The evaluation also sought to know the number of household members categorized by age groups that
benefited from CA Ebola emergency response project. The result shows that 23.3 percent of the project
beneficiaries were within the age cohort 0 -5 years, 21.9 percent of the beneficiaries were within the cohort
of 6 – 17 years, 17.6 percent were in the age group 51-56, and 16.0 percent of the beneficiaries were 65
years and above.
Figure 1:Percentage of beneficiaries by age groups
3.3 Project Management and Distribution Methods For Christian Aid to achieve their objectives, there were six key areas of intervention
Distribution of food items
Distribution of non-food items
6
Textbox 1(3.4) ‘’…most people received food support from SLSAV most
especially the survivors and elderly caregivers of vulnerable children in Kambia. The supply reached the people, especially the orphans’’
FGD Adult Female, Kambia
Direct cash transfers
Distribution of teaching and learning materials
Mobilisation and training of traditional healers
Engaging vulnerable groups in income-generation activities For analysis purposes, some of these areas will be merged. The evaluation assessed Christian Aid’s general management of the intervention and found that although CA monitored their Implementing Partners (IPs), it was however weak and they could improve on it. For example, up to the time of the evaluation, CA did not have a comprehensive list of beneficiaries for all districts. They had to rely on the individual IPs for a list which was eventually never comprehensive. Expectedly, CA should be in possession of such list as it is an audit issue. Secondly, there seemed to be limited coordination between and amongst IPs. As a result, there was hardly any sharing of lessons-learnt by the IPs during programme implementation. Although this could be done after the evaluation, it would have been more useful if done in real time. One IP could not learn from the mistake of another when lessons learnt are shared after programme implementation. It should however be noted that for the most part internal records were kept by individual IPs. Another management concern was the selection of the items in the package. One would have expected CA to review the items as the outbreak unfolded and new issues arose but been that the procurement process was centralized, CA stuck with the same package thereby limiting variety and choice. IPs were hardly involved in the planning phase of the project thereby including activities and items that were not context-sensitive. If IPs were involved in the procurement process, for example, appropriate items and potential delivery delays would have been addressed. Lastly, there were capacity issues with IPs. Although they mostly were able to cope, they were however not very prepared, capacity-wise, to handle a complex emergency situation as that of the Ebola response. CA would want to consider providing some training for their IPs in similar circumstances in future.
3.4 Target for Food Items From the quantitative data, interviews and focus group discussions, it was evident that the project was
successful in targeting quarantined homes, survivors, elderly caregivers and other categories of beneficiaries.
86.9 per cent of the respondents confirmed that they received support from one of the Christian Aid Ebola
emergency response project implementing partners. 13.1 per cent of the respondents reported that they did
not get any support from the CA project IPs.
Further analysis showed that 60.4 per cent of the respondents interviewed in Kailahun District acknowledged
receipt of food support from a project IP. In Bo
District, 94.6 per cent of the respondents
confirmed support from the project IP whilst in
Port Loko, Kambia and Tonkolili districts all the
respondents interviewed acknowledged receipt
of FIs from the project IPs. For respondents
interviewed in the Western Area Rural, 95 per
cent confirmed receipt of food support from the project IP. Overall, the proportion of respondents that
acknowledged receipt of food support is over 80 per cent and this shows that the project met its objectives.
7
This point is illustrated by a sentiment expressed by an adult female participant in a focus group discussion in
Kambia exemplified in textbox 1 above.
As a possible case for the low percentage of respondents in Kailahun District, it was gathered from the IDIs
conducted with the IPs from that district (MCSL and SEND) that support services were initially given by MCSL
but later transferred to SEND because the former could not submit to CA some required financial documents
in time. This change, as was acknowledged by the IPs, might have affected the smooth running of the project.
In that vein, after Social Welfare and the relevant IPs had initially agreed on a list of beneficiaries (which was
forwarded to CA), the IPs found out in the field that some these beneficiaries were already on other
agencies’ (such as World Health Organisation: WHO) distribution lists. To avoid duplication and waste of
resources, these prospective beneficiaries were not given the CA supplies. Other survivors and OVCs were
therefore given the items. The list of the additional/replaced beneficiaries were not supplied to CA—one
reason why some beneficiaries denied receiving food items as those were the ones dropped for appearing on
other IP lists for food and non-food distribution.
3.5 Food Items Component
3.5.1 The Efficiency of Food Items Distribution
The efficiency of FIs distribution component of CA’s Ebola emergency response project was assessed against
timeliness of the support and how long quarantined homes had to wait to receive FIs from project IPs.
Timeliness of the support: The evaluation found that 52.1 per cent of the respondents stated that the food
assistance they received from the project implementing partners was timely and 47.9 per cent of the
respondents reported that the support was untimely. Figure 2 below shows the respondents’ opinion about
the timeliness of the food support.
Figure 2: Timeliness of food support by district
Respondents were further asked how long they had to wait during quarantine period to receive food items
from IPs. Overall, the result shows that 14.1 per cent of the respondents stated that they received food
supply from IPs the same day of quarantine whilst 15.0 per cent of the respondents reported that they
received the FIs one day after the quarantine. 12.5 per cent of the respondents interviewed confirmed that
they received FIs two days after their houses were quarantined whilst 58.1 per cent of the respondents
8
acknowledged that they received FIs after two weeks into the quarantine period. 0.3 per cent of the
respondents reported that they received their FIs before their homes were placed under quarantine. The
chart below shows how long quarantined homes had to wait to receive FIs.
Figure3: How long quarantine homes had to wait to receive food supply
In-depth analysis of the evaluation data indicates that in Kambia District 2.2 per cent of the respondents
reported that they received FIs before their houses were placed under quarantine. 82.9 per cent of the
respondents in Port Loko District confirmed that they had to wait more than one week into the quarantine
time before receiving FIs from project IPs; and similarly, 62.5 per cent of the respondents in Bo District stated
that they had to wait for more than a week before receiving FIs from RADA. A good proportion of
respondents in Tonkolili and Western Rural confirmed receipt of FIs the same day their homes were placed
under quarantine, although this negates the IP interviews. The table below shows the percentage of
respondents and the time they had to wait to receive food supply form project IPs.
Table2: How long quarantine homes have to wait to receive food supply
Source: Evaluation household questionnaire data
Textbox 2 below is a statement made by an adult male in a Focus Group Discussion (FGD) in Port Loko District
expressing sentiments about when quarantined homes received food supply.
District Before quarantine
Same day of quarantine
One day after quarantine
Two days after quarantine
More than One week into quarantine time
Percentage of respondents confirming food supply
Kailahun 1.1% 2.2% 57.1% 60.4%
Bo 14.3% 8.9% 8.9% 62.5% 94.6%
Port Loko 8.6% 8.6% 82.9% 100.0%
Kambia 2.2% 11.1% 40.0% 37.8% 8.9% 100.0%
Tonkolili 37.0% 34.8% 13.0% 15.2% 100.0%
Western Rural
35.0% 10.0% 15.0% 35.0% 95.0%
Textbox 2 (3.5.1) ‘’The quarantined people received food after the quarantine period. When the first houses were quarantined the homes received food from local authorities most especially the Paramount Chief. When the second phase of quarantine happened, the SLSAV started supplying food items. SLSAV was the first NGO who intervened during the second phase of the quarantine. They did a lot in this community.” FGD Adult Male, Port Loko District
9
Textbox 4 (3.5.2) ‘’The CA project implementing partner SLSAV did a
very good job in the distribution of food items. Even though we were under quarantine and our neighbours were afraid to come close to give us help the SLSAV field staff took the food items to us and talk to nicely’’’ FGD Adult Female, Port Loko
Textbox 3 (3.5.2) ‘’The food supplied to us was good and enough, most
especially the rice and sardine. I regained my normal body status after three weeks when I was discharged from the treatment canter. I was eating enough and fine’’. FGD Mixed Youth, Bo District
3.5.2 Food Items Distribution Effectiveness
From the inception of the CA Ebola emergency response project, one of the desired goals of the FIs distribution component of the project was to address immediate food needs for quarantined homes, EVD survivors, orphans and other vulnerable people. It was also geared towards improving the nutritional status of beneficiaries, most especially orphans. This was mostly achieved as demonstrated in Textbox 3. The overarching5 goal of the FIs distribution by CA was to ensure that Ebola transmission is reduced by ensuring that quarantined homes had access to adequate and appropriate food and to ensure that quarantined homes didn’t have to break quarantine restrictions in search of food. The evaluation assessed the effectiveness of the FIs distribution by looking at the quality of the FIs distributed, the method used by implementing partners in the distribution FIs, the availability and adequacy of food items distributed and how organized were the field staff in the distribution process. The foregoing factors were examined in the paragraphs below. Quality of Food items distributed: Nine6 food items that were in the FIs distribution package were assessed to know the quality of FIs distributed to beneficiaries. Respondents were asked about the quality of the FIs they received. From the evaluation result, majority (92.6%) of the respondents stated that the quality of the FIs distributed by CA project IPs was good. 7.0 per cent of the respondents stated that the quality of the FIs distributed was reasonable, and 0.4 per cent of the respondents reported that the quality of the FIs distributed was poor. The textbox (3) above is a statement made by an FGD participant which reflects the level of satisfaction with the quality of FIs supplied by IPs. Assessing the distribution method: Beneficiaries were asked to judge the method used by IPs in the distribution of FIs to quarantined homes and other project beneficiaries. From the household data 24.6 per cent of the respondents interviewed stated that the method used by IPs in the distribution of FIs to project beneficiaries was very good whereas 61.7 per cent of respondents confirmed that the method used by IPs was good. 13.1 per cent of the respondents admitted that the distribution method of FIs by IPs was reasonable and 0.7 reported that the distribution method was poor. Out of the six evaluation districts, only in Kailahun about 2.2 per cent of those who received food items in the district reported that the distribution method used by IPs was poor. Majority of the food aid recipients in Tonkolili and Bo (78.3 and 73.2% respectively) confirmed that the food distribution method by IPs was good. Textbox 4 presented above is a statement by one of the female FGD participants in Port Loko District summarising the sentiments expressed by respondents.
5 Christian Aid –Crisis Response outcome: ICCO 1 Act formal proposal narrative.
6 Maggi, Tea leaf, Sardine, Vegetable oil, Powdered Milk, Salt, Sugar, Rice and Tomato Paste.
10
Figure 4: Percentage respondents judging the food distribution methods by project IPs
Availability and adequacy of most needed FIs in the package: The evaluation investigated to know if there
were any additional FIs that were mostly needed by beneficiaries at the time of the distribution and how
adequate were these FIs in the distribution package. The availability of the FIs was assessed at the household
levels while the adequacy of the needed food items was investigated during FGDs, IDIs and KIIs.
At household level, respondents were asked if most needed FIs were not available in food distribution
package. Analysis of the result indicates that, 43.1 per cent of the respondents admitted that there were
additional items that were mostly needed by them at the time of the distribution but were not included in
the food package, whilst 56.9 per cent stated that distribution package contained their most needed FIs. This
point is illustrated by statements expressed by participants during FGDs and KIIs as exemplified by the quotes
in textboxes 5 and 6 below.
How organised were the field staff in the distribution of FIs: During the Ebola crisis many people, including
humanitarian workers, had serious security and safety concerns to go close to communities or homes that
had been quarantined. This created a lot of food shortage for quarantined homes and other beneficiaries.
With this background, the evaluation sought to know how organised the IP field staff during the distribution
of FIs. When asked to describe the IP field staff and distribution process as either ‘very organised’,
‘organised’, or ‘not organised’, 36.4 per cent of the respondents reported that the IP field staff were very
organised
Textbox 5(3.5.2) ‘’The package was very rich and it contained about 19 items, mostly food stuff. At some point in time we complained about standardisation from government and other NGOs. But I often tell them that’s how the project was designed by CA and we cannot
alter it in the field’’- IDI, Bo District.
t alter it in the field’’- a statement by a project field staff in Bo District.
Textbox 6(3.5.2) ‘’Yes there was enough food for us, so we don’t need to go out in search
of food’’ FGD Adult Female, Western Area
Textbox 7(3.5.2) ‘’The distributors conducted the process in a fair and organized manner. They talk to people nicely
and they were also sympathetic to survivors and those that lost their love ones in the community’’. FGD Youth Participant in Bo Districts
11
Textbox 8(3.5.3) “I will prefer RADA because the one supply given to us was more than all the other supplies we received from other NGOs”.
FGD Adult Female, Bo District
while 62.3 per cent of interviewed beneficiaries affirmed that the field staff were organised. However, 1.3
per cent reported that they were disorganised. A similar question was posed to participants in FGDs to gauge
a wider view of direct and indirect beneficiaries of the project. One of the FGD participants’ statements is
illustrated by a quote in the textbox (7) below.
3.5.3 The appropriateness of Food items distributed
One of the objectives of the FIs distribution component of the CA Ebola emergency response project was to cover the immediate basic food needs of the quarantined homes, survivors and vulnerable groups in Ebola affected communities. The evaluation sought to know if the food items distribution was the right thing to do in the right way and at the right time. On this note, the appropriateness of the CA Ebola emergency response project was assessed based on: if beneficiaries’ expectation was in conformity with the FIs received from project IPs. At household levels, beneficiaries were asked if FIs they received from CA project IPs were in line with their expectation and also if urgently needed FIs were not included in the food package.
Food items received were in line with beneficiaries’ expectation: one of the objectives of the food items distribution component was to provide the immediate basic food needs of survivors and quarantined homes so as to eliminate the desires of quarantined people to break quarantine lines in search of food. The result shows that 63.6 per cent of the respondents indicated the FIs they received were in line with what they were expecting according to the prior information they received, whereas 36.4 per cent of the respondents stated that the food package received was not in conformity with their food expectation. One food item that many respondents thought was missing is fish. During FGDs similar questions were posed to participants to have a wider view of beneficiaries’ perception about the appropriateness of the food distribution. Several statements were made by participants. The text box above is a statement made by one female FGD participant in Bo District. However other statements were made by other participants disagreeing with the appropriateness of the food supply. Below is a statement made by a local authority (Town Chief) in Port Loko District expressing his disapproval of the manner in which a field worker treated them. Urgently needed FIs included in the food package: During the Ebola outbreak there were many food aid interventions by NGOs but people continued to question the type of food items distributed to quarantined homes, survivors and OVCs. The evaluation attempted to investigate the content of the food package distributed by the CA project IPs. In this regard, respondents were asked if there were any additional FIs that they urgently needed at the time of the distribution that were not included in the food package. The result shows that 56.9 per cent of the respondents stated no and 43. 1 per cent confirmed that there were additional FIs that they urgently needed but were not included in the food package. Further analysis shows
Textbox 9(3.5.3) “One of the field workers first visited us and informed us that he wants to come with food supply
for survivors and quarantined homes. Later he came back in a hurry and took the survivors’ cards and he collected the supply on behalf of this town but took the supply to Mile 36. He took the remaining food items to waterloo”. IDI, Port Loko District
12
Textbox 10 (3.5.3) ‘’When the quarantined homes received the food items from SEND, they stopped going out to hustle for food. Before the SEND food distribution, the virus was spreading faster in this community but when people received food items the spread of the virus stopped quickly in our community.’’ IDI, Kailahun District
that 14.1 percent of the respondents reported that they mostly needed corn soya blend (CSB) but was not included in the food package, 11.2 percent of the respondents claimed that they needed vegetable oil urgently but was not included in the package whilst 8.0 percent of them acknowledged that they urgently needed fish but was not included in the food package. The table below shows the percentage of respondents reporting the urgently needed food items but were not included in the food package.
Table3: percentage respondents reporting the food items the urgently needed A similar question was posed in FGDs to gauge a wider view of beneficiaries and the textbox above is an excerpt of the sentiments expressed by a community leader in a key informant interview. This further illustrates how the response was able to slow the spread of the disease, hence contributing to combatting the disease.
3.6 Beneficiary Target for Non Food Items (NFIs) The intervention was designed to target closed contacts of people infected by the disease especially in quarantined homes, survivors and care givers providing assistance/care to orphans who lost their parents to the disease. The evaluation found that overall, 83.1 per cent of the respondents acknowledged receipt of NFIs from CA implementing partners. Of this, 55.1 % were survivors while 10.2% were elderly care givers. About 18 per cent out of the 83.1 per cent were other beneficiaries.
Food Items Percent of respondents
Corn Flour 14.1 Cooking Utencils 1.9 Fish 8.0 Ground Nut 1.9 Onions 0.3 Palm Oil 2.6 Pepper 0.3 Rice 0.3 Sardines 1.6 Soft Drinks 0.6 Tea Leaf 0.3 Vegetable Oil 11.2 TOTAL 43.1
13
Table 4: Proportion of recipients of Non Food Items
3.6.1 Appropriateness of Non Food Items
As a result of the contagious nature of the Ebola scourge and the efforts by the government and its partners in containing and breaking the chain of transmission, all property formerly owned by Ebola survivors would be destroyed in order to decontaminate the environment and prevent further transmission. In that case, survivors usually have nothing left, as their property, including personal effects and kitchen paraphernalia, would have been destroyed by concentrated chlorine solution or burnt down. The expectation is that survivors would be provided new items to replace the lost ones. In many cases though, survivors were provided with a standard package across the board. The evaluation result shows that a one-size fits all package was given to all beneficiaries within the same category, hence not catering for special needs and situations. The evaluation asked if there were items they urgently needed but were not included in their packages. The study found that about six out of ten people (60.4%) reported that the items were appropriate while four out of ten (39.6%) said there were missing items they urgently needed. This indicates that a stakeholder consultation (including potential beneficiaries) prior to distribution of items is advisable.
Figure 5: Percentage of respondents reporting the availability of needed items in the package
39.6
60.4
Yes No
Of these, about six in ten respondents (57.1%) in Kailahun reported urgently needing items that were missing in the package. However, in Bo and Kambia districts at least eight in ten respondents reported that their urgent needs were met by the package.
Category of Beneficiaries Proportion of Respondents
Survivors 55.1%
Elderly Care Givers 10.2% Other Beneficiaries 17.8%
Overall 83.1%
14
Table 5: Percentage of Respondents reporting the unavailability of urgently needed NFIs in the package
District Yes No
Kailahun 57.1% 42.9%
Bo 19.6% 80.4%
Port Loko 51.4% 48.6%
Kambia 15.6% 84.4%
Tonkolili 47.8% 52.2%
Western Rural 35.0% 65.0%
The evaluation further asked respondents if the items received were in line with what they expected, according to the information they received before the distribution. Overall, about 64% reported that the items they received were in tandem with their expectations, while about 36% reportedly received less than they expected. In Kambia (93.3%) and Western Rural (92.5%) at least 9 out of 10 respondents said they received what they expected, whilst in Tonkolili (60.9%) and Kailahun (56.0%) about six in ten respondents said that they did not receive what they expected. Some respondents expected the package to include bedsheets/bedcovers.
Figure 6: Percentage of respondents reporting that the items were in line with their expectation
3.6.2 Effectiveness of NFIs Distribution
The effectiveness of the project intervention provides an indication of the quality of its implementation measures. As a result of urgency of the response, many partners provided different supports to ensure that the needs of the affected beneficiaries are addressed in a timely and effective manner. To this end, the evaluation assessed the timely provision of the food and non-food items during the emergency response period of the outbreak. Quality of NFIs received by beneficiaries The evaluation noted that 21.4% of the respondents reported that the quality of the NFIs distributed was
excellent while 68.7% rated it as good. Only about one per cent of the respondents rated the quality as poor.
15
Figure 7: Percentage of respondents reporting on the quality of NFIs
Proportion of respondents rating the quality of NFIs received by District When disaggregated by districts, 71% of the respondents in Kambia District rated the items as excellent while
only 3.3% of the respondents in Kailahun rated the items as excellent. It is worthy to note that in the Western
Area (Rural) 5% of the respondents felt that the quality of the NFIs was poor. Those respondents who
thought the quality of the NFIs was poor said that the items broke easily, shortly after they had been
distributed.
Table 6: Percentage of respondents by districts reporting the quality of NFIs District Excellent Good Reasonable Poor
Kailahun 3.3% 81.3% 15.4% -
Bo 10.7% 89.3% -
Port Loko 37.1% 54.3% 8.6% -
Kambia 71.1% 24.4% 4.4% -
Tonkolili 4.3% 91.3% 4.3% -
Western Rural 27.5% 47.5% 20.0% 5.0%
3.6.3 The Efficiency of NFIs Distribution
Efficiency is the extent to which the programme converted or was expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results in order to achieve the maximum possible outputs, outcomes, and impacts with the minimum possible inputs. Timeliness of the NFIs
Respondents were asked about how timely they thought the distribution of the NFIs was in relation to their
needs. Over half of them (52.1%) reported that the receipt of the items was timely. However, a statistically
significant percentage of people (47.9) reported that the receipt of the items was untimely. This is
particularly important because people’s perception of humanitarian assistance goes a long way to
determining their appreciation of it. This indicates that efficiency of the project could be improved. Below is
an indication of how timely the respondents thought of the NFIs.
16
Figure 8: Percentage of respondents reporting the timeliness of receipt of NFIs
Disaggregated by districts, while 73.3% of respondents in Kambia reported that it was timely, 94.3% of the
Port Loko district respondents felt that the time the non-food items were distributed was untimely.
Table 8: Percentage of respondents reporting the timeliness of receipt of NFIs
District Timely Untimely
Kailahun 69.2% 30.8%
Bo 33.9% 66.1%
Port Loko 5.7% 94.3%
Kambia 73.3% 26.7%
Tonkolili 58.7% 41.3%
Western Rural 47.5% 52.5%
3.7 Cash Transfer Component
3.7.1 Target for Cash Transfer
The cash transfer component of the CA Ebola emergency response project was designed to benefit 250 families (2,100 individuals) that have lost bread winners, 200 EVD survivors and 150 Other Vulnerable Children in 6 districts. The evaluation investigated to know if the beneficiaries received cash support from CA project IPs. 67.1 per cent of the respondents that received financial support from CA project IPs were survivors, 12.8 per cent were elderly caregivers and 20.1 per cent were other7 categories of beneficiaries interviewed.
Figure 9: Percentage of respondents acknowledging receipt of cash from project IPs
Categories of beneficiary
Did you receive money from project IPs during the Ebola outbreak as a support to
your family
Total
Yes No
Survivor 34.5% 32.6% 67.1%
Elderly Caregiver 9.6% 3.2% 12.8%
Other 18.5% 1.6% 20.1%
Source: Evaluation household questionnaire
7 They include widows, those that have lost breadwinners and women in agriculture.
17
Textbox 12 (3.7.2) ‘’Late last year when things were really difficult for me and not expecting any financial assistance
from anybody and as a widow and a caregiver for an Ebola orphan, one SEND worker informed me that my name was on their list to receive cash’’. FGD Adult Female, Kailahun District.
3.7.2 Cash Transfers Effectiveness
Ideally, to effectively implement and achieve the objectives of a project, the project should have an approved
policy to be followed. The project policy stipulates: who should be a beneficiary, the selection and targeting
of beneficiaries and beneficiaries to have pre-knowledge of their entitlement to increase the effectiveness of
the project. The evaluation of the CA Ebola emergency response project is based on the above mentioned
factors and these factors are examined below.
Availability of Stipulated project policy on beneficiaries: During the review of the project documents,
information sourced from project documents indicates that CA’s Ebola emergency response project cash
transfer component was designed to provide cash to families that had lost breadwinners, EVD survivors and
elderly caregivers of vulnerable children. This seems to indicate that the project had a specific policy guide in
place regarding intended target beneficiaries.
To find out how the cash transfer effectively responded and was appreciated by the beneficiaries, FGD
participants were asked to express their opinions on the extent to which the cash transfer benefits target
groups in the community. The text box below is the statement made by one elderly caregiver as a participant
in an FGD in Bo District.
In this regard, the project has a policy/guideline for implementing the cash transfer component and the availability of this policy helps to ensure project effectiveness and also its efficiency. Selection and Targeting of Beneficiaries: Considering the lucrative nature of the cash transfer component of the project coupled with limited livelihood strategies and opportunities in most EVD affected communities at the time of the project implementation, everyone would want to be included as beneficiary for the cash transfer. To safeguard this, the evaluation found that the IPs collaborated with the MSWGCA and District Ebola Response Committee (DERC) to identify beneficiaries. The text box below is a sentiment expressed one FGD participant showing how the technique was effective.
In addition to the above sentiment expressed by the FGD participant, household respondents were asked if they gave anything to anybody in return for the cash they received and 94.4 per cent of the beneficiaries interviewed indicated that they did not give anything to anybody in return of the cash they received. Beneficiaries having pre-knowledge of cash entitlement: In any project implementation, it is ideal for beneficiaries to have foreknowledge of their entitlement, as this will help to ensure effectiveness and efficiency of the project. The evaluation sought to know at household level if beneficiaries had pre-
Textbox 11 (3.7.2) “I am aware of the cash transaction given by RADA to orphans and survivors last year after the quarantine period. I was part of it; we were given Le 500, 000 each as widows and Le200, 000 for orphans. That was the highest money I handled at a time last year for myself and my two surviving orphans”. FGD Adult Female, Bo District
18
knowledge of the amount of money they were supposed to receive from CA project IPs. 75.6 per cent (149) of the respondents interviewed who received money from the IPs stated they had no pre-knowledge of the amount of money they were entitled to and 24.4 per cent (48) confirmed that they had pre-knowledge of the amount of cash they were supposed to receive. Respondents were further asked about who informed them about the amount they were supposed to receive. The figure below illustrates the percentage of respondents indicating the source from which they had the information about the amount of money they are supposed to receive from project IPs.
Figure 10: Who informed the beneficiaries about the amount money they are supposed to
receive.
3.7.3 Cash Transfer Component’s Relevance and Appropriateness
In recent times many critics have often questioned the cost-effectiveness of the distribution of FIs during
emergencies and also the inability to reach the intended beneficiaries. This idea is getting widely accepted by
many relief agencies to resort to cash transfers in emergency situations.
3.7.4 Justifying the Cash Transfers in the Ebola Emergency Response
Various stakeholder discussions were held at district level, to gauge the perception of the respondents on the justification and appropriateness of the CA Ebola emergency response project cash transfer component. Most of the stakeholders interviewed believed that the rationale on the part of CA to have chosen cash transfers as a humanitarian response for EVD victims was justifiable because of the following reasons:
The scale of suffering caused by the EVD: Many stakeholders highlighted that the EVD caused a widespread of human suffering and many people suffered severe consequences as a result of the EVD outbreak in country and the need to provide humanitarian assistance in the affected communities for survivors, elderly caregivers, widows, orphans and people that have lost breadwinners to meet their basic needs and enable these victims to re-establish their livelihood is unquestionable. Cash was suggested as a way to help alleviate that suffering, especially when they could choose what to do with it.
Many interventions provided in the form of food and non-food items by other projects: During the EVD outbreak, there was huge humanitarian response by many NGOs to the EVD victims in many communities. It was highlighted by many stakeholders during evaluation field visits that CA’s cash transfer initiative to EVD victims is unique and as such, the cash transfer component was considered to be highly complementary with other humanitarian relief interventions undertaken by many other NGOs such as the distribution of FIs and NFIs. The text box below indicates the view of a key informant in Port Loko District.
19
3.8 Observing Humanitarian Standards
Humanitarian assistance rendered during emergencies has certain standards to respect which are essentially
to ensure that humanity was respected and basic standards were met during implementation. The evaluation
examined three key issues in this respect:
i. Payment of cash or kind to receive items
ii. General respect accorded to beneficiaries
iii. Knowledge of complaint procedures
3.8.1 Payment of cash or in-kind for the FIs received
If project beneficiaries did give any cash or in-kind to anybody for the food items they received was gauged by the evaluation. Respondents were asked if any cash or in-kind was given to anybody by beneficiaries for the items they received. From the analysis, the result revealed that 97.8 per cent of the beneficiaries interviewed stated that they did not give any cash or in-kind to anybody for the food items they received whilst 2.2 per cent of the respondents interviewed confirmed that they did give cash or in-kind to someone for the food items they received. Further analysis of the result shows that 5.7 per cent of beneficiaries who received FIs in Port Loko District alleged that they paid cash or kind to someone for the food items they received and such allegation was not reported by beneficiaries in the Western Rural. 2.2 per cent of beneficiaries that received FIs from CA project IPs in Kailahun, Kambia and Tonkolili stated that payment of cash or in-kind was given to someone for the food items they received. The table below shows a percentage summary of beneficiaries that reported paying cash or in-kind to someone for the FIs they received.
Table 9: Percentage of respondents that reported giving cash or in-kind for items received
District Did you give any cash or in-kind to anybody for the food items you received during the distribution?
Percentage of respondents receiving FIs
Yes No
Kailahun 2.2% 58.2% 60.4%
Bo 1.8% 92.9% 94.6%
Port Loko 5.7% 94.3% 100.0%
Kambia 2.2% 97.8% 100.0%
Tonkolili 2.2% 97.8% 100.0%
Western Rural 0.0% 95.0% 95.0%
Textbox 13 (3.7.4): ‘’I have witnessed in many situation the sales of Ebola emergency relief items most especial the food items and also many cases the relief items provided by NGOs do not match the beneficiary needs and one of the main advantage of cash transfers intervention is that the beneficiaries have the power to decide what they want to buy with the money’’. IDI, Kailahun District
20
3.8.2 General respect accorded to beneficiaries
Asked how respected beneficiaries felt with the way the IP treated them during distribution, 99.4% of the respondents reported that IPs treated them respectfully. It is worthy of note that Tonkolili (2.2%) and Western Rural (2.5%) respondents reported being treated disrespectfully by IPs. Details of ‘respect’ and ‘disrespect’ were not probed.
Figure 11: Percentage of respondents reporting how respectful the IPs treated them
3.8.3 Knowledge of Complaint Procedures
The evaluation sought to know if the beneficiaries were aware of any complaint procedure if they felt their
rights were violated. Overall, only 8.9% of the respondents acknowledged that they knew where to lodge
complaints if they wanted to. This is a little worrying as it points to management and monitoring lapses.
Ideally, this should have been embedded as a key sensitisation component of the project, as instances
abound across the world where aid workers violate their beneficiaries’ rights. It is worthy to note that 100%
of respondents in Kailahun and Western Rural districts reported that they knew nowhere to complain, in case
they had any issues with the process.
Figure 12: Percentage of respondents having knowledge of complaint procedure
21
4.0 Limitations of the Evaluation The evaluation was carried with a few challenges. Below are the key ones and how we mitigated them:
Rainy Season: Considering that data was collected in November (where it still rained a little), data collectors
were provided with plastic folders to protect the documents.
Bad Road Network: Many of the roads in the provinces are very bad. In that regard, we discouraged night
travels and over-speeding to prevent accidents.
Transportation: Although SADRS had a pre-contract agreement for Christian Aid to provide vehicles to drop
off and pick up enumerators, in one chiefdom, for each district, Christian Aid expressed at last minute that
they experienced constraints and could only provide one vehicle. SADRS then resorted to hiring commercial
vehicles which incurred extra loss to the firm.
Beneficiary List: Up to the eve of the training Christian Aid/IPs could not provide a comprehensive list of
beneficiaries. This was a very serious challenge as it prevented the consultancy from employing a complete
randomised sampling, as planned. One of the beneficiary lists provided was completely illegible and
therefore not usable. In many instances, the closest identifying information against names is chiefdoms, after
we had stressed the need to have specific addresses, localities and phone numbers, where possible. CA
suggested that IP representatives would help us identify these beneficiaries (which was less than ideal) but
eventually did in some cases. The team pointed out that this could possibly influence the response and a
likely cause for social desirability8.
CA Rep at Training: In spite of calls for a CA representative to be at the training to give an overview of the
project and its activities, especially those changes/amendments that may have occurred post-proposal
period, including method of distribution, CA responded that it’s against their policy as it could cause bias,
although the team thought otherwise. SADRS’ technical team leads played this role.
Other Logistics: Also, in our pre-contract agreement, CA promised to provide a projector for the training,
however, and on the eve of the training the agreement was reneged. This resulted to the Consultancy hiring
a projector throughout the period, at a financial cost to the firm.
8 Simply put, social desirability can be described in research as a situation where the research participant tells the
researcher what the participant thinks the researcher would like to hear, even if it’s not the truth.
22
5.0 Conclusion and Recommendations
5.1 Conclusion Christian Aid’s ICCO-funded Ebola response project as implemented through various partners across the four
regions of the country was considered by many to have been well-conceived by the project designers,
especially within the context of an emergency situation. It was evident that the project was successful in
targeting quarantined homes, survivors, elderly caregivers and other categories of beneficiaries which
injected a feeling of care and hope in a situation of fear and despondency. This position is further buttressed
in that majority of the respondents confirmed that they received support from one of Christian Aid’s Ebola
implementing partners and huge percentage (92.6%) of the respondents were also satisfied with the quality
of the FIs distributed; with 52.1 percent of the respondents stating that the food assistance they received
from the project implementing partners were timely. SADR observes that although 47.9 percent of the
respondents confirmed the support was untimely, yet this could not have largely affected the project’s
implementation and objectives.
The findings showed that the EVD response project indeed provided emergency relief assistance and through which, succeeded in contributing to reducing the stigma surrounding those who had contracted the disease and enhanced their reintegration back into their families and communities. The latter was especially true for the children, whose non-food items like toys and clothes easily attracted the company of others of their age group. However, SADRS observed that despite that CA’s Ebola response packages were (in comparison to other partners’) bigger and holistic (comprising many items), the supply could not march the demand and as a result, many “would-have-been beneficiaries” went without. Like that of the food and non-food items, majority of the beneficiaries received the cash assigned. However, CA would want to consider increasing the amount as to be able to kick-start any enterprise and the notion of giving cash to beneficiaries to start agriculture at the time when planting season was almost over was a huge challenge. With the exception of few9, most of the beneficiaries saw the cash as an extension of the food items as monies were mostly used to provide food in the homes. On project management, the selection technique of beneficiaries of all of the IPs contacted was to a large
extent effective and transparent. All of the IPs maintained internal reporting system, especially between the
field and office staff of the IPs and periodic reports were sent to CA. However, consistent field monitoring of
IPs’ activities by CA was weak, and being an emergency situation, it was sometimes difficult to sit and wait for
feedback before actions were taken.
It was further observed that most IPs depended strongly on social mobilisation (or partnership with other donors) to implement project; despite the fact that the figure heads of most of the IPs had had experience in some kind of relief work, yet there were limited staff strength for most of the IPs to cope with the immediacy of the job they had to do. Although such partnership made implementation more transparent and less stressful for partners, working with multi-social institutions sometimes caused delays; IPs were requested to attend several meetings in a week which sometimes affected timely distribution10. Maybe as a result of the emergency situation, SADRS observed that there was weak and inadequate assessment of IPs and this further
9 There were however one or two isolated stories of successful impact ; for instance like the story of a female survivor at Baomahun
at Bo District – she was reported to have used the cash to buy one jerry can of palm oil (5 gallons) and is now trading that by pint at the local market. 10
It should be noted that in an emergency situation of that magnitude, it would have been difficult (at least for information sharing purposes) for any one partner to work without attending these meetings.
23
affected planning, implementation and communication. For instance there was a case of an IP’s contract being terminated and services transferred to another that was not even shortlisted by CA for the same contract. During the implementation CA was more concerned with strategic level activities (e.g. procurement) and less involved in the monitoring process of field activities. Similarly, procurement was centralised in Freetown and as a result, a one size-fits-all package was delivered to beneficiaries. For instance SADRS observed that men were supplied sanitary pads, pots supplied without covers and some children had to exchange their sandals because their sizes were never supplied. Though few, some IPs reported that some of the foods were spoiled before getting to the beneficiaries.
5.2 Recommendations General Management
I. CA to Monitor IPs during implementation: The Ebola response project was widely acknowledged and considered appropriate by both IPs and majority of the beneficiaries. However for future programmes, project monitoring should be more robust on the part of CA. CA should not just depend on the field reports of IPs but their M&E Department should assign staff to periodically monitor projects implementation. This will make it easier to address field-related challenges, share lessons learnt in real time, and improve on coordination and monitoring of the overall implementation.
II. IPs to be carefully selected and trained: The selection of IPs for future projects should be more thorough to avoid termination of contracts midway project implementation. Being an emergency situation (which may have the tendency of affecting thoroughness), focus should be on IPs that are resident in the localities rather than contracting IPs at central level. In a case where an IP at local level has the requisite experience but has limited staff capacity to absorb services required, project should support such IPs with training packages or to embark on short term employment contracts to scale up capacity. Although cost might be slightly higher with such arrangement, yet CA would have helped improved the local partner and CA or other NGOs needing IPs might benefit from the services of such IPs in the future.
III. IPs’ involvement in planning could be improved: Once contracts are signed, IPs should be granted some leverage to make input into the planning process at the strategic level, especially on issues relating to challenges encountered in the field. For instance the procurement and distribution of food and non-food items would have been less challenging if the input of the IPs was constantly sought. Future needs assessments of beneficiaries would be better done with more involvement of IPs as ‘partners’.
IV. More post-Ebola follow- up programmes for sustainability: To sustain impact, more post-Ebola follow-up programmes should be considered, especially for the women and children who stand to suffer most when the income of the household drops or fails. There were glaring stories of survivors, orphans and OVCs whose livelihoods have been threatened since they returned to live with extended family members. CA may wish to follow-up to enhance the re-integration of these children or liaise with child focused NGOs to achieve such goal.
V. Complaint procedures to be made known to beneficiaries: Because of the duty of care, where feasible, CA should consider increasing awareness of their prospective relief aid beneficiaries of complaint procedures in case they suspect abuse of the process.
24
Food and Non-Food Items
VI. Procurement to be decentralised: In line with recommendation (III) above, where possible, the procurement of food and non-food items in the future should not only involve the IPs but done at regional or district levels to minimise damage to non-food items, spoilage of food items and delays in supplies to the final beneficiaries. Or where procurement is done at central level, suppliers should be informed that goods are considered acceptable when delivered at regional or district levels. This could improve on project’s effectiveness and efficiency.
VII. Packages to be need-specific: Although most of the respondents generally reported that most needed items were available and adequate in their packages, yet CA should in the future disaggregate items and packages by sex and age categories to enhance beneficiaries’ usage and satisfaction with their packages. A one size fits all package support should be avoided; better inventory of the needs of the beneficiaries will be required.
Cash Transfers
VIII. Volume of cash and knowledge of IP’s purpose for cash transfer be increased: One very laudable intervention of the Ebola response project was the cash transfers. However, to deepen the impact of such in the future, the amount of cash should be increased to an amount that can feasibly start a business. The amount of cash transferred was reportedly small and could not start any “meaningful business”; most ended up using it to buy food. CA in the future may want to encourage beneficiaries to form corporative groups around particular business proposals from beneficiaries, and a basket or pool fund made available to the groups rather than to individuals. This could be under the supervision of the IPs or directly supervised by CA M&E department. Similarly, cash was handed to beneficiaries (most of whom had limited or no business acumen) without any clue as to what could be done. Future cash transfers could be done with business orientation discussions or talks that would motivate, re-orientate and help beneficiaries better understand the purpose of the cash transfers.
25
6.0 ANNEX ANNEX 1
ICCO-funded Christian Aid Ebola Response Programme, Sierra Leone
1 Nov 2014 to 30 November 2015
End of Programme External Evaluation
Terms of Reference
I. Background
Christian Aid in Sierra Leone with funding from ICCO (1 November 2014 to 31 August 2015) was able to
respond to the Ebola Virus Disease (EVD) epidemic through supporting partners in 9 districts to provide
emergency relief assistance and to carry out various activities in support of Ebola survivors, people in
quarantined households, and orphans and other vulnerable children (OVCs).
By providing water, food and Non-Food Items (NFI’s) to quarantined homes the need to break the quarantine
to seek essential supplies was removed thereby reducing the risk of the infection spreading further. Providing
survivors with food and other basic needs aimed to reduce the stigma surrounding those who had contracted
the disease and supported their reintegration into their families and communities.
ICCO Funding
Phase 1 100,000
Phase 2 200,000
Addendum 101,882
Total in Euro 401,882
ICCO-Funded Activities
Phase 1:
Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners
Phase 2:
Outcome 1: The basic needs of 250 families (2,100 individuals) that have lost bread winners, 200
survivors, and 150 OVCs are provided in 6 districts with cash and food packages;
Outcome 2: 150 survivors and Ebola widows have established income generation activities (IGAs),
started agricultural activities, and successfully reintegrated in their communities.
Outcome 3: The spread of EVD is mitigated and stigma reduced by providing food and NFIs to 150
quarantined households and mobilising 50 traditional healers as community health workers for
infection prevention and control (IPC)
26
Additional Activities funded under the Addendum:
Outcome 4: Support to elderly caregivers (provision of micro grants with related training for
livelihoods inputs such small animals, seeds and vegetables to allow elderly caregivers to sustainably
support vulnerable children)
Outcome 5: 285 orphans and vulnerable children in Ebola-affected communities in Tonkolili District
provided with primary and junior school learning materials and supplementary food to enable them to
restart their schooling
RMA
Research: ‘Keeping the Faith’ research on the role of faith leaders in the Ebola response (in
collaboration with Tear Fund, CAFOD and Islamic Relief)
Disaster Risk Reduction (DRR): 1-2 day workshop for partners and CASL, DCI & RADA staff
27
Partner Activities
District Partner Activities Quarantined & Survivor Households,
Families that have lost Breadwinners, Widows
Orphans & Vulnerable Children
(OVC)
Elderly Caregivers Traditional Healers
Planned Achieved Planned Achieved Planned Achieved Planned Achieved
PHASE 1 1 Nov 2014- 31 Jan 2015
Kailahun Methodist Council Sierra Leone (MCSL)
Food & NFIs 210 180 0 30
Bo & Pujehun
Rehabilitation and Development Agency (RADA)
Food & NFIs 200 100 0 100
Port Loko
Sierra Leone Social Action Volunteers (SLSAV)
Food & NFIs 100 100 0 0
PHASE 2 +Addendum 1 Feb 2015- mid-Sept 2015
Bombali Community Action for Human Security (CAHSec)
Outcome 3: Food & NFIs to quarantined households; Mobilising 50 traditional healers for IPC (Infection prevention & control)
50
50
50 50
Port Loko
SLSAV Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners
100 100 50 50
Kambia
SLSAV Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners
100 100 50 50
Tonkolili Real Women in Action Programme (REWAP)
Outcome 4: Support to elderly caregivers (extension to 15 Sept) Outcome 5: Support to 285 orphans and vulnerable children (180 primary; 105 junior/secondary) with primary and junior/secondary school learning materials and supplementary food to enable them to restart their schooling
285
285
50
50
Western Area Rural & Western Area Urban
Network of People Living with HIV (NETHIPS)
Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools
50
50
50
50
28
Kailahun SEND-SL
Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools
100
50
100
50
100 100
Bo
RADA Outcome 1: Food, NFIs, cash to survivors, OVCs, & families that have lost breadwinners Outcome 2: Survivors & widows supported with IG activities - seeds & agricultural tools
100
50
100
50
50 50
29
II. Purpose of the Evaluation
This final over-all programme evaluation will have the following two-fold purpose:
1. To assess and validate the outcomes and impact of the response from the standpoint of beneficiaries
and partners, and;
2. To capture and synthesize Christian Aid (CA) and partner’s learning from the response.
III. Key questions the evaluation must address
1. On outcomes and impact :
a. What has been achieved by the response, where and for whom?
b. What intended and unintended changes did the programme bring to target communities and
partners?
c. How has the response addressed local needs and priorities in combatting the spread of Ebola?
2. On learning :
a. What were the learnings of CA and partner? How have these been documented, shared and used to
improve the response? What can be replicated for future humanitarian responses?
b. What were the downward accountability mechanisms established within the programme delivery
process? What and how can they be improved?
c. What strengths, weaknesses, benefits and constraints resulted from engaging a consortium of
partners? What value has been added by working with new partners?
IV. Evaluation Outputs
1. An inception report by the external consultant setting out an evaluation plan, methodology/design,
and timetable
2. Draft and final report
3. Management response to the evaluation on the draft report
4. Learning event to present the key findings of the evaluation
The final evaluation report should be no longer than 10 pages, excluding any appendices, and include an
executive summary. This should be addressed to the evaluation steering group and will be shared with the
Donor and/or publicly on Christian Aid’s website with the full synthesis and country reports available upon
request.
V. Methodology
The final detailed methodology will be developed by the external evaluator and reviewed by and agreed with
the evaluation steering group. It will include but not limited to the following:
A desk review of existing documents (partner proposals and reports, and other related documentation)
Quantitative and qualitative analysis using participatory methods with a sample of direct and indirect
beneficiaries
30
Interviews, discussions and debriefing sessions with partners and CA staff in Sierra Leone and UK via
Skype
VI. Evaluation Team
The evaluation team will be made up of:
1. A team leader/external lead evaluator with a strong background in humanitarian evaluation
practice specifically using participatory approaches and with an excellent understanding of the Sierra
Leone Ebola context. S/he will work with an external assistant evaluator with strong understanding of
emergency response and application of humanitarian standards to assist in country with data
gathering / interviews
VII. Management
The evaluation team will report to Christian Aid’s Regional Emergency Manager (REM) for West Africa.
The evaluation steering group will oversee the management of the evaluation and ensure it fulfils its
purpose. This group will sign off the ToR and then meet as required. This is likely to be at least thrice – to
agree and sign off the ToR and inception note and to review and collate the feedback on the initial findings
from the evaluation. They will also assist in communicating the findings of the evaluation.
Suggested membership: REM, ICCO representative, and CASL Country Manager/Policy & Programme
Manager and Emergency Response Programme Manager.
VIII. Timeframe
Recruitment of evaluation team – 1 week
Desk review – 1 week
Inception report drafting and finalisation
Logistics and admin preparation for field work
Field work – 1 week
Draft report submission – 1 week
Steering group review of report & drafting of management response
Final report submission – 20th November 2015
Top Related