UNDP_GEF_SGP_Project_Impact_Evaluation_Research_Application of the Propensity Score Matching...
-
Upload
yohannes-mengesha-phd-candidate -
Category
Environment
-
view
305 -
download
0
description
Transcript of UNDP_GEF_SGP_Project_Impact_Evaluation_Research_Application of the Propensity Score Matching...
\
By: Yohannes Mengesha, Jema Haji and Yonas Abera
Dire Dawa Administration Environmental Protection Authority
Socioeconomic Impact of the UNDP/GEF-SGP Funded Community-Based Climate Change Mitigation Projects in Ethiopia: A Case from
Dire Dawa Administration
April 2013
Dire Dawa
1
ABSTRACT
Since July 2006, UNDP/GEF-SGP has supported 75 projects in Ethiopia, out of which, 13 of
them were being implemented in Dire Dawa Administration. Taking the already phased out
four community projects as a reference, this study examines the socioeconomic impacts of the
program in the Administration. A cross sectional survey of 160 households (80 from the project
beneficiaries and 80 from non-beneficiaries) was undertaken to examine and evaluate the
impacts of the program on households’ livelihood. Descriptive statistics coupled with an
econometric model was used to analyze the data collected from different sources. The
descriptive analysis of this study indicates that the societies are becoming better off in their
livelihood due to the existence of the project despite the fact that it is associated with many
challenges. Applying a propensity score matching technique, the study has figured out that the
level of monthly income, asset and monthly consumption expenditure of the program
beneficiaries are higher than that of non-beneficiaries. Generally, the major findings of the
study showed the average effect of the program to be positive and statistically significant;
suggesting that the program has achieved its stated objectives of improving the socio economic
conditions of the local community and the environment. It is also suggested that the project
would have the capacity to improve the livelihood of the beneficiaries further if corrective
measures are taken to tackle the challenges faced by the project.
2
I) INTRODUCTION
A. Background
Dire Dawa Administration is situated in the eastern part of Ethiopia. The region is known to be
a low land with much of its land lying in semi-arid and arid lands with series of escarpments at
the southern, south eastern and south western directions. Within the region, there is high
altitudinal variation consisting of mountain ranges with slopes, hills, valley bottoms, river
terraces and flat plains and most of the series of barren hills and mountains seen with their
natural vegetation depleted and bed rocks exposed (DDEPA, 2012).
The region is also characterized dominantly with warm and dry climatic situations associated
with arid and semi- arid in nature. Rainfall deficiency and absence of surface water, highly
eroded and shallow soil depth, and significant decline in the overall forest resources and bio-
mass cover mainly due to unregulated deforestation process are common features observed in
the region (DDEPA, 2012).
Dire Dawa Administration is organized under 9 urban and 38 rural local Kebelles.
Accordingly, in the region, there are two distinct, rural and urban, economic activities both of
which are faced with very serious food insecurity problem due to a combination of socio-
economic factors and increasing fragility of the ecosystem (ibid).
Environmental degradation in its different forms, erosion and deforestation in particular
together with increasing population pressure has led to a decline in productivity and serious
disturbance on the natural ecosystem. The situation has resulted in significant natural disasters
due mainly to flash flood hazards in the rainy seasons. In addition to the flash flood hazards,
the unsustainable utilization of natural resources and the absence of appropriate policy
measures contributed towards increasing vulnerability of the community (DDEPA, 2013).
Land degradation coupled with vegetation cover loss is being the most serious problem facing
the rural communities of Dire Dawa Administration which is aggravating the consequence of
climate change disaster and biological diversity loss. The areas are vulnerable to the impacts of
climate change because of factors such as wide spread poverty, recurrent drought and over–
dependence on rain–fed agriculture and other socio economic factors (DDEPA, 2013).
3
The recurrent drought and associated absence of alternative source of income has forced the
local communities to clear more vegetation to generate income through the selling of fuel
wood and charcoal. Hence, this forced the area to become heavily reliant on massive food aid
and a front line victim of drought and continuous famine that has jeopardizing development
interventions in the area. These all, directly or indirectly lead towards over exploitation of
natural resources contributing to the problem of climate change and poverty (ibid).
The Local Government is making efforts to address these adverse conditions and has designed
coping mechanisms. In fact, some of these efforts have been promising and have brought about
strategies that have induced changes in the attitude of the affected local communities. Some
strategic measures include the development and implementation of national environmental
initiatives, as well as policy/program and project initiatives that directly and/or indirectly
address climate change and adaptation mechanisms. These initiatives could be capitalized for
mitigating the undesirable consequences of climate related hazards, while lobbying for and
seeking international solidarity and assistance in the form of financial, technical and
technological resources (NAPA, 2007, as cited in DDEPA, 2013).
With some assistance from non-governmental organizations and the government, small-scale
farmers and pastoralists are adopting a variety of coping mechanisms. In the farming areas,
many are shifting to more drought tolerant crops and varieties, improved forest management
practices, diversified energy sources, and alternative means of income from off-farm activities
(DDEPA, 2013).
B. The UNDP – GEF/SGP Project
The UNDP funded GEF- Small Grants Program (SGP) is a global cooperate program
implemented by UNDP on behalf of Implementing Agencies (IAs) and Executing Agencies
(EAs) of the Global Environment Facility and executed by the United Nations Office for
Project Services (UNOPS). Launched in 1992, GEF SGP is rooted in the belief that global
environmental problems can best be addressed if local people are involved and direct
community benefits and ownership are generated (DDEPA, 2013).
4
Since its inception in 1992, the Global Environment Facility Small Grants Program (SGP) has
been promoting grassroots action to address global environmental concerns. SGP aims to
deliver global environmental benefits in the GEF focal areas of biodiversity conservation,
climate change mitigation, protection of international waters, prevention of land degradation,
and elimination of persistent organic pollutants through community-based or bottom-up
approaches. Along with these five major objectives, the project is targeted on achieving
poverty reduction and local empowerment of the societies. Special concern is also given to
local and indigenous communities as well as gender concerns. GEF/SGP supports the larger
sustainable development goals and the achievement of key components of the Millennium
Development Goals (DDEPA, 2013).
In Ethiopia, SGP was officially launched in June 2006 and funds community-based initiatives
aimed to build the capacity of organizations as a necessary element for sustainable use of their
natural resources and deliver global environmental benefits in the GEF focal areas. Ethiopia
has been benefiting greatly from the financial support of the Global Environmental Facility-
Small Grant Program, with the particular aim of increasing the capacity of local communities
to conserve biological diversity, abate climate change, protect international waters, reduce the
impact of persistent organic pollutants and prevent land degradation, through a variety of
bottom-up approaches (ibid).
Ever since its establishment in the country, the GEF-SGP provides funding to more than 1,250
local CBOs and NGOs that are designed to improve livelihood of the community especially,
poor and vulnerable people, without regard to their ethnic, political, or religious association
and contribute positively to the global environment through local level actions. Communities
can thus improve their own quality of life while pioneering approaches that may become
increasingly important on global scales. Out of the total project found in our country, 13
projects found in the Dire Dawa Administration and Grants so far approved for these projects
worth more than 4 million birr (US$ 333,400.00) (ibid).
In Dire Dawa Administration, during the implementation period of the project, (from
December 2009 until November 2011) various planned activities have been accomplished by
each CBO in collaboration with other partners in order to achieve the expected and desired
outcomes and results of the projects. Most of the activities are, of course, related to combating
5
land degradation, mitigation of climate change and livelihood improvements which include one
or more of the following activities (DDEPA, 2013):
Construction of physical soil and water conservation activities such as terrace, check
dam and drainage on closed areas;
Raising and planting of agroforestry, fruits and forage seedlings respectively;
Procurement and distribution of nursery and SWC equipments to beneficiary groups;
Provision of agroforestry, fruits and forage seeds;
Provision and distribution of seedlings;
Popularization and familiarization of fuel efficient stoves;
Spring developments to improve the irrigable capacity;
Construction of water wells for use in vegetables production;
Preparation and transferring of compost on selected farm lands;
Training on fuel efficient technologies, compost making, fodder development bee
keeping and other activities;
Capacity building for administrative bodies on financial and human resource
management;
Undertaking community travel workshops for community representatives; and
Conducting continuous community discussion to empower sense of ownership during
the implementation of the project and etc.
Given the activities implemented by the project, it is expected that the societies in reach of the
project benefit from the activities performed by the project thereby enjoy improvement in their
livelihood. However, the significance of the impact of the project on the livelihood of the
beneficiaries has to be empirically justified. In cognizant of this, this study was undertaken to
figure out the significance of the impact of the project as well as the extent to which livelihood
of the beneficiaries has changed due to the existence of the project. Moreover, attempts were
made to identify the major challenges facing the project.
II) METHODOLOGY
A. Data Sources and Methods of Data Collection
6
For this study, both primary and secondary data were used. The primary data was collected
using a survey method. This survey constitutes sampled households from the project
participants as well as from non-participants. In addition, existing project documents, including
annual and periodic progress reports (from DDEPA & CBOs offices) and other related written
and audiovisual documents were also reviewed to gather secondary data.
B. Sampling Design
This study has focused on four city and rural kebelles of the Administration including Gorro,
Gendegara Gendegola, Koriso, and Halole – Iffa based on the terms of reference as requested
by the Environmental Protection Authority Office of Dire Dawa (DDEPA). Using the
probability proportional to size (PPS) sampling, 80 beneficiary households were randomly
selected from a list of all the beneficiary households in the four kebelles. Similarly, 80 non-
beneficiary households were selected using the same approach from a list of all non-
beneficiary households in the four kebelles. In other words, using this method, a total of 160
households were sampled; of which, 80 are direct participants of the project and 80 are non-
participant households. The detail is stated in table 1 as shown below.
Table 1: Population and sample size of each kebelle
No Name of
Village/Kebelle
Number of Households
Total
CBO
members
Non
members
Sampled
members
Sampled
non members
1 Gorro 2100 380 1720 13 29
2 Gendegara
Gendegola 1600 680 920 23 15
3 Koriso 2250 760 1490 25 25
4 Halole-Iffa 1250 565 685 19 11
Total 7200 2385 4815 80 80
Source: Own compile, 2013
C. Methods of Data Analysis
7
Descriptive and inferential statistics as well as econometric impact evaluation technique was
applied to analyze the quantitative data. The study has employed an econometric model called
Propensity Score Matching (PSM) method in assessing the impact of the program on the
livelihood of the society.
Propensity Score Matching Method
The central aim of the study is to identify the causal effect of participation in the program. To
examine the casual effect of the program, the difference in the community’s net wealth, income
per capita and expenditure per capita, due to participation in the program, was considered. This
was analyzed using the P-score Matching method. This method is preferred to the traditional
regression methods in several ways. Matching involves pairing treatment and comparison units
that are similar in terms of observable characteristics. Matching can yield unbiased estimates of
the treatment impact, (Sadek and Wahba, 2001). Moreover, the PSM method is used to correct
sample selection bias due to the observable difference between the treatment and comparison
group.
This method involves three steps
i) Estimation of the propensity score for each household
ii) Matching the households in common support values using the value of the propensity
score
iii) Test of overlapping and unconfoundedness assumptions
iv) Estimating the average treatment effect on the treated
i. Estimation of P-scores
In technical terms, suppose there are two types of individuals: those that are beneficiaries of
the program (Di =1) and those that do not (Di=0). Individuals with program (treated group) are
matched to those without (comparison group) on the basis of the propensity score. The
propensity score for individual i is defined as:
P (Xi) = P (Di =1| Xi) (0< P (Xi) <1) (1)
8
Where, Xi is a vector of pre-treated explanatory variables such as (age of schooling, age of
respondent, access to alternative credit service, etc.).
Rosenbaum and Rubin (1983) show that if the Di’s are independent over all i, and outcomes
are independent of program participation given Xi (i.e. unobserved differences across the
treated and the comparison groups do not influence being in a specific group), then outcomes
are also independent of program participation given P (Xi), just as they would be if
participation is assigned randomly.
A logit model was used to estimate the p-score using composite pre-intervention characteristics
of the sample households (Rosenbaum and Robin, 1983). In the specification of the Logistic
model, the dependent variable is the probability to participate in the program by the households
whereas the independent variables are of both types that are continuous or categorical. The
cumulative logistic probability model is econometrically specified as follows:
ip = if = i
iif 1
1
(2)
Where: ip is the probability that an individual is a beneficiary of GEF-SGP given i ,
Denotes the base of natural logarithms, which is approximately equal to 2.718;
i represents the thi
explanatory variables; and and i are parameters to be estimated.
ii. Choice of Matching Algorithm
After estimation of the propensity scores, seeking an appropriate matching estimator is the
major task of a program evaluator. There are different matching estimators in theory. However,
the most commonly applied matching estimators are Nearest Neighbor matching (NN), Caliper
matching, and Kernel matching. In this study, we have made comparisons among these
matching estimators to select the best fitted one.
The question here is how and which method to select. Clearly, there is no single answer to this
question. The choice of a given matching estimator depends on the nature of the available
dataset (Bryson et al., 2002). In other words, it should be clear that there is no `winner' for all
situations and that the choice of a matching estimator crucially depends on the situation at
hand. The choice of a specific method depends on the data in question, and in particular on the
9
degree of overlap between the treatment and comparison groups in terms of the propensity
score. When there is substantial overlap in the distribution of the propensity score between the
comparison and treatment groups, most of the matching algorithms will yield similar results
(Dehejia and Wahba, 2002).
The matching quality depends on the ability of the matching procedure to balance the relevant
covariates. Rosenbaum and Rubin (1983) proposed standardized bias that is commonly used
method to quantify the bias between treated and control groups. According to Dehejia and
Wahba (2002), choice of a matching estimator can be tested by different criteria, such as,
balancing test, pseudo-R2 and matched sample size.
Balancing test is a test conducted to know whether there is statistically significant difference in
mean value of per-treatment characteristics of the treatment and the control group of the
households. After matching the treatment and the control group should have similar household
characteristics; to check whether this is the case, two types of balancing test were employed to
check the matching quality. A simple t-test is used to examine whether the mean of each
covariate differs between the treatment and the control group after matching and to supplement
the simple t-test, the Hotel ling’s T-squared test is performed to jointly test the equality of the
mean between the two groups for all covariates and preferred when there is no significant
difference.
Additionally, Sianesi (2004) suggests re-estimating the propensity score on the matched
sample, i.e. only on participants and matched nonparticipants, and comparing the pseudo-R2s
before and after matching. The pseudo-R2 indicates how well the regressors X explain the
participation probability. After matching there should be no systematic differences in the
distribution of covariates between both groups and therefore the pseudo-R2 should be fairly
low. Furthermore, one can also perform a likelihood ratio test on the joint significance of all
regressors in the probit or logit model. The test should not be rejected before, and should be
rejected after matching. Pseudo-R2 before and after matching is used as comparison to ensure
that there is no systematic differences in the distribution of the covariates between both groups,
and the pseudo-R2 should be fairly low after matching (Sianesi 2004).
10
Common support: The common support region is the area which contains the minimum and
maximum propensity scores of treatment and control group households, respectively. It
requires deleting of all observations whose propensity scores is smaller than the minimum and
larger than the maximum of treatment and control, respectively (Caliendo and Kopeinig, 2005).
Only the subset of the comparison group that is comparable to the treatment group should be
used in the analysis. Hence, an important step is to check the overlap and the region of
common support between treatment and comparison group. Observations which lie outside
common support region are discarded from analysis (Caliendo and Kopeinig, 2005). No
matches can be made to estimate the average treatment effects on the ATT parameter when
there is no overlap between the treatment and non-treatment groups.
Failure of Common Support: We have presented possible approaches to implement the
common support restriction. Those individuals that fall outside the region of common support
have to be disregarded. But, deleting such observations yields an estimate that is only
consistent for the subpopulation within the common support. However, information from those
outside the common support could be useful and informative especially if treatment effects are
heterogeneous (Lechner, 2001b).
iii. Testing the Overlap and Unconfoundedness Assumptions
The first method to detect lack of overlap is to plot distributions of covariates by treatment
groups (Imbens and Wooldridge, 2008). A more direct method is to inspect the distribution of
the propensity score in both treatment groups, which can reveal lack of overlap in the
multivariate covariate distributions.
The unconfoundedness assumption implies that beyond the observed covariates, there are no
unobserved characteristics of the individual associated both with the potential outcome and the
treatment (Imbens and Wooldridge, 2008). Although the unconfoundedness assumption is not
directly testable, this study assesses its plausibility by estimating a pseudo causal effect that is
known to be zero.
iv. Average Treatment Effect on the Treated
11
The parameter of interest in the estimation of propensity score is the Average Treatment Effect
on the Treated (ATT) which can be estimated as follow:
Let Di1 be a dummy variable indicating individual i’s beneficial of the program. The outcome
variable, an indicator of economy of an individual i is denoted by iY 1 which depends on
whether or not the individual participated in the program. Then effect of the program
participation in year t on the outcome in t + s is given by;
01 ,, stisti YY (3)
The major difficulty in examining this effect is that 0, stiY is not observable if individual i
participated in the program or if it is in treatment group, while 1, stiY is not observable if
individual i does not participate in the program or if it is in the control group. Therefore,
existing studies on impact evaluation often estimates the average effect of the treatment on the
treated (ATT), defined as:
1,, ,,1/01 tiiitstisti XPDYYEATT
1,,1, ,1/0,,1/1
tiiitstttiitsti XPDYEXPDYE (4)
Where:
Xi denotes pre- program characteristics of individual i in year t-1
P(Xi) is the p-score
Yi1 and Yi
0 are the potential outcomes in the two counterfactual situations of receiving
treatment and no treatment.
D. Variable Definitions and Working Hypothesis
The literature on the impact of a program on poverty reduction makes it clear that the choice of
dependent and independent variables had been identified by the researchers. This section
describes the variables used in the econometric analysis.
Dependent variables
Participation in the program, which is the dependent variable for the logit analysis, is a
dichotomous/dummy/ variable which was represented in the model by 1 for beneficiaries of the
12
program and 0 for non-beneficiaries. There are three outcome variables which were used in the
analysis of Average Treatment Effect on the Treated.
Asset: a number of measures of household worth including refrigerator, housing condition,
TV, etc.
Income: this includes various categories of income including selling of goods and services, etc
Expenditure: this measures the total amount of expenditure on food and non-food items of a
household per month.
Independent variables
The independent (explanatory) variables that were expected to have relationship with
participation in the GEF-SGP were selected based on available literature and scientific
research. The following explanatory variables are hypothesized to explain the dependent
variable. These are: Households’ socio-economic characteristics which include family size; age
of schooling; age; amount saved, marital status; training and experience in credit use were
selected for the logit analysis.
Family size: This is the total number of adult equivalent to represent total family sizes that live
together under the same household. An increase in household size implies more expenditure
from limited resources which intern results in higher tendency of these households to
participate in such programs to finance their increased household expenditure. Hence, it is
expected to have a positive relationship with the probability of the participation in the GEF-
SGP.
Age of schooling: It refers to the educational level of the head of the household and is
measured in years of formal schooling. The hypothesis is based on the assumption that
education has a direct relationship with the utilization of information that could help for
adapting to climate change impacts which makes this variable to be positively related with the
outcome variables.
Marital status: it is a dummy variable which takes value 1 if a household head is married and
0 other wise. Marriage is biological and social engagement to support each other both socially
and economically. It is established with a view of helping each other and married people pool
13
their resources together and reduce cost that would be spent separately. Hence, in this study,
married household heads are assumed to have a positive relationship with the probability of
participating in the program.
Age: It is defined as the period from the respondent birth to the time of the survey and is
measured in years. Empirical evidence show that with increase in age, household’s motivation
to engage in such programs decreases and hence it was hypothesized in the study that age of
household heas has negative relationship with program participation.
Amount saved: This variable is a continuous variable that measures in Birr. According to
Cohen (2001) savings can help households deal with income shocks; it provides a basic
indicator of household security. The more they save the less they tend to be engaged in
environmental protection activities. Therefore, it is expected that this variable would have
negative influence on participation in the program.
Training taken: This is a dummy variable showing whether the individual has taken trainings
related to the program. Hence, it is expected that taking related training will created access for
the individual as well as motivate him/her to participate in the program.
Credit use experience: This is a dummy variable showing that whether the individual has
access to credit service or not. It is expected that the possibility of individual’s participation in
the program increases as the individual has access to credit use; for the reason that he is
expected to have more experience to participate in such things.
III) FINDINGS OF THE STUDY
A. Descriptive Analysis of Impact of the Program
Table 2 presents descriptive statistics results of sample households on outcome variables, such
as, income, asset and expenditure. The survey results show that participant and nonparticipant
households have average monthly income of 1765 and 1391.54 respectively. In terms of
average asset value of the respondents, average asset of participants is 21319.47 while that of
14
nonparticipants is 10171.52; and the estimated average monthly expenditure results indicate
that it is 1494.69 for participants and 1184.2 for nonparticipants.
As indicated in the table, the difference in mean of monthly income, asset and expenditure of
the two groups was found to be significant at 10% , 5% and 10% level of significance,
respectively, as revealed by the t-statistic result. Result of the t-statistics for the variables was
found to be 1.7942, 2.5895, and 1.7710, respectively. This means that households in the
program are better off in all these variables.
Table 2: Descriptive results of outcome variables
Outcome
variable
Program Non-program T-value
Mean SD Mean SD
Income 1765 .69 1608.54 1391.54 932.3 -1.7942*
Asset 21319.47 37406.45 10171.52 8141.66 -2.5895**
Expenditure 1494.69 1328.15 1184.2 825.07 -1.7710*
Source: own computation result, 2013
***, ** and * Significant at 1%, 5% and 10% level of significance respectively
According to the explanation of DDEPA officials, The GEF-SGP is playing a major role in
Poverty Alleviation by creating new business activities which generates alterative income for
poor rural households. It has made a significant positive impact on the economic and social
conditions of the beneficiaries. The project activities have resulted in increased personal
income, creating employment opportunity, and permit personal spending on children’s
education, health, nutrition and improved housing. The program has helped very poor
households meet basic needs and protect against risks. Generally, the use of the program by
low-income households is associated with improvements in household economic welfare. The
program is said to help to smooth their consumption levels and significantly reduce the need to
sell assets to meet basic needs.
B. Econometric model results
15
This section discusses the results of Propensity Score Matching in detail. To measure the
average treatment effect on the treated (ATT) for intended outcome variables, a logit model
was estimated in order to get the propensity scores. Next a matching estimator that best fit to
the data was selected. Then based on those scores estimated and matching estimator selected,
matching between participants and non-participants was done to find out the impact of the
program on the mean values of the outcome variables. Therefore, this section illustrates all the
required algorithms to calculate the average treatment effect on the treated, which helps us to
identify the impact of the program.
i. Propensity scores
Table 3: Logit results of household program participation
Covariate Coefficient St.error Z-value
Age .1906755 .068377 2.79 **
Family size -.7453037 .3388641 -2.20 **
Age of schooling .1626885 .1371936 1.19
Amount saved .0005103 .0008589 0.005
Experience in credit
use
.2798052 .12721 2.20**
Training 0.121953 0.534323 3.99***
Marital status -.9969225 .622401 -1.60*
_cons -8.787451 3.279208 -2.68**
NN 160
LR chi2(9) 184.99
Prob > chi2 0.0000
Pseudo R2 0.8341
Log likelihood -18.394881
Source: own survey result, 2013
***, ** and * Significant at 1%, 5% and 10% level of significance, respectively
Table 3 shows the estimation results of the logit model. The estimated model appears to
perform well for our intended matching exercise. The pseudo-R2 value is 0.8341. Looking into
16
the estimated coefficients (Table3), the results indicate that program participation is
significantly influenced by five explanatory variables. Age, family size, experience in credit
use, training and marital status were found to be significant variables to affect the participation
of the household to the program, at 5%, 5%, 5%, 1%, and 10%, level of significance,
respectively.
As indicated by sign of the coefficients, as age of the household head is getting older, the
possibility of participation in the program is higher; households who have larger family size
are less likely to participate in the program; access to credit increases the possibility of
participation; training in related activities increases the possibility of participation, and marital
status of being married reduces the possibility of participation.
Here, our interest is to estimate predicted values of program participation (propensity score) for
all households in the program and outside the program, before launching the matching. Then, a
common support condition should be imposed on the propensity score distributions of
household with and without the program and finally drop observations whose predicted
propensity scores fall outside the range of the common support region. The data set resulted in
good matches in the case of minima and maxima approach. Therefore, this approach was
employed to identify the common support region.
As shown in Table 4 below, the estimated propensity scores vary between 0.087 and 0.992
(Mean=0.936) for program or treatment households and between 0.00021 and 0.978 (mean =
0.0655) for non-program (control) households. The common support region would then lie
between 0.087 and 0.978. In other words, households whose estimated propensity score is less
than 0.087 and larger than 0.978 are not considered for the matching exercise.
Table 4: Distribution of estimated Propensity Score
Group Obs. Mean S.D Min Max
Treated 80 0.936 0.174 0.087 0.992
Control 80 0.0655 0.164 0.00021 0.978
Total 160 0.506 0.468 0.000 0.992
Source: own estimation data, 2013
17
ii. Matching of participant and non-participant households
Alternative matching estimators were tried in matching the treatment program and control
households in the common support region. The final choice of a matching estimator was
guided by different criteria such as equal mean test referred to as the balancing test (Dehejia
and Wahba, 2002), pseudo-R2 and matched sample size. Specifically, a matching estimator
which balances all explanatory variables (i.e. results in insignificant mean differences between
the two groups), bears a low R2
value and also results in large matched sample size is
preferable.
In line with the above indicators of matching quality, caliper matching with radius 0.5 is
resulted in relatively low pseudo R2, with relatively better balancing test (with insignificant
mean difference of 7 explanatory variables) and large matched sample size, as compared to
other alternative matching estimators (indicated in Table 5). Then it was selected as a best fit
matching estimator for data at hand.
Table 5: performance criteria
Matching criteria Match
sample
size
Balancing
test
Pseudo R2
Before matching After
matching
NN Matching
Without replacement 83 7 0.836 0.342
With replacement 101 4 0.836 0.443
Radius Caliper
0.1 88 4 0.836 0.443
0.25 108 4 0.836 0.374
0.5 108 7 0.836 0.342
Kernel
0.1 106 4 0.836 0.342
0.25 108 4 0.836 0.443
0.5 108 6 0.836 0.496
Source: own estimation data, 2013
Testing the balance of propensity score and covariates
18
After choosing the best performing matching algorithm, the next task is to check the balancing
of propensity score and covariate using different procedures by applying the selected matching
algorithm (in our case Caliper matching). As indicated earlier, the main purpose of the
propensity score estimation is not to obtain a precise prediction of selection into treatment, but
rather to balance the distributions of relevant variables in both groups. The balancing powers of
the estimations are determined by considering different test methods such as the reduction in
the mean standardized bias between the matched and unmatched households, equality of means
using t-test and chi-square test for joint significance for the variables used.
The process of matching thus creates a high degree of covariate balance between the treatment
and control samples that are ready to use in the estimation procedure. t-values in Tables 6 show
that before matching half of chosen variables exhibited statistically significant differences
while after matching all of the covariates are balanced.
Table 6: Propensity score and covariate balance
Variable Mean Bias t-test
Sample Treated Control
Age unmatched 49.704 35.114 61.4 3.88***
Matched 37.345 40.624 -30.6 -1.07
Family size Unmatched 4.4691 4.3165 8.2 0.52
Matched 4.7241 5.1908 -25.0 -0.82
Age of
schooling
unmatched 6.1605 6.3797 -5.0 -0.32
Matched 5.0345 4.829 4.7 0.20
Marital unmatched 1.6296 1.3418 30.7 1.94*
Matched 1.6552 1.9597 -32.5 -1.18
Experience unmatched 3.8765 0.56962 137.1 8.68***
Matched 2.931 4.1981 -52.5 -1.09
Training unmatched 0.8888 0.2532 345.4 11.76***
Matched 0.75862 0.44456 125.6 1.61
Amount
saved
Unmatched 185.1 99.57 16.9 1.07
19
Source: own estimation data, 2013
iii. Testing the Overlap Assumption and Unconfoundedness
As can be seen from the tables 5, the value of pseudo R2 is fairly low after matching, denoting
that the unconfoundedness assumption is plausible. Moreover, the study uses p-score graph to
test the plausibility of the overlap assumption. The following figure shows the distribution of
propensity scores of both treatment and control observations before the common support
condition. The figure showed that there were unmatched observations in both of the treated and
untreated groups before the common support condition is imposed.
Matched 95.825 62.555 6.6 0.04
PesudoR2 0.283
20
.2 .4 .6 .8Propensity Score
Untreated Treated: On support
Treated: Off support
Figure 1: Distribution of propensity scores of treated and untreated households before common
support
However, as can be seen from figure 2, after matching the data using the Radius Caliper
matching with 0.5 Caliper, the common support condition has trimmed out some observations
from the model. The figure shows that there is great tendency of overlapping. Hence we can
conclude that the overlap assumption is plausible for this estimator.
Figure 2: Distribution of propensity scores of treated and untreated households after common
support
21
iv. Treatment effect on the treated
In this section, the research provides evidence as to whether or not the GEF/SGP has brought
significant impact on household’s living condition. The estimation result presented in Table 7
provides a supportive evidence of statistically significant effect of the program on the
livelihood of the households. On average, the program has increased income of the
participating households by 312.18 birr per month, assets by 3710.71; and expenditure by
249.358, at 1%, 5%, and 10%, level of significance.
Table 7: Average Treatment effect on Treated (ATT)
Variable Treated Control Difference S.E T-test
Asset 16593.65 12882.94 3710.71 5984.32 2.05**
Income 1634 1321.8193 312.18 419.96 3.74***
Expenditure 1265.034 1015.676 249.358 37751 1.66*
Source: own estimation data, 2013
C. Challenges of the Project
As acquired from the secondary information, problems faced by the project, specifically by
Community Based Organizations can be classified into four major areas: These include gaps in
project management skills, gaps in logistics management, gaps in administration and
governance, and threats from poor infrastructure.
i. Gaps in project management skills
Project Management is the process that provides a framework for information gathering,
analysis, planning, implementation, monitoring and evaluation of a project. It is a dynamic
process using the appropriate resources of the organization in a controlled and structured
manner, employed to achieve a change clearly defined within specific objectives identified as
strategic needs. Project management is therefore a powerful tool for improving the
22
effectiveness and efficiency of a project by helping an organization to set project goals and
objectives; and to guide implementation, monitoring and evaluation of a project.
Project management provides a framework within which projects are implemented and ensure
that scarce resources are used for project activities that address the defined objectives. It also
helps to establish a link between proposal preparation, review and approval mechanism, and
ensures that the project is completed within defined scope, quality, time and cost limits.
For an organization to implement its programs in an effective and efficient manner as well as
to keep its self sustainability, the staff implementing the project needs to have knowledge and
skills in various elements of project management. However, project staffs of the largest
majority of CBOs say that they lack such skills, a factor that has been an impediment in the
ability of CBOs to mobilize resources and fill the existing gap in their environment and
livelihoods.
ii. Gaps in logistics management
Logistics entails planning on availability and utilization of resources and commodities in order
to achieve the set targets effectively. Logistics involves the integration of information,
transportation, inventory, warehousing, material-handling, and packaging, and occasionally
security. Logistics is a channel of the supply chain which adds the value of time and place
utility”. Community Based Organizations lack the training in logistics management to control
the in-flow and out-flow of resources placed at their disposal, and to produce supportive
documentation to assist their groups to effectively run their projects.
iii. Gaps in administration and governance
Administration and governance issues have been stumbling blocks in the running of
community organizations. The roles of various office holders in the organization are often
muddled and unclear, which ultimately affects the performance of the CBO. Furthermore
communication channels within the organization are often lacking and internal conflicts are
frequent.
Administration and governance were not covered in this survey, but it was clear that, with a
skeleton staff, the CBOs often have one person playing dual roles. For example on the question
23
on who provides financial services in the organization, it was clear that most CBO Leaders/
Chairpersons were also involved in keeping project accounts and reporting on project
activities. Such multiple roles often lead to a lack of transparency, for example in reporting on
project finances.
iv. Threats from poor infrastructure
Rural Dire Dawa is known for its poor infrastructure, for a lack of roads, a lack of appropriate
communication for information and education technologies, a lack of electricity, low levels of
telephone lines and very long distances to be travelled by project coordinators to get to the
project areas. This was the main problem lying under the poor and infrequent monitoring and
evaluation practices done by the project coordinating bodies.
IV) CONCLUDING REMARKS
Dire Dawa Administration is among those regions which are affected by natural catastrophes
such as frequent damaging floods, desertification, and frequent drought, as the result of the
climate change. Consequently, households living in the region, particularly rural households
are expected to be impoverished which has the tendency to exacerbate the environmental
destruction.
In cognizant of this, both the government and concerned non-governmental organizations are
striving to combat the situation through implementation of different projects. The UNDP –
GEF/SGP Project is among these projects working in the region through establishment of
community based organizations. The primary goal of this project is changing the living
standard of particularly the poor societies thereby mitigating environmental problems, through
provision of different services to the society.
This study was undertaken to figure out the socio-economic impact of this project on the
livelihood of the society. The study found out that the program intervention leads to change
that is different from that would have happened without the intervention. The program
increases the probability of improvement in income, asset and expenditure of the clients. The
changes more likely occur with program participation than without program participation. The
24
program has enabled the participants to generate income that could be spent on better facilities,
which could improve their living standard. This implies these households are expected to be
enabled to mitigate or to take adaptation strategies for the existing environmental problems.
However, different challenges were identified which may retard the impact of the project.
These problems include gaps in project management skills, gaps in logistics management, gaps
in administration and governance, and threats from poor infrastructure. It is expected that the
impact of the project would have even been much higher than its current impact had these
problems not exist. With the existence of these problems, self sustainability of the
impact/output of the project by the Community Based Organization is in question, in the
future. Therefore, emphasis should be given to bring about solutions for these problems.
REFERENCES
Bryson , Jennie and Halaby, Charles, 2002. "Propensity Score Matching to Estimate the Effects
of Elite College Attendance on Career Outcomes" Paper presented at the annual meeting of the
American Sociological Association, Atlanta Hilton Hotel, Atlanta, GA,
http://www.allacademic.com/meta/p105891_index.html
Caliendo and Kopeining, 2005. Child Labor Response to Household Participation in Credit
Schemes and Households, Income-Generating Activities in Peru, Master thesis. Washington
D.C.
Dehejia, R., and S. Wahba, 2002. Causal effects in non-experimental studies: re-evaluating the
evaluation of training programs. Journal of the American Statistical Association, 94: 1053–
1062.
Dire Dawa Environmental Protection Authority (DDEPA), 2012. Regional State of the
Environment and Outlook. Supporting the Delivery of MDG. A Preliminary Report.
Dire Dawa Environmental Protection Authority (DDEPA), 2013. Terminal Evaluation Report
of the UNDP/GEF SGP Funded Community-Based Sustainable Environmental Management
25
Program in Dire Dawa.
Imbens, G. and J. Wooldridge, 2008. Recent developments in the econometrics of program
evaluation. The Institute for Fiscal Studies Department of Economics, UCL, cemmap working
paper CWP24/08
Lechne, G.D., J.S, 2001b. Attitudes and Farmer Conservation Behavior. American Journal of
Agricultural Economics 70:12-19
Rosembaum, P.R. and Robin, D.B. 1983. The Central Role of the Propensity Score in
Observational Studies for Causal effects, Biometrika, Vol.70, No.1, pp. 41–55.
Sadek, M. and and W. Nagarjan, 2007. Predicting Creditworthiness: Evidence from ASCRAs
and ROSCAs in the Gambia. Saving and Development, Quarterly review. 22(4): 399-413.
Sianesi, K., 2004. Microfinance and Women’s Economic Empowerment: Bridging the Gap,
Redesigning the Future, Working Paper Series No. 1, WCCN (Wisconsin Coordinating
Council on Nicaragua).