Post on 28-Aug-2018
Impact Evaluation Design Kenya
Country Team 1. Anne Amadi 2. Justice Daniel Musinga 3. Dr. Paul Kimalu 4. Kennedy Mugonyi
ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015
: #ieGovern
What is the problem?
What is the problem? - Case backlog & case delays Intervention(s) - Regular feedback on performance (case backlog and
clearance rates to the courts). - Emails, phone calls, heads-of-court stations quarterly meetings, visits
by Judiciary officials. - *Considering tailoring printed feedback with influential images or
message (such as sending report with picture of the Chief Justice).
- Enhanced Court User Committees (CUCs) engagement - Greater stakeholders engagement – (Representatives from police,
prisons, advocates, CSOs, probation dpt etc)
1
Research questions • What is the most effective way of reducing case backlog and
improving case clearance rates? • Discuss mechanisms (e.g. multiple arms) - Monitoring and feedback without rewards or sanctions - Court stations stakeholders engagement. • Briefly mention the Theory of Change: Program -> A -> B -> C -
> X -Feedback Appreciation, recognition, managerial assistance, career progression/promotion Reduction in backlog -Stakeholders engagement Improved coordination, peer pressure Reduction in backlog.
Research Questions
2
Method Randomized Control Trial (RCT)
Clusters / sample size How many clusters? 160 courts How big is the sample size? 160 courts X 12 months = 1920 court months.
Design Cross-cutting design.
Methods & Design
3
Feedback
Stakeholders consultations
(CUC) 40 40
40 40
Outcomes:
- Speed -Case clearance rate.
-Disposition time.
-Case turnover ratio.
- Quality -Court users and employee satisfaction survey.
Unit(s) of observation: - Courts
Outcomes & Measurement
4
#ieGovern Impact Evaluation Workshop
Istanbul, Turkey January 27-30, 2015
Thank You!
facebook.com/ieKnow
#impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/impact_evaluation
http://dime.worldbank.org WEB
Post Crisis Needs Assessment Impact Evaluation FATA & KPK
ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015
: #ieGovern
Background • Post Crisis Needs Assessment Strategic Objectives, mainly
Citizen – State Trust
• Impact Evaluation & capacity building of country systems
• Rule of law (Judicial)
• Governance
• Infrastructure
• Health
• CDD
Evaluation Objectives
• To measure impact of key PCNA interventions with credible data and correctly specified indicators
• To establish credible evidence and inputs for governance and service delivery reforms for further programming decisions
• To institutionalize evaluation in country systems for evidence based decision making and development planning
Projects Evaluation
Type Evaluation Method Sample Size (with Over
Sample)
Off-Season Vegetable - Agriculture Ex-Ante Randomize Control Trials RCT
430 Farmers (Adopters & Non Adopters) were interviewed in survey against 1381 Farmers in Khyber & Mohmand Agency which is 30% of the total farmers.
Anti-corruption Hotline - Governance
Ex-Ante Randomize Control Trials RCT
16,000 Individuals
TB Control Program FATA - Health Ex-Ante Randomize Control Trials RCT
124 Villages with 871 Respondents
MNCHP - Health Ex-Post Randomize Control Trials RCT
22 CMWs , 22 LHWs
Right to Information - Governance Ex-Ante Randomize Control Trials RCT
240,000 Individuals
e-Citizen Grievance - Justice Ex-Ante Randomize Control Trials RCT
220,000 Individuals
Right to Access to Public Services - Governance
Ex-Ante Randomize Control Trials RCT
215,000 Individuals
Review Request
• Define Citizen Trust Restoration
• Quality of service satisfaction (17 measures)
• Confidence in institutions (9 measures)
• Social satisfaction – self/family/other (17 measures)
• Use of technology mobile/internet for transparency and increase in trust in the government (43 measures)
What is the problem? • The effectiveness and utilization of the formal
judicial system • Back log of cases • Parallel judicial systems
• Build capacity of judicial system through trainings • Exploit variations in the timing of training (both experimental and
non-experimental)
• Exploit variations in where trained judicial officials are located
• Exploit variation in the type of training received
• Information about the availability of improved alternatives to formal court cases
Research Questions
• Who are the key actors underlying an effective Judicial System?
• What are barriers to citizen utilization of state supported Judicial System?
• Poorly trained Judicial staff leads to bad decisions and that leads to citizens distrust and under utilization
• On the supply side the judicial staff is trained
• On the demand side use awareness campaigns • Improved efficiency • Low cost mediation
Methods & Design • Triple Difference for the impact of judicial trainings
• Across judges
• Across time
• Across judicial staff
• RCT on the awareness campaign
• 350 judges across 27 districts
• Universe of cases that are brought forward and cases that are brought forward for mediation
Outcomes &
Measurement • Case specific outcomes
• Mediation vs formal court litigation • Time to completion • Number of hearings • Third party evaluation of decisions (random order)
• Litigant-specific outcomes • Satisfaction with decision • Cost incurred • Willingness to bring cases forward in the future
• Citizen outcomes • Willingness to access state institutions • Most recent form of dispute resolution • Trust in the state
• Court level outcomes • Backlog of cases • Case load per Judge
#ieGovern Impact Evaluation Workshop
Istanbul, Turkey January 27-30, 2015
Thank You!
facebook.com/ieKnow
#impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/impact_evaluation
http://dime.worldbank.org WEB
Judicial Development Project KYRGYZ REPUBLIC
ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015
: #ieGovern
What is the problem?
What is the problem? - Problems of access and efficiency deter (M)SMEs from using
courts to resolve their disputes and enforce their rights. Interventions - Arm 1: SMEs receive training on laws/procedures that commonly
affect SMEs (staggered) - Arm 2: SMEs receive court infrastructure (staggered) - Arm 3: SMEs receive infrastructure & training
1
Research questions • How do improvements in court infrastructure impact access &
perceived efficiency of justice for (M)SMEs? • How does targeted training impact (M)SMEs? • Which of the interventions provides the greater impact on (M)SMEs? Theory of change: • Demand-side interventions that raise awareness of law/procedure
among potential court users are more likely to increase capacity to resolve legal disputes than supply-side courthouse infrastructure.
Research Questions
2
Method - RCT built into quasi-natural experiment (courthouse)
Sample size - Baseline survey of 500 (M)SMEs + repeat surveys - Control for revenue, no. employees, distance to courthouse, gender
and ethnicity of manager/owner - Caseload data on timeliness, project data, case tracking
Spillovers: - Information spillovers - Understanding of preferences/behaviors of (M)SMEs on all topics - Increase in formality of (M)SMEs
Methods & Design
3
Outcomes:
- Project level: inform Kyrgyz Republic Phase 2 project
- World Bank level: inform design of future projects
- National/international level: inform decision-making by Kyrgyz (and other governments) on how to improve justice / business climate
Units of observation:
- Firm level
Outcomes & Measurement
4