Post on 28-Dec-2015
Impact Assessment Monitoring & Evaluation for e-Government
ProjectsSubhash Bhatnagar
As part of the Capacity Building Workshop under the Joint Economic Research Program (JERP)
This session will focus on the need for assessing impact of e-government projects and will described a methodology of how such an assessment can be carried out. Results of an impact assessment study sponsored b the World Bank will be discussed in detail illustrating the methodology and the value that could be derived from such assessments. Some of the pitfalls that should be avoided in making assessment will be described.
Presentation Structure
• Why assess impact? • Learning from past work on assessment• Proposed Methodology• Results from a Bank Study of 8 projects
– Study objectives– Projects covered in the study– Analysis of Results
• Are investments in eGovernment worthwhile?• Lessons for assessment work
Why Impact Assessment?
• To ensure that funds deployed in eGovernment provide commensurate value.
• To create a bench mark for future projects to target• To identify successful projects for replication and
scaling up• To sharpen goals and targeted benefits for each
project under implementation• To make mid course correction for projects under
implementation• To learn key determinants of economic,
organizational, and social impact from successful and failed projects
Evaluation of Impact: Key Issues
• Macro versus Micro Approach- unit of analysis• Assessment from whose perspective?• Dimensions on which impact can be assessed for
different stakeholders• Can all costs and benefits be monetized?• How to isolate the effect of ICT use from
different interventions ?• Degree of quantification versus qualitative
assessment• Measurement issues: sampling, questionnaire
design, analysis of internal data, triangulation
Learning from Past Assessments
• Variety of approaches have been used-client satisfaction surveys, expert opinion, ethnographic studies
• Client satisfaction survey results can vary over time as bench mark changes - need for counterfactuals
• Often studies have been done by agencies that may be seen as being interested in showing positive outcome
• Lack of credibility of results-different studies of the same project show very different outcomes
• Lack of rigor in sampling-results can not be easily generalized• Lack of rigor in controlling for external influence-need for
counterfactuals ignored.• Lack of a standard methodology-making it difficult to compare
projects• Hardly any projects do a benchmark survey
Critique of Existing Frameworks• Biased towards quantification of short term direct
cost savings- quality of service, governance and wider impacts on society not studied.
• Conceptual in nature-hardly any frameworks have been applied to assess impact of real projects
• Variety in delivery models has not been recognized. Impact is a function of the delivery model and the nature of clients being served
• Practical issues of paucity of data have not been taken into account-particularly in a developing country context where baseline surveys are not done and M&E systems are weak
Measurement Framework
Stakeholders Key Dimension of Impact
Client Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation)Quality of Service (Decency, Fairness, Convenience, etc.)Over all satisfaction
Agency(Including Partners in Implementation)
Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation) Performance on Key Non-economic Objectives Process ImprovementsWork life of employees
SocietyOther DepartmentsGovernment as a WholeCivil Society
Desirability of investments in e-GovernmentImpact on vulnerable groupsImage of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness)Impact on development Goals
Proposed Framework
• Focuses on retrospective assessment of e-delivery systems(B2C and B2B)
• Balanced approach between case study and quantitative analysis
• Recognizes that some part of the value to different stakeholders can not be monetized
• Understand how inputs lead to outputs and outcomes in different project contexts
• A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data
Methodology for Assessment
• Select mature, wide scope and scale projects of e-delivery of services.• Collect data through structured survey from clients, employees,
supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system)
• Customize survey instrument to each project, adapt in local language• Data can be collected through Internet survey, face to face interviews and
focus groups• Use professional market research agencies with trained investigators for
face to face int• Determine sample frame and size so that results can be extrapolated to the
entire population (often 300 clients may be sufficient). Select respondents randomly from locations stratified by activity levels and remoteness
• Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies.
• Develop a case study-organizational context, process reform, change management.
A study sponsored by World Bank Done by Indıan Instıtute of Management Ahmedabad
andLondon School of Economıcs
Preliminary Resultsfrom Projects in India
Study Team
• Study Coordinator: Subhash Bhatnagar• Indian Institute of Management, Ahmedabad
(IIMA)– Subhash Bhatnagar, Rama Rao, Nupur Singh,
Ranjan Vaidya, Mousumi Mandal
• London School of Economics– Shirin Madon, Matthew Smith
• ISG e-Gov Practice Group– Deepak Bhatia, Jiro Tominaga
• Sponsors– World Bank,IIMA, Department of IT
Projects of e-delivery of Services
• Issue of land titles in Karnataka (Bhoomi): 180 Kiosks, Launched February 2001 (2-01)
• Property registration in Karnataka (Kaveri): 230 offices (3-03)
• Computerized Treasury (Khajane): 240 locations (11-02)• Property Registration in Andhra Pradesh: AP 400 offices.
(11-98)• eSeva center in Andhra Pradesh: 250 locations in 190
towns, Used monthly by 3.5 million citizens (8-01) • e-Procurement in Andhra Pradesh (1-03)• Ahmedabad Municipal Corporation (AMC): 16 Civic
Service Centers (9-02)• Inter State Check Posts in Gujarat: 10 locations (3-2000)• e-Procurement in Chile (Comprasnet)• Income Tax on-line in Chile
Dimensions to be Studied to Evaluate Impact• Project context: basic information on the project and its context• Inputs (technology, human capital, financial resources);• Process outcome (reengineered processes, shortened cycle time,
improved access to data and analysis, flexibility in reports);• Customer results (service coverage, timeliness and
responsiveness, service quality and convenience of access);• Agency outcomes (transparency and accountability, less
corruption, administrative efficiency, revenue growth and cost reduction) and
• Strategic outcomes (economic growth, poverty reduction and achievement of MDGs).
• Organizational processes: institutional arrangements, organizational structure, and other reform initiatives of the Government that might have influenced the outcome for the ICT project.
Profile of Respondents Bhoomi Card e-Seva AMC Kaveri Number of respondents 242 232 253 239 237 Nature of clients Farmer Property
owner Urban dweller
Urban dweller
Property owner
Percentage Break up: Education Illiterate 36.78 9.91 3.95 2.09 27.00 Schooled 52.07 58.62 57.71 63.18 55.70 Graduate 11.16 31.47 38.34 34.73 17.30 Profession Workers 90.91 38.79 33.99 17.15 69.20 Business 4.13 34.91 27.27 45.19 12.24 White collar 1.65 10.34 22.53 17.15 6.75 Supervisor 3.31 15.95 16.21 20.50 11.81 Average Income <5000 92.15 43.10 40.71 22.18 71.73 5000-10000 5.79 44.40 42.69 60.25 19.83 >10000 2.07 12.50 16.60 17.57 8.44 Urban / Rural Urban 28.93 70.26 100.00 100.0 32.49 Rural 71.07 29.74 0.00 0.00 67.51
Improvement Over Manual System
AMC CARD Check Post
E-Proc E-Seva
Kaveri
Bhoomi
Total Travel Cost per transactıon (Rs.)
21.07 67.71 3430.60
7.40 89.22 0.15
Number of trips 0.65 1.38 5.16 0.28 1.18 0.47
Wage Loss (Rs.) 36.84 28.46 15.63 120.55
(39.22)
Waiting Time (Minutes)
14.69 97.00 8.94 114.95 18.50 61.81 33.97
Governance Quality - 5 point scale
1.08 1.01 0.25
Percentage paying bribes
2.51 4.31 6.25 11.77 0.40 12.71 18.83
Service Quality- 5 point scale
0.52 0.40 0.58 0.24 0.76 0.27 0.85
Error Rate 0.42 0.86 1.58 3.80 0.03
Preference for Computerization (%)
97.49 96.98 91.25 83.71 96.84 98.31 79.34
Savings in Cost to CustomersEstimates for entire client population
Projects Million Transactio
ns
Travel Cost
Saving (Rs.
Million)
Wage Loss(Rs.
Million)
Waiting Time (Hours
)
Bribes (Rs.
Million)
Additional
Revenue (Rs.
Million)
Bhoomi RTC-2.6645
MUT-0.1777
(73.96)
(0.086)
66.40
KAVERI 1.0277 91.69 123.89 1.059 (49.40)
CARD 1.0295 69.71 29.30 1.665 (95.99)
e-Seva 37.20 275.45 581.40 11.468 NA
e-Procurement
.0264 90.73 0.0507 3.38
AMC 0.6171 13.43 22.70 0.151 0.15
Check Post 16.4075 2.444 270.37
1613.00
Projects: Descending Order Of Improvement in Composite Scoreson a 5 point scaleProject Manual Computer Differenc
e Average
Average
S.D. Average S.D.
BHOOMI 2.86 0.86 4.46 0.51 1.60
e-SEVA 3.39 0.65 4.66 0.39 1.27
e-PROCUREMENT 3.22 0.58 4.26 0.58 1.03
CHECK POST 3.48 0.79 4.32 0.59 0.84
AMC 3.37 0.61 4.12 0.90 0.75
KAVERI 3.35 0.86 3.90 0.74 0.55
CARD 3.78 0.49 3.93 0.38 0.15
Descending Order Of Post Computerization Composite Scoreon a 5 point scaleProject Manual Computer Differen
ce AverageAverage S.D. Average S.D.
e-Seva 3.39 0.65 4.66 0.39 1.27
Bhoomi 2.86 0.86 4.46 0.51 1.60
Check Post 3.48 0.79 4.32 0.59 0.84
e-Procurement 3.22 0.58 4.26 0.58 1.03
AMC 3.37 0.61 4.12 0.90 0.75
CARD 3.78 0.49 3.93 0.38 0.15
KAVERI 3.35 0.86 3.90 0.74 0.55
Client Perception (Rating on 5 Point Scale in AMC)
Difference
weighted Scores
ComputerizedFrequency Distribution
ManualFrequency Distribution
High Low High Low
Less costs 0.98 78.2 10.5 11.3 36.4 36.8 26.8
Good Waiting Facilities
0.97 79.5 13.0 7.5 41.4 31.8 26.8
Time and effort 0.90 85.8 3.8 10.5 46.4 37.7 15.9
Complaint Handling
0.77 75.7 13.0 11.3 43.1 32.6 24.3
Greater Transparency
0.68 75.7 16.7 7.5 46.0 36.0 18.0
Fair Treatment 0.65 77.8 15.1 7.1 51.0 33.5 15.5
No need for Agents
0.52 71.5 23.4 5.0 57.3 25.9 16.7
Equal Opportunity 0.47 74.5 15.5 10.0 55.2 29.3 15.5
Top Four Attributes Desired in the Application
AMC Less time and effort required
Less corruption
Greater transparency
Good complaint handling system
CARD Less time and effort required
Less waiting time
Less corruption
Fair Treatment
e-Procurement
Less corruption
Easy access Equal opportunity to all
Transparent system of tender valuation
e-Seva Less time and effort required
Less waiting time
Convenient time schedule
Fair Treatment
Check Post
No Delay in transactions
Error FreePayment receipts
Error-free transactions
Fair Treatment
Bhoomi Error free transaction
No delay in transaction
Less time and effort required
Less waiting time
KAVERI Less Corruption
Greater transparency
Error free transaction
Less waiting time
Impact on Agency AMC Civic
Center
CARD
e-Seva
Bhoomi
KAVERI Check
post
eProcureme
nt
Khajane
Total Project Investment (Rs. million)
250.00 300.00
537.00
330.00 185.00
3.5050.4
338.00
Operating Expenses 168.9 52.7 24.3 64.9
Annual Transactions (million)
0.71 1.03 37.20 2.84 2.47 16.73
0.03 15.69
Clients Served (million) 0.29 0.33 1.89 1.67 1.33 6.12 0.00
Tax Revenue in 2005-06 for Computerized (Rs. million)
1974.2 17282
19245 3109.4
Tax Revenue in Last Year of Manual (Rs. million)
42.05
9033 702.68
Growth Rate in Tax Revenue for Computerized
31.95 50.17
17.00 15.10
Transaction Fees in 2005-06 Computerized (Rs. million)
53.32 1130.8
203.59
274.19 2626.9 0
Transaction Fees in Last Year of Manual (Rs. million)
2.53 1890.46
Growth Rate in Transaction Fees for Computerized
12.86 50.90
83.51 22.76 16.71 113.06
8.06
Agency: Growth of Tax and Transaction Fee
Tax Revenue, Transaction Fee
0
5000
10000
15000
20000
25000
1996-97 1997-98 1998-99 1999-00 2000-01 2001-02 2002-03 2003-04 2004-05 2005-06
Year
Am
ount
( (
Rs.
milli
on))
AMC Tax Revenue CARD Tax Revenue KAVERI Tax Revenue
AMC Transaction Fee CARD Transaction Fee e-Seva Transaction Fee
Bhoomi Transaction Fee KAVERI Transaction Fee Checkpost Transaction Fee
Economic Viability of ProjectsAgency Perspective
Yearly Operating Expense per Transaction
Investment per Cumulative
Transactions for 4 years
AMC Civic Center
109.42
CARD 95.94
e-Seva 4.56 6.57
Bhoomi 18.54 9.48
KAVERI
Checkpost 2.76
eProcurement 918.85 43.70
Khajane 4.14 5.48
Attitude to e-Government
The Knowledge Society Mean SDE gov makes an impact on the knowledge of society 4.03 0.73E gov makes an impact on the literacy level of society 3.97 0.82Improvement in GovernanceEnhances citizens convinence in availing govet services 4.14 0.73Reduces corruption in delivery of public services 3.98 0.89Increases accountability & transprency of govet 4.03 0.79E gov has helped to improve the image of govt 4.12 0.80More investment in e-GovernanceGovernment should make more investment on E gov 4.12 0.82More govt department/ public agencies should be computerised 4.13 0.83Computerisation of govt deprtments is a waste of resources 3.85 0.96Investments in Development Schemes vesus eGovernmentMoney spent in e gov should be used for other govt activities 2.96 1.17Building schools, roads, dispensaries is more useful than E gov projects 3.22 1.14Digital InclusionE gov services puts the poor at disadvantage 3.73 0.92E gov services benefits only the rich and influential 3.74 0.98E gov services benefits only the urban people 3.50 1.01Rural citizens benefit greatly from e-Government services 3.30 1.20Rural and urban poverty level have changed 3.22 1.08
Preliminary Observations• Overall Impact
– Significant positive impact on cost of accessing service– Variability across different service centers of a project – Strong endorsement of e-Government but indirect preference for private participation
• Reduced corruption-outcome is mixed and can be fragile– Any type of system break down leads to corruption– Agents play a key role in promoting corruption– Private operators also exhibit rent seeking behavior given an opportunity– Systematizing queues by appointments helps prevent break down
• Small improvements in efficiency can trigger major positive change in perception about quality of governance.
• Challenges– No established reporting standards for public agencies- In case of treasuries, the AG office
has more information on outcome.– What is the bench mark for evaluation-improvement over manual system, rating of
computerized system (moving target), or potential?– Measuring what we purport to measure: design of questions, training, pre test, field checks,
triangulation– Public agencies are wary of evaluation-difficult to gather data
Questionnaire Design and Survey
• Design of analytical reports prior to survey. Often key variables can be missed if the nature of analysis in not thought through prior to the study.
• Pre code as many items in the questionnaire as possible.• Consistent coding for scales -representing high versus low or positive versus
negative perceptions.• Differently worded questions to measure some key items/ perceptions.• Wording of questions should be appropriate to skill level of interviewer and
educational level of respondent.• Local level translation using colloquial terms.• Feedback from pre-testing of questionnaire should be discussed between study
team and investigators. The feedback may include: the length of questionnaire, interpretation of each question and degree of difficulty in collecting sensitive data.
• Quality of supervision by MR agency is often much worse than specified in the proposal. Assessing the quality of investigators is a good idea.
• Involvement of study team during the training of investigators.• Physical supervision by study team of the survey process is a good idea, even if it
is done selectively
Establishing Data Validity
• Check extreme values in data files for each item and unacceptable values for coded items.
• Cross check the data recorded for extreme values in the questionnaire.
• Check for abnormally high values of Standard Deviation.• Even though a code is provided for missing values, there can
be confusion in missing values and a legitimate value of zero.• Look for logical connections between variables such as travel
mode and travel time; bribe paid and corruption.• Poor data quality can often be traced to specific investigators
or locations.• Random check for data entry problems by comparing data
from questionnaires with print out of data files.• Complete data validity checks before embarking on analysis