Deepak T. Bhatia Measuring the impact of Egovernment Deepak T. Bhatia Sectoral and E-government...
-
Upload
kenneth-anthony -
Category
Documents
-
view
215 -
download
2
Transcript of Deepak T. Bhatia Measuring the impact of Egovernment Deepak T. Bhatia Sectoral and E-government...
Deepak T. Bhatia
Measuring the impact of Egovernment
Deepak T. BhatiaSectoral and E-government Applications Practice Lead
Global ICT department (GICT)The World Bank
Deepak T. Bhatia
It is no longer acceptable for technology to be utilized simply for technology’s sake. Individual e-government projects need to demonstrate their contribution to overall
government objectives – with the focus being on results and impact.
Deepak T. Bhatia 3
Why e-Government?
•Battling bureaucracy
•Improving Efficiency
•Better service delivery
Strengthening governance and accountability
Deepak T. Bhatia 4
About 150 contain ICT components (out of 250 new WB projects each year).
0% 1% 2%
8%11%
22%
40%
47%
55%
61%57%
51%
0%
10%
20%
30%
40%
50%
60%
70%
Bank projects w ith ICTcomponents (%)
0% 1% 2% 8% 11% 22% 40% 47% 55% 61% 51% 57%
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
In the World Bank the Number of Projects with ICT is growing
57% of active projects have ICT components (this is 930 projects)
ICT components’ combined volume is $6.2 billion
Deepak T. Bhatia 5
Analysis of e-Government Impact
• World Bank commissioned study to explore impact of e-government on citizens, in collaboration with the Indian Institute of Management, Ahmedabad, India and the London School of Economics
• Effort to go beyond anecdotal assessments commonly available
• Sample of 5 mature, wide scope/scale projects selected for analysis from India
– Chosen to represent different stages of overall IT readiness and evolution
Deepak T. Bhatia
Learning from Past Assessments
• Variety of approaches have been used for assessments -client satisfaction surveys, expert opinion
• Often studies have been done by agencies that may be seen as being interested in showing positive outcome
• Lack of credibility of results-different studies of the same project show very different outcomes
• Lack of rigor in sampling-results can not be easily generalized
• Lack of rigor in controlling for external influence-need for counterfactuals ignored.
• Lack of a standard methodology-making it difficult to compare projects
Deepak T. Bhatia 7
Dimensions of measurement - Stakeholders
• Impact on Clients:• All measurements on the basis of a sample of clients for each major service
availed by the client. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service. In cases where alternate (non electronic modes) are currently being used by the same set of users in similar contexts elsewhere, measurement would be recorded for such usage
• Impact on Agencies (Including Partners in Implementation):
• All measurements on the basis of data collected from agency records, a sample of employees or a group of managers. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service
• Impact on Society:
• Some of the questions to be framed as degree of improvements resulting from the implementation of a specific project
Deepak T. Bhatia
Measurement Framework - Dimensions
Stakeholders Key Dimension of Impact
Client Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation)Quality of Service (Decency, Fairness, Convenience, etc.)Over all satisfaction
Agency
(Including Partners in Implementation)
Economic (Direct & Indirect)Governance (Corruption, Accountability, Transparency, Participation) Performance on Key Non-economic Objectives Process ImprovementsWork life of employees
Society
Other Departments
Government as a Whole
Civil Society
Desirability of investments in e-Government
Impact on vulnerable groups
Image of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness)
Impact on development Goals
Deepak T. Bhatia
Proposed Framework
• Focuses on retrospective assessment of e-delivery systems(B2C and B2B)
• Balanced approach between case study and quantitative analysis
• Recognizes that some part of the value to different stakeholders can not be monetized
• Understand how inputs lead to outputs and outcomes in different project contexts
• A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data
Deepak T. Bhatia
Methodology for Assessment
• Select mature implementations , of e-delivery of services.
• Collect data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system)
• Customize survey instrument to each project, adapt in local language
• Data can be collected through Internet survey, face to face interviews and focus groups
• Determine sample frame and size so that results can be extrapolated to the entire population (often 300 clients may be sufficient).
• Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies
Deepak T. Bhatia 11
Projects assessed
• Bhoomi project. 2001. delivering two online services: issuance of records of rights, tenancy, and crop inspection register (RTC) and filing of requests for mutation (changes in land ownership) for affecting changes in land records.
• Karnataka Valuation and E-Registration (KAVERI) project. 2003 delivering three key services: registration of property sale and purchase deeds, issuance of nonencumbrance certificates, and issuance of copies of previously registered deeds.
• Computer-aided Administration of Registration Department (CARD). 1998. same three online services provided by KAVERI.
• eSeva. 2002, These one-stop service centers now deliver 135 services from central, state, and local governments and public utilities, and are used by 3.1 million citizens a month
• Ahmedabad Municipal Corporation (AMC) Civic Centers project. 2002, deliver three important services: annual payment of property taxes, issuance of birth and death certificates, and issuance of shop licenses.
Deepak T. Bhatia 12
Caveat
• largely relies on the survey of stakeholder perceptions of public services before and after these projects.
• Thus not an impact evaluation of investments in narrowly defined ICT, which typically includes hardware, software, and communication equipment.
• Rather, a review of the outcomes of packaged efforts involved in the transition from manual services to those using ICT as a key component.
Deepak T. Bhatia 13
Impact: Computerized service delivery is preferable to manual
Preference for Computerization (%)
0
20
40
60
80
100
Bhoomi KAVERI CARD e-Seva AMC
Preference for Computerization (%)
Deepak T. Bhatia 14
-100
-80
-60
-40
-20
0
20
40
60
80
100
120
Bhoomi
RTC
Bhoomi
Mutation
KAVERI CARD eSeva AMC
User reports of changes in travel costs, by project (number of trips * distance travelled)
Rupees per service
• Bhoomi costs went up due to change in location of service centers from villages to “taluka” headquarters
Deepak T. Bhatia 15
User reports of reductions in waiting times, by projectMinutes
• decreased in all five projects.
• With manual systems, users often had to wait in long lines to receive services. Automation has increasedthe efficiency of document processing.
0
10
20
30
40
50
60
70
80
90
100
Bhoomi
RTC
Bhoomi
Mutation
KAVERI CARD eSeva AMC
Deepak T. Bhatia 16
0.0 0.2 0.4 0.6 0.8 1.0
Bhoomi-RTC
Bhoomi-Mutation
KAVERI
CARD
eSeva
AMC
User perception of improvement in service quality
• Rating on elements such as service delivery location, accessibility, convenience, cost, transparency, and service orientation
Deepak T. Bhatia 17
Project Features
Bhoomi Error-free transactions Reduced delays in transactions
Shorter waiting times Fewer visits
KAVERI Less corruption Increased transparency Error-free transactions Shorter waiting times
CARD Increased convenience Shorter waiting times Less corruption Fair treatment
eSeva Increased convenience Shorter waiting times More convenient time schedules
Fair treatment
AMC Increased convenience Less corruption Greater transparency Good complaint handling system
Users’ top four desired features of services, by project
Goals Examples of performance indicators
Key features of the enabling environment
Outcomes Increased efficiency Increased transparency and
accountability Higher-quality public
services Better access to services
Financial and time savings in government activities
Public perceptions, such as user satisfaction and score cards
Financial and time savings for citizens
Increased public service timeliness and responsiveness
Reduced errors Financial saving per
transaction
Overall e-government strategies
Political and popular support for cross agency coordination and public sector reform
Telecommunications infrastructure and cost structures for increasing ICT access
Supportive legal and regulatory frameworks
Balances with competing priorities (such as roads and education)
Macroeconomic changes
Outputs Reengineered processes New ICT systems Increased service coverage
Comparisons of old and new business processes
Technical reviews of IT infrastructure, applications, and performance
Variety of available services IT support capacity Service training
Proposed conceptual framework
Deepak T. Bhatia 19
Additional conclusions
• Out of the three projects where significant corruption was reported in the manual system, one project was able to eliminate corruption through computerization
• Incremental operating costs for agencies per transaction are less than the reduction in the direct cost of accessing the service reported by the clients.
• Negative aspect is the observed variability of impacts across different delivery centers in a project
• Agencies: ability to cope with growth in transactions was enhanced
• Citizens strongly supported the idea that more agencies need to be computerized, but not necessarily through government investments
Deepak T. Bhatia 20
Conclusions
• Undertake ex-post evaluation of successful ongoing projects.
• Can be used for for ex-ante evaluation, with following assumptions
– extent and adoption rate of user demand need to be estimated– Project designs and investment levels must be commensurate with
anticipated benefits to users. – quality of the enabling environment should be assessed to gauge the
expected impacts of planned projects – Detailed surveys at the initial stages of project planning, would help
establish a baseline.
Deepak T. Bhatia 21
Lessons Learned
• IT cannot be a goal in itself - projects must have a direct link to process reform
• E-Government is a long term endeavor: not a silver bullet, not for immediate gains
• ‘E’ stands for ‘Everything-but-IT’: institutional change, political support, capacity development.. technology is the easy part
• Ultimately can generate significant cost and efficiency gainsBut requires…
Deepak T. Bhatia 22
Focusing on results
Deepak T. Bhatia 23
Credits
• Impact Study team
• Subhash Bhatnagar , IIM, Ahmedabad, India (Primary Author)
• Jiro Tominaga, WB
• Deepak Bhatia, WB
• Shirin Madon, London School of Economics
• Presentation research
• Asheeta Bhavnani, WB
• Mikhail Bunchuk, WB
• Reviewers
• David L. Cieslikowski• Christine Zhen-Wei Qiang
Deepak T. Bhatia 24
Annex
Deepak T. Bhatia 25
How have governments been measuring results of e-government investments?
• France – Mareva - A Method of Analysis and Value Enhancement
• Germany -WiBe - Economic Efficiency Assessment Methodology
• USA – Performance Reference Model Framework
• Australia: Demand and Value Assessment Methodology
• All these government methodologies broadly focus on the same dimensions.
• They offer two levels of impact assessment:
– first, in terms of how projects provide a business case justification for expenditure and whether they meet the targets set for them,
– and second, in terms of how projects meet the goals of the agencies concerned and, in turn, how that helps achieve wider government strategies.
Deepak T. Bhatia 26
World Bank and Impact measurement experience
• What is the WBs experience with measuring impact?
• Independent evaluation group - IEG measures impact of Bank programs and projects, and draws lessons from experience. The Bank also has a large program of impact evaluation although there may not be any that have centered around the ICT sector.
• Whether WB prescribes ONE methodology to all countries? How much tailoring is needed?
• IEG generally uses objective based methodology to measure impact. But if this question refers to tailoring strategies then the key finding from IEG work is that strategies including ICT must be tailored to country circumstances.