Evaluating global open government data: methods and status

Evaluating global open government data: methods and status

Lei Zheng Lab for Digital and Mobile Governance

Fudan University China

[email protected]

Wai-Min Kwok United Nations Department of Economic and Social Affairs

United States [email protected]

Vincenzo Aquaro United Nations Department of Economic and Social Affairs

United States [email protected]

Xinyu Qi Lab for Digital and Mobile Governance


[email protected]

Wenzeng Lyu Lab for Digital and Mobile Governance


[email protected]

ABSTRACT1 This paper first examines seven representative open government data evaluation practices by systematically comparing and analyzing their frameworks, indices and methods. Based on this analysis, the paper then builds up a framework to evaluate the open government data performances of all Member States of the United Nations. The paper finds that most current evaluation programs focus on Data and Foundation and give less attention to Platform, Use and Impact. The paper finds that in 2018, 34 countries (18%) scored “Very High”, 40 countries (21%) scored “High”, 43 countries (22%) scored “Middle”, while 76 countries (39%) scored “Low”. Recommendations are provided based on the findings.

CCS CONCEPTS • Applied computing → Computers in other domains → Computing in government → E-government

KEYWORDS Open Government Data, Evaluation, Performance

ACM Reference format:

Lei Zheng, Wai-Min Kwok, Vincenzo Aquaro, Xinyu Qi, Wenzeng Lyu. 2020. Evaluating global open government data: methods and status. In Proceedings of the 13th International Conference on Theory and Practice of Electronic Governance (ICEGOV 2020), 23-25 September 2020, Athens, Greece, 11 pages. https://doi.org/10.1145/3428502.3428553

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ICEGOV'20, September 23–25, 2020, Athens, Greece © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7674-7/20/09…$15.00 https://doi.org/10.1145/3428502.3428553

1. INTRODUCTION This paper discusses the current status and global trends of open government data (OGD) by analyzing data related to open government data as observed in the United Nations E-Government Survey. In the following sections, Section 2 introduces the background of global development of open government data, and elaborates the benefits and risks brought about by opening government data. Section 3 reviews existing literature related to evaluation on open government data. In Section 4, the paper constructs an open government data evaluation framework and describes the methods of evaluating global open data performances with data collected in the United Nations E-Government Survey. Section 5 displays the findings of the analysis and evaluation. Based on the findings, Section 6 elaborates the conclusions and makes corresponding policy recommendations. Finally, Section 7 discusses the limitations of this research that could be further explored in the future.

2. BACKGROUND In the era of big data, the ability of humans to acquire, manage and use data has never been greater. The growing proliferation of data, generated by the development of information and communication technologies (ICTs), requires increased data openness to transform itself into a powerful force capable of changing the socio- economic fabric. During the recent years, it is evident that there is a fast-evolving global trend of open government data towards a more inclusive, interactive, contextualized and innovative stage that supports the achievement of the 2030 Agenda for Sustainable Development.

381

mailto:[email protected]





https://doi.org/10.1145/3428502.3428553


https://doi.org/10.1145/3428502.3428553

ICEGOV 2020, 23-25 September 2020, Athens, Greece Zheng, Lei et al.

OGD has become popular not only due to the novelty of its practice, but also the capacity for it to generate both social and economic values.

OGD first became a worldwide movement through policy changes in the late 2000s. Since the publication of the “8 Open Government Data Principles”, more governments have realized the untapped potential of the huge quantities of data produced or commissioned by public bodies to promote transparency, collaboration and participation by making government data accessible to all without restrictions [1]. Governments across the globe are beginning to become increasingly attentive of OGD due to its potential in improving government functions, increasing public awareness and promoting sustainable development to generate tremendous economic and social value. In the public sector, OGD can aid in better policy-making and public services, as well as stronger public trust since it can be used to help the public better understand what the government does, how well it performs, and to hold it accountable for wrongdoing or unfulfilled promises [2]. Moreover, OGD will further promote the development of technology, such as artificial intelligence (AI) and robotics.

The development of OGD in recent years has shown the following trends: There is an increased number of national OGD portals with major points of growth in developing regions and Middle to Low E-Government Development Index (EGDI) countries, according to the E-government Survey. From 2014 to 2018, the total number of countries with a national or local OGD portal has reached 139 in 2018, comprising 72% of the United Nations Member States [3]. Compared with 46 countries in 2014 and 106 countries in 2016, the number of countries with national open government data portals has seen a remarkable increase [4][5]. Most OGD portals are comprised of a directory or metadata repositories depicting the fundamental concepts, methodology and structure of the data [3]. Meanwhile, more and more OGD portals are using advanced data analytics and interactive visualization tools and providing application programming interfaces (APIs) and developer networks to build a ‘developer friendly’ environment. Furthermore, a trend of regional OGD portal coalitions is appearing including such examples as Open Data for Africa and European Data Portal.

OGD is growing in popularity with governments across the globe working to improve its strategy and implementation. Likewise, international organizations are also involved, evaluating countries’ open government data and making other efforts such as launching open data initiatives and programs, hosting events, and providing capacity development toolkits and platforms.

OGD has been expanding rapidly not only in terms of the number of participating countries and international organizations, but also the quantity, coverage, and variety of subject matter of opened datasets. According to a recent publication from The World Wide Web Foundation, 71% of existing data is available online, though not all data is in machine-readable form [6]. However, the actual quantity as well as quality of released datasets varies greatly among countries.

Furthermore, governments have begun to set policies and legislative frameworks to ensure an equal and enabling environment for both citizens and developers to use open data. Most but not all countries have some kind of Open Data License indicated on their respective portals to guide users in how to use and reuse available data. Some OGD portals provide guidance about the complicated datasets and additional features such as allowing users to request new open datasets, encouraging collaborative computer programming and stimulating the creation of online Apps about public open data. Although OGD can yield many benefits, it also comes with risks. When the legal and effective use of data cannot be ensured, personal privacy and information security may be especially threatened. Moreover, with the prevalence of opening and using data, people who cannot access and use data will be left behind, especially for the poor and vulnerable, women, children and the elderly, which is inconsistent with the overarching principle of the 2030 Agenda for Sustainable Development, to “leave no one behind”.

3. LITERATURE REVIEW At present, there are mainly two types of research on open government data evaluation. One is to construct evaluation frameworks from a theoretical perspective, and the other focuses on comparative analysis of existing evaluation programs.

Various open data evaluation frameworks are constructed with different focuses. Veljković, N., et al. (2014) proposed benchmark proposals for open government, which evaluate through the openness, transparency, participation and collaboration of data [7]. Sayogo, D. S., et al. (2014) divided the results of open government data portal assessments into three sections: data representation, participatory and engagement capability and online capabilities [8]. Viscusi, G., et al. (2014) discussed a quality-based open government data compliance assessment framework. The criteria include three different quality dimensions: completeness, accuracy and timeliness [9]. Zuiderwijk, A., et al. (2014) established an open data framework to compare open data policies, which includes the factors of environment and context, policy content, performance indicators and public values [10]. Donker, F. W., et al. (2017) developed a framework to comprehensively assess open data provision, open data governance, and open data user characteristics [11].

There is also research on comparative analysis of existing evaluation programs. Susha, I., et al. (2015) compared five open data benchmarks, including Open Data Readiness Assessment, Open Data Barometer, Global Open Data Index, European PSI Scoreboard and Open Data Economy Benchmarking Research, through metadata and meta-theories [12]. World Wide Web Foundation and The Governance Lab at NYU (2015) reviewed the current landscape of open data assessment and outlined a common assessment framework developed through collaborative workshops for open data activities [13]. Xia Yikun (2015) categorized two types of open government data evaluation programs: readiness evaluation and developmental evaluation [14].

382

Evaluating global open government data: methods and status ICEGOV 2020, 23-25 September 2020, Athens, Greece

4. METHDOLOGY

4.1. Comparing current international practices on evaluating OGD

Along with the global development of open government data practices, evaluations on open data practices emerge accordingly. There have been some internationally recognized evaluation reports.

Considering the influence of the evaluation programs and the credibility of the evaluation organizations, this paper chooses and examines seven representative open government data evaluation programs to systematically compare and analyze their frameworks, indexes and methods (Table 1). These projects are divided into two categories. The first are actual evaluation programs which have been conducted, such as the Open Data Barometer and the Global Open Data Index which are worldwide projects, and the Open Government Data Index, the European PSI Scoreboard and the Open Data Monitor, which are regional projects that primarily focus on OECD countries and the European Union member states. The second are evaluation frameworks and tools, such as the Open Data Readiness Evaluation and the Open Data Maturity Model.

Table 1: summaries of existing OGD evaluation programs

ID EVALUA-

TION PROJECT

EVALUATI-ON

SUBJECT

EVALUATION OBJECT METHODOLOGY

P1 Open Data Barometer

[15]

World Wide Web

Foundation

2017 (4th edition): 115

countries;

2015 (3rd edition: 92 countries;

2015 (2nd edition): 86 countries;

2013: 77 countries;

Government self-evaluation, Peer-

review, Expert survey, Secondary data, Quantitative

data and qualitative evaluation

P2 Global

Open Data Index [16]

Open Knowledge

2016/2017: 94 places;

2015: 122 places;

2014: 97 places;

2013: 60 places;

Snowball sampling, Volunteer surveys

and interviews, Expert evaluation,

Peer evaluation

P3 OURdata Index [18] OECD

OECD member and

partner countries

Self-evaluation survey (Survey

respondents were predominantly

chief data officers in these countries).

P4

European PSI

Scoreboard [19]

EU EU Countries

Crowdsourced data collection, expert

interview and quantitative

analysis with seven indicators

P5 Open Data

Monitor [20]

EU EU Countries

Automated analysis of open data directories,

etc. of data portals, Providing real-time data and methods

P6

Open Data Readiness

Assessment [21]

The World Bank's Open Government

Data Working Group

readiness of a government or

individual agency

Surveys and interviews with

government agencies and other

stakeholders

P7 Open Data Maturity

Model [22]

Open Data Institute

Organizations that plan to or have conduct

open data activities

Self-evaluation

The comparison of the seven existing data evaluation programs identifies five key dimensions in current evaluation frameworks, which are Foundation, Platform, Data, Use and Impact. Foundation indicates the internal and external environments of opening government data, particularly the policy and institutional framework, that supports open data initiatives. Platform refers to the online platform of open data. Data indicates the standards, amount, quality and types of datasets, which form the core of open data practices. Platform and Data present the outputs of open data practices. Use signifies how open data have been used, including demands, purposes and forms. Among these five dimensions, Foundation, Platform and Data all affect the Use, and Impacts presents the ultimate effects and value of the open data usage. (Figure 1)

Figure 1: index framework of current OGD practices

The analysis finds that most current evaluation programs focus on Data and Foundation. Indicators related to Data emphasize both the quantity, quality and standards of data. Indicators related to Foundation emphasize laws and regulations, political will and leadership as well as organization and management. Relatively speaking, the current evaluation programs give less attention to Platform, Use and Impact, probably because of the difficulties in the assessing and immaturity of open government data practices.

The paper also finds that both global and regional evaluation projects highlight the evaluation of Foundation and Data, while

383


regional projects also involve Use and Impact. One possible reason may be that it is more convenient and feasible to conduct more in-depth observations on a limited number of countries within a specific area.

4.2. Evaluating global OGD performance

4.2.1. Evaluation framework The analysis also finds that none of the current international evaluation programs covers all member states of the United Nations to reflect a global picture of open government data movement. Therefore, based on findings and lesson learned from the above comparison and analysis, this paper builds up a framework to evaluate the open data performances of all 193 member states of United Nations.

As shown in Table 2, the evaluation framework is composed of 11 questions related to open government data in the UN E-government Survey 2018, which are grouped into three dimensions: Policy and Institutional Framework, Platform and Data. In the evaluation framework, Data is given the highest weight, and the weight of Platform and Policy and Institutional Framework rank the second and the third respectively.

Table 2: evaluation indicators and framework

LEVEL WEIGHT INDICATOR QUESTIONNAIRE

Policy and Institutional

Framework

20% POLICY

Is there an Open Government Data

policy available online?

Platform 25%

PORTAL Is there an open government data

portal?

GUIDE

Is there guidance available for using Open Government

datasets?

DATAREQUEST

Can the public propose/request new

open datasets be made available

online?

Data 55%

HEALTH

Are there Open Government datasets available on HEALTH? These could be either

i) Not machine-readable (PDF,

scanned files etc.) or ii) Machine-readable

(CSV, JSON, XML etc.)

EDUCATION

Are there Open Government datasets

available on EDUCATION? These could be either i) Not

machine-readable (PDF, scanned files etc.) or ii) Machine-

readable (CSV, JSON, XML etc.)

EMPLOYMENT


available on EMPLOYMENT? i)

Not machine-readable (PDF, scanned files etc.) ii) Machine-


SOCIALSECURITY

Are there Open Government datasets available on SOCIAL PROTECTION? i) Not

machine-readable (PDF, scanned files etc.) ii) Machine-


ENVIRONMENT


available on ENVIRONMENT? i)

Not machine-readable (PDF, scanned files etc.) ii) Machine-


METADATA

Does the Government’s Open Data Portal have a data dictionary or

metadata repository?

PROMOTION

Does the government encourage open data

app competitions (hackathons)?

4.2.2. Data collection The evaluation is based on data from 2018 Online Service Index (OSI). OSI data was collected from an independent survey questionnaire, conducted by the Department of Economic and Social Affairs of the United Nations (UN DESA), which assesses the national online presence of all 193 United Nations Member States. This data is collected by a group of researchers under the supervision of UN DESA.

For the 2018 Online Service Index (OSI) data, a total of 206 online United Nations Volunteer researchers from 89 countries with coverage of 66 languages were organized to assess each country’s national website in its native language using a Questionnaire including questions on open government data. In addition, all United Nations Member States were requested (through the Member State Questionnaire) to provide information regarding their website addresses (URL) for different government ministries and the national portal(s).

4.2.3. Index calculation The encoding of 11 questions and answers are shown in Table 3. Six questions’ answers are encoded in a binary format, where 1 means the country meets the requirement of the indicator and 0 means it did not. The other five questions include two sub-questions, which are encoded in 1/0.5/0 format as scores. “1” indicates both sub-questions are Yes and “0.5” means only one sub-question is Yes.

Moreover, this paper modified the encoding of the question of ID 1 (different from the encoding in UN E-government Survey

384


2018), the country will be given score 1 in this question when the national open data portal can be found. The country will not score if there is no national open data platform, only local government open data platforms, or the platform is owned by the department of the government.

By encoding and evaluating these 11 questions, this paper produces a new index, Open Government Data Index (OGDI), which is a weighted average of scores for each country. The OGDI ranges from 0 to 1 to indicates scores from the lowest to the highest level.

Table 3: 14 questions and answer’s coding

ID QUESTION SCORES

1 Is there an open government data portal? 0 indicates None;

1 indicates Yes

2 Is there an Open Government Data policy available online?

0 indicates None;

1 indicates Yes

3 Is there guidance available for using Open Government datasets?

0 indicates None; 1 indicates Yes

4 Can the public propose/request new open datasets be made available online?

0 indicates None;

1 indicates Yes

5 Does the government encourage open data

app competitions (hackathons)? 0 indicates None;

1 indicates Yes

6 Does the Government’s Open Data Portal

have a data dictionary or metadata repository? 0 indicates None;

1 indicates Yes

7

Are there Open Government datasets available on HEALTH? These could be either i) Not

machine-readable (PDF, scanned files etc.) or ii) Machine-readable (CSV, JSON, XML etc.)

0 indicates None;

0.5 indicates the first question is Yes;

1 indicates Yes for all

8

Are there Open Government datasets available on EDUCATION? These could be either i) Not machine-readable (PDF, scanned files etc.) or ii) Machine-readable (CSV, JSON, XML etc.)

0 indicates None;



9

Are there Open Government datasets available on EMPLOYMENT? i) Not machine-readable (PDF, scanned files etc.) ii) Machine-readable


0 indicates None;



10

Are there Open Government datasets available on SOCIAL PROTECTION? i) Not machine-

readable (PDF, scanned files etc.) ii) Machine-readable (CSV, JSON, XML etc.)

0 indicates None;



11

Are there Open Government datasets available on ENVIRONMENT? i) Not machine-readable (PDF, scanned files etc.) ii) Machine-readable


0 indicates None;



5. FINDINGS

5.1. Overall performance In 2018, 34 countries (18%) score “Very High”, with OGDI values of 1, which is full mark. 40 countries (21%) score “High”, with OGDI values in the range of 0.80 to 0.97. 43 countries (22%) score “Middle”, with OGDI values in the range of 0.41 to 0.80, while 76 countries (39%) score “Low”, with OGDI values lower than 0.41 (Table 5).

Table 5: all countries grouped by OGDI levels in alphabetical order

VERY HIGH

OGDI 2018 (1, FULL MARK)

HIGH OGDI 2018

(0.80-0.97)

MIDDLE OGDI 2018

(0.41-0.80)

LOW OGDI 2018 (Lower than 0.41)

Australia Albania Andorra Afghanistan Micronesia

Austria Azerbaijan Argentina Algeria Monaco

Belgium Bahrain Armenia Angola Mozambique

Brazil Bangladesh Bahamas Antigua and Barbuda Myanmar

Brunei Darussalam Burkina Faso Barbados Belize Nauru

Bulgaria Chile Belarus Benin Nicaragua

Canada China Bhutan Bosnia and Herzegovin

a Niger

Denmark Colombia Bolivia Botswana Pakistan

Finland Croatia Burundi Cambodia Palau

France Cyprus Costa Rica Cameroon Papua New

Guinea

Germany Czech Republic

Côte d'Ivoire Cape Verde Saint Kitts

and Nevis

India Ecuador Dominica Central African

Republic Saint Lucia

Ireland El Salvador Dominican Republic Chad

Saint Vincent and the

Grenadines

Italy Estonia Ethiopia Comoros Samoa

Japan Georgia Guatemala Congo San Marino

Kazakhstan Ghana Honduras Cuba Sao Tome

and Principe

Luxembourg Greece Iceland

Democratic People's

Republic of Korea

Seychelles

Mexico Hungary Islamic

Republic of Iran

Democratic Republic of the Congo

Solomon Islands

Netherlands Indonesia Jordan Djibouti Somalia

New Zealand Israel Kuwait Egypt South Sudan

Norway Kenya Latvia Equatorial Guinea Sudan

Peru Kyrgyzstan Malawi Eritrea Suriname

Poland Liechtenstei

n Malta Fiji Swaziland

Portugal Lithuania Mauritius Gabon Syrian Arab Republic

Republic of Korea Malaysia Mongolia Gambia Tajikistan

Republic of Moldova Morocco Montenegr

o Grenada Timor-Leste

Russian Federation Oman Namibia Guinea Turkmenista

n

Singapore Paraguay Nepal Guinea-Bissau Tuvalu

Sweden Philippines Nigeria Guyana Vanuatu

385


Switzerland Romania Panama Haiti Venezuela

North Macedonia Serbia Qatar Iraq Vietnam

United Arab Emirates

Slovak Republic Rwanda Jamaica Yemen

United Kingdom of

Great Britain and Northern Ireland

South Africa Saudi Arabia Kiribati Zimbabwe

Uzbekistan Spain Senegal Lao People's Democratic

Republic

Togo Sierra Leone Lebanon

Tunisia Slovenia Lesotho

Turkey Sri Lanka Liberia

Ukraine Thailand Libya

United

States of America

Tonga Madagascar

Uruguay Trinidad

and Tobago Maldives

Uganda Mali

United

Republic of Tanzania

Marshall Islands

Zambia Mauritania

It reflects two characteristics: first, most countries (about 61%)

are in the middle and low OGDI levels and there is a large space for improvement. Second, the gap in open data development levels is large. A minority of countries with high levels could meet all the requirements of the evaluation index, whereas there are more countries failed to get scores on most indicators.

Figure 2 shows a thematic global map of OGDI score levels. Different colors stand for the four types of score levels of countries. It can be seen from the figure that countries with OGDI in middle or low level are mainly concentrated in central and western Asia, middle Americas and the whole African region. Countries with high OGDI are mainly concentrated in Europe, East Asia, North America and parts of South America.

Figure 2: global map of OGDI score level

a) Very high and High OGDI Group

The percentage of ‘very high’ and ‘high’ level countries (OGDI score higher than 0.80) is 39%, accounts for more than one-third of the UN Member States (74 countries). These meet the requirements of most indicators in policy and institutional framework, platform and data, which shows really good performance on open government data.

b) Middle OGDI Group

43 countries in the Middle OGDI level group with scores between 0.41 and 0.80, which is around 20% of UN Member States, have generally lower scores in some indicators.

c) Low OGDI Group

The percentage of countries with scores lower than 0.41 is 39% (76 countries), which is roughly equivalent to the ‘very high’ and ‘high’ group. Most are developing and least developed countries.

5.2. Leading countries Those 34 countries with very high score levels are the leading

countries. The list of the top-ranking countries in open government data are presented in Table 6 with corresponding OGDI scores and its three components scores: foundation (i.e. policy and institution framework), platform and data. The OGDI score of ‘1’ for these 34 countries is a full score and shows that all requirements of each indicator have been met. Twenty of all the top countries are from Europe, while eight countries are from Asia and four countries are from the Americas. Australia and New Zealand are the only two countries from Oceania. No African countries are in the top leading countries.

Table 6: 34 top countries’ OGDI score board

COUNTRY NAME REGION SCORE POLICY &

INSTITUTION PORTAL DATA

Australia Oceania 1 0.2 0.25 0.55

Austria Europe 1 0.2 0.25 0.55

Belgium Europe 1 0.2 0.25 0.55

Brazil Americas 1 0.2 0.25 0.55

Brunei Darussalam Asia 1 0.2 0.25 0.55

Bulgaria Europe 1 0.2 0.25 0.55

Canada Americas 1 0.2 0.25 0.55

Denmark Europe 1 0.2 0.25 0.55

Finland Europe 1 0.2 0.25 0.55

France Europe 1 0.2 0.25 0.55

Germany Europe 1 0.2 0.25 0.55

India Asia 1 0.2 0.25 0.55

386


Ireland Europe 1 0.2 0.25 0.55

Italy Europe 1 0.2 0.17 0.55

Japan Asia 1 0 0.25 0.55

Kazakhstan Asia 1 0.2 0.25 0.55

Luxembourg Europe 1 0.2 0.25 0.55

Mexico Americas 1 0.2 0.25 0.55

Netherlands Europe 1 0.2 0.25 0.55

New Zealand Oceania 1 0.2 0.25 0.55

Norway Europe 1 0.2 0.25 0.55

Peru Americas 1 0.2 0.25 0.55

Poland Europe 1 0.2 0.25 0.55

Portugal Europe 1 0.2 0.25 0.55

Republic of Korea Asia 1 0.2 0.25 0.55

Republic of Moldova Europe 1 0.2 0.25 0.55

Russian Federation Europe 1 0.2 0.25 0.55

Singapore Asia 1 0.2 0.25 0.55

Sweden Europe 1 0.2 0.25 0.55

Switzerland Europe 1 0.2 0.25 0.55

The Former Yugoslav Republic

of Macedonia Europe 1 0.2 0.25 0.55

United Arab Emirates Asia 1 0.2 0.25 0.55

United Kingdom of Great Britain and Northern Ireland

Europe 1 0.2 0.25 0.55

Uzbekistan Asia 1 0.2 0.25 0.55

5.3. Indicators compliance rate Figure 3 shows the compliance rate of the indicators (excluding those related to datasets). Policy has the highest rate, and 64 percent of all UN Member States provide open data policies online. The number of countries with national Open Government Data (OGD) portals has reached 90, comprising 47 percent of all UN Member States.

Figure 3: compliance rate of indicators

Nearly 60 percent of national portals also have a directory or metadata repositories describing the data underlying concepts, methodology and structure. Around 63 percent of countries that have OGD portals and websites also provide guidance on using and navigating the complex datasets, encourage users to request new datasets, initiate hackathons and use public open data for creating online Apps.

5.4. Most common open datasets There are five kinds of datasets that countries commonly open: health, education, employment, social security and environment. The average scores of five datasets are shown in Figure 4. Scores of health, employment and education are relevantly higher, and scores of social security and environment are comparatively lower.

Figure 4: average score of five kinds of open datasets

In Figure 5, regions are analyzed as group to compare their performance of opening different kinds of datasets. The gap between regions is reflected in every aspect. Europe’s scores are higher than other regions in all datasets, and balance well in all aspects. While the scores of Africa and Oceania are not balanced in different datasets, especially the scores in social security and environment are quite low.

64%61%

54% 53% 53%47%

Policy Metadata Datarequest

Guide Promotion Portal

00,20,40,60,8

1Health

Education

EmploymentSocial Security

Environment

387


Figure 5: average score in 5 kinds of open data sets grouped by region

5.5. Regional performances Figure 6 highlights asymmetries and distributional impacts of open government data development within these regions. All the regions currently have low OGDI level countries. Most countries in Africa (62%) and Oceania (79%) have low OGDI scores. No countries in Africa are in the ‘very high’ level and no countries in Oceania are in the ‘high’ level. European countries have an excellent performance in open government data. In Europe, around 70% of countries have very high or high OGDI scores, while less than 10% of countries are in low level. Asia and the Americas have similar distribution in OGDI score levels, but more percent of Asian countries are in groups of very high and high level.

Figure 6: OGDI score level grouped by region

Figure 7 shows the breakdown of OGDI in each region. Europe scores the highest on each component of OGDI while Oceania is the lowest. Data has the greatest difference in all the regions, whereas scores of policy and institution and platform are very close in all regions.

Figure 7: breakdown of OGDI per geographical region

Africa has large gaps in infrastructure, including broadband infrastructure and access to broadband services. This is the evidence of the low scores of OGDI in every aspect. However, some African countries have good performance in open government data. Table 7 shows the top 10 countries in scores of OGDI in Africa. South Africa has the highest scores 0.97, which is close to 1. For Burkina Faso, Togo, Kenya and Ghana, even though they are low income countries, they still have good performance on open data.

Table 7: top 10 African countries in OGDI

COUNTRY NAME SCORE POLICY &

INSTITUTION PLATFORM DATA

South Africa 0.97 0.2 0.25 0.52

Burkina Faso 0.90 0.2 0.25 0.45

Togo 0.90 0.2 0.25 0.45

Kenya 0.86 0.2 0.25 0.41

Ghana 0.80 0.2 0.25 0.35

United Republic of Tanzania

0.76 0.2 0.25 0.31

Ethiopia 0.73 0.2 0.25 0.28

Uganda 0.70 0.2 0.08 0.42

Zambia 0.70 0 0.15 0.55

Rwanda 0.66 0.2 0.08 0.38

The Americas have higher performance in open data than

Africa and Oceania, with more countries in very high and high OGDI levels. Table 8 shows the top 10 countries in scores of OGDI

HEALTH

EDUCATION

EMPLOYMENTSOCIALSECURITY

ENVIRONMENT

Africa

Americas

Asia

Europe

Oceania

0%11% 18%

43%

14%10%

24%27%

30%

0%27%

30%20%

17%

7%

63%

35% 36%

9%

79%

Africa Americas Asia Europe Oceania

VeryHigh High Middle Low

388


in Americas. All top 10 countries are in very high and high level. Brazil, Canada, Mexico and Peru have the best performance with full mark.

Table 8: top 10 American countries in OGDI

COUNTRY NAME SCORE

POLICY & INSTITUTION PLATFORM DATA

Brazil 1.00 0.2 0.25 0.55

Canada 1.00 0.2 0.25 0.55

Mexico 1.00 0.2 0.25 0.55

Peru 1.00 0.2 0.25 0.55

Ecuador 0.97 0.2 0.25 0.52

United States of America 0.97 0.2 0.25 0.52

Chile 0.93 0.2 0.25 0.48

Croatia 0.93 0.2 0.25 0.48

Uruguay 0.90 0.2 0.25 0.45

Colombia 0.86 0.2 0.25 0.41

Asia is not only the most populous region, but also the largest continent in terms of land mass. The open government data development trend is highly diverse across the countries in Asia. Eight countries are in very high OGDI score level. India and Kazakhstan are the only two countries with lower middle income but in very high score level. There are still 16 countries in Asia that are in the low OGDI score level, which shows the large gap between countries in Asia in open government data (Table 9).

Table 9: top 10 Asian countries in OGDI

COUNTRY NAME SCORE

POLICY & INSTITUTION PLATFORM DATA

Brunei Darussalam 1.00 0.2 0.25 0.55

India 1.00 0.2 0.25 0.55

Japan 1.00 0.2 0.25 0.55

Kazakhstan 1.00 0.2 0.25 0.55

Republic of Korea 1.00 0.2 0.25 0.55

Singapore 1.00 0.2 0.25 0.55

United Arab Emirates 1.00 0.2 0.25 0.55

Uzbekistan 1.00 0.2 0.25 0.55

Israel 0.97 0.2 0.25 0.515

Georgia 0.93 0.2 0.25 0.48

Malaysia 0.93 0.2 0.18 0.55

Oman 0.93 0.2 0.25 0.48

Philippines 0.93 0.2 0.25 0.48

Europe has the best performance in all regions in every aspect of open government data. Twenty countries in Europe are in the very high score level (nearly 60%), including the Republic of Moldova, the only lower middle-income country in this level. While there are still a few countries have middle and low OGDI scores (Table 10).

Table 10: top 10 European countries in OGDI

COUNTRY NAME SCORE POLICY &

INSTITUTION PLATFORM DATA

Austria 1.00 0.2 0.25 0.55

Belgium 1.00 0.2 0.25 0.55

Bulgaria 1.00 0.2 0.25 0.55

Denmark 1.00 0.2 0.25 0.55

Finland 1.00 0.2 0.25 0.55

France 1.00 0.2 0.25 0.55

Germany 1.00 0.2 0.25 0.55

Ireland 1.00 0.2 0.25 0.55

Italy 1.00 0.2 0.25 0.55

Luxembourg 1.00 0.2 0.25 0.55

Netherlands 1.00 0.2 0.25 0.55

Norway 1.00 0.2 0.25 0.55

Poland 1.00 0.2 0.25 0.55

Portugal 1.00 0.2 0.25 0.55

Republic of Moldova 1.00 0.2 0.25 0.55

Russian Federation 1.00 0.2 0.25 0.55

Sweden 1.00 0.2 0.25 0.55

Switzerland 1.00 0.2 0.25 0.55

The Former Yugoslav

Republic of Macedonia

1.00 0.2 0.25 0.55

United Kingdom of Great Britain and Northern

Ireland

1.00 0.2 0.25 0.55

Oceania has two developed countries, Australia and New Zealand. Other countries are island-States with smaller populations and economies. Table 1 shows the top 10 Oceania countries with very high OGDI score. Tonga is the only country in Oceania that has a middle OGDI score. The other countries are all in low OGDI score levels. This illustrates the wide gap comparing Oceania and other regions in open government data development, as well as within Oceania region itself.

389


Table 11: top 10 Oceania countries in OGDI

COUNTRY NAME

SCORE POLICY & INSTITUTION

PLATFORM DATA

Australia 1.00 0.2 0.25 0.55

New Zealand 1.00 0.2 0.25 0.55

Tonga 0.63 0.2 0.08 0.35

Palau 0.38 0.2 0 0.18

Solomon Islands

0.29 0 0.08 0.21

Vanuatu 0.28 0 0 0.28

Fiji 0.22 0 0.15 0.07

Micronesia 0.21 0 0 0.21

Tuvalu 0.21 0 0.07 0.14

Kiribati 0.14 0 0 0.14

5.6. National income and open data development

Countries with different GDP levels and income levels differ in the performance of open data, as shown in Figure 8. The line between two colors in each bar identify the average score of each group. The bottom and top rims of each box indicate the percentile 25 and percentile 75 of score. Generally, the higher the income of one country, the better the performance in open data. Most countries in high and upper middle-income groups tend to have higher OGDI scores than average OGDI scores. High-income countries, with advanced levels in telecommunications infrastructure and human capital development, develop faster by expanding the scope and quality of the open data on online platforms. But there are exceptions: Equatorial Guinea is in a high-income group, but has a low OGDI score (0, low level). Though in low income group, Burkina Faso has a high OGDI score (0.9, high level).

Figure 8: distribution of OGDI 2018 scores across income level

Moreover, there is a positive correlation between the country’s OGDI scores and its GDP, as presented in Figure 9. The fitting

straight line, of which R square equals 0.42 and p-value is less than 0.00, significantly identify the positive relation between open data and GDP. But there are distinguished outliers with high OGDI scores but comparatively low GDP, such as Republic of Moldova (1, lower middle income), The Former Yugoslav Republic of Macedonia (1, upper middle income), Togo (0.9, low income) and Kyrgyzstan (0.8, lower middle income). These are exemplary cases of optimizing OGD values with limited resources.

Figure 9: positive correlation between OGDI 2018 with GDP

6. POLICY RECOMMENDATIONS From a global perspective, open data is still in its early development stage in most countries with untapped potential in supporting sustainable development. Only 47 percent of all United Nations Member States have launched national open government data portals, and around 61% countries’ level of OGD development are below medium and low.

There is a large gap in the level of open data development among countries. Only a few countries with higher open data levels meet the requirements of all evaluation indicators, while some countries do not receive any scores on most indicators. The most common open datasets are education, health and employment. as such, the paper suggests countries enhance openness of datasets in the field of environment, equality and social security.

From a regional perspective, Europe demonstrates the best performance in open government data among all regions. It receives high scores in each indicator and the gap among countries is small. Countries in Africa and Oceania have a relatively weak foundations of open government data. Countries in Asian and Americas regions are divided in OGD development level, as countries in high and very high level of are as many as countries in middle and low level.

The correlations between open government data and national income and GDP are significantly positive. Countries with higher national income and GDP are more likely to have a better performance in open data. However, there are, outliers, low

390


income countries with high OGDI score, which can be used as successful cases for low income countries to develop open data with limited resources or know-how.

According to the findings and conclusions, this paper provides the following recommendations:

• Policy and institutional framework: national governments should introduce policies and regulations that are focused on open government data at the national or central government level, so as to guarantee the continuous and effective opening of government data, and clarify the relevant rights and obligations of open data in the policy and legal agreement.

• Platform: national governments should build national open data platforms to provide users with raw, primary and machine-readable datasets. Statistical data from statistical and other departments, which are mostly aggregated data, should not be the primary sources of open data. Government should also provide guidelines on the platform to help users better understand and use open data. The functions of data requests should be provided on the platform, so that users can apply for data they need from governments.

• Data: national governments should provide metadata on platforms to help users better understand open data. A data directory/dictionary is also needed to help users to know better about datasets. Opening more high-value government datasets in areas such as health, education, environment, employment, and social security, which are closely related to people's work and life are critical to the success.

• Utilization: national governments should encourage users to fully use open data. Governments could organize activities such as open data innovation competitions to promote the utilization of open data, and work with a variety of social organizations to enhance the effectiveness and sustainability of such activities.

7. DISCUSSION The novelty of this evaluation is that it has simultaneously assessed the global open data performances from the three dimensions: Policy and Institutional Framework, Platform and Data. There are 193 countries evaluated, and the evaluation scope is wide, which can reflect the current status and trends of open data worldwide. However, due to the limitations of the data from the United Nations E-Government Survey, it’s hard to conduct more detailed and comprehensive analysis. The indicators on policy, platform and data are not sufficient, and the analysis of use and impact has not been conducted.

Through analysis, this paper also found some limitations of the United Nations E-Government questionnaire: too few survey questions for open data; survey questions are not specific with more details; there is no strict distinction between specialized national open data platforms and other national and sub-national government portals; scoring formats are binary (0/1), focusing

only on Yes or No without concerning the level and quality of open data.

REFERENCES [1] OECD. Open Government Data. Retrieved from

https://www.oecd.org/gov/digital-government/open-government-data.htm.

[2] OECD. 2013. Open Government Data Towards Empirical Analysis of Open Government Data Initiatives. Retrieved from https://www.oecd-ilibrary.org/governance/open-government-data_5k46bj4f03s7-en. PDF.

[3] United Nations Department of Economic and Social Affairs. 2018. 2018 United Nations E-Government Survey. United Nations: New York. Retrieved from https://publicadministration.un.org/egovkb/en-us/Reports/UN-E-Government-Survey-2018.



[6] World Wide Web Foundation. 2017. Open Data Barometer Global Report 4th Edition. . Retrieved from https://opendatabarometer.org/doc/4thEdition/ODB-4thEdition-GlobalReport.pdf.

[7] Veljković, N., et al. 2014, Benchmarking open government: an open data perspective, Government information quarterly, 31(2): 278-290.

[8] Sayogo, D. S., et al. 2014, A framework for benchmarking open government data efforts, Proceedings of the 2014 47th Hawaii international conference on system sciences, 1896-1905.

[9] Viscusi, G., et al. 2014, Compliance with open government data policies: an empirical assessment of Italian local public administrations, Information polity, 19(3,4): 263-275.

[10] Zuiderwijk, A., et al. 2014, Open data policies, their implementation and impact: a framework for comparison, Government information quarterly, 31(1): 17-29.

[11] Donker, F. W., et al. 2017, How to assess the success of the open data ecosystem?, International Journal of Digital Earth, 10(3): 284-306.

[12] Susha, I., et al. 2015, Benchmarks for evaluating the progress of open data adoption: usage, limitations, and lessons learned, Social science computer review, 33: 613-630.

[13] World Wide Web Foundation and The Governance Lab at NYU. 2016. Towards common methods for assessing open data: workshop report & draft framework. Retrieved from http://opendataresearch.org/sites/default/files/posts/Common%20Assessment%20Workshop%20Report.pdf.

[14] Xia Yikun. 2015, Comparison and Analysis on the Open Government Data Assessment Methods of International Organizations, Library and Information Services, 19: 75-83.

[15] World Wide Web Foundation. Open Data Barometer. Retrieved from http://opendatabarometer.org.

[16] Open Knowledge Foundation. Global Open Data Index. Retrieved from http://index.okfn.org.

[17] United Nations Department of Economic and Social Affairs. United Nations E-Government Survey. Retrieved from https://publicadministration.un.org/egovkb/en-us/Reports/UN-E-Government-Survey-2018.

[18] OECD. OURdata Index. Retrieved from https://www.oecd.org/internet/digital-government/open-government-data.htm.

[19] ePSI Platform PSI Scoreboard. Retrieved from https://www.europeandataportal.eu/en/content/epsi-platform-psi-scoreboard.

[20] Open Data Monitor. Retrieved from https://opendatamonitor.eu.

[21] Open Data Readiness Assessment. Retrieved from http://opendatatoolkit.worldbank.org/en/odra.html.

[22] Open Data Institute. Open Data Maturity Model. Retrieved from https://theodi.org/article/open-data-maturity-model-2/.

391

Evaluating global open government data: methods and status

Documents

Transcript of Evaluating global open government data: methods and status