POLICY IMPACT EVALUATION THROUGH PROSPERITY METRICS AND OPEN DATA SOURCES

Ourania MarkakiCostas KoutrasPanagiotis KokkinakosSotiris KoussourisJohn PsarrasNational Technical University of Athens

1-2 June 2015 EMCIS 2015 – Athens, Greece

Evaluating Policy Making via Prosperity Metrics

•Which metrics?

•What kind of indicators?

•For which kind of applications?

•How do we exactly interpret concepts like• Quality of life• Prosperity• Performance measurement

1-2 June 2015 EMCIS 2015 – Athens, Greece 2

?

The Social Indicators Zoo

• Aggregate (summary) indicators• e.g. GDP

• Composite Indicators• Combine individual indicators into a single composite index, possibly weighting

the indicators according to their relative importance – e.g. HDI

• Headline Indicators• Sustainability Indicators

• International, National, Regional and Local indicators

1-2 June 2015 EMCIS 2015 – Athens, Greece 3

Issues on the use of indicators

• Institutionalization: setup routine procedures and practices to ensure the continuing existence of an indicator and to legitimize the method and concept of the measure. (Drawbacks identified).

• Avoiding bias: indicators should be produced by professional statistical agencies that have a strong awareness of policy issues, without having responsibility for them.

• Maintaining the sensitive balance between (global) standardization and local democracy (identifying the relevant measure to the specific local circumstances)

1-2 June 2015 EMCIS 2015 – Athens, Greece 4

A methodological framework of indicator development [Coombes, Wong, 1994]

1-2 June 2015 EMCIS 2015 – Athens, Greece 5

Step 1: Conceptual consolidationClarifying the concept to be represented by the analysis

Step 2: Analytical StructuringProviding an analytical framework within which indicators will be collated and analysed

Step 3: Identification of IndicatorsTranslation of key factors identified in step 2 into specific measurable indicators

Step 4: Synthesis of indicator valuesSynthesizing the identified indicators into composite index/indices or into analytical summary

How can Policy Compass contribute?

• "Open data" movement and the "social computing" capabilities of today open a fantastic observatory of experimentation with (and for) Social Indicators

• Assess the limits of `causal analysis‘ in policy making, combining indicators with causal models

• Assess the feasibility of assisting policy design and decision-making with quantitative handles (Social Indicators) at the global & local level

• Evaluate the ramifications and the consequences of mixing the previous items with eGov techniques, eParticipation tools & Social Media.

1-2 June 2015 EMCIS 2015 – Athens, Greece 6

The Problem• Internet has evolved into a rich source

for information but also to an instrument of spreading misinformation and propaganda

• Lack of consensus about a suitable metric for measuring progress

• Difficulty of objectively assessing the impacts of government policies

The Proposed Approach Open Public Data Prosperity Indicators Fuzzy Cognitive Maps Argumentation Technology Deliberation Platforms and Social

Media

The Context

1-2 June 2015 EMCIS 2015 – Athens, Greece 7

Improve the quality and transparency of the policy

analysis and evaluation phases of the policy cycle for a variety of stakeholders, ranging from

citizens to policy makers

Policy cycle

Analysis

Implementation/Monitoring

The Proposed Approach: Overview

• A research prototype of an easy-to-use, highly visual and intuitive tool for:

• Constructing prosperity and other policy metrics with an easy-to-use visual language for defining variables and functions over open data sources.

• Constructing graphs and charts visualizing metrics for selected geographical regions and time periods.

• Annotating graphs and charts with political or policy events.• Constructing causal models with an easy-to-use visual tool for Fuzzy

Cognitive Maps (FCM).• Sharing and debating prosperity graphs and FCM across popular social

media platforms.• Summarizing and visualizing the debates in argument maps and

conducting structured surveys about policy issues• Aggregating opinions on policy issues, to formulate a common position in a

party or interest group.

1-2 June 2015 EMCIS 2015 – Athens, Greece 8

Pillar I: Prosperity Indexes

• Prosperity metrics capture the level of welfare and quality of life in a given region or society.

• Prosperity is a vague and subjective concept with essential psychological, social and economic aspects.

• There is no consensus about how to objectively measure prosperity• Indicators of economic growth:

• Gross Domestic Product (GDP)• Genuine Progress Index (GPI)• Index of Sustainable Economic Welfare (ISEW)• GINI Index

• Alternatives:• Human Development Index (HDI)• Legatum Prosperity Index• “Healthy life years statistics” by Eurostat

1-2 June 2015 EMCIS 2015 – Athens, Greece 9

Pillar II: Open Public Data

• Open and unrestricted access to large scale data sets is essential for political engagement and scientific research

• Available large scale data sets have nowadays their own self-contained existence rules.

• Micro-data can be used to construct new indicators of multifaceted nature.

• Sources of micro-data:• Eurobarometer surveys• European Union Statistics on Income and Living Conditions (EU-SILC) by Eurostat• Urban Audit (the European cities Eurostat)

1-2 June 2015 EMCIS 2015 – Athens, Greece 10

Pillar III: Fuzzy Cognitive Maps

• Fuzzy Cognitive Maps (FCMs) provide a well-founded, general-purpose and intuitive method for modelling and simulating relationships between variables.

• FCMs have been introduced by B. Kosko (1986) as a fuzzified version of Cognitive Maps, originally introduced by political scientist R. Axelrod (1976).

• An FCM is a fuzzy directed graph of nodes and edges, where nodes represent fuzzy concepts, describing behavioral characteristics of a system that occur to some degree, and directed edges represent the causal relationships among these concepts.

1-2 June 2015 EMCIS 2015 – Athens, Greece 11

• The graph edges are weighted by a real value from the interval [-1, 1], which expresses the strength of the relation between two concepts.

• FCMs have been widely used to model and simulate policies and their effects.

Pillar IV: Argumentation Technology

• Argumentation support systems are computer software for helping people participate in various kinds of goal-directed dialogues in which arguments are exchanged.

• The idea of using argumentation support systems for e-Participation can be traced back at least to Horst Rittel’s pioneering work in the early 1970s who used visual maps of arguments, to help people collaborate and find solutions to what he called “wicked problems”.

• “Wicked problems” have no algorithmic, scientific or objectively optimal solutions for a variety of reasons, including the lack of consensus among stakeholders about utilities and values.

• Typically, e-Participation projects make use of generic groupware systems (e.g. discussion fora, online surveys, etc.) not providing though specific technical support for argumentation.

1-2 June 2015 EMCIS 2015 – Athens, Greece 12

Pillar V: Deliberation Platforms and Social Media

• Deliberation platforms incarnate the efforts taken by government agencies, to increase citizens’ engagement in their decision and policy making processes.

• The first wave of deliberation platforms has witnessed extensive information on government activities, decisions, plans and policies, the proliferation of e-voting and e-consultation spaces, along with various types of e-fora.

• Still, the first generation of deliberation platforms did not meet the original expectations.

• The advent of Web 2.0 tools has created a more vivid environment and the popularity of the social media has set a new battlefield for the concept of e-Participation.

1-2 June 2015 EMCIS 2015 – Athens, Greece 13

Use Case Scenarios (1/2)

1-2 June 2015 EMCIS 2015 – Athens, Greece 14

Use Case Scenarios (2/2)

1-2 June 2015 EMCIS 2015 – Athens, Greece 15

Lessons Learnt

• There exists a growing concern on the limits of GDP-like measures for measuring societal welfare. GDP is very successful, but it is impossible to derive deep results on many aspects of everyday life with simple aggregate indicators alone.

• It makes perfect sense to attempt constructing aggregate, headline or composite indicators, to measure important social phenomena at the regional /local level.

• There do not exist ‘objective’ or ‘neutral’ indicators of any kind. The community does not pursue objective indicators, it rather attempts to construct useful ones.

• A broad experimentation on the calculation and exploitation of indicators, given the multitude of data available, is definitely interesting scientifically and will provide useful "social" feedback

1-2 June 2015 EMCIS 2015 – Athens, Greece 16

www.PolicyCompass.eu

www.twitter.com/PolicyCompassEU

www.facebook.com/PolicyCompass

PolicyCompass

Thank you!

Ourania Markaki, omarkaki@epu.ntua.gr