Procedures for the comparison of policy options: Scryer

The ex-ante evaluation of policies: The case of food safety regulations

Corso per dottorandi Economia e Statistica Agro-alimentare

Maddalena RagonaDipartimento di Scienze Statistiche, Università di Bologna

Bologna, febbraio-marzo 2012

Contents

• Fuzzy sets

• Scryer

Fuzzy logic

Classifying statements in «true» and «false» may be too restrictive…

Any statement may have a certain degree of truth• E.g. is your coffee bitter or sweet?

• 0.8 sweet

• 0.2 bitter

When linguistic variables are exploited, there are specific functions to manage different degrees of truth

Fuzzy logic - Coffee example

Truth

Quantity of sugar

1

0

Bitter Sweet Very sweet

Note that the fuzzy membership functions may have very different shapes, which also depend on how large they are (how uncertain is the judgement)

Fuzzy vs. Probabilistic Logic

The distinction is philosophical• Fuzzyness as «degree of belonging» to different

sets• (Subjective) probability: how much it is probable

that the element belongs to that set (it belongs to one set only, but there are different degrees of perception)

• Probability as a sub-set of fuzzy logic?• Fuzzy probability?

Scryer (MoniQA socio-economic evaluation tool)

Fuzzy multi-criteria tool to support decision-making Steps

1. Qualitative assessment of each impact for each policy option (coding/scoring procedure based on expert(s) judgement)

2. Feasibility filter (data availability, time, costs) to evaluate the possibility of quantitative assessment

3. Quantitative assessment of some impacts accounting for statistical error

4. Fuzzy multi-criteria comparison of options

Computer-based• Currently Excel spreadsheet, to be implemented into web-application

Characteristics Analysis of impacts based on the directions of the EC Impact

Assessment Guidelines (2009) Ranking of policy options based on NAIADE software, developed at JRC-

EC for environmental impact assessment It allows for both synthesis of quantitative (model-based) and qualitative

assessments without the need for monetisation It may take into account public sensitiveness It accounts for uncertainty in outcomes evaluations (including lack of data

/ external uncertainty like weather / expert internal uncertainty) Weighting of impacts is allowed for Sensitivity analysis of the policy ranking

Advantages of a fuzzy multicriteria approach (Scryer)

qualitative assessment – data entering

qualitative assessment – data entering

qualitative indicator X on a 1-9 scale

uncertainty indicator U on a 1-5 scale

qualitative assessment – coding procedure

1 5 9

No impactStrong negative impact

Strong positive impact

1 5

Very good informationVery low or no uncertainty

No informationHigh uncertainty

case study – qualitative assessment

case study – feasibility filter

Is quantitative evaluation needed / feasible?

Step 2 - Feasibility filter

Step 3: quantitative assessment

For each policy option, insert:• Estimated impact• Standard error of estimate

user weight

The scoring system tends to privilege impacts with high probability of occurrence and high level of information certainty.

One prominent impact on the benefit side & several important impacts on the cost side

case study – final ranking

Fuzziness in Scryer

What’s the impact of a specific regulation on public healt?• Negative and weak?• Neutral?- Positive and weak?- Strong and positive?

- The qualitative evaluation may belong to a single statement or to several ones with different degree of membership

- It depends on uncertainty- Qualitative fuzzy evaluations may be aggregated with

quantitative statistical (probabilistic, model-based) evaluations

The starting impact matrix for fuzzy multi-criteria calculations

dimension 14*(2n) X 14n matrix

whose elements xij : ordinal values (between 1 and 9) which measure the impact of policy j for the impact i

U a 14n matrix whose elements uij : corresponding uncertainty assessments (values between 1 and 5)

SUMMARY TABLEType of

assessmentImpact

relevance User weightX(Y) U(E) X(Y) U(E) X(Y) U(E)

PUBLIC HEALTH Quantitative 10 0.133 45.0 4.0 56.0 5.0 44.0 7.0FIRM COMPETITION Quantitative 3 0.040 56.0 67.0 24.0 6.0 499.0 6.0CONDUCT OF BUSINESSES/SMEs Qualitative 5 0.067 4.5 2.3 2.8 3.0 2.5 3.0ADMINISTRATIVE BURDENS ON BUSINESSES Qualitative 5 0.067 5.0 5.0 1.0 4.0 1.0 4.0PUBLIC AUTHORITIES Qualitative 7 0.093 5.0 3.5 1.0 3.5 1.0 3.5INNOVATION AND RESEARCH Qualitative 5 0.067 5.0 3.3 7.0 3.3 7.7 3.3CONSUMERS Qualitative 6 0.080 5.0 3.0 5.0 3.0 5.0 3.0INTERNATIONAL TRADE Qualitative 3 0.040 5.0 3.2 5.5 2.7 5.5 2.7MACROECONOMIC ENVIRONMENT Qualitative 6 0.080 5.0 3.0 5.0 3.0 5.0 3.0LABOUR MARKETS Qualitative 7 0.093 5.0 3.0 5.0 3.0 5.0 3.0ENVIRONMENT Qualitative 6 0.080 5.0 3.0 4.0 2.0 3.0 3.0DISTRIBUTIVE EFFECTS negative Qualitative 4 0.053 5.0 4.0 4.0 4.0 4.0 4.0DISTRIBUTIVE EFFECTS positive Qualitative 5 0.067 5.0 4.0 6.0 4.0 6.0 4.0SOCIAL SENSITIVITY Qualitative 3 0.040 5.0 4.0 5.0 4.0 5.0 4.0

Policy option 1 Policy option 2 Policy option 3

policy j (j: 1,…,n) impact category i (i: 1,…,14)

Steps

1) Transform qualitative variables into Gaussian fuzzy sets

2) Compute distances between pairs of policy options for each specific impact category (distance between two fuzzy sets or stochastic variables)

3) Produce a pairwise comparison between policy options based on the above distances and the weights assigned to impact categories

4) Rank the policies based on their performance in pairwise comparisons

Gaussian fuzzy sets

2

2

( )( ) exp

2k

ijijS

k

k xx q

If element xij is a qualitative score, it needs to be transformed into a fuzzy set

Gaussian fuzzy sets Fuzzy sets defined through a membership function for each of its elements A degree of membership is needed for each of the 9 values of X Fuzzy set Sk where k: 1,…,9 are the potential values that xij may assume q actual assessment of xij, where q is a single value between 1 and 9 The membership function is defined as follows:

K centre of the fuzzy set Sk

k width of the fuzzy set Sk (i.e. a measure of dispersion around the centre) The Gaussian membership functions return a value between 0 and 1, where

when q=k

The «variance» (uncertainty)

function of the centre k of each fuzzy set and of the stated uncertainty level u

assumption dispersion is larger for assessments around 5 and for smaller values of uij

standard deviation for a continuous uniform distribution ranging from 1 to 9 is 2.58 we adopt this value as the maximum variability level

with k=1,…,9

Example

xij=3 score for a given impact uij=4 level of uncertainty The membership function is computed for all sets Sk with k ranging

from 1 to 9, considering the relative dispersion value k. Consider the first fuzzy set S1, for which k=1

1

2

2

(1 3)( 3) exp 0.04

2 0.774ijSx

Distance between two fuzzy sets

o xi1 qualitative impact of the first policy for the i-th category of impact

o xi2 impact of the second policy for same category

o The comparison depends on two fuzzy sets S(xi1=q) and S(xi2=h)

1)Rescale the membership functions through a constant c so that their integral equals to 1, for example, for S(xi1=q)

2) Compute the distance

9 9

1 1 11 1

( ) 1 ( ) 1 ( )k k ki i iS S Skk k

c x q c x q c x q

9 9

1 1 2 2 1 1 2 21 1

( ) ( ) ( ) ( )MAX MAX

l m l mMIN MIN

l m

i i i iS S S Sl ml m

D l m c x q c x h dldm l m c x q c x h

weighted average of all potential distances between the linguistic values, weighted by their

membership functions

Distance - quantitative

when the impact is quantitative distance between two impacts assuming a normal

distribution and exploiting the Hellinger distance

s1 and s2 standard errors of the estimated impacts xi1 and xi2

Pairwise comparison (by impact)

Credibility values are computed for a set of preference relations between 2 options for each impact category2 policy options P1 and P2 6 statements:

• P1 is much better than P2 (according to criterion i)• P1 is better than P2

• P1 is more or less like P2

• P1 is identical to P2

• P1 is worse than P2

• P1 is much worse than P2

range between 0 (not credible at all) and 1 (maximum confidence)

Computation of credibility values (1)

elements needed

(a) semantic distances (also considering the “sign” of the relationship);

(b) cross-over values • parameter which

indicates the distance for which credibility is set at 0.5 (i.e. the confidence that the statement is credible equals the confidence that it is not credible)

• must be fixed (or left to the user)

2, 1 2

2

0 if

1 if

( , ) 2 11

i

x y

x yc P P

D

, 1 2 2

2

0 if

1 if ( , )

1i

x y

x yc P P

D

ln 2

, 1 2( , )D

ic P P e

Computation of credibility values (2)

22

ln 2

, 1 2( , )D

ic P P e

, 1 2 2

2

0 if

1 if ( , )

1i

x y

x yc P P

D

2, 1 2

2

0 if

1 if

( , ) 2 11

i

x y

x yc P P

D

Pairwise comparison (across impacts – aggregation)

wi [0,1] (with i:1,…,c) weights assigned to each criterion

Aggregate preference intensity index for each of the 6 preference statements

entropy

29

*, 1 2

*, 1 2*, 1 2 *, 1 2

0 if ( , ) ( , )

( , ) if ( , )i

ii i

c P PP P

c P P c P P

* 1 2 *, 1 2 *, 1 2 *, 1 2 *, 1 21

1( , ) [ ( , ) ( , ) (1 ( , )) ( , )]

cA B

i i i ii

H P P P P P P P P P Pc

*,

*, 1 22 *,

0 if 0( , )

log otherwise iA

ii

P P

*,

*, 1 22 *,

0 if 1( , )

log (1 ) otherwise iB

ii

P P

preference intensity indices may hide very heterogenous situations, in terms of consistency across the credibility indices for the various criteria entropy measure, to ‘weigh’ the preference intensity indices in the final policy ranking step

Adjusted membership function for each policy comparison, considering a threshold to rule out very small preference intensities:

increases as the basic credibility values concentrate around 0.5 (i.e. uncertainty)

tends to 0 when most of the basic credibility values are 0 or 1 (i.e. certainty)

extremes : • H=0 when all basic credibility values are 0 or 1,

•H=1 when all basic credibility values are 0.5

Ranking of policy optionsThe final step consists in producing two overall indices to rank the various policy alternatives. These two indices + and - aggregate – respectively – the pairwise indices B and W for a given policy option I relative to all policy alternatives.

( , ) [1 ( , )] ( , ) [1 ( , )]

( )2 ( , ) ( , )

p

i j i j i j i jj i

i p

i j i jj i

P P H P P P P H P P

PH P P H P P

( , ) [1 ( , )] ( , ) [1 ( , )]

( )2 ( , ) ( , )

p

i j i j i j i jj i

i p

i j i jj i

P P H P P P P H P P

PH P P H P P

All policy alternatives can now be ranked according to + and -, which are included between 0 and 1. They can be interpreted as a degree of membership to the statements that “Policy alternative i is the best policy option” and “Policy alternative j is the worst policy option”.

entropy can be considered distance between two impacts assuming a normal

distribution and exploiting the Hellinger distance

without entropy omit terms in square brackets and uss 2(p-1) as denominator.

For each policy option, the equations aggregate the much better (much worse) and better (worse) preference intensity indexes, to generate an aggregate preference index for the best/worst policy option.

degree of membership to the statements that ‘Policy alternative i is the best policy option’ and ‘Policy alternative j is the worst policy option’

range between 0 and 1

Multi-Criteria Analysis vs. Cost-Benefit Analysis

MCA CBA

more comprehensive approach less comprehensive approach (only monetary values)

based on experts’ preferences (subjectivity)

measures individual preferences (objectivity), even though biased by income

objectives and criteria are more clearly stated

objectives and criteria are often implicitly assumed

has not a rigorous approach to include time discounting

has a rigorous approach to include time discounting (but difficult to choose appropriate discount factor)

distributional impacts are more clearly considered

distributional impacts are less clearly considered

Final considerations

There is no optimal procedure:• Scale of measurement of impacts• Decision aim

References

Figueira, J., Greco, S., and Ehrgott, M., 2005. Multiple criteria decision analysis: state of the art surveys. International Series in Operations Research and Management Science. Springer

Munda, G., Nijkamp, P., and Rietveld, P., 1992. Comparison of fuzzy sets: A new semantic distance. Serie Research Memoranda. Free University, Amsterdam

-----, 1995. Qualitative multicriteria methods for fuzzy evaluation problems: An illustration of economic-ecological evaluation. European Journal of Operational Research 82, 79-97

Ragona, M., Mazzocchi, M., Zanoli, A., Alldrick, A.J., Solfrizzo, M., and van Egmond, H.P. (2011). Testing a toolbox for impact assessment of food safety regulations: Maximum levels for T-2 and HT-2 toxins in the EU. Quality Assurance and Safety of Crops & Foods, 3(1):12-23

Zadeh, L.A., 1965. Fuzzy sets. Information and Control 8, 338-353