Taking seriously the political dimension of

Policy evaluation

Taking seriously the political dimension of

Policy evaluation

Vincent SpenlehauerÉcole nationale des ponts et chaussées /

IFRISSéminaire ASIRPA 13 juin 2012 Inra Paris

IntroductionIntroduction• Policy evaluation stems from the encounter of 1) the

policy-something movement in US social sciences and 2) the multiplication of federal public interventions in the 1960’s

• The history of PE is marked by great hopes, few practical successes and plenty of practical failures

• One main reason of this disappointing balance-sheet is knowledge/power clutching is not an easy and a normal game to play for social scientists and that most of them usually prefer the Leon Fetzinger’s solution (reduction of cognitive dissonance, 1957, 1993) to the Chris Argyris’ solution (double-loop learning, 1978)

• My message here is simple : sorry my friends PE presents a political dimension and you’d better cope with it !

Knowledge/Power Clutching (I)

Knowledge/Power Clutching (I) • A common definition of policy evaluation (D1) : “an official

process launched by public authorities and by which sound knowledge is produced about the evaluated policy’s general quality, implementation scheme, goal attainment, costs, outputs and outcomes”

• To be really complete one should add (A2): « in order to ameliorate the policy under evaluation »

• Obviously, this poses the question of Knowledge/Power Clutching, because PE constitutes one aspect of policy-making which is one aspect of political action

• But why D1 more frequent than D1+A2 ? H1 : Enlightenment ideology or faith saying Good Knowledge will necessarily have positive impact. H2 : in big modern democratic societies politics and knowledge production are professionalized, hence quite separated

• As a result the history of policy evaluation practice is paved with cases of non-influential evaluations. Because their actors globally decided to comply with D1 and ignore A2. • A good and politically useful policy evaluation A good and politically useful policy evaluation

is necessarily hybridis necessarily hybrid

Knowledge/Power Clutching (II)Illustration from Road Safety

evaluation

Knowledge/Power Clutching (II)Illustration from Road Safety

evaluation• Context: Prime Minister service, Transport Minister,

Road Safety agency director… (Spenlehauer 2011)

Policy folds Evaluanda envisaged

Evaluanda chosen

Evaluanda evaluated

vehicles Car, truck, motorcycle, bicycle

roads Urban, rural

Govern’tal campaigns

Gore, informative, TV, radio, press

education Schools, driving schools

Involvement and empowerment

Infra-n’al tiers of gvt, corporate world, associations

Law enforcement Speed, alcohol, safety belt, helmet

Speed, alcohol, safety belt, helmet

Mostly speed

Knowledge/Power Clutching (III)evaluanda setting

Knowledge/Power Clutching (III)evaluanda setting

• Complex evaluanda = dispersion of evaluative energy = weakening of evaluation impact on policy

• Thinking pol. Rate of return = eval. Impact/difficulty• The hybrid exercise of good evaluanda setting• Policy analysis as a main preliminary step. The evaluanda

is not a given object. Beyond measurement methodology : what could and should be measured, and related to what values, objectives, norms. Political considerations.

• The inescapable political game to play, even for professional scientists (if they want to have a rationalization impact)

• Choose

• Rather than P KP P KP P

P KP P

The knowledge production issueThe knowledge production issue• Evaluative KP = capturing, probing, gathering,

assembling information about the evaluanda to answer in a convincing manner three basic questions : how (much) the evaluanda works, why does it work like this, is it possible to improve the way it works ?

• As a professional group academic social scientists can (should) play a crucial role because of their wonderful tool box and their institutionalized freedom of thought and speech > super-skill to design KP global scheme BUT

• The evaluanda’s stakeholders “own” a large part of info needed, don’t share it easily, in as much informal and vernacular knowledge can be very relevant for the PE

• A good KP system is then multi-actors. Its construction and management shows also a political dimension: use of authority, alliance building, bargaining (participation against info).

• A balance must be found between 1) the access to, reliability and pertinence of the info processed and 2) the scientific robustness of the process

Combining earth, wind and fire !

Combining earth, wind and fire !

• Traditional input/output & outcome attributive measurement (renaissance with evidence-based policy and randomized experimental design) (Campbell & Julian 1966 ; Pawson 2006)

• Utilization-focused evaluation (the governance of the evaluation relies on an assembly of good-will stakeholders who transform the policy on the basis of the evaluation (Patton 1978)

• Theory-driven evaluation recommending focus on the opening of the evaluanda “black-box” to understand the action theories behind (what policy actors have in mind?) (Chen & Rossi 1989 ; Stame 2004)

• The question is…• Instead of doing this why don’t we do

this ?!?

BiblioBiblio• Argyris Ch. 1993, Knowledge for Action. A guide to overcoming

barriers to organizational change, Jossey Bass• Baumgartner F. & Jones B. 2005, The Politics of Attention. How

Government Prioritizes Problems, University of Chicago Press• Campbell, D.T. , and Julian C.S. 1966. Experimental and quasi-

experimental designs for research. Rand McNally.• Chen H., Rossi P., 1989, “Issues in the Theory-Driven Perspective”,

Evaluation and Program Planning, (12)4:299-306• Festinger L. 1957. A theory of cognitive dissonance. Row, Peterson • Lindblom C.E. 1958. 'Policy Analysis'. American Economic Review

48(3):298–312 • Lowi T 1964 "American Business, Public Policy, Case-Studies, and

Political Theory", World Politics, 16(4):677-715 • Palumbo D. Eds 1987, The Politics of Program Evaluation, Sage• Patton M.Q.1978, Utilization-Focused Evaluation, Sage Publications• Pawson R. 2006, Evidence-Based Policy. A Realist Perspective, Sage • Spenlehauer V. 2011, Des sciences sociales engagées : l’évaluation

des politiques publiques, (mémoire d’HDR consultable sur demande)

• Spenlehauer V.1995, L’évaluation de politique, usages sociaux. Trois études de cas d’évaluation, L’Harmattan

• Stame N. 2004, “Theory-Based Evaluation and Types of Complexity”, Evaluation (10)1:58-76