Evaluation and the

Science of Complexity

Evaluating Complexity Conference

NORAD

29th -30th May 2008

Agenda

Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

The history of M&E in the international development sector is in four distinct phases 1960s to 1979: Early developments

1979-1984: Rapidly growing interest

1984 to 1988: M&E matures

1988 to the present: the crossroads

Wealth of tools, techniques and approaches are now available Logical framework analysis Results-based management Needs Assessments, Impact Assessments Ex ante and ex post assessments ZOPP GANTT Social Network Analysis Appreciative Enquiry Most Significant Change Outcome Mapping Many many more!!

For many organisations, evaluations are at the centre of a vicious circle...

Increased competition

Increased pressure to show results and impact

Lack of professional norms and standards

Poor learning and accountability

Growing need for high profile

fundraising and advocacy work

...causing problems for M&E

Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes

Reflection, learning and analysis are threatened by existing agency cultures and processes

Org. Learning

Existing culture & process

Evaluation

Accountability

Evaluations, like other similar initiatives, often sit on top of existing organisational silos, inefficiencies and power imbalances, rather than resolving them

Silos

Evaluation

KM

Agencies plough the same evaluation “field”, but stick to their own furrows

Understanding of the effectiveness and use of evaluations is weak at best…

Where’s the data??!

Evaluations tend to be based on wish lists, not strategies, and therefore are often overloaded

Leadership and political buy-in to evaluation is rare and unreliable, with two common reactions

...all of which means that (1) evaluation efforts resemble this iceberg...

What is planned

What actually happens

And (2) evaluators spend most of their time feeling like this...

working against this...

Agenda

Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

“Exploring the science of complexity”

Primary aim was to explore the potential value of complexity science for those who work on change and reform initiatives within the aid sector

Drew on scientific and experimental literature – physiology, physics, mathematics, public sector reform, sociology, economics, organisational theory, plus case studies, reports and evaluations from the aid sector

Reviewed over 250 articles, books, reports and evaluations

10 peer reviewers, including 5 directors of leading research institutes

Published February 2008 Available to download from www.odi.org.uk

A man was walking home one dark and foggy night. As he groped his way through the murk he nearly tripped over someone crawling around by a lamp post.

“What are you doing?” asked the traveler.

“I’m looking for my keys” replied the other.

“Are you sure you lost them here?” asked the traveler.

“I’m not sure at all,” came the reply, “but if I haven’t lost them near this lamp I don’t stand a chance of finding them.”

A (well-known) story…

A closer inspection of the light under the lamp revealed…

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

Logical frameworks / results chains articulate a clear world view and theory of change: “The light under the lamp”

The machine metaphor - universe as clockwork

The future is knowable given enough data points

Phenomena can be reduced to simple cause & effect relationships

Dissecting discrete parts would reveal how the whole system worked; science was the search for the search for the basic building blocks

The role of scientists, technologists & leaders was to predict and control - increasing levels of control (over nature, over people, over things) would improve processes, organisations, quality of life, entire human societies

Key Assumptions

Assumptions about systems Ordered Reductionist - parts would reveal the whole

Assumptions about how systems change Linear relationships influence as direct result of force from one object to

another - predictable Simple cause & effect

Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”

...Reality of aid is a little different...

But a new light is being turned on (slowly, unevenly, using a dimmer switch)…

Complexity science is a science of understanding change A loosely bound collection of ideas, principles and

influences from a number of other bodies of knowledge, including chaos theory fractal geometry cybernetics complex adaptive systems postmodernism systems thinking

Discovery of similar patterns, processes and relationships in a wide variety of phenomena related to the nature and dynamics of change

From microscopic chemical reactions…

...to the evolution of galaxies...

Complexity scientists use a range of ideas and concepts (familiar, challenging and baffling) to make distinctions between simple, complicated and complex systems and phenomena

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical

and necessary

Sending one rocket increases assurance that next will be ok

High level of expertise in many specialized fields + coordination

Separate into parts and then coordinate

Rockets similar in critical ways

High degree of certainty of outcome

Formulae have only a limited application

Raising one child gives no assurance of success with the next

Expertise can help but is not sufficient; relationships are key

Can’t separate parts from the whole

Every child is unique

Uncertainty of outcome remains

Complicated ComplexSimple

The recipe is essential Recipes are tested to

assure replicability of later efforts

No particular expertise; knowing how to cook increases success

Recipe notes the quantity and nature of “parts” needed

Recipes produce standard products

Certainty of same results every time

The claims of complexity scientists

The complexity of real world systems is (usually) not recognised or acknowledged by scientists and policy makers

Or, that if it is not acknowledged, they don’t deal with them

Or, that if they do deal with them, they don’t do so effectively

Or, that if they do deal with them effectively, it’s because they used an specific approach / framework

...that is also available to you, dear client, at a reasonable daily rate plus a per diem

[JOKE]

There have been diverse efforts to apply ideas to social, economic and political analysis and practice Arthur, Ormerod - Economics Stacey, Snowden - Organisations Jervis, Urry, Cutler - Intl relations De Mancha - History Gilchrist - Community development Education policy - Sanders and McCabe Health policy - Zimmerman Government reform - Chapman Strategic thinking – Saunders Evaluation - Williams

Complexity and systems approaches have already proved useful in re-thinking aid and development issues Uphoff, 1990s Chambers, 1997 Sellamna, 1999 IDRC, Outcome Mapping, 2001 Warner, 2001 Rihani, 2002 Lansing and Miller, 2003 Inclusive Aid, 2004 ECDPM, 2004-06 Eyben, 2006 Guijt, various Davies, Network Analysis, various

`

10 key concepts and implications

8 Adaptive Agents

10 Co-Evolution

4 Non-Linearity

3 Emergence from

SimpleRules

6 Phase space and attractors

5 Sensitivity to initial

conditions

1 Interconnected and interdependent

elements and dimensions

2 Feedback processes

7 Strange

attractors and the edge of

chaos

9 Self organisation

Features of systems

Dynamics of change

Behaviours and relationships

Complex systems

Collection of parts, which collectively have a range of dimensions

Parts share an physical or symbolic environment / space

Action by any part can affect the wholeE.g. individuals, families, communities, cities,

markets, societies, populations, economies, nations, planets

Complex systems are interconnected and interdependent to different degrees Interconnectedness may occur between any elements,

dimensions, systems and environments

This interconnectedness leads to interdependence between the elements and the dimensions of a system, and gives rise to complex behaviour

Complex systems can be tightly or loosely coupled, internally and with their environment, giving rise to different kinds of complex behaviours Tightly coupled: Global FOREX markets Loosely coupled: US University system, global construction industry

Systems can be understood via mapping techniques, followed by analysis to understand the dynamics and interactions of the system

Foot and mouth disease: an example of failure caused by focusing on one part of the system and ignoring the links between sub-systems (biology, geography, economics)

Economic rationalisation of abattoirs and EU subsidies increased the interconnectedness of herds to a critical point.

Changes to foot and mouth reporting rules delayed the isolation of infectious animal

The relationship between these actions and the epidemiology of F&M was not appreciated in advance where it mattered because the livestock industry was not viewed as a interconnected, interdependent system

Complexity also means that systems need to be understood at different scales

Atom

Molecule

Cell

Tissue

Organs

Organisms

Communities

E.g. evaluating the effectiveness of child health programmes

A.N. NGO

Other NGOs

Private Sector

Community and

Family

Church

Civil Society

DevelopingCountry Govmts

Local partners

Example: evaluating resource flows in the humanitarian system

Issues of interconnectedness, interdependence and scale usually do not become apparent until a crisis... Foot and mouth disease, UK

Economics, cattle management, disease September 11th

Globalisation and terrorism Climate change

Western consumerism and Southern disasters Credit crunch

US mortgage market mis-selling and the world economy Food prices

Biofuels and food consumption Vulnerability to natural disasters

Sichuan earthquakes and dams Human trafficking

Desire, economics and rights abuses

...indicating that we have biases in the way we view the world...

Three different kinds of problems have been identified, along with some common biases in dealing with them

They are:“Messes”“Problems” “Puzzles”

“Messes” are issues that do not have a well defined form or structure. NB not a value statement, but a description There is often not a clear understanding of the problem faced

Messes often involve economic, technological, ethical and political issues It has been suggested that all of the really important issues in the world

start out as messes.

For example, how was rising HIV/AIDS incidence in Brazil dealt with in the 1990s? concerned money, technology, ethics, social relations, politics, gender

relations, poverty all of these dimensions of the problem had to be dealt with

simultaneously, and as a whole

Many of the major problems we face are “messes”! Credit crunch

US mortgage market mis-selling and the world economy Food prices

Biofuels and Vulnerability to natural disasters

Sichuan earthquakes and dams Climate change

Western consumerism and Southern disasters September 11th

Globalisation and terrorism Human trafficking

Sexual preferences and human rights abuses Arms trade

Economics and war Etc, etc, etc

“Problems” are issues that have a known or knowable form or structure

The key dimensions and variables of a problem are known and the interaction of dimensions may also be understood, even if only partially.

With problems, there is no single clear cut way of doing things there are many alternative solutions, depending on the constraints faced Expertise matters

For example, dealing with the sewage system in a particular city may rely on amount of money available, technology, political stance of leaders, climatic conditions, urban development, the road system, and so on

Puzzles are well defined and well structured known problems with a specific “best” solution Solutions can be worked out and improved

Solutions are replicable - “best practices” are possible

Policy and traditional science is biased towards puzzle-solving Real-world, complex, messy nature of

systems is frequently not recognised

Simple puzzle-based solutions are applied to complex messesE.g. Global War on Terror has been applied as

the single best solution to the mess of terrorism

“Some of the greatest mistakes have been made when dealing with a mess, by not seeing its

dimensions in their entirety, carving off a part, and dealing with this part as if it were a problem, and then solving it as if it were a puzzle, all the while ignoring the linkages and connections to

other dimensions of the mess”

Or to put it another way: dividing a cow in half does not give you two smaller cows

Implications: analyse and deal with the reality of the system Multidimensionality, interdependence and interconnectedness of

poverty and humanitarian crises (and responses to them) should be recognised by those designing, managing and evaluating aid interventions

Analysis may need to be in line with historical research - not ‘did x cause y?’ but ‘what happened and why?’, building narratives about events and processes.

The task of selection and synthesis of data becomes as important as analysis

Different perspectives on what the system is need to be taken into account, especially when these perspectives differ as to the nature, interconnectedness and scale of the system

Messes, problems and puzzles need to be identified and dealt with using appropriate approaches

Interconnectedness and interdependence gives rise to a range of phenomena and

behaviours

`

To find out more, read the paper!

8 Adaptive Agents

10 Co-Evolution

4 Non-Linearity

3 Emergence from

SimpleRules

6 Phase space and attractors

5 Sensitivity to initial

conditions

1 Interconnected and interdependent

elements and dimensions

2 Feedback processes

7 Strange

attractors and the edge of

chaos

9 Self organisation

Features of systems

Dynamics of change

Behaviour of agents

Agenda

Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

Evaluation of Complex Systems is NOT new Educational systems, social initiatives and

government interventions are complex social systems where effective evaluation is seen as a key process in measuring success

But there is an increasing recognition that for evaluation of complex social systems to be more effective, evaluations may need to take into account the theoretical understanding of complex systems

Implications for evaluation are at four levels

1. Implications for evaluation methodologies and approaches

2. Implications for the focus of evaluations

3. Implications for the purpose and scope of evaluations

4. Implications for evaluations as a complex system in their own right

Implications for evaluation methodologies and approaches

Linear models dominate...

If you deliver the

product and / or service to the extent

Intended, then your participants

will benefit in certain ways

If these benefits to participants

are achieved, then certain changes in

communities, organisations and systems

might be expected to occur

If you accomplish your planned

activities, then hopefullyyou will deliver the amount of the product

and / or service that

you intended

Certain resources

are needed to operate

your program

If you haveaccess to

them, then you can use

them to achieve your

plannedactivities

Outcomes / Purpose

Impacts / Goal

OutputsInputs Activities

MONITORING EVALUATION

Many evaluation “results chains” visualize change as linear, based on multiple cause-effect logic models

X Y

Linear, Predictable

Focused on the end result

The program (X) gets the credit!

Inputs Activities Outputs Outcomes Impact

When applied to development, “the results chain” can lead to Faulty thinking Misguided data collection Misleading reporting of results Gives a false sense of reality to senior

managers and donors who are far from where change is taking place

Complexity science sees change as…

Interconnected (multiple actors and factors)

Non-linear (unexpected results occur)

Incremental, cumulative, with tipping points

Beyond the control of the project / programme

Two-way (program also changes)

Continuous (not limited to the life of the project)

Therefore

Develop new theories of change, adapt existing theories of change to challenge assumptions of linearity

Design evaluations to allow for interdependencies and interconnections in the system the program is trying to influence, and capture the resulting dynamics

Implications for evaluation focus

Focus of “traditional” evaluations

Formal project / programme / organisation

Environment is outside the organisation evolves separately until programme is implemented

Level of analysis is single or at most a few, relatively independent levels

Implications of complexity Features of systems: evaluate from perspective of

multiple, nested levels of interconnected systems, study feedback between the organisation and its environment, look for emergent rather than planned change

Dynamics and nature of change: Look for non-linearity, anticipate surprises and unexpected outcomes, analyse the system dynamics over time and frame the “space for possible change”, look for changes in conditions that facilitate systemic change, and how well matched the programme is to the wider system

People, motivations and relationships: Study patterns of incentives and interactions among agents, study quality of relationships, study individuals and informal / shadow coalitions, vs. formal organisation, study co-evolution of organisation and environment

Implications for evaluation purpose and scope

M&E is seen as standing in contrast to creative dynamism of field work

Purpose and scope of traditional evaluations vs complexity-oriented evaluations

Traditional Complexity-oriented

Measure success against predetermined goals

Develop new measures and monitoring mechanisms as goals emerge & evolve

Render definitive judgments of success or failure

Provide feedback, generate learning, support direction or affirm changes in direction

Aim to produce generalisable findings across time & space

Aim to produce context-specific understandings that inform ongoing innovation

Creates fear of failure Supports hunger for learning

Implications for evaluation as a complex system

Key Assumptions

Assumptions about systems Ordered Reductionist - parts would reveal the whole

Assumptions about how systems change Linear relationships influence as direct result of force from one object to

another - predictable Simple cause & effect

Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”

Goals ResultsActivities

Monitoring

Evaluation

Evaluations traditionally seen as a rational, technical, information-generating process

...Reality is a little different...

Evaluation systems are themselves complex systems, with many interconnected parts and dimensions; no two evaluations are the same

Focus of evaluation Policy / Guidelines Scope Project vs Policy Demand, goals Timing, quantity Preparation, TOR Management and team selection Methods and tools Engagement with stakeholders Dissemination of findings and utilisation Costs Quality maintenance mechanisms

“...To be effective an evaluation program must match the dynamics of the system to

which it is applied....”

Eoyang and Berkas (1998)

Implications of evaluations as a complex systemTraditional Complexity-oriented

Position the evaluator outside to assure independence and objectivity

Position evaluation as an internal, team function integrated into action and ongoing interpretive processes

Accountability to control and locate blame for failures

Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond strategically

Accountability focused on and directed to external authorities and funders

Accountability centred on fundamental values and commitments

Evaluator controls the evaluation and determines the design based on the evaluator’s perspective about what is important

Evaluator collaborates in the change effort to design a process that matches philosophically and organisationally

All systems can be placed on a spectrum between extremes of ordered and chaotic E.g. solids and gases

In solids, atoms are locked into place In gases they tumble over one another at random

In between the two extremes, at a phase transition, a phenomenon called the ‘edge of chaos’ occurs This phenomenon describes systems behaviours where the evolution of

the system never quite locks into place and never quite dissolve into turbulence either.

In human organisations, the simplest example is of a system that is neither too centrally controlled (order) nor too bottom-up and therefore disorganised (chaos)

Physiology on the Edge of Chaos

Healthy Heart – on the edge of chaos

Severe Congestive Heart Failure – orderly

Cardiac Arrhythmia, Atrial Fibrillation - chaotic

Dynamics

Ad

ap

tab

ility

Ordered Disordered

ZONE of HEALTH

Point of Maximum Adaptability

Threshold Threshold

High

Low

Dynamic adaptability is the key to system health

Changelessness is a sign of death,transformation a sign of life.- Commentary on the I Ching

Evaluation at the

edge of chaos?

Agenda

Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Final points

Complexity science gives additional weight to calls for re-thinking

The wider contexts of aid work The nature of the processes involved in aid work The dynamics of change involved in aid work The real influence of aid work The role of partner organisations and

beneficiaries in aid work The tools and techniques for planning,

monitoring, learning and evaluating aid work

Given the resistance to change in the power dynamics of aid, real world applications may continue to be “innovative”, “under the radar”

and outside the mainstream of aid policy and practice...

There are a number of common criticisms of complexity... Theoretical: adds nothing new

E.g. the ideas of complexity science have all been identified elsewhere But complexity brings them together

Practical: doesn’t specify what should be done E.g makes no specific recommendations as to how best to act in complex systems See next slide

Supports managerial “snake-oil” / “complexologists” / re-warmed ideas E.g. is abused and misused What isn’t?

Political: emergence and self organisation support neo-liberal stances Just because self-organisation happens, doesn’t mean it is good

Credit crisis Rwandan genocide

Complexity is more centre-ground, for example, “edge of chaos” systems are seen as most robust and resilient because are the optimally combination of control and flexibility

...and different perspectives on the value of complexity science Deep paradigmatic insights

Champions

Interesting parallels and useful approaches, but not the only way to see things Pragmatists

Meaningless coincidences Critics

Source of weakness is also the source of strength These ideas are not about “what you dos”, but about

“how you dos” Not “solutions for problems”, but “approaches to problems” Tools for furthering understanding, for opening up new ways of

seeing and thinking

They point to the personal, professional, institutional, political mindsets, attitudes and conditions which need to be in place to work effectively in and with complex systems

Complexity concepts can support the intuition and navigation of practitioners

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

Four suggestions

Develop collective intellectual openness to ask a new, potentially valuable, but challenging set of questions of our mission and their work

Develop collective intellectual and methodological restraint to accept the limitations of a new and potentially valuable set of ideas not misuse or abuse or let them become part of the ever-swinging pendulum of

aid approaches

Need to be humble and honest about the scope of what can be achieved through ‘outsider’ interventions, about the kinds of mistakes that are so often made, and about the reasons why such mistakes are repeated

Need to develop the individual, institutional and political courage to face up to the implications of complexity

...We can't solve problems by using the same kind of thinking

we used when we created them..

Final points (1)

Everybody thinks to change the

world; nobody thinks to

change themselves

Final points (2)

Thank you!

Get in touchb.ramalingam@alnap.org