Me plenary final final
57
Facilitating Systemic Change: Improving Evaluation to Improve Practice Elizabeth Dunn, Impact LLC SEEP Annual Conference November 7, 2012
description
2012 AC Wed Morning Plenary
Transcript of Me plenary final final
Facilitating Systemic Change:Improving Evaluationto Improve Practice
Elizabeth Dunn, Impact LLCSEEP Annual Conference
November 7, 2012
Why We Need Systemic Thinking
• My journey from small farmers to microcredit to inclusive market systems…and back again
• Practitioners, evaluators and donors need systemic thinking to meet new challenges:– Measuring participation under facilitation– Maintaining accountability with flexibility– Evaluating sustainability in evolving systems
Beneficiaries and Boundaries
• Identifying participants– Currently inconsistent
• Understanding spillover– Good for impact– Bad for evaluators?
• Relates to boundaries of system
• 3 types of participants– direct, indirect, imitators Source: Outreach, Outcomes and Sustainability
in Value Chain Projects by Creevey et al., Sept. 2011, USAID AMAP microREPORT #171.
Credibility and Accountability• Issue: Systems change and
projects must adapt• Accountability to donors
– Not going away– But targets can constrain
project effectiveness
• Credibility of evidence– Baselines, control groups
and attribution
• Causal modeling– Useful and essential– Should be flexible
Sustainability as Systemic Change
• Markets as systems– Value chain as network of
firms/actors relationships
• Sustainability as systemic change– New, better relationships– Learning and adaptation– Broadening of benefits
• Sustainability as an emergent property
Evaluation Challenges
1. Adopt consistent and comprehensive measures of participation under facilitation.
2. Agree with donors on ways to demonstrate accountability under flexible interventions.
3. Adapt causal modeling to guide practice and to evaluate evolving systems.
4. Advance knowledge of sustainability through meaningful indicators of systemic change.
Measuring Impacts in Market Systems:
Rethinking the Current Paradigm
Dr Shamim Bodhanya
Academic Leader: Higher Degrees and Research, Graduate School of Business and Leadership, University of KwaZulu-Natal
Chairperson – Institute of Natural Resources
South AfricaTel: +27 31 260 1493
Email: [email protected]://www.linkedin.com/in/shamimbodhanya
Bounded Rationality
“The capacity of the human mind for formulating and solving complex problems is
very small compared with the size of the problem whose solution is required for
objectively, rational behaviour in the real world or even for a reasonable approximation to such
objective rationality.”
(Simon, 1957, p 198)
Systems Thinking
Wicked problems (mess)
• Unbounded• Ill-defined• Multiple, conflicting goals• Goals may also be ill-defined• Multiple perspectives, values
Complex Adaptive System
“A complex adaptive system (CAS) is a system comprised of heterogeneous agents that interact locally with each
other based on local schema, such that the behavior of the system arises as a
result of feedback relationships between the agents, and the system
evolves as the schemata of the agents adapt based on the feedback.”
Bodhanya, 2008
Complex Adaptive System
“A complex adaptive system (CAS) is a system comprised of heterogeneous agents that interact locally with each
other based on local schema, such that the behavior of the system arises as a
result of feedback relationships between the agents, and the system
evolves as the schemata of the agents adapt based on the feedback.”
Bodhanya, 2008
Agents with schemata
Emergence
Self Organisation
Sensitive dependence
Path dependence
HistoryFar from equilibriumFitness Landscapes
Co-evolution
Edge of Chaos
Egality
Persistence
Artefacts
Characteristics of Complex
Adaptive Systems
Immune system
Nervous system
Cities
Regions
Countries
Firms
Industries
Economies
Open source
Markets
EXAMPLES
Paradigm changing
Rethinking the paradigm
• Social systems are complex adaptive systems• Radically challenges our worldviews• Flat earth versus spherical earth• Ability to predict and control• What does this mean for human agency and
volition?• Changes our entire conception of planning• Societal change• Markets This does not in anyway imply that
human actors must be fatalistic.
Arc de Triomphe
Towards a way forward• We are on a journey• Context – Contingent on local conditions• Theoretical frameworks <-> Social Reality• Learning systems• Models –
– Soft models – Narratives / Metaphor
• Language is world constituting• Strategic conversations• Facilitation• What is measurement?• Agency in measurement tools• Artifacts• Generative relationships
Thank you
Simplifying complexity through systems thinking
Panel: Measuring impact in market systems (November 7th 2012)
Richard HummelbrunnerÖAR Regionalberatung
Graz, Austria
Systemic or Systematic?
Systemicfocus on the whole and the parts Three core dimensions: Interrelationships Perspectives Boundaries
Systematicfocus on the parts, step-by-step
= +
Trivial or non-trivial simplification ?
Transformation
Internal state(s)
Input
Input Output ?
Output
Context
Consequences for monitoring
• Regard interventions as social systems Unit of observation: intervention and context Observe relevant contextual factors (scanning) during
implementation, in particular relevant actions of others Look beyond intended routes and effects, avoid tunnel
view, capture broader range of effects (irrespective of intentions)
• Different approach towards deviations from plan Do not per se regard as negative (‘correction reflex’) Do not treat as isolated phenomena, but connect with
intervention logic Information to understand the internal dynamics and self-
organising forces at work within target social system
Linear or ‚circular‘ logic models ?
Inputs Outputs Results
Mechanisms
Context
Needs /ProblemsIssues
Impact
Inputs Outputs Results
Single or multiple logics ?
Measuring Impacts in Market Systems:
Rethinking the Current Paradigm
Dr Shamim Bodhanya
Academic Leader: Higher Degrees and Research, Graduate School of Business and Leadership, University of KwaZulu-Natal
Chairperson – Institute of Natural Resources
South AfricaTel: +27 31 260 1493
Email: [email protected]://www.linkedin.com/in/shamimbodhanya
2nd economyMANY FARMERS•Simple technology•Small markets (mainly family & neighbours)•Unprocessed goods
Communal land tenure
1st economy $$FEW FARMERS•Highly mechanised•Involves agro-processing•Large markets (diverse consumers and products)
entrepreneurscorporates
Slow reform
Freehold tenure
NB. Markets:•Land•Finance•Physical•Labour
Communal ownership L
Land – ownership entity
Keeping land productive
80/20 - 20% of people produce 80% of the food
InfrastructurePrivate sector:
•Primary production•Transport•Processing
Organised agric. /
National governmentPolicy & programmes
DEA, DWA, DAFF
NAFUNERPO
Commodity groups (SASA, FSA, Grain
SA, etc)
Provincial & local
govt
Policy
Farmers
Financial orgs:•Landbank•Private banks•Donors•Microfinance
WATERGLOBAL WARMING Increasing prices:
•Fuel (oil) – supply too•Fertilizer
MAFISA•Landbank•DAFF•DFIs
Cheap importsTariff protection
WTO
Education & training
Research
Issues
• Practitioners have a good sense of real world complexity
• Overly formalistic tools• Attempt to straightjacket that experience into
inappropriate tools• Unintended consequences• Multiple perspectives• Policy resistance• Boundary judgments• Bounded rationality
Systemic change
• Events – Pattern - Structure• Structure• Relationships between actors• Mental models• Incentives• Information flows• Changing feedback loops• Strength of the loops
Considerations
• Systemic M&E is conceptually challenging• ...yet we must stay rooted to practice• Rigour and credibility without overly
mechanistic, formalistic approaches to M&E• Donor centric – How to get accountability
while shifting the centre of gravity to change on the ground and systemic change
Iceberg
Events
Patterns
Structure
Behaviour over time
What are the causes of …?
Event-oriented View of the World
Goals
Situation
Problem Decision Results
Source: Sterman 2001
Actions of Others
Feedback View of the World
Goals
Decisions
Environment
Goals of Other Agents
Source: Sterman 2001
Side Effects
Closed loop thinking
Poor communicatio
n
Poorperformance
Conflict
Role confusion
Nosupport
Lack of resources
Poormanagement
Low morale
Micro-managing
No training
Lack of skills
Laws of the Fifth Discipline (Senge)• Today’s problems come from yesterday’s solutions• The harder you push the harder the system pushes back (policy resistance)• Behaviour grows better before it grows worse• The easy way out usually leads back in• The cure can be worse than the disease• Faster is slower• Cause and effect not closely related in time and space• Small changes can produce big results – but the areas of highest leverage
are often the least obvious• You can have your cake and eat it too – but not at once• Dividing an elephant in half does not produce two small elephants• There is no blame
Characteristics of Complex Adaptive Systems
• Agents with schemata• Emergence• Self-organisation• Sensitive dependence on initial conditions• History • Path dependence• Far-from-equilibrium• Co-evolution• Fitness Landscapes• Edge of chaos• Artifacts• Persistence• Egalitarianism
Simplifying complexity through systems thinking
Panel: Measuring impact in market systems (November 7th 2012)
Richard HummelbrunnerÖAR Regionalberatung
Graz, Austria
• Replace impacts chains with configurations/networks, permitting to Link elements at the same level Connect different levels (e.g. Outputs - Results) Capture reciprocal or feed-back relationships Allocate activities or assumptions with effects Show different strategy options, impact pathways Identify leverage points for interventions
• More refined modelling (if appropriate/useful) E.g. Influence or Multiple Cause Diagrammes (all or partly) Causal Loop Diagrammes (identify Feedback Loops) Represent qualitative features (z.B. intensity, duration, delays)
Consequences of a systemic view of effects
Example of outcome configuration(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
LEISTUNG
LEISTUNG
Outcome
Outcome
Outcome
Example of outcome configuration(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome Outcome
Objective
Outcome
Outcome
Outcome
Outcome
Example of outcome configuration(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Objective
Outcome
Outcome
Outcome
INSTRUMENTS
OUTPUTS
ACTIVITIES
INSTRUMENTS
OUTPUTS
ACTIVITIES
INSTRUMENTS
OUTPUTS
ACTIVITIES
INSTRUMENTS
OUTPUTS
ACTIVITIES OUTPUTS
ACTIVITIES
Challenges for capturing effects in complicated and complex situations
• Main challenges Contribution to objectives through many factors / outputs Difficult to establish clear causalities and relationships
between outputs and further effects (results, impacts) Temptation to attribute effects irrespective of contribution Impact not appropriate to hold program actors accountable
• Limited utility of many monitoring systems Predominant focus on inputs or outputs (easy to capture) Predominant use of quantitative indicators (easy to
measure, capture only narrow part of reality) Information on result / impact indicators comes often (too)
late to change course during implementation
Lines of influence in a program(example: EU Structural Fund - Programs)
High
Low
INPUTS OUTPUTS RESULTS IMPACTS
Project owners, partners, External factors
Programme Actors
Funding conditions
Project development
Project selection
Observe implementation of projects
Influence actions of others
Logical Framework RBMOutcome Mapping
The ‚Process Monitoring of Impacts‘ approach
Theory-based monitoring approach making use of logic models Focus on processes, which should lead to results / impacts Logic models are considered as hypotheses (to be modified
during implementation), not as ‘blue-prints Perspectives and observations of various stakeholders are
captured and reflected, applying use of systemic methods Core rationale of the approach
Provide information for programme actors as early as possible on the likeliness of achieving results/impacts
Particular emphasis on domains that can be influenced by them or for which they are responsible.
Monitoring of change processes
Basic assumptions for change (along a result chain)
Inputs are used to produce outputs
(= projects)
Outputs are used (by someone, in a specific manner) to reach results
Results will lead to (expected) impacts
in a plausible manner
Inputs Outputs Results Impacts
External expertise,Process consulting
Services to sensitize for innovations
Enterprises (esp. SME) collaborate in networks (also with large enterprises)
Enterprises (esp. SME) carry out reorganisation processes
Enterprises (esp. SME) carry out product finding processes
Enterprises (esp. SME) introduce new technologies
Enterprises (esp. SME) gain new markets
Enterprises (esp. SME) conceive innovation-/ investment projects
Increased linkages between enterprises at regional level/ scale
Sustainable stabilisation of enterprises
New/ improved services, products and production processes
Adaption to international competition
Increase in employment/ new jobs
New contacts with clients, new orders
External expertise,Process consulting
Services to sensitize for innovations
Enterprises (esp. SME) carry out reorganisation processes
Enterprises (esp. SME) carry out product finding processes
Enterprises (esp. SME) introduce new technologies
Enterprises (esp. SME) gain new markets
Enterprises (esp. SME) prepare innovation-/ investment projects
Sustainable stabilisation of enterprises
New/ improved services, products and production processes
Adaption to international competition
Increase in employment/ new jobs
New contacts with clients, new orders
Advise for co-operation
I
I
I
OUTPUT (TYPES) USE of OUTPUTS RESULT IMPACT
I Quantifiable indicator
Example of Logic model (Enterprise support Scheme)
Result Based Management (RBM)Emerging tendencies
• Shift focus of performance information from outputs (goods and services produced) to outcomes (benefits)
• Set performance expectations for outcomes Clarify conceptional issues (function, purpose and location of targets)
• Different approach to accountability Influencing outcomes (not achieving them)
• Assess contributions to outcomes (instead of claiming attribution) Take account of other contributing factors, gestation period of outputs
• Beware of straightforward links between performance and budgets/costs
Managing for outcomes requires authority for managers to do so,
i.e. more flexibility on activities, resources and outputs
Adapting Logframes to deal with complexity:Differentiate effects in line with situation
• Categorize outputs via portfolio matrix: locate in one of the three domains: simple, complicated, complexhas implications for completing other elements of logframe
• If outputs predominantly lie in the ‘complicated’ domain: carefully identify indicators and assumptions to enable
monitoring of unfolding practice, relevant factors and context conditions
• If many (or majority of) outputs are considered to be ‘complex: Identify indicators that allow documenting initial conditions and -
in combination with assumptions - capturing emerging phenomena
Not all interventions (or parts thereof) are treated as ‘simple’!
Output Portfolio
C O M P L E X
C O M P L I C A T E D
S I M P L E
DEGREE OF CERTAINTY ( About what to do)
DE
GR
EE
OF
AG
RE
EM
EN
T (
Bet
wee
n S
take
hold
ers)
High agreement Some disagreement High disagreement
Hig
h ag
reem
ent
Som
e di
sagr
eem
ent
Hig
h di
sagr
eem
ent
Types of indicators
Output
Type ofIndicators
Lagging
Coincident
Leading
Time
Example
Services delivered by Centre
Service Centre established
Building permits obtained, Work contracted
