Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental...
Transcript of Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental...
![Page 1: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/1.jpg)
Program theory and logic models
for systemic evaluation
International Conference on Systemic Approaches in Evaluation
Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ)
Eschborn, Germany 25-26 January 2011
Patricia Rogers [email protected]
![Page 2: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/2.jpg)
2
Overview
• Presentation:
– Simple, complicated and complex systems
– 7 aspects to consider for evaluation
• Examples
– How these aspects have been addressed in some recent examples of program theory
• Discussion
What recommendations of practical relevance can be made?
– How to develop it further?
– How to create the preconditions for systemic evaluations (if it is considered desirable)
Recommendations for
– Evaluators
– Managers of evaluations
– Donors
![Page 3: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/3.jpg)
3
Sources for this presentation
![Page 4: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/4.jpg)
4Funnell and Rogers Purposeful Program Theory 4
Program theory An explicit theory of how an intervention contributes to the
intended or observed outcomes, which has 2 components:
Theory of
change
The process by which change comes about (for an
individual, organization or community)
Theory of action How the intervention is constructed to activate the theory of
change
Logic model A visual representation of a program theory, usually in a
diagram but sometimes in a table
Program theory
evaluation
An evaluation that is at least partly guided by an explicit
program theory. It is not necessarily „driven‟ by the theory,
since it should be driven by its intended purpose and the
needs of its intended users
Some definitions
![Page 5: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/5.jpg)
5Funnell and Rogers Purposeful Program Theory 5
What people sometimes assume you mean by logic models that
address complexity
![Page 6: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/6.jpg)
66
Two framings of simple, complicated and complex
Glouberman and Zimmerman 2002 Kurtz and Snowden 2003
Simple Tested „recipes‟ assure
replicability
Expertise is not needed
The domain of the „known‟,
Cause and effect are well
understood,
Best practices can be confidently
recommended,
Complicated Success requires high level of
expertise in many specialized
fields + coordination
The domain of the „knowable‟
Expert knowledge is required,
Complex Every situation is unique –
previous success does not
guarantee success
Expertise can help but is not
sufficient; relationships are key
The domain of the „unknowable‟,
Patterns are only evident in
retrospect.
Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like?
Ottawa: Commission on the Future of Health Care in Canada, 2002. http://www.healthandeverything.org/fi les/Glouberman_E.pdf.
Kurtz, C. F. and D. J. Snowden (2003) „The New Dynamics of Strategy: Sense-making in a Complex and Complicated World‟, IBM
Systems Journal 42(3): 462–83. ( who also discuss chaotic and disordered)
![Page 7: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/7.jpg)
7
Different types of systems
A simple system
A complicated system
A complex system
1. http://commons.wikimedia.org/wiki/File:Ford_assembly_line_-_1913.jpg 2 .Heath Robinson 3. Oriolus
http://www.flickr.com/photos/28556257@N00/441814998/
![Page 8: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/8.jpg)
8
What constitutes a systemic approach to evaluation?
Attention to:
• Inter-relationships
• Perspectives
• Boundaries
Williams B. Imam I. (2007) (eds) Systems Concepts in Evaluation - An Expert Anthology
EdgePress/AEA Point Reyes CA.
Williams, B and Hummelbrunner, R (2010) Systems Concepts In Action: A Practitioner's Toolkit„
Stanford: Stanford University Press.
![Page 9: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/9.jpg)
9
Funnell, S.C. and Rogers, P.J. (2011) Purposeful Program Theory. San Francisco: Jossey-Bass/Wiley.
9
Aspects of complicated and complex situations and interventions with
potentially important implications for evaluation
1) Focus
2) Governance
3) Consistency
4) Necessariness
5) Sufficiency
6) Change trajectory
7) Unintended outcomes
![Page 10: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/10.jpg)
1010
(1) Focus
Simple Single set of intended outcomes/impacts
Complicated Different intended outcomes/impacts intended by
different partners/stakeholders
Different intended outcomes/impacts at different levels
Complex Emergent intended outcomes/impacts
![Page 11: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/11.jpg)
1111
Simple focus
Intervention Longer term
outcomes
Shorter term
outcomes
Intervention that produces single set of outcomes
![Page 12: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/12.jpg)
1212
Complicated focus (1)
Intervention that produces different outcomes valued by different
stakeholders
INTERVENTION SHORT-TERM
OUTCOMES
LONG-TERM OUTCOMES
DIFFERENT LONG-TERM OUTCOMES
![Page 13: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/13.jpg)
1313
Intervention Longer term
outcomes Shorter term outcomes at
system level
Activities at
system level
Activities at site
level
Activities at
client level
Shorter term outcomes at
site level
Shorter term outcomes at
client level
Complicated focus (2)
Intervention that produces different outcomes at different levels
![Page 14: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/14.jpg)
14Funnell and Rogers Purposeful Program Theory 14
Complex focus (1)
Intervention that produces emergent intermediate outcomes
![Page 15: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/15.jpg)
15Funnell and Rogers Purposeful Program Theory 15
Complex focus (2)
Intervention that produces emergent long-term outcomes
![Page 16: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/16.jpg)
16Funnell and Rogers Purposeful Program Theory 16
(2) Governance
Simple Single organization
Complicated Specific organizations with formalized requirements
Complex Emergent organizations working together in flexible
ways
![Page 17: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/17.jpg)
17
(3) Consistency
Simple Implement what has been identified as „best practice‟ or
„evidence-based practice‟ – what works
Complicated Classify the situation and implement what has been
adapted for that context – what works for whom in what
situation
Complex Ongoing adaptation to emerging conditions – what is
working here
![Page 18: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/18.jpg)
18
Complex consistency
Adaptive, responsive intervention
SHORT-TERM OUTCOMES
LONG-TERM OUTCOMES
INTERVENTION
![Page 19: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/19.jpg)
19Funnell and Rogers Purposeful Program Theory 19
(4) Necessariness
Simple Only way to achieve the intended impacts
Complicated One of several ways to achieve the intended impacts – which can be
identified in advance
Complex One of several ways to achieve the intended impacts – which are only
evident in retrospect
![Page 20: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/20.jpg)
20Funnell and Rogers Purposeful Program Theory 20
(5) Sufficiency
Simple Sufficient to produce the intended impacts. Works the same for
everyone
Complicated Only works in conjunction with other interventions (previously,
concurrently, or subsequently) and/or only works for some people
and/or only works in some circumstances – which can be identified
in advance
Complex Only works in conjunction with other interventions (previously,
concurrently, or subsequently) and/or only works for some people
and/or only works in some circumstances – which is only evident in
retrospect
Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)
![Page 21: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/21.jpg)
21Funnell and Rogers Purposeful Program Theory 21
Ways in which an intervention can work with other interventions
Stronger Families and Communities Strategy evaluation 2000-2004 Final Reporthttp://www.rmit.edu.au/casr/sfcse
![Page 22: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/22.jpg)
22Funnell and Rogers Purposeful Program Theory 22
Complicated necessariness (1)
Intervention
Longer term
outcomes
Shorter term
outcomes
A different
intervention
Multi-stage intervention (eg Outcome Mapping)
![Page 23: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/23.jpg)
23Funnell and Rogers Purposeful Program Theory 23
Complicated necessariness (2)
INTERVENTION SHORT-TERM
OUTCOMES
OUTCOMES/ IMPACTS
ANOTHER INTERVENTION
SHORT-TERM OUTCOMES
ANOTHER INTERVENTION
SHORT-TERM OUTCOMES
Intervention that works in combination with other interventions
![Page 24: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/24.jpg)
24Funnell and Rogers Purposeful Program Theory 24
Complicated necessariness (3)
INTERVENTION SHORT-TERM
OUTCOMES
PARTICULAR PARTICIPANT CHARACTERISTICS
LONG-TERM OUTCOMES
INTERVENTION
DIFFERENT PARTICIPANT CHARACTERISTICS
DIFFERENT SHORT-TERM OUTCOMES
DIFFERENT LONG-TERM OUTCOMES
Intervention that works differently for different types of participants
![Page 25: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/25.jpg)
2525
(6) Change trajectory
Simple Constant, linear relationship between effort and results (eg
twice the investment produces twice the results)
Complicated Well understood but not linear relationship between effort and
results (eg curvilinear dose-response relationship such as
diminishing returns or too much of a good thing)
Complex Emergent relationship between effort and results (eg unknown
tipping points)
![Page 26: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/26.jpg)
26Funnell and Rogers Purposeful Program Theory 26
(7) Unintended outcomes
Simple Unintended outcomes can be anticipated and monitored
Complicated Different unintended outcomes are likely in particular
combinations of circumstances – expertise is needed to
anticipate them and identify them
Complex Unintended outcomes cannot be anticipated but only
identified (and addressed) as they emerge or in retrospect
Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)
![Page 27: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/27.jpg)
27Funnell and Rogers Purposeful Program Theory 27
Some thoughts on how program theory might address
complicated and complex aspects
Issues that may need to
be addressed
1. Focus
2. Governance
3. Consistency
4. Necessariness
5. Sufficiency
6. Change trajectory
7. Unintended
outcomes
Possible evaluation methods, approaches and
methodologies
• Emergent evaluation design that can
accommodate emergent program objectives and
emergent evaluation issues
• Collaborative evaluation across different
stakeholders and organisations
• Non-experimental approaches to causal
attribution/contribution that don‟t rely on a
standardized „treatment‟
• Realist evaluation that pays attention to the
contexts in which causal mechanisms operate
• Realist synthesis that can integrate diverse
evidence (including credible single case studies)
in different contexts
• „Butterfly nets‟ to catch unanticipated results
![Page 28: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/28.jpg)
28
Examples
Examples are on a separate handout, along with pages for recording
comments in terms of the 7 aspects and broader implications
![Page 29: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/29.jpg)
29
1. VECO Indonesia: Sustainable Agricultural Chain
Development, Deprez and Van Steenkiste (2010)
![Page 30: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/30.jpg)
30
2. New Zealand Department of Labor: Recognised Seasonal
Employer policy (Nunns and Roorda, 2009)
![Page 31: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/31.jpg)
31
3. International Center
for Tropical
Agriculture (CIAT)
Striga programs
(Douthwaite, Kuby,
van de Fliert and
Schulz, 2003)
![Page 32: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/32.jpg)
32
4. Waawiyeya Evaluation Tool, Tending the Fire
program (Johnston, 2010)
![Page 33: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/33.jpg)
33
5. Stronger Families and Communities Strategy (CIRCLE, 2006)
CIRCLE „Third Newsletter. Evaluation of the Stronger
Families and Communities Strategy 2000-2004.
Funnell, S.C., Rogers, P.J. and Scougall, J. „Issues Paper on
Community Capacity Building for the Evaluation of the
SFCS‟. Canberra: Department of Family and Community
Services.
Both available at http://rmit.edu.au/casr/sfcse
![Page 34: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/34.jpg)
34
Discussions
What recommendations of practical relevance can be made?
– How to develop it further?
– How to create the preconditions for systemic evaluations (if it is considered
desirable)
Recommendations for
– Evaluators
– Managers of evaluations
– Donors
![Page 35: Program theory and logic models for systemic evaluation€¦ · Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford](https://reader033.fdocuments.us/reader033/viewer/2022050607/5faebbc7f14108410f53b353/html5/thumbnails/35.jpg)
35
References and further readingFunnell, S. and Rogers, P.J. (2011) Purposeful Program Theory. San Francisco: Jossey-
Bass/Wiley.
Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada, 2002. http://www.healthandeverything.org/fi les/Glouberman_E.pdf.
Kurtz, C. F., and Snowden, D. F. “The New Dynamics of Strategy: Sense-Making in a Complex and Complicated World.” IBM Systems Journal, 2003, 42(3), 462–483.
Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford Press.
Rogers, P. J. “Using Programme Theory for Complicated and Complex Programmes.” Evaluation, 2008, 14 (1), 29–48.
Rogers, P. J., Guijt, I., and Williams, B. “Thinking Systemically: Seeing from Simple to Complex in Impact Evaluation.” Presented at the 3IE/African Evaluation Association Impact Evaluation Conference, Cairo, Egypt, 2009.
Rogers, P.J. and Williams, B. 2010 „‟Using Systems Concepts in Evaluation‟‟ in Beyond Logframe; Using Systems Concepts in Evaluation. Tokyo: FASID http://www.fasid.or.jp/shuppan/hokokusho/pdf/h21-3.pdf
Snowden, D. J., and Boone, M. “A Leader‟s Framework for Decision Making.” Harvard Business Review, Nov. 2007, pp. 69–76.
Williams B. Imam I. (2007) (eds) Systems Concepts in Evaluation - An Expert Anthology EdgePress/AEA Point Reyes CA.
Williams, B and Hummelbrunner, R (2010) Systems Concepts In Action: A Practitioner's Toolkit„ Stanford: Stanford University Press