Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain...
Transcript of Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain...
![Page 1: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/1.jpg)
www.3ieimpact.org Lindsey Novak
Theory of Change
International Initiative for Impact Evaluation
Lindsey Novak International Initiative for Impact
Evaluation (3ie)
![Page 2: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/2.jpg)
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
![Page 3: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/3.jpg)
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
vs the “Black box” approach
![Page 4: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/4.jpg)
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
vs the “Black box” approach
Ultimate goal: Answer the “why?” question
![Page 5: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/5.jpg)
www.3ieimpact.org Lindsey Novak
1. What is the program?
Questions
![Page 6: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/6.jpg)
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve?
Questions
![Page 7: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/7.jpg)
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve? 3. What intermediate steps lead to those
outcomes?
Questions
![Page 8: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/8.jpg)
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve? 3. What intermediate steps lead to those
outcomes? 4. What assumptions are associated with
each link in the causal chain?
Questions
![Page 9: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/9.jpg)
www.3ieimpact.org Lindsey Novak
1. What is the program?
Campaign for Clean Water
School-Based Malaria Intervention
![Page 10: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/10.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Malaria Intervention
![Page 11: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/11.jpg)
www.3ieimpact.org Lindsey Novak
2. What outcomes does the program aim to achieve?
School-Based Malaria Intervention
Village of Love
![Page 12: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/12.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Malaria Intervention
I N C R E A S E D K N O W L E D G E
![Page 13: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/13.jpg)
www.3ieimpact.org Lindsey Novak
3. What intermediate steps lead to those outcomes?
School-Based Malaria Intervention
![Page 14: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/14.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
I N C R E A S E D K N O W L E D G E
![Page 15: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/15.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
I N C R E A S E D K N O W L E D G E
![Page 16: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/16.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
I N C R E A S E D K N O W L E D G E
![Page 17: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/17.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
![Page 18: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/18.jpg)
www.3ieimpact.org Lindsey Novak
1. What outcomes do you want to achieve? 2. What program will induce those
outcomes? 3. What pathways theoretically lead to those
outcomes? 4. What assumptions are associated with
each link in the causal chain?
School-Based Malaria Intervention
![Page 19: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/19.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Children adhere to full med regimen
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Not considered substitution for
preventative measures
![Page 20: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/20.jpg)
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
![Page 21: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/21.jpg)
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis
Data Collection
![Page 22: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/22.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Data Collection
![Page 23: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/23.jpg)
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis – Factual analysis
Data Collection
![Page 24: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/24.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Data Collection
![Page 25: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/25.jpg)
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis – Factual analysis
– Some outcomes and assumptions cannot be measured or tested
Data Collection
![Page 26: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/26.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Higher Test
Scores
Test accurately measures knowledge
Medicine is effective
Students correctly targeted
- RDT devices accurate, Teachers
categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Binding constraint is addressed
Child labor doesn’t increase
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Data Collection
![Page 27: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/27.jpg)
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program
![Page 28: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/28.jpg)
www.3ieimpact.org Lindsey Novak
Campaign Against Child Marriage in Bihar & Jharkhand Breakthrough & Catalyst Management Services
![Page 29: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/29.jpg)
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered
![Page 30: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/30.jpg)
www.3ieimpact.org Lindsey Novak
I N C R E A S E D K N O W L E D G E
Better Labor Market
Outcomes
Healthier & Smarter
Children Next Generation
Higher Incomes
Long Term Effects
Reduced Poverty
![Page 31: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/31.jpg)
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered
![Page 32: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/32.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Unintended Consequences
Not considered substitution for
preventative measures
Child labor doesn’t increase
![Page 33: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/33.jpg)
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered • Does not consider that program to
outcomes is not uni-directional
![Page 34: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/34.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Feedback
![Page 35: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/35.jpg)
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered • Does not consider that program to
outcomes is not uni-directional • Heterogeneous effects are not considered
![Page 36: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/36.jpg)
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Heterogeneous Effects Binding
constraint is addressed
Disaggregate by Distance to School
![Page 37: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/37.jpg)
www.3ieimpact.org Lindsey Novak
• What is the best time to create a theory of change? – During the program design – Before data collection – Before data analysis
Timing
![Page 38: Theory of Change · Lindsey Novak Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model vs the “Black box” approach](https://reader034.fdocuments.us/reader034/viewer/2022042920/5f648b9320399a017b263481/html5/thumbnails/38.jpg)
www.3ieimpact.org Lindsey Novak
Closing Remarks
• Ultimate goal: Answer the “why?” question • Identifies assumptions and creates
testable hypotheses • Identifies areas of potential weakness in
the program • Can be created for new and existing
programs