Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of...
-
Upload
harry-hicks -
Category
Documents
-
view
215 -
download
0
Transcript of Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of...
![Page 1: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/1.jpg)
Costs, Cost-Effectiveness and Cost-Benefit Analysis
Brooks BowdenCenter for Benefit-Cost Studies of Education
Teachers College, Columbia University
Clive BelfieldQueens College, City University of New York
IES PI Meeting December 11 2015
![Page 2: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/2.jpg)
Outline
① Foundations of CA+CEA+CBA = “EA”② Need for EA③ Case studies④ Future of EA
Context: • EA research at CBCSE.org• EA teaching: IES Methods Training Grant
(2014-17, #R305B140003)
![Page 3: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/3.jpg)
![Page 4: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/4.jpg)
Case Studies: EA
Socio-Emotional Learning
interventions
Talent Search
Reading Partners
City Connects
Early literacy
4
![Page 5: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/5.jpg)
Case Studies: Displacement
• Outcome specification
Socio-Emotional Learning
interventions• Program fidelityTalent Search
• Treatment contrastReading Partners
• Service mediationCity Connects
• Power of the testEarly literacy
5
![Page 6: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/6.jpg)
COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS
FOUNDATIONS
4
![Page 7: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/7.jpg)
Basic Ideas
• Money is scarce; must consider if program is affordable
① Costs Analysis (CA): a basic description of how much resource is needed for a program
• Cannot just look at if a program works; must also look at resources needed for it to work
② Cost-Effectiveness Analysis (CEA): the link between resources needed for programs and their outcomes
• Must look at the economic consequences of a program
③ Cost-Benefit Analysis (CBA): test whether economic returns justify the cost
5
![Page 8: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/8.jpg)
Costs Analysis
• See if program can be afforded
• Enumerate all resources to implement a policy
• Incremental costs over “next best alternative” or “business-as-usual”
• This is an actual research project using ‘ingredients’ method:o Samplingo Quantities of ingredientso Prices of ingredientso Testing for difference in costs
• Average Cost per participant
• AC must be within budget
6
![Page 9: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/9.jpg)
Cost Example
High school mentoring program for 500 students:
4 teachers @ $100,000 per
4 classrooms @ $25,000 per
After-school program for 1,000 students:
6 counselors @ $80,000 per
1 gym @ $120,000 per
Mentoring Program
After-school
Program
Total number of students
500 1,000
Personnel $400,000 $480,000
Facilities $100,000 $120,000
Total Cost $500,000 $600,000
Average Cost $1,000 $600
7
![Page 10: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/10.jpg)
Cost-Effectiveness Analysis
• Find cheapest way to do the thing we have decided to do
• Compares policy alternatives, based on ratio of their costs to a quantifiable (but not monetized) effectiveness measure
• Compare alternative policies, not rule on single policy
• Compare similar policies: scale, target group, expected effects
• CE ratio = $Cost/Unit of Effectiveness
• Lowest CE ratio is most cost-effective
10
![Page 11: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/11.jpg)
CEA Example
HS Dropout rate = 20%
School size = 500
Mentoring program:o Costs $1,000 per studento Reduces dropout rate by 5%
After-school program:o Costs $600 per studento Reduces dropout rate by 2%
Mentoring
program
After-school
program
Total cost $500,000 $300,000
Baseline dropouts 100 100
Yield of new high school graduates
5 2
CE ratio $100,000 $150,000
9
![Page 12: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/12.jpg)
Cost-Benefit Analysis
• See if it is worth spending money on a program
• Compare policy alternatives, based on ratio of their costs to quantifiable and monetized effectiveness measures
• Monetized effectiveness measures are “benefits” (e.g. earnings gains, lower special education spending)
• BC ratio = $Benefits/Cost or B−C = Benefit-Costs
• Worth it if BC ratio > 1 or B−C > 0
12
![Page 13: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/13.jpg)
CBA Example
Mentoring program reduces dropout rate by 5%
Each new high school graduate compared to a dropout over lifetime: o Earns $120,000 more
o Saves taxpayer $30,000 in spending on crime
Mentoring program
Total cost over baseline C $500,000
Yield of new high school graduates 5
Total earnings gain B $600,000
Total crime saving B $150,000
Net Benefit (B-C) $250,000
Benefit-Cost Ratio (B/C) 1.50
13
![Page 14: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/14.jpg)
COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS
NEED FOR EA
12
![Page 15: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/15.jpg)
Need for EA in Education
• Many interventions, reforms, programs, policies
• Intensively investigate “What Works”
• Very little guidance on
“..At What Cost?”
“..Is it Worth the Cost?”
13
![Page 16: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/16.jpg)
Examples E
Effects
Class size reduction ✓✓
Reading Recovery ✓✓
High VA teachers ✓✓
Vouchers ✓
Web-learning =
16
![Page 17: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/17.jpg)
Examples E C
Effects Costs
Class size reduction
✓✓ Expensive
Reading Recovery ✓✓ Very expensive
High VA teachers ✓✓ No idea
Vouchers ✓ Cheaper (?)
Web-learning = Cheaper (?)
17
![Page 18: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/18.jpg)
Examples E C E?
Cost-Effective or Efficient?
Class size reduction
Perhaps
Compared to what?
From whose perspective?
Is it affordable?
Reading Recovery
High VA teachers
Vouchers
Web-learning
18
![Page 19: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/19.jpg)
EA Methods
• They are methods:o CA: ingredients methodo CEA: CA + evaluation impactso CBA: CA + shadow pricing methods
• There are methodological standards
• But not like impact evaluations because hypothesis / statistical testing is not determinative
19
![Page 20: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/20.jpg)
EA Practice
• EA is becoming more common
• Slowly and from a low base
• Assumptions rarely harmonized
• Research questions rarely related to each other
• Limited hypothesis testing
20
![Page 21: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/21.jpg)
Value of EA
Conventional answer: • It will help decision-making• If it does not help us decide between X and Y,
it is not needed
New answer: • It will improve education research
generally
19
![Page 22: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/22.jpg)
SESSION II
APPLICATIONS & RECOMMENDATIOS
21
![Page 23: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/23.jpg)
Case Studies
• Outcome specification
Socio-Emotional Learning
interventions• Program fidelityTalent Search
• Treatment contrastReading Partners
• Service mediationCity Connects
• Power of the testEarly literacy
23
![Page 24: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/24.jpg)
OUTCOME SPECIFICATION
CASE STUDY: SOCIO-EMOTIONAL LEARNING
21
![Page 25: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/25.jpg)
SEL Interventions
• Two full reviews of interventions that enhance SE skills (Durlak et al., 2011; Sklad et al., 2012)
• Focus on SE skills or test scores? (Can’t do both)
• SE skills are broad-based, not task-specific
• Need to look at broad set of outcomes• Need to weigh those outcomes• This implies CBA or CBA framework25
![Page 26: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/26.jpg)
The Economic Value of Social and Emotional Learning
26
![Page 27: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/27.jpg)
Outcomes of SEL interventions?
Across the literature outcomes were:• Inconsistent• More or less• Student, teacher, classroom, school
level• Durable or ephemeral• Fast-acting or delayed
24
![Page 28: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/28.jpg)
25
![Page 29: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/29.jpg)
Conclusions
• Better “benefit maps”
• Focus on school-level or class-level effects of SEL
• Examine how fast-acting and durable interventions are
• Generally, pre-specify outcomes (Social Impact Bonds)
29
![Page 30: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/30.jpg)
Clincher
• We propose a cost-effectiveness analysis
• Question always asked is:
How are we measuring effectiveness?
• We don’t know – we are economists
30
![Page 31: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/31.jpg)
PROGRAM FIDELITY
CASE STUDY: TALENT SEARCH
38
![Page 32: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/32.jpg)
• Helps low-income and first-generation college students complete high school and gain access to college
• Many services: academic achievement, access to financial aid, test taking and study skills assistance, academic advising, tutoring, career development, campus visits
Talent Search
32
![Page 33: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/33.jpg)
Program Flexibility
Site-specific variation:• Sites can serve students in G6-12 or cohorts
from 6G through 12G; at few or many schools
• Sites are at schools and at colleges• Programs vary with respect to services
Cost analysis needed to:① Describe the program② Estimate in-kind resources③ Identify most cost-effective sites33
![Page 34: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/34.jpg)
EA of Talent Search
• Cost Analysis: Collected cost data using ingredients method from 6 TX sites and 3 FL sites
• CEA: Linked cost data to effects from Constantine et al. (2006)
• CBA: Shadow priced estimates of earnings gains from high school completion and college attendance
34
![Page 35: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/35.jpg)
35
![Page 36: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/36.jpg)
36
PER YEAR
FL3
TX4
TX3
FL1
TX2
FL2
TX6
TX4
TX1
$640
$2,730
$5,190
Cost per Student per Site(PV at age 18 2010$)
![Page 37: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/37.jpg)
37
FL1
FL3
FL2
TX2
TX4
TX1
TX3
TX6
TX5
$- $30,000 $60,000 $90,000 $120,000 $150,000
$10,830
$148,090
Infinity
Cost-effectiveness Ratios: Cost per HS Completer
![Page 38: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/38.jpg)
Read 180
• For struggling readers G1-12 to improve decoding, fluency, and comprehension skills
• Prescriptive program:o 90-minutes daily of whole-group, small-group and
individualized instruction + videos + feedbacko Class size 15
• Collected cost data using ingredients method from three sites (Levin et al., 2007)
38
![Page 39: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/39.jpg)
Read 180 Cost per Student
Site 1 Site 2 Site 3
Students served 6,701 1,080 2,400
Personnel (teachers) $320 $950 $70
Personnel (administrators, technicians, coordinators)
$50 $400 $60
Equipment/materials(computers, licenses) $250 $150 $140
Other (prof. dev., sub teachers, other) $- $10 $10
Average Cost $610 $1,510 $280 46
![Page 40: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/40.jpg)
JOBSTART
• Education/training program for low-income youth, including remedial education, GED preparation, and job placement assistance
• Youth can remain in the program for up to two years
• MIS data from Cave et al. (2003)
40
![Page 41: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/41.jpg)
41
Corpus Christi
EGOS Denver
San Jose
Chicago
LA Jobs Corps
NY City
$4,470
$4,710
$6,650
$14,760
$16,190
$20,800
JOBSTART Cost per Youth per Site (PV at 18, 2014$)
![Page 42: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/42.jpg)
Mean = $14,900SD = $13,400N =5050
![Page 43: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/43.jpg)
Job Corps: Line Item Costs
Cost per Mean SD Max
Counseling $4,620 4360 $50,020
Admin $3,640 3410 $53,540
Voc. training $2,930 2760 $33,540
Allowance $2,200 1940 $11,110
Child care $1,810 1670 $17,450
Outreach $1,630 1440 $9,860
Education $1,600 1460 $15,210
Med/dental $1,270 1180 $13,190
Food $1,050 1020 $12,280
![Page 44: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/44.jpg)
Conclusions
Cost Analysis yields information on:
• What the program is
• Actual program implementation
• Fidelity• Many evaluations include (idiosyncratic?)
investigation of program fidelity
• Base fidelity on resource use indicators
44
![Page 45: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/45.jpg)
TREATMENT CONTRAST
CASE STUDY: READING PARTNERS
28
![Page 46: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/46.jpg)
Reading Partners
• Supplemental pull-out reading
program:o One-on-one tutoring (2x45 minutes per week)o Dedicated school space and materialso Structured and individualized curriculumo Data-driven instructiono Rigorous ongoing trainingo Instructional supervision and support
• For struggling students in K-5
• Split responsibility:o Run with volunteers supported by AmeriCorps Site
Coordinatorso Site Coordinators supervised weekly by Program
Managers
46
![Page 47: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/47.jpg)
Mobilizing Volunteer Tutors
47
www.mdrc.org/sites/default/files/ReadingPartners_2015_FR.pdf
![Page 48: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/48.jpg)
Evaluating RP
• Multi-site randomized design: o 1,250 students randomly assigned at 19 school siteso Grades 2-5, eligible for Reading Partnerso Evaluated during school year 2012-13o Baseline equivalence of treatment and control
• Cost analysis was part of evaluation:o What resources were needed to implement RP as
per the evaluation?o What proportion of RP costs were borne by the
school?o How did these resource requirements compare with
the resources required to implement other supplemental reading services in these schools?48
![Page 49: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/49.jpg)
RP Context
• Cost ≠ Price
• School selects RP as one option for supplemental reading services
• Children in need must be served – impact relative to other supplemental services (not no intervention)
• Cost Study Sample: 6 of 19 study siteso Chosen based on strong implementation, geographic diversity, and ability to collect
reliable data
• Data collected via implementation observations, teacher surveys, and interviews with RP staff, teachers, reading support staff, and school personnel
• Total social cost calculated by summing the costs of all the resources utilized to implement program (Levin & McEwan, 2001)
o Included both in-kind and financial costs
o Used national prices
o 2013 $USD
• Same approach used to assess the cost of the other supplemental reading programs offered at these six schools49
![Page 50: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/50.jpg)
RP Costs
• Total social cost calculated by summing costs of all resources utilized to implement program (Levin & McEwan, 2001)o Included both in-kind and financial costso Used national priceso 2013 $USD
• Same approach used to assess the cost of the other supplemental reading programs offered at these six schools
50
![Page 51: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/51.jpg)
RP Cost Analysis
Cost per student
RP staff $690
AmeriCorps members $930School staff $90
Volunteer time + transport $1,520
Facilities $300
Materials/equipment $80
Total Ingredients $3,610
51
![Page 52: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/52.jpg)
RP Cost per Student
Cost per student
Cost to school
Cost to others
RP staff $690 $320 $370
AmeriCorps members $930 $930
School staff $90 $90Volunteer time + transport $1,520 $1,520
Facilities $300 $300
Materials/equipment $80 $80
Total Ingredients $3,610 $710 $2,900
20% 80%
52
![Page 53: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/53.jpg)
53
Cost per Cost to Cost to Cost to Cost to Cost Student School Volunteers Reading Partners AmeriCorps
IngredientsReading Partners staff 690 690AmeriCorps members 930 930School staff 90 90Volunteer time and transportation 1,520 1,520Facilities 300 300Materials and equipment 80 80Total ingredients 3,610 390 1,520 1,700 0
Fee for service ($) 320 -320
AmeriCorps grant ($) -270 270
Net cost per student (Total ingredients +fee for service + AmeriCorps grant) ($) 710 1,520 1,110 270
Portion of net cost per student (%) 20 42 31 7
Cost of Reading Partners per Program Group Student
Distribution of Cost per Student
![Page 54: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/54.jpg)
Cost of Other Supplemental Reading Services
Total Cost Per Student
Reading Partners Other Supplemental Services
Site A $3,450 $1,840
Site B $3,420 $2,230
Site C $3,570 $2,680
Site D $5,190 $1,050
Site E $4,210 $1,980
Site F $2,740 $4,890
Pooled $3,610 $1,780
54
![Page 55: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/55.jpg)
RP Cost versus BAU Cost
RP BAU
$710
$1,700
$2,900 $80
School funded Other resources
55
![Page 56: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/56.jpg)
Conclusions
• Schools receive a resource rich intervention [RP] for a fraction of the cost
• Cost-effective option for under-resourced schools
• Important to consider treatment contrast and BAU in costs as well as effects.
56
![Page 57: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/57.jpg)
SERVICE MEDIATION
CASE STUDY: CITY CONNECTS
56
![Page 58: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/58.jpg)
Service Mediation Interventions
These SMIs have attributes of:
• Diagnosis/assessment
• Information
• Referral
• Assignment to services
• Tracking
• Service management
58
![Page 59: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/59.jpg)
Service Mediation Interventions
Examples:• Diagnosis/assessment: Interventions to change
college remediation
• Information: Letter grades and accountability systems
• Referral: Texts to motivate class attendance
• Assignment: Grade retention
• Tracking effect: FAFSA support to direct college choice
• Tracking effect: Head Start to motivate future parental investment
• Service management: City Connects
59
![Page 60: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/60.jpg)
Implications (1)
• Impact of intervention is the sum of diagnosis + service receipto New placement test impact depends on the test +
remedial services received based on the test result o Grade retention policy impact depends on whether
student is retained + services received by retained v. non-retained students
• Not possible to disentangle the impact attributable to the intervention from the impact attributable to service receipt; service change must be measured
60
![Page 61: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/61.jpg)
Implications (2)
• Interventions might save resources in the long runo New placement tests have reduced enrollment in
remediation (Scott-Clayton et al., 2014)
• Interventions with no positive effect might be justified purely in resource terms
• Illustrate using economic framework
61
![Page 62: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/62.jpg)
City Connects (CCNX)
• CCNX diagnoses student need, refers students to most appropriate support services, manages support services, and provides some information/services to students directly
• Walsh et al. (2014b, Table 7) identify significant gains in academic achievement from CCNX
• CCNX yields average gain of 0.39 standard deviations in English Language Arts and math by 5th Grade
62
![Page 63: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/63.jpg)
BCA of City Connects
63
![Page 64: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/64.jpg)
Sample, Data, and Methods
• Sample: o 2 representative K-8 schools in Bostono 5 community partners per site, based on service
intensity
• Data from interviews: o Site coordinators, teachers, principals, City Connects
central office staff, and community partners
• Costs estimated with ingredients method:o Opportunity cost not accounting costs/ expenditureso Separate analyses of quantities/prices and by funder
64
![Page 65: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/65.jpg)
Costs for K-5 years per site(PV at K, 2013$)
City Connects Mediated ServicesPersonnel $4,868,900Central Program Staff $174,580 School Site Coordinators $900,040 School Staff $93,140 Parental Involvement $4,990 Materials $5,040 $42,070Facilities $16,410 $186,910Other $774,900Total $1,204,560 $5,872,800Students 780 780Average Cost $1,540 $7,530
65
![Page 66: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/66.jpg)
Average Cost per StudentCore +
No Extra Service
Core + Full Services
Core + Estimated
Service Change
City Connects program $1,540 $1,540 $1,540
Mediated services $7,530 $3,030
Full Cost (C) $1,540 $9,070 $4,570
Effects (E) 0.39 0.39 0.39
C/E ratio $3,950 $23,260 $11,72066
![Page 67: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/67.jpg)
Service Mediation
• City Connects is effective – large gains in academic achievement
• If only direct costs are counted, it is very cost-effective
• But if all service costs are counted, it is unclear• Need to perform CBA to fully evaluate City
Connects• Require different impact evaluation (outcomes,
durations)
67
![Page 68: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/68.jpg)
POWER OF TEST
CASE STUDY: EARLY LITERACY
50
![Page 69: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/69.jpg)
Power of Test
• Before identifying impacts, specify power of test
• But economists are not focused on statistical significance
• And distribution of ratio of two variables is not same as distribution of each variable
• So use “power of economic test”
69
![Page 70: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/70.jpg)
Early Literacy
• Estimated costs for all effective early literacy interventions
• Costs based on ingredients of effective implementations of interventions
• Related costs to effectiveness from independent studies
70
![Page 71: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/71.jpg)
Improving Early Literacy
71
![Page 72: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/72.jpg)
72
Cost per Effect Size Gain
K-PALS $40
Stepping Stones $570
Sound Partners $600
Reading Recovery $2,090
Fast ForWord $1,480
Corrective Reading $6,360
Wilson Reading Systems $13,390
![Page 73: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/73.jpg)
Power of Economic Test
• How much more effective would Wilson have to be to be as cost-effective as K-PALS?
• How imprecise would estimates of these effects have to be for K-PALS to lose status as most cost-effective?
• Minimum detectable effect (MDE): the smallest true effect that has a “good chance” of being found to be statistically significant (Bloom)
• MDCE: the smallest true CE-ratio that has a “good chance” of being found to be statistically significant? 73
![Page 74: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/74.jpg)
CA: General Conclusions
• Gives better understanding of the program• Programs typically cost more than
guesstimated or proposed or in budget documents
• Substantial variation in costs for a given program
• Most programs are not high cost given what is expected to happen
74
![Page 75: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/75.jpg)
CEA: General Conclusions
• No expectation more costly programs are more
effective
• Very low cost programs are unlikely to be
effective
• Cost-effectiveness varies across programs,
across sites, and over time
• Targeted interventions are likely to be more
cost-effective
75
![Page 76: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/76.jpg)
CBA: General Conclusions
• Targeted interventions are likely to have higher BC ratios
• Effective programs tend to have very high benefits
• Almost always undercount benefits• Test scores not as valuable as other skills
76
![Page 77: Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive.](https://reader035.fdocuments.us/reader035/viewer/2022062500/5697c0081a28abf838cc6ae7/html5/thumbnails/77.jpg)
Friday afternoon! Thank you!
Brooks Bowden and Clive Belfieldwww.cbcse.org
77