Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education
Prepared by:
Michael Duffin, PEER Associates, Inc.
With support from:
the Place-based Education Evaluation Collaborative (PEEC)
For presentation at:
the NAAEE Pre-conference Research SymposiumOctober 10, 2006
For more details and related information, see
www.PEECworks.orgor
www.PromiseOfPlace.org(as of Dec. 2006)
The Place-based EducationEvaluation Collaborative (PEEC)
Wellborn Ecology Fund
Community Mapping Program
Sustainable Schools Project Forest For Every
Classroom
Litzsinger Road Ecology Center
Trail To Every Classroom
CO-SEED
PEEC’s Dose-Response Measurement Strategy
If participants with less dose report lower outcomes, and those with more dose report higher outcomes, then the program is likely to be an active ingredient
Coleman (1966) claimed that schooling accounted for only 10% of the variance in student achievement (or R2 = .10)
Marzano (2003) claims that that number is actually closer to 20%, with 13% deriving from teacher-level factors, and 7% attributable to school-level factors
Wang (2002) found that weight status predicts 17-19% of cost for treating cardiovascular disease
Program “Dose”
(Exposure + Implementation)P
rogr
am “
Res
pon
se”
(Mea
sure
s of
Int
ende
d O
utco
me)
Less More
Low
er
H
ighe
r
Constructing Program “Dose”
Survey items estimating # of times participated in each part of the program delivery
Each item multiplied by staff estimate for average duration of typical event resulted in rough estimate of total hoursConverted program exposure to 1 to 4 scale and averaged with responses to items for implementation level to get dose compositeVery blunt instrument, but does seem to capture variability in dose (for the regression calculation, ordinal sequence was more important than gross accuracy of construct)
Measuring Program “Response” or Outcomes
Survey items reflecting behavior changes represented in program logicModules reflect major ideas
Indices represent a range of sub-dimensions of major ideas
Two to six items for each index
Used Bonferoni correction to be conservative about statistical significance
Indices, modules, and overall modules rationally constructed (factor analysis and systematic internal reliability tests still to be done)
PEEC Cross-Program Survey Results 2003-2004Changes in Educator Practice
342 educator surveys
Very diverse sample(4 programs in 55 schools; Whole school change & Prof. development models; Urban, rural, suburban; Grades K-12)
0.0 1.0 2.0 3.0 4.0
Dosage compositeScale: No exposure to a PEEC program = 0,
Very high PEEC program exposure and implementation = 4
1.0
2.0
3.0
4.0
Ave
rage
sco
res
for item
s ab
out t
each
er col
labor
atio
n, m
eeting
of
curr
iculu
m g
oals
, confi
den
ce, e
ner
gy,
gro
wth
as a
teac
her
, and
use
of lo
cal peo
ple
and p
lace
s fo
r te
achin
g(teapracmea)
Scal
e: S
tron
gly
Dis
agre
e =
1, to
Stro
ngl
y A
gree
= 4
Scal
e: T
wic
e pe
r ye
ar o
r less
= 1
, to
Once
a w
eek
or m
ore = 4
The best fit multiple regressionline above shows that 19% of thevariability in survey response ispredicted by dose of a PEEC program.The result is statistically significant.R2= .19, p = .000, n = 342.
Figure S4. Teacher Practice (overall module) From PEEC educators surveys, 2003-04
Averages from an aggregate of 12 survey items show PEEC dose accounts for 19% of variance in Overall Educator PracticeAlso at ΔR2 ≥ .10: Educator Engagement/ Growth; Use of Local Places for Teaching; Student Engagement in Learning; Student Civic Engagement; Student Time Spent Outdoors, Student Stewardship Behavior; Community Civic Engagement; Community planning/decision making process
Approx 200 more educator survey responses to be added in 04-06(2 more programs, approx. 20 more schools)
Whole School “Tipping Point” Hypothesis
Survey Analysis Provides Finer-grained View of Cultural “Tipping” at Haley Elementary, 2006
Pre-post measures spanning 3 years show large, significant effects
Dose-response calculation for aggregate of the pre- and post responses shows CO-SEED dose accounts for 45% of variance in Overall Educator Practice ChangeDose-response calculation for post-only responses shows CO-SEED dose accounts for 12% of variance in same outcomeComparing zero dose only: in 2003 Mean=1.9, n=20, SD=.51; by 2006 Mean =2.8, n=4, SD=.60.“Tipped-but-not-completely” still need some mechanism to sustain
Reflections on UsingDose-Response Measurement Strategy
Single measurement event (without giving up possibility for pre-post comparisons, either of independent groups or matched pairs)
Conducive to both aggregation and disaggregation (across time and across programs)
Effect size is readily apparent (ΔR2 or percent variance)
Requires another level of understanding of statistics for users of the evaluation data
Be open to site-specific construction of dose
First Grade Academic Achievementas a Function of CO-SEED/ Community-Based Units
(Young Achievers School, 2005)
Principal says “One thing we know is that kids’ writing is much more interesting, complex, and detailed if they’ve had rich experience…The current first grade has about a third of the kids who didn’t have Kindergarten here and in general it is breathtaking the difference in the academic achievement. Our Kindergarten has the strongest place-based education in the school, especially with language development.” First grade is also strong.
3 measures (Direct Reading Assessment, TERC Math, YA Writing Assessment) tracked in YA’s assessment database
Compared 1st graders with one v. two years of exposure to strong PBE teachers
Design:
Young Achievers School, Jamaica Plain, MA
First Grade Academic Achievementas a Function of CO-SEED/ Community-Based Units
(Young Achievers School, 2005)
1st graders w/ more place-based education outperformed peers on all measures
Findings:
DRA Assessment: 1 year vs. 2 year tenure at Y.A.
Grade 1 only 9.57
Grades K-1 13.2
0
4
8
12
16
20
24
28
32
36
DRA
Assessment
Mea
n S
core Grade 1 only
(N=14)
Grades K-1(N=25)
Writing Assessment: 1 year vs. 2 year tenure at Y.A.
1.63
1.55
1.44
1.61
1.54
1.71.65
1.61
1.681.64
1
1.25
1.5
1.75
2
2.25
2.5
Writing-Content Writing-Organization Writing-Language Writing-Mechanics Writing-Overall
Assessment
Mea
n s
core Grade 1 only
(N=14)
Grades K-1(N=25)
TERC Math: 1 year vs. 2 year tenure at Y.A.
Grade 1 only 60.71%
Grades K-168.60%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
TERC Math
Assessment
Mea
n S
core Grade 1 only
(N=14)
Grades K-1(N=25)
Young Achievers School, Jamaica Plain, MA
Effects of CO-SEED onStandardized Test Scores (MCAS) at the
Beebe Health & Environmental Magnet School(Beebe School, 2005, Massachusetts)
CO-SEED worked with Beebe 1999-2003, helped secure CSR funding to continue work 2002-2005
Several lines of evidence suggest that the environmental theme has become embedded in the school culture
Before analyzing MCAS scores, we predicted that Beebe would deviate from the typical pattern and increase performance relative to district and/or state in the following content areas:
– Math (mostly near 3rd and 4th grade)– English Language Arts – Writing– Life Science– Earth Science
Design:
Beebe School, Malden, MA
Effects of CO-SEED onStandardized Test Scores (MCAS) at the
Beebe Health & Environmental Magnet School(Beebe School, 2005, Massachusetts)
Typical pattern: State performs highest, then Beebe, then district
Findings:
MCAS - Grade 5 Earth Science
3035404550556065707580
2000 2001 2002 2003 2004 2005
year
% c
orre
ct BeebeDistrictState
Beebe School, Malden, MA
Effects of CO-SEED onStandardized Test Scores (MCAS) at the
Beebe Health & Environmental Magnet School(Beebe School, 2005, Massachusetts)
Only a few deviations from the typical pattern (6th & 8th grade Math, 8th grade Life & Earth science)
Findings:
MCAS - Gr. 8 Math, Open Response
3035404550556065707580
2000 2001 2002 2003 2004 2005
year
% c
orre
ct BeebeDistrictState
MCAS - Gr. 8 Life Science
3035404550556065707580
2000 2001 2002 2003 2004 2005
% c
orre
ct BeebeDistrictState
Beebe School, Malden, MA
Effects of CO-SEED onStandardized Test Scores (MCAS) at the
Beebe Health & Environmental Magnet School(Beebe School, 2005, Massachusetts)
Analysis mildly supported the prediction for two areas (Math & Earth Science)
Analysis strongly supported the prediction in one area (Life Science)
Analysis did not support the prediction in one area (Writing, the typical pattern persisted in both grades 4 and 7)
Future prediction: Strongest results will continue to show up in the upper grades (i.e. where students have the highest cumulative dose of the environmental/ place-based theme integration)
Findings:
Beebe School, Malden, MA
References
Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfield, F. D., & York, R. L. (1966). Equality of educational opportunity. U.S. Government Printing Office, Washington, DC.
Duffin, M., Powers, A. L., Tremblay, G., & PEER Associates. (2004). Place-based Education Evaluation Collaborative: Report on cross-program research and other program evaluation activities, 2003-2004. Retrieved October 6, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/S0019440A.
Duffin, M., & PEER Associates. (2006). Portrait of an urban elementary school: Place-based education, school culture, and leadership; An evaluation of Project CO-SEED at the Dennis C. Haley Elementary School, 2003-2006. Retrieved October 6, 2006 from http://www.peecworks.org/PEEC/PEEC_Reports/S00FE7771-0100A9B6
Marzano, R. J. (2003). What works in schools: Translating research into action. Association for Supervision and Curriculum Development, Alexandria, VA.
Wang, G., Zheng, Z., Heath, G., Macera, C., Pratt, M., & Buchner, D. (2002). Economic burden of cardiovascular disease associated with excess body weight in U.S. adults. American Journal of Preventive Medicine, 23 (1), 1-6.
Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education
Prepared by:
Michael Duffin, PEER Associates, Inc.
With support from:
the Place-based Education Evaluation Collaborative (PEEC)
For presentation at:
the NAAEE Pre-conference Research SymposiumOctober 10, 2006
For more details, see
www.PEECworks.org or www.PromiseOfPlace.org
Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education
•Duffin, M., Powers, A. L., Tremblay, G., & PEER Associates. (2004). Place-based Education Evaluation Collaborative: Report on cross-program research and other program evaluation activities, 2003-2004. Retrieved October 6, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/S0019440A.•Duffin, M., & PEER Associates. (2006). Portrait of an urban elementary school: Place-based education, school culture, and leadership; An evaluation of Project CO-SEED at the Dennis C. Haley Elementary School, 2003-2006. Retrieved October 6, 2006 from http://www.peecworks.org/PEEC/PEEC_Reports/S00FE7771-0100A9B6
For more details and related information, see
www.PEECworks.org or www.PromiseOfPlace.org
Program “Dose”(Exposure + Implementation)
Pro
gram
“R
esp
onse
”(M
easu
res
of I
nten
ded
Out
com
e)
Less More
Low
er
Hig
her
0.0 1.0 2.0 3.0 4.0
Dosage compositeScale: No exposure to a PEEC program = 0,
Very high PEEC program exposure and implementation = 4
1.0
2.0
3.0
4.0
Ave
rage
sco
res
for item
s ab
out t
each
er col
labor
atio
n, m
eeting
of
curr
iculu
m g
oals
, confi
den
ce, e
ner
gy,
gro
wth
as a
teac
her
, and
use
of lo
cal peo
ple
and p
lace
s fo
r te
achin
g(teapracmea)
Scal
e: S
tron
gly
Dis
agre
e =
1, to
Stro
ngl
y A
gree
= 4
Scal
e: T
wic
e pe
r ye
ar o
r less
= 1
, to
Once
a w
eek
or m
ore = 4
The best fit multiple regressionline above shows that 19% of thevariability in survey response ispredicted by dose of a PEEC program.The result is statistically significant.R2= .19, p = .000, n = 342.
Figure S4. Teacher Practice (overall module) From PEEC educators surveys, 2003-04
Summary notes from October 10, 2006 presentation at NAAEE pre-conference symposium by Michael Duffin, PEER Associates
Top Related