Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education

Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education

Prepared by:

Michael Duffin, PEER Associates, Inc.

With support from:

the Place-based Education Evaluation Collaborative (PEEC)

For presentation at:

the NAAEE Pre-conference Research SymposiumOctober 10, 2006

For more details and related information, see

www.PEECworks.orgor

www.PromiseOfPlace.org(as of Dec. 2006)

The Place-based EducationEvaluation Collaborative (PEEC)

Wellborn Ecology Fund

Community Mapping Program

Sustainable Schools Project Forest For Every

Classroom

Litzsinger Road Ecology Center

Trail To Every Classroom

CO-SEED

PEEC’s Dose-Response Measurement Strategy

If participants with less dose report lower outcomes, and those with more dose report higher outcomes, then the program is likely to be an active ingredient

Coleman (1966) claimed that schooling accounted for only 10% of the variance in student achievement (or R2 = .10)

Marzano (2003) claims that that number is actually closer to 20%, with 13% deriving from teacher-level factors, and 7% attributable to school-level factors

Wang (2002) found that weight status predicts 17-19% of cost for treating cardiovascular disease

Program “Dose”

(Exposure + Implementation)P

rogr

am “

Res

pon

se”

(Mea

sure

s of

Int

ende

d O

utco

me)

Less More

Low

er

H

ighe

r

Constructing Program “Dose”

Survey items estimating # of times participated in each part of the program delivery

Each item multiplied by staff estimate for average duration of typical event resulted in rough estimate of total hoursConverted program exposure to 1 to 4 scale and averaged with responses to items for implementation level to get dose compositeVery blunt instrument, but does seem to capture variability in dose (for the regression calculation, ordinal sequence was more important than gross accuracy of construct)

Measuring Program “Response” or Outcomes

Survey items reflecting behavior changes represented in program logicModules reflect major ideas

Indices represent a range of sub-dimensions of major ideas

Two to six items for each index

Used Bonferoni correction to be conservative about statistical significance

Indices, modules, and overall modules rationally constructed (factor analysis and systematic internal reliability tests still to be done)

PEEC Cross-Program Survey Results 2003-2004Changes in Educator Practice

342 educator surveys

Very diverse sample(4 programs in 55 schools; Whole school change & Prof. development models; Urban, rural, suburban; Grades K-12)

0.0 1.0 2.0 3.0 4.0

Dosage compositeScale: No exposure to a PEEC program = 0,

Very high PEEC program exposure and implementation = 4

1.0

2.0

3.0

4.0

Ave

rage

sco

res

for item

s ab

out t

each

er col

labor

atio

n, m

eeting

of

curr

iculu

m g

oals

, confi

den

ce, e

ner

gy,

gro

wth

as a

teac

her

, and

use

of lo

cal peo

ple

and p

lace

s fo

r te

achin

g(teapracmea)

Scal

e: S

tron

gly

Dis

agre

e =

1, to

Stro

ngl

y A

gree

= 4

Scal

e: T

wic

e pe

r ye

ar o

r less

= 1

, to

Once

a w

eek

or m

ore = 4

The best fit multiple regressionline above shows that 19% of thevariability in survey response ispredicted by dose of a PEEC program.The result is statistically significant.R2= .19, p = .000, n = 342.

Figure S4. Teacher Practice (overall module) From PEEC educators surveys, 2003-04

Averages from an aggregate of 12 survey items show PEEC dose accounts for 19% of variance in Overall Educator PracticeAlso at ΔR2 ≥ .10: Educator Engagement/ Growth; Use of Local Places for Teaching; Student Engagement in Learning; Student Civic Engagement; Student Time Spent Outdoors, Student Stewardship Behavior; Community Civic Engagement; Community planning/decision making process

Approx 200 more educator survey responses to be added in 04-06(2 more programs, approx. 20 more schools)

Whole School “Tipping Point” Hypothesis

Survey Analysis Provides Finer-grained View of Cultural “Tipping” at Haley Elementary, 2006

Pre-post measures spanning 3 years show large, significant effects

Dose-response calculation for aggregate of the pre- and post responses shows CO-SEED dose accounts for 45% of variance in Overall Educator Practice ChangeDose-response calculation for post-only responses shows CO-SEED dose accounts for 12% of variance in same outcomeComparing zero dose only: in 2003 Mean=1.9, n=20, SD=.51; by 2006 Mean =2.8, n=4, SD=.60.“Tipped-but-not-completely” still need some mechanism to sustain

Reflections on UsingDose-Response Measurement Strategy

Single measurement event (without giving up possibility for pre-post comparisons, either of independent groups or matched pairs)

Conducive to both aggregation and disaggregation (across time and across programs)

Effect size is readily apparent (ΔR2 or percent variance)

Requires another level of understanding of statistics for users of the evaluation data

Be open to site-specific construction of dose

First Grade Academic Achievementas a Function of CO-SEED/ Community-Based Units

(Young Achievers School, 2005)

Principal says “One thing we know is that kids’ writing is much more interesting, complex, and detailed if they’ve had rich experience…The current first grade has about a third of the kids who didn’t have Kindergarten here and in general it is breathtaking the difference in the academic achievement. Our Kindergarten has the strongest place-based education in the school, especially with language development.” First grade is also strong.

3 measures (Direct Reading Assessment, TERC Math, YA Writing Assessment) tracked in YA’s assessment database

Compared 1st graders with one v. two years of exposure to strong PBE teachers

Design:

Young Achievers School, Jamaica Plain, MA

First Grade Academic Achievementas a Function of CO-SEED/ Community-Based Units

(Young Achievers School, 2005)

1st graders w/ more place-based education outperformed peers on all measures

Findings:

DRA Assessment: 1 year vs. 2 year tenure at Y.A.

Grade 1 only 9.57

Grades K-1 13.2

0

4

8

12

16

20

24

28

32

36

DRA

Assessment

Mea

n S

core Grade 1 only

(N=14)

Grades K-1(N=25)

Writing Assessment: 1 year vs. 2 year tenure at Y.A.

1.63

1.55

1.44

1.61

1.54

1.71.65

1.61

1.681.64

1

1.25

1.5

1.75

2

2.25

2.5

Writing-Content Writing-Organization Writing-Language Writing-Mechanics Writing-Overall

Assessment

Mea

n s

core Grade 1 only

(N=14)

Grades K-1(N=25)

TERC Math: 1 year vs. 2 year tenure at Y.A.

Grade 1 only 60.71%

Grades K-168.60%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

TERC Math

Assessment

Mea

n S

core Grade 1 only

(N=14)

Grades K-1(N=25)

Young Achievers School, Jamaica Plain, MA

Effects of CO-SEED onStandardized Test Scores (MCAS) at the

Beebe Health & Environmental Magnet School(Beebe School, 2005, Massachusetts)

CO-SEED worked with Beebe 1999-2003, helped secure CSR funding to continue work 2002-2005

Several lines of evidence suggest that the environmental theme has become embedded in the school culture

Before analyzing MCAS scores, we predicted that Beebe would deviate from the typical pattern and increase performance relative to district and/or state in the following content areas:

– Math (mostly near 3rd and 4th grade)– English Language Arts – Writing– Life Science– Earth Science

Design:

Beebe School, Malden, MA



Typical pattern: State performs highest, then Beebe, then district

Findings:

MCAS - Grade 5 Earth Science

3035404550556065707580

2000 2001 2002 2003 2004 2005

year

% c

orre

ct BeebeDistrictState




Only a few deviations from the typical pattern (6th & 8th grade Math, 8th grade Life & Earth science)

Findings:

MCAS - Gr. 8 Math, Open Response

3035404550556065707580

2000 2001 2002 2003 2004 2005

year

% c

orre


MCAS - Gr. 8 Life Science

3035404550556065707580

2000 2001 2002 2003 2004 2005

% c

orre





Analysis mildly supported the prediction for two areas (Math & Earth Science)

Analysis strongly supported the prediction in one area (Life Science)

Analysis did not support the prediction in one area (Writing, the typical pattern persisted in both grades 4 and 7)

Future prediction: Strongest results will continue to show up in the upper grades (i.e. where students have the highest cumulative dose of the environmental/ place-based theme integration)

Findings:


References

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfield, F. D., & York, R. L. (1966). Equality of educational opportunity. U.S. Government Printing Office, Washington, DC.

Duffin, M., Powers, A. L., Tremblay, G., & PEER Associates. (2004). Place-based Education Evaluation Collaborative: Report on cross-program research and other program evaluation activities, 2003-2004. Retrieved October 6, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/S0019440A.

Duffin, M., & PEER Associates. (2006). Portrait of an urban elementary school: Place-based education, school culture, and leadership; An evaluation of Project CO-SEED at the Dennis C. Haley Elementary School, 2003-2006. Retrieved October 6, 2006 from http://www.peecworks.org/PEEC/PEEC_Reports/S00FE7771-0100A9B6

Marzano, R. J. (2003). What works in schools: Translating research into action. Association for Supervision and Curriculum Development, Alexandria, VA.

Wang, G., Zheng, Z., Heath, G., Macera, C., Pratt, M., & Buchner, D. (2002). Economic burden of cardiovascular disease associated with excess body weight in U.S. adults. American Journal of Preventive Medicine, 23 (1), 1-6.

http://www.peecworks.org/PEEC/PEEC_Reports/S0019440A


Prepared by:

Michael Duffin, PEER Associates, Inc.

With support from:

the Place-based Education Evaluation Collaborative (PEEC)

For presentation at:

the NAAEE Pre-conference Research SymposiumOctober 10, 2006

For more details, see

www.PEECworks.org or www.PromiseOfPlace.org

http://www.peecworks.org/

http://www.promiseofplace.org/


•Duffin, M., Powers, A. L., Tremblay, G., & PEER Associates. (2004). Place-based Education Evaluation Collaborative: Report on cross-program research and other program evaluation activities, 2003-2004. Retrieved October 6, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/S0019440A.•Duffin, M., & PEER Associates. (2006). Portrait of an urban elementary school: Place-based education, school culture, and leadership; An evaluation of Project CO-SEED at the Dennis C. Haley Elementary School, 2003-2006. Retrieved October 6, 2006 from http://www.peecworks.org/PEEC/PEEC_Reports/S00FE7771-0100A9B6

For more details and related information, see

www.PEECworks.org or www.PromiseOfPlace.org

Program “Dose”(Exposure + Implementation)

Pro

gram

“R

esp

onse

”(M

easu

res

of I

nten

ded

Out

com

e)

Less More

Low

er

Hig

her

0.0 1.0 2.0 3.0 4.0

Dosage compositeScale: No exposure to a PEEC program = 0,

Very high PEEC program exposure and implementation = 4

1.0

2.0

3.0

4.0

Ave

rage

sco

res

for item

s ab

out t

each

er col

labor

atio

n, m

eeting

of

curr

iculu

m g

oals

, confi

den

ce, e

ner

gy,

gro

wth

as a

teac

her

, and

use

of lo

cal peo

ple

and p

lace

s fo

r te

achin

g(teapracmea)

Scal

e: S

tron

gly

Dis

agre

e =

1, to

Stro

ngl

y A

gree

= 4

Scal

e: T

wic

e pe

r ye

ar o

r less

= 1

, to

Once

a w

eek

or m

ore = 4

The best fit multiple regressionline above shows that 19% of thevariability in survey response ispredicted by dose of a PEEC program.The result is statistically significant.R2= .19, p = .000, n = 342.

Figure S4. Teacher Practice (overall module) From PEEC educators surveys, 2003-04

Summary notes from October 10, 2006 presentation at NAAEE pre-conference symposium by Michael Duffin, PEER Associates

http://www.peecworks.org/

http://www.promiseofplace.org/

Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education

Documents

Transcript of Using a Dose-Response Analysis Strategy to Measure Outcomes of Place-based Education