Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole...
-
Upload
kenneth-cameron -
Category
Documents
-
view
213 -
download
0
Transcript of Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole...
![Page 1: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/1.jpg)
Early Learning in Mathematics (ELM)The Efficacy of a Kindergarten Curriculum
Implemented in Whole Classroom Settings
Scott K. Baker, PhDPacific Institutes for Research / University of Oregon
Ben Clarke, PhDPacific Institutes for Research
Hank Fien, PhDUniversity of Oregon
Keith Smolkowski, PhDOregon Research Institute
Chris Doabler, PhDPacific Institutes for Research
David Chard, PhDSouthern Methodist University
IES Conference June 2010
![Page 2: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/2.jpg)
Acknowledgements
Institute of Education SciencesThe research reported here was supported by the Institute of
Education Sciences, U.S. Department of Education, through Goal 2 development grant, #R305K040081, and a Goal 3 efficacy grant, #R305A080114, to Pacific Institutes for Research. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Additional Oregon Project StaffKathy Jungjohann / Karen Davis: Curriculum developmentMari Strand Cary / Rhonda Griffiths: Coordination and research Chris Doabler: Observation measurement and research
![Page 3: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/3.jpg)
Early Learning in Mathematics (ELM)
• 4-year randomized efficacy control trial• Measuring the efficacy of a kindergarten
mathematics curriculum in Oregon and Texas.
• Research Design• Randomized Controlled Block Design• Classrooms within school matched on full / half
day and randomly assigned to treatment (ELM) or control conditions
3
![Page 4: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/4.jpg)
Purpose of the 4-year project• Study 1: Efficacy trial of the whole group
curriculum (ELM) on kindergarten students’ mathematics achievement
• Study 2: Efficacy trial of small group ELM component – Roots – on the achievement of at-risk students
• Examine potential mediation variables, dose-response variables, and moderation factors
![Page 5: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/5.jpg)
Structure of the Curriculum
• Daily Calendar Lessons / Activities– 15 minutes daily, whole class “circle” time– Monthly booklets with objectives and
application activities
• 120 Core Lessons divided into 4 quarters– 30 minutes whole class instruction– 15 minutes teacher directed written work– End of quarter assessment of progress
![Page 6: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/6.jpg)
ELM Instructional Content• National Math Advisory Panel (2008) recommends
a focused, coherent progression of mathematics learning with emphasis on proficiency with key topics
• ELM focuses on key strands rather than a broad array of mathematical content– Numbers and Operations– Geometry– Measurement– Vocabulary (NCTM Process Standard, 2000)
NCTM Curriculum Focal Points for K (2006)
![Page 7: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/7.jpg)
ELM Conceptual Framework
Development of Mathematical
Concepts / Models
Mathematics-relatedVocabulary and
Discourse
Mathematics-relatedVocabulary and
Discourse
Procedural Fluency
![Page 8: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/8.jpg)
Research questions• What is the immediate impact of ELM taught in general
education kindergarten classrooms on mathematics achievement compared to standard district practice?
• Is impact moderated by student level of risk for mathematics difficulties?
• Does rate of teacher models or student practice opportunities mediate a condition effect?
• Is there evidence of an interaction between condition and student practice on student outcomes?
![Page 9: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/9.jpg)
Study Sample• Assignment at the classroom level blocked on
school– Districts = 3– Schools = 24
• Intervention classrooms = 34 • Control classrooms = 30
• Students nested in classrooms– Whole class instruction– N = 1,349
![Page 10: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/10.jpg)
Student Demographics
• 56.3% eligible for free or reduced lunch• 38.4% English Learners• 8.4% special education
• 49.5% White• 36.4% Latino• 4.8% Asian / Pacific Islander• 2.3% African American
![Page 11: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/11.jpg)
Intervention Mediators Proximal Outcomes
DistalOutcomes
Hypothesized Model of ELM Impact
![Page 12: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/12.jpg)
Descriptive data on implementation
• Classroom fidelity observations – treatment and control
• Classroom observations focusing on instructional interactions – treatment and control
• Teacher logs addressing content coverage – treatment and control classrooms
![Page 13: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/13.jpg)
Implementation Fidelity
• General ratings (8 items)– Models skills/concepts appropriately and with
ease– Engages students in learning throughout the
lesson
• Uses ELM / completes all lesson activities (dichotomous)
• For each ELM activity (range 1-7 per lesson): Full (2) / Partial (1) / Not Taught (0)
![Page 14: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/14.jpg)
Implementation Fidelity Data: ELM Lesson Activities
• 81 ELM (fidelity) observations during the year• Fall : mean = 1.71 (SD = .19)• Winter: mean = 1.65 (SD = .33)• Spring: mean = 1.62 (SD = .43)• Overall:mean = 1.65 (SD = .36) (83% of Full)• 2 of 81 lessons had a mean below Partial (1)
level of implementation
![Page 15: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/15.jpg)
Student measures of impact
• Test of Early Mathematics Ability (TEMA)• Early Numeracy – CBM
– Oral Counting– Number Identification– Quantity Discrimination– Missing Number
![Page 16: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/16.jpg)
Method and Analysis Framework
• Competing curricula– All students received instruction– Time balanced across conditions
• Sample– At risk (some or high risk)
• < 40th percentile• 66% of student sample
– No risk • ≥ 40th percentile• 34% of student sample
![Page 17: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/17.jpg)
Nested Time × Condition Analysis• Outcome: net differences from pre to post• Nested students within classrooms
– Control for nonindependence (e.g., ICCs)– Controls for teacher effects
• Maximum likelihood (restricted)– Includes all cases with data at either T1 or T2– Reduces bias from missing data
• Moderation added Time × Risk × Condition interaction
• Effect sizes: Hedges’ g
![Page 18: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/18.jpg)
Sample Means, SDs, and NsT No Risk Some Risk
ELM Control ELM Control
TEMA Raw T1 29.0 (6.8) 28.8 (7.3) 14.1 (6.0) 14.6 (6.2)
T2 39.6 (6.3) 39.1 (7.9) 28.6 (8.4) 26.9 (8.0)
TEMA Percentile T1 66.5 (16.4) 65.1 (17.4) 14.5 (11.4) 15.4 (11.7)
T2 70.4 (18.3) 68.1 (20.8) 35.5 (23.6) 31.0 (22.4)
CBM Total T1 120.3 (44.0) 116.7 (43.5) 45.0 (32.6) 45.5 (35.5)
T2 193.1 (35.8) 187.9 (38.1) 138.4 (50.3) 126.6 (50.5)
Sample Size T1 203 181 397 343
T2 190 174 341 312Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms.
![Page 19: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/19.jpg)
TEMA Percentile Scores Gains by Condition
• Gains– Control: 10.94– ELM: 14.73– Difference: 3.79
• Test of Condition– t = 2.10– df = 61– p = .0396– ES = +0.14
• T1 differences were not statistically significant (t = 0.57)
![Page 20: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/20.jpg)
CBM Total Scores Gains by Condition
• Gains– Control: 77.20– ELM: 84.87– Difference: 7.67
• Test of Condition– t = 1.99– df = 61– p = .0509– ES = +0.14
• T1 differences were not statistically significant (t = 0.60)
![Page 21: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/21.jpg)
TEMA Raw ScoresCondition by Risk Status
• Main Effects– Difference in gains: 1.32– t = 2.41, df = 61, p = .0190
• Condition by Risk Status– t = 2.47, df = 61, p = .0162– No Risk ≥ 40th %tile
• Difference in gains: 0.04• t = 0.51, df = 61, p = .9586
– Risk < 40th %tile• Difference in gains: 1.98• t = 3.29, df = 61, p = .0017
![Page 22: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/22.jpg)
CBM Total ScoresCondition by Risk Status
• Main Effects– Difference in gains: 7.67– t = 1.99, df = 61, p = .0509
• Condition by Risk Status– t = 2.24, df = 61, p = .0289– No Risk ≥ 40th %tile
• Difference in gains: -0.27• t = -0.05, df = 61, p = .9570
– Risk < 40th %tile• Difference in gains: 10.81• t = 2.54, df = 61, p = .0138
![Page 23: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/23.jpg)
Effect Sizes (Hedges’ g)
Measure Not At Risk At Risk
Tema Raw Score +0.006 +0.242**
EN-CBM Total +0.014 +0.215*
*p < .05; **p < .01
![Page 24: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/24.jpg)
Summary• ELM classrooms outperformed controls
– TEMA raw and percentile scores– EN-CBM Total
• Students at risk– Improve on all measures more than no-risk students– Control students at risk catching up on no-risk students
• TEMA: 14.0 percentile gain on no-risk students• EN-CBM: 9.6 point gain on no-risk students
– ELM students at risk catching no-risk students faster• TEMA: 18.6 percentile gain on no-risk students• EN-CBM: 20.63 point gain on no-risk students
• No condition effects for students with no risk
![Page 25: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/25.jpg)
Intervention Mediators Proximal Outcomes
DistalOutcomes
Preliminary Analysis of Association between Observation Data and Student Outcomes
![Page 26: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/26.jpg)
Coding of Academic Teacher-Student Interactions (CATS) Observation
Instrument CATS uses a frequency count approach to measure
teacher-student instructional interactionsObservers code behavior occurrences in a continual, serial fashion.Utilizes a strict coding structure
CATS based on evidence of effective instruction in early literacy and beginning mathematics, and adapted from the STICO observation instrument (Smolkowski & Gunn, 2010)
26
STUDENT BEHAVIORS TEACHER BEHAVIORS
Individual responsesGroup responsesCovert responsesMistakes
Teacher modelsAcademic feedback
![Page 27: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/27.jpg)
Hypothetical Case of a Instructional Interaction
![Page 28: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/28.jpg)
![Page 29: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/29.jpg)
The Role of Teacher Modeling and Student Practice in Student
Outcomes
![Page 30: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/30.jpg)
Preliminary Mediation Analysis
• Does rate of teacher models or student practice opportunities mediate condition effect?
• Rates of (a) teacher models, (b) student group practice opportunity and (c) individual student practice opportunities entered as mediators to determine if they decreased condition effect– Condition effect was still significant– No evidence to support this mediation hypothesis
![Page 31: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/31.jpg)
Secondary Analysis
• If student practice overall is not mediating impact, perhaps the value (quality) of practice differs by classroom and is related to condition
![Page 32: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/32.jpg)
Interaction between Rate of Practice and Condition
• By condition do rates of practice opportunities show the same pattern of impact on student outcomes?
• Are treatment – control differences on student outcomes similar in classrooms with high rates of practice vs. low rates of practice?
![Page 33: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/33.jpg)
Interaction between Rate of Practice and Condition
• Number of Classrooms by Treatment Condition and Median Rate of Individual and Group Practice Opportunities
• Practice Opportunity Quartiles in Rate per Minute
Control ELM Total
Above Median
6 26 32
Below Median
24 8 32
Total 30 34 64
Control ELM Total
Minimum 0.4 1.0 0.4
25th %ile 0.8 1.9 1.2
Median 1.3 2.3 1.9
75th %ile 1.7 3.0 2.4
Maximum 4.1 4.1 4.1
![Page 34: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/34.jpg)
TEMA Scores by Rate of Practice
• Within ELM condition– High-practice classrooms outperform low-practice classrooms– Difference = 2.59, t = 2.58, df = 29, p = 0.0151
• Within control condition– No difference between high- and low-practice classrooms– Difference = 0.39, t = 0.31, df = 29, p = .7575
• Within classrooms with an above-median practice rate– ELM classrooms (might) outperform control classrooms– Difference = 2.33, t = 1.85, df = 29, p = .0747
• Within classrooms with a below-median practice rate– No difference between ELM and control classrooms– Difference = 0.14, t = 0.14, df = 29, p = .8935
![Page 35: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/35.jpg)
EN-CBM Scores by Rate of Practice
• Within ELM condition– No difference between high- and low-practice classrooms– Difference = 9.83, t = 1.46, df = 29, p = 0.1538
• Within control condition– No difference between high- and low-practice classrooms– Difference = -7.59, t = -0.92, df = 29, p = .3635
• Within classrooms with an above-median practice rate– ELM classrooms outperform control classrooms– Difference = 19.37, t = 2.35, df = 29, p = .0257
• Within classrooms with a below-median practice rate– No difference between ELM and control classrooms– Difference = 1.95, t = 0.29, df = 29, p = .7731
![Page 36: Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes.](https://reader031.fdocuments.us/reader031/viewer/2022013011/56649efd5503460f94c11fa4/html5/thumbnails/36.jpg)
Next steps
• Have just completed implementation of Study 1 in Dallas, Texas
• Have just completed implementation of Study 2 in Oregon
• Will implement Study 2 in Dallas, Texas in 2010-11
• Ongoing analysis to investigate impact of condition and mediation and moderation variables associated with impact