Washington State LASER2008 Evaluation Report
Dave WeaverKelly Jarvis
RMC Research Corporation111 SW Columbia, Suite 1200
Portland, Oregon [email protected]
(800) 788–1887
2
Presentation Contents
• Analysis of 2007 Science WASL Results– School-Level Analysis– Student-Level Analysis– Longitudinal Analysis
• Module-Level Assessment Results• Sentinel Site Visits Preliminary Results• Recommendation• Upcoming Activities
3
Analysis of 2007 ScienceWASL Results
• Evaluation Question– To what extent did teacher professional
development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?
4
School-Level Analysis• Data Sources
– School-Level Science WASL Data for 2007– School-Level Mathematics WASL Data for 2006– School-Level Demographic Data– School-Level Professional Development (PD) Index
• Total PD Hours of Teaching Staff for 2-Years Per FTE
• Methods: Linear Regression Analysis – Dependent Variable:
• Percent of Students Who Met Science Standard on 2007 WASL– Independent Variable:
• School-Level 2-Year PD Index– Control Variables:
• Percent of Students Eligible for Free or Reduced Price Lunch (FRL)• Students Per Classroom Teacher
5
Student-Level Analysis• Data Sources
– 2007 Student-Level Science WASL Results for Grade 5 & 8 with Student Demographics
– 2006 Student-Level Mathematics WASL Results for Grades 4 & 7– Teacher-Level 3-Year PD Index
• Total PD Hours Teacher for 3-Years Prior to WASL
• Methods: Linear Regression Analysis– Dependent Variable
• 2007 Science WASL Scale Score
– Independent Variable• Teacher-Level 3-Year PD Index
– Control Variables• FRL• Previous Grade Mathematics WASL Scale Score
6
Student-Level Data Map
7
Longitudinal Analysis• Data Sources
– 2007 Student-Level Science WASL Results for Grade 8 with Student Demographics
– 2004 Student-Level Science WASL Results for Grades 5– School-Level 3-Year PD Index
• Total PD Hours Teacher for 3-Years Prior to WASL
• Methods: Linear Regression Analysis– Dependent Variable
• 2007 Science WASL Scale Score– Independent Variable
• School-Level 3-Year PD Index– Control Variables
• FRL• Previous Grade Mathematics WASL Scale Score
8
Longitudinal Data Map
9
Finding 1
• The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL).
• This finding is consistent with earlier studies.
10
School-Level Analysis Results
30.1%
33.8%34.6%
37.0%
20%
22%24%
26%
28%
30%32%
34%
36%38%
40%
Adjusted Means
2-Year PD Per FTE
Per
cent
of
Gra
de 5
Stu
dent
s W
ho
Met
/Exc
eede
d Sc
ienc
e St
anda
rd
Less Than or Equal to 3Hours
Greater Than 3 andLess Than or Equal to7.5 hours
Greater Than 7.5 andLess Than or Equal to13.4 hours
Greater Than 13.4hours
Means adjusted by previous year math scores,
class size, and FRL.
11
Longitudinal Analysis Results
394.8394.3
395.5
397.1
390391392393394395396397398399400
Adjusted Means
School-Level PD Index Category (Secondary)
Gra
de 8
Sci
ence
WA
SL S
cale
Sc
ore
Greater Than 0 andLess Than or Equal To1.25
Greater Than 1.25 andLess Than 3
Greater Than or Equalto 3 and Less than 6.3
Greater than or Equalto 6.3
Means adjusted by Grade 5 Science Scale Score, FRL,
Non-Minority, Special Education, and Gifted.
12
Finding 2
• Grade 5 teachers must have participated in at least 18 hours of professional development before an impact on student achievement was measurable, and the impact of the professional development on student achievement increased as the number of professional development hours increased.
Grade 5 Multiple Regression Analysis Across Teacher-Level PD Index Categories
PD Index Interval N Beta t p Adj. R2 SE
More than 0 but less than 9 hours
5,633 -.013 -1.410 .158 .542 14.682
At least 9 hours but less than 18 hours
5863 .011 1.279 .201 .584 14.105
At least 18 hours but less than 36 hours
7589 .025 3.176 .001* .538 15.318
At least 36 hours but less than 54 hours
2740 .034 2.658 .008* .547 14.710
54 hours or more 1366 .093 5.009 <.001* .534 15.010
NOTE: Dependent Variable is the Grade 5 science WASL scale score. Control variables are the previous year math scale score and FRL Results are weighted by the number of students assessed.* denotes significance at the .05 level.
14
Finding 3• Grade 5 teachers who had participated in
science professional development served:– A larger proportion of students eligible for free
or reduced-price lunch;– A greater proportion of Hispanic and Latino
students; and– A smaller proportion of White students.
• These differences may account for the differences in student achievement.
Variable No PD Any PD
Total Number of Students 41,400 30,006
Student Achievement
Mean 2007 Science WASL Score 391.17 389.53
Mean 2006 Math WASL Score 409.26 406.71
Ethnicity Hispanic/Latino 12.3% 17.6% White (Not Hispanic Origin) 69.0% 63.5%
Special Program
Free or Reduced Price Lunch 37.1% 43.1%
Differences in Demographics and Achievement of PD Participants vs. Non-Participants
16
Module-Level Assessment of Student Content Knowledge • Evaluation Questions
– What science content knowledge gains do students make as a result of their use of the inquiry-based instructional modules?
– What is the relationship between the science PD of the teacher and the gains in student content knowledge?
17
Module-Level Assessment of Student Content Knowledge
• Assessment– Obtained Permission to use Horizon Research
Assessment– Selected Horizon Research Instruments
• Human Body from FOSS• Levers and Pulleys from FOSS• Rocks and Minerals from STC• Electric Circuits from STC
18
Module-Level Assessment of Student Content Knowledge
• Administration Plan– Recruit teachers with a range of PD levels– Data collection for each student
• Pre and post student scores• Student demographics
– Gift certificate compensation to teachers– Data collected spring 2008
• 53 classes completed both pre and post assessments• 1087 students
56 Teachers RecruitedKit PD Level Teachers
Electric Circuits High (>54 hours) 12
Medium (>18 but <54 hours) 5
Low (<18 hours) 5
None 4
Total: 26
Human Body High (>54 hours) 3
Medium (>18 but <54 hours) 4
Low (<18 hours) 3
None 7
Total: 17
Rocks and Minerals High (>54 hours) 2
Medium (>18 but <54 hours) 5
Low (<18 hours) 5
None 1
Total: 13
20
Finding 4
• Students demonstrated significant improvement between the preassessment and the postassessment on all 3 modules: on average, student scale scores increased more than 19 points.
21
Module-Level Student Assessment Results
42.7 42.537.4
62.1 62.456.5
0102030405060708090
100
Electric Circuits Human Body Rocks and Minerals
Mea
n Sc
ale
Scor
e
Pre Post
22
Finding 5• The relationship between student performance and teacher
professional development was inconclusive.– The performance of the students whose teachers participated in
fewer than 18 hours of professional development was significantly lower than the performance of students whose teachers participated in 18 or more hours professional development.
– Students whose teachers did not participate in professional development outperformed students whose teachers participated in some professional development.
• Additional data will be collected during the 2008–2009 school year.
Student Assessment Gains as a Function of Professional Development
24
Sentinel Site Selection• Schools With Significant LASER Participation• Identified 36 Schools
– 13 Schools Refused Participation– Added 11 New Schools
– Visited 34 Schools
• Defined 2 Groups of Schools Based on Science WASL Change Between 2006 and 2007 – Demonstrated Significant Positive Gains– Demonstrated Little, No, or Negative Gains
25
Sentinel Site Visits• 10 Site Visitors• Each site visit: 1 ½ to 2 days
– Principal Interview– At Least 5 Teacher Survey & Interviews– At Least 3 Classroom Observations
• Data Collection Instruments– http://www.rmccorp.com/LASERSiteVisits/
• Conducted 3 web-based training session• Round 1: Spring 2008• Results Reported As Rubric Scores
26
Finding 6• The site visitors gave the vast majority of the
schools visited very high ratings in several areas indicating that the schools served by the LASER Alliances had:– Successfully established a core sequence of inquiry-
based instructional modules across all of the grade levels, and
– Had materials support system that delivered modules to teachers on time and ready to use.
27
Finding 7
• There were several significant differences between schools whose students demonstrated increased performance on the science WASL compared to schools whose students did not.
28
Schools with Grade 5 Students• Schools whose Grade 5 students’ science
WASL scores increased had significantly greater:– Participation in professional learning communities– Time during the workday for school-based
professional development– Evidence of positive parent and community support– Evidence of teachers integrating literacy with science
29
Schools with Grade 8 Students• Schools whose Grade 8 students’ science WASL
scores increased had:– Time during the work day for school-based professional
development– Greater participation in professional development that
involved examining student work– Lessons that more often engaged students in
discussion based on scientific evidence– Lessons whose content was more significant, accurate
and worthwhile
30
Schools with Grade 8 Students (cont.)• Schools whose Grade 8 students’ science WASL scores
increased had:– Lessons that more often built on prior ideas and
experiences– Lessons whose content was more closely connected to
the lesson activities– Lessons that were effective at fostering student
understanding– Lessons that included mechanisms to monitor students’
understanding of science content– Lessons that helped students make sense of scientific
concepts– Lessons that routinely provided opportunities for students to
reflect on their new understanding
31
Finding 8• The schools that demonstrated an increase in the
percentage of students who met the science standards had teachers with significantly fewer years of education experience than the schools with no gains.
• This difference was more pronounced in the schools that served Grade 8 students.
– Testing for demographic differences between the schools that improved their science WASL scores and schools that did not could not account for the findings. Factors examined include:
• School size• Ethnic/racial composition of the student population• Program membership of the students• Teacher education and experience
32
Conclusion• The infrastructure to support the use of a core
curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited.
• Although these conditions are necessary for the implementation of inquiry-based science instruction, they are not sufficient to raise student achievement as measured by the science WASL.
33
Recommendation 1• Ensure that the professional development on
research-based instructional practices is consistent and explicit across all of the LASER Alliances– Help teachers understand the elements of effective
science instruction and use the modules as a means of carrying out the element with their students.
34
Elements of Effective Science Instruction• Intrinsic Motivation
– Relevant, interest to student, resolution of dissonance• Eliciting Student’s Prior Knowledge
– Articulating what they already believe about a concept• Intellectual Engagement
– Students do the thinking• Use of Evidence
– Discourse: Justifications and explanations based on evidence• Sense-Making
– Closure, meta-cognition, reflection
Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction.
35
Recommendation 2
• Increase support for school-based professional development that helps teachers:– Assume accountability for student learning
that results from the use of the modules, and
– Collaboratively implement the elements of effective science instruction.
– Ample structure and leadership for success
36
Evaluation Activities for 2008-09• Sentinel Site Visits (30 to 35)• Student Content Assessments
– Fall, Winter, & Spring– 7 Instructional Modules
• Analysis of 2008 WASL Data• Online Surveys
– School Principal Survey– Alliance Directors
• PD Provider Online Surveys (3)
Rocks & Minerals
Human Body
Electric Circuits
Motion & Design
Levers & Pulleys
Environments
Variables
37
ASK-IT Schools Show What Science Engaged Schools Can Do
ASK IT schools had teachers who: Used formative assessment to
determine student understanding of key concepts
Met in professional learning communities to inform their teaching based on student data
Increased their science teaching time so most teachers completed each unit.
Environments Proficiencies 5th Grade Matched Pair 1
-2
-1.5
-1
-0.5
0
Pre Post
Treatment
Control
Human Body Overall Proficiencies
-2.5
-2
-1.5
-1
-0.5
0
Pre Post
Treatment
Control
Magnetism & Electricity Overall Proficiencies
-2
-1.5
-1
-0.5
0
0.5
Pre Post
Treatment
Control
38
Questions ???Dave Weaver
RMC Research Corporation111 SW Columbia, Suite 1200
Portland, Oregon [email protected]
(800) 788–1887
Top Related