A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut...
-
Upload
clarence-nelson -
Category
Documents
-
view
215 -
download
1
Transcript of A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut...
![Page 1: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/1.jpg)
A Statistical Linkage Between
NAEP and ECLS-K Grade Eight Reading Assessments
Enis Dogan
Burhan Ogut
Young Yee Kim
Sharyn Rosenberg
NAEP Education Statistics Services InstituteAmerican Institutes for Research
A Statistical Linkage Between
NAEP and ECLS-K Grade Eight Reading Assessments
Enis Dogan
Burhan Ogut
Young Yee Kim
Sharyn Rosenberg
NAEP Education Statistics Services InstituteAmerican Institutes for Research
![Page 2: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/2.jpg)
Purpose
In Spring of 2007, about 1300 students took the NAEP and ECLS-K grade eight reading assessments
The purpose of this study is to establish a statistical link between these two assessments using this common sample
We also test the validity of the link by comparing the model projected results to reported NAEP results
![Page 3: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/3.jpg)
BACKGROUND
![Page 4: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/4.jpg)
Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K)
Provides longitudinal data on students’ academic achievement in reading, mathematics, and science
Began in the fall of 1998 with a national sample of 21,000 kindergartners
Each student is tested by a series of two-stage, 30-minute adaptive tests, administered face to face, on seven occasions
Item parameters and individual scores are estimated using IRT
![Page 5: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/5.jpg)
NAEP 8th Grade Reading 2007
Provides cross-sectional data on students’ academic achievement in reading
Nationally representative sample of more than 350,000 students participated in the 2007 reading assessment
Each student takes just a portion of the test (BIB Design), consisting of two 25-minute sections or one 50-minute section
Item parameters are estimated using IRT, but results are reported at group level. NAEP does not report individual scores
![Page 6: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/6.jpg)
Differences between NAEP and ECLS-K
The two assessments have similar frameworks, but do not share any common items
NAEP uses a nationally representative sample ECLS-K is representative of children enrolled in first grade
in the U.S. in 1999-2000
![Page 7: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/7.jpg)
ESTIMATING THE PROJECTION EQUATION
![Page 8: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/8.jpg)
Linking through a common sample
In Spring of 2007, about 1300 public school students took NAEP and ECLS-K grade 8 reading assessments. o About ¼ were part of the operational NAEP sample o The rest took the NAEP assessment for the purposes of
the common sample linking study
![Page 9: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/9.jpg)
Method and challenges
NAEP scores in the form of plausible values were not available for about ¾ of the linking sample since they were not part of the operational NAEP assessment
Using item level data from the entire common sample, MML regression was used (with the AM software) to estimate a projection equation predicting NAEP score distributions from ECLS-K scores
The projection equation was
where y is the NAEP score and x is the ECLS-K score
i i iy a bx e
![Page 10: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/10.jpg)
The projection equation
The projection equation was estimated using two different weights:o the original ECLS-K sampling weights, and o poststratification weights
Parameter estimates predicting NAEP scale scores from ECLS-K scale scores
1 Using original weights: RMSE = 18.84, F(1,168) = 326.47
2 Using poststratification weights: RMSE = 18.60, F(1,168) = 370.64
Parameter Original estimate1 Poststratified estimate2
Intercept 79.32 (10.46) 75.86 (9.92)
Slope 1.10 (0.06) 1.12 (0.06)
![Page 11: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/11.jpg)
Variance of the projection
For the projection equation, the a, b parameter estimates are denoted
and their covariance as
The projected NAEP average reading score is
The variance of the projection at mean is
i i iy a bx e
2
2
ˆ ˆˆˆ( , )ˆ ˆa ab
ab b
Var a b
ˆˆ ˆy a bx
2 2 2 2ˆˆ ˆ ˆ ˆVar 2 Vara ab by x x b x
![Page 12: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/12.jpg)
TESTING THE VALIDITY OF PROJECTION EQUATION
![Page 13: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/13.jpg)
Comparing reported and projected NAEP mean
National average scores and 95% confidence intervals for projected and reported 2007 NAEP reading scores: grade 8, public
![Page 14: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/14.jpg)
Scores by gender and race/ethnicity
Left bar: Confidence Interval for the mean based on the projection using the original weights
Right bar: Confidence Interval for the reported NAEP mean
N
AE
P S
cale
![Page 15: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/15.jpg)
AN APPLICATION OF THE LINK: READING PERFORMANCE AT EARLIER GRADES AND PROFICIENCY IN NAEP
![Page 16: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/16.jpg)
What is the relationship between reading performance at first, third and fifth grades and Proficiency in eighth-grade NAEP reading assessment?
Early childhood reading performance and proficiency in NAEP
![Page 17: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/17.jpg)
Show an overall understanding of the text, including inferential as well as literal information
Extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences—including other reading experiences
Identify some of the devices authors use in composing text
Proficiency in eighth-grade NAEP reading assessment
![Page 18: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/18.jpg)
Level 1 : Letter recognition: identifying upper- and lower-case letters by name
Level 2 : Beginning sounds
Level 3 : Ending sounds
Level 4 : Sight words
Level 5 : Comprehension of words in context
Level 6 : Literal inference
Level 7 : Extrapolation
Level 8 : Evaluation
Level 9 : Evaluating nonfiction
Level 10: Evaluating complex syntax: evaluating complex syntax and understanding high-level nuanced vocabulary in biographical text.
ECLS-K Reading performance levels
![Page 19: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/19.jpg)
Percentage of students at Levels 5 through 9: Grade 5
Reading performance at grade 5 and Proficiency in 8th grade NAEP
![Page 20: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/20.jpg)
Percentage of students at Levels 5 through 9: Grade 5
Reading performance at grade 5 and Proficiency in 8th grade NAEP
33 %
48 %
72 %Evaluation: demonstrating understanding of author’s craft …
13 %
![Page 21: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/21.jpg)
Percentage of students at Levels 4 through 8: Grade 3
Reading performance at grade 3 and Proficiency in 8th grade NAEP
![Page 22: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/22.jpg)
Percentage of students at Levels 4 through 8: Grade 3
Reading performance at grade 3 and Proficiency in 8th grade NAEP
11 %25 %
46 % 62 %
Extrapolation: identifying clues used to make inferences, …
![Page 23: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/23.jpg)
Percentage of students at Levels 3 through 7: Grade 1
Reading performance at grade 1 and Proficiency in 8th grade NAEP
![Page 24: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/24.jpg)
Reading performance at grade 1 and Proficiency in 8th grade NAEP
13 %
34 %
49 %
73 %
85 %
Percentage of students at Levels 3 through 7: Grade 1
Comprehension of words in context: reading words in context
![Page 25: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/25.jpg)
Summary of findings
Using a common sample, a linking equation was estimated
Using the equation, projected NAEP scores were computed for ECLS-K 8th graders
Mean projected NAEP scores for the nation and the gender and racial/ethnic groups were close to reported NAEP results
Mean projected NAEP scores were used to examine the relationship between proficiency in grade eight NAEP reading and earlier reading performance
![Page 26: A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649f515503460f94c749e1/html5/thumbnails/26.jpg)
Limitations
Composition of the ECLS-K sample and the
population it represents
Projection results for Hispanic students
Future analyses
Further validation of the link
Modeling growth in reading on the NAEP scale
Discussion