Sept. 29 th, 2005 Investigating Learning over Time Mingyu Feng Neil Heffernan Longitudinal Analysis...
-
date post
21-Dec-2015 -
Category
Documents
-
view
212 -
download
0
Transcript of Sept. 29 th, 2005 Investigating Learning over Time Mingyu Feng Neil Heffernan Longitudinal Analysis...
Sept. 29th, 2005
Investigating Learning over Time
Mingyu FengNeil Heffernan
Longitudinal Analysis on Assistment Data
Purpose
To answer two type of questions: Do students learn over time?
To characterize each person’s pattern of change over time
Within-individual change over time
Do students differ on learning rate? If yes, what impact learning?
To examine if people differ on within-individual change and ask for the association between predictors and patterns of change
Inter-individual differences in change
Important Features of Study of Change
Three or more waves of data Why three?
Describe process the change Tell the shape of individual growth trajectory Is change steady?
The more the better A sensible metric for time
Make sure you still got enough waves of data An outcome whose values change
systematically over time
Method
Our work follows the approach presented in Applied Longitudinal Data
Analysis (Modeling change and event occurrence), Singer & Willett (2003)
Statistical software
package - SPSS was used to run all the analysis.
Data from the Assistment System
Log data of year 2004 841 students 2 Worcester Public Schools 8 Teachers with 4 from each school
Students went to labs about every other week from Sept., 04 to June,05 On average, 5.70 measurement occasions ranges from 1 to 9
The Three Features in the Data
Longitudinal data - “Person-period” structured Each student has multiple records-one for each
measurement occasion Metric for time
“CenteredMonth” – Month centered around Sept. (value = # of months since Sept. [0 … 10])
multiple sessions in one month are aggregated into one
The outcome % correct on the main question. Transformed to “MCASScore”
MCASScore = % correct * 54 (full score of MCAS test)
Sample Data
2 73 n n 104 y 74 1 950 m n n 2 16.61538
3 73 n n 104 y 74 1 950 m n n 3 20.25
4 73 n n 104 y 74 1 950 m n n 5 41.72727
5 73 n n 104 y 74 1 950 m n n 6 23.14286
6 73 n n 104 y 74 1 950 m n n 7 32.4
7 73 n n 104 y 74 1 950 m n n 8 19.63636
9 73 n n 104 y 74 1 951 f y y 2 18
10 73 n n 104 y 74 1 951 f y y 3 33.42857
11 73 n n 104 y 74 1 951 f y y 5 31.90909
School ID Teacher ID Class ID Class Level
Student ID Gender Free lunch?
Special Ed.Centered
Month
MCASScore
Note: 1. class level was determined class average initial score in Oct. : If avg (class score in Oct.) > global mean, then class_level = 1; else class_level = 0.2. Given the way class level is calculated, we filtered out data waves in Sept. and Oct.
Explore the data set
Mean (MCASScore) increased across time
0 1 2 3 4 5 6 7 8 9
CenteredMonth
0.00
5.00
10.00
15.00
20.00
25.00
30.00
Me
an
MC
AS
Sc
ore
School 73 School 75
Mean24.3295 20.6751
Std. Dev. 13.69403 13.60057
T-test showed that students from School 73 has got significant higher scores (p < .001)
Individual Change over Time
Empirical growth plots for 24 students
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
239 240 243 244 245
246 247 248 314 315
316 320 321 327 331
666 667 668 669 805
806 807 809 810
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0 2 4 6 8
CenteredMonth
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0 2 4 6 8
CenteredMonth
0 2 4 6 8
CenteredMonth
0 2 4 6 8
CenteredMonth
Individual Change over Time
Smooth nonparametric summaries of how individuals change over time
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
239 240 243 244 245
246 247 248 314 315
316 320 321 327 331
666 667 668 669 805
806 807 809 810
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0 2 4 6 8
CenteredMonth
0.00
9.00
18.00
27.00
36.00
45.00
54.00
MC
AS
Sco
re
0 2 4 6 8
CenteredMonth
0 2 4 6 8
CenteredMonth
0 2 4 6 8
CenteredMonth
Individual Change over Time
Fitted trajectories (linear regression line)
Mean (Intercept) = 16.555Mean (Slope) = 1.4586 > 0Correlation (intercept, slope) = -.81
Parameters of regression line for some students
Use Multilevel Model for Change
We need a model that embodies two types of research questions Level-1 question about within-person change Level-2 question about between-person
differences Multilevel statistical model
Level-1 submodel that describes how individuals change over time
Level-2 submodel that describes how these changes varies across individuals
A multilevel model Level-1 submodel (individual growth model):
Level-2 submodel:
Composite model:
ijijiiij TIMEY 10
iii LEVEL 111101 iii LEVEL 001000
InterpretationYij : score for person i at time j, a linear function of TIMEij (CenteredMonth here)
∏0i/ ∏1i : Intercept/Slope of true change trajectory of student i (∏0i :score in Sept.)
r00/r10: Population average of level-1 intercept/slope
r01/r11: Population average difference in ∏0i/ ∏1i for a unit difference in level-2 predictor (LEVELi)
εij: random measurement error for person i at occasion j
ζ0i, ζ1i: parameter residual which permits the level-1 parameters of one person to differ stochastically from those of others
)*(*** 1011100100 ijiiijijiijiij TIMETIMELEVELTIMELEVELY
2110
0120
1
0 ,0
0~
Ni
i
2,0~ Nij
Fixed effect
Random effect (Variance
Components)
Mixed Effect Model
Baseline: Unconditional means model
Just means and variations (no predictor)
Fit Multilevel Model to Assistment Data
ijiijY 0 ii 0000
Estimates of Fixed Effects (a)
Parameter Estimate Std. Error df t Sig.
95% Confidence Interval
Lower Bound Upper Bound
Intercept 24.1687396 .3258244 768.537 74.177 .000 23.5291282 24.8083511a Dependent Variable: MCASScore.
This tells us MCASScore varies over time and students differ from each other on MCASScore (p < .001)– sufficient variation at both levels to warrant further analysis
Estimates of Covariance Parameters(a)
Parameter Estimate Std. Error Wald Z Sig. 95% Confidence Interval
Lower Bound Upper Bound
Residual 126.8945794 3.1537978 40.235 .000 120.8613877 133.2289377
Intercept [subject = studentID]
Variance55.8742819 4.2160555 13.253 .000 48.1929604 64.7799047
estimated overall average MCASScore
within-person variancebetween-person
variance
More models
Introduce first predictor: TIME Unconditional growth model
This factor is important: BIC diff = 84
Try more factors as predictors school, teacher, class, class_level
ijijiiij TIMEY 10 ii 0000 ii 1101
Estimates of Fixed Effects(a)
Parameter Estimate Std. Error df t Sig. 95% Confidence Interval
Lower Bound Upper Bound
Intercept 20.8622499 .5366397 677.170 38.876 .000 19.8085722 21.9159277
CenteredMonth .6415105 .0948756 630.321 6.762 .000 .4552001 .8278210
estimate of students
average MCASScore in Sep.
Average score increase .64 points
every month
Reject hypothesis of no relationship between
score and TIME
What did We Learn?
School matters? Teacher Class Class level
Fit Multilevel Model to Assistment DataModel Predictors BIC #params
Model A 31711.79 3
Model B CenteredMonth 31627.67 6
Model D CenteredMonth+SchoolID 31616.67 8
Model E CenteredMonth+TeacherID 31671.87 20
Model F CenteredMonth+ClassID 31668.08 70
Model K CenteredMonth+ClassLevel 31457.92 8
Model L CenteredMonth+ClassLevel+SchoolID 31454.602 10
Model M CenteredMonth+ClassLevel+SchoolID (intercept) 31449.059 9
Model N CenteredMonth+ClassLevel+TeacherID 31516.433 22
Model O CenteredMonth+ClassLevel+TeacherID(intercept) 31485.309 15
Model A
Model B
Model D
Model E
Model F
Model K
Model L
Model M
Model N
Model O
31300 31400 31500 31600 31700 31800
Model A
Model D
Model F
Model L
Model N
* BIC of Model M (highlighted in bold) is the lowest among all models A through O. * Predictor: TIME, SCHOOL and Class_Level* School was only used as predictor of intercept (changing rate is not distinguishablebetween schools (p > .05, ns)
Result of Model M Tells
Interpretation Estimate of average initial score of students from lower level classes of school 73 is
17.1389; the score is 14.7419 for students from lower level classes of school 75 From higher level class adds 9 points to average initial score Estimate of change rate of lower level classes is .8173 It seems students from higher level classes learns slower (.3473 points lower every
month) How to use this to calculate student’s score?
Students from 73: Score at Month = 17.1389+9.45*Level + (0.817-0.347*Level)*Month Students from 75: Score at Month = 14.7419+9.45*Level + (0.817-0.347*Level)*Month
Estimates of Fixed Effects (a)
Parameter Estimate Std. Error df t Sig. 95% Confidence Interval
Lower Bound Upper Bound
[schoolID=73.0] 17.1389235 .7693093 868.727 22.278 .000 15.6290012 18.6488458
[schoolID=75.0] 14.7419300 .7969344 968.325 18.498 .000 13.1780125 16.3058475
ClassLevel 9.4549425 1.0083900 678.784 9.376 .000 7.4750039 11.4348810
CenteredMonth .8172511 .1385169 755.640 5.900 .000 .5453273 1.0891749
CenteredMonth * ClassLevel
-.3473341 .1899717 650.737 -1.828 .068 -.7203657 .0256974
a Dependent Variable: MCASScore.
A New Data Set Include transfer model information
Does learning rate differ on knowledge components?
Use the basic model: “MCAS5” Outcome: MCASScore for a single standard Time metric: season ( every 3 months)
For more stable estimate of student performance on different knowledge status
Include pretest score from 09/2004 Paper and pencil test given in original format of MCAS ’04
Sample Data (II)
schoolID Teacher ID Class ID studentID Season KC Name MCASScore Pretest
73 104 74 950 0 G-Geometry 30.375 8
73 104 74 950 0 M-Measurement 27 8
73 104 74 950 0 N-Number-Sense-Operations 34.2692 8
73 104 74 950 0 P-Patterns-Relations-Algebra 24.3 8
73 104 74 950 0 D-Data-Analysis-Statistics-Probability 40.5 8
73 104 74 950 1 G-Geometry 43.2 8
73 104 74 950 1 N-Number-Sense-Operations 45 8
73 104 74 950 1 P-Patterns-Relations-Algebra 28.4211 8
73 104 74 950 1 D-Data-Analysis-Statistics-Probability 36 8
73 104 74 950 2 G-Geometry 27 8
73 104 74 950 2 M-Measurement 0 8
73 104 74 950 2 P-Patterns-Relations-Algebra 33.9429 8
For one student: 950
Fit Multilevel Model to Assistment Data (II)
MODEL BIC#param
s Predictors
Model A2 66207.548 3
Model B2 66016.383 6 Season
Model C2 65406.461 10 season + KC name (intercept)
Model C2' 65722.122 10 season + KC name (slope)
Model D2 65287.17 14 season + KC name
Model E2 44588.375 8 season + pretest
Model F2 44580.103 7 season + pretest (intercept)
Model G2 44042.376 15 season + pretest (intercept) + KC name0 20000 40000 60000 80000
Model A
Model B
Model C
Model C'
Model D
Model E
Model F
Model G
TIME is still significant (BIC diff = 191) Knowledge Components as a predictor lead to a big improvement (more
than 700 BIC decrease) Pretest is a even better differentiator (see the big gap between Model E
and Model D)
Why?
Result of Mode G2Estimates of Fixed Effects(a)
Parameter Estimate Std. Error df t Sig. 95% Confidence Interval
Lower Bound Upper Bound
[KCName=D-Data-Analysis-Statistics-Probability]
9.2048856 .9340674 1058.357 9.855 .000 7.3720511 11.0377201
[KCName=G-Geometry] 12.6313817 .9237818 1020.453 13.674 .000 10.8186526 14.4441108
[KCName=M-Measurement ] 8.0682218 .9397700 1082.318 8.585 .000 6.2242445 9.9121992
[KCName=N-Number-Sense-Operations ] 18.8854008 .9026647 937.580 20.922 .000 17.1139238 20.6568779
[KCName=P-Patterns-Relations-Algebra ] 19.0073128 .9173294 988.721 20.720 .000 17.2071766 20.8074491
PretestScore .6272369 .0438227 498.598 14.313 .000 .5411371 .7133367
Season ([KCName=D-Data-Analysis-Statistics-Probability])
5.2260148 .5272740 3239.974 9.911 .000 4.1921905 6.2598390
Season ([KCName=G-Geometry ]) -.0723080 .5603555 3488.254 -.129 .897 -1.1709659 1.0263499
Season ([KCName=M-Measurement]) 1.4231592 .6093210 3864.869 2.336 .020 .2285379 2.6177805
Season ([KCName=N-Number-Sense-Operations])
2.5131848 .4729417 2715.492 5.314 .000 1.5858227 3.4405468
Season ([KCName=P-Patterns-Relations-Algebra ])
-1.3867194 .4529750 2490.712 -3.061 .002 -2.2749657 -.4984732
a Dependent Variable: MCASScore.
The estimate of average initial score
on “Patterns” is 19.007, the highest
Pretest score increases by 1, the
estimated initial score increases by .627
estimate of average rate of change on Data-Analysis is
5.226
Negative learning rate indicates “un-learning”
Future work
Try other predictors Gender, class_level, finer grained
transfer models Use fitted model to predict post-test
score or even further real MCAS score
Introduce “Assistment” metrics Performance on “scaffolds”, #hints… Weigh outcome by time spent
Thank you!
Details about this analysis are available at http://www.cs.wpi.edu/~mfeng/analysis