How Big was Response Bias in England to PISA 2003? John Micklewright & Sylke V. Schnepf July 2008.
-
Upload
todd-owens -
Category
Documents
-
view
213 -
download
0
Transcript of How Big was Response Bias in England to PISA 2003? John Micklewright & Sylke V. Schnepf July 2008.
How Big was Response Bias in England to PISA 2003?
John Micklewright & Sylke V. Schnepf
July 2008
2
Motivation• PISA – Programme of International Student Assessment
• Reports from OECD for 2003 exclude the UK due to perceived non-response bias in England
• Simon Briscoe, Economics Editor at The Financial Times: the exclusion is among the ‘Top 20’ recent threats to public confidence in official statistics in the UK
Presentation draws on:
• Response Bias in England in PISA 2000 and 2003, DfES Research Report 771
3
PISA response rates (%): England
2000 2003
England OECD average
England OECD average
School response before replacement
59 86 64 90
School response after replacement
82 92 77 95
Student response 81 90 77 90
4
‘Bias’ in what?
• Mean
• Variance
• % beneath a given threshold
5
Five groups of 15 year olds
i. all pupils in England (less permitted exclusions from the
target population)
ii. all pupils in sampled schools (initial, 1st repl., 2nd repl.)
iii.all pupils in responding schools
iv. sampled pupils in responding schools
v. responding pupils
6
What we do
• Show how (i) mean, (ii) variance, and (iii) % <
thresholds of KS3 and KS4 scores change from
group to group
• Estimate biases in these 3 measures of PISA scores in group v, focusing on problem of pupil response
• Look at both 2000 and 2003
7
KS4 v. PISA scores - respondents
R = 0.78
KS4 score
PISA maths score
8
Means and SDs of KS4 scores, 2003
9
% below or above KS4 thresholds, 2003
group % < bottom decile, KS4 score, group (i)
% > top decile of KS4 score,
group (i) i 10.0 10.0 ii 9.7 10.5 iii 9.5 9.3 iv 7.6 9.3 v 4.9 9.9
10
Creating response weights
Model probability of pupil response for sampled pupils in responding schools (group iv)
Prob(response)i = F(domestic test scorei, other characteristicsi)
Predicted probability of response
KS4 score
11
Bias due to pupil response
reading maths science
Mean +6.5 +6.2 +6.9
SD -3.6 -2.9 -3.0
Extent of bias, calculated as number of PISA score points
(PISA international mean = 500, international SD = 100
Compare with the estimated standard errors (SEs) of the mean and SD for England in 2003:
SE (mean) = 2.9 (reading and maths) and 3.0 for science
SE (SD) = 1.6 (reading and maths) and 1.7 for science
12
However…
• Adjusting the mean by the estimated bias of about 6 points moves England one place in the country rankings
• ‘Post-stratification’ weights are used in many surveys – they could be provided with the data for England
• Think of possible biases in countries just above the 80% student response threshold (Australia, Austria, Canada, Ireland, Poland and the USA)
13
Conclusions
1. It is possible to assess reliably both the direction and magnitude of response biases in England in 2003, and hence compare UK scores with those for other countries
2. The extent of the bias in means and SDs of scores is equal to about double the SEs, but the impact on ‘league table’ rankings is small
3. Biases were similar in 2000, when pupil response in England just reached the required threshold