Overview of ICCS field trial National Research Coordinators Meeting Windsor, June 2008.
-
Upload
marsha-floyd -
Category
Documents
-
view
217 -
download
0
Transcript of Overview of ICCS field trial National Research Coordinators Meeting Windsor, June 2008.
Overview of ICCS field trial
National Research Coordinators Meeting Windsor, June 2008
NRCMeetingWindsor
June 2008
Content of presentation
• Participation in field trial• General outcomes of field trial• Feedback from national centres
– Summary in NRC(June08)1.pdf
• Problems identified in field trial• Consequences of field trial
NRCMeetingWindsor
June 2008
Field trial outcomes
• Field trial data analysis– revealed that generally the instruments worked
well• Cognitive test well targeted to ability & reliable• Questionnaire scales reliable and fit structure
– informed the selection of item material for main survey
– confirmed general feasibility of on-line surveys for teacher and schools
• Review of implementation procedures– Generally procedures worked well– Confirmed the feasibility of using regional
instruments– Some possibilities for improvement &
streamlining
NRCMeetingWindsor
June 2008
Field trial participation
• 32 countries undertook field trial
• One country to field trial in June 2008
• 31 data sets included in international field trial analysis report
• Approximately– 19,000 students – 9,000 teachers– 700 schools
NRCMeetingWindsor
June 2008
School response rates
• 21 countries had school response rates of at least 85 percent BEFORE replacement
• 4 countries had school responses rates of at least 85 percent AFTER replacement
• Only 7 countries had school response rates below 85 percent
• Period for field trial close to the end of the year contributed to problems with school participation
NRCMeetingWindsor
June 2008
Student response rates
• Overall, about 90 percent of students participated within schools
• Response rates (slightly) below 85 percent were found in only two countries
NRCMeetingWindsor
June 2008
Teacher response rates
• Across countries, there was a median teacher response rate within schools of 90 percent
• When combined with school response rates the overall response rate for the teacher survey was 89 percent (median across countries)
• In 10 out of 31 countries the overall response rate was below 75 percent
NRCMeetingWindsor
June 2008
Results of NRC Survey
• Overall satisfaction with manuals
• Main problems encountered:– Convincing schools to participate– Explaining sampling procedures– Implementing some test administration
procedures– Reaching a high student and teacher
participation within schools
NRCMeetingWindsor
June 2008
Issues raised regarding survey instruments
• Length of student test and questionnaires (7 countries)
• Questions in instruments seen as too difficult for students of this age (4 countries)
• General directions too long and repetitive (5 countries)
NRCMeetingWindsor
June 2008
Issues raised regarding the teacher sampling design
• Listing of all teacher teaching in target grade seen as a problem
• Two national centres saw the procedures as a major obstacle for school participation
• Data about teachers’ month/year of birth seen as inappropriate
NRCMeetingWindsor
June 2008
Issues raised regarding teacher participation
• Teachers not teaching CCE saw questionnaire as irrelevant to them
• Two national centres viewed questionnaire as too long
• Teacher participation more difficult to obtain than student participation
NRCMeetingWindsor
June 2008
Issues raised regarding regional modules
• Problems with layout of European instrument (part 2 being upside-down and back-to-front)
• Problems with administration and length of general directions
NRCMeetingWindsor
June 2008
Issues raised regarding manuals and software
• More than 80% found the manuals “very helpful” but suggestions to:– Simplify and shorten manuals – Refine software tools for sampling and
data entry
• Need to improve the scoring guides for coding of open-ended questions
NRCMeetingWindsor
June 2008
Issues identified by the ISC
• Need for improvement of adaptation notes
• Provision for countries to record their national adaptations to general directions
• Inclusion of checklist in manual for submission for verification
NRCMeetingWindsor
June 2008
Looking ahead
• Reduction in assessment time– Testing time now 45 minutes– Student questionnaire time now about 35
minutes– Regional instruments shorter– Some reduction in teacher and school
questionnaires
• Reduction in teacher sample size– Cluster size of 15 for main survey– On average, no more than 15 teachers in target
population at schools
NRCMeetingWindsor
June 2008
Expert advice
• Project Advisory Committee Meeting 7-8 June here in Windsor
• Discussion of outcomes and proposals
• Some changes to the proposed instruments following PAC advice– PAC suggestions will be included in
presentations
NRCMeetingWindsor
June 2008
Questions or comments?