Data Management Seminar, 8-11th July 2008, Hamburg 1 ICCS 2009 – On-line Data Collection in the...

Data Management Seminar, 8-11th July 2008, Hamburg1

ICCS 2009 – On-line Data Collection

in the Main Survey


Content

Field Trial ExperienceIntegration with Main Survey OperationsGuidelinesRestrictionsMain Survey Time Planning


Field Trial Experience

In general, Field Trial analysis showed that on-line data collection can be used as an optional mode for the main survey.During the NRC meeting in Windsor, individual feedback was given including recommendations for the main survey.


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring

Done by SurveySystem.

Convert for ODC after process is complete

for paper instruments.

Determine and track administration mode.

Monitor participation on-line

and follow-up.

No data to enter, just participation

status.

Send “cover letters” instead of

questionnaires.

No data to send. It’s at the IEA DPC already.


Integration – Instruments

Instrument Preparation– International definition files prepared by IEA DPC.– National adaptations to structure within

countries.– “Copy and Paste” conversion AFTER paper

translation, within country and international translation verification, and layout verification.


Integration – Instruments

Instrument Verification– Within country content comparison and optical

check (final paper versions vs. on-line)– NRCs send localized definitions to IEA DPC– Final optical / logical checks against paper and

NAF– Approval or request for corrections from IEA DPC– IEA DPC sets up survey for the Internet

Codebook Adaptation– WinDEM codebooks are automatically created!


Integration – Sampling

School sampling is not affected in terms of coverage and selectionWithin-School Sampling– WinW3S allows you to assign the

questionnaire type (paper or on-line) per default for principals and/or teachers

– WinW3S allows you to track the questionnaire type for individuals that require the other type (e.g., a school that was supposed to fill all questionnaires on-line, but in which some teachers requested paper questionnaires)


Integration – Administration

On-line respondents, usually entire schools, receive “cover letters” instead of questionnaires.Same WinW3S labels used.Credentials created with WinW3S are distributed to respondents together with a culture-specific and typo-safe login address, for instance: http://www.iccs2009.net/odc/71792Administration period applies to both modes.


Integration – Data Entry and Verification

Trivial: No need to enter data!Track participation in WinW3S from downloadable ‘Monitor’ reports (“returned, data available” vs. “no data available”) and teacher tracking forms (for other participation indicators, like “left school permanently”).‘On-line’ data merged with ‘paper’ data from WinDEM at IEA DPC.Verification: Done at the IEA DPC.


Guidelines

Overall goal: highest possible response rates, both on questionnaire as well as variable level.Decision and responsibility of usage lies with NRCs, who are generally free to decide about type and scope of implementation, i.e. as the default mode, for principals and/or teachers, or for specific schools or individuals.


Guidelines

Individual restrictions in terms of computer/Internet penetration, poor on-line response rates in previous surveys or other obstacles should be taken into account.Eventually, the results of the field trial in terms of response rate and drop-out should become the basis for decision.


Restrictions

On-line data collection is administered with the IEA SurveySystem ONLY. The IEA DPC can not manage the verification and data conversion for other systems.


Restrictions

On-line mode may become the default mode in your country for the MS, but it is MANDATORY to provide paper questionnaires to individuals/schools without Internet access, equipment, or that simply refuse.For the MS National Centers should therefore check with schools whether the on-line mode is acceptable for them BEFORE send-out of materials. In other words, school coordinators should not need to ask for paper questionnaires if they prefer them. Enquiries should take place during the first contact/notification of schools or together with the send-out of teacher listing forms.


Main Survey Time Planning

Documentation of software is part of the ICCS 2009 Survey Operations Procedures Manual Unit 3.International definition files distributes together with SurveySystem software on August 18, 2008.“Copy-and-Paste” conversion may start AFTER translation AND layout verification for paper questionnaires is completed.


Main Survey Time Planning

Submit zipped definition files to IEA DPC after layout verification approval, and latest two week before the first day of survey administration.Allow at least one week for review and paper/on-line comparison at IEA DPC.On-line administration at the same time as paper administration.


Summary

Participation is optional and should be guided by the level of confidence that participants have with the mode.Fall-back paper questionnaires are mandatory (ODC can not be the exclusive mode for the time being).More tasks going on in sequence and parallel in terms of preparation and monitoring.Challenge to assign and track the correct questionnaire mode (on-line or paper) to individuals in schools. Some may agree to participate on-line, but others will require paper versions.Advantages gained from reduced printing and shipping costs, less workload during data entry and preparation.Software and hosting will be provided by the IEA DPC.


Thank you for your attention!

Data Management Seminar, 8-11th July 2008, Hamburg 1 ICCS 2009 – On-line Data Collection in the...

Documents

Transcript of Data Management Seminar, 8-11th July 2008, Hamburg 1 ICCS 2009 – On-line Data Collection in the...