Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Design Issues Section B...

Copyright 2010, The World Bank Group. All Rights Reserved.

Questionnaire Design Issues Section B

Disclaimer: The examples used are not necessarily good or recommended but are used for illustrative purposes only.


Structure and Format

Layout can:

• Facilitate data collection, transformation and capture

• Differentiate between questions, responses, sections and clarify skip patterns

• Encourage better response rates

2


Structure and Format1. Abbreviations and punctuation

a. Spell out abbreviationsb. Do not use commas or periods inappropriatelyc. Abbreviated wording only with trained interviewersd. Underline or bold critical words

2. Response categoriesa. Margin-justify pre-coded and self-coded responsesb. Margin-justify code boxes for write-in responses

3. Spacea. Avoid cluttered appearanceb. Provide enough space for write-in responsesc. Avoid too many thick black lines - try thin gray lines

3



4. Answer spacesa. Should be clearly identifiable for each questionb. Use columnar or tabular format for answer spacesc. Ensure each response category box is clearly identifiable

5. Interviewer working spacea. Provide space and instructions on the questionnaire for any

calculations to be performed

6. Differentiationa. Differentiate between the types of answers required and

provide guides to skip patterns whenever possible.

4



7. Colora. Drop-out colors for image data captureb. Differentiate between sections and clarify skip patternsc. Colorful graphics can help motivate respondents to complete

self-enumerated questionnaires

8. Question numberinga. Each question should be numbered sequentially (1, 2, 3)b. Each section should be numbered sequentially (I, II, III, A, B,

C…)

5



9. Instructionsa. Should be placed (in order of preference):

– Above the question– At the beginning of a questionnaire section– On the front of the questionnaire– On a separate sheet or in an instructional booklet

b. Instructions should have a differentiated and standardized font, location or separation (box)

10.Languagea. Various languages may affect spacing and layoutb. Translations should be rigorously checked

11. Consistencya. Formatting decisions must be consistent throughout.

6


Processing Considerations

Keyed data entry

1. Appropriate number of code boxes per response

2. One box per character

3. Answer boxes margin justified

4. How many responses should be marked for each multiple choice question?

5. How will “not stated” be marked?

6. How will questions not applicable to a particular respondent be marked?

7


Processing Considerations

Scanned data capture

1. Forms must be kept in good condition

2. If multiple pages, must be separated (bar codes, cutters)

3. Paper color, printing clarity, drop out colors

4. Sufficient white space around answer boxes

5. Proper writing instrument used to complete forms

6. More rigorous testing of questionnaires, early enough to make changes to software, equipment, and/or forms.

7. Compare against a keyed sample

8


Questionnaire Quality Issues with Scanned Data Capture

Questions should be pre-coded as much as possible.

The major areas requiring quality control are:

1. Paper (color, weight and size)

2. Printing (color and clarity)

3. Format (page layout, response position(s), and clear zones)

9


Questionnaire Quality Issues with Scanned Data Capture

Interviewers, and respondents for self-enumeration, must be trained, instructed in how to complete the form correctly.

1. OMR bubbles

2. OCR characters

10


Methods for Drafting Questionnaires

1. Unstructured Individual Interviewing

2. Qualitative Group Interviews

3. Participant Observation

11


Unstructured Individual Interviewing

A discussion of the proposed survey topics

Guided by a topic outline

Used to gain insights into questionnaire structure

Time required: 2 - 6 weeks, Cost: Salary and per diems

12


Qualitative Group Interviews

Informal discussions of selected topics

Aids in developing conceptual framework and data specification

Can also be used post survey to interpret the data.

Time: 2 - 4 months, Cost: Salary and per diems

13


Participant Observation Research

Anthropological technique for gathering information

Field researchers live with the persons of interest.

Can be used to ensure questionnaire will provide enough information and help phrase questions

Time: 6 months - 1 year, Cost: Support of researcher and incidentals

14


Procedures for Testing the Questionnaire Draft

1. Informal Testing - subjective evaluations of the questionnaire,

2. Formal Testing - relies on statistical evaluations

a. Pilot Studies

b. Split sample tests

15


Informal Testing

A questionnaire field test with a small number of interviews

Depends on subjective information provided by interviewers and observers.

Not designed to be evaluated on a rigorous statistical basis.

Usually done as a first test of the draft or as a final step to ensure that revisions work well.

16


Informal Testing: personnel, cost and time

Coordinator.

Interviewers

Personnel to evaluate the results.

50 to 300 respondents.

Respondents are selected purposively rather than randomly.

Can evaluate entire questionnaire or a portion. More than one questionnaire evaluated with split-sample testing.

Time: Dependent on questionnaire and sample but operational aspects around 4 months. Cost: also variable but generally low.

17


Formal Testing

Relies on probability sampling and results are evaluated on a rigorous statistical basis.

1. Pilot study - a prototype of the survey conducted to observe all of the proposed survey operations working together.

2. Split-Sample Test - conducted to evaluate two or more alternate versions of the questionnaire.

18


Formal Testing - Pilot Study

Reduces time between final survey and results.

Often little time for a pilot test, evaluation and refinement of survey tools and procedures.

Large, complicated, or repetitive surveys need a pilot study

19


Pilot Study – personnel, time and cost

Probability sample necessitates larger numbers of personnel, including a sampling statistician, data processing staff and analysts.

Different respondents should be used for pilot and survey.

Not finished until data analysis is completed.

Evaluation plan must be decided in advance.

Time and costs are variable but are likely greater than for an informal test.

20


Formal Testing - Split-sample

Experimental design incorporated into the data collection process.

Used as methodological studies to advance knowledge of questionnaire design and the survey research process.

Personnel requirements are similar for a pilot test but require more sophisticated statistical and analytical expertise.

Time and cost are similar to pilot test but the time may be distributed differently for a split-sample with more planning and less editing and coding.

21


Techniques for Evaluating a Questionnaire

1. Investigating Respondents Interpretations of Questions

2. Observation and Monitoring of Interviews

3. Learning from Interviewers

4. Using Record Checks

5. Response Analysis Surveys

22


Questionnaire Review Worksheet Exercise

Questionnaire organizationQuestion wordingQuestion groupingSkip patternsCoding Questionnaire length

23

Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Design Issues Section B...

Documents

Transcript of Copyright 2010, The World Bank Group. All Rights Reserved. Questionnaire Design Issues Section B...