Survey Sampling - 2

Survey Sampling - 2

Survey Design

Specification of the research goals• The most critical stage in designing any survey but

often one of the most difficult• Need to turn a fairly simple question e.g. ‘What is the

impact of social capital on people’s life chances?’ into a series of hypotheses which you want to test– E.g., People who have an extended structure of formal

social networks better able to find jobs than those who do not have such networks.

• Need to first ask yourself which is the most appropriate survey design for the hypotheses being tested

Decision processes• Survey design depends on the research questions being

asked• Need to choose an appropriate survey design

– What is the population of interest?– Is there a way of selecting a sample that will be unbiased and

represent the population you are interested in?– What is your unit of observation – the household, families,

individuals, institutions, countries, something else?– What is your unit of analysis – e.g. you may observe

individuals but your unit of analysis may be the number of spells they have had in paid employment

– Do you need cross-sectional data or longitudinal data to answer your research questions?

Remember…Who responds?• Individuals• Households• Establishments/

organizations• Land• Animals or plants

About…• Individuals• Events• Households• Other people in the

household• The local area• An industry

Unit of OBSERVATION Unit of ANALYSIS

Decision processes• Survey design depends on the research questions being

asked• Need to choose an appropriate survey design

– What is the population of interest?– Is there a way of selecting a sample that will be unbiased and

represent the population you are interested in?– What is your unit of observation – the household, families,

individuals, institutions, countries, something else?– What is your unit of analysis – e.g. you may observe

individuals but your unit of analysis may be the number of spells they have had in paid employment

– Do you need cross-sectional data or longitudinal data to answer your research questions?

Cross sectional designs• Cross sectional surveys provide a snap-shot of a population at one

point in time• Often used for large government surveys, for one-off projects,

political polling• If the research questions are about describing or explaining current

status a cross-sectional design is what you need• Are many design and data collection variants within a cross-

sectional design determined by the research questions– Sampling sub-populations, age group or by geographical location– Questionnaire administration only– Diaries – time-use, nutrition, or travel– Responses to visual stimulus of some kind e.g. market research on

branding– An experimental design using randomized control and treatment groups

(can apply to longitudinal surveys as well)

Cross-sectional designs used for trend analysis

• Repeated cross sectional surveys (e.g. Family Resources Survey, General Household Survey, Labour Force Survey)– Common design for large government surveys– New sample drawn for each survey– Carry similar questions each year– Used for trend or time series analysis at aggregate level

• A cross-sectional design often includes a longitudinal element– Retrospective life histories of employment, marriage and

fertility, housing, migration– Questions about a reference period in the past e.g. the last

month, year, or since leaving full-time education etc.

When do we need longitudinal data? (1)• Longitudinal designs collect data about different time points for

the same units of observation or cases (usually individuals but could be dwellings or employers etc.)

• Information on transitions and change over time• To measure inherently longitudinal phenomena

– Change• movements into or out of poverty• labor market transitions and durations• longer term social mobility (sosyal hareketlilik)• household formation and dissolution• residential mobility

– Stability• persistence of poverty over time• persistence of people’s attitudes over time• Inter-generational transmission

When do we need longitudinal data? (2)• Temporal element central to inferences about causality

– If X and Y: Prove an association between variables– If X and Y and not Z: Isolate the relationship– If X then Y: Temporal ordering

• To control for the effects of unmeasured fixed differences (heterogeneity) between subjects

• To study social change by separating out age, period and cohort effects

• To establish the effect of a treatment e.g. a policy intervention by comparing periods before and after the introduction of the policy

Designs for collecting longitudinal data• Retrospective studies

– Cross sectional collection of historical information– Past events and transitions are recalled by respondents e.g. life time

employment history• Record linkage panels• From data collected for administrative purposes e.g. taxation or

social welfare system or from Censuses– Need to reliably identify individuals and allow them to be linked over

time– Provide powerful data but limited in scope for analysis of social

change• Prospective studies

– same individuals are re-interviewed in successive years– suffer less from recall error than retrospective studies

From concepts to indicators to questions• The type of data you need to collect depends on

what you need to know to answer your research questions

• Steps in the process:– Defining the key concepts you are need to measure– Operationalize those concepts, i.e., what indicators

will measure the concept of interest?– Develop indicators into questions in your

questionnaire• You will examine measurement error in greater

detail over the coming weeks

Working with Abstract Concepts

Standardization• The aim is to produce as standardized

questionnaire that will produce data for quantitative analysis– “Standardized” – all participants are asked the same

questions in the same format in a consistent manner• Why standardize?

– Every participant receives the same treatment or stimulus, any variation in the resulting data will represent real differences between cases

• This ideal is very difficult to achieve!

Reliability and validity• Aim: To have questions which are…

– Reliable: They are understood the same way at each time they are asked and therefore yield consistent answers on separate occasions

– Valid: They are measuring what we think they are measuring and are understood in the same way by each respondent

Basic rules of question design• What do you think some basic rules of

question design might be?• What be might some characteristics of

unreliable questions that we can spot simply by reading them?

THE BASICS OF WRITING GOOD QUESTIONS

Good Design Helps Minimize Nonresponse and Measurement

Error • Instrument should be easy to understand, to navigate, and to

complete – Encourage response

• Questions must be: – Clear – Unambiguous – Minimize possible bias

• E.g., questions with socially desirable answers • The harder it is for the respondents, the higher the survey

nonresponse rate

How Hard Can It Be? • That’s everyone’s first reaction • Turns out (and you will likely experience it in this class), it’s

harder than it seems • Here are some reasons (and there are more):

– The devil’s in the details: Differences in words and question wording matter – sometimes a lot

– It’s not just asking questions, but asking the right questions (with the proper response scales) that get at what you’re trying to learn

– The questions need to be robust to lots of factors such as how different people (mis)interpret the questions (including response scales, question ordering, etc.) – consistency is key

The reality: It’s just downright hard to write questions that are perfectly clear and unambiguous, yet reasonably simply worded, and that get at what you want to know

Anecdote• Two Jesuit novice priests both wanted a

cigarette while they prayed. They decided to ask their superior for permission. The first asked but was told no. A little while later he spotted his friend smoking and praying. "Why did the superior allow you to smoke and not me?" he asked. His friend replied, "Because you asked if you could smoke while you prayed, and I asked if I could pray while I smoked!"

An Example – What’s “Truth?”

Dilmann 2007.

?• Why did these results for the alternative sets of

categories and the two questions differ so dramatically?

• Which answers should be trusted? • How can you ever be sure that whatever

questions are asked in surveys obtain answers that provide the best possible estimates of the distribution of the characteristic of interest in the population?

Eliciting Correct Information Not As Simple As One Might Expect

Groves et al.

CRITERIA FOR ASSESSING EACH SURVEY QUESTION

1. Does the question require an answer?2. To what extend do survey recipients already have an

accurate, ready made answer for the question they are being asked to report?

3. Can people accurate recall and report past behavior?4. Is the responded (katılımcı) willing to reveal the required

information?5. Will the responded feel motivated to answer each question?6. Is the respondents understanding of response categories

likely to be influenced by more than words?7. Is survey information being collected by more than one

mode?8. Is changing a question acceptable to the survey sponsor?

• A survey question is more than a general inquiry. Survey questions make it possible to determine the distribution of a characteristic (an attitude, belief, behavior, or attribute of each respondent) in the survey population. Neither of the following questions meet that criterion.

• If you fixed dinner at home last night, did you eat meat as part of that meal?– Yes– No

• When you go out to eat, which type of food do you most prefer?– Turkish– Italian– Chinese– Other

All are acceptable depending on the precision of the information you require.

Respondent’s opinion on issue for which wording changes produce substantial inconsistencies in response choice:

“Tall people are more likely to be elected Prime Minister of Turkey.”

Do you strongly agree, somewhat agree, somewhat disagree, or strongly disagree with this statement?

(or) Do you very strongly agree, strongly agree, agree, disagree, strongly disagree, or very strongly disagree?

(or) On a scale of 1 to 7, where 1 means entirely agree and 7 means entirely disagree, use a number to

indicate how strongly you agree or disagree.

• People do not have ready-made responses to opinion and belief questions to nearly the same degree that they possess answers to questions about their educational level, the kind of car they drive, or whether they own their home.

• The vaguer the question, the vaguer the categories (e.g., using only numbers, as in a 1–7 scale), and the more remote these items are from people’s experiences, the more likely a question is to produce inconsistent responses if we ask the same person to answer this question at different times.

Asking how many hours a person watched television on the first Sunday of the previous month is an example of a behavior that is hard to remember. Respondents are unlikely to be able to recall something so precise from that long ago. To solve this problem, surveyors often ask how many hours per day a person “usually” watches television. To answer, the respondent must recall what she “usually” does and estimate. Recalling the frequency of these routine or ordinary behaviors is subject to considerable potential error, as evidenced by the effect of the category choices offered as part of the survey question.

This tendency for recall and reporting to decline as a function of length of recall has yield an important measurement error model.

If there were no recall problems, the ith person would report in response to the question.The model specifies that instead of , the respondent reports

𝑦 𝑖=𝜇𝑖 (𝑎𝑒−𝑏𝑡 )+𝜀𝑖: is the proportion of events that are reported despite concerns about sensitivity and social desirability.b : is the rate of decline in reporting as a function of time: is the deviation from the model for the ith individual.

The model specifies that the proportion of events correctly reported exponentially declines.The model implies that • for events that are distinctive, near another easily recalled

event and important in the life of the respondent, a is close to 1.0 and b is close to 0.0.

• for nonsensitive events that are easily forgotten, a may be close to 1.0 but b is large.

𝑦 𝑖=𝜇𝑖 (𝑎𝑒−𝑏𝑡 )+𝜀𝑖

• The fact that respondents know an answer does not mean they are willing to provide it. Many respondents are reluctant to reveal certain information about themselves, for example, their income. Others may be unwilling to answer questions about previous drug use, or having shoplifted when they were teenagers.

• Considerable evidence suggests that people are more likely to give honest answers to self-administered than to interview questionnaires For example, when asked the question, “How often have you driven a car after drinking alcoholic beverages?” only 52% responded “never” to the self-administered questionnaire versus 63% for the comparable telephone survey (Dillman and Tarnai, 1991).

• Designers of interview surveys often write questions independent of any motivational considerations, leaving it to the interviewer to encourage or persuade respondents to carefully select and report complete answers.

• Questionnaires are sometimes constructed in a way that requires respondents to consult a separate instruction booklet to understand unclear questions.

• Motivation can be encouraged in many ways, ranging from incentives and follow-up reminders to respondent-friendly questionnaire design.

• Responses to the questions about studying and presented at the beginning of this section were clearly influenced by more than words alone.

• Attitudinal and belief questions typically rely on vague quantifiers, such as strongly favor to strongly oppose, high priority to low priority, agree to disagree, or even vaguer ones that rely on numbers such as –3 to +3, 1–7, or 1–10.

• Such numerical scales require respondents to give a certain amount of definition to any category they choose to use. The vaguer the question and answer categories, the greater the potential for measurement error.

• This means that data collected by each mode needs to be comparable with that collected by another. An illustration is the fact that 42% of the telephone respondents versus only 23% of the mail respondents chose more than 2 1⁄2 hours in response to the hours of study question posed at the beginning of this section.

• These challenges of mixed-mode surveys will be discussed later.

• Often sponsors want information about which respondents have virtually no knowledge and for which formulating a meaningful answer is difficult.

• Political considerations may also dictate the selection of question and answer categories. Thus, it is important to ask sponsors whether questions that appear troublesome are subject to change and if so, how much.

• Writing questions is a difficult challenge precisely because so many factors simultaneously influence whether a proposed question obtains accurate answers.

• The eight questions listed here constitute the mental checklist when, surveyors asking for help thrust a draft and ask, “What do you think?” The search for tools and principles for writing questions is a means of overcoming each of these potential problems.

• This is the topic to which we now turn.

WHICH QUESTION STRUCTURE ISMOST APPROPRIATE?

Types of Question Content• Dillman (1978) distinguishes five types of

question content– Behavior: what people do– Beliefs: what people think to be true or false– Knowledge: what people know to be true or false– Attitudes: what people think to be desirable– Attributes: information about a person’s

characteristics• Clarity about what you are trying to measure

with your question is essential

Data for Quantitative Analysis• When designing a question you need to think

about how that variable will be used in analysis• Different levels of measurement needed for

different types of descriptive or inferential statistical analysis

• The way we ask a question will produce data at nominal, ordinal, or interval levels

• What are these levels of measurement?• How are they used in analysis?

Nominal Level of Measurement• A variable (question) where the different

response options have no set rank-order• We cannot order the categories from low to

high, lesser to greater in any meaningful way– Religious affiliation or ethnicity have no natural

rank-order• Another type of categorical variable

Ordinal Level of Measurement• Rank-order categories from low to high, lesser

to greater• Cannot specify in numeric terms how much

difference there is between categories– Response options of “child”, “teenager”, “young

adult”, “middle aged”, “elderly”• Inherent ordering, but we do not know the

numerical gap between categories• A type of categorical variable

Interval Level of Measurement• Variable (question) in which categories can be

ranked from low to high in some meaningful way• Can specify the amount of difference between

values of categories, e.g., age in years from youngest to oldest and we know that the difference between age 20 and 25 is 5 years

• Many interval questions in surveys– Hours of work, income, height and weight – Anything that is collected as a number and produces a

distribution• Often described as continuous variables

Types of questions and response formats• Open (verbatime) questions

– How many cigarettes have you smoked in the last month?– In what religion were you brought up?– Response is in the own words of the respondent– Answer is field coded or coded post fieldwork

• Closed– Predefined set of response options– “Forced choice”, “Multiple choice”, “Code All”

• Scales, Scores and Rankings– Agree-Disagree (Likert Scales)– Rankings of preferences

Open vs Closed Ended Questions• Open-ended questions are good when response

options are not well-known/structured or when want to get the logic or reasoning behind answer– Use sparingly: Hard to analyze and can put excess

burden on respondents– Don’t use for you are too lazy to determine

appropriate response categories

Open vs Closed Ended Questions• Closed-ended questions are good when

response options are well-known and mutually exclusive– Easier to analyze than open-ended responses– If done poorly results in numerous errors (will be

covered in the following weeks)

Teaching and Learning Lab - MIT

PRINCIPLES FOR WRITING SURVEY QUESTIONS:THE COMBINING OF WORDS AND STRUCTURE

Dillman, D.A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York: John Wiley & Sons, Inc.

• Words are the building blocks for all question structures, but deciding which words to use and in what order is far from simple.

• The wrong choice of words can create any number of problems, – from excessive vagueness to too much precision, – from being misunderstood to not being understood at all, and– from being too objectionable to being uninteresting and

irrelevant. • Making the right choice in any given situation and

knowing when you have achieved it are issues on which agreement is far less likely.

Stanley Payne• Perhaps no one has summarized the dilemma of writing

questions as elegantly as Stanley Payne (1951) nearly 50 years ago in The Art of Asking Questions.

• In one chapter he presents 41 versions of a single question before finding one that he considers acceptable for a survey. Even this question is cautiously labeled “passable,” and the reader is admonished that pretesting might shoot this 41st version full of holes.

• The concluding chapter of the book summarizes Payne’s rules for wording questions, with the subtitle of the chapter, “A Concise Check List of 100 Considerations,” describing the surveyor’s dilemma.

`SELECTED` PRICIPLES1. Choose simple over specialized words2. Choose as few words as possible to pose the question3. Use complete sentences to ask questions4. Avoid vague quantifiers when more precise estimates can be obtained5. Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made

answer.6. Use equal numbers of positive and negative categories for scalar questions7. Distinguish undecided from neutral by placement at the end of the scale8. Avoid bias from unequal comparisons9. State both sides of attitude scales in the question stems10. Eliminate check-all-that-apply question formats to reduce primacy effects11. Develop response categories that are mutually exclusive12. Use cognitive design techniques to improve recall13. Provide appropriate time referents14. Be sure each question is technically accurate15. Choose question wordings that allow essential comparisons to be made with previously collected

data16. Avoid asking respondents to say yes in order to mean no17. Avoid double-barreled questions18. Soften the impact of potentially objectionable questions19. Avoid asking respondents to make unnecessary calculations

• When a word exceeds six or seven letters, chances are that a shorter and more easily understood word can be substituted, although it should not automatically be assumed that all shorter words are acceptable.

• Nearly 20% of the respondents listed the number of years they had lived in the city or town and the county.

And so forth for the remaining concepts…

One of the shortcomings of agree/disagree is that there is a cultural tendency in many societies for people to agree rather than disagree, or to acquiesce in their responses. Consequently, the second revision, which maintains the use of closed-ended ordered categories but changes to an “easier, the same, more difficult” format, would seem to be a better solution.

Remember requiring a response to each stimulus rule!

Not only response is obtained but also the information is increased!

The information is further increased!

>7 Rule!

Not only exhaustive but also mutually exclusive categories.

Continues on the next slide…

CONSTRUCTING THE QUESTIONNAIRE

COVER LETTERStudy Title:PI:Dear Respondent, I am inviting you participate in a research project to study xxxx. [This research project is funded by xxxx.] Along with this letter is a short questionnaire that asks a variety of questions about xxxx. I am asking you to look over the questionnaire and, if you choose to do so, complete it and send [or give] it back to me. It should take you about xxxx minutes to complete. You must be 18 years of age to participate. The results of this project will be [say what it’s for]. Through your participation I hope to understand xxxx. I hope that the results of the survey will be useful for [xxxx] and I hope to share my results by [publishing them in a scientific journal][presenting them on the web where students all over the world can use them]

I do not know of any risks to you if you decide to participate in this survey and I guarantee that your responses will not be identified with you personally (or make some statement about confidentiality.) I promise not to share any information that identifies you with anyone outside my research group which consists of me [and xxxx]. [You should not put your name on the questionnaire.] If you do not feel comfortable handing in your survey to me (or your teacher), you may also . . .[put option here, e.g., mail it to the following address, drop it off at X office]I hope you will take the time to complete this questionnaire and return it. Your participation is voluntary [and there is no penalty if you do not participate]. Regardless of whether you choose to participate, please let me know if you would like a summary of my findings. To receive a summary, xxxx. If you have any questions or concerns about completing the questionnaire or about being in this study, you may contact me at xxxx. You may also contact my research advisor at xxxx. This project has been approved by the Institutional Review Board (IRB) at North Carolina A&T State University. If you have any questions about your rights as a research study participant, you may contact the chair of the IRB through Compliance Office at (336) 334-7995 or [email protected]. You must be at least 18 years old in order to participate.If you agree to participate, you may keep this form and complete the survey.If you wish, you may stop at any time.You do not have to place your name on the survey. Sincerely. Your Name Here…

mailto:[email protected]

1. Start with a salient/interesting question2. Don’t switch topics in ordering the questions3. Place items with same response categories in a series

1. Group questions with same answer type on the same topic together into matrix questions

4. Keep objectionable/personal and open-ended questions until the end of the survey

5. List answer categories vertically rather than horizontally6. Maintain consistency throughout survey in direction of

scales displayed7. Place instructions exactly where needed8. Number questions simply and consecutively from

A questionnaire is like a conversation which typically evolves in accordance with societal norms (Schwarz, 1996). Constantly switching topics makes it appear that the questioner is not listening to the respondent’s answers. Each answer seems to stimulate a response on an unrelated topic, as though the person’s answer was not heard.

• What was your total family income in 1998?• Do you like to play golf?• What is your opinion on global warming?• Are you married?• How many times have you gone bowling during the past year?• What is your political party preference?• Do you favor or oppose these measures to reduce environmental pollution?• What is your occupation?• Please describe your favorite recreational activity.• How adequate is your present health care?• Which political party does the best job of promoting economic growth?• How old are you?• Has your health gotten better or worse during the past year?

AN EXAMPLE OF A QUESTIONNAIRE WITH DISCONNECTED QUESTIONS

Suppose, for example, that a cover letter has informed recipients that they are being asked to respond to an important questionnaire about preferences for possible change in the health care program to which they belong. The questionnaire cover carries out that theme with a title such as, “What Changes Would You Like to See Made in Your Present Health Maintenance Program?” These words develop respondents’ expectations that they are going to be asked for their opinions about desired changes. Thus, it seems important that early in the questionnaire they are asked to do that. A salient beginning to a questionnaire is partly a matter of meeting respondent expectations and partly a matter of identifying questions that the respondent will find interesting. One might reasonably begin by asking how long respondents have participated in the current program and how well they like it.

However, one should not begin by asking a series of disjointed demographic questions—education, age, income, home ownership, veteran status, or even prior medical history. These demographics may be highly relevant to the study objectives and easy to answer, but respondents will not see obvious relevance to the topic.

The first pages are also not the place to include a long series of abstract attitudinal scales, such as those designedto measure a respondent’s orientation towards government versus private sector involvement in meeting health care needs.

• An effort is also made to order questions in a way that will be logical to the respondent. When respondents are asked why they left a job, it makes sense to ask such questions after asking why they accepted that job and how long they were employed in that position. In general, this suggestion implies asking people about things in the order that they happened.

• It is also helpful to ask people descriptive questions about an activity before requesting evaluations of the experience, thus encouraging more complete recall of past events.

• For example, a question about the main reason a person left a job might best be asked after a series of items about the nature of that job.

• For example, if within a general topic area there are several yes/no questions, some that require unfavorable-favorable scalar responses, and others that require agree-disagree responses, the task of the respondent can be eased by grouping those requiring the same answer categories together.

• Our purpose is to ease the cognitive burden of responding. Responding to questionnaires in which each new question means having to think about a new topic, as well as a new type of response category, requires more respondent effort.

• Objectionable questions are placed near the end of the questionnaire, where they are likely to be seen after the respondent has had an opportunity to become interested in the questionnaire.

• A respondent who has spent five or 10 minutes already answering questions is less likely to respond to an objectionable question by quitting. Moreover, some questions may seem less objectionable in light of previous questions already answered.

• One of the questions that people often object to answering is the question of income.

• In addition to reporting income, people are often uncomfortable answering questions about their sexual behavior or any laws they may have broken.





scales displayed 7. Place instructions exactly where needed8. Number questions simply and consecutively from

Exercise• Write 6 survey questions based on what you

have learned in the readings. One question should measure the respondent’s sex. The remaining 5 questions should be on the topic of smoking tobacco products. One should tap beliefs about smoking tobacco products, the others should be designed to measure attitudes, attributes, knowledge and behavior respectively.

How to measure the respondent’s sex

Behaviour I

Behaviour II

Beliefs

!

Attitudes

Attributes

Knowledge

Project 1• You have been asked to carry out a survey on

ethnicity and paid employment in Izmir. The main aim is to understand how ethnicity is associated with employment opportunities in Izmir.– What are the key concepts you will have to measure?– What are some possible indicators?– Design 5 questions:

• Beliefs: What is thought to be true or false• Attributes: Characteristics• Knowledge: What is known to be true or false• Attitudes: What is desirable• Behavior: What is done

Survey Sampling - 2

Documents

Transcript of Survey Sampling - 2