Research Methods

Research Methods

SURVEY RESEARCH
Research Methods

Survey Research
Survey research is one of the most important areas of measurement in applied social research. The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. A "survey" can be anything from a short paper-and-pencil feedback form to an intensive one-on-one in-depth interview.

Roadmap
We'll begin by looking at the different types of surveys that are possible. These are roughly divided into two broad areas: Questionnaires and Interviews. Next, we'll look at how you select the survey method that is best for your situation. Once you've selected the survey method, you have to construct the survey itself. Here, we will be address a number of issues including: the different types of questions; decisions about question content; decisions about question wording; decisions about response format; and, question placement and sequence in your instrument. We turn next to some of the special issues involved in administering a personal interview. Finally, we'll consider some of the advantages and disadvantages of survey methods.

Questionnaires vs. Interviews
Surveys can be divided into two broad categories: the questionnaire and the interview. Questionnaires are usually paper-and-pencil instruments that the respondent completes. Interviews are completed by the interviewer based on the respondent says. Sometimes, it's hard to tell the difference between a questionnaire and an interview. For instance, some people think that questionnaires always ask short closed-ended questions while interviews always ask broad open-ended ones. But you will see questionnaires with open-ended questions (although they do tend to be shorter than in interviews) and there will often be a series of closed-ended questions asked in an interview.

Questionnaire
Mail Survey+ Wide Coverage+ Inexpensive- Response rates low - biased samplesGroup-Centered Questionnaire: A sample of respondents is brought together and asked to respond to a structured sequence of questions.+ questions can be explained+ response rate high- Narrow Coverage

Questionnaire
Household drop-off s: In this approach, a researcher goes to the respondent's home or business and hands the respondent the instrument. In some cases, the respondent is asked to mail it back or the interview returns to pick it up. Like the mail survey, the respondent can work on the instrument in private, when it's convenient. Like the group administered questionnaire, the interviewer makes personal contact with the respondent -- they don't just send an impersonal survey instrument. And, the respondent can ask questions about the study and get clarification on what is to be done. Generally, this would be expected to increase the percent of people who are willing to respond.

Interviews
Interviews are a far more personal form of research than questionnaires. In the personal interview, the interviewer works directly with the respondent. Unlike with mail surveys, the interviewer has the opportunity to probe or ask follow-up questions.General types include face-to-face and elite interviews Interviews can be very time consuming and they are resource intensive. The interviewer is considered a part of the measurement instrument and interviewers have to be well trained in how to respond to any contingency.

Interviews
Telephone interviews enable a researcher to gather information rapidly. Most of the major public opinion polls that are reported were based on telephone interviews. Like personal interviews, they allow for some personal contact between the interviewer and the respondent. And, they allow the interviewer to ask follow-up questions. But they also have some major disadvantages:Many people don't have publicly-listed telephone numbers. Some don't have telephones. People often don't like the intrusion of a call to their homes. And, telephone interviews have to be relatively short or people will feel imposed upon.

Selecting the Survey Method
Checklist of Selection Issues: Population & AccessibilityCan the population be enumerated? Is the population literate? Are there language issues? Will the population cooperate? What are the geographic restrictions?

Sampling IssuesWhat data is available? Can respondents be found? Who is the respondent? Can all members of population be sampled? Are response rates likely to be a problem?

Question IssuesWhat types of questions can be asked? How complex will the questions be? Will screening questions be needed? Can question sequence be controlled? Will lengthy questions be asked? Will long response scales be used?

Content IssuesCan the respondents be expected to know about the issue? Will respondent need to consult records? Bias IssuesCan social desirability be avoided? Can interviewer distortion and subversion be controlled? Can false respondents be avoided?

Administrative IssuesCostsFacilitiesTimePersonnel / Training

Comparative Evaluation of Survey Methods

Criterion

PersonalInterview

Mail

Telephone

Internet

Cost

High

Low

Moderate

Low

Response Rate

High

Low

High to Moderate

Low

Control of Interview Situation

High

Low

Moderate

Low

Probing Ability and Collection of Detail

High

Low to Moderate

Moderate

Low to Moderate

Speed

Low

Low to Moderate

High

Moderate

Conducting the Survey
Constructing a survey instrument is an art in itself. There are numerous small decisions that must be made -- about content, wording, format, placement -- that can have important consequences for your entire study. While there's no one perfect way to accomplish this job, we do have lots of advice to offer that might increase your chances of developing a better final product. There are three areas involved in writing a question:determining the question content, scope and purpose choosing the response format that you use for collecting information from the respondent figuring out how to word the question to get at the issue of interest

Dichotomous Questions
When a question has two possible responses, we consider it dichotomous. Surveys often use dichotomous questions that ask for a Yes/No, True/False or Agree/Disagree response. There are a variety of ways to lay these questions out on a questionnaire:

Nominal Questions
We can also classify questions in terms of their level of measurement. For instance, we might measure occupation using a nominal question. Here, the number next to each response has no meaning except as a placeholder for that response. The choice of a "2" for a lawyer and a "1" for a truck driver is arbitrary -- from the numbering system used we can't infer that a lawyer is "twice" something that a truck driver is.

Ordinal Questions
We might ask respondents to rank order their preferences for presidential candidates using an ordinal question.We want the respondent to put a 1, 2, 3 or 4 next to the candidate, where 1 is the respondent's first choice. Note that this could get confusing. We might want to state the prompt more explicitly so the respondent knows we want a number from one to 4 (the respondent might check their favorite candidate, or assign higher numbers to candidates they prefer more instead of understanding that we want rank ordering).

Interval Questions
We can also construct survey questions that attempt to measure on an interval level. One of the most common of these types is the traditional 1-to-5 rating (or 1-to-7, or 1-to-9, etc.). his is sometimes referred to as a Likert response scale. Here, we see how we might ask an opinion question on a 1-to-5 bipolar scale (it's called bipolar because there is a neutral point and the two ends of the scale are at opposite positions of the opinion):

Cumulative Questions
Finally, we can also get at interval measures by using what is called a cumulative or Guttman scale Here, the respondent checks each item with which they agree. The items themselves are constructed so that they are cumulative -- if you agree to one, you probably agree to all of the ones above it in the list:

Contingency Questions
Sometimes you have to ask the respondent one question in order to determine if they are qualified or experienced enough to answer a subsequent one. This requires using a filter or contingency question. For instance, you may want to ask one question if the respondent has ever smoked marijuana and a different question if they have not. in this case, you would have to construct a filter question to determine whether they've ever smoked marijuana:

Contingency Questions
Filter questions can get very complex. Sometimes, you have to have multiple filter questions in order to direct your respondents to the correct subsequent questions. There are a few conventions you should keep in mind when using filters: try to avoid having more than three levels (two jumps) for any question Too many jumps will confuse the respondent and may discourage them from continuing with the survey.if only two levels, use graphic to jump (e.g., arrow and box) The example above shows how you can make effective use of an arrow and box to help direct the respondent to the correct subsequent question.if possible, jump to a new page

Open vs. Closed
Open-ended QuestionsWhat is the most important problem facing the nation today?Closed-ended QuestionsDo you feel you are asked to pay more than you should in federal income taxes, about the right amount, or less than you should?

Survey Designs
Cross-sectional Surveys

Longitudinal Surveys

-Trend Studies

-Cohort Studies

-Panel Studies

Question Wording
One of the major difficulty in writing good survey questions is getting the wording right. Even slight wording differences can confuse the respondent or lead to incorrect interpretations of the question. Here, I outline some questions you can ask about how you worded each of your survey questions.

Questions a/b Question Wording
Can the Question be Misunderstood?The survey author has to always be on the lookout for questions that could be misunderstood or confusing. For instance, if you ask a person for their nationality, it might not be clear what you want (Do you want someone from Malaysia to say Malaysian, Asian, or Pacific Islander?). Or, if you ask for marital status, do you want someone to say simply that they are either married or no married? Or, do you want more detail (like divorced, widow/widower, etc.)?Some terms are just to vague to be useful. For instance, if you ask a question about the "mass media," what do you mean? The newspapers? Radio? Television?

Misunderstandings
Here's one of my favorites. Let's say you want to know the following:

What kind of headache remedy do you use?

Do you want to know what brand name medicine they take? Do you want to know about "home" remedies? Are you asking whether they prefer a pill, capsule or caplet?

Assumptions
What Assumptions Does the Question Make?Sometimes we don't stop to consider how a question will appear from the respondent's point-of-view. We don't think about the assumptions behind our questions. For instance, if you ask what social class someone's in, you assume that they know what social class is and that they think of themselves as being in one. In this kind of case, you may need to use a filter question first to determine whether either of these assumptions is true.

Time
Is the time frame specified?Whenever you use the words "will", "could", "might", or "may" in a question, you might suspect that the question asks a time-related question. Be sure that, if it does, you have specified the time frame precisely. For instance, you might ask:

Do you think Congress will cut taxes?

or something like

Do you think Congress could successfully resist tax cuts?

Neither of these questions specifies a time frame.

Personal Wording
How personal is the wording?With a change of just a few words, a question can go from being relatively impersonal to probing into your private perspectives. Consider the following three questions, each of which asks about the respondent's satisfaction with working conditions:Are working conditions satisfactory or not satisfactory in the plant where you work? Do you feel that working conditions satisfactory or not satisfactory in the plant where you work? Are you personally satisfied with working conditions in the plant where you work? The first question is stated from a fairly detached, objective viewpoint. The second asks how you "feel." The last asks whether you are "personally satisfied." Be sure the questions in your survey are at an appropriate level for your context. And, be sure there is consistency in this across questions in your survey.

Etc.
Other Wording IssuesThe nuances of language guarantee that the task of the question writer will be endlessly complex. Without trying to generate an exhaustive list, here are a few other questions to keep in mind: Does the question contain difficult or unclear terminology? Does the question make each alternative explicit? Is the wording objectionable? Is the wording loaded or slanted?

Question Placement
Decisions About PlacementOne of the most difficult tasks facing the survey designer involves the ordering of questions. Which topics should be introduced early in the survey, and which later? If you leave your most important questions until the end, you may find that your respondents are too tired to give them the kind of attention you would like. If you introduce them too early, they may not yet be ready to address the topic, especially if it is a difficult or disturbing one. There are no easy answers to these problems - you have to use your judgment. Whenever you think about question placement, consider the following questions:Is the answer influenced by prior questions? Does question come too early or too late to arouse interest? Does the question receive sufficient attention?

Placement
A Checklist of ConsiderationsThere are lots of conventions or rules-of-thumb in the survey design business. Here's a checklist of some of the most important items. You can use this checklist to review your instrument: start with easy, nonthreatening questionsput more difficult, threatening questions near endnever start a mail survey with an open-ended questionfor historical demographics, follow chronological orderask about one topic at a timewhen switching topics, use a transitionreduce response set (the tendency of respondent to just keep checking the same response)for filter or contingency questions, make a flowchart

Bad Questions 1
Suppose the budgets of your state and local governments have to be curtailed, which of these parts would you limit most severely? (ACIR 1982)Aid to the Needy8%Public Welfare39%

Bad Questions 2
Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened? (Roper 1994a)22% Possible, 12% Not sureDoes it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened? (Roper 1994b)1% Possible

Bad Questions 3
As you may know, in 1974, Jerry Springer, who had gotten married six months earlier, was arrested on a morals charge with three women in a hotel room. He also used a bad check to pay for the womens services, and subsequently resigned as mayor of his city. Does this make you feel much more likely, somewhat more likely, somewhat less likely, or much less likely to support Jerry Springer for governor this year? (Cambridge Survey Research 1974)

Bad Questions 4
Does your employer or his representative resort to trickery in order to defraud you of a part of your earnings?If you are paid piece rates, is the quality of the article a pretext for fraudulent deductions from your wages?Marx 1880: 208

Question Wording & Response Set
Do you approve or disapprove of the job President Clinton is doing? (Morin 1993)Approve 53%, Disapprove 38%

Do you strongly approve, somewhat approve, somewhat disapprove, strongly disapprove of the job President Clinton is doing? Approve (strongly and somewhat) 62%, Disapprove (strongly and somewhat) 32%

Question Wording and Order 1
Would you say that traffic contributes more or less to air pollution than industry? (Wanke et al. 1995)Traffic 45%; Industry 32%Would you say that industry contributes more or less to air pollution than traffic?Industry 57%; Traffic 24%

Question Order 2a
Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it? Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it? 36% Yes (Hyman and Sheatsley 1950)

Question Order 2B
Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?73% Yes

Response Validity:Over- & Under-reporting Behavior
Face-to-Face PhoneSelf-AdministeredLow Threat/NormativeRegistered to Vote+15+17+12Voted in Primary+39+31+36Have own library card+19+21+18

High ThreatBankruptcy-32-29-32Drunk driving-47-46-54

(Bradburn and Sudman 1980; Neuman 2006)

Research Methods

Documents

Transcript of Research Methods