Surveys and Questionnaires

Research Methods and Data

College of Advancing Studies

Brendan Rapple

Surveys Provide Important Knowledge

Economists, psychologists, health professionals, political scientists, and sociologists conduct surveys to study such topics as:

Income and expenditure patterns among households;

Voting behavior;

Effects on family life of women working outside the home, etc.

Auto manufacturers use surveys to find out how satisfied people are with their cars.

And a multitude of other topics

This presentation owes much to the American Statistical Association brochure series on survey research:

http://www.amstat.org/sections/srms/whatsurvey.html

Specific Purpose Essential

Objectives of a survey should be as

Specific

Clear-cut

Unambiguous as possible

"Men's Health Practices" is a very nebulous topic.

Better:How often do African-American males aged 40-49 visit the dentist?

Or

A survey of 50-60 year old male professors at BC about their weekly exercise habits

Steps in Conducting A Survey

Define precise purpose

Specify population

Specify appropriate sample

How to administer survey?

Draft of survey instrument

Pretest it

Revise it

Administer survey

Analyze, write it up, and communicate the results

Use results meaningfully

Decide on Mode of Data Collection

Mailed Questionnaire

Telephone

In Person Interview

Computer Questionnaire

Use Mailed/Computer Questionnaires or Not?

Perhaps “better” responses by phone -- but phone interviewing is very time-consuming.

On other hand, people tend to be more truthful with anonymous questionnaires.

Generally cheaper than one-on-one interviews.

Perhaps a mix of questionnaire and interviewing?

For All Surveys it is Essential to

explain purpose of survey very clearly and precisely.

explain any potential use the results will have for the respondents themselves.

stress voluntary nature - respondents are doing a favor to the researcher.

use letter-head – official – for a mailed/computer questionnaire.

explicitly promise confidentiality.

mention a clear expression of thanks.

Whatever Format Used . . . . . .

Important to specify a deadline for response.

Follow-up letter (e-mail) often advisable for a mailed/computer questionnaire

Always essential for PILOT STUDY or PRE-TEST

Main problems revolve about:

Question content, e.g. confusion with overall meaning of question as well as misinterpretation of individual terms or concepts

Formatting, e.g. problems with how to skip or navigate from question to question may result in missing data and frustration for both interviewers and respondents.

Population → Sample

Usually the population to be surveyed is too large.

Accordingly, one must select a smaller, representative sample.

“This sample is usually just a fraction of the population being studied.”

Samples

The quality of the sample – whether it is up-to-date and complete – is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed.

Must be representative of population.

Are the distributions of attributes, opinions, and beliefs in the sample the same as in the population?

You want to be able to make inferences about the population as a whole based on what you find to be true of the sample.

Variability

Variability is large, then sample should be large

Converse also true

2 Barrels of Apples

Barrel A (low variability) -- all apples about 5 ins. in diameter (range 5.1 to 4.8 ins.)

Barrel B (high variability) -- apples range from 2 to 6 ins. in diameter

Picking 3 apples from Barrel B might give result well below (above) average.

Still, Size of Sample Isn't Everything

Large numbers do not, in and of themselves, increase the representativeness of a sample.

Most professional survey conductors hold that a moderate sample size is enough statistically and operationally.

Representative Sample

Survey: Success of unwed teenage mothers in a specific community in raising children?

To be representative, sample must contain same proportion of unwed teenage mothers at

--each age level

--each educational level

--each socio-economic statusin the community

Population -- Sample

It is essential that you select sample in such a way that every name on the population list has an equal chance of being included in the sample.

Random Sample

Example: 500 part-time students in Advancing Studies

Sample of 20% is required

Assign each student a number from 1 to 500

Randomly select 100 numbers

Systematic Random Sampling

Example 1. 2,000 in population and you want a sample of 200, then you might

select every 10th name

Example 2. 500 part-time students in Advancing Studies Sample of 20% is required

--Randomly Select a Number from 1 to 5

--Select Every 5th Person

--002, 007, 012. 017, 022, and up to 497.

Possible Problem

Staff in govt. agency may be listed unit by unit

Each unit has 9 line-level workers and 1 supervisor.

The supervisor is the 10th person on the list.

It’s a survey of 20% -- every 5th person is selected.

If first no. selected is 1, 2, 3, or 4 then no supervisor will be selected, though they comprise 10% of population.

If first number selected is 5, then supervisors will be greatly overrepresented.

Thus, possibility of bias due to periodicity or patterns.

Stratified Sampling

Population: 2,000 (800 females; 1,200 males)

Sample required: 200

If gender is an important variable in your survey, then both females and males should be included in appropriate numbers, that is, in proportions that correspond to their presence in the population.

Strategy:

Treat both sexes as separate populations and take 10% sample from each.

OR

Make sure that all females are listed first and then take every tenth name.

Either way you will end up with 80 females and 120 males

Convenience Sampling

Could ruin an otherwise well-conceived survey.

It’s simple and cheap to select a sample of names from a phone-directory to find out which candidate people intend to vote for.

However, this sampling procedure could give incorrect results since persons without telephones or with unlisted numbers would have no chance to be reflected in the sample,

Their voting preferences might be quite different from persons who have listed telephones.

Confidentiality

Confidentiality of data supplied by respondents is of prime concern to all reputable survey organizations.

Important that individual respondents are not identified in reporting survey findings.

All of the survey’s results should be presented in totally anonymous summaries, such as statistical tables and charts.

Problems with Volunteers

Example:

TV programs asking viewers to vote.

people call who are most committed to issue.

“stuffing of ballots” by multiple calls.

Time of day is important – who’s available?

Margin of Error

Error margin of 1,000 randomly chosen individuals is said to be 3.1%.

Thus, if a random sample of 1,000 indicates that 59% will vote for Obama, the actual number could range from 55.9% to 62.1%.

Questions in a Questionnaire Should Be

Woven together

Flow smoothly

Avoid confusion

A good questionnaire forms an integrated whole.

Different Understandings

Everyone should see/understand the exact same question – no ambiguity.

But people from different backgrounds, with diverse frames of reference, may have different perceptions of the same question.

KISS PrincipleKeep It Simple, Statistician

Questions should be, as far as possible: Simple Clear, Easy to answer Personally relevant to them

Often recommended that questionnaires be written at the 5th grade reading level

Avoid Ambiguity

Do you favor governmental involvement in health care?

“What is your income?” __________________ do you mean:

weeklymonthlyannualpretaxafter taxfrom salary or from all sources

Another Example:

(A blouse manufacturer wants to ascertain what type of sleeves teenage females prefer in their blouses)

1. Do you like short sleeve blouses?YES__ NO__

2. Do you like long sleeve blouses?YES__ NO__

3. Do you like sleeveless blouses?YES__ NO__

Another Example

“Do you jog regularly?”

Problem, of course, lies with REGULARLY

Avoid jargon, slang, abbreviations

Plumbers talk about “snakes”

Psychologists about “oedipus complex”

Lawyers about “mens rea”

“NATO” usually means North Atlantic Treaty Organization

But some respondents might take it to mean:

--National Auto Tourist Organization

--Native Alaskan Trade Orbit

--North African Tea Office

Types of Questions

Open-Ended Questions:

What is your age?

What is the total turnover in your company?

Which of the four seasons do you prefer?

How would you spend a a $1,000,000 lottery win?

Disadvantages of Open-ended Questions:

Variation in answers make coding/scoring difficult

Advantages of Open-ended Questions

Do not impose researcher's opinion on respondent

Can lead to a very precise answer

Forced-Choice Questions

How many books do you read each year? Please check as appropriate below:

___ none ___ 1 to 5___ 6 to 10___ 11 to 20___ more than 20

What do you recall about the frequency of snowfall in Boston during the winter of 1994?

___ It snowed almost every day___ It snowed about once a week___ It snowed about once every two weeks___ It snowed about once a month

Importance of Wording

Take a very simple question:

How many drinks do you have each day? (Check one of the following)

____ 5 or more____ 4____ 3____ 2____ 1____ none

Better to Ask

Are your daily drinking habits reasonably consistent --i.e. do you take about the same number of alcoholic drinks each day?

_____ YES

_____ NO (if you mark "NO," skip the following question).

Checking Respondents' Consistency

Q. 4. Check one of the following:_x_ I believe that manufacturing should be increasingly

computerized, even if layoffs ensue.

___ Preserving the jobs of workers is more important than computerizing manufacturing.

Later on, the Questionnaire might ask:

Q. 30. Check one of the following:___ I support G.M.'s increasing use of robotics,

despite the resulting massive layoffs.

_x_ The government should force G.M. to curtail computerization to ensure a reduction in layoffs.

Avoid Emotional Language

“What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving people?”

Problematic Words: murderous

freedomsstealpeace

Avoid Loaded Questions:

"Should the mayor spend even more tax money trying to keep the streets in top shape?"

"Should the mayor fix the pot-holed and dangerous streets in our city?"

What is your present marital status?

1 never married 2 married 3 divorced 4 separated 5 widowed

Arrangement Is Important

A Better Arrangement Would Be:

What is your present marital status? (circle number)

1. never married2. married3. divorced4. separated5. widowed

Danger of Overlapping

Check How Many Children are in Your Family:

a. 0-1 children

b. 1-2 children

c. 2 or more children

Avoid Prestige Bias

“Most doctors say that cigarette smoke causes lung disease for those near a smoker”. Do you agree?”

“Do you support the president’s policy regarding Zanozui?”

Avoid Double-Barreled Questions:

"Does your company have pension and health insurance benefits?"

Ranking Answers (Likert Scale Format):

Women should automatically receive three months maternity leave in your company

strongly agree partly agree disagree strongly

agree disagree

___ ___ ___ ___ ___

How often do you attend church?

Never Yearly Monthly Weekly Daily

____ ___ ___ ___ ___

Visual Analog Scales BIPOLAR scales

I see myself as:

Extremely Extremely interested in uninterested politics ______:______:______:______:______ in politics