“Items”

Writing Survey Questions

OverviewCharacteristics of good survey itemsTwo basic kinds of questionsSome Common item writing problems Scale selectionEvaluating questions after you write them

Its harder than you thinkDespite your best efforts, you will probably

write questions that raise more questions than they answer

There are no real hard and fast rules, but rather a set of guidelinesCommon sense is the real rule

CAVEAT of all survey research - - - surveys record second hand information, communicated perceptions, opinions, beliefs. Direct observation is always better (but not always possible)

What will you measureAlways begin by asking yourself what you intend

to measure or evaluate. This serves as the guide to what items you need to make a decision.

Ask yourself what covariates you will need (blocking variables)Common ones include, age, gender, SES, race,

ethnicity, but you may identify some that are relevant to your study (Fowler, 1995)

A guiding question should be “on what might the answers depend” (Yovanoff, 2005)

More is usually better than less – you can always throw them away if you don’t need them, but you usually cant go back and get more info

5 Basic Characteristics There are some basic characteristics of questions and

answers that are fundamental to good measurement (Fowler, 1995)

1.Question need to be consistently understood – the same for each person

2.Questions need to be consistently administered – sometimes an issue with in person interviews, less so with online surveys

3.Respondents need to be capable of understanding the question, and understanding how to answer

4.Respondents need to be willing to answerIn general – we want to maximize the degree to which

an question produces answers that measure something

Two Basic Kinds of QuestionsThose that aim to gather facts (objective)

e.g., In what month were you born? e.g., What is your approximate annual income in

thousands of CDN dollars?

Those that aim to gather opinion (subjective) e.g., Indicate on the following scale how much you

liked last night’s math homework. e.g., What is the top reason you think an SDSU prof

works for UofO is in BC?

Questions to Gather Facts(e.g., month born, annual salary, etc.)

Begin by clearly defining what you want to measure – overall and with each item. This definition will guide you in item development.Decide how you want the response to look – on what scale (more on that later)

Questions to Gather FactsThink of a common question you might see

on a survey designed to gather a ‘fact’

Questions to Gather Opinion(e.g. liking homework, why SDSU for UO in BC, etc.)

When you gather opinion, you are gathering subjective dataThere is no right or wrong answerSometimes these look like factual questions, but if people will respond in different ways, they are note.g., How friendly is your teacher?

Open Ended QuestionsThey are needed when …

you cannot put all possible responses on a scale, e.g., what are your work duties?

You want to understand thinking, e.g., how would you solve this problem?

When you really don’t understand what the answer might look like, e.g., how did you become homeless?

Unlike with scales or MC questions, open ended questions allow for more respondent flexibility – but it is is coupled with difficult analysis laterPeople are not constrained to an area of response, so

there is a lot of variability in the dataLeads to subjectivity in coding / imposition of scale

Common Item Writing Problems(and how to deal with them)

Multi-dimensionalityAmbiguous StemsResponse RestrictionSensitive ItemsDistorted Responses

Item Writing - DimensionalityA single dimension in a question is usually

best – both in the stem and possible responsee.g., How would you rate your health?

In this case “health” is ill defined, and may mean different things to different people. Is it level of fitness? Absence of disease? Weight? This construct has multiple dimensions

e.g., How would you rate your teacher?a)smart and confidentb)smart and not confidentc)no smart and confidentd)not smart and not confident

Item Writing - DimensionalitySometimes this is referred to as double

barreledIn questions watch out for coordinating

conjunctions, because they are designed to join clauses"and," "but," "or," "nor," "for," "so,“, "yet“

e.g., Although the system of education in BC is of good quality, it really should not be mirrored in other provinces. Agree or Disagree?You may agree the BC system is good or not goodYou may think it should be mirrored, or not.

Item Writing - AmbiguousAvoid ambiguous wordsBe concrete, define key terms in questionsDo you spend a lot of time studying in the

Master’s Program?On average, how many hours do you study

each day ?Do you usually prepare before class?

On average, how many hours do you spend preparing for class each week ?

Item Writing – Restricted ResponseDuring the semester, on average, how many hours

do you spend preparing for this class each week? Less than 1 hour Between 1 and 2 hours Between 2 and 4 hours More than 4 hours

VS

During the semester, on average, how many hours do you spend preparing for this class each week?

Enter response _____

Item Writing - LeadingWatch out for statements of supposed fact

e.g., Overall, would you agree with most people that our federal government is corrupt?

e.g., better – Overall, is our federal government corrupt?

e.g., This year, wild dogs killed more cats than in any other year, ever. Do you agree that dogs ownership should be better regulated?

e.g., better – Should dog ownership be regulated?

Item Writing - SensitivitySimple fact - some sensitive questions will

not elicit truthful responsesMany respondents will answer positively to

avoid the questione.g., Do you love your children?Better to triangulate with…

How much time do you spend with your children?Do you play games with your children?How much do you cuddle your children?Have you ever cried because of your child’s

behavior?Have you ever hit your child with a wooden bat?

Item Writing - SensitivityTake note – sensitivity is in the eye of the

beholder – it is not so much the question, but the way a person will answere.g. Have you been hospitalized in the last

year? a. For those who have not, it isnt sensitive – truthful

responseb. For those who have been to hospital for the flu, it

probably isnt – probably truthful responsec. For those who have been to hosptial for teenage

pregnancy or an STD – it probably is – probably untruthful response

Item Writing - SensitivitySome studies seem to indicate that

respondents tend to answer questions in a way that might make them look better, to the surveyor or the public (Locander, Sudman, Bradburn, 1976)Tend to under-report disease - when questioned

about health (Cannell, Fisher, Bakker, 1965)Tend to report status quo - when questioned

about voting (Madow, 1967)Tend to highly underreport - when questioned

about masturbating (Sudman & Bradburn, 1982)

Item Writing - SensitivityTo reduce sensitivity

Assure confidentiality, use blind data analysisBetter yet, don’t ask for names /personally

identifying info (or do so at the end of the survey)

Communicate the importance of accurate responses

Reduce the role of the interviewer (if face to face)

Reduce the amount of detail you ask for, so it is less obtrusive

Item Writing– DistortionsThere are two primary reasons we might not get

the answer we want outside of social desirability – “distortions in answers”The respondent may have forgottenThe respondent may not have the information

Be aware of respondents and how these factors may play into creating measurement error. Try to design around them. e.g., lead into a question with a brief description

imagine yourself at home when you are eating dinner as you answer this question

Keep in mind your responses are completely anonymous as you answer the following questions

ScalesWith your question stem, you also need to

design a response format. It might be a selection of choices, might be open ended, but often is on a scale.

Scale SelectionYour selection of response format (the scaling) is

determined, in part, by the kind of analysis you might want to useNominal – words/names – no order, no comparative

meaningeg., red, blue, green, Mark, Mary, ChrisGood for descriptive data analysis only

Ordinal – ordered, less to moreeg., no experience, a little experience, some experience, a lotGood for descriptive data, too, lends itself to proportions

analysis like % or chi-square Interval (no zero) / ration (zero) – equal interval scales

eg., month experienceLends itself to parametric statistics -t-tests, correlations,

regression, ANOVA, etc.

Scale SelectionNo matter what scale you choose, that scale

is open to subjectivity. This is a constant source of measurement error, and is difficult to quantify.e.g., if I ask you to rate this class on a scale

from 1 to 10, one person’s 6 might be pretty good, where another’s 6 might mean just palatable.

Often, we use numbers, words, and pictures to let people express their feelings and opinions, each may be more or less appropriate in different settings

Response Scale ExamplesNumbers

eg., on a scale of 1 to 10…Words

eg., a lot, some, only a little, not at all Pictures

Scales – the detailsHow many categories?

Fewer categories leads to less variance (less data to analyze)

But, research points to <10 (Andrews, 1984) being adequateIn truth, 5 or 7 is sufficient (10 point scale just

seems to be as much as people can handle) An even number will force one way or the

otherAn odd number permits sitting on the fence

Evaluating QuestionsAfter creating questions, there are several

methods we can use to evaluate their efficacy/quality

1.Focus group discussions did you like this question? Why or why not?

2.Interviews that focus on how a person answers a question (a cognitive approach) what were you thinking when you answered this

question

3.Field test under realistic conditions (pilot testing) e.g., give to 10 middle school age students first,

before giving to the whole middle school

Evaluating QuestionsWe can also evaluate the degree to which our

questions work by looking at data after they are administered

1.Predictive relationships of responses known in theory (e.g., how much do you like your job? What is your level of job stress?)

2.Comparison of data from questions that ask almost the same thing (but are worded differently)

3.Comparison of answers to records (when data available)

4.Consistency of answers from a respondent at two time points

ActivityWith a partner or small group, design a good question

ActivityWith a partner or small group, design a good question

Now, make it bad in just one of the ways I covered

ActivityWith a partner or small group, design a good question

Now, make it bad in just one of the ways I covered

Come type on my computer when ready

Some Bad Questions1. Xxx2. Xxx3. Xxx4. Xxx5. Xxx6. Xxx7. Xxx8. Xxx9. Xxx10. xxx

Activity – fix the questions1. Xxx2. Xxx3. Xxx4. Xxx5. Xxx6. Xxx7. Xxx8. Xxx9. Xxx10. xxx

Another Activity(pick an area to evaluate)Write 3-5 questions that fit your topicIn the same groups, review the items to

make sure they are good.Consider covariates (usually demographic

variables) you might want – these are also called blocking variables