Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications...
-
Upload
derick-berry -
Category
Documents
-
view
215 -
download
0
Transcript of Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications...
![Page 1: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/1.jpg)
Surveys 101: Methodology & Good QuestionsPart I of II
Joyce Chapman, Consultant for Communications & Data AnalysisState Library of North Carolina, 4-16-2014
1
![Page 2: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/2.jpg)
AgendaPlanning a surveyDistributing a surveyBest practices for survey design
This webinar will be recorded and made available here:http://statelibrary.ncdcr.gov/ld/webinars.html
Much info for this webinar comes from the book "Designing & Conducting Survey Research: A Comprehensive Guide." By Louis Rea & Richard Parker.
2
![Page 3: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/3.jpg)
What is a survey and when do we use it
?
3
![Page 4: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/4.jpg)
Before you beginWhat problems are you trying to solve? What questions are you trying to answer? What data will help you answer the questions? Does any of this data already exist?For colleges and universities: do you need IRB approval?
4
![Page 5: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/5.jpg)
Clearly define the target population for the survey
5
![Page 6: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/6.jpg)
How do we determine sample size?
This is where it gets tricky for us non experts!
Try something simple like a sample size calculator:
http://www.surveysystem.com/sscalc.htm
6
![Page 7: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/7.jpg)
How do I select participants who accurately reflect my population?
Ideally you want a random sample, not a convenience sample
7
![Page 8: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/8.jpg)
Types of survey distributionMail-out surveysWeb-based surveysIn-library paper-based surveysTelephone surveysIn-person interviewsIntercept surveys
8
![Page 9: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/9.jpg)
Types of survey distributionMail-out surveysWeb-based surveysIn-library paper-based surveysTelephone surveysIn-person interviewsIntercept surveys
Most common types used in libraries
9
![Page 10: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/10.jpg)
Web-based surveys: advantagesConvenienceRapid data collectionCost-effectiveAmple timeConfidentiality and security
10
![Page 11: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/11.jpg)
Web-based surveys
Web-based surveys: disadvantagesLimited respondent baseSelf-selectionLack of interviewer involvement
11
![Page 12: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/12.jpg)
In-person surveys: advantagesAbility to clarify and probeAbility to contact hard-to-reach populationsAssurance the instructions are followedObserved data (watch out!)
12
![Page 13: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/13.jpg)
In-person surveys
In-person surveys: disadvantagesInterviewer-induced biasLess anonymity
13
![Page 14: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/14.jpg)
Survey design
14
![Page 15: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/15.jpg)
15
The introductory statement
![Page 16: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/16.jpg)
The introductory statement
Mention the organization conducting the survey, state the relationship between the sponsoring institution and the respondent. "The Duplin County Public Library is conducting a
survey of residents in order to assess community opinions about services provided by the library."
Make a general statement establishing the objectives and goals of the survey and the significance of results to the respondents. "The purpose of this study is to identify those needs
that undergraduate students feel should be addressed in order to maintain a competitive 21st academic library."
16
![Page 17: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/17.jpg)
The introductory statement
Clarify the basis of the sample selection. "The library is particularly interested in the opinion of
patrons who live on the west side of town, and as such you have been selected at random."
State that participation is valued and answers are neither correct nor incorrect. Participation is confidential. “Your opinions are very important to us. You should know
that there are no right or wrong answers and that your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents."
If you are conducting an in-person or phone survey, identify yourself by name and job title.
17
![Page 18: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/18.jpg)
18
Question sequence
![Page 19: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/19.jpg)
Question sequence
Begin with "introductory questions." Sensitive questions should be placed near the
end. Consecutive questions that tend to evoke
reflexive responses should be minimized or respondents should be sensitized to the differences.
19
![Page 20: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/20.jpg)
20
Question format
![Page 21: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/21.jpg)
Question format
Advantages of closed-responseAnswer sets are uniform; facilitates comparison.Makes the question clearer to the respondent. Categories might remind the respondent of alternatives that otherwise would have been forgotten.Sensitive questions are often better addressed this way because you are pre-establishing an implicitly "acceptable" range of answers.They are easier on the respondent, increasing the likelihood that the response rate will be higher.
21
![Page 22: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/22.jpg)
Question format
Disadvantages of closed-responseParticipants may select a response randomly rather than in a thoughtful way.Increased possibility that simplicity of format leads to entry errors.Compel respondents to choose a "closest representation" of their actual response. Subtle distinctions among respondents cannot be detected within the pre-established categories.
22
![Page 23: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/23.jpg)
Question format
Open-responseRecommended to use sparingly / only when needed.Analysis will requires time-consuming categorization of open-ended response data by the researchersRequires greater communicative skills for respondent (spelling, writing, computer usage)Good compromise: use a close-ended question with a follow-up that's open-ended.
Do you want X? Yes__ No__. If no, why not? ____________
23
![Page 24: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/24.jpg)
24
Question phrasing
![Page 25: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/25.jpg)
Jargon, colloquialisms, word choiceThink of your audience. Wording should be simple. Avoid technical words.
X "How many times in a typical week do you use the library's Internet-enabled public access
computers?"O "How many times in a typical week do you use the
library's computers [to access the Internet]?"
25
![Page 26: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/26.jpg)
Abbreviations and acronymsAssume your audience does not know any of these!
X “CCPL is interested in replacing its OPAC."O “Cleveland County Public Library is interested in
replacing its online catalog.”X “How frequently do you use our ILL services?”
O “How frequently do you use our Interlibrary Loan services? [This service allows us to request
material from another library for you if we do not have it here].”
26
![Page 27: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/27.jpg)
Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "With which ethnic group do you most closely identify?"
[someone might interpret this as "which group do I best get along with?" instead of "which ethnic group am I a
member of?"]
O "Please indicate your race or ethnicity."
27
![Page 28: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/28.jpg)
Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "What is your income?"
O "What is your income before taxes? Include salary as well as other sources of income."
X "How many people are there in your household?"
O "Including yourself, how many people are there in your household?"
28
![Page 29: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/29.jpg)
Confusingly phrased questionsMake sure your wording is not unnecessarily confusing. The respondent should not have to spend time re-reading/interpreting the question.
X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"
It seems likely __ It seems unlikely __
O "If the library had a Maker Space, would you use it?"
Yes__ No__ Unsure__
29
![Page 30: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/30.jpg)
Avoid “double barreled" questionsA double barreled question introduces two or more issues with the expectation of a single responseX "Are you satisfied with opening hours at the West branch
and at the main library?
O "Are you satisfied with opening hours at the West branch?"
"Are you satisfied with opening hours at the main library?"
O "Are you satisfied with opening hours at the West branch?"
"Are you satisfied with opening hours at the main library?"
30
![Page 31: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/31.jpg)
Avoid “double barreled" questionsA double barreled question introduces two or more issues with the expectation of a single response
X "Is our staff friendly and professional?"
O "Is our staff friendly?"
"Is our staff professional?"
31
![Page 32: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/32.jpg)
Do not use manipulative informationCertain questions require some explanation/background. Be careful that explanatory statements do not unduly influence responses.
X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for
our library by designating only $1.15 per resident?"
O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per
resident?"
32
![Page 33: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/33.jpg)
Do not use manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info.
O "Do you believe that the county government is adequately allocating funds for our library by designating
$1.15 per resident?"
O "If you were to learn that the county government spends approximately $10 per resident on landscaping public
areas, would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"
33
![Page 34: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/34.jpg)
34
Response formatting
![Page 35: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/35.jpg)
Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer.
X Group study rooms O Digital media lab
Digital media lab E-books
Laptop lending Group study rooms
E-books Laptop lending
Printing/copying Printing/copying
35
![Page 36: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/36.jpg)
Interval scale categoriesShould be as equal as possible in size. You also want each category to have an equitable number of respondents. Do not allow to overlap. Provide an unbounded final category.
X Age 0-10 O Age 0-9
Age 10-15 Age 10-19
Age 15-20 Age 20-29
Age 20-50 Age 30-39
Age 50-75 Age 40-49
Age 50+
36
![Page 37: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/37.jpg)
Multiple responseSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.
X For which of the following reasons do you use the library?
X__ Y__ Z__
O For which of the following reasons do you use the library? Choose
all that apply.
X__ Y__ Z__
37
![Page 38: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/38.jpg)
Appropriate response choicesSurveys can be very frustrating when the questions are fixed response but do not have all the appropriate answer choices.Make sure you provide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate.
X Does the laptop lending program meet your needs?
Yes__ No__
O Does the laptop lending program meet your needs?
Yes__ No__ N/A__ [or “I’ve never used this program__”]
38
![Page 39: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/39.jpg)
You've drafted the survey!
39
![Page 40: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/40.jpg)
Review each question against a checklistDo you have an introduction with all the parts we talked about? Are the first questions easy/factual? Are sensitive questions near the end?Are reflexive response questions minimized or is respondent sensitized?Have you clarified multiple v. single choice responses?Are you using closed- v. open-ended responses appropriately? Do you have "other" open-ended responses where needed?Have you done away with all acronyms/abbreviations?Have you avoided jargon, ambiguously worded, confusing, and double barreled questions? Do you have any manipulative language?
40
![Page 41: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/41.jpg)
41
![Page 42: Surveys 101: Methodology & Good Questions Part I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014.](https://reader030.fdocuments.us/reader030/viewer/2022032805/56649edf5503460f94bef6f5/html5/thumbnails/42.jpg)
Questions?
Contact: [email protected]
Find this Powerpoint/recording: http://statelibrary.ncdcr.gov/ld/webinars.html
Sign up for part II of the Survey 101 webinar series here: http://statelibrary.ncdcr.libcal.com/event.php?id=399040
42