CBR 303: A Hands-On Approach to Developing Survey Tools in Community-Based Research.
-
Upload
eric-sanders -
Category
Documents
-
view
212 -
download
1
Transcript of CBR 303: A Hands-On Approach to Developing Survey Tools in Community-Based Research.
CBR 303: CBR 303: A Hands-On Approach to Developing Survey A Hands-On Approach to Developing Survey
Tools Tools in Community-Based Researchin Community-Based Research
Welcome & Introductions
Name & affiliation
Experience with CBR and/or survey tools
What you hope to learn today
Objectives
Upon completion of this workshop you will be able to:
Differentiate between what you need to know and what is nice to know
Use various strategies for asking survey questions
Apply different types of response options
Recognize various approaches for administering surveys
Describe how to increase response-rates
Develop a strategy to manage data
Identify various strategies for presenting survey results
Agenda
Review of CBRDefinition and purposes of a surveyIdentify features of ‘best’ surveysSteps in the survey processEstablish research objectives; identify what you
need to know vs. what is nice to knowBegin to develop research questionsRefine research questions
What is CBR?
When you hear “community-based research,” what words or phrases come to mind?
How would you define it?
CBR definition
CBR is a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities.
Minkler & Wallerstein (2003)Community-Based Participatory Research for HealthJossey-Bass (2004)
Principles of CBR
ethicalreview
process oriented
community relevance
social actionoutcomes
joint dataownership
varied methods
capacity building
community initiation
CBR
Introduction to surveys
Why do we do surveys?
What questions can they answer?
What questions can’t they answer?
When is doing a survey a bad idea?
What is a survey?
A process for gathering information, without detailed verification, on the activity being examined.
The systematic collection, analysis and interpretation of information about some area of study.
Information is collected primarily by means of sampling and interviews with the selected individuals.
Survey Goals
To produce detailed data that will allow researchers to:|
describe the characteristics of the group under study,
test theories about the group, and
generalize results beyond just those people who responded to the survey
Purpose for Surveys
Understand the specific activity under review
Identify significant areas warranting special emphasis
Obtain information for use in making organizational changes
Identify future changes/goals for the organization
Features of the “BEST” Surveys *
Specific objectives
Straightforward questions
Sound research design
Sound choice of population or sample
Reliable & valid instruments
Appropriate analysis
Accurate reporting of survey results
Reasonable resources* From The Survey Handbook, Arlene Fink, 1995
Survey process
Assemble your team
Establish your research objectives and questions
Determine your sample
Choose interview methodology
Design questionnaire
Pre-test the questionnaire
Administer survey
Collect and enter data
Analyze data
Present data
1. Assemble your team
Statistician +/ Social Scientist +/Epidemiologist
Community members +/ Community leaders +/ Service providers
Remember to consult your team early and often during planning, implementation analysis and dissemination.
2. Establish Research Objectives & Questions
What do you want to know?
What do you need to know?
What will you measure? Why this factor?
How will the information be used?
Is a survey the best way to get the information you need?
What kinds of resources (time, staff, money) do you have in place to make this happen?
Try not to mix too many objectives into a single survey. Keep it focused.
Example – Research Objective
To identify the factors that lead to early school withdrawal for lower income youth in the community
Don’t Reinvent the Wheel
Before you develop your own tools see what other people have done.
Is it worth redoing a study that has been done 100 times?
What makes your research new? How do others ask similar types of questions?
(check the literature) Can you use or adapt a pre-existing instrument to
meet your needs?
Standardized vs. Homemade Instruments
Standardized Homemade
can compare your results hard to compare
less costly more costly
may not be totally appropriate
can tailor to your needs, may be more relevant
higher credibility credibility depends of investigators
may not exist! always creatable!
Next … Hypothesis
X Y
Lower Socio-economic Status Early School Withdrawal
‘Operationalizing’ Variables
Lower Socio-economic Status
Actual Income Occupation Work
Shift/Conditions Time Spent with
children Ability to provide
educational supports
Early School Withdrawal
Withdrawal from school prior to completion of Grade 12
Marketable skills Relationship with school
staff Relationship with peers Social attitudes
From Concept to Questions Example: Socio-economic Status
Construct Socio-economic status
Indicator Income level
Operationalized Variables
What was your total family income during the past 12 months?
Measurement a) Under $10,000 b) $10,001-$25,000 c) $25,0001 - $50,000d) $50,001-100,000 e) more than $100,001
From Concept to Questions Example: Socio-economic Status
Construct Socio-economic status
Indicators Educational attainment
Operationalized Variables
What is the highest level of school you have completed?
Measurement a) Primary Schoolb) Some High Schoolc) Completed High Schoold) Some College e) Completed Collegef) Graduate/Professional Degree
Exercise: From concept to questions
On your Survey Development Template:
1. Identify the hypothesis (e.g. X causes Y) underlying your objective.
2. List the related factors for each variable.
3. Develop a related research question, operationalizing your variables.
4. Develop a specific measurement for the question.
Important Tips!
Each question on your survey needs to be JUSTIFIED and LINKED to your research objectives
Obtain ALL of the ESSENTIAL information
Obtain ONLY the ESSENTIAL information
Example: Possible question topics
AgeSexSexualityExperiences in schoolLevel of academic completionExperiences of successSESSchool
NeighborhoodWhat their parents do
Extracurricular activities
Hobbies
Relationships with teachers
Relationship with peers
Relationship with parents
Part-time employment
Triaging Questions
High: Absolutely essential
Medium: Very valuable for decision making
Low: Supportive data to enhance understanding
Strategies for asking questionsDifferent Types of Questions
Open-Ended
The respondent answers in their own words
More demanding and time consuming for respondents
Lower response rate
Harder to categorize, analyze and interpret
Good when you don’t know how people will respond, useful in the preliminary pilot stage
E.g. Please describe any barriers that you face accessing our service?
Closed/Structured
The respondent answers from a range of choices
Less time consuming for respondents
Higher response rate
Easier to categorize, analyze and interpret
Requires more front-end effort (designing the questions)
Can oversimplify or make assumptions
Response categories must be inclusive
Limits answers to options presented
Can compromise with ‘other:____________’ category
Types of questions: Two choices
Do you like ice cream?
Yes No
Types of questions: Multiple choice
Which is your favourite ice cream flavour?
Chocolate Vanilla Strawberry Banana Other: __________ (please specify)
Types of questions: Checklist
Please select all of the ice cream flavours that you have eaten in the last month?
Chocolate Vanilla Strawberry Banana Other: __________ (please specify)
Types of questions: Ranking
Please order the following ice cream flavours by preference (where 1=your favourite & 4 = your least favourite)?
Chocolate Vanilla Strawberry Banana
Types of questions: Rating/Likert Scale
Please rate the degree to which you agree/disagree with the following statements:
1. Ice cream is my favourite dessert.
Strongly agree Agree Disagree Strongly Disagree
2. Ice cream's only redeeming feature is that it can have chocolate.
Strongly agree Agree Disagree Strongly Disagree
Helpful options
Excellent – good – fair – poor
Approve/Disapprove, Favour/Oppose,
Too many – about right – not enough
Often – Regularly – Seldom
Always – most of the time- some of the time- rarely – never
More likely – no difference – less likely
(Agree) 1 2 3 4 5 (Disagree)
Evens or Odds
Even numbered scales force a choice
Odd numbered scales allow for a neutral stance
Beware of the forced choice: research shows that those who would have gone for neutral are more likely to go for positive choices when forced!
Beware of neutral categories: research shows that people ‘revert to the mean’ (ie they are most likely to choose neutral).
Types of questions: Numerical
How many ice cream cones have you eaten in the last week? _____
How old are you? ______
What year were you born?_____
Tips for Question Design
Don’t ask questions that you would not answer
Focus on behaviours not labels
Keep your values/assumptions out of questions
Make sure that questions are not too demanding
Avoid abbreviation or professional jargon
Use simple, direct and familiar words/terms (slang can be OK)
Tips for question design cont’d …
Make questions clear, specific and as short as possible
Make sure everyone will interpret the question the same way
Specify the frame of reference (e.g. time, place)
Avoid leading questions
Be aware of ‘social desirability’
Make sure that questions are applicable to all respondents
Ordering your questions
Always give an intro (title, sponsor, purpose etc)
Start with easy questions that everyone can answer (builds trust & confidence)
Introduce sensitive stuff gradually
Ordering your questions
There should be a logic to sequence and flow, use transition statements or subheadings
Move from general to specific
Ask about things in chronological order
Always say thank you at the end and leave room for questions/comments/feedback
Common Survey Errors!
Beware!
Problem questions: The double barrel
Beware of the “ANDs”
Do you like to watch TV andand eat potato chips?
How would you rewrite?
Problem Questions: The Incompetent Experts
Sometimes people don’t know enough to answer the questions or may not be able to estimate the answers
To what extent do you agree with the economic foreign policy of Latvia?
How many cans of soup have you bought in the last year?
Problem questions: The long list
Sometimes having questions with too many response categories can be overwhelming, especially when you are asking folks to rank order them:
Please rank order these TV shows (where 11 is your favourite and 1717 is your least favourite)…
Problem questions: The double negative
I disagree that the statement is false…
Problem questions: Not mutually exclusive
How old are you?
20-25 25-30 30-35 35-45
Problem questions: Vague wording
Do you have a partner?
Yes No
Problem questions: The mismatch
Do you work out?
Strongly agree Agree Disagree Strongly disagree
The Vague Response
When is your birthday? __________
Answer: tomorrow
Sex, drugs and all the good stuff
How do we Ask about Sensitive Topics?
Asking sensitive questions
Don’t ask what you would not answer!
Consider social desirability bias
Some strategies for asking sensitive questions include:
Self-administered questionnaires Ensuring anonymity or confidentiality Careful wording Providing appropriate response ranges
Ways of Asking Threatening Questions
Various techniques have been developed to ask embarrassing or sensitive questions in non-threatening ways.
The following are examples of several such techniques as applied to the question:
"Did you kill your boss?"
Ways to ask: “Did you kill your boss?”
The Casual Approach "Did you happen to have killed your boss?“
The Numbered Card “Please read the number which corresponds with
what happened to your boss.“
1. Natural death2. I killed him/her3. Other (please specify)
“Did you kill your boss?”
The “Everybody’s Doing It” Approach "As you may be aware, many people have been killing their
bosses these days. Would you happen to have killed yours?"
The “Sandwich” Method "Have you ever done any of the following?"
a. Fire-bombed the Parliament buildingsb. Murdered your bossc. Detonated a nuclear device
“Did you kill your boss?”
The Assumption Method "How many of your bosses have you killed in the last
ten years of work?“
The “Other People” Approach
a. "Do you know of other people who have killed their bosses?"
b. “Have you done that yourself?“
“Did you kill your boss?”
The Illustrative Technique
“Which of the following pictures most accurately depicts what happened to your boss?"
Designing a good survey is hard work
Give Yourself Plenty of Time!
Pilot Testing
TEST! TEST! TEST! TEST! TEST! Test with colleagues & participants – the more
the better. This will save you time in the long run!
Discover Poor question wording or ordering or
instructions Suggest additional response categories Time issues
Getting your questions answered….
Who are you going to talk to?
3. Determine Your Sample
Who will you interview?
Who is your clientele?
Will this group provide you with the necessary data to meet the goals of the project?
Keep the sample large enough for a sufficient amount of data but small enough so that it is manageable.
Sampling
Eligibility or inclusion/exclusion criteria
Who do you wantwant to talk to?
Who do you not wantnot want to hear from?
Inclusion & Exclusion
Research Question
How does socio-economic factors impact on students’ ability to remain in school?
Target Population
High School Students
Inclusion Criteria
13-18 years oldStudents in school & Early school leaversParental and youth consent
Exclusion Criteria
Use alcohol, marijuana or other drugs more than once a week
Sampling
Probability Sampling (ideal)Probability Sampling (ideal) Simple random sampling Stratified random sampling Systematic random sampling Cluster random sampling
Non-random sampling (hard-to-reach, pilot)Non-random sampling (hard-to-reach, pilot) Whole group Convenience Snowball Quota
Sample Size
Best way to determine this is to talk to a statistician!Considerations include:
CostEffect Size
Response rate
Getting your questions answered….
How are you going to do it?
4. Choose an Interview Methodology
How will you conduct the interview?
mail out Questionnaires individual interviews case study focus group on-line surveys
Consider the make up of your clientele: language; accessibility language; accessibility options; age; educationoptions; age; education
Types: Face to Face (door to door)
BENEFITS
Interviewer administers survey
Ensures consistency & complete response
Allows for clarification & probing
Good if there are lots of ‘skips’
High response rateTakes care of literacy issues
CONSIDERATIONS
ExpensiveConfidentiality & privacy
concernsInterviewer biasSocial desirability bias may
be stronger
Types: Telephone
BENEFITS
Interviewer administers survey
Ensures consistency & complete response
Allows for clarification & probing
Good if there are lots of ‘skips’
Less expensive than face to face
CONSIDERATIONS
Less response rate than face to face
Miss those without a telephoneHard in this climate of
‘telemarketers – must be SHORT!
Types: Self-completed (mail, web)
BENEFITS
Survey completed by respondent
Low-costCan be good if there is lots
of ‘confidential’ or ‘embarrassing’ questions
Increase anonymity of responses
CONSIDERATIONS
Requires literacyVariable completeness of answers
Bad if there are lots of ‘skips’ or confusing parts
Lower response rate
Factors to Consider when Choosing a Method
Cost, budgetHuman resourcesEquipmentExpertiseWho is likely to be reached/not reachedWhat is your target population most likely to respond to?BiasResponse rateParticipant burdenFollow-up necessary?TimelinesSample size
Getting your questions answered….
But nobody wants to talk to me?
Improving Response Rate
Keep things short & simple
Ensure confidentiality
Target the appropriate population
Make it easy/convenient
Clearly communicate purpose, importance & relevance
Give advance notice
If you can – provide rewards or incentives
Increasing response rate: Layout matters!
Create a bookletPrint on white paper, don’t look like an adNo questions on 1st page – just title No questions on last page – room for commentsOrder fun/easy questions 1st, harder in the middle
and demographics at the endThe first question matters: it should be easily
answerable!The layout should be neat and clean and easy to
followFrom Dillman handout
Getting your questions answered….
What do I with all these surveys?
8. Manage Your Data
Expect to collect data 2-3 weeks after distributing
Stay organized
Remember confidentiality
Determine ahead of time (before or while survey is being developed) data management program (e.g. NVIVO or NuDist for qualitative; SPSS, SAS, Stata for quantative)
Determine ahead of time who will enter data; provide them with adequate training
Talk to your statistician!
9. Analyze Data
Identify themes & patterns
Frequencies (counts of things, how many times things happen; get mean, median, mode)
Descriptives (percentages or raw numbers of demographics and other factors that would be nonsensical to take an average such as gender or ethnicity)
Outliers (things that fall outside the expected values)
My questions are answered….
Presenting Results
Numbers can be deceiving…
10. Report Results
Make sure to acknowledge both the strengths & limitations of your data beginning with strengths!
ALWAYS CONTEXTUALIZE YOUR DATAALWAYS CONTEXTUALIZE YOUR DATA
How many people did you talk to? How did you decide who to talk to?
(random, convenience) Who did you talk to? How easy is it to generalize your results?
Presentation Matters
TablesGraphsPicturesWords
Features of the ‘bestbest’ Surveys
Specific objectives
Straightforward questions
Sound research design
Sound choice of population or sample
Reliable & valid instruments
Appropriate analysis
Accurate reporting of survey results
Reasonable resourcesFrom The Survey Handbook, Arlene Fink, 1995
Reflection & Next Steps
Reflect on and record 1-3 things you learned today.
Record 1-3 related goals you have going forward.
Record 1-3 related next steps you plan to take upon leaving the workshop.
Objectives
Having completed this workshop you should now be able to:
Differentiate between what you need to know and what is nice to know
Use various strategies for asking survey questions
Apply different types of response options
Recognize various approaches for administering surveys
Describe how to increase response-rates
Develop a strategy to manage data
Identify various strategies for presenting survey results
Acknowledgements
Several of the slides presented here today were developed by San Patten, MSc., Community-Based Research Coordinator for the Alberta Community Council on HIV
CBR 303: CBR 303: A Hands-On Approach to Developing Survey A Hands-On Approach to Developing Survey
Tools Tools in Community-Based Researchin Community-Based Research