Non-Experimental designs: Surveys & Quasi-Experiments Psych 231: Research Methods in Psychology.
-
date post
19-Dec-2015 -
Category
Documents
-
view
219 -
download
1
Transcript of Non-Experimental designs: Surveys & Quasi-Experiments Psych 231: Research Methods in Psychology.
Announcements Lab attendance is critical this week because group projects are being administered Attendance will be taken.
Turn in the group project rating sheet 1
Non-Experimental designs Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc). • Surveys• Quasi-Experiments• Developmental designs• Small-N designs
This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each
Surveys What are they? (review chapter 7)
Questionnaires and interviews that ask people to provide information about themselves
Why conduct them? To compliment experimental work
• Good/common first step, can collect a lot of data about a lot of variables
• Do not have to directly observe behaviors Best way to collect some kinds of information:• Descriptive, behavioral, and preferential
• (e.g. demographic information, recreational behavior, and attitudes)
Surveys Advantages
One can investigate internal events (for example, attitudes & opinions)
Can generalize about an entire population based on relatively small samples of individuals
Large amounts of data can be collected quickly with relatively little cost (effort, time, etc.)• But they’re often not as “cheap” as you may think
Surveys Disadvantages
Correlational: causal claims shouldn’t be made
Non-response bias• Why doesn’t everybody respond?• Does response rate interact with variables of interest?
Large data sets are sometimes difficult to analyze
Self-reports may not be truthful• Response set - tendency to respond from a particular perspective (e.g., how a “moral” person would answer)
Stages of survey research Stage 1) Identify the focus of the study and select your research method What are the objectives of the research?
Is a survey method the best approach? What kind of survey should be used?
Surveys methods Many different methods are used to administer surveys Group administration (e.g. MASS testing session)
Mail surveys Internet surveys Telephone surveys Face-to-face interviews Focus group interviews
Stages of survey research cont. Stage 2) Determining the research schedule and budget
Stage 3) Establishing an information base Find out what’s been done, what’s known
• E.g., Find other related surveys
Stage 4) Identify the sampling frame The actual population that the sample is drawn from (as opposed to the ideal population)• Think of it as operationalizing the conceptual level population
Stages of survey research cont. Stage 5) Determining the sample size and sampling method Review Probability and Non-Probability methods
(Oct. 10 lecture)
Voluntary response methods A kind of convenience sampling methods commonly used
•Problem: You typically get only individuals with strong opinions to respond, so the results are often extremely biased
Should leftover Halloween candy be given out to students who get an A on the exam?
Should leftover Halloween candy be given out to students who get an A on the exam?
Call 123-NONO if you think NOCall 123-4YES if you think YES
Importance of sample size
Sampling error - how is the sample different from the population? Confidence intervals
• An estimate of where the mean or percentage in the overall population is, based on the sample data • “John Doe has 55% of the vote, with a margin of error ± 3%”
• Margin of error (that “± 3%” part)• Which would you be more likely to believe
• We asked 10 people …• We asked 1000 people …
• The larger your sample size, the smaller your margin of error will be.
Stages of survey research cont. Stage 6) Designing the survey instrument Question construction: How the questions are written is very important• Clearly identify the research objectives
• Do your questions really target those research objectives?
• Take care wording of the questions• Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc.
• How should questions be answered?
Good and poor questionsGoodPoor
Was the FDC negligentby ignoring the warningsabout Vioxx during testingand approving it for sale?
a) Yesb) Noc) Unsure
If the FDC knew that
Vioxx caused serious
side effects during testing,
what should it have done?
a) Ban it from ever being sold
b) Require more testing before approving it
c) Unsure
Problem: emotionally charged words
Problem: emotionally charged words
Good and poor questionsGoodPoor
Are you against same sexmarriage and in favor of aconstitutional amendmentto ban it?
a) Yesb) Noc) Unsure
What is your view on same sex marriage?
a) I think marriage is a matter of personal choice
b) I’m against it but don’t want a constitutional amendment
c) I want a constitutional amendment banning it
Problem: Biased inmore than one direction
Problem: Biased inmore than one direction
Survey Questions Question types
Open-ended (fill in the blank, short answer)• Can get a lot of information, but• Coding is time intensive and potentially ambiguous
Close-ended (pick best answer, pick all that apply)• Easier to code• Response alternatives are the same for everyone
Rating scales • Used for “how much” judgments
• e.g., Likert scale – measures attitudes, agree/disagree
• Take care with your labels• Range of scores, anchors
Stages of survey research cont.
Stage 7) Pre-testing the survey instrument Fix what doesn’t seem to be working
Stage 8) Selecting and training interviewers For telephone and in-person surveys Need to avoid interviewer bias
Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a final report
Error in survey research
Measurement error Same old issues of validity and reliability• Are your questions really measuring what you want them to?
• Do you get similar answers with repeated questions
Error in survey research
Are there differences in your sample (compared to the population as a whole)?
Response rate • What proportion of the sample actually responded to the survey?• Hidden costs here - what can you do to increase response rates
• Non-response error (bias)• Is there something special about the data that you’re missing? From the people who didn’t respond
Sampling error
Measurement error
Quasi-experiments What are they?
Almost “true” experiments, but with an inherent confounding variable
General types1) An event occurs that the experimenter
doesn’t manipulate• Something not under the experimenter’s control
• (e.g., flashbulb memories for traumatic events)
2) Interested in subject variables– high vs. low IQ, males vs. females
3) Time is used as a variable
Quasi-experiments Advantages
Allows applied research when experiments not possible
Threats to internal validity can be assessed (sometimes)
Quasi-experiments Disadvantages
Threats to internal validity may exist Designs are more complex than traditional experiments
Statistical analysis can be difficult•Most statistical analyses assume randomness
Quasi-experiments Program evaluation
– Research on programs that is implemented to achieve some positive effect on a group of individuals.– e.g., does abstinence from sex program work
in schools– Steps in program evaluation
– Needs assessment - is there a problem?– Program theory assessment - does program
address the needs?– Process evaluation - does it reach the
target population? Is it being run correctly?
– Outcome evaluation - are the intended outcomes being realized?
– Efficiency assessment- was it “worth” it? The the benefits worth the costs?
Quasi-experiments Nonequivalent control group designs
with pretest and posttest (most common)(think back to the second control lecture)
participants
Experimentalgroup
Controlgroup
Measure
Measure
Non-Random Assignment
Independent Variable
Dependent Variable
Measure
Measure
Dependent Variable
– But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity)
Quasi-experiments Interrupted time series designs
Observe a single group multiple times prior to and after a treatmentObs Obs Obs Obs Treatment Obs Obs Obs Obs• Look for an instantaneous, permanent change
Variations of basic time series design Addition of a nonequivalent no-treatment control group time seriesO O O T O O O & O O O _ O O O
Interrupted time series with removed treatment• If treatment effect is reversible