Common Designs and Quality Issues in Quantitative Research Research Methods and Statistics.
-
date post
20-Dec-2015 -
Category
Documents
-
view
220 -
download
3
Transcript of Common Designs and Quality Issues in Quantitative Research Research Methods and Statistics.
Common Designs and Quality Issues in Quantitative Research
Research Methods and Statistics
Intended Learning Outcomes
To familiarise yourself with the different types of quantitative research designs commonly used in occupational psychology research
To understand the concepts of validity and reliability and why these are important to consider when designing research studies
What is Research Design
“A design specifies the logical structure of a research project and the plan that will be followed in the execution. It determines whether a study is capable of obtaining an answer to the research question in a manner consistent with the appropriate research methodology and the theoretical and philosophical perspectives underlying the study.”
(Sim & Wright, 2000: 27)
Elements of Research Designs phenomena/variables to be researched
how will these phenomena/variables be measured? (what method/technique?)
who/where will the data be collected from?
when will the data be collected?
what type of data will I have as a result?
what will be the consequences of this for data analysis?
Elements of Research Designs phenomena/variables to be researched
how will these phenomena/variables be measured? (what method/technique?)
who/where will the data be collected from?
when will the data be collected?
what type of data will I have as a result?
what will be the consequences of this for data analysis?
Common Designs
Group differences
Relationships between variables: correlations regression models
Surveys / questionnaires
Time series
Other designs
Group Differences
INTERVENTIONPRE
CONTROLPRE
INTERVENTIONPOST
CONTROLPOST
e.g. to determine the effect of a training intervention on scores
0
10
20
30
40
50
60
PRE POST
Ou
tco
me
Mea
sure
INTERVENTION
CONTROL
Group Differences Designs - Variations
No control group
More than two groups
More than one outcome measure
No time element
More than two time points
Etc.
Relationships between Variables
Bivariate relationships each participant is measured on two or
more variables (either both are categorical or both are ordinal or above)
Regression models based on linear correlations various predictor variables and one
outcome variable
Bivariate Relationships – Categorical Data
PUBLIC SCHOOL
PRIVATE SCHOOL
READING DIFFICULTIES
8 2
NO READING DIFFICULTIES
24 22
e.g. to find out whether the proportion of pupils with reading difficulties varies from public to private schools
Bivariate Relationships – Ordinal, Interval or Ratio Data
e.g. to find out how the amount of TV viewing is correlated with academic performance
0
1
2
3
4
5
6
0 5 10 15 20
Hours of Weekly TV Viewing
En
gli
sh
SA
T S
co
re
Regression
e.g. which are the best predictors of academic performance?
e.g. which are the best predictors for whether a child will get a statement of educational needs?
PREVIOUS SAT SCORE GENDER
FREE MEALS
TV VIEWING
ATTENDANCE RECORD
ACADEMIC PERFORMANCE
Surveys / Questionnaires
May be used: as an outcome measure (evaluation) to describe (the attitudes of) a
particular group – SURVEY
Surveys can be used to check for: differences between groups relationships between variables
Time Series multiple data points (50+) – recorded data
useful for evaluation when trend and/or seasonality are existent
0
5
10
15
20
25
30
35
Se
p-0
2
De
c-0
2
Ma
r-0
3
Jun
-03
Se
p-0
3
De
c-0
3
Ma
r-0
4
Jun
-04
Se
p-0
4
De
c-0
4
Ma
r-0
5
Jun
-05
Se
p-0
5
De
c-0
5
Ma
r-0
6
Jun
-06
Se
p-0
6
De
c-0
6
Ma
r-0
7
Nu
mb
er o
f R
efer
rals
Other Designs
Single case designs
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
Days
Ou
tco
me
Mea
sure
A A
B B
Which One to Choose…???Your choice of study design needs to take into account:
your research question
data available / feasible tests available
other details: trends / seasonality existent?
Making Sure your Study is a Good Quality One
Just two thing to worry about…
High internal and external validity
Validity and reliability of instruments
Internal and External Validity
Interval validity refers to the lack of confounding variables (related to design)(e.g. can we really conclude the children’s reading performance has improved because of our IV – intervention we introduced?)
External validity refers to whether we can generalise our results to our target population (related to sampling)
Threats to Internal Validity
REGRESSION TO THE MEAN
MORTALITY
COMPENSATORY RIVALRY
EXPERIMENTER BIAS
DIFFUSION OF BENEFIT
MATURATION
External Validity
Can we generalise our findings to other people/places/settings/conditions/etc.?
Related to: artificiality
does the experimental situation resemble the real world?
sample selection is our sample different from the
population you want to apply our findings to?
High Quality Instruments Validity: Does your test measure what it claims to? Reliability: Does it measure it consistently?
Not reliable therefore not
valid
Reliable but not valid
Both reliable and valid
Reproduced from Trochim (2002) on http://www.socialresearchmethods.net/kb/reliability.htm
Relationship between Validity and Reliability
RELIABILITY
VALIDITY
random error
systematic error
When Is Quality Compromised?
Ethics Practical issues
THINK ABOUT… How do validity and ethics relate to
one another? Is it ethical to sacrifice validity in a
study to make it more ethical?