Statistical Issues in Data Collection and Study Design For

Community Programs and Research

October 11, 2001

Elizabeth Garrett

Division of Biostatistics

Department of Oncology

esg@jhu.edu

Overview

• Goals of Data Collection and Study Design

• Key concepts– Reliability – Validity

• “Latent” Constructs

• Study Designs

• Potential Biases

Goals of Data Collection

Two broad goals*

– evaluation of intervention• controlled

• uncontrolled

– summary of population• demographics

• attitudes

The Goal of Study Design

To devise a model for some complex etiologic or clinical process that

gives valid and precise inference.

Issue Specific to Interventions

• Outcomes tend to be “soft”

• Not always an easily quantified response

• We often use one or more “surrogates” to measure outcomes.

Key Concepts• Reliability: Is the data that you are collecting a reliable or

reproducible measure?• Has to do with how closely your measure correlates with the

underlying construct you want to measure.

• Truth = Observed Data + Error– If you collected the same data tomorrow, would you get the same

answer?– If you ask a related question, will the two questions have

correlated answers?– If two different “raters” (i.e. collectors) evaluate the same

individual, do they get the same data?

Validity

• Validity: Is what you are collecting measuring what you want to measure?– Face validity– Construct validity– Criterion validity– Etc.

Valid

Invalid

Reliable(but still invalid!)

Valid and Reliable

Validity

• Internal Validity:– Valid for the population from which you

sampled

• External Validity– Generalizable to a broader population

Latent Constructs

• Definition: A latent variable is a variable that cannot be directly measured.

• Examples:– Quality of Life– Socio-economic status– Distress– Depression

Latent Constructs

• Need to be measured using multiple variables• Variables, taken together, should “define” the construct• Methods should be decided upon ahead of time and

data collection needs to be considered.• Examples: latent class analysis, factor analysis• Coding is important

– likert scale: “On a scale of 1 to 7…..”– binary: yes/no, present/absent– continuous: age, income

Some Study Design Types• Cross-sectional, No Intervention

– Attributes• quantify community

• “summarize” attitudes, demographics, etc.

• descriptive statistics– means, medians, standard deviations

– “pictures” of the sample: histograms, boxplots

• “hypothesis generating”, and NOT “hypothesis testing”

• simplest conceptually

• Cross-sectional, No Intervention (cont.)– Issues to think about

• sampling– Who?

– When?

– Where?

• Data (this is general to ALL study designs)– format?

– Binary versus continuous versus ordinal versus categorical?

– open-ended?

• Pre-post Design, One group (uncontrolled)– Was intervention successful?– Attributes:

• Compare baseline to follow-up

• simplest when only two time points are collected.

• Convenient that each individual serves as his/her own control

• Hypothesis testing: – Ho: intervention worked

– Ha: intervention did not work

• Some methods: binomial tests, signed rank test, paired t-test, regression methods

– Issues to think about• when should “success” be measured?

– 1 week? 1 month? Both?

– What if effect at 1 month but “washed out” by 6 months?

• How is success measured?

– Yes/no? Continuous change in score?

• Learning effect

– bias of questionnaires

– is this the most appropriate design if there is a potential learning effect?

• “Placebo” effect could play a role.

• Adherence!

– Is the effect of intervention smaller than anticipated because some study participants did not adhere?

• Confounders and effect modifiers!

– Are there some individuals that respond more strongly to the intervention than others in such a way that is predictable (e.g. age? weight? political views?)

• Pre-Post, Two Groups (Controlled)– Does intervention group improve more than the

control group?– Attributes

• similar to pre-post, one group

• can quantify placebo and learning effects (caveat)

• hypothesis testing:– Ho: effect in control group = effect in treatment group

– Ha: effect in control group effect in treatment group

• Some methods: 2 sample t-test, rank sum test, fisher’s exact test, regression methods

– Issues to think about• We have a measure of placebo effect

• blinding or masking? Is it possible?

• Randomization– Balance?

– Stratification necessary?

– Matched?

• ITT versus Treatment received?

• Drop out

• Adherence

Other Study Designs

• Case-Control Studies

• Cohort Studies (aka Prospective Study)

• Ecologic Study

Potential Biases to Keep in Mind• Selection Bias (IV)

– individuals who join the study are not representative of the population in a way that affects the outcome.

• Information Bias (IV)– measures tend to be biased in one direction

• Confounding (IV)– Mixing of effects leads to wrong inference

• Effect Modification (IV)– effect of treatment depends on another factor (e.g. age)