1

Basics of Research Study Design

Karen H. Hagglund, M.S.

Medical Education

Ascension St. John Hospital

Email: karen.hagglund@ascension.org

Research:

How Do I Begin???

Take It “Bird by Bird”Anne Lamott

Let’s Take it Step by Step...

Identify topic

Literature review

Variables of interest

Research hypothesis

Design study

Power analysis

Write proposal

Design data tools

Committees

Collect data

Set up spreadsheet

Enter data

Statistical analysis

Graphs

Slides / poster

Write paper /

manuscript

Preliminary Steps to Research

Design: Identify Topic

Can be a difficult first step

Your interest & attitude

Observation

Discussion

Hot topics

Faculty research

Preliminary Steps to Research

Design: Research Hypothesis

Topic research question

Research question hypothesis

– Null hypothesis (H0)

• Predicts no effect or difference

– Alternative hypothesis (H1)

• Predicts an effect or difference

2

Hypothesis Testing

What Question are you trying to answer?

Will routine use of procedure checklists

improve patient safety?

Hypothesis (H1): Procedures consistently aided

by checklist use will cause

fewer patient complications.

Hypothesis (H0): Consistent checklist use

does not alter procedural

complications

Null

Hypothesis Testing

Goal of any Research Study

and of

the Statistical Analysis:

Disprove the Null Hypothesis

(H0)

Hypothesis Testing

TRUTH

Difference Exists No Difference(H1) (H0)

Difference Exists True Type I or

(Reject H0) Conclusion error

Power (1-)

No Difference Type II or True(Do not reject H0) errorConclusion

(1-)

Preliminary Steps to Research

Design: Variables of Interest

• Independent variables

• groups

• cause

• Dependent variables• end points

• effect or outcome

• These should be defined operationally

Operational Definitions

Define a hypothetical construct in

terms of the operations used to

measure it

Operational definitions are precise,

measurable & concrete

First used by Harvard physicist

Percy Bridgman in the 1920’s

Some are easily defined...

Height

Weight

Temperature

Blood Pressure

3

Some are NOT so easily

defined...

Pain

Patient satisfaction

Professional competency

?

How to approach difficult-to-

define variables of interest

Operational definitions are the best we can do

Look to the experts for their operational

definitions

Use standardized tests that measure the

construct of interest

– reliability measurements

– validity measurements

Example

Topic: Cancer & Smoking

Research Question: Is there a

relationship between smoking &

cancer?

H0: Smokers are not more likely to

develop cancer compared to non-

smokers.

H1: Smokers are more likely to

develop cancer than are non-smokers.

Let’s Design the Study!

4

Who are the patients that may

participate?

Subject Selection

Subject inclusion criteria

– conditions which must be met to enroll patients

Subject exclusion criteria

– conditions which preclude enrollment even if

inclusion criteria are met

– may interfere with measurements

– unsafe to enroll in study

Example

Subject Criteria

Inclusion Criteria

– Adults (>18 years)

– Smoke >1 pack

cigarettes/day

– Smoker for >1 year

Exclusion Criteria

– History of cancer

– Other co-morbid

conditions

Subject Selection

Include a wide range of patients if possible

(gender, race, socioeconomic groups) to be

representative of larger population

(generalizability, external validity)

Random selection from population

How Many Subjects Do I Need ?

5

Power Analysis

Make your best guess about results

– Base estimate on published studies

The smaller the effect you think you will

observe, the more subjects necessary

What is the Time Frame?

Estimate how long it will take to:

– Enroll patients

– Collect data

– Enter data into spreadsheet

– Analyze data

– Interpret data & prepare for presentation

Seasonal variation in outcome

Allow plenty of time!

Who will conduct the study?

Yourself

Fellow residents

Attending physicians

Nurses

In-service

Incentive

Role of Investigators

Clarify the role & duties of each

investigator & assistant

Do this early on to avoid later conflicts

Role of Investigators

Authorship (from Uniform Requirements)

– substantial contributions to:

• conception & design,

• or acquisition of data,

• or analysis & interpretation of data

– drafting the article or revising it critically for

important intellectual content

– final approval of the version to be published

Will the patients want to

participate?

Clear explanation of study

Personal benefit

– Likely benefit outweighs risk to patient

Contribution to advancement of medical

knowledge

Reimbursement for time & expenses

6

Hawthorne Effect

Studies conducted at Western Electric Plant, Illinois in 1927 by Elton Mayo

Workers assembling telephone relays

What increases worker productivity?

Working conditions manipulated in separate room

– Breaks

– Hours

– Lighting

– Type of work

So what happened?

Productivity increased

Employees felt special when part of study

Underscores need for control group

Methodological Issues

Strive to eliminate bias in study

– Subject selection bias

– Procedural bias

Must insure patient anonymity & confidentiality of records

Determine what information you are collecting & how you will collect it

– Demographics

– Risk factors, outcomes

– Don’t collect everything!

Let’s Poll the Audience !

What type of analysis should be

conducted to determine the number of

subjects needed for a research study?

Pow

er analy

sis

Analys

is of v

ariance

Estim

ation o

f the a

moun...

t-te

st

0% 0%0%0%

1. Power analysis

2. Analysis of variance

3. Estimation of the amount

of time you have before

your project is due

4. t-test

Types of Research Design

7

Research Design Terms

Prospective – Retrospective

Controlled – Uncontrolled

Randomized – Nonrandomized

Blinded - Unblinded

Considerations

Hypothesis should dictate your design

But often the time remaining in your

training program does!

– Prospective vs. retrospective

Should choose the best design your

hypothesis & time will allow

Study Goal

Descriptive

Explanatory

Descriptive Designs

Case report

– Presentation of one case

Case series

– Group of related consecutive cases

Population

– Describe characteristics of a large population

Program or course evaluation

– Evaluate success of program or course

Explanatory Designs

Observational

Experimental

Observational Designs

Case-control

Cohort

Cross-sectional

8

Case-Control Design

Retrospective data collection

Starting point is the adverse event

Group according to presence or absence of outcome or disease

Investigator attempts to find risk factors that predict disease status

Example

Smoking & Cancer

Researcher goes through charts of

patients who were diagnosed with

breast cancer (cases) over a 10 year

period

Examines smoking history

Compares to patients in system without

breast cancer (controls)

Example: Green Tea

Question: What is the association between green

tea and breast cancer risk in Asian Americans?

Interviewed Chinese, Japanese, and Filipino women

in Los Angeles

– 501 with breast cancer (cases)

– 594 with NO breast cancer (controls)

Asked to recall dietary habits, including intake of

black and green tea

Found that green tea drinkers had a decreased risk

of breast cancer

Wu AH, Yu MC, Tseng CC, Hankin J, Pike MC. Green tea and risk of breast cancer in Asian Americans. Int J

Cancer. 2003 Sep 10;106(4):574-9.

Resident Example: Bacteremia

Objective: To evaluate the efficacy of tigecycline in

patients with Gram-negative bacteremia

Chart review:

– 29 cases received tigecycline for bacteremia

– 87 controls matched for bacterial isolate

Cases were more likely to have been admitted from

an ECF, have a higher Charlson Weighted Index of

Comorbidity score, and a longer time to effective

antibiotic administration

Those receiving tigecycline had a poorer immediate

outcome and higher mortality

Assessment of Case-Control

Designs

Advantages

– good initial exploratory study

– can explore multiple risk factors

simultaneously

– efficient, relatively inexpensive & quick

– data may be available through record

review

– well suited for rare diseases

9

Assessment of Case-Control

Designs

Disadvantages

– questionable data quality

– recall bias (if interview patients)

– sampling bias

– can study only one outcome at a time

– cannot determine prevalence or incidence

– cannot demonstrate cause & effect

Cohort Design

Follow patients over period of time

Grouping according to risk factor

Later, compare incidence of the

outcome between those with risk factor

& those without

Example

Smoking & Cancer

Researcher chooses a group of people

including smokers & nonsmokers

Records smoking data

10 years later, determines incidence of

cancer

Example: Breakfast Cereal

What is the association between breakfast cereals

and risk of heart failure?

Asked 721,000 male physicians to report their

consumption of whole grain cereal at baseline, 18

wks., 24, 48, 72, 96, and 120 mos.

Surveyed every year for new medical diagnoses

Found that eating whole grain cereal was associated

with a decreased risk of heart failure

Djoussé L, Gaziano JM. Breakfast cereals and risk of heart failure in the physicians' health study I. Arch Intern

Med. 2007 Oct 22;167(19):2080-5.

Assessment of Cohort Designs

Advantages

– powerful study design to define incidence

– recall bias not a problem

– good choice when randomization not possible

Disadvantages

– expensive in time, money & number of subjects

– confounding variables

– attrition

10

Cross-sectional Design

Data are collected at one point in time

(“snapshot” of study population)

Includes record reviews & patient

surveys

Risk factors & outcome are determined

simultaneously

Example

Smoking & Cancer

Researcher distributes a survey to a randomly selected group of patients in a primary care clinic’s waiting room

Asks questions about smoking status & history of cancer

Determines if there is a relationship between smoking & cancer

Resident Example: Burnout

Objective was to assess burnout among resident

physicians in the Detroit area

Sent the Maslach Burnout Inventory to 71 family

medicine residents in 8 programs

Found that 51% reported burnout

– Attributed to heavy workload

– Affected patient care

Assessment of Cross-sectional

Designs

Advantages

– relatively inexpensive & quick

– data may be available through existing records

Disadvantages

– cannot determine cause & effect

– ineffective for rare conditions

– can determine prevalence, not incidence

– may not be generalizable

Experimental Designs

Clinical Trial

Educational Intervention

11

Clinical Trial Design

Patients receive an intervention or treatment

(independent variable)

Effect (outcome, dependent variable) is

measured & attributed to intervention or

treatment

Can be single site or multi-center

Can be between-subjects or within-subjects

Parallel & Crossover Designs

Treatment A

Treatment B

Treatment C

Treatment A

Treatment B

Baseline Baseline

Hallmarks of the Randomized

Controlled Trial (RCT)

Gold standard of study design

Random assignment to experimental or

control group

Excluding the intervention, the two

groups are treated equally

Placebo controlled

Hallmarks of the Randomized

Controlled Trial (RCT)- Blinding

Single: only patient is blind to group assignment

Double: both patient & researcher are blind to group assignment

Double dummy:

– used when two drugs cannot be made to look alike (brown pill, red capsule)

– a placebo is made to look like each of the active drugs

– each patient then takes both one of the active drugs & the placebo form of the other drug

Example

Smoking & Cancer

Researcher randomly assigns

nonsmokers to smoking or nonsmoking

groups

10 years later, determines which group

had greater incidence of cancer

12

Resident Example: Jelly Bean Study

Question: Can 18 Brach’s jelly beans substitute for

glucola in gestational diabetes testing?

Enrolled 157 pregnant women

– Took 1 hour glucola test (50 gms. glucola)

– Repeated 1 hour test (jelly beans) 2 wks. later

– Took 3 hour glucose tolerance test 2 wks. later

Found that the jelly beans could substitute for the

glucola and were better tolerated

– Similar sensitivity, specificity, positive/negative predictive

values compared to gold standard test

Assessment of Clinical Trial

Designs

Advantages

– can manipulate dose & treatments

– groups have comparable confounding

variables

– knowledge of temporality is assured

– fewest methodological biases

– permits causal inferences

Assessment of Clinical Trial

Designs

Disadvantages

– ethical issues may limit ability to randomly

assign groups

– risk of sample selection bias

– attrition

– noncompliance

– expensive

Educational Intervention Design

Researcher tests effect of an educational

intervention on the outcome of interest

– Lecture

– Handout

– Video

Pre- post-test design (within subjects)

Education group compared to control group

(between subjects)

Example

Smoking & Cancer

Researcher gives a pre-test to a group of

smokers which measures their knowledge of

the evidence linking smoking to cancer

The smokers watch a video & receive

educational materials

Researcher gives a post-test to see if

knowledge has improved

13

Assessment of Educational

Intervention Designs

Advantages– relatively inexpensive & quick

– can manipulate intervention

– groups have comparable confounding variables

– knowledge of temporality is assured

Disadvantages– risk of sample selection bias

– attrition

– noncompliance

– may not be generalizable to “real world”

– an increase in knowledge may not mean behavior change

Let’s Poll the Audience !

Experimental research

designs include:

Cro

ss-se

ctio

na...

Clin

ical t

rial..

.

Case

reports

Educa

tional in

...

0% 0%0%0%

1. Cross-sectional studies

2. Clinical trials

3. Case reports

4. Educational interventions

Which type of research seeks

to make a cause-and-effect

statement?

Experim

ental

Desc

riptiv

e

Corre

latio

nal

All

of the a

bo...

0% 0%0%0%

1. Experimental

2. Descriptive

3. Correlational

4. All of the above

A journalist reports a study that

shows smoking to be correlated with

poor grades in college. What can

you conclude?

Smokin

g tends .

..

Smokin

g and po...

Incr

eased sm

ok...

Students

with

...

0% 0%0%0%

1. Smoking tends to cause

poor grades in many

students.

2. Smoking and poor grades

tend to be found together.

3. Increased smoking will

cause your grades to fall.

4. Students with poor grades

are anxious, causing them

to smoke more.

What is the best research

design if you want to study

patients with a rare disease?

Longit

udinal s

...

Pro

spect

ive st

...

Experim

ental s

...

Cro

ss-se

ctio

na...

Retro

spect

ive ..

.

0% 0% 0%0%0%

1. Longitudinal study

2. Prospective study

3. Experimental study

4. Cross-sectional study

5. Retrospective study

14

So I have designed my study,

where do I go from here?!

Post- Research Design

Write proposal

Design data tools

Committees

Write a Research Proposal

Include:

– Introduction

– Research question / hypothesis

– Methods

– Plan for statistical analysis

• Don’t forget power analysis & predicted results

– Budget

– References

Design Data Tools

Develop data collection tools– Pilot test

– Keep ease of data entry in mind

Use standardized tools

– Examples: pain scale, depression scale

– Reliability

– Validity

Sample QuestionnaireDate of Birth ___/___/___ Age today: _____

Height: ___ft ___in Weight: ____lbs

Marital status: Single Married Gender: Male Female

Race: Highest Level of Education:

African-American Some High School

Am. Indian/Alaskan Native High School Graduate/GED

Asian Some College

White College Graduate

Hawaiian/Pacific Islander Postgraduate

Other____________

Household Income: Occupation:

<$10,000 Student

$10,000-$24,999 Work outside the home

$25,000-$49,999 Unemployed currently

>$50,000

Committees

GME Research Committee

– funding

– assessment of importance of study

– assessment of appropriate study design

Institutional Review Board (IRB)

– protects the rights & welfare of human subjects

– all studies need review

– need approval before data collection begins

15

Remember...

Allow plenty of time- Plan early

Find a mentor & support staff

Have fun with your project

Communicate your results

– “Correlation does not imply causation”

– Change in policy or clinical practice

– Direction for further studies

Strive for publication

Purpose of Resident Research

The goal is to understand & appreciate the research process– Fellowship

– Clinician

Patients are informed “research consumers”

– Internet

– Media reports of research

Continuing medical education

– Life-long habit of reading & evaluating research studies