1

People and People and Measurements—The Measurements—The

Nuts & Bolts of Nuts & Bolts of ResearchResearch

Optimizing Subjects & Variables Optimizing Subjects & Variables and Introduction to Kaiser and Introduction to Kaiser

Division of ResearchDivision of ResearchAlan S. Go, M.D.Alan S. Go, M.D.

Division of Research, Kaiser Permanente of Northern CaliforniaDivision of Research, Kaiser Permanente of Northern California

Depts. of Epidemiology, Biostatistics, and Medicine, UCSFDepts. of Epidemiology, Biostatistics, and Medicine, UCSF

August 11, 2009August 11, 2009

2

Today’s ObjectivesToday’s Objectives

Brief Introduction to Research in Kaiser Brief Introduction to Research in Kaiser Permanente of Northern CaliforniaPermanente of Northern California Gain a better understanding of the Kaiser Gain a better understanding of the Kaiser

Division of Research, population, and databasesDivision of Research, population, and databases Selecting the PeopleSelecting the People

Develop systematic approach to optimize Develop systematic approach to optimize subject selectionsubject selection

Choosing the MeasurementsChoosing the Measurements Understand the implications of exposure & Understand the implications of exposure &

outcome variable/measurement choicesoutcome variable/measurement choices Application to a Real Research Question: Application to a Real Research Question:

The ATRIA Study The ATRIA Study

3

AHA Cardiovascular Outcomes AHA Cardiovascular Outcomes Research Center Fellowship Research Center Fellowship

OpportunityOpportunity

Two-year fellowship sponsored through the Two-year fellowship sponsored through the AHA Pharamceutical Roundtable focused on AHA Pharamceutical Roundtable focused on training the next generation of outcomes training the next generation of outcomes researchersresearchers

Kaiser Permanente-Stanford University AHA Kaiser Permanente-Stanford University AHA CV Outcomes Research CenterCV Outcomes Research Center

Contact: Alan S. Go, MD (alan.s.go@kp.org)Contact: Alan S. Go, MD (alan.s.go@kp.org)

http://www.americanheart.org/presenter.jhtml?identifier=9713

4

Kaiser Division of Research & Kaiser Division of Research & PopulationPopulation

Regional research division focused on Regional research division focused on epidemiology and health care epidemiology and health care effectiveness researcheffectiveness research

Kaiser Northern California population:Kaiser Northern California population: >3.2 million members (>2 million adults); >3.2 million members (>2 million adults);

~52% women~52% women

*2005 Kaiser Permanente Members Health Survey (N. Gordon)

Race/EthncityRace/Ethncity Overall, %Overall, % Men, %Men, % Women, %Women, %

HispanicHispanic 12.312.3 12.612.6 12.012.0

Native Amer.Native Amer. 1.71.7 1.51.5 1.91.9

AsianAsian 17.017.0 17.817.8 16.316.3

Hawaiian/PIHawaiian/PI 0.60.6 0.60.6 0.60.6

Black/AABlack/AA 6.36.3 5.25.2 7.37.3

White/Eur.White/Eur. 61.861.8 61.561.5 62.162.1

OtherOther 12.612.6 13.413.4 11.911.9

5

Kaiser Administrative Kaiser Administrative DatabasesDatabases

Demographic & membership Demographic & membership characteristicscharacteristics Unique lifetime medical record number to Unique lifetime medical record number to

track information across all major databasestrack information across all major databases Age; gender; and race/ethnicityAge; gender; and race/ethnicity Membership, drug benefit, and insurance Membership, drug benefit, and insurance

statusstatus Physician identifiers & characteristicsPhysician identifiers & characteristics Clinic and medical center characteristicsClinic and medical center characteristics

6

Kaiser Clinical Kaiser Clinical Databases/RegistriesDatabases/Registries

Inpatient diagnoses/proceduresInpatient diagnoses/procedures Ambulatory diagnoses/proceduresAmbulatory diagnoses/procedures Outpatient pharmacy prescriptionsOutpatient pharmacy prescriptions Inpatient and outpatient laboratory testsInpatient and outpatient laboratory tests Pathology findingsPathology findings Selected Kaiser disease registries:Selected Kaiser disease registries:

Chronic kidney disease, heart failure, Chronic kidney disease, heart failure, diabetes mellitus, GDM, cancer, HIV/AIDS, diabetes mellitus, GDM, cancer, HIV/AIDS, PCOS, etc.PCOS, etc.

At end of 2008, regional EMR based on At end of 2008, regional EMR based on EpicEpic

7

After deciding a great research After deciding a great research question, figuring out question, figuring out WHOWHO you you want to study and want to study and WHATWHAT you want you want to measure are the next key steps…to measure are the next key steps…

Subjects and Variables: The Subjects and Variables: The Nuts and Bolts of the Nuts and Bolts of the

Research QuestionResearch Question

8

Selecting Your Selecting Your Subjects…Subjects…

9

Optimizing Subject Optimizing Subject Selection: Selection: A Delicate A Delicate

Balancing ActBalancing Act

FeasibilityFeasibility

AccessibilityAccessibility

CostCost

Time/EfficiencyTime/Efficiency

GeneralizabilityGeneralizability

AccuracyAccuracy

Diversity Diversity

Adequate SizeAdequate Size

10

Subject Selection: The Subject Selection: The Nitty GrittyNitty Gritty

ExplicitlyExplicitly Define Inclusion Criteria Define Inclusion Criteria Demographic features (e.g., age, gender, Demographic features (e.g., age, gender,

race)race) Clinical criteriaClinical criteria Geographic/administrative characteristics Geographic/administrative characteristics Sampling time frameSampling time frame

ExplicitlyExplicitly Define Exclusion Criteria Define Exclusion Criteria Minimum number needed to be feasible Minimum number needed to be feasible

with acceptable generalizability to target with acceptable generalizability to target populationpopulation

11

Subject Sampling Subject Sampling Techniques:Techniques:

How to Get the “People?” How to Get the “People?” (1)(1)

Convenience SamplesConvenience Samples True convenience True convenience (e.g., 25 clinic patients I (e.g., 25 clinic patients I

know well)know well) Consecutive Consecutive (e.g., next 100 patients (e.g., next 100 patients

undergoing liposuction)undergoing liposuction)

Probability SamplesProbability Samples Simple random Simple random (e.g., using random number (e.g., using random number

table)table) Stratified or weighted random Stratified or weighted random (e.g., (e.g.,

by gender)by gender) Cluster Cluster (e.g., by clinic or neighborhood)(e.g., by clinic or neighborhood)

12

Subject Recruitment:Subject Recruitment:How to Get the “People?” How to Get the “People?”

(2)(2) Successful Recruitment Generally Means…Successful Recruitment Generally Means…

response, generalizable sample, adequate response, generalizable sample, adequate size, completed on time (or early!)size, completed on time (or early!)

For database only studies—For database only studies—Not usually a big Not usually a big problemproblem

For hands-on studies For hands-on studies ((e.g., surveys, cohorts, e.g., surveys, cohorts, trialstrials)) Expect that it will be harder than you think!Expect that it will be harder than you think! Use reasonable inclusion/exclusion criteriaUse reasonable inclusion/exclusion criteria Acceptable subject burden/potential benefitsAcceptable subject burden/potential benefits Efforts to minimize subject non-responseEfforts to minimize subject non-response

13

Applying These Principles Applying These Principles to Answer My Research to Answer My Research

Question:Question:

What is the association between use of What is the association between use of the blood thinner, warfarin, and the the blood thinner, warfarin, and the risk of ischemic stroke & bleeding in risk of ischemic stroke & bleeding in patients with atrial fibrillation treated patients with atrial fibrillation treated

in a usual clinical care setting?in a usual clinical care setting?

14

Warfarin for Warfarin for Stroke Prevention Stroke Prevention

in AFin AF Atrial fibrillation (AF) is most common Atrial fibrillation (AF) is most common

clinically significant arrhythmiaclinically significant arrhythmia11 and ↑ and ↑ stroke risk ~5-foldstroke risk ~5-fold2,32,3

RCTs in selected nonvalvular AF (NVAF) RCTs in selected nonvalvular AF (NVAF) patients showed warfarin patients showed warfarin stroke by 68% stroke by 68% but but bleeding bleeding33

Aspirin much less effective (RRR ~20%)Aspirin much less effective (RRR ~20%)

Warfarin recommended for most NVAF Warfarin recommended for most NVAF patients, but patients, but concerns about whether trial concerns about whether trial results can be applied to the “real worldresults can be applied to the “real world””

1 Go AS et al. JAMA. 2001;285:2370-75.2 Wolf PA et al. Stroke 1991;22:983-88.3 Atrial Fibrillation Investigators. Arch Intern Med 1994;154:1449-57

15

AAnnTTicoagulation and icoagulation and RRisk Factors isk Factors IIn n AAtrial trial

Fibrillation Fibrillation The ATRIA StudyThe ATRIA Study

16

Atrial Fibrillation

Warfarin

TE/Bleeds

ATRIA StudyATRIA Study

17

ATRIA Study: SubjectsATRIA Study: Subjects

All adults with nonvalvular AF

in U.S.

Ambulatory adults with diagnosed

nonvalvular AF in Kaiser No.Calif.

18

ATRIA Study: Inclusion ATRIA Study: Inclusion CriteriaCriteria

Sampling Frame GoalSampling Frame Goal: Identify all : Identify all ambulatory adults with diagnosed chronic ambulatory adults with diagnosed chronic nonvalvular AFnonvalvular AF

Inclusion criteriaInclusion criteria:: DemographyDemography: : >>18 years, M/F, all 18 years, M/F, all

race/ethnicitiesrace/ethnicities Clinical CriteriaClinical Criteria: Diagnosed AF from outpatient : Diagnosed AF from outpatient

& ECG databases (& ECG databases (1 outpatient AF dx + 1 outpatient AF dx + 1 1 ECG with AF ECG with AF oror 2 outpatient AF dx only)2 outpatient AF dx only)

Geography/AdministrativeGeography/Administrative: Received care in : Received care in Kaiser Permanente of Northern CaliforniaKaiser Permanente of Northern California

Time PeriodTime Period: AF diagnosis found in 1996-1997: AF diagnosis found in 1996-1997

19

ATRIA Study: Exclusion ATRIA Study: Exclusion CriteriaCriteria

Exclusion criteria:Exclusion criteria: No health plan membershipNo health plan membership Transient perioperative atrial fibrillationTransient perioperative atrial fibrillation Concomitant hyperthyroidismConcomitant hyperthyroidism Diagnosed valvular heart disease Diagnosed valvular heart disease No outpatient care during 12 months No outpatient care during 12 months

after index dateafter index date No drug benefit surrounding index dateNo drug benefit surrounding index date

20

ATRIA Cohort AssemblyATRIA Cohort Assembly

Suspected AFSuspected AF

13,559 Ambulatory Adults with Diagnosed Chronic 13,559 Ambulatory Adults with Diagnosed Chronic Nonvalvular AFNonvalvular AF** and Known Warfarin Status and Known Warfarin Status

No membership after AF dxNo membership after AF dx

Valvular heart diseaseValvular heart disease

Transient AF after cardiac surgeryTransient AF after cardiac surgery

No outpatient care after index date No outpatient care after index date or drug benefitor drug benefit

Concomitant hyperthyroidismConcomitant hyperthyroidism

<18 years old<18 years old

1 outpatient AF dx only1 outpatient AF dx only Identified by ECG onlyIdentified by ECG only

1 Outpatient AF Dx1 Outpatient AF DxN=15,570N=15,570

1 ECG with AF1 ECG with AFN=13,052N=13,052

*Validation studies suggest ≥87% of cohort w/ECG-confirmed AF

21

ATRIA: Baseline ATRIA: Baseline CharacteristicsCharacteristics

Mean age ± SDMean age ± SD 71 ± 12 71 ± 12

yryr

WomenWomen 43 %43 %

Previous strokePrevious stroke 9 %9 %

Previous heart failurePrevious heart failure 29 %29 %

HypertensionHypertension 50 %50 %

Diabetes mellitusDiabetes mellitus 18 %18 %

Previous coronary diseasePrevious coronary disease28 %28 %The ATRIA cohort is older, has more women, and greater

comorbid burden than RCT populationslikely generalizable to AF patients in typical practice

22

Making the Making the Measurements:Measurements:

Implications for Exposure Implications for Exposure & Outcome& Outcome Variable Variable

ChoicesChoices

23

““The most elegant The most elegant design of a clinical design of a clinical

study will study will notnot overcome the overcome the

damage caused by damage caused by unreliable or unreliable or

imprecise imprecise measurement.”measurement.”J.L. Fleiss (1986)J.L. Fleiss (1986)

Fleiss, JL. The design and analysis of clinical experiments. pp. 1-5. 1986. John Wiley and Sons, New York.

24

““Accuracy must be balanced Accuracy must be balanced against practical against practical considerations, and that considerations, and that method chosen which will method chosen which will provide the provide the maximal accuracy maximal accuracy within the bounds of the within the bounds of the investigator’s resources and investigator’s resources and other practical limitationsother practical limitations.”.”J.H. Abramson (1984)J.H. Abramson (1984)

Abramson, JH. Survey methods in community medicine (3rd Ed.), p. 121. 1984. Churchill Livingstone, Edinburgh.

25

PredictorPredictor** OutcomeOutcome

Confounding Confounding VariablesVariables**

Effect ModifiersEffect Modifiers**

Planning the Planning the MeasurementsMeasurements

Relationship of Key ExposuresRelationship of Key Exposures

*Often generally categorized as “exposures”

26

Additional “Exposure” Additional “Exposure” ConsiderationsConsiderations

Dose Issues Dose Issues Cumulative Cumulative

exposureexposure Exposure rateExposure rate

Time IssuesTime Issues Start of exposureStart of exposure When it endedWhen it ended Exposure Exposure

distributiondistribution

Alcohol UseAlcohol Use Total # of drinksTotal # of drinks # Drinks/day# Drinks/day

Date of first Anchor Date of first Anchor SteamSteam

Date of last margaritaDate of last margarita Daily vs. binge Daily vs. binge

drinkingdrinking

27

General Variable TypesGeneral Variable Types

ContinuousContinuous Quantitative Quantitative

intervals with intervals with typical rankingtypical ranking

Examples:Examples: Cholesterol levelCholesterol level Number of drinksNumber of drinks Day supply of drugDay supply of drug Waist sizeWaist size TimeTime

CategoricalCategorical Dichotomous (yes/no) Dichotomous (yes/no)

(e.g., death, diabetes)(e.g., death, diabetes)

Nominal (no order) Nominal (no order) (e.g., ethnicity, (e.g., ethnicity, occupation)occupation)

Ordinal (ordered Ordinal (ordered rank) rank) (e.g., NYHA HF (e.g., NYHA HF Class I-IV)Class I-IV)

28

Typical Data SourcesTypical Data Sources

Survey/Survey/questionnairequestionnaire

Interviews Interviews DiariesDiaries Direct Direct

observationobservation Environmental Environmental

measurementsmeasurements

Databases/registriesDatabases/registries Medical recordsMedical records Physiologic Physiologic

measuresmeasures Biomarkers Biomarkers (e.g., DNA, (e.g., DNA,

sera)sera)

Imaging testsImaging tests PathologyPathology

Goal: choose the source that gives Goal: choose the source that gives data closest to the “gold standard” data closest to the “gold standard”

while being feasible to collectwhile being feasible to collect

29

General Measurement General Measurement Goals…Goals…

You get the same result when You get the same result when measured repeatedly—within the same measured repeatedly—within the same subject, between subjects, and over subject, between subjects, and over timetimemaximize PRECISIONmaximize PRECISION

It represents what it’s really supposed It represents what it’s really supposed to beto be maximize ACCURACY/VALIDITYmaximize ACCURACY/VALIDITY + high sensitivity & specificity+ high sensitivity & specificity

30

The Measurement The Measurement SpectrumSpectrum

After deciding the exposure/outcome After deciding the exposure/outcome of interest, “measurement” includes:of interest, “measurement” includes: Written instructions for applying the Written instructions for applying the

method for measuring the variablemethod for measuring the variable Doing the measurement method itselfDoing the measurement method itself Spelling out collected data for analysisSpelling out collected data for analysis Implementing quality control Implementing quality control

procedures throughout procedures throughout (i.e., making sure you (i.e., making sure you get what you meant to get)get what you meant to get)

31

Improving Precision and Improving Precision and Accuracy of Variables & Accuracy of Variables &

Reducing BiasReducing Bias Standardize methodsStandardize methods Pretest, pretest, Pretest, pretest,

pretestpretest Refine/automate Refine/automate

instrumentinstrument Train & evaluate Train & evaluate

staffstaff Timely editing, Timely editing,

coding & correcting coding & correcting of formsof forms

Multiple Multiple measurementsmeasurements

Use or validate against Use or validate against “gold standard” “gold standard”

Less obtrusive Less obtrusive measuresmeasures

For outcomes, blinding For outcomes, blinding to exposure statusto exposure status

Institute quality Institute quality control measures control measures during data collection, during data collection, processing, and processing, and analysisanalysis

32

Applying These Principles Applying These Principles to Answer My Research to Answer My Research

Question:Question:

What is the association between use of What is the association between use of warfarin and the risk of ischemic warfarin and the risk of ischemic stroke & bleeding in patients with stroke & bleeding in patients with atrial fibrillation treated in usual atrial fibrillation treated in usual

clinical care?clinical care?

33

ATRIA Study: ATRIA Study: MeasurementsMeasurements

All adults with NVAF

in U.S.

Ambulatory adults with diagnosed NVAF in KPNC

Warfarin

TE/Bleeds

-Longitudinal warfarin use-Hospitalized ischemic stroke or other systemic embolism

-Hospitalized bleeding event

34

PredictorPredictor OutcomeOutcome

Confounding Confounding VariablesVariables

Effect ModifiersEffect Modifiers

Planning ATRIA Planning ATRIA MeasurementsMeasurements

-Demographic features-Stroke risk factors-Warfarin contraindications

(?)

35

Exposure Example: Exposure Example: WarfarinWarfarin

Warfarin use (main predictor)Warfarin use (main predictor) Baseline “warfarin useBaseline “warfarin use”—At least one of the ”—At least one of the

following within 3 months of index AF dx following within 3 months of index AF dx date:date:

1 filled R1 filled Rxx for warfarin in pharmacy database for warfarin in pharmacy database ““Coumadin therapy” in outpatient db (ICD-9 Coumadin therapy” in outpatient db (ICD-9

V58.61)V58.61) >1 outpatient INR measurement in lab database>1 outpatient INR measurement in lab database

Longitudinal “warfarin useLongitudinal “warfarin use”—time-dependent ”—time-dependent exposure based on warfarin Rexposure based on warfarin Rxx and INR tests and INR tests

Validation study of method for baseline Validation study of method for baseline useuse Chart review of random sample of “users” & Chart review of random sample of “users” &

“non-users”: 96% raw agreement (“non-users”: 96% raw agreement (=0.92)=0.92)

36

Outcome Example: Outcome Example: Ischemic StrokeIschemic Stroke

Ischemic stroke (main outcome)Ischemic stroke (main outcome) Identification methodIdentification method: searched databases : searched databases

Primary discharge ICD-9 codes for possible acute Primary discharge ICD-9 codes for possible acute ischemic stroke (e.g., 433.x, 434.x, 436.0) found ischemic stroke (e.g., 433.x, 434.x, 436.0) found in hospital discharge and billing claims databasesin hospital discharge and billing claims databases

Validation method:Validation method: reviewed medical records reviewed medical records Obtain Kaiser/non-Kaiser hospital recordsObtain Kaiser/non-Kaiser hospital records 3-physician review (+/- Neurology consultant)3-physician review (+/- Neurology consultant) Unable to blind warfarin status at time of eventUnable to blind warfarin status at time of event ““Valid stroke” required documented acute Valid stroke” required documented acute

neurological deficit lasting >24 hours not due to neurological deficit lasting >24 hours not due to other etiologyother etiology

37

What Did We Find?What Did We Find?

Among 13,559 adults with nonvalvular Among 13,559 adults with nonvalvular atrial fibrillation, longitudinal use of atrial fibrillation, longitudinal use of warfarin therapy was associated with…warfarin therapy was associated with… 49% adjusted decrease in risk of ischemic 49% adjusted decrease in risk of ischemic

strokestroke Modest absolute increase in risk of Modest absolute increase in risk of

intracranial hemorrhage (0.51 vs. 0.33 per intracranial hemorrhage (0.51 vs. 0.33 per 100 person-years)100 person-years)

Net benefit of warfarin greatest among Net benefit of warfarin greatest among patients at the highest risk for ischemic patients at the highest risk for ischemic strokestrokeGo AS, Hylek EM, Chang Y, et al. Anticoagulation therapy for stroke

prevention in atrial fibrillation: how well do randomized trials translate into clinical practice? JAMA 2003; 290:2685-92.

38

Questions?Questions?

Alan S. Go, M.D.Alan S. Go, M.D.Director, Comprehensive Clinical Director, Comprehensive Clinical

Research UnitResearch Unit

Assistant Director for Clinical ResearchAssistant Director for Clinical Research

Division of ResearchDivision of Research

Kaiser Permanente of Northern CaliforniaKaiser Permanente of Northern California

2000 Broadway St, Oakland CA 946122000 Broadway St, Oakland CA 94612

Tel: 510-891-3553/Email: Tel: 510-891-3553/Email: Alan.S.Go@kp.orgAlan.S.Go@kp.org