The good, the bad, and the ugly The good, the bad, and the ugly of multiple data sourcesof multiple data sources

Barbara J. McMorris, PhDBarbara J. McMorris, PhDONRS Seminar May 19, 2011ONRS Seminar May 19, 2011

UMN School of NursingUMN School of Nursing

Types of Types of TriangulationTriangulation

MethodologicalMethodological

AnalysisAnalysis

InterdisciplinaryInterdisciplinary

InvestigatorInvestigator

DataData

Why is Data Triangulation Why is Data Triangulation Important?Important?

Better research study & results

Reviewers getting savvier; asking for multiple sources of data

More “tools” for your

research tool box

Study 1: Administrative Claims Data + Self Reports

Rhinitis Study GoalsRhinitis Study Goals

Assess burden of allergic and non allergic Assess burden of allergic and non allergic rhinitis rhinitis – What is the cost and utilization impact of What is the cost and utilization impact of

treatment for rhinitis? treatment for rhinitis?

Evaluate providers’ perceptions of rhinitis Evaluate providers’ perceptions of rhinitis burden on patients and severity of burden on patients and severity of symptomssymptoms

Rhinitis Survey Data AnalysisRhinitis Survey Data Analysis

63

63

40

79

75

59

0 50 100

Rhinitis Tximproves QOL

Patients' symptseverity extrmly

impt

AR & NAR nottreated/managed

the same

Allergy Specialist

No AllergySpecialist

% Agree/Strongly Agree

Rhinitis Claims Data AnalysisRhinitis Claims Data Analysis

Count of and Costs of Health Care Services Utilized during 1 Yr Follow-up

Mean (SD)

Mild Patients

(N = 5,670)

ModeratePatients

(N = 22,216)

SeverePatients

(N = 7,217)

Total(N=35,103)

Number of Rhinitis-Related ER Visits

0.03 (0.75) 0.05 (0.89) 0.04 (0.69) 0.05 (0.83)

Number of Rhinitis-Related Office Visits

5.61 (10.1) 8.12( 11.69) 9.39 (12.85) 7.98 (11.75)

Number of Rhinitis-Related Specialist Office Visits

3.08 (8.50) 5.65 (10.43) 7.22 (11.79) 5.56 (10.52)

Total Rhinitis-Related Medical Cost

$358 (725) $514 (919) $556 (952) $497 (900)

Total Rhinitis-Related Pharmacy Costs

$364 (417) $435 (474) $440 (478) $424 (467)

Reflection: “good-bad”Reflection: “good-bad”

Known sampling frame Known sampling frame

““Clean” processed dataClean” processed data

Timing of data collection differentTiming of data collection different– Lag in health care claims being loadedLag in health care claims being loaded– Physical report of severityPhysical report of severity

Low response rateLow response rate

Results used in marketing

http://www.gsk.com/media/pressreleases/http://www.gsk.com/media/pressreleases/2007/2007_10_19_GSK1134.htm2007/2007_10_19_GSK1134.htm

Study 2: Self Reports + Provider Care Flow Sheets

RA Infusion Study GoalsRA Infusion Study Goals

Describe patient-reported outcomes due to Describe patient-reported outcomes due to bDMARD infusions: bDMARD infusions: – satisfactionsatisfaction– lost productivity lost productivity – pain/discomfortpain/discomfort

Quantify the amount of time providers spend for Quantify the amount of time providers spend for initiating infusions (3 types)initiating infusions (3 types)

Describe provider satisfaction with infusionsDescribe provider satisfaction with infusions

Data SourcesData Sources

Health care claims databaseHealth care claims database

Provider surveyProvider survey

Patient Survey Patient Survey

Case Report Form Case Report Form

Infusion Care Flow SheetInfusion Care Flow Sheet

RA Infusion ResultsRA Infusion Results

Patient and Provider time associated with infusions

Means & %’s

Infusion 1(N = 102)

Infusion 2(N = 74)

Infusion 3(N = 29)

Total patient time away from work/home (min.)

172.6 267.9 446.9

% patients able to return to work 45% 17% 29%

% had friend/family member miss work

31% 24% 43%

Total provider labor time (min.) 86.7 128.2 237.9

Reflection: “good”Reflection: “good”

Good example of triangulation Good example of triangulation

Comprehensive picture of “burden” Comprehensive picture of “burden”

Small sample sizeSmall sample size

Results used in marketing

http://www.orencia.com/rheumatoid-arthritihttp://www.orencia.com/rheumatoid-arthritis-medication/orencia-infusion.aspxs-medication/orencia-infusion.aspx

Study 3: Self Reports + Process Data + Official Records

Broadway HS Study GoalsBroadway HS Study Goals

Provide a descriptive profile of pregnant Provide a descriptive profile of pregnant and parenting teen girls and parenting teen girls

Compare outcomes for Broadway and Compare outcomes for Broadway and comparison girls:comparison girls:

progress toward graduationprogress toward graduationdelaying a repeat birthdelaying a repeat birth keeping children fully immunizedkeeping children fully immunized

Data SourcesData Sources

Girls’ self reports on pre/post surveysGirls’ self reports on pre/post surveys

Case manager reports of timeCase manager reports of time

Official RecordsOfficial Records

–Program Program

–School districtSchool district

–CountyCounty

Evaluation ResultsEvaluation Results

%

County DataCounty Data

MFIP and Other AssistanceBroadwayStudents(N = 158)

ComparisonGroup

(N = 97)

Ave counted months of MFIP 9.7 10.76

Ave used months of MFIP 25.8 26.9

2 or more children 27% 27%

Used emergency assistance 47% 41%

Case closed due to sanction limit 9% 13%

Reflection: kinda “ugly”Reflection: kinda “ugly”

Using survey created/revised by othersUsing survey created/revised by others

Linking datasets – what is the link?Linking datasets – what is the link?

Consent to access official records?Consent to access official records?

What were the data collected for? What were the data collected for? – Who entered the data?Who entered the data?

Budget enough time & money for both Budget enough time & money for both data management & analysis?data management & analysis?

Final Cautions about Final Cautions about Data TriangulationData Triangulation

Need well-defined goals (no “fishing”)Need well-defined goals (no “fishing”)

Costly both in time and $$$Costly both in time and $$$

May not solve bias/error issues May not solve bias/error issues

Replication is difficultReplication is difficult

Know your “official” records sources and Know your “official” records sources and their limitationstheir limitations

Do you have “permission” to link data Do you have “permission” to link data sets? sets?

““triangulation state of mind”*triangulation state of mind”*

““If you If you self consciouslyself consciously set out to collect set out to collect and double-check findings using multiple and double-check findings using multiple sources and modes of evidence, the sources and modes of evidence, the verification process will largely be built into verification process will largely be built into the data-gathering process, and little more the data-gathering process, and little more need be done than to report on one’s need be done than to report on one’s procedures.”procedures.”

*Miles & Huberman (1984: *Miles & Huberman (1984: p.235)p.235)

Thank you!Thank you!

Useful citations:Begley, CM. (1996). Using triangulation in nursing research. Journal of Advanced Nursing 24:122-128.

Thurmond, VA. (2001). The Point of Triangulation. Journal of Nursing Scholarship 22(3): 253-258.