"Virtual" Clinical Trials:Case Control Experiments Utilizing a Health Services Research Workstation

Mark G. Weiner, M.D., Alan L. Hillman, M.D., M.B.A.Division of General Internal Medicine

University of Pennsylvania School ofMedicine

We created an interface to a growing repository ofclinical and administrative information to facilitatethe design and execution of case-controlexperiments. The system enables knowledgeableusers to generate and test hypotheses regardingassociations among diseases and outcomes. Theintuitive interface allows the user to specify criteriafor selecting cases and defining putative risks. Therepository contains comprehensive administrativeand selected clinical information on all ambulatoryand emergency department visits as well as hospitaladmissions since 1994. We tested the workstation'sability to determine relationships between outpatientdiagnoses including hypertension, osteoarthritis andhypercholesterolemia with the occurrence ofadmissions for stroke and myocardial infarction andachieved results consistent with published studies.Successful implementation of this Health ServicesResearch Workstation will allow "virtual" clinicaltrials to validate the results of formal clinical trialson a local population and may provide meaningfulanalyses of data when formal clinical trials are notfeasible.

INTRODUCTION

The increasing focus on quality of care in medicinerequires new systems to recognize patients at riskfor serious illness and to promote interventions thatdemonstrably improve health status. Traditionally,formal experimental methods have been required toprove an association between a disease and itsputative risk factors. Once a risk is discovered, thenfurther clinical trials are required to determinewhether management of the risk factor can reducethe likelihood of disease occurrence. Unfortunately,concerns about the generalizability of the resultsoften hampers their acceptance by practicingclinicians. Such concerns are often well-foundedsince, in our experience, a physician's patientpopulation often has different characteristics thanthe formal study population and effectiveness oftherapy in the real world can often differ fromefficacy achieved within a tightly-monitored,controlled study.

In this paper, we describe the development andtesting of a Health Services Research Workstation

which is a specialized interface to an integratedclinical practice database. It can be used to test therelationships among diseases and perform "virtual"clinical trials to examine the impact of differenttreatment modalities on the course of a disease.Since the tool utilizes population-based informationfrom patients within a local community, results ofthese experiments should be directly applicable tothose populations and acceptable to physicianspracticing in the region.

BACKGROUND

A growing body of literature demonstrates theclinical and research value of relational databasetechnology that integrates medically relatedinformation from disparate legacy systems.Electronic access to comprehensive, longitudinalinformation on patients has improved the process ofpatient care as measured by efficiency,' reduction ofadverse drug events,2 and more appropriate medicaldecision-making.3

In addition to these direct clinical applications, theuse of databases to support clinical research hasbeen debated and tested for years. Notablesuccesses include the Regenstrief Medical RecordsSystem which has demonstrated successfully avariety of risk factors for development of renalinsufficiency in hypertensives.4 The HealthEvaluation through Logical Processing (HELP)system at the Latter Day Saints (LDS) Hospital wasable to determine risk factors for nosocomialinfections.5 Despite these successes, criticism ofthis methodology focuses on the observational,largely administrative nature of the information.Within these databases, inaccurate data can berecorded and subtle differences in patientcharacteristics can be overlooked. However, whenthe administrative data are coupled with clinical datain the form of practice databases, the impact of someof these errors can be ameliorated.6 For example,the Duke Cardiovascular Disease Databasedemonstrated the power of such integration of databy generating prediction models of survival aftercoronary artery bypass surgery that agreed with theresults of randomized controlled trials.7 Theseexamples demonstrate that, while not a substitute for

1091-8280/98/$5.00 C 1998 AMIA, Inc. 300

clinical trials, population research using practicedatabases can generate hypotheses warrantingfurther study and can test the findings of clinicaltrials on local populations. Importantly, clinicalresearch using practice databases can providevaluable analyses when a trial is unethical, orinfeasible. Models are being developed todetermine the characteristics that determine ifdatabase research or randomized clinical trials aremore appropriate."

Whereas the use of databases for clinical practiceand research9 has met with great success, only a fewnotable efforts have attempted to automate themethodology and structure interfaces to facilitateaccess to data necessary for the experiment designand revision process.'0 Therefore, an additionalgoal of our work was to incorporate analysis toolswith an experiment design interface to help interpretpopulation data. Our system enables rapid designand execution of case-control studies. Wehypothesize that the interface, and the datasupporting it, can be used to identify knownassociations and reject implausible relationships.

METHODS

A. Database DevelopmentWe developed a comprehensive, integrated view ofthe diverse population cared for by the University ofPennsylvania Health System. This group of 500,000patients accounts for over 1 million ambulatoryvisits/year and 33,000 hospital admissions/year toour main tertiary care hospital. The data elementsaccessible at this time include basic demographicsand information on all office visits and inpatientadmissions including diagnoses assigned,procedures performed and charges assessed sinceJanuary 1994. This information had existedpreviously only within two distinct databases, IDXfor outpatients and PHS for inpatients.Additionally, we have integrated laboratory resultsfrom Cerner and the clinical findings of cardiaccatheterization, nuclear cardiology andechocardiography procedures that are currentlystored within distinct custom-built databases.Finally, we incorporated pharmacy data on a subsetof the Health System population. All the data existcentrally on a DEC AlphaServer, on a UNIXplatform running Oracle 7.3. The data areaccessible through both direct ODBC connections toPentium workstations running Microsoft Access andusing web browsers which interact with the databasevia a Pentium II, 266 MHz machine with MicrosoftWindows NT running Internet Information Server4.0 and active server pages. Security is achievedthrough password and firewall protection of the

ODBC connection and through password and SecureSocket Layer encryption of the Web interface.

B. Interface DevelopmentWe developed the interface using Microsoft Accessand subsequently made it available via webbrowsers using Active Server Pages constructedusing Microsoft's Visual InterDev. In addition tothe traditional interface which enables the selectionof patients with specified demographic and clinicalcriteria, this interface is structured to facilitate case-control experiments that allow the researcher to testfor an actual association between disease occurrenceand reputed or hypothesized risk factors. In theseexperiments, the researcher calculates an odds ratiowhich quantifies the relative occurrence of aputative risk in "case" patients known to have adisease compared with "control" patients who do nothave the disease. Figure 1 shows a two-by-two tablewhich illustrates all the possible combinations of thepresence or absence of a disease and the presence orabsence of a risk.

Case ControlRisk Present [ Xa | bI ]Risk Absent c d

Figure 1: A sample 2 x 2 table

The degree of association between a risk and adisease is related to the magnitude of the odds ratiowhich is calculated as (a-d)/(b-c). Values greaterthan 1 support a positive association between therisk and the disease; values less than 1 support anegative association with the disease, i.e., thepresence of the putative risk protects the individualfrom the disease. Statistical significance of theassociation is determined by deriving the 95%confidence intervals around the odds ratio. Theassociation is defined as significant (p<0.05) if therange of values between the confidence intervalsdoes not include 1. The formula used to calculatethe confidence interval (CI) is as follows:

CI = (a d/b c)exp(±z (1/a + 1/b + 1/c + I/d)o 5)where z=l.96 to produce a 95% confidenceinterval.s

The user conducts the case control experimentdesign in five simple steps which are illustrated inFigure 2. Cases and putative risks, respectively, arefirst designated by entering the appropriateInternational Classification of Diseases-9-ClinicalModification (ICD-9-CM) diagnostic codes in thetext boxes. The "*" is a wildcard character torepresent multiple related codes. The user can varythe number of controls in a third text box. When the

301

information is entered, the user executes theexperiment by pressing a button to locate all casesand another to locate controls chosen at randomfrom a similar population. Retention ofthe case andcontrol finding as two separate steps stems from thefrequent need to select the number of desiredcontrols AFTER the number of cases has beendetermined. Typically, the number of controls is atleast as great as the number of cases found.

Results are displayed as an odds ratio withconfidence interval, as well as the demographiccharacteristics of both cases and controls. Analysisof the impact of different demographics should bethe subject of further investigation.

C. Selection of cases and putative risksIdeal case diseases for study include, for example,hospital admissions for stroke (ICD-9=434.*) andmyocardial infarction (ICD-9=410.*) because oftheir relatively high prevalence in the populationand the existence of well-defined criteria fordiagnosis. We drew cases from the data derivedfrom the inpatient database. To test the ability ofthe system to reach conclusions that requireintegration of information, we chose risks fromwithin the distinct ambulatory care database.Ambulatory diagnoses of hypertension (ICD-9 =

401.9), osteoarthritis (ICD-9 = 715.*) andhypercholesterolemia (ICD-9 = 272.0) were chosenas putative risks of stroke and myocardial infarction

Cam

645

because of their relative frequency in the populationand because they illustrate a spectrum of possiblerisk associations with the "case" diagnoses. Forinstance, the relationship between hypertension andboth myocardial infarction and stroke is very welldefined,'2 and we expected to reaffirm thatassociation. Conversely, there is no known linkbetween the presence of arthritis and either stroke or

myocardial infarction, so we did not expect togenerate a significant odds ratio. Finally, therelationship between elevated cholesterol andmyocardial infarction has been well establishedsince the original Framingham study,'3 although alink between an elevated cholesterol and stroke hasnot been demonstrated.14 We expected the oddsratios generated by our system to be consistent withthese predictions from peer-reviewed, publishedliterature.

The choice of the population from which cases andcontrols are drawn requires special consideration.Many of our health system's ambulatory patientsmay be hospitalized at locations for which we haveno data. Similarly, many patients admitted to theHospital of the University of Pennsylvania are seenby primary care physicians outside our healthsystem. Therefore to enable appropriatestratification of patients, the interface queries a datamart consisting of the subset of Health Systempatients for whom both inpatient and ambulatorydata exist.

Control

IOdds Rad 1.46393557

Cl: 1.17593 to 1.8224694

650

MeanAg 64.2573531085

Age Range 30.98

Germbr: Male 383 Feale 262

Race: Cauasion 403Back* 185

598.346153845154

30.99

Male 268 Female 392

Caucation 314 Black 308

302

Stopi DanStao Stop2: DeneCmbRks Step3: L ateC_ Stp4: Set sz d Stop5 Seldcr*olsCagDDefinko conlbaCaseIhilion Risk Dehb"io ~(Hospi D/C Ox) l i Dx)1T .ZZ IL401.......__ W TTTT..WI w~ I wnC lI

Figure 2: The Case-Control design and results reporting form

339 290Ms 370

Table 1: Odds ratios associated with putative risks and outcomes (OR = odds ratio; *p<O.05)PUTATIVE RISK OUTCOMIE OR CONFIDENCE INTERVALAmbulatory Diagnosis of Admission for Myocardial 1.64 1.24-2.13*Hypertension InfarctionAmbulatory Diagnosis of Admission for Stroke 2.73 2.04-3.65*HypertensionAmbulatory Diagnosis of Arthritis Admission for Myocardial 1.20 0.87 - 1.65

InfarctionAmbulatory Diagnosis ofArthritis Admission for Stroke 1.41 0.95-2.04Ambulatory Diagnosis of Admission for Myocardial 3.47 2.69-4.48*Hypercholesterolemia InfarctionAmbulatory Diagnosis of Admission for stroke 0.93 0.62-1.38HypercholesterolemiaIaI I

RESULTS

Figure 2 displays the user interface screen, includingthe design and results of one case-controlexperiment to examine the association betweenhypertension and myocardial infarction. Table 1summarizes the results of the other experiments. Asanticipated, the odds ratios generated by evaluationof hypertension as a risk for both myocardialinfarction and stroke showed a statisticallysignificant positive association. However, nosignificant association was found betweenosteoarthritis and these significant medicaloutcomes. We found a strong association betweenan ambulatory diagnosis of hypercholesterolemiaand an admission for myocardial infarction. Finally,as expected, no association was demonstratedbetween elevated cholesterol and stroke. All of ourhypotheses were confirmed.

DISCUSSION

This work demonstrates the face validity of a datamodel and interface of a Health Services ResearchWorkstation that can be used to design and carry out"virtual" clinical trials. The statistical associationsdiscovered by the workstation are consistent withthose of published clinical trials. Slight differencesin the actual odds ratios are attributable to the natureof the study population. Since the populationanalyzed, by definition, required at least oneinpatient stay, the patients, both cases and controls,are probably less healthy than the generalpopulation. However, the impact of this bias wouldlikely blunt any apparent relationship between adisease and its risk; a sicker population may be morelikely to have the risk than the general population,even ifthey did not have the "case" disease.

We recognize, and continue to address, statisticalissues raised by the results shown. For example, the

difference in demographic characteristics of thecases and controls may confound the results.Logistic regression can determine the impact ofthese differences on the significance of the oddsratios. We anticipate that improved sources of dataand incorporation of additional analytical methodswill address such issues and we will incorporatethese into the workstation in the near future.

The risk factors used in this project weredichotomous variables; the risk factor was eitherpresent or absent. Future research should addresswhether continuous variables can be identified asputative risks. For example, instead of comparingpatients with hypercholesterolemia to those who donot carry that diagnosis, average cholesterol levelsassociated with patients who develop myocardialinfarction could be compared with the average levelof controls who have not had a myocardialinfarction. Furthermore, likelihood ratios can bederived which reflect the risk of myocardialinfarction for different ranges of cholesterol level.This information will be particularly useful forpatient assessment and clinical decision-making.

As the breadth, depth and longitudinal scope of ourdata collection continue to grow, it will becomepossible to determine the impact of risk factormodification on the course of a disease. Forinstance, while we have shown an associationbetween hypertension and myocardial infarction, weshould also be able to show a reduced association asblood pressure improves over time.

The goal of health services research extends beyondthe integration and analysis of traditional clinicaland administrative parameters. Validatedmeasurements of health related quality of life canprovide important, additional insights into the well-being of a population. Paul Ellwood, in his 1988Shattuck Lecture,'5 envisioned a new era in

303

outcomes research wherein health servicesresearchers could rapidly design and execute clinicaltrials that could detect when different treatmentstrategies result in meaningful improvements in thecourse of a given disease. Although outcomes ofinterest have traditionally been assessed usingconcrete measures such as mortality or health careutilization, these outcomes do not necessarily reflectthe impact of other non-fatal, but life-altering effectsof disease and its treatment. The Short Form 12 (SF-12),16 a brief 12-question survey that evaluatespatients' perceptions of their health status, is one ofmany instruments that capture this essential clinicaldetail. Patients' ability to complete the survey inunder two minutes, and the existence ofcomputerized methods of administration shouldenable data collection on a large scale, within theworkflow of a clinical encounter.

We have shown that a Health Services ResearchWorkstation can provide preliminary answers toimportant clinical questions in real-time, and at farless cost than traditional studies. It may alsovalidate results of randomized controlled trialsperformed on a specific population. As a result, thisnew integrated methodology represents an importantadvance in the feasibility, applicability andacceptability of outcomes research. Furtherenhancements, including the addition of moresophisticated analytical techniques and healthrelated quality of life measures, will provide apowerful mechanism for continuous monitoring andimprovement ofthe quality of care.

AcknowledgmentsDr. Weiner was supported by the National LibraryofMedicine Independent Fellowship in AppliedInformatics, grant number LM00051.

References

Garrett LE, Hammond WE, Stead WW. Theeffects of computerized medical records on providerefficiency and quality of care. Meth Inform Med1986;25:151-7.2 McDonald CJ, Hui SL, Smith DM. Reminders tophysicians from an introspective computer medicalrecord: A two year randomized trial. Ann InternMed 1984;100:130-8.3 Evans RS, Burke JP, Pestotnik SL, Classen DC,Menlove RL, Gardner RM. Prediction of hospitalinfections and selection of antibiotics using anautomated hospital database. Proceedings, SCAMC1990;14;663-8.

' Tierney WM, McDonald CJ, Luft FC. Renaldisease in hypertensive adults: Effect of race andtype II diabetes. American Journal of KidneyDiseases 1989:13;485-493.5 Evans R, Gardner RM, Bush AR, Burke JP,Jacobson JA, Larson RA, et al. Development of acomputerized infectious disease monitor.Computers in Biomedical Research 1985;18:103-113.6 Tierney WM, McDonald CJ. Practice databasesand their uses in clinical research. Statistics inMedicine 1991;10:541-57.7 Hlatky MA, CaliffRM, Harrell FE, Lee KL, MarkDB, Pryor DB. Comparison of predictions based onobservational data with results of randomizedcontrolled clinical trials of coronary artery bypasssurgery. Journal of the American College ofCardiology 1988;1 1:237-45.8 Hornberger, J, Wrone E. When to base clinicalpolicies on observational versus randomized trialdata. Ann Intern Med 1997; 127:697-703.9 Scully KW, Pates RD, Desper GS, Connors AF,Harrell FE, Pieper KS, et al. Development of anEnterprise-Wide Clinical Data Repository: MergingMultiple Legacy Databases. Proceedings, SCAMC1997;21 ;32-36.10 Safran C, Chute CG. Exploration and exploitationof clinical databases. International Journal ofBiomedical Computing 1995;39:151-6.'" Hennekens CH, Buring JE. Epidemiology inMedicine. Boston: Little Brown and Co, 1987.12 SHEF Cooperative Research Group. Preventionof stroke by antihypertensive drug treatment in olderpersons with isolated systolic hypertension. Finalresults of the Systolic Hypertenstion in the ElderlyProgram. JAMA 1991 ;265:3255-64.'3 Kannel WB, Dawber TR, Kagan A, Revotskie N,Stokes J. Factors of risk in the development ofcoronary heart disease -six-year follow-upexperience: the Framingham Study. Ann Intern Med1961 ;55:33-50.14 Atkins D, Psaty BM, Koepsell TD, LongstrethWT, Larson EB. Cholesterol reduction and the riskfor stroke in men: A meta-analysis of randomizedcontrolled trials. Ann Intern Med 1993;1 19:136-145.15 Ellwood PM. Shattuck lecture--Outcomesmanagement. A technology of patient experience.New England Journal of Medicine. 318:1549-56,1988.16 Ware JE Jr. Kosinski M, Keller SD. A 12-itemshort-form health survey: Construction of scales andpreliminary tests of reliability and validity. MedicalCare, 34:220-33, 1996.

304