March 4, 2008: I. SimInformatics for Clinical Research Epi 206 – Medical Informatics Ida Sim, MD,...

March 4, 2008: I. Sim Informatics for Clinical ResearchEpi 206 – Medical Informatics

Ida Sim, MD, PhD

March 4, 2008

Division of General Internal Medicine, and Center for Clinical and Translational Informatics

UCSF

Informatics for Clinical Research

Copyright Ida Sim, 2008. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.

2

Draft

for N

RC com

mitt

ee re

port,

do

not c

ite o

r circ

ulat

e Big Picture of Health Informatics

Virtual Patient

Transactions

Raw data

Medical knowledge

Clinical research

transactions

Raw research

data

Dec

isio

n su

ppor

t

Med

ical

logi

c

PATIENT CARE / WELLNES RESEARCH

Workflow modeling and support, usability, cognitive support, computer-supported cooperative work (CSCW), etc.

CTMSs


Outline

• Clinical Trial Management Systems (CTMS)

– NCI/NIH vision

– case study: UCSF Cancer Center

• Naming data• Running trials on the Web• Summary


Biomedical Informatics Spheres

Administrative Clinical Care Research

ClinicalBilling

Physical Networking

Standard Communications Protocols (e.g., HL-7)

Standard Vocabulary

PracticeManagement

Systems

Medical BusinessData Model

ElectronicMedicalRecord

Clinical CareData Model

??

??


CTMS

Administrative Clinical Care Research

ClinicalBilling

Physical Networking

Standard Communications Protocols (e.g., HL-7)

Standard Vocabulary

PracticeManagement

Systems

ElectronicMedicalRecord

Clinical Trial Management

Systems

Medical BusinessData Model

Clinical CareData Model

Clinical StudyData Models


EHR vs. CTMS

• EHR• Patient demographics• Chart notes

– problem list

• Visit and assessment• Lab and other orders• Results reporting• Clinical decision-making• Discharge summary

• CTMS• Title, NCT #, IRB #• Protocol document

– interventions, design, outcomes, etc.

• Study assessment• Outcomes assessment• Case report forms• Data analysis• Trial reporting/publication


Clinic 2008

FrontDesk

Radiology

MedicalInformationBureau

Walgreens

Pharm BenefitManager

Benefits Check(RxHub)

HealthNet

B&T

UCare

Specialist

ReferralAuthorization

Internet Intranet Phone/Paper/Fax

Lab

UniLab

(HL-7)

IRB

Trial Design

Protocol

Funding Agency

Site 1 Site 2 Site 3

Site Management Organization (SMO)

Study DB

Data analysis

Results reporting

Contract R

esearch Organ

ization

(CR

O)

SponsorsAcademic PIs

?

Clinical Research Today

• >80% on paper


Market Facts

• In early 90s, 80% trials done in academia, now reversed

• Huge business– Big Pharma: $15.9 billion on clinical trials [2006 PhRMA Industry

Profile]

– NIH budget $28.6billion 2005

• Fragmented, global industry– estimated 1200 organizations involved in clinical research in 2004 in

US (Sponsors, CROs, SMOs, AHCs...)

– CROs >$8b market in 2002, enrolled 20 million subjects in 2001

– “top US firms projecting...within 2 to 3 years as much as 65% of their FDA-regulated clinical trials will be conducted abroad” [Tufts Outlook 2006]


Clinic 2008

FrontDesk

Radiology

MedicalInformationBureau

Walgreens

Pharm BenefitManager

Benefits Check(RxHub)

HealthNet

B&T

UCare

Specialist

ReferralAuthorization

Internet Intranet Phone/Paper/Fax

Lab

UniLab

(HL-7)

IRB

Trial Design

Protocol

Funding Agency

Site 1 Site 2 Site 3

Site Management Organization (SMO)

Study DB

Data analysis

Results reporting

Contract R

esearch Organ

ization

(CR

O)

SponsorsAcademic PIs

?

Want “Interoperability”


Interoperability

• Ability of two or more systems or components to exchange information and to use the informationthat has been exchanged [IEEE Standard Computer Dictionary, 1990]

– syntactic: grammar, composition of what is said• e.g., using an exchange protocol

• e.g., HL7, DICOM, XML Document Type Definition (DTD)

– semantic: meaning of what is said• e.g., using a controlled vocabulary aka dictionary

• e.g., SNOMED, ICD-9


2 Routes to Interoperability

• “Microsoft” approach– one end-to-end system from trial inception to end

– used by all PIs, industry, CROs, worldwide

• Modular, component-based, interfaces approach– define common terms, models, interchange

protocols

– provide software components for assembly

– provide way to “certify” compatibility of systems


Clinical Trial Software Components

Clinical Trial Management Infrastructure

Adverse Event Reporting

Study Calendar

Routine Data Exchange Financial Billing

Structured Protocol

Representation

Laboratory Interfaces

Clinical Trials DatabaseSystems

Interoperation

CRF Standardization

InvestigatorAnd

Site Credentialing


NCI Priority for Interoperation

Clinical Trial Management Infrastructure

Adverse Event Reporting

Study Calendar

Routine Data Exchange Financial Billing

Structured Protocol

Representation

Laboratory Interfaces

Clinical Trials DatabaseSystems

Interoperation

CRF Standardization

InvestigatorAnd

Site Credentialing


NCI caBIG Vision

caBIGTM Architecture WorkspacecaBIGTM Architecture Workspace

caBIGTM Vocabularies and Common Data Elements WorkspacecaBIGTM Vocabularies and Common Data Elements Workspace

Strategic Planning

Workspace

Strategic Planning

Workspace

Training Workspace

Training Workspace

Integrative Cancer

Research

Workspace

Integrative Cancer

Research

Workspace

In Vivo Imaging

Workspace

In Vivo Imaging

Workspace

Tissue Banks &

Pathology Tools

Workspace

Tissue Banks &

Pathology Tools

Workspace

Data Sharing & Intellectual

Capital Workspace


Capital Workspace

Clinical Trials

Management Systems

Workspace

Clinical Trials

Management Systems

Workspace

15

CTMS Workspace Goals

• Facilitate the planning and instantiation of clinical trials,

(and monitoring of trials once they are instantiated)

• Facilitate the conduct of clinical trials

• Facilitate the reporting and sharing of clinical trial data

to existing/ new destinations

• Achieve interoperability

• Increase the ability of systems to access and use the

data and functionality of other systems

• Facilitate the integration of new sources and destinations

of data


Critical Components

• Terminologies/vocabularies– base terms used to describe biomedical concepts

• e.g., SNOMED, NCI Thesaurus

• Common Data Elements– clinicallly-agreed upon data items for research

• e.g., “menopause” defined a certain way

• Common data model of study protocol– study information: e.g., eligibility criteria, treatment, outcomes

• CTOM, SDTM, BRIDG, etc. etc.

• Common interchange standards– e.g., CDISC (“HL7 for clinical research”)– so design, monitoring, reporting systems, etc. can talk


NCI-->CTSA-->NIH xxBIG??

caBIGTM Architecture WorkspacecaBIGTM Architecture Workspace

caBIGTM Vocabularies and Common Data Elements WorkspacecaBIGTM Vocabularies and Common Data Elements Workspace

Strategic Planning

Workspace

Strategic Planning

Workspace

Training Workspace

Training Workspace

Integrative Cancer

Research

Workspace

Integrative Cancer

Research

Workspace

In Vivo Imaging

Workspace

In Vivo Imaging

Workspace

Tissue Banks &

Pathology Tools

Workspace

Tissue Banks &

Pathology Tools

Workspace


Capital Workspace


Capital Workspace

Clinical Trials

Management Systems

Workspace

Clinical Trials

Management Systems

Workspace


Outline


– NCI/NIH vision


• Naming data• Running trials on the Web• Summary


Commercial CTMSs• Functions

– document management (protocol, CRFs (Case Report Forms))

– finances, IRB– study calendar (what to do to whom when)– data management and analysis– reporting

• Examples– proprietary

• OracleClinical, C3D, Velos, etc. etc.

– open source• OpenClinica


Cancer Center Needs

• Reports need to continue as an NCI-designated Cancer Center– e.g., trials initiated, pt demographics

• UCSF CC needed good data for reports– bought Velos eResearch in 2003 for reporting

– required all CC investigators to use Velos• licenses issued to PIs in CC and some in GCRCs


What Gets into Velos?

• Study summary data, as free text– administrative (PI, contact info, etc)– protocol (treatments, outcomes, etc)– IRB-related

• submitted protocol, approval status, current protocol• tied into Cancer Center PRS review, but not CHR

• Patient data– PI can define CRFs, data entered would be stored

in a Velos database• basic report functions• export to SAS or Excel for analysis


How is it Working?• Adoption

– rocky initially, better since integrated with PRS • For NCI reporting, successful• For data hosting and analysis

– just started this in 2007(!)– “easy to get data in, hard to get data out”

• biostatisticians are not too happy

• underlying data schema and access are opaque, by design

• less reponsive/customizable to reporting and analysis needs

• Not used for finances, study calendar, standardization of terms, etc.


Outline


– NCI/NIH vision


• Naming data• Running trials on the web• Summary


Data Elements• To ensure sharable data, research data

variables should be standardized– standardized in naming

• terms should be from a controlled vocabulary (e.g., SNOMED, NCI Thesaurus)

– standardized to be common across studies• e.g., menopause with full clinical definition, to be used in

all breast cancer trials

• Both can be standardized through common Case Report Forms (CRFs)

Use “Add” button to list multiple symptoms

SNOMED Concept has been added to formRichesson R, 2007

SNOMED-CT Browser for Research

Implementation Status• > 300 electronic case report forms (CRFs)

• > 26,000 questions

• ~ 400 SNOMED CT and RxNorm browser contexts– 75% represent current findings (e.g., context is physical exam,

clinical assessment forms) and 25% represent historical findings (e.g., medical history context).

• 200 investigators and research staff have been trained on this tool

– Compliance is high (up to 90%)

– Satisfaction seems high (few complaints)

Richesson R, 2007


Standardization Across Studies

• Studies fundamentally done to demonstrate differences between interventions, by types of patients [Clarke M, Trials 2007]

– common outcome measures necessary for pooling / systematic reviews

• e.g., 5-year cancer free survival, common asthma measures– also common eligibility criteria, e.g., Post-menopause

• post (Prior bilateral ovariectomy, OR >12 mo since LMP with no prior hysterectomy and not currently receiving therapy with LH-RH analogs [eg. Zolades])

• post (Prior bilateral ovariectomy, OR >12 mo since LMP with no prior hysterectomy)

• pre (<6 mo since LMP AND no prior bilateral ovariectomy, AND not on estrogen replacement)

• above categories not applicable AND Age >=50


NCI Approach in Cancer

• NCI caDSR (Data Standards Repository)– library of Common Data Elements (CDEs) that

others have defined– you can define new CDEs using terms from NCI

Thesaurus

• Let’s go search...– http://cdebrowser.nci.nih.gov/CDEBrowser/

– http://cdebrowser.nci.nih.gov/CDEBrowser/common/help/cdeBrowserHelp.html

http://cdebrowser.nci.nih.gov/CDEBrowser/common/help/cdeBrowserHelp.html




Case Report Forms

• Why reinvent the wheel for common forms?• caDSR has a Forms Builder

– pull together CDEs into CRFs• can download as HTML, XML, Excel, Word, PDF

• NCI now building library of CRFs – e.g., Demographics CRF built from CDEs

• PDF, Word, etc. or directly to CTMS for direct data entry

• Velos– can design CRFs in Velos, with direct access to NCI’s CDEs– if caBIG compatible, could in future access NCI’s CRF

library directly


Outline


– NCI/NIH vision




Internet vs. Web

itsa

medicine

ucsf.edu

nci.nih.gov cochrane.uk myhome.com

Main Trunk Cables

local trunk cablethrough Berkeley

amazon.com

at homedial-in to itsa.ucsf.edu via modem

pacbell.net

aol.com

Internet Service Provider (ISP)via DSLor cable

LAN


Internet vs. Web

• Internet = network of networks– computers and cables all linked to one another

and talking to one another using protocols

– supports lots of different internet protocols• e.g., http, ftp, smtp, https, rdf, doi, etc. etc.

• Web is the internet traffic that uses http– servers send out information in HTML

• Hypertext Markup Language

– web browsers can decode HTML and display it


Clients and Servers

itsa

medicine

ucsf.edu

nci.nih.gov cochrane.uk myhome.com

Main Trunk Cables

amazon.com

at home

pacbell.net

aol.com

LAN

Server

Client


Research IT on Internet/Web

• Research IT using Internet– uses Internet network of networks to send data and

commands back and forth– servers and clients do the storage, query, retrieval,

computation, reporting– may have nothing to do with a web browser

• Research IT using Web– web servers send HTML content over the Internet using

HTTP– web browsers and other “clients” receive that content for

display or computation• What are logistical and methodological issues?


Eg: Smoking Cessation Trial

• Target audience– English and Spanish-speaking smokers

• Pre- and post demographic, etc. survey• Randomized Interventions

– downloadable brochure vs. brochure + email reminders + diary

• Outcome– quit rate

Slides from Ricardo Muñoz, [email protected] World Health Research Center, www.health.ucsf.edu

mailto:[email protected]


Surveys on the Web

• Survey design and deployment• Recruitment

– selection, sampling, non-response bias

• Sample size calculation


Web Surveys are Cheaper

• Web surveys have higher fixed cost but cost per additional respondent is much lower– marginal cost per mail survey respondent $1.93– phone $40 to $100– web $0

• Buy or build?– buy: many companies offer survey design,

deployment, and data management services– build: do-it-yourself


Buying Survey Services

• Many, many companies exist• Survey Monkey www.surveymonkey.com

– free for 10 questions, 100 responses per survey– professional subscription $19.95/mo or $200/yr

• up to 1000 responses per month, $0.05 per additional response

• DatStat’s Illume – web-based survey creation and management– real-time data access and complex query capabilities– exports data to SAS, SPSS, etc. – Internet World Health Research Center is beta user

• $7000/yr first year, $3000/yr thereafter

• $4000 license/user (e.g., you)

Disclosure: I have no ties to SurveyMonkey or DatStat

http://www.surveymonkey.com/


Survey Design• Usual survey design issues apply, PLUS• Technical design of survey

– platform (e.g., Mac) and browser (e.g., Safari) incompatibilities

– use Flash, Java, etc requiring plug-ins or version compatibility

– readiblity (font too small), need to scroll, confusing navigation, bugs

• What technology does respondent group use?– check some browser statistics sources

• e.g., http://www.w3schools.com/browsers/browsers_stats.asp

– need to test and double-test in various platforms and browsers used, various versions of HTML, Java, Flash, etc.


Measurement Bias

• What you designed may not be what respondent sees

• Client’s browser displays the survey based on – platform, browser, monitor, screen/window size

– different users see different survey, e.g., • small screen/window size makes “Next” button not visible

• text doesn’t fit on small window, or requires scrolling for some respondents and not others

• colors, graphics (e.g., visual analog scales) may appear differently


Non-Completion Bias

• Influenced by– respondent familiarity with web (e.g., click on link)– technical design of survey– bandwidth– convenience (return to finish?)

• Can use mixed-mode surveys to address– e.g., combined web/phone, web/mail


Subject Recruitment

• Recruitment is biggest bottleneck of clinical research– 30-40% of clinical trial costs – >80% of trials have recruitment delays– 1/20 recruited patients actually enroll

• Web-based recruitment can be international, cheap, fast– e.g., www.stopsmoking.ucsf.edu Dec 05 - Feb 07

• 350,000 hits, 60,000 entered data, 20,000 enrolled• 2/3 Spanish-speaking, 1/3 English• 131,517 visits from 121 countries Jan 12, 05 to April 5,

06

http://www.stopsmoking.ucsf.edu/


Threats to Validity

• Selection bias: who is on the web? who isn’t?– digital divide

• Sampling error– non-biased sampling of respondent population

• Non-response bias– enrollees not completing the survey

• Measurement error– poor question wording, variation in how survey

appears on various browsers, non-completion


Digital DivideInternet Access Broadband Access

<$30,000 41% 8%

$30-49,000 71% 16%

>$50,000 89% 39%

No children 59% 16%

Children in home 76% 29%

White 69% 23%

African-American 56% 15%

Hispanic 48% 14%

"Digital Divide" Still Shapes Media Landscape (10/19/04, Knowledge Networks/SRI); http://www.knowledgenetworks.com/info/press/releases/2004/101904_htmtrends.htm

http://www.knowledgenetworks.com/info/press/releases/2004/101904_htmtrends.htm


Digital Health Divide

• Spanish-language sites have lower quality– 45% of English-language sites vs. 22% with minimal

coverage & complete accuracy (JAMA 2001; 285:2612-2621)

• Broadband more available to higher-income white households with children– uneven potential access to Flash, tele-consultation,

etc.

• Most of divide attributable to income, not to race


Reducing Sampling Error

• Social sciences and marketing are most advanced in web survey methodology– e.g., Joint Statistical Meetings of the American

Statistical Association

– http://www.knowledgenetworks.com/dmg/index.html

• Recruit a representative sample• Use a pre-assembled representative cohort

Disclosure: I have no relationship with KnowledgeNetworks


Recruit Representative Sample

• Random digit dialing (RDD) analog equally representative as (land-line) telephone RDD– RDD sampling

– if respondent agrees, provide them with free Internet access (via MSNTV, aka WebTV) or other necessary hardware for duration of participation

– e.g.,http://knowledgenetworks.com/

http://knowledgenetworks.com/




Representative Cohorts

• Maintained by e.g., large survey and marketing firms– www.knowledgenetworks.com

• KnowledgePanel is representative of US• can target specific respondents, “response rates of 65-

75%, abandonment rate <2%”

– www.surveysampling.com• panels in 17 countries totaling 3.8 million respondents

– http://experimentcentral.org/ • NSF-funded representative panel for social science

research

http://www.knowledgenetworks.com/

http://www.surveysampling.com/

http://experimentcentral.org/


Enrollment Rates

• Response rates typically 30-60%• Affected by

– number of (pre) contacts, whether personalized• most influential factors

– incentives (e.g., Amazon certificate)

– population surveyed, nature of topic, official sponsorship, etc.


Other Recruitment Methods

• Higher risk of sampling bias– search engines

• need seach engine optimzation (SEO) techniques, e.g., buying Google keywords

– links from related pages– email lists, social networking sites, chat rooms,

newsgroups– other Internet communities

• friends, webrings

– blend traditional and web• give website on radio, TV, print, brochures


Note on Sample Size

• Estimating sample size– e.g., Google provides traffic history for various

keywords (adwords.google.com)

• Since incremental cost often negligible, less pressure to minimize sample size– not unusal to get large samples (>10,000)

• But high sample size = high accuracy!– may be precise but inaccurate if sample is non-

representative


Outline


– NCI/NIH vision




Summary

• Clinical research fragmented, global, essentially separate from clinical care

• Clinical reseach informatics in 2 worlds– most still paper, commercial CTMSs mostly document

centered (PDFs) rather than data or concept-centered

– movement towards caBIG-like world with • standard data elements (CDEs) and case report forms (CRFs)

• common data models (e.g., BRIDG) and interchange exchange standards (CDISC)

• Surveys on web offer promises and methodologic pitfalls

March 4, 2008: I. SimInformatics for Clinical Research Epi 206 – Medical Informatics Ida Sim, MD,...

Documents

Transcript of March 4, 2008: I. SimInformatics for Clinical Research Epi 206 – Medical Informatics Ida Sim, MD,...