REVIEW

Interpretation and Impact of Real-World Clinical Datafor the Practicing Clinician

Lawrence Blonde . Kamlesh Khunti . Stewart B. Harris .

Casey Meizinger . Neil S. Skolnik

Received: July 5, 2018 / Published online: October 24, 2018� The Author(s) 2018

ABSTRACT

Real-world studies have become increasinglyimportant in providing evidence of treatmenteffectiveness in clinical practice. While ran-domized clinical trials (RCTs) are the ‘‘goldstandard’’ for evaluating the safety and efficacyof new therapeutic agents, necessarily strictinclusion and exclusion criteria mean that trialpopulations are often not representative of the

patient populations encountered in clinicalpractice. Real-world studies may use informa-tion from electronic health and claims data-bases, which provide large datasets from diversepatient populations, and/or may be observa-tional, collecting prospective or retrospectivedata over a long period of time. They cantherefore provide information on the long-termsafety, particularly pertaining to rare events,and effectiveness of drugs in large heteroge-neous populations, as well as information onutilization patterns and health and economicoutcomes. This review focuses on how evidencefrom real-world studies can be utilized tocomplement data from RCTs to gain a morecomplete picture of the advantages and disad-vantages of medications as they are used inpractice.Funding: Sanofi US, Inc.

Keywords: Clinical practice; Real-world data;Real-world study

INTRODUCTION

Real-world studies seek to provide a line ofcomplementary evidence to that provided byrandomized controlled trials (RCTs). WhileRCTs provide evidence of efficacy, real-worldstudies produce evidence of therapeutic effec-tiveness in real-world practice settings [1]. The

Enhanced digital features To view enhanced digitalfeatures for this article go to https://doi.org/10.6084/m9.figshare.7156850.

L. Blonde (&)Ochsner Diabetes Clinical Research Unit,Department of Endocrinology, Frank RiddickDiabetes Institute, Ochsner Medical Center, NewOrleans, LA, USAe-mail: lblonde@ochsner.org

K. KhuntiDiabetes Research Centre, Leicester GeneralHospital, University of Leicester, Leicester, UK

S. B. HarrisSchulich School of Medicine and Dentistry, Centrefor Studies in Family Medicine, Western Centre forPublic Health and Family Medicine, WesternUniversity, London, ON, Canada

C. MeizingerDepartment of Family Medicine, Abington JeffersonHealth, Abington, PA 19001, USA

N. S. SkolnikAbington Family Medicine, Jenkintown, PA, USA

Adv Ther (2018) 35:1763–1774

https://doi.org/10.1007/s12325-018-0805-y

RCT is a well-established methodology forgathering robust evidence of the safety andefficacy of medical interventions [2]. In RCTs,the investigators are able to reduce bias andconfounding by utilizing randomization andstrict patient inclusion and exclusion criteria.This internal validity is often achieved at theexpense of external validity (generalizability),since the populations enrolled in RCTs maydiffer significantly from those found in every-day practice. Real-world evidence has emergedas an important means to understanding theutility of medical interventions in a broader,more representative patient population. Thestrict exclusion criteria for RCTs may excludethe majority of patients seen in routine care;therefore, real-world evidence can give vitalinsight into treatment effects in more diverseclinical settings, where many patients havemultiple comorbidities [3, 4].

Data from real-world studies can provideevidence that informs payers, clinicians, andpatients on how an intervention performs out-side the narrow confines of the research setting,providing essential information on the long-term safety and effectiveness of a drug in largepopulations, its economic performance in anaturalistic setting, and for assessment of com-parative effectiveness with other treatments.With improvements in the rigor of methodol-ogy being applied to real-world studies, alongwith the increasing availability of higher-qual-ity, larger datasets, the importance of findingsfrom these studies is growing. The value of real-world data has been recognized by regulatorybodies such as the US Food and Drug Adminis-tration (FDA) and the European MedicinesAgency (EMA) [5, 6]. These bodies acknowledgethe importance of real-world data in supportingmarketed products and their potential role insupporting life cycle product develop-ment/monitoring and decision-making for reg-ulation and assessment [5, 6]. A survey of thepharmaceutical and medical devices industry inthe European Union and the USA determinedthat 27% of real-world studies are conducted byindustry, performed ‘‘on request’’ by regulatoryauthorities [7]. Real-world data form a keycomponent of healthcare technology assess-ments used by national and regional bodies,

such as the National Institute for Health andCare Excellence (NICE) in the UK and Ger-many’s Institute for Quality and Efficiency inHealth Care (EQWiG), to guide clinical deci-sion-making [8]. The data from real-worldstudies are also increasingly utilized by payers.In a US survey, the majority of payers whoresponded reported using real-world data toguide decision-making, in particular on utiliza-tion management and formulary placement [9].Such data usage may have profound effects; forexample, the reversal of a decision by theEQWiG that analogue basal insulins showed nobenefit over human insulin, which restoredmarket access and premium pricing for insulinglargine in Germany [10]. The increase in thenumber of real-world studies has resulted inmore clinical evidence being available to guidetreatment decisions, and can allow assessmentof the impacts of off-label use. In this paper, wereview the impact of real-world clinical data andhow their interpretation can assist clinicians toassess clinical evidence appropriately for theirown decision-making.

The Association of the British Pharmaceuti-cal Industry defines real-world data as ‘‘data thatare collected outside the controlled constraintsof conventional RCTs to evaluate what is hap-pening in normal clinical practice’’ [11]. Real-world studies can be either retrospective orprospective, and when they include prospectiverandomization, they are called ‘‘pragmatic trialdesign’’ studies (Table 1) [12]. The clearest dis-tinction between RCTs and real-world studies isbased on (a) the setting in which the research isconducted and (b) where evidence is generated[2]. RCTs are typically conducted with preciselydefined patient populations, and patient selec-tion is often contingent on meeting extensiveeligibility (i.e., inclusion and exclusion) criteria.Participants in such trials (and the data theyprovide) are subject to rigorous quality stan-dards, with intensive monitoring, the use ofdetailed case-report forms (to capture additionalinformation that may not be present in ordi-nary medical records), and carefully managedcontact with research personnel (who areresponsible for ensuring protocol adherence)being commonplace. Real-world evidence, incontrast, is often derived from multiple sources

1764 Adv Ther (2018) 35:1763–1774

that lie outside of the typical clinical researchsetting: these can include offices that are notgenerally involved in research, electronic healthrecords (EHRs), and patient registries andadministrative claims databases (sometimesobtained from integrated healthcare deliverysystems). Despite these differences, real-worldevidence can also be used retrospectively asexternal control arms for RCTs, to providecomparative efficacy data [13]. Consequently,this article is based on previously conductedstudies and does not contain any studies withhuman participants or animals performed byany of the authors.

Large ‘‘pragmatic trials’’ are an increasinglycommon real-world data source. Such trials aredesigned to show the real-world effectiveness ofan intervention in a broad patient group [14].

They incorporate a prospective, randomizeddesign and collect data on a wide range ofhealth outcomes in a diverse and heterogeneouspopulation (i.e., they are consistent with clini-cal practice) [15–17]. Pragmatic trials are con-ducted in routine practice settings [1], include apopulation that is relevant for the interventionand a control group treated with an accept-able standard of care (or placebo), and describeoutcomes that are meaningful to the popula-tion in question [14]. Aspects of care other thanthe intervention being studied are intentionallynot controlled, with clinicians applying clinicaldiscretion in their choice of other medications[11]. Pragmatic trials may focus on a specifictype of patient or treatment, and study coordi-nators may select patients, clinicians, and clin-ical practices and settings that will maximize

Table 1 Comparison of randomized controlled trials and real-world studies

Randomized controlledtrials

Real-world studies

Type of study Experimental/

interventional

Observational/non-interventional Interventional/pragmatic

Design Prospective Retrospective/prospective Prospective

Primary focus Efficacy, safety, quality, cost-

effectiveness

Efficacy, safety, quality, cost-effectiveness, natural history, compliance

and adherence, service models, patient preferences, comparative

Patient population Narrow, restricted,

motivated

Diverse, large, and unrestricted

Monitoring Intense (ICH-GCP

compliant)

Not required (?) Reflects usual care

Comparators Gold standard/placebo None/standard clinical

practice/multiple iterations

Standard practice/

placebo/multiple iterations

Outcomes Clear sequence Wide range

Data collection

confounders

Standardized, controlled Routine, recruitment bias (?), recall/interviewer bias

Randomization Yes No Yes

Blinding Yes No Sometimes (participants or

outcome assessment)

Follow-up Generally short Reflects usual care Long

ICH-GCP International Conference on Harmonisation of Good Clinical Practice

Adv Ther (2018) 35:1763–1774 1765

external validity (i.e., the applicability of theresults to usual practice) [16]. As such, prag-matic trials are able to provide data on a rangeof clinically relevant real-world considerations,including different treatments, patient- andclinician-friendly titration and treatment algo-rithms, and cost-effectiveness, which in turnmay help address practice- and policy-relevantissues. These studies can focus specifically onthe outcomes which are most important topatients, and take into account real-worldtreatment adherence and compliance on thedirect impact of a medication or treatmentregimen for patients.

UNDERSTANDING THE STRENGTHSAND WEAKNESSES OF REAL-WORLD STUDIES

Compared with RCT data, real-world evidencehas the potential to more efficiently provideanswers that inform outcomes research, qualityimprovement, pharmacovigilance, and patientcare [2]. As they are performed in clinical set-tings and patient populations that are similar tothose encountered in clinical practice, real-world studies have broader generalizability.Specifically, RCTs provide evidence of efficacy,while real-world studies give evidence of effec-tiveness in real-world practice settings [1].Additionally, observational, retrospective real-world studies are generally more economicaland time efficient than RCTs [18] as they useexisting data sources such as registries, claimsdata, and EHRs to identify study outcomes [16].

Key to the utility of real-world studies is theirability to complement data from RCTs in orderto fill current gaps in clinical knowledge.Specific trial criteria may cause RCTs to excludea particular group of patients commonly seen inclinical practice; for example, RCTs frequentlyexclude older adults. In the case of diabetes,while many RCTs focus primarily on the safetyand glucose-lowering efficacy of antihyper-glycemia drugs [19], it is desirable to have real-world effectiveness outcomes data in patientswith type 2 diabetes (T2D) that take intoaccount issues such as adherence [20, 21] andthe frequency of side effects in less controlled

settings (which may affect outcomes). Suchstudies suggest that the difference betweenglycated hemoglobin reduction in RCTs and inpractice may be related to adherence and pointto the potential value of real-world studiesassessing clinical-practice effectiveness. Inaddition, real-world evidence can addressimportant issues such as the impact of treat-ment on microvascular disease and cardiovas-cular (CV) events [22] and enable theexamination of outcomes, which are difficult toassess in RCTs, such as the utilization ofhealthcare resources by patients receiving dif-ferent therapies. In the DELIVER-3 study, forexample, insulin glargine 300 U/ml (Gla-300)was associated with reduced resource utilizationcompared with other basal insulins [23]. Anexample, which demonstrates the utility ofpragmatic trial design, is the exploration ofpatient-driven insulin titration protocols thathighlight the practical need that patients face ineveryday life, rather than reflecting the needs ofa highly controlled, well-motivated RCT popu-lation [24–26].

Real-world studies have a number of limita-tions. Retrospective and non-randomized real-world studies are subject to bias and con-founding factors, problems that are controlledfor in randomized blinded trials [27]. Electronicdata may be inconsistently collected, withmissing data elements that can eventually resultin reduced statistical validity and a decreasedability to answer the research question [16]. Thetypes of bias seen in real-world trials includeselection bias (e.g., therapies may be differentlyprescribed depending upon patient and diseasecharacteristics, e.g., severity of disease and/orother patient characteristics), information bias(misclassification of data), recall bias (caused byselective recall of impactful events bypatients/caregivers), and detection bias (wherean event is more likely to be captured in onetreatment group than another) [28]. While sys-tematic reviews have found little evidence tosuggest that treatment effects or adverse eventsin well-designed observational studies are eitheroverestimated or qualitatively different fromthose obtained in RCTs, each real-world studymust be examined individually for sources ofbias and confounding [29–31]. Indeed, caution

1766 Adv Ther (2018) 35:1763–1774

should be exercised when using data from real-world studies (particularly retrospective studies)to influence change in clinical practice [18]because of confounding and bias. Techniquessuch as propensity score matching (PSM) can beused to reduce selection bias by matching thecharacteristics of patients entering differentarms of studies (see below) [32].

Properly designed, prospective, interven-tional pragmatic trials have the potential toovercome many of the limitations of observa-tional and retrospective real-world studies.However, the main limitation of pragmatic tri-als is that they do not often place constraints onpatients and clinicians, which may result ininconsistent or missing data in source docu-ments such as EHRs. This, together withheterogeneity in terms of clinical practice andassociated documentation, may lead to areduced capability of the study to answer theresearch question [16]. In addition, hetero-geneity of clinical practice and patient popula-tions reduces the translatability of pragmatictrial data to different settings and locations [33].There are also numerous challenges inherent inpragmatic trial design. These are illustrated bythe trade-off between blinding of results toreduce bias and the desire to create a fullypragmatic design where the intervention isdelivered as in normal practice [14]. Pragmatictrials, in producing evidence of effectiveness inreal-world-practice settings, may trade aspectsof internal validity for higher external validity,which ultimately means that they are moregeneralizable than RCTs [1].

LEARNING FROM REAL-WORLDFINDINGS: EXAMPLES

Retrospective Observational Studies

A real-world study that had a definite effect onprescribing practice concerned a live attenuatednasal spray influenza vaccine in the USA. Onthe basis of results from a number of RCTs,which showed the superior efficacy of this vac-cine over the inactivated influenza vaccine, theAdvisory Committee for Immunization Prac-tices (ACIP) issued a guidance for its use in

children [34]. However, because of data fromreal-world observational studies showing worseperformance compared with the RCT data andnear zero performance against some pandemicinfluenza strains, the ACIP subsequently chan-ged its guidance and recommended against theuse of the live attenuated vaccine [34]. Retro-spective, observational real-world data canconfirm or refute the findings of RCTs. Forexample, the DELIVER-2 and DELIVER-3 studieswere conducted in a broad population ofpatients with T2D on basal insulin, includingat-risk older adults, and showed that those whoswitched to Gla-300 experienced significantlyfewer hypoglycemia events—including eventsassociated with hospitalization or emergencyroom visits—than those who switched to otherbasal insulins, without compromising bloodglucose control [23, 35, 36], corroborating theresults obtained in the EDITION RCTs [37–39].

Prospective Observational Studies

The importance of prospective observationalstudies has been clearly illustrated. For example,the Framingham Heart Study, initiated almost70 years ago [40]. This study has provided sub-stantial insight into the epidemiology of car-diovascular disease (CVD) and its risk factors,and has significantly influenced clinical think-ing and practice. In the case of diabetes,prospective observational studies have providedkey evidence that has guided the developmentof treatment guidelines worldwide. Ten years oflong-term follow-up after the completion of theUK Diabetes Study confirmed and extendeddata on the importance of glycemic control inpreventing the development of the microvas-cular and macrovascular complications of T2Din a real-world population [41]. The Epidemi-ology of Diabetes Interventions and Complica-tions (EDIC) prospective observational follow-up study of the Diabetes Control and Compli-cations Trial (DCCT) has described the long-term effects of prior intensive therapy comparedwith conventional insulin therapy on thedevelopment and progression of microvascularcomplications and CVD in type 1 diabetes [42].

Adv Ther (2018) 35:1763–1774 1767

The prospective observational ReFLeCTstudy is looking at rates of hypoglycemia, gly-cemic control, patient-reported outcomes, andquality of life under normal clinical practiceconditions in approximately 1200 Europeanpatients with either type 1 or 2 diabetes forwhich they are prescribed insulin degludec. Ananalysis of data from the Cardiovascular RiskEvaluation in people with type 2 Diabetes onInsulin Therapy (CREDIT) study found thatimproved glycemic control in patients begin-ning insulin resulted in significant reductionsin CV events such as stroke and CV death; nodifferences were observed between differentinsulin regimens, suggesting that it was goodglycemic control that was the most importantfactor [43].

Pragmatic Prospective Randomized Trials

A number of pragmatic randomized trials havebeen completed or are underway to investigatea range of real-world diabetes patient-careissues, including the long-term effectiveness ofmajor antihyperglycemia medications [44],glucose monitoring [45, 46], insulin initiation[47], and support strategies [48]. Since 2008, theFDA and subsequently the EMA have requiredsponsors of new antihyperglycemia therapies toevaluate their CV safety. This has resulted in anumber of large-scale CV outcome trialsincluding pragmatic trials such as the TrialEvaluating Cardiovascular Outcomes with Sita-gliptin (TECOS) [49] and the Exenatide Study ofCardiovascular Event Lowering (EXSCEL) trial[50].

REAL-WORLD STUDIES:ADDRESSING GENERALIZABILITY

RCT exclusion criteria may rule out a significantproportion of real-world patients. As previouslymentioned, patients excluded from RCTs areolder, have more medical comorbidities, andhave more challenging social and demographicissues than those included in these trials. Real-world studies have the potential to assess whe-ther results seen in RCTs would be generalizable

to real-world patient populations. The EMPA-REG OUTCOME RCT selected T2D patients withestablished CVD and, for those treated with thesodium-glucose co-transporter-2 (SGLT2) inhi-bitor empagliflozin vs placebo, reported a sig-nificant reduction in the primary compositeendpoint of a three-point major adverse cardiacevent (MACE) (CV death, non-fatal myocardialinfarction, and non-fatal stroke), as well as theindividual endpoints of CV death, all-causedeath, and hospitalization for heart failure [51].The CANVAS RCT investigating the SGLT2inhibitor canagliflozin, which included a lowerpercentage of patients at high CV risk thanEMPA-REG, also reported a significant reductionin the primary composite endpoint of a three-point MACE and the individual endpoint ofhospitalization for heart failure but did notshow a significant benefit for CV mortality orall-cause mortality alone [52]. Evidence from afurther real-world study may support andexpand upon the RCT data. The CVD-REALstudy in over 300,000 patients with T2D, bothwith (13% of the total) and without establishedCVD, showed a consistent reduction in hospi-talization for heart failure suggesting a real-world benefit of the SGLT2 inhibitor drug classas a whole in patients with T2D, irrespective ofexisting CV risk status or the SGLT2 inhibitorused [53].

IMPROVING QUALITY OF EVIDENCEGENERATED FROM REAL-WORLDSTUDIES

Criteria for the design of observational studieshave been developed and, if followed, shouldresult in higher-quality studies (Table 2) [28].The STROBE guidelines (STrengthening theReporting of OBservational studies in Epidemi-ology) provide a reporting standard for obser-vational studies [54]. An extension to theCONSORT guideline for RCTs provides specificguidance for pragmatic trials and provides areporting checklist that covers background,participants, interventions, outcomes, samplesize, blinding, participant flow, and generaliz-ability of findings [55]. Adherence to such cri-teria should improve not only the quality but

1768 Adv Ther (2018) 35:1763–1774

Table 2 Quality criteria for comparative observational database studies

Section Quality criteria

Background Clear underlying hypotheses and specific research question(s)

Methods

Study design Observational comparative effectiveness database study

Independent steering committee involved in a priori definition of the study methodology (including

statistical analysis plan), review of analyses, and interpretation of results

Registration in a public repository with a commitment to publish results

Database(s) High-quality database(s) with few missing data for measures of interest

Validation studies

Outcomes Clearly defined primary and secondary outcomes, chosen a priori

The use of proxy and composite measures justified and explained

The validity of proxy measures checked

Length of

observation

Sufficient duration to reliably assess outcomes of interest and long-term treatment effects

Patients Well-described inclusion and exclusion criteria, reflecting target patients’ characteristics in the real

world

Analyses Study groups compared at baseline using univariate analyses

Avoidance of biases related to baseline differences using matching and/or adjustments

Sensitivity analyses are performed to check the robustness of results

Sample size Sample size calculations based on clear a priori hypotheses regarding the occurrence of outcomes of

interest and target effect of studied treatment versus comparator

Results Flow chart explaining all exclusions

Detailed description of patients’ characteristics, including demographics, characteristics of the disease

of interest, comorbidities, and concomitant treatments

Characteristics of patients lost to follow-up are compared with those of patients remaining in the

analyses

Extensive presentation of results obtained in unmatched and matched populations (if matching was

performed) using univariate and multivariate, as well as unadjusted and adjusted, analyses

Sensitivity analyses and/or analyses of several databases go in the same direction as primary analyses

Adv Ther (2018) 35:1763–1774 1769

also the validity of real-world study data inclinical practice.

A number of methods have also been devel-oped to reduce the effects of confounding inobservational studies, including PSM. Thismethod aims to make it possible to compareoutcomes of two treatment or managementoptions in similar patients [32]. It does this byreducing the effects of multiple covariates to asingle score, the propensity score. Comparisonof outcomes across treatment groups of pairs orpools of propensity-score-matched patients canreduce issues such as selection bias [32].Although a powerful and widely used tool, thereare limits to the degree in which propensityscore adjustments can control for bias andconfounding variables. An example of this canbe seen in RCT versus real-world data for mor-tality in patients with severe heart failure trea-ted with the aldosterone inhibitorspironolactone [56]. While RCT data consis-tently showed a reduction in mortality, in areal-world study using PSM, spironolactoneappeared to be associated with a substantiallyincreased risk of death [57]. The authors of thestudy suggest that concluding that spironolac-tone is dangerous on the basis of the real-worldstudy is not legitimate because of issues ofunknown bias and confounding by indication(i.e., confounding due to factors not in thepropensity score or even not formally mea-sured) [57]. This illustrates a major limitation ofPSM: it can only include variables that are in theavailable data [58]. A further major limitation isthat the need for grouping or pairing data in

PSM narrows the patient population analyzed,limiting generalizability and thereby reducingone of the main values of real-world studies.

‘‘Big data’’ have emerged as a cutting-edgediscipline that uses capture of data from EHRsand other high-volume data sources to effi-ciently generate hypotheses about the relation-ship between processes and outcomes. Thisdemands an increased emphasis on the integ-rity of the data, with ‘‘high-quality’’ datadefined in terms of their accuracy, availabilityand usability, integrity, consistency, standard-ization, generalizability, and timeliness [59, 60].Missing data may represent a significant chal-lenge in some datasets. For example, the UShealthcare system (unlike many Europeancountries) relies on a number of different labo-ratory companies to supply laboratory resultsdata, which may result in inconsistencies in therecording of results in EHRs. The technical andmethodological challenges presented by thesenew data sources are an active area of endeavorby key stakeholders moving towards harmo-nization of data collected from high-volumedata sources, with the aim of creating a unifiedmonitoring system and implementing methodsfor incorporating such data into research [2].Artificial intelligence (AI) is the natural partnerof big data, and the increased availability ofthese data sources is already allowing AI toimprove clinical decision-making. AI tech-niques have used raw data gleaned from radio-graphical images, genetic testing,electrophysiological studies, and EHRs toimprove diagnoses [6].

Table 2 continued

Section Quality criteria

Discussion Summary and interpretation of findings, focusing first on whether they confirm or contradict a priori

hypotheses

Discussion of differences with results of efficacy RCTs

Discussion of possible biases and confounding factors, especially related to the observational nature of

the study

Suggestions for future research to challenge, strengthen, or extend study results

Reprinted with permission of the American Thoracic Society. Copyright � 2018 American Thoracic Society [28]RCT randomized control trial

1770 Adv Ther (2018) 35:1763–1774

As a final caveat, with the increasing avail-ability of real-world data, there may be somediscrepancies in information derived from dif-ferent sources. As with all data, be it from RCTsor real-world practice, consideration should begiven to the limitations and generalizability ofresults when interpreting individual study out-comes and applying them to everyday clinicalpractice.

CONCLUSIONS

Real-world studies provide important informa-tion that can complement and/or even expandthe information obtained in RCTs. RCTs set thestandard for eliminating bias in determiningefficacy and safety of medications, but havesignificant limitations with regard to generaliz-ability to the broad population of patients withdiabetes receiving health care in diverse clinicalpractice settings. Because real-world studies areperformed in actual clinical practice settings,they are better able to assess the actual effec-tiveness and safety of medications as they areused in real-life by patients and clinicians. Withimproving study designs, methodologicaladvances, and data sources with more compre-hensive data elements, the potential for real-world evidence continues to expand. Moreover,the limitations of real-world studies are betterunderstood and can be better addressed. Real-world evidence can both generate hypothesesrequiring further investigation in RCTs and alsoprovide answers to some research questions thatmay be impractical to address through RCTs.

ACKNOWLEDGEMENTS

KK acknowledges support from the NationalInstitute for Health Research (NIHR) Collabo-ration for Leadership in Applied HealthResearch and Care-East Midlands (CLAHRC-EM)and the NIHR Leicester Biomedical ResearchCentre.

Funding. Funding, including article pro-cessing charges and Open Access fee, was

provided by Sanofi US, Inc. All authors had fullaccess to all of the data in this study and takecomplete responsibility for the integrity of thedata and accuracy of the data analysis.

Medical Writing and Editorial Assis-tance. The authors received writing and edito-rial support in the preparation of thismanuscript. This support was provided by GraceRichmond, PhD, of Excerpta Medica, funded bySanofi US, Inc.

Authorship. All named authors meet theInternational Committee of Medical JournalEditors (ICMJE) criteria for authorship for thisarticle, take responsibility for the integrity ofthe work as a whole, and have given theirapproval for this version to be published.

Disclosures. Lawrence Blonde receivedgrant/research support and honoraria fromAstraZeneca, Intarcia Therapeutics, JanssenPharmaceuticals, Lexicon Pharmaceuticals,Merck, Novo Nordisk, and Sanofi. KamleshKhunti received honoraria and research supportfrom AstraZeneca, Boehringer Ingelheim, EliLilly, Janssen, Merck Sharp & Dohme, Novartis,Novo Nordisk, Roche, and Sanofi. Stewart Harrisreceived honoraria and grants/research supportfrom Abbott, AstraZeneca, Boehringer Ingel-heim, Eli Lilly, Intarcia, Janssen, Merck, NovoNordisk, and Sanofi, honoraria and consultingfees from Abbott, AstraZeneca, BoehringerIngelheim/Lilly, Janssen, Novo Nordisk, andSanofi, and honoraria from Medtronic andMerck. Casey Meizinger has nothing to disclose.Neil Skolnik served on advisory boards forAstraZeneca, Boehringer Ingelheim, JanssenPharmaceuticals, Intarcia, Lilly, Sanofi, andTeva, has been a speaker for AstraZeneca andBoehringer Ingelheim, and received researchsupport from AstraZeneca, Boehringer Ingel-heim, and Sanofi.

Compliance with Ethics Guidelines. Thisarticle is based on previously conducted studiesand does not contain any studies with humanparticipants or animals performed by any of theauthors.

Adv Ther (2018) 35:1763–1774 1771

Open Access. This article is distributedunder the terms of the Creative CommonsAttribution-NonCommercial 4.0 InternationalLicense (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercialuse, distribution, and reproduction in anymedium, provided you give appropriate creditto the original author(s) and the source, providea link to the Creative Commons license, andindicate if changes were made.

REFERENCES

1. Luce BR, Drummond M, Jonsson B, et al. EBM, HTA,and CER: clearing the confusion. Milbank Q.2010;88:256–76.

2. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us?N Engl J Med. 2016;375:2293–7.

3. Barnish MS, Turner S. The value of pragmatic andobservational studies in health care and publichealth. Pragmat Obs Res. 2017;8:49–55.

4. Fortin M, Dionne J, Pinho G, Gignac J, Almirall J,Lapointe L. Randomized controlled trials: do theyhave external validity for patients with multiplecomorbidities? Ann Fam Med. 2006;4(2):104–8.

5. FDA. Developing a framework for regulatory use ofreal-world evidence; Public Workshop. https://www.gpo.gov/fdsys/pkg/FR-2017-07-31/pdf/2017-16021.pdf. Accessed 08 Sep 2017.

6. EMA. Update on real world evidence data collec-tion. 10 March 2016. https://ec.europa.eu/health//sites/health/files/files/committee/stamp/2016-03_stamp4/4_ real_world_evidence_ema_presenta-tion.pdf. Accessed 08 Sep 2017.

7. Batrouni M, Comet D, Meunier JP. Real worldstudies, challenges, needs and trends from theindustry. Value Health. 2014;17:A587–8.

8. Goodman CS. National Information Center onHealth Services Research and Health Care Tech-nology (NICHSR): HTA 101, 2017. https://www.nlm.nih.gov/nichsr/hta101/ta10103.html. Acces-sed Feb 2018.

9. Malone DC, Brown M, Hurwitz JT, Peters L, Graff JS.Real-world evidence: useful in the real world of USpayer decision making? How? When? And whatstudies? Value Health. 2018;21(3):326–33.

10. Cattell J, Groves P, Hughes B, Savas S. How canpharmacos take advantage of the real-world dataopportunity in healthcare? McKinsey and Com-pany, 2011. https://www.mckinsey.com/*/media/mckinsey/dotcom/client_service/Pharma%20and%20Medical%20Products/PMP%20NEW/PDFs/Pharma%20%20RWD%20opportunity%20October%202011.ashx. Accessed Feb 2018.

11. ABPI. The vision for real world data—harnessingthe opportunities in the UK. Demonstrating valuewith real world data 2017. http://www.abpi.org.uk/media/1378/vision-for-real-world-data.pdf. Acces-sed 22 Jan 2018.

12. Schwartz D, Lellouch J. Explanatory and pragmaticattitudes in therapeutical trials. J Clin Epidemiol.2009;62(5):499–505.

13. Davies J, Martinex M, Martina R, et al. Retrospectiveindirect comparison of alectinib phase II data vsceritinib real-world data in ALK ? NSCLC afterprogression on crizotinib. Ann Oncol. 2017;28(suppl_2): ii28-ii51. 10.

14. Ford I, Norrie J. Pragmatic trials. N Engl J Med.2016;375:454–63.

15. Dang A, Vallish BN. Real world evidence: an Indianperspective. Perspect Clin Res. 2016;7:156–60.

16. Sox HC, Lewis RJ. Pragmatic trials: practical answersto ‘‘real world’’ questions. JAMA. 2016;316:1205–6.

17. Tunis SR, Stryer DB, Clancy CM. Practical clinicaltrials: increasing the value of clinical research fordecision making in clinical and health policy.JAMA. 2003;290:1624–32.

18. Dubois RW. Is the real-world evidence or hypothe-sis: a tale of two retrospective studies. J Comp EffRes. 2015;4(3):199–201.

19. Clinicaltrials.gov. Studies for ‘‘Diabetes Mellitus,Type 2’’. https://clinicaltrials.gov/ct2/results?cond=Diabetes?Mellitus%2C?Type?2&term=&cntry=&state=&city=&dist=. Accessed 21 Aug 2018.

20. Carls GS, Tuttle E, Tan RD, et al. Understanding thegap between efficacy in randomized controlled tri-als and effectiveness in real-world use of GLP-1 RAand DPP-4 therapies in patients with type 2 dia-betes. Diabetes Care. 2017;40:1469–78.

21. Edelman SV, Polonsky WH. Type 2 diabetes in thereal world: the elusive nature of glycemic control.Diabetes Care. 2017;40:1425–32.

22. McGovern A, Hinchliffe R, Munro N, de Lusignan S.Basing approval of drugs for type 2 diabetes on realworld outcomes. BMJ. 2015;351:h5829.

1772 Adv Ther (2018) 35:1763–1774

23. Zhou FL, Ye F, Gupta V, et al. Older adults with type2 diabetes (T2D) experience less hypoglycemiawhen switching to insulin glargine 300 U/mL (Gla-300) vs other basal insulins (DELIVER 3 study).Poster 986-P, American Diabetes Association (ADA)77th Scientific Sessions, San Diego, CA, US, June 10,2017.

24. Blonde L, Merilainen M, Karwe V, Raskin P. Patient-directed titration for achieving glycaemic goalsusing a once-daily basal insulin analogue: anassessment of two different fasting plasma glucosetargets-the TITRATETM study. Diabetes Obes Metab.2009;11:623–31.

25. Gerstein HC, Yale JF, Harris SB, et al. A randomizedtrial of adding insulin glargine vs. avoidance ofinsulin in people with type 2 diabetes on either nooral glucose-lowering agents or submaximal dosesof metformin and/or sulphonylureas. The CanadianINSIGHT (Implementing New Strategies with Insu-lin Glargine for Hyperglycaemia Treatment) Study.Diabet Med. 2006;23:736–42.

26. Meneghini L, Koenen C, Weng W, Selam JL. Theusage of a simplified self-titration dosing guideline(303 Algorithm) for insulin detemir in patients withtype 2 diabetes—results of the randomized, con-trolled PREDICTIVETM 303 study. Diabetes ObesMetab. 2007;9:902–13.

27. Garrison LP Jr, Neumann PJ, Erickson P, Marshall D,Mullins CD. Using real-world data for coverage andpayment decisions: the ISPOR Real-World DataTask Force report. Value Health. 2007;10:326–35.

28. Roche N, Reddel H, Martin R, et al. Quality stan-dards for real-world research. Focus on observa-tional database studies of comparativeeffectiveness. Ann Am Thorac Soc. 2014;11(Suppl2):S99–104.

29. Benson K, Hartz AJ. A comparison of observationalstudies and randomized, controlled trials. N Engl JMed. 2000;22(342):1878–86.

30. Concato J, Shah N, Horwitz RI. Randomized, con-trolled trials, observational studies, and the hierar-chy of research designs. N Engl J Med.2000;342:1887–92.

31. Golder S, Loke YK, Bland M. Meta-analyses ofadverse effects data derived from randomised con-trolled trials as compared to observational studies:methodological overview. PLoS Med.2011;8:e1001026.

32. McMurry TL, Hu Y, Blackstone EH, Kozower BD.Propensity scores: methods, considerations, andapplications. J Thorac Cardiovasc Surg.2015;150:14–9.

33. Patsopoulos NA. A pragmatic view on pragmatictrials. Dialogues Clin Neurosci. 2011;13:217–24.

34. Frieden TR. Evidence for health decision making—beyond randomized, controlled trials. N Engl J Med.2017;377:465–75.

35. Ye F, Agarwal R, Kaur A, et al. Real-world assessmentof patient characteristics and clinical outcomes ofearly users of the new insulin glargine 300U/mL.Poster 943-P, American Diabetes Association (ADA)76th Scientific Sessions, New Orleans, LA, US. June11, 2016.

36. Zhou FL, Ye F, Berhanu P, et al. Real-world evidenceconcerning clinical and economic outcomes ofswitching to insulin glargine 300 units/mL vs otherbasal insulins in patients with type 2 diabetes usingbasal insulin. Diabetes Obes Metab.2018;20(5):1293–7.

37. Bolli GB, Riddle MC, Bergenstal RM, et al. Newinsulin glargine 300 U/ml compared with glargine100 U/ml in insulin-naıve people with type 2 dia-betes on oral glucose-lowering drugs: a randomizedcontrolled trial (EDITION 3). Diabetes Obes Metab.2015;17:386–94.

38. Riddle MC, Bolli GB, Ziemen M, et al. New insulinglargine 300 units/mL versus glargine 100 units/mLin people with type 2 diabetes using basal andmealtime insulin: glucose control and hypo-glycemia in a 6-month randomized controlled trial(EDITION 1). Diabetes Care. 2014;37:2755–62.

39. Yki-Jarvinen H, Bergenstal R, Ziemen M, et al. Newinsulin glargine 300 units/mL versus glargine 100units/mL in people with type 2 diabetes using oralagents and basal insulin: glucose control andhypoglycemia in a 6-month randomized controlledtrial (EDITION 2). Diabetes Care. 2014;37:3235–43.

40. Mahmood SS, Levy D, Vasan RS, Wang TJ. TheFramingham Heart Study and the epidemiology ofcardiovascular disease: a historical perspective.Lancet. 2014;383:999–1008.

41. Stratton IM, Adler AI, Neil HA, et al. Associationwith glycaemia with macrovascular and microvas-cular complications of type 2 diabetes (UKPDS 35):prospective observational study. BMJ.2000;321:405–12.

42. Diabetes Control and Complications Trial/Epi-demiology of Diabetes Interventions and Compli-cations Research Group. Effect of intensive therapyon the microvascular complications of type 1 dia-betes mellitus. JAMA. 2002;287:2563–9.

43. Freemantle N, Danchin N, Calvi-Gries F, Vincent M,Home PD. Relationship of glycaemic control andhypoglycaemic episodes to 4-year cardiovascular

Adv Ther (2018) 35:1763–1774 1773

outcomes in people with type 2 diabetes startinginsulin. Diabetes Obes Metab. 2016;18:152–8.

44. Nathan DM, Buse JB, Kahn SE, et al. Rationale anddesign of the glycemia reduction approaches indiabetes: a comparative effectiveness study(GRADE). Diabetes Care. 2013;36:2254–61.

45. Wermeling PR, Gorter KJ, Stellato RK, et al. Effec-tiveness and cost-effectiveness of 3-monthly versus6-monthly monitoring of well-controlled type 2diabetes patients: a pragmatic randomised con-trolled patient-preference equivalence trial in pri-mary care (EFFIMODI study). Diabetes Obes Metab.2014;16:841–9. https://doi.org/10.1111/dom.12288.

46. Young LA, Buse JB, Weaver MA, et al. Threeapproaches to glucose monitoring in non-insulintreated diabetes: a pragmatic randomized clinicaltrial protocol. BMC Health Serv Res. 2017;17:369.

47. Furler J, O’Neal D, Speight J, et al. Supportinginsulin initiation in type 2 diabetes in primary care:results of the Stepping Up pragmatic cluster ran-domised controlled clinical trial. BMJ.2017;356:j783.

48. Choudhry NK, Isaac T, Lauffenburger JC, et al.Rationale and design of the Study of a Tele-phar-macy Intervention for Chronic diseases to ImproveTreatment adherence (STIC2IT): a cluster-random-ized pragmatic trial. Am Heart J. 2016;180:90–7.

49. Green JB, Bethel MA, Armstrong PW, et al. Effect ofsitagliptin on cardiovascular outcomes in type 2diabetes. N Engl J Med. 2015;373:232–42.

50. Holman RR, Bethel MA, Mentz RJ, et al. Effects ofonce-weekly exenatide on cardiovascular outcomesin type 2 diabetes. N Engl J Med. 2017;377:1228–39.

51. Zinman B, Wanner C, Lachin JM, et al. Empagli-flozin, cardiovascular outcomes, and mortality intype 2 diabetes. N Engl J Med. 2015;373:2117–28.

52. Neal B, Perkovic V, Mahaffey KW, et al. Canagli-flozin and cardiovascular and renal events in type 2diabetes. N Engl J Med. 2017;377(7):644–57.

53. Kosiborod M, Cavender MA, Fu AZ, et al. Lower riskof heart failure and death in patients initiated onsodium-glucose cotransporter-2 inhibitors versusother glucose-lowering drugs: the CVD-REAL study(comparative effectiveness of cardiovascular out-comes in new users of sodium-glucose cotransporter-2 inhibitors). Circulation. 2017;136(3):249–59.

54. STROBE. STROBE Statement: Strengthening thereporting of observational studies in epidemiology.https://www.strobe-statement.org/index.php?id=available-checklists. Accessed 26 Sep 2018.

55. Zwarenstein M, Treweek S, Gagnier JJ, et al. Prag-matic Trials in Healthcare (Practihc) group. Improv-ing the reporting of pragmatic trials: an extension ofthe CONSORT statement. BMJ. 2008;337:a2390.

56. Pitt B, Zannad F, Remme WJ, et al. The effect ofspironolactone on morbidity and mortality inpatients with severe heart failure. N Engl J Med.1999;341(10):709–17.

57. Freemantle N, Marston L, Walters K, et al. Makinginferences on treatment effects from real worlddata: propensity scores, confounding by indication,and other perils for the unwary in observationalresearch. BMJ. 2013;347:f6409.

58. Penning de Vries BBL, Groenwold RHH. Cautionarynote: propensity score matching does not accountfor bias due to censoring. Nephrol Dial Transplant.2017;1–3.

59. Zhang R, Wang Y, Liu B, et al. Clinical data qualityproblems and countermeasure for real world study.Front Med. 2014;8:352–57.

60. Chen JH, Asch SM. Machine learning and predic-tion in medicine—beyond the peak of inflatedexpectations. N Engl J Med. 2017;376(26):2507.

1774 Adv Ther (2018) 35:1763–1774