Adaptive Design of Confirmatory Trials

Advances and Challenges

Tze Leung Laialowast Philip W Lavorib Ka Wai Tsangc

aDepartment of Statistics Stanford University Stanford CA USAbDepartment of Health Research and Policy Stanford University Stanford CA USA

cInstitute for Computational and Mathematical Engineering Stanford UniversityStanford CA USA

Abstract

The past decade witnessed major developments in innovative designs of con-firmatory clinical trials and adaptive designs represent the most active areaof these developments We give an overview of the developments and associ-ated statistical methods in several classes of adaptive designs of confirmatorytrials We also discuss their statistical difficulties and implementation chal-lenges and show how these problems are connected to other branches ofmainstream Statistics which we then apply to resolve the difficulties andbypass the bottlenecks in the development of adaptive designs for the nextdecade

Keywords Adaptive design Adaptive randomization Bayesian inferenceEarly stopping Hybrid resampling Multi-arm bandits

1 Introduction

Because of the lack of information on both the magnitude and the sam-pling variability of the treatment effect of a new treatment at the designstage there has been increasing interest from the biopharmaceutical indus-try in adaptive designs that can adapt to the information collected duringthe course of the trial Beginning with Bauer [1] who introduced sequential

lowastCorresponding author at Department of Statistics Sequoia Hall 390 Serra Mall Stan-ford CA 94305-4065 USA Tel +1 6507232622Email address laitstanfordedu (Tze Leung Lai)

Preprint submitted to Contemporary Clinical Trials May 5 2015

adaptive test strategies over a planned series of separate trials and Wittesand Brittain [2] who considered internal pilot studies a large literature hasgrown on adaptive design of clinical trials In Section 2 we review several di-rections of development and basic methodologies in that literature Despitethe vibrant research activities and the attractiveness of adaptive designs thatprovide a promising alternative to and major advance over standard clini-cal trial designs which are handicapped by insufficient information at theplanning stage these adaptive designs are fraught with statistical and imple-mentation difficulties which have been impediments to their widespread useSection 3 discusses these difficulties and reviews in this connection relatedaspects of the FDA Draft Guidance for Industry on Adaptive Design fordrugs and biologics in 2010

In Section 4 we describe some new advances in adaptive designs to addressthese difficulties and to respond to certain issues raised by the FDA DraftGuidance We also use an adaptive clinical trial currently being plannedat the Stanford Stroke Center to illustrate the new methodologies and theirimplementation Section 5 gives some concluding remarks and further dis-cussion of the challenges and opportunities of adaptive designs for Phase IIIclinical trials in drug development

2 Adaptive designs Overview of methods and developments

In this section we give an overview of the developments of adaptive designof clinical trials together with the associated statistical methods that havebeen used or introduced The overview is divided into two parts the firstof which is on frequentist methods reviewed in Sections 21 and 22 Thesecond part is on Bayesian adaptive designs which are reviewed in Section23 and which are arguably the most active area of clinical trial innovationsfor testing cancer treatments

21 Sample size re-estimation

In standard clinical trial designs the sample size is determined by thepower at a given alternative but in practice it is often difficult for investi-gators to specify a realistic alternative at which sample size determinationcan be based Although a standard method to address this difficulty is tocarry out a preliminary pilot study the results from a small pilot study maybe difficult to interpret and apply as pointed out by Wittes and Brittain[2] who proposed to treat the first stage of a two-stage clinical trial as an

2

internal pilot from which the overall sample size can be re-estimated Thespecific problem considered by [2] as an example of internal pilots actuallydated back to Steinrsquos two-stage procedure [3] introduced in 1945 for test-ing hypothesis H0 microX = microY versus the two-sided alternative microX 6= microYfor the means of two independent normal distributions with common un-known variance and based on iid observations X1 X2 middot middot middot sim N(microX σ

2)and Y1 Y2 middot middot middot sim N(microX σ

2) Let tνα denote the upper α-quantile of the t-distribution with ν degrees of freedom In its first stage Steinrsquos proceduresamples n0 observations from each of the two normal distributions and com-putes the usual unbiased estimate s20 of σ2 In the second stage it samplesup to

n1 = n0 or[(t2n0minus2α2 + t2n0minus2β

)2 2s20δ2

](1)

observations from each population where α is the prescribed type I errorprobability and 1 minus β is the prescribed power at the alternatives satisfying|microX minus microY | = δ The null hypothesis H0 microx = microY is then rejected if |Xn1 minusYn1 | gt t2n0minus2α2

radic2s20n1 Steinrsquos two-stage procedure is modified in [2 4]

as follows Viewing |Xn1 minus Yn1|radic

2s21n1 as a fixed-sample test statisticbased on a sample of size n1 from each population the test statistic has thenon-central t-distribution with 2n1minus2 degrees of freedom and non-centralityparameter δ

radicn1(2s21) at the alternative microX minus microY = δ Fixing α β and δ

let n(σ2) denote the smallest n1 for which the probability exceeds 1minus β thatan observation from this distribution exceeds the critical value t2n1minus2α2 Anestimate of the total desired sample size based on a pre-trial estimate σ2

0

of σ2 is n(σ20) Following a pilot study of size n0 per arm which results in

the variance estimate s20 the total sample size can be re-estimated as n(s20)At this point there are many options for how to proceed In particular [2]recommends taking the maximum of n(σ2

0) and n(s20) as the new total samplesize while [4] recommends retaining n(σ2

0) unless n(s20) is substantially largerThe aforementioned papers and subsequent refinements [5 6 7] represent

the ldquofirst generationrdquo of adaptive designs The second-generation adaptivedesigns adopt a more aggressive viewpoint of re-estimating the sample sizefrom the estimate of δ (instead of the nuisance parameter σ) based on thefirst-stage data starting with Fisher [8] for the case of normally distributedoutcome variables with known common variance σ2 which can be assumed toequal 12 without loss of generality If n is the original sample size per treat-ment then after rn pairs of observations (0 lt r lt 1) nminus12S1 sim N (rδ

radicn r)

3

where S1 =sumrn

i=1(Xi minus Yi) If it is now desired to change the second-stage sample size from (1 minus r)n to γ(1 minus r)n for some γ gt 0 then con-ditional on the first-stage data (nγ)minus12S2 sim N

((1minus r)δradicγn 1minus r

) where

S2 =sumnlowast

i=rn+1(Ximinus Yi) and nlowast = rn+ γ(1minus r)n is the new total sample size

per treatment Note that under H0 δ = 0 (nγ)minus12S2 has the N(0 1 minus r)distribution regardless of the (data-dependent) choice of γ thus Fisherrsquos teststatistic

nminus12(S1 + γminus12S2

)(2)

has a N(0 1) distribution under H0 The corresponding test has been calleda variance spending test because 1 minus r is the remaining part of the totalvariance 1 not spent in the first stage Denne [9] proposed a test thatalso allows data-dependent updates of the total sample size but maintainsthe type I error probability by a seemingly different method Dennersquos testchooses a critical value for S2 that maintains the conditional type I errorrate Pδ=0(S1 + S2 gt zα

radicn | S1 = s1) Jennison and Turnbull [10] showed

that this test is actually equivalent to Fisherrsquos test which they found toperform poorly in terms of expected sample size and power in comparisonto group-sequential tests Tsiatis and Mehta [11] independently came to thesame conclusion attributing this inefficiency to the use of the non-sufficientldquoweightedrdquo statistic (2)

Working in terms of the z-statistic that divides a sample sum by its stan-dard deviation Proschan and Hunsberger [12] noted that any non-decreasingfunction C(z1) with range [0 1] can be used as a conditional type I error func-tion to define a two-stage procedure as long as it satisfiesint infin

minusinfinC(z1)φ(z1) dz1 = α (3)

and suggested certain choices of C(middot) Having observed the first-stage dataZ1 H0 δ = 0 is rejected in favor of δ gt 0 after the second stage if Z2 gtΦminus1(1 minus C(z1)) Condition (3) ensures that the type I error probability ofany test of this form is α The tests proposed earlier by Bauer and Kohne[13] can be represented in this framework as noted by Posch and Bauer [14]The basic idea underlying these representations dated back to Bauer [1] whoused it to develop sequential adaptive test strategies over a planned series ofseparate trials

Assuming normally distributed outcomes with known variances Jennisonand Turnbull [15] introduced adaptive group sequential tests that choose the

4

jth group size and stopping boundary on the basis of the cumulative samplesize njminus1 and the sample sum Snjminus1

over the first j minus 1 groups and thatare optimal in the sense of minimizing a weighted average of the expectedsample sizes over a collection of parameter values subject to prescribed errorprobabilities at the null and a given alternative hypothesis They showed howthe corresponding optimization problem can be solved numerically by usingbackward induction algorithms They also showed in [16] that standard (non-adaptive) group sequential tests with the first stage chosen approximately arenearly as efficient as their optimal adaptive tests

A new approach was developed by Bartroff and Lai [17 18] in the generalframework of multiparameter exponential families It uses efficient general-ized likelihood ratio (GLR) statistics in this framework and adds a third stageto adjust for the sampling variability of the first-stage parameter estimatesthat determine the second-stage sample size The possibility of adding athird stage to improve two-stage designs dated back to Lorden [19] WhereasLorden used crude upper bounds for the type I error probability that are tooconservative for practical applications Bartroff and Lai overcame this diffi-culty by using new methods to compute the type I error probability and alsoextended the three-stage test to multiparameter and multi-armed settingsthus greatly broadening the scope of these efficient adaptive designs

22 Seamless Phase IIIII trials with hypotheses selection at interim

Bretz and his collaborators [20 21] at Novartis have extended Bauerrsquosseminal ideas in [1] to develop a second generation of adaptive designs that areof much greater interest to drug development than sample size re-estimationHighlighting the need for more efficient and effective drug development pro-cesses to translate the ongoing revolution in biomedical sciences to break-throughs in treating diseases [20] notes the inefficiency of contemporaryPhase III trials that are ldquostand-alone confirmatory trials ignoring informa-tion from previous phasesrdquo and argues for innovation through seamless PhaseIIIII designs that ldquoaim at interweaving these (phases) by combining theminto one single study conducted in two stagesrdquo The advantages of theseadaptive seamless designs (ASDs) noted in [20 p 624] are that they

(i) reduce the time to decide on plan and implement the next phase

(ii) save costs through the combination of evidence across two studies and

(iii) get long-term safety data earlier as a direct consequence of followingup the Phase II patients

5

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

adaptive test strategies over a planned series of separate trials and Wittesand Brittain [2] who considered internal pilot studies a large literature hasgrown on adaptive design of clinical trials In Section 2 we review several di-rections of development and basic methodologies in that literature Despitethe vibrant research activities and the attractiveness of adaptive designs thatprovide a promising alternative to and major advance over standard clini-cal trial designs which are handicapped by insufficient information at theplanning stage these adaptive designs are fraught with statistical and imple-mentation difficulties which have been impediments to their widespread useSection 3 discusses these difficulties and reviews in this connection relatedaspects of the FDA Draft Guidance for Industry on Adaptive Design fordrugs and biologics in 2010

In Section 4 we describe some new advances in adaptive designs to addressthese difficulties and to respond to certain issues raised by the FDA DraftGuidance We also use an adaptive clinical trial currently being plannedat the Stanford Stroke Center to illustrate the new methodologies and theirimplementation Section 5 gives some concluding remarks and further dis-cussion of the challenges and opportunities of adaptive designs for Phase IIIclinical trials in drug development

2 Adaptive designs Overview of methods and developments

In this section we give an overview of the developments of adaptive designof clinical trials together with the associated statistical methods that havebeen used or introduced The overview is divided into two parts the firstof which is on frequentist methods reviewed in Sections 21 and 22 Thesecond part is on Bayesian adaptive designs which are reviewed in Section23 and which are arguably the most active area of clinical trial innovationsfor testing cancer treatments

21 Sample size re-estimation

In standard clinical trial designs the sample size is determined by thepower at a given alternative but in practice it is often difficult for investi-gators to specify a realistic alternative at which sample size determinationcan be based Although a standard method to address this difficulty is tocarry out a preliminary pilot study the results from a small pilot study maybe difficult to interpret and apply as pointed out by Wittes and Brittain[2] who proposed to treat the first stage of a two-stage clinical trial as an

2

internal pilot from which the overall sample size can be re-estimated Thespecific problem considered by [2] as an example of internal pilots actuallydated back to Steinrsquos two-stage procedure [3] introduced in 1945 for test-ing hypothesis H0 microX = microY versus the two-sided alternative microX 6= microYfor the means of two independent normal distributions with common un-known variance and based on iid observations X1 X2 middot middot middot sim N(microX σ

2)and Y1 Y2 middot middot middot sim N(microX σ

2) Let tνα denote the upper α-quantile of the t-distribution with ν degrees of freedom In its first stage Steinrsquos proceduresamples n0 observations from each of the two normal distributions and com-putes the usual unbiased estimate s20 of σ2 In the second stage it samplesup to

n1 = n0 or[(t2n0minus2α2 + t2n0minus2β

)2 2s20δ2

](1)

observations from each population where α is the prescribed type I errorprobability and 1 minus β is the prescribed power at the alternatives satisfying|microX minus microY | = δ The null hypothesis H0 microx = microY is then rejected if |Xn1 minusYn1 | gt t2n0minus2α2

radic2s20n1 Steinrsquos two-stage procedure is modified in [2 4]

as follows Viewing |Xn1 minus Yn1|radic

2s21n1 as a fixed-sample test statisticbased on a sample of size n1 from each population the test statistic has thenon-central t-distribution with 2n1minus2 degrees of freedom and non-centralityparameter δ

radicn1(2s21) at the alternative microX minus microY = δ Fixing α β and δ

let n(σ2) denote the smallest n1 for which the probability exceeds 1minus β thatan observation from this distribution exceeds the critical value t2n1minus2α2 Anestimate of the total desired sample size based on a pre-trial estimate σ2

0

of σ2 is n(σ20) Following a pilot study of size n0 per arm which results in

the variance estimate s20 the total sample size can be re-estimated as n(s20)At this point there are many options for how to proceed In particular [2]recommends taking the maximum of n(σ2

0) and n(s20) as the new total samplesize while [4] recommends retaining n(σ2

0) unless n(s20) is substantially largerThe aforementioned papers and subsequent refinements [5 6 7] represent

the ldquofirst generationrdquo of adaptive designs The second-generation adaptivedesigns adopt a more aggressive viewpoint of re-estimating the sample sizefrom the estimate of δ (instead of the nuisance parameter σ) based on thefirst-stage data starting with Fisher [8] for the case of normally distributedoutcome variables with known common variance σ2 which can be assumed toequal 12 without loss of generality If n is the original sample size per treat-ment then after rn pairs of observations (0 lt r lt 1) nminus12S1 sim N (rδ

radicn r)

3

where S1 =sumrn

i=1(Xi minus Yi) If it is now desired to change the second-stage sample size from (1 minus r)n to γ(1 minus r)n for some γ gt 0 then con-ditional on the first-stage data (nγ)minus12S2 sim N

((1minus r)δradicγn 1minus r

) where

S2 =sumnlowast

i=rn+1(Ximinus Yi) and nlowast = rn+ γ(1minus r)n is the new total sample size

per treatment Note that under H0 δ = 0 (nγ)minus12S2 has the N(0 1 minus r)distribution regardless of the (data-dependent) choice of γ thus Fisherrsquos teststatistic

nminus12(S1 + γminus12S2

)(2)

has a N(0 1) distribution under H0 The corresponding test has been calleda variance spending test because 1 minus r is the remaining part of the totalvariance 1 not spent in the first stage Denne [9] proposed a test thatalso allows data-dependent updates of the total sample size but maintainsthe type I error probability by a seemingly different method Dennersquos testchooses a critical value for S2 that maintains the conditional type I errorrate Pδ=0(S1 + S2 gt zα

radicn | S1 = s1) Jennison and Turnbull [10] showed

that this test is actually equivalent to Fisherrsquos test which they found toperform poorly in terms of expected sample size and power in comparisonto group-sequential tests Tsiatis and Mehta [11] independently came to thesame conclusion attributing this inefficiency to the use of the non-sufficientldquoweightedrdquo statistic (2)

Working in terms of the z-statistic that divides a sample sum by its stan-dard deviation Proschan and Hunsberger [12] noted that any non-decreasingfunction C(z1) with range [0 1] can be used as a conditional type I error func-tion to define a two-stage procedure as long as it satisfiesint infin

minusinfinC(z1)φ(z1) dz1 = α (3)

and suggested certain choices of C(middot) Having observed the first-stage dataZ1 H0 δ = 0 is rejected in favor of δ gt 0 after the second stage if Z2 gtΦminus1(1 minus C(z1)) Condition (3) ensures that the type I error probability ofany test of this form is α The tests proposed earlier by Bauer and Kohne[13] can be represented in this framework as noted by Posch and Bauer [14]The basic idea underlying these representations dated back to Bauer [1] whoused it to develop sequential adaptive test strategies over a planned series ofseparate trials

Assuming normally distributed outcomes with known variances Jennisonand Turnbull [15] introduced adaptive group sequential tests that choose the

4

jth group size and stopping boundary on the basis of the cumulative samplesize njminus1 and the sample sum Snjminus1

over the first j minus 1 groups and thatare optimal in the sense of minimizing a weighted average of the expectedsample sizes over a collection of parameter values subject to prescribed errorprobabilities at the null and a given alternative hypothesis They showed howthe corresponding optimization problem can be solved numerically by usingbackward induction algorithms They also showed in [16] that standard (non-adaptive) group sequential tests with the first stage chosen approximately arenearly as efficient as their optimal adaptive tests

A new approach was developed by Bartroff and Lai [17 18] in the generalframework of multiparameter exponential families It uses efficient general-ized likelihood ratio (GLR) statistics in this framework and adds a third stageto adjust for the sampling variability of the first-stage parameter estimatesthat determine the second-stage sample size The possibility of adding athird stage to improve two-stage designs dated back to Lorden [19] WhereasLorden used crude upper bounds for the type I error probability that are tooconservative for practical applications Bartroff and Lai overcame this diffi-culty by using new methods to compute the type I error probability and alsoextended the three-stage test to multiparameter and multi-armed settingsthus greatly broadening the scope of these efficient adaptive designs

22 Seamless Phase IIIII trials with hypotheses selection at interim

Bretz and his collaborators [20 21] at Novartis have extended Bauerrsquosseminal ideas in [1] to develop a second generation of adaptive designs that areof much greater interest to drug development than sample size re-estimationHighlighting the need for more efficient and effective drug development pro-cesses to translate the ongoing revolution in biomedical sciences to break-throughs in treating diseases [20] notes the inefficiency of contemporaryPhase III trials that are ldquostand-alone confirmatory trials ignoring informa-tion from previous phasesrdquo and argues for innovation through seamless PhaseIIIII designs that ldquoaim at interweaving these (phases) by combining theminto one single study conducted in two stagesrdquo The advantages of theseadaptive seamless designs (ASDs) noted in [20 p 624] are that they

(i) reduce the time to decide on plan and implement the next phase

(ii) save costs through the combination of evidence across two studies and

(iii) get long-term safety data earlier as a direct consequence of followingup the Phase II patients

5

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

internal pilot from which the overall sample size can be re-estimated Thespecific problem considered by [2] as an example of internal pilots actuallydated back to Steinrsquos two-stage procedure [3] introduced in 1945 for test-ing hypothesis H0 microX = microY versus the two-sided alternative microX 6= microYfor the means of two independent normal distributions with common un-known variance and based on iid observations X1 X2 middot middot middot sim N(microX σ

2)and Y1 Y2 middot middot middot sim N(microX σ

2) Let tνα denote the upper α-quantile of the t-distribution with ν degrees of freedom In its first stage Steinrsquos proceduresamples n0 observations from each of the two normal distributions and com-putes the usual unbiased estimate s20 of σ2 In the second stage it samplesup to

n1 = n0 or[(t2n0minus2α2 + t2n0minus2β

)2 2s20δ2

](1)

observations from each population where α is the prescribed type I errorprobability and 1 minus β is the prescribed power at the alternatives satisfying|microX minus microY | = δ The null hypothesis H0 microx = microY is then rejected if |Xn1 minusYn1 | gt t2n0minus2α2

radic2s20n1 Steinrsquos two-stage procedure is modified in [2 4]

as follows Viewing |Xn1 minus Yn1|radic

2s21n1 as a fixed-sample test statisticbased on a sample of size n1 from each population the test statistic has thenon-central t-distribution with 2n1minus2 degrees of freedom and non-centralityparameter δ

radicn1(2s21) at the alternative microX minus microY = δ Fixing α β and δ

let n(σ2) denote the smallest n1 for which the probability exceeds 1minus β thatan observation from this distribution exceeds the critical value t2n1minus2α2 Anestimate of the total desired sample size based on a pre-trial estimate σ2

0

of σ2 is n(σ20) Following a pilot study of size n0 per arm which results in

the variance estimate s20 the total sample size can be re-estimated as n(s20)At this point there are many options for how to proceed In particular [2]recommends taking the maximum of n(σ2

0) and n(s20) as the new total samplesize while [4] recommends retaining n(σ2

0) unless n(s20) is substantially largerThe aforementioned papers and subsequent refinements [5 6 7] represent

the ldquofirst generationrdquo of adaptive designs The second-generation adaptivedesigns adopt a more aggressive viewpoint of re-estimating the sample sizefrom the estimate of δ (instead of the nuisance parameter σ) based on thefirst-stage data starting with Fisher [8] for the case of normally distributedoutcome variables with known common variance σ2 which can be assumed toequal 12 without loss of generality If n is the original sample size per treat-ment then after rn pairs of observations (0 lt r lt 1) nminus12S1 sim N (rδ

radicn r)

3

where S1 =sumrn

i=1(Xi minus Yi) If it is now desired to change the second-stage sample size from (1 minus r)n to γ(1 minus r)n for some γ gt 0 then con-ditional on the first-stage data (nγ)minus12S2 sim N

((1minus r)δradicγn 1minus r

) where

S2 =sumnlowast

i=rn+1(Ximinus Yi) and nlowast = rn+ γ(1minus r)n is the new total sample size

per treatment Note that under H0 δ = 0 (nγ)minus12S2 has the N(0 1 minus r)distribution regardless of the (data-dependent) choice of γ thus Fisherrsquos teststatistic

nminus12(S1 + γminus12S2

)(2)

has a N(0 1) distribution under H0 The corresponding test has been calleda variance spending test because 1 minus r is the remaining part of the totalvariance 1 not spent in the first stage Denne [9] proposed a test thatalso allows data-dependent updates of the total sample size but maintainsthe type I error probability by a seemingly different method Dennersquos testchooses a critical value for S2 that maintains the conditional type I errorrate Pδ=0(S1 + S2 gt zα

radicn | S1 = s1) Jennison and Turnbull [10] showed

that this test is actually equivalent to Fisherrsquos test which they found toperform poorly in terms of expected sample size and power in comparisonto group-sequential tests Tsiatis and Mehta [11] independently came to thesame conclusion attributing this inefficiency to the use of the non-sufficientldquoweightedrdquo statistic (2)

Working in terms of the z-statistic that divides a sample sum by its stan-dard deviation Proschan and Hunsberger [12] noted that any non-decreasingfunction C(z1) with range [0 1] can be used as a conditional type I error func-tion to define a two-stage procedure as long as it satisfiesint infin

minusinfinC(z1)φ(z1) dz1 = α (3)

and suggested certain choices of C(middot) Having observed the first-stage dataZ1 H0 δ = 0 is rejected in favor of δ gt 0 after the second stage if Z2 gtΦminus1(1 minus C(z1)) Condition (3) ensures that the type I error probability ofany test of this form is α The tests proposed earlier by Bauer and Kohne[13] can be represented in this framework as noted by Posch and Bauer [14]The basic idea underlying these representations dated back to Bauer [1] whoused it to develop sequential adaptive test strategies over a planned series ofseparate trials

Assuming normally distributed outcomes with known variances Jennisonand Turnbull [15] introduced adaptive group sequential tests that choose the

4

jth group size and stopping boundary on the basis of the cumulative samplesize njminus1 and the sample sum Snjminus1

over the first j minus 1 groups and thatare optimal in the sense of minimizing a weighted average of the expectedsample sizes over a collection of parameter values subject to prescribed errorprobabilities at the null and a given alternative hypothesis They showed howthe corresponding optimization problem can be solved numerically by usingbackward induction algorithms They also showed in [16] that standard (non-adaptive) group sequential tests with the first stage chosen approximately arenearly as efficient as their optimal adaptive tests

A new approach was developed by Bartroff and Lai [17 18] in the generalframework of multiparameter exponential families It uses efficient general-ized likelihood ratio (GLR) statistics in this framework and adds a third stageto adjust for the sampling variability of the first-stage parameter estimatesthat determine the second-stage sample size The possibility of adding athird stage to improve two-stage designs dated back to Lorden [19] WhereasLorden used crude upper bounds for the type I error probability that are tooconservative for practical applications Bartroff and Lai overcame this diffi-culty by using new methods to compute the type I error probability and alsoextended the three-stage test to multiparameter and multi-armed settingsthus greatly broadening the scope of these efficient adaptive designs

22 Seamless Phase IIIII trials with hypotheses selection at interim

Bretz and his collaborators [20 21] at Novartis have extended Bauerrsquosseminal ideas in [1] to develop a second generation of adaptive designs that areof much greater interest to drug development than sample size re-estimationHighlighting the need for more efficient and effective drug development pro-cesses to translate the ongoing revolution in biomedical sciences to break-throughs in treating diseases [20] notes the inefficiency of contemporaryPhase III trials that are ldquostand-alone confirmatory trials ignoring informa-tion from previous phasesrdquo and argues for innovation through seamless PhaseIIIII designs that ldquoaim at interweaving these (phases) by combining theminto one single study conducted in two stagesrdquo The advantages of theseadaptive seamless designs (ASDs) noted in [20 p 624] are that they

(i) reduce the time to decide on plan and implement the next phase

(ii) save costs through the combination of evidence across two studies and

(iii) get long-term safety data earlier as a direct consequence of followingup the Phase II patients

5

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

where S1 =sumrn

i=1(Xi minus Yi) If it is now desired to change the second-stage sample size from (1 minus r)n to γ(1 minus r)n for some γ gt 0 then con-ditional on the first-stage data (nγ)minus12S2 sim N

((1minus r)δradicγn 1minus r

) where

S2 =sumnlowast

i=rn+1(Ximinus Yi) and nlowast = rn+ γ(1minus r)n is the new total sample size

per treatment Note that under H0 δ = 0 (nγ)minus12S2 has the N(0 1 minus r)distribution regardless of the (data-dependent) choice of γ thus Fisherrsquos teststatistic

nminus12(S1 + γminus12S2

)(2)

has a N(0 1) distribution under H0 The corresponding test has been calleda variance spending test because 1 minus r is the remaining part of the totalvariance 1 not spent in the first stage Denne [9] proposed a test thatalso allows data-dependent updates of the total sample size but maintainsthe type I error probability by a seemingly different method Dennersquos testchooses a critical value for S2 that maintains the conditional type I errorrate Pδ=0(S1 + S2 gt zα

radicn | S1 = s1) Jennison and Turnbull [10] showed

that this test is actually equivalent to Fisherrsquos test which they found toperform poorly in terms of expected sample size and power in comparisonto group-sequential tests Tsiatis and Mehta [11] independently came to thesame conclusion attributing this inefficiency to the use of the non-sufficientldquoweightedrdquo statistic (2)

Working in terms of the z-statistic that divides a sample sum by its stan-dard deviation Proschan and Hunsberger [12] noted that any non-decreasingfunction C(z1) with range [0 1] can be used as a conditional type I error func-tion to define a two-stage procedure as long as it satisfiesint infin

minusinfinC(z1)φ(z1) dz1 = α (3)

and suggested certain choices of C(middot) Having observed the first-stage dataZ1 H0 δ = 0 is rejected in favor of δ gt 0 after the second stage if Z2 gtΦminus1(1 minus C(z1)) Condition (3) ensures that the type I error probability ofany test of this form is α The tests proposed earlier by Bauer and Kohne[13] can be represented in this framework as noted by Posch and Bauer [14]The basic idea underlying these representations dated back to Bauer [1] whoused it to develop sequential adaptive test strategies over a planned series ofseparate trials

Assuming normally distributed outcomes with known variances Jennisonand Turnbull [15] introduced adaptive group sequential tests that choose the

4

jth group size and stopping boundary on the basis of the cumulative samplesize njminus1 and the sample sum Snjminus1

over the first j minus 1 groups and thatare optimal in the sense of minimizing a weighted average of the expectedsample sizes over a collection of parameter values subject to prescribed errorprobabilities at the null and a given alternative hypothesis They showed howthe corresponding optimization problem can be solved numerically by usingbackward induction algorithms They also showed in [16] that standard (non-adaptive) group sequential tests with the first stage chosen approximately arenearly as efficient as their optimal adaptive tests

A new approach was developed by Bartroff and Lai [17 18] in the generalframework of multiparameter exponential families It uses efficient general-ized likelihood ratio (GLR) statistics in this framework and adds a third stageto adjust for the sampling variability of the first-stage parameter estimatesthat determine the second-stage sample size The possibility of adding athird stage to improve two-stage designs dated back to Lorden [19] WhereasLorden used crude upper bounds for the type I error probability that are tooconservative for practical applications Bartroff and Lai overcame this diffi-culty by using new methods to compute the type I error probability and alsoextended the three-stage test to multiparameter and multi-armed settingsthus greatly broadening the scope of these efficient adaptive designs

22 Seamless Phase IIIII trials with hypotheses selection at interim

Bretz and his collaborators [20 21] at Novartis have extended Bauerrsquosseminal ideas in [1] to develop a second generation of adaptive designs that areof much greater interest to drug development than sample size re-estimationHighlighting the need for more efficient and effective drug development pro-cesses to translate the ongoing revolution in biomedical sciences to break-throughs in treating diseases [20] notes the inefficiency of contemporaryPhase III trials that are ldquostand-alone confirmatory trials ignoring informa-tion from previous phasesrdquo and argues for innovation through seamless PhaseIIIII designs that ldquoaim at interweaving these (phases) by combining theminto one single study conducted in two stagesrdquo The advantages of theseadaptive seamless designs (ASDs) noted in [20 p 624] are that they

(i) reduce the time to decide on plan and implement the next phase

(ii) save costs through the combination of evidence across two studies and

(iii) get long-term safety data earlier as a direct consequence of followingup the Phase II patients

5

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

jth group size and stopping boundary on the basis of the cumulative samplesize njminus1 and the sample sum Snjminus1

over the first j minus 1 groups and thatare optimal in the sense of minimizing a weighted average of the expectedsample sizes over a collection of parameter values subject to prescribed errorprobabilities at the null and a given alternative hypothesis They showed howthe corresponding optimization problem can be solved numerically by usingbackward induction algorithms They also showed in [16] that standard (non-adaptive) group sequential tests with the first stage chosen approximately arenearly as efficient as their optimal adaptive tests

A new approach was developed by Bartroff and Lai [17 18] in the generalframework of multiparameter exponential families It uses efficient general-ized likelihood ratio (GLR) statistics in this framework and adds a third stageto adjust for the sampling variability of the first-stage parameter estimatesthat determine the second-stage sample size The possibility of adding athird stage to improve two-stage designs dated back to Lorden [19] WhereasLorden used crude upper bounds for the type I error probability that are tooconservative for practical applications Bartroff and Lai overcame this diffi-culty by using new methods to compute the type I error probability and alsoextended the three-stage test to multiparameter and multi-armed settingsthus greatly broadening the scope of these efficient adaptive designs

22 Seamless Phase IIIII trials with hypotheses selection at interim

Bretz and his collaborators [20 21] at Novartis have extended Bauerrsquosseminal ideas in [1] to develop a second generation of adaptive designs that areof much greater interest to drug development than sample size re-estimationHighlighting the need for more efficient and effective drug development pro-cesses to translate the ongoing revolution in biomedical sciences to break-throughs in treating diseases [20] notes the inefficiency of contemporaryPhase III trials that are ldquostand-alone confirmatory trials ignoring informa-tion from previous phasesrdquo and argues for innovation through seamless PhaseIIIII designs that ldquoaim at interweaving these (phases) by combining theminto one single study conducted in two stagesrdquo The advantages of theseadaptive seamless designs (ASDs) noted in [20 p 624] are that they

(i) reduce the time to decide on plan and implement the next phase

(ii) save costs through the combination of evidence across two studies and

(iii) get long-term safety data earlier as a direct consequence of followingup the Phase II patients

5

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

The basic idea underlying the ASDs in [20] is to extend to multiple testingof k hypotheses H1

0 Hk0 the methods used in [1] [13] and [14] for combin-

ing the p-values over the two stages in a two-stage procedure of a directionalnull hypothesis on treatment effects Here k represents the number of doselevels or treatment regimens of a new drug considered in a typical Phase IItrial At the interim analysis after the first stage (which corresponds to thePhase II component of the ASD) only one dose level (or treatment regimen)say the jth one is selected for continuation in the second stage (correspond-ing to the Phase III component) of the ASD Bretz et al [20] apply theclosed testing principle to all intersection hypotheses involving Hj

0 using theSimes method [22] to define for each intersection hypothesis the adjusted firststage p-values of the hypotheses in the intersection and thereby keeping thefamily-wise error rate (FWER) controlled at a prespecified level Althoughit controls the FWER this way of combining the first- and second-stage datais very inefficient as pointed out in [10 11] More efficient two-stage designshave been introduced by Stallard and Todd [23] and Wang et al [24] for thecase of normally distributed outcome variables with known variances andnearly optimal ASDs have recently been developed by using deeper conceptsand more powerful techniques that are described in Section 41 The pro-cedures proposed in [20 21] however are widely regarded as being moreflexible than those to [23 24] and easier to extend to more complicated set-tings In fact Brannath et al [25] have extended them to survival outcomesand Jenkins Stone and Jennison [26] provide a further extension by allowingan intermediate endpoint (such as progression-free survival) to be used toguide hypothesis selection at the end of the first stage while using overallsurvival as the primary endpoint for the second stage Because of the com-plexity of the problem [26] does not discuss the inefficiency of combining thep-values from the two separate stages even though one of its authors has ad-vocated to use more efficient group sequential test statistics for considerablysimpler testing problem in [10 15]

23 Bayesian approach

Adding confusion to the debate between the ldquoefficiency camprdquo repre-sented by [10 11 15] on efficient designs under restrictive assumptions thatinvolve sufficient statistics and nearly optimal stopping rules and the ldquoflex-ibility camprdquo that focuses on combining information in a flexible way fromdifferent stages of the trial to tackle complicated settings is the ldquoBayesiancamprdquo that purportedly has both efficiency and flexibility Since Bayesian

6

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

inference is based on the posterior distribution it does not need adjustmentsfor learning and adaptation during the course of the trial Acknowledgingthat the statistical benchmark to gain regulatory approval of a new treat-ment is to have a statistically significant result in the frequentist sense ata specified Type I error Bayesian adaptive designs for Phase III trials relyon Monte Carlo simulations under a chosen parameter configuration belong-ing to the null hypothesis but there is no guarantee that the Type I error ismaintained by this approach at other parameter configurations for a compos-ite hypothesis Another regulatory issue with Bayesian designs is the choiceof the prior distribution which may be difficult to justify

Despite the regulatory difficulties with the Bayesian approach it is ar-guably the most active area of development in adaptive design of clinicaltrials Berry [27] and Berry et al [33] point out the flexibility and natu-ral appeal of applying the Bayesian approach to midcourse adaptation ina trial such as dropping an unfavorable arm of the new treatment beingtested or modifying the randomization scheme incorporation of historicaland other related information treatment of multiple endpoints and multiplesub-groups missing data and deviations from the original study plan SinceBayesian inference can be updated continually as data accumulate and isnot tied to the design chosen early stopping for safety and futility or effi-cacy are allowed Besides early stopping based on the posterior probabilityof an efficacious outcome Bayesian adaptive designs also use the posteriorprobabilities to determine randomization proportions to improve the out-comes of patients accrued to a trial This idea dated back eight decadesago to Thompson [29] who proposed to randomize patients to one of twotreatments with probability equal to the current posterior probability thatit is the better treatment He was motivated by the ethical consideration ofexposing a minimal expected number of patients in the trial to the inferiortreatment Meuer Lewis and Berry [30] point out how this outcome-adaptiverandomization can be used to close the ldquotherapeutic misconceptionrdquo gap forrecruiting patients to a trial which is particularly important for ldquotrials intime-sensitive conditions that require rapid decision making by patients orsurrogates and by physiciansrdquo

Some trial participants and family members believe that the goalof a clinical trial is to improve their outcomes mdash a misconceptionoften reinforced by media advertising of clinical research Clin-ical trials have primarily scientific aims and rarely attempt to

7

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

collectively improve the outcomes of their participants Anybenefit to an individual trial participant is a chance effect of ran-domization and the true but unknown relative effects of treat-ments Thus even though serving as a research participant isessentially an altruistic activity many clinical trial volunteers donot participate in research out of altruism An adaptive clini-cal trial design can be used to increase the likelihood that studyparticipants will benefit by being in a clinical trial

Adaptive randomization based on posterior probabilities features promi-nently in many Bayesian adaptive oncology trials particularly those at theMD Anderson Cancer Center One such trial is the BATTLE (Biomarker-integrated Approaches of Targeted Therapy for Lung Cancer Elimination)trial of personalized therapies for non-small cell lung cancer (NSCLC) It usesan adaptive randomization scheme to select K = 5 treatments for n = 255NSCLC patients belonging to J = 5 biomarker classes Let ymjk denote theindicator variable of disease control of the mth patient in class j receivingtreatment k The adaptive randomization scheme is based on a Bayesianprobit model for pjk = P (ymjk = 1) The posterior mean γ

(t)jk of pjk given

all the observed indicator variables up to time t can be computed by Gibbssampling Letting γ

(t)jk = max(γ

(t)jk 01) the randomization probability for a

patient in the jth class to receive treatment j at time t+ 1 is proportional toγ(t)jk Moreover a refinement of this scheme allows suspension of treatment k

from randomization to a biomarker subgroup Simulations of the frequentistoperating characteristics at some parameter configurations are used to de-termine the threshold for treatment suspension [31] The BATTLE designwhich ldquoallows researchers to avoid being locked into a single static protocolof the trialrdquo that requires large sample sizes for multiple comparisons of sev-eral treatments across different biomarker classes can ldquoyield breakthroughsbut must be handled with carerdquo to ensure that ldquothe risk of reaching a falsepositive conclusionrdquo is not inflated as pointed out in an April 2010 editorialin Nature Reviews in Medicine on such designs

Besides BATTLE another design mentioned in the editorial is that ofI-SPY2 [32] The I-SPY1 and I-SPY2 (Investigation of Serial Studies to Pre-dict Your Therapeutic Response with Imaging and Molecular Analysis) trialsrepresent innovative approaches to multi-center clinical trials for the devel-opment and testing of biomarker-guided therapies for breast cancer I-SPY1involved ten cancer centers and the National Cancer Institute (NCI SPORE

8

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

program and cooperative groups) to identify the indicators of response tochemotherapy that would best predict the survival of women with high-riskbreast cancer During the four-year period 2002ndash2006 I-SPY1 monitored 237breast cancer patients serially using MRI and tissue samples to study the can-cer biology of responders and non-responders to standard chemotherapy ina neoadjuvant (or pre-surgery) setting It found that tumor response evalu-ated in this manner was a good predictor of the patientsrsquo overall survival andthat tumor shrinkage during the treatment was a good predictor of long-termoutcome and response to treatment The findings of I-SPY1 set the stagefor the I-SPY2 trial an ongoing adaptive clinical trial of multiple Phase 2treatment regimens (in combination with standard chemotherapy) in patientswith tumors with varying genetic signatures I-SPY2 was launched in 2010as a collaborative effort between the NCI and cancer centers the FDA andindustry The overall trial design includes a multi-institutional multi-armadaptively randomized framework with therapeutic arms introduced as olderarms graduate or are dropped due to their high or low Bayesian predictiveprobability of being more efficacious than standard therapy with pathologiccomplete response as the study endpoint The study tests novel therapeuticarms against a standard therapy arm Drugs that are found during the trialto have a sufficiently low predictive probability of being successful in a sub-sequent confirmatory phase III study are dropped from the study As moremature treatmentbiomarker signature combinations drop out for futility orgraduate to a subsequent confirmatory phase newer arms are designated totake their place thereby generating increased efficiency in a dynamic andflexible framework

Berry [27 p33] acknowledges that ldquothe flexibility of the Bayesian ap-proach can lead to complicated trial designsrdquo and that ldquoinstitutional reviewboards and others involved in clinical research including regulators whenthe trial is for drug or medical device registration require knowing the trialdesignrsquos operating characteristicsrdquo Markov chain Monte Carlo (MCMC) sim-ulations are used to compute these operating characteristics but there is adelicate computational issue because convergence of the MCMC algorithmto the stationary distribution which is the target (posterior) distributionis a theoretical concept that has not been accompanied by error bounds forpractical implementation Although there are diagnostics as described in [33p48ndash49] one can only implement simple checks in the simulation programto compute the operating characteristics for which an upper bound on thenumber of iterations has also to be imposed to avoid getting into an infi-

9

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

nite loop Therefore although Berry [27 p34] argues that ldquoone can use theBayesian approach to build a design and modify it to deliver predeterminedfrequentist characteristics such as 5 false positive rate and 90 power ata particular difference in treatment effectsrdquo resulting in an ldquoessentially fre-quentistrdquo modified Bayesian design the complex and somewhat fuzzy TypeI error claim of this approach is not easy to gain regulatory acceptance Formore complicated trials such as those involving censored survival data thesefrequentist modifications become formidable and the usual frequentist anal-ysis without modification for data-dependent adaptation is carried out inthese Bayesian adaptive designs as in [33] that reports a trial comparingrelapse-free survival for standard chemotherapy to that for capecitabine us-ing the hazard ratio for disease recurrence or death of the capecitabine groupto the standard chemotherapy group The trial was discontinued for futil-ity at the first interim analysis and usual p-values and confidence intervalswere reported in [33] ignoring that this was a group sequential design witha Bayesian stopping rule

3 Statistical and regulatory issues

As we have reviewed in Sections 21 and 22 the flexibility and otheradvantages of adaptive designs over traditional clinical trial designs gainedincreasing acceptance during the period 1990ndash2010 by the pharmaceuticalindustry for which a major concern was regulatory acceptance of these noveldesigns and the associated analyses Accordingly much effort was devotedto maintaining the Type I error probability of the confirmatory test compar-ing the new treatment to an active control or placebo based on data froma data-dependent adaptive design An ldquoefficiency camprdquo of biostatisticiansfrom academia subsequently raised issues with the efficiency of the methodsintroduced and questioned whether the inefficiency of most of these methodsto protect the Type I error probability (eg by weighting the test statis-tics or p-values at different stages of the adaptive design) might outweighthe advantages of adaptation and proposed to use standard group sequen-tial designs instead A Bayesian camp also from academia who felt thatworking with Bayesian posterior probabilities could deliver both flexibilityand efficiency then emerged Although Bayesian designs were used in somehighly visible oncology trials of biomarker-guided personalized therapies thepharmaceutical industry was leery of using them for new drug applications

10

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

because of potential regulatory difficulties and needed guidance from regula-tory agencies on what would be acceptable

31 The 2010 FDA Draft Guidance for Industry on Adaptive Design

In February 2010 this much awaited FDA Draft Guidance [34] appeared212

years after the EMA (European Medicines Agency) reflection paper onadaptive designs [35] Both documents discuss newly developed statisticalmethods that allow confirmatory research with data-driven changes of thedesign The FDA Draft Guidance focuses mainly on ldquoadequate and wellcontrolledrdquo study settings and defines an adaptive clinical study as one thatincludes a prospectively planned opportunity for modification of specifiedaspects of the study design and hypotheses based on analysis of data (usuallyinterim data) from subjects in the study Analyses of the accumulating dataare performed at prospectively planned timepoints within the study in whichldquoprospectiverdquo means that the adaptation was planned (and details specified)before examining the data Avoiding increased rates of false positive studyresults (increased Type I error rate) is emphasized in the Draft Guidancewhich also highlights the importance of minimizing statistical and operationalbiases introduced by adaptation In particular the Draft Guidance advisesshielding the investigators as much as possible from knowledge of the chosenadaptive changes and from the unblinded data used in the interim analysesbecause knowledge of the specific adaptation decisions can lead investigatorsto treat and evaluate patients differently leading to operational bias

Despite the warnings about potential biases and possible inflation of TypeI error probability the Draft Guidance acknowledges the value of adaptivedesigns to innovate clinical trials in the development of new drugs and bi-ologics It points out that ldquowell-understoodrdquo methods represent in manycases well-established and relatively low-risk means of enhancing study effi-ciency and informativeness that may deserve wider use It describes poten-tial benefits of using Bayesian methods in clinical trials for medical devicesparticularly with respect to their flexibility use of all available evidence andpredictive probability for decision making and efficient data-dependent sam-ple sizes and randomization schemes On the other hand it also points outpotential challenges in using the Bayesian approach which requires extensivepre-planning and model building besides specific computational and statis-tical expertise and which may have substantial Type I error inflation ItsSection VIID says ldquoUsing simulations to demonstrate control of the Type Ierror rate however is controversial and not fully understoodrdquo It also points

11

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

out ldquosimulation biasrdquo that can arise when simulations are set up by the mod-eling group who can decide on the choice of simulation scenarios to mask theadvantages of competing designs and on the parameter configurations in thenull hypothesis to mask Type I error inflation

Two years after the FDA Draft Guidance on Adaptive Design the Pres-identrsquos Council of Advisors on Science and Technology (PCAST) issued areport on ldquoPropelling Innovation in Drug Discovery Development and Eval-uationrdquo in September 2012 The report points out that clinical trials con-stitute the largest single component (representing nearly 40) of the RampDbudget of major biopharmaceutical companies and yet are inefficient in termsof cost time organization and delivery of evidence supporting or against thenew medical product It says that time is ripe for improving the efficiencyof clinical trials because ldquoit is increasingly possible to obtain clear answerswith many fewer patients and with less timerdquo by focusing studies on ldquospe-cific subsets of patients most likely to benefit identified based on validatedbiomarkersrdquo It also says

Another approach is to use innovative new approaches for trial designthat can provide more information more quickly Bayesian statisticaldesigns potentially allow for smaller trials with patients receiving onaverage better treatments These and other modern statistical designscan improve on current protocols which have only a very limited abil-ity to explore multiple factors simultaneously Such factors importantlyinclude individual patient responses to a drug the effects of simultane-ous multiple treatment interventions and the diversity of biomarkersand disease sub-types

It refers to [27] and cites the I-SPY trials as examples of ldquoexciting innova-tive models for clinical trialsrdquo It also recommends that the FDA ldquorun pilotprojects to explore adaptive approval mechanisms to generate evidence acrossthe life cycle of a drug from pre-market through the post-market phaserdquo

32 Response to the FDA Draft Guidance and industryrsquos perspectives

After the 3-month public comment period following the FDA Draft Guidancea special issue on adaptive designs of clinical trials appeared in the Journalof Biopharmaceutical Statistics The papers [36 37 38 39 40] in this specialissue are viewpoints from biostatisticians from the pharmaceutical industryon the Draft Guidance while [41 42 43] represent those from academiaIn addition [44 45] summarize the highlights and a panel discussion in the

12

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

Basel Conference on Perspectives on the Use of Adaptive Designs in ClinicalTrials (March 12 2010) The editorial [46] to this special issue says

The overwhelming interests in adaptive designs in lieu of group sequen-tial designs will generate more challenges and invite more controversiesDifferent from the fixed design and the group sequential design theadaptive design not only provides the opportunities to change or selectfrom the initial null hypotheses but also gives the flexibility to modifythe maximum statistical information prospectively planned which canalter the alternative hypothesis of ultimate interest The main chal-lenge of unblinding in group sequential designs has been extended toadaptive designs due to the potential of inviting changes in the futurecourse of the remaining trial or the related trials that are either under-way or ongoing The critical concerns which need to be scrutinized arethe likely implications due to the unblinded data-dependent changeswhich are prone to operationally and statistically induce the bias Useof the DMC or DSMB solely for interim adaptive monitoring and adap-tive recommendation in addition to safety monitoring with an adaptivedesign though it has been proposed is under scrutiny for its ability tobe completely objective Meanwhile other trial logistics models egcombination of the DMC and an independent statistical analysis centerhave also been proposed

A follow-up note [46] by the Editor of the journal says

In practice while we enjoy the flexibility of the adaptive trial designsthe quality integrity and validity of the trial may be at a greater riskespecially for those less-well-understood designs as described in theFDA draft guidance From a regulatory perspective there is alwaysconcern about whether the p-value or confidence interval regarding thetreatment effect under an adaptive trial design is reliable or correctIn addition the misuse or abuse of adaptive design methods in a clin-ical trial may lead to a totally different trial that is unable to addressscientificmedical questions that the trial is intended to answer

The paper [36] also summaries the work of a PhRMA (PharmaceuticalResearch and Manufacturers of America) Working Group on adaptive clinicaltrial designs prior to the FDA Draft Guidance It agrees with the DraftGuidance that ldquothe number of aspects of a trial that are subject to adaptationshould be quite limitedrdquo On the other hand it argues that seamless PhaseII-III designs for which the Draft Guidance says that ldquothese terms provide

13

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

no additional meaning beyond the term adaptiverdquo have great advantages incertain studies and ldquodeserve further emphasisrdquo in the Guidance Concerningunblinding issues in an adaptive trial it says

The PhRMA group had proposed in the Executive Summary the pos-sibility of limited and carefully controlled sponsor involvement in thedecision and adaptation processes in situations where that perspectivewas felt to be needed for the decision but always totally insulated fromtrial personnel This required clear justification of the need and purposeof the sponsor involvement ldquominimalrdquo access to information in termsof the number of individuals involved the times they would receiveinformation and the amount of information they would receive totalseparation of these personnel from trial operations and strong firewallsand documentation of processes This would clearly not be a ldquoone-size-fits-allrdquo model and would be highly dependent on case-by-case details The following points are made clear to sponsors Interim resultsmust remain inaccessible to personnel involved in trial conduct stan-dard operating procedures and charters will need to be more detailedand specific than in familiar monitoring settings detailed descriptionswill be required as to how the analysis will be performed who will haveaccess to the results and under what conditions it will be prospec-tively described and retrospectively documented how compliance withthe specified procedures will be monitored

The papers [37] [38] and [39] provide further discussions on blinding andoperational bias together with other aspects of the Draft Guidance Liuand Chi [40] give an insightful discussion of the history of regulatory re-search legal basis bias and blinding in connection with the Draft GuidanceIn addition they also raise a number of statistical issues with widely ac-cepted frequentist methods in group sequential and adaptive clinical trialsIn particular they cite the critiques of Cox [48] and Armitage [49] on errorspending conditional power and stochastic curtailment

Cook and DeMets [41] commented that the Draft Guidelines ldquoare ex-tremely useful and should provide both industry and academia valuable in-formation concerning regulatory perspective on the design conduct andanalyses of adaptive clinical trialsrdquo They also note that ldquoin exchange forpotential efficiencies in resource utilization adaptive trials suffer from lim-itations in scientific conclusions complications and inefficiencies in the sta-tistical analysis and logistical difficulties relative to fixed sample or fixed

14

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

duration trialsrdquo Emerson and Fleming [42] discuss ldquothe extent to which theadaptive designs do not meet the goals of having greater efficiency beingmore likely to identify truly effective treatments being more informativeand providing greater flexibilityrdquo and support the FDArsquos requirement ofldquoadequate and well-controlled confirmatory studies complete with prospec-tive detailed specification of the entire randomized clinical trial design in away that allows accurate and precise estimation of treatment effectivenessrdquoCheng and Chow [43] highlight the Draft Guidancersquos distinction betweenwell-understood and less well-understood adaptive designs and further clas-sify the less well-understood adaptive designs into ldquoflexiblerdquo and ldquowildlyflexiblerdquo ones and recommend the latter not be used

4 Towards flexible and efficient adaptive designs satisfying regu-latory requirements

41 Efficiency flexibility and validity via mainstream statistical methods

Mainstream statistical methods form the core of graduate programs inStatistics and have well-established theories efficiency properties and im-plementation detailssoftware Bayesian methods are clearly part of thiscore but so are parametric semiparametric and nonparametric (empirical)likelihood methods While Bayesian inference is based on the posterior dis-tribution likelihood inference is based on the likelihood function For largesample sizes and under mild regularity conditions the Bayesian and likeli-hood approaches yield asymptotically equivalent estimates and tests Theargument of the Bayesian camp that the Bayesian approach is the only ef-ficient way to predict outcomes at the end of the trial given the data atinterim analysis ignores the fact that adaptive predictors which replace theunknown parameters by sequential maximum likelihood estimates have alsobeen found to be asymptotically optimal in time series and control systems[50 51 52] Note that time series analysis and forecasting is another core inmainstream Statistics and likelihood methods are workhorses of this coreas is Bayesian methodology

Closely related to time series analysis is sequential analysis and dynamicstochastic optimization which form another core in mainstream StatisticsApplying efficient test statistics is only using half of the toolbox for buildingan efficient adaptive design and the other half consists of dynamic stochasticoptimization In fact the three-stage design proposed in [17 18] for samplesize re-estimation as described in Section 21 was derived as an approximate

15

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

solution to the associated optimal stopping problem It turns out that thecomplicated Bayesian designs proposed in the literature are sub-optimal be-cause they are myopic in the sense of minimizing the current posterior lossrather than the sum of current and future posterior losses whereas the opti-mal solutions can be well approximated by relatively simple frequentist de-signs such as those in [17 18] Another example can be found in the classicalmulti-armed bandit problem which addresses the dilemma between ldquoexplo-rationrdquo (to generate information about unknown system parameters) andldquoexploitationrdquo (to set system inputs in order to maximize expected rewardsfrom the outputs)

Suppose there are K treatments of unknown efficacy to be chosen se-quentially to treat a large class of n patients How should we allocate thetreatment to maximize the mean treatment effect Lai and Robbins [53] andLai [54] consider the problem in the setting in which the treatment effecthas a density function f(x θk) for the kth treatment where θk are unknownparameters There is an apparent dilemma between the need to learn theunknown parameters and the objective of allocating patients to the besttreatment to maximize the total treatment effect Sn = X1 + +Xn for the npatients If θk were known then the optimal rule would use the treatmentwith parameter θlowast = arg max1lekleK micro(θk) where micro(θ) = Eθ(X) In ignoranceof θk Lai and Robbins [53] define the regret of an allocation rule by

Rn(θ) = nmicro(θlowast)minus Eθ(Sn) =sum

kmicro(θk)ltmicro(θlowast)

(micro(θlowast)minus micro(θk))EθTn(k)

where Tn(k) is the number of patients receiving treatment k They show thatadaptive allocation rules can be constructed to attain the asymptoticallyminimal order of log n for the regret in contrast to the regret of order nfor the traditional equal randomization rule that assigns patients to eachtreatment with equal probability 1K A subsequent refinement by Lai [54]shows the relatively simple rule that chooses the treatment with the largestupper confidence bound U

(n)k for θk to be asymptotically optimal if the upper

confidence bound at stage n with n gt k is defined by

U(n)k = inf

θ isin A θ ge θk and 2Tn(k)I(θk θ) ge h2(Tn(k)n)

inf empty =infin

where A is some open interval known to contain θ θk is the maximum likeli-hood estimate of θk I(θ λ) is the KullbackLeibler information number and

16

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

the function h has a closed-form approximation It is noted in [55] that the

upper confidence bound U(n)k corresponds to inverting a generalized likeli-

hood ratio (GLR) test based on the GLR statistic Tn(k)I(θk θ) for testingθk = θ

The multi-arm bandit problem has the same ldquolearn-as-we-gordquo spirit ofthe BATTLE trial described in Section 23 Making use of these ideas LaiLiao and Kim [55] have recently introduced a frequentist alternative to theBayesian adaptive design of the BATTLE trial While the spirit of the BAT-TLE trial focuses on attaining the best response rate for patients in the trialit does not establish which treatment is the best for future patients witha guaranteed probability of correct selection A group sequential design isintroduced in [55] for jointly developing and testing treatment recommenda-tions for biomarker classes while using multi-armed bandit ideas to providesequentially optimizing treatments to patients in the trial Thus the designhas to fulfill multiple objectives which include (a) treating accrued patientswith the best (yet unknown) available treatment (b) developing a treatmentstrategy for future patients and (c) demonstrating that the strategy devel-oped indeed has better treatment effect than the historical mean effect ofstandard of care plus a predetermined threshold Because of the need forinformed consent the treatment allocation that uses the upper confidencebound rule for multi-arm bandits is no longer appropriate It is unlikely forpatients to consent to being assigned to a seemingly inferior treatment forthe sake of collecting more information to ensure that it is significantly infe-rior (as measured by the upper confidence bounds) Instead randomization

in a double blind setting is required and the randomization probability π(i)jk

determined at the ith interim analysis of assigning a patient in group j totreatment k cannot be too small to suggest obvious inferiority of the treat-ments being tried that is π

(i)jk ge ε for some 0 lt ε lt 1K The unknown

mean treatment effect microjk of treatment k in biomarker class j can be esti-mated by the sample mean microijk at interim analysis i Let kj = arg maxk microjk

which can be estimated by kij = arg maxk microijk at the ith interim analysisMulti-arm bandit theory suggests assigning the highest randomization prob-ability to treatment kij and randomizing to the other available treatments inbiomarker class j with probability ε This adaptive randomization schemeis called ldquoε-greedyrdquo in the machine learning literature and is much simplerthan the Thompson-type adaptive randomization scheme based on posteriorprobabilities This frequentist design is shown to be asymptotically opti-

17

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

mal in [55] which also carries out simulation studies that show its superiorfinite-sample performance

Biomarker-guided personalized therapies studied in the BATTLE trial arean example of personalized medicine Personalization in medical treatmentsand in web-based recommender systems and electronic marketing belongsto the emerging area of contextual bandit theory in sequential analysis andexperimentation of ldquobig datardquo Contextual multi-armed bandits also calledmulti-armed bandits with side (covariate) information provide a mathemat-ical framework for this stochastic dynamic optimization problem Whereasthe BATTLE trial assumes that the cut-points defining the biomarker classesare available and thereby reduce treatment allocation for each class to amulti-armed bandit problem contextual bandit theory allows learning thesecut-points (or more general regression functions of treatment response on thecovariates) To illustrate with an example the stochastic optimization prob-lem of web-based personalization in showing online ads for each user withthe goal of maximizing its effectiveness measured in terms of click-throughrate or total revenue has been formulated as a contextual multi-armed ban-dit problem with the page request of each user as side (covariate) informationand layouts of ads available for the requested page as arms We have recentlysolved the dynamic stochastic optimization problem associated with contex-tual bandits and adaptive randomization of the type in [55] together witharm elimination again plays a key role in our solution which is presentedelsewhere

The comments of Liu and Chi [40 Sections 74 75 81 84] on theplethora and controversies of ad hoc adaptation methods in the burgeoningliterature on adaptive clinical trials demonstrate the importance of combin-ing the principles and methods from different cores of mainstream Statisticsto address the inherently complex and difficult problems in adaptive design ofclinical trials In particular consider their comments on Tsiatis and Mehtarsquosclaim [11] on the inefficiency of the sample size re-estimation designs in thesecond paragraph of Section 21 They say that the claim is ldquologically flawedat the fundamental levelrdquo because [11] does not consider expected sample sizeproperties and only focuses on power at a pre-specified alternative for all testswith the same type I error and error-spending function at a simple null Inthe terminology of this section [11] only dwells on the first half of the toolboxfor building an efficient adaptive design but does not consider the second halfof the toolbox opening up the possibility of a flawed overall design that [40]discusses The second half of the toolbox on dynamic stochastic optimiza-

18

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

tion is arguably much more difficult than the first half In principle this canbe formulated as a Bayesian sequential decision problem that can be solvedby dynamic programming Specifically given a prior distribution of the un-known parameters one can formulate the dynamic stochastic optimization asa dynamic programming problem in which the state at time t is the posteriordistribution of the parameters given the observations up to t However thedynamic programming equations are often prohibitively difficult to handleboth computationally and analytically Moreover it may also be difficult tospecify a reasonable prior distribution Chapter 3 of [56] gives an overview ofstochastic optimization over time dynamic programming and approximatedynamic programming together with their applications to Phase I cancertrial designs and sequential tests of composite hypotheses In particular forthe sequential testing application analytic approximations to the Bayes rulesare developed by using Laplacersquos asymptotic formula to relate the posteriordistributions to GLR statistics and large or moderate deviations approxima-tions to boundary crossing probabilities A review of the multi-arm banditproblem is given in [57] which explains the suboptimal nature of the myopicrule and demonstrates that the rules in [53 54] achieve asymptotic efficiencyby introducing relatively simple uncertainty adjustments to the myopic ruleand thereby attaining certain lower bounds on the expected sample size fromthe inferior arms for ldquouniformly goodrdquo rules

Developing information-theoretic asymptotic lower bounds such as thosein [53 54] and finding procedures to attain them bypass the difficulties of dy-namic programming but require deep insights and synthesis of several main-stream cores of Statistics This approach has proved useful in more compli-cated design problems than those reviewed in Section 2 In particular it wasrecently used in [58] for the problem of adaptive choice of patient subgroupfor comparing two treatments motivated by a clinical trial design problemposed to us by colleagues of the Stanford Stroke Center that will be explainedin Section 43 Adaptive (data-dependent) choice of the patient subgroup tocompare the new and control treatments is a natural compromise between ig-noring patient heterogeneity and using stringent inclusion-exclusion criteriain the trial design and analysis Section 2 of [58] first provides an asymptotictheory for trials with fixed sample size in which n patients are randomized tothe new and control treatments and the responses are normally distributedwith mean microj for the new treatment and micro0j for the control treatment if thepatient falls in a pre-defined subgroup Πj for j = 1 J and with commonknown variance σ2 Let ΠJ denote the entire patient population for a tra-

19

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

ditional randomized controlled trial (RCT) comparing the two treatmentsSince there is typically little information from previous studies about thesubgroup effect size microj minus micro0j for j 6= J [58] begins with a standard RCT tocompare the new treatment with the control over the entire population butallows adaptive choice of the patient subgroup I in the event HJ is not re-jected to continue testing Hi microi le micro0i with i = I so that the new treatmentcan be claimed to be better than control for the patient subgroup I if HI isrejected

Letting θj = microjminusmicro0j and θ = (θ1 θJ) the probability of a false claimis the type I error

α(θ) =

Pθ(reject HJ) + Pθ(θI le 0 accept HJ and reject HI) if θJ le 0

Pθ(θI le 0 accept HJ and Reject HI) if θJ gt 0

for θ isin Θ0 Subject to the constraint α(θ) le α [58 Appendix A] establishesthe asymptotic efficiency of the procedure that randomly assigns n patientsto the experimental treatment and the control rejects HJ if GLRi ge cα fori = J and otherwise chooses the patient subgroup I 6= J with the largestvalue of the generalized likelihood ratio statistic

GLRi = nin0i(ni + n0i)(microi minus micro0i)2+σ

2

among all subgroups i 6= J and rejects HI if GLRI ge cα where microi(micro0i) isthe mean response of patients in Πi from the treatment (control) arm andni(n0i) is the corresponding sample size The test statistic GLRi is the sampleestimate of the Kullback-Leibler information (npi4)(microi minus micro0i)

2+σ

2 notingthat nin0i(ni + n0i) asymp npi as study subjects are equally likely to receivethe new treatment or control After establishing the asymptotic efficiencyof the procedure in the fixed sample size case [58] proceeds to extend it toa 3-stage sequential design by making use of the theory of Bartroff and Lai[17 18] reviewed in Section 21 It then extends the theory from the normalsetting to asymptotically normal test statistics such as the Wilcoxon ranksum statistics that are commonly used for analyzing the clinical endpoints(Rankin scores) of stroke patients A modification of this argument detailsof which are given elsewhere can be used to derive asymptotically efficientseamless Phase II-III designs reviewed in Section 22 for which the hypothesisHj now corresponds to the jth dose or treatment regimen of the new drug

The discussion in this section up to now has focused on the efficiency andflexibility aspects in the title and we have highlighted how different cores

20

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

of mainstream Statistics have provided concepts and techniques to addressthese aspects We now move to ldquovalidityrdquo in the title which is related tosatisfying regulatory requirements for the trial design and analysis The ldquofre-quentist twistrdquo [27 p33] to modify a Bayesian adaptive design as describedin the last paragraph of Section 23 still falls short of FDArsquos requirement ofvalid frequentist false positive rate for all parameter configurations in thenull hypothesis that we have referred to the second paragraph of Section31 The approximation of dynamic Bayes rules by sequential proceduresbased on GLRs which is called ldquopseudo-maximizationrdquo in [59 p240] has animportant bonus that makes it particularly suited to frequentist inferencenamely that GLR is an approximate pivot [59 pp207 216] under a generalnull hypothesis which ensures that the distribution of the GLR statistic isapproximately the same irrespective of the parameter configuration in thenull hypothesis As explained in [59 Chapter 16] using GLR or generalStudentized statistics in bootstrapping is important for the second-order ac-curacy of the bootstrap method because they are asymptotically pivotalNote that the bootstrap and other resampling methods form another corein mainstream Statistics For group sequential or adaptive designs the ap-proximate pivotal property of Efronrsquos bootstrap [60 61] breaks down buta more general resampling scheme called hybrid resampling can be used asthe sequential GLR or Studentized statistics are still approximately pivotalunder hybrid resampling see [62 63 64]

42 Hybrid resampling survival outcomes and more complicated settings

The results of the BATTLE trial whose Bayesian adaptive design hasbeen reviewed in Section 23 are reported by Kim et al [65] Despite applyingBayesian adaptive randomization and arm elimination ldquostandard statisticalmethods included Fisherrsquos exact test for contingency tables and log-rank testfor survival datardquo and standard confidence intervals and p-values based onnormal approximations without adjustments for Bayesian adaptive random-ization and possible treatment suspension This paper shows the dilemmafaced by the clinical investigators who bought into Bayesian design but hadto carry out frequentist inference such as p-values and confidence intervals topublish their results in medical journals What previous research to attainfrequentist validity for regulatory approval seemed to have missed is thathybrid resampling already provides a versatile and accurate method for validfrequentist inference in Bayesian and other adaptive designs One may ar-gue however that the MCMC computations of posterior probabilities may

21

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

be too computationally intensive to carry out hybrid resampling On theother hand the code for performing the frequentist operating characteris-tics in the frequentist twist [27] can be readily modified to carry out hybridresampling Furthermore as noted in the preceding section the Thompson-type adaptive randomization scheme based on posterior probabilities in theBATTLE design is suboptimal and a much simpler and yet asymptoticallyoptimal adaptive sampling scheme based on the truncated MLE p

(t)jk can be

used instead Hybrid resampling is especially suited for primary and sec-ondary analysis in a regulatory environment of complex data in adaptiveclinical trials A detailed discussion is given in [66] for the case of censoredsurvival data the complexity of which has led to inefficient adaptive de-signs as reviewed in the second paragraph of Section 22 A new approach isalso developed in [66] that can adapt the choice of the test statistics to theobserved survival pattern

43 An illustrative example on adaptive subgroup selection

In 2014 our colleagues at the Stanford Stroke Center came to us witha request to help them design a clinical trial evaluating a new method foraugmenting usual medical care with endovascular removal of the clot aftera stroke resulting in reperfusion of the area of the brain under threat withthe intent of salvaging tissue and improving outcomes compared to standardmedical care with intravenous tissue plasminogen activator (tPA) alone Theinvestigators were also interested in tailoring the reference population tooptimize the power to detect a significant effect In the course of discussionsit emerged that there were a nested sequence of six subsets of patients definedby a combination of elapsed time from stroke to start of tPA and an imaging-based estimate of the size of the unsalvageable core region of the lesion Thesequence was defined by cumulating the cells in a two-way (3 volumes times 2times) cross-tabulation as described in [58 p195] In the upper left cell c11which consisted of the patients with a shorter time to treatment and smallestcore volume the investigators were most confident of a positive effect whilein the lower right cell c23 with the longer time and largest core area there wasless confidence in the effect The six cumulated groups Π1 Π6 give rise tocorresponding one-sided null hypotheses H1 H6 for the treatment effectsin the cumulated groups This is the trial that motivated the problem ofadaptive choice of patient subgroup for comparing two treatments discussedin the penultimate paragraph of Section 41

22

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

Shortly before the final reviews of the protocol for funding were com-pleted four randomized controlled trials of endovascular reperfusion therapyadministered to stroke patients within 6 hours after symptom onset demon-strated decisive clinical benefits [67 68 69] As a result the equipoise ofthe investigators shifted making it necessary to adjust the intake criteria toexclude patients for whom the new therapy had been proven to work bet-ter than the standard treatment The subset selection strategy became evenmore central to the design since the primary question was no longer whetherthe treatment was effective at all but for which patients should it be adoptedas the new standard of care Moreover besides adapting the intake criteria tothe new findings another constraint was imposed by the NIH sponsor whicheffectively limited the total randomization to 476 patients Recall that inthe original design if HJ was accepted at an interim stage the study wouldgo on to recruit to the maximum sample size in the selected subgroup Thelimit on total randomization is an example of a constraint on adaptive designthat reflects more conservative budgeting at the NIH after positive resultsare reported by other studies with more restrictive inclusion criteria

We redefine the design as in [58] using the maximum randomization limitbut with the futility stopping rule based on GLR testing of the one-sidednull at the alternative implied by the maximum sample size which is nowa random variable since it depends on whether HJ is rejected at an interimanalysis We estimate the expected value of the maximum sample size bysimulation under the null then recalculate the implied alternative (whichbecomes larger since the expected maximum is smaller) and replace it in thedesign The operating characteristics of the adjusted design are simulatedunder various scenarios shown in Table 1 in which each result is based on5000 simulations The projected overall effect of endovascular therapy isbased on (a) the observed 90-day outcomes in an earlier study of similar pa-tients treated gt 6 hrs after symptom onset and (b) the assumption that earlyreperfusion will be achieved in 75 of the endovascular arm vs 20 of themedical therapy arm Using these projections we calculate an expected stan-dardized effect size of 036 (normal approximation to the Wilcoxon statisticcomparing the modified Rankin scores across treatments) If this effect isuniform in all 6 cells the fixed sample size non-adaptive design requires 376patients per group (overall) to have 90 power at a two-sided alpha of 5(Wilcoxon test)100 additional patients are added for the adaptive designfor a maximum randomization of 476

Table 1 compares the performance of a traditional fixed sample-size design

23

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

Table 1 Operating characteristics of adaptive design and fixed-sample con-ventional design

ScenarioEffect (standardized) in cells Avg Adaptive Conventionalc11 c12 c21 c22 c31 c32 effect Avg Power Power

0 0 0 0 0 0 0 0 361 22 476 251 03 03 03 03 03 03 03 354 80 476 892 05 04 03 0 0 0 02 400 86 476 553 05 05 0 0 0 0 017 403 87 476 41

(fixed n = 476) for testing HJ to the adaptive design (random sample sizen le 476) that allows choice of Hj (j 6= J) if HJ is accepted The effect sizesin the table are standardized Under the null hypothesis of no treatmenteffect (Scenario 0) the adaptive design controls the total Type 1 errorbelow 25 stops early for futility 63 of the time and the mean numberof patients randomized (Avg ) is 361 If the effect is uniform across cells(scenario 1) the fixed-sample design is optimal but the adaptive designresults in only a small loss of power (from 89 to 80) The adaptive designperforms much better (higher power and smaller expected sample size) thanthe fixed-sample conventional trial when the effect size distribution acrossthe subgroups is in accord with the biological assumptions (scenarios 2 and3) If the effect is concentrated in two cells with small core volumes (scenario3) the adaptive design maintains 87 power while the conventional designcollapses (41 power) The adaptive design also performs well compared toa non-adaptive fixed-sample design in [58] that allows adaptive subgroupchoice at the end of the study

5 Discussion

The FDA document [70] on the critical path to new medical productspoints out that todayrsquos advances in the biomedical sciences offer new possi-bilities to treat many diseases but that the number of new drug and biologicapplications submitted to the FDA has been declining The PCAST re-port described in the last paragraph of Section 31 argues that innovationsin clinical trial design and improvements in efficiency and cost effectivenessof clinical trials are needed for biomedical advances to reach their full po-tential in treating diseases Although it is widely recognized that adaptivedesigns represent promising innovations that can reverse the ldquostagnationrdquo

24

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

trend pointed out in [70] the overview of the state of the art in adaptive de-signs in Sections 2 and 3 shows that despite the advances reviewed thereinmajor hurdles still remain and need to be resolved before adaptive designscan meet the desiderata of efficiency flexibility and validity from a regula-tory perspective Section 4 describes our recent work at the Stanford Centerfor Innovative Study Design to address these challenges In particular weshow how the methodological problems underlying these hurdles are relatedto other branches of mainstream Statistics from which we can synthesizeand further develop the methods and results to yield (a) efficient and flex-ible adaptive designs of confirmatory clinical trials and (b) valid statisticalanalyses of the data that can meet regulatory requirements Berry [71] hasgiven a cogent summary of the advantages and disadvantages (ldquopromiserdquoand ldquocautionrdquo) of adaptive designs He points out that their potential ad-vantages are greatest in complicated settings and mentions I-SPY2 as anldquoexample of a complicated trial made possible by adaptive designrdquo in whichldquothe adaptive randomization provides information about which drugs benefitwhich patientsrdquo and ldquotrial participants receive better treatmentrdquo Since headvocates using the Bayesian approach that analyzes the data via posteriordistributions the increased complexity of a trial only increases the complex-ity of the MCMC simulations On the other hand this increased complexitywould make it even harder for the ldquofrequentist twistrdquo mentioned in Section41 to be able to satisfy the regulatory requirements However it should benoted that Section 41 and 42 have developed methods which have frequen-tist validity while sharing the efficiency and flexibility of Bayesian adaptivedesigns We are therefore in agreement with Berry that adaptive designshave greatest advantages over the traditional designs in ambitious and com-plicated trials that translate todayrsquos advances in the biomedical sciences intoeffective treatments of complex diseases

An important concern with adaptive designs mentioned by Berry [71] isldquothe issue of trends in patient populations during a period of timerdquo We haveillustrated in Section 43 how this concern can be handled at interim analysesOne can clearly adapt to these changes in an adaptive design as sequentialchange-point detection and estimation methods form another core of main-stream Statistics This issue and the related statistical methodologies willbe treated elsewhere Another major concern that has also been mentionedin the 2010 FDA Draft Guidance and in the review in Section 32 is aboutblinding and operational bias Berry [71] says ldquoModifications (of the trial)that occur during the trial may convey information outside the sphere of con-

25

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

fidentiality of the DSMB and affect the types of patients who are accrued tothe trialrdquo On the other hand ldquoall DSMB reports of interim analyses conveysome information about the relative performance of the arms of the trialsrdquoWe propose to keep whatever is disclosed to the DSMB within its sphere ofconfidentiality Blinding should be maintained outside the DSMB throughoutthe trial and the protocolrsquos adaptation rule should also be kept confidentialby the DSMB We can regard the adaptation that is built into the protocolas an algorithm that is executed by a computer analogous to algorithmictrading in financial markets with electronic platforms Since the computertechnology is already available to carry out automated high-frequency trad-ing the same should be true for implementing adaptive clinical trials If thedesign has been well thought through in advance and its performance char-acteristics are well understood there is no need for the sponsor of the trialto participate in the adaptive decisions at interim analyses similarly to howtrades are executed by computers without human intervention in algorithmictrading

The principles and methods discussed herein for adaptive design of PhaseIII trials can be extended to adaptation in clinical trial plans (CDPs) Asnoted in [72] in the development of a new drug in the pharmaceutical indus-try an important component of the effort and costs involves clinical trials toprovide clinical data to support a beneficial claim of the drug and in casethis is not valid to support the termination of its development The clinicaltrials progress in steps and are labeled Phase I II and III trials A projectteam steers the operations in which intensity cost and duration increasewith the phase and there is a core team that makes decisions guided by aCDP The CDP maps out the clinical development pathway beginning withfirst-in-man studies and ending with submission to the regulatory agency ortermination of development It defines the number and type of clinical stud-ies and their objectives determines the time sequence of the studies some ofwhich may be carried out in parallel identifies major risk areas and sets keydecision points and gono-go criteria An important objective of the CDP isto build a clinical data package to support a beneficial claim or terminationof further development These data should start to be collected from PhaseI dose rangingselection studies and provide evidence in favor of stopping orcontinuing at various decision points in subsequent Phase II and III trials Atthe planning stage of a CDP there is usually inadequate information aboutessential parameters for designing the Phase I II and III clinical trials or foroptimizing the sequence of clinical trials in an overall plan It is therefore

26

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

inevitable that strong assumptions need to be made at the planning stageto come up with over-simplified plans and standard clinical trials designswhich are well understood and relatively simple to present to senior manage-ment are good ways to start Because of the uncertainties and the strongassumptions underlying the ldquosimplifiedrdquo CDP [72] proposes to modify it byusing adaptive designs in place of standard designs in the simplified CDPwhich is the essence of the adaptive CDP developed therein

References

[1] Bauer P Multistage testing with adaptive designs (with Discussion)Biom Inform Med Bio 1989 20130ndash148

[2] Wittes J Brittain E The role of internal pilots in increasing the effi-ciency of clinical trials Stat Med 1990 965ndash72

[3] Stein C A two-sample test for a linear hypothesis whose power is inde-pendent of the variance Ann Math Stat 1945 16243ndash258

[4] Gould AL Shih WJ Sample size re-estimation without unblinding fornormally distributed outcomes with unknown variance Comm Stat SerA 1992 212833ndash2853

[5] Herson J Wittes J The use of interim analysis in sample size adjust-ment Drug Inform J 1993 27753ndash760

[6] Birkett M Day S Internal pilot studies for estimating sample size StatMed 1994 132455ndash2463

[7] Denne JS Jennison C A group sequential t-test with updating of samplesize Biometrika 2000 87125ndash134

[8] Fisher L Self-designing clinical trials Stat Med 1998 171551ndash1562

[9] Denne S Sample size recalculation using conditional power Stat Med2001 202645ndash2660

[10] Jennison C Turnbull BW Mid-course sample size modification in clini-cal trials based on the observed treatment effect Stat Med 2003 22971ndash993

27

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

[11] Tsiatis AA Mehta C On the efficiency of the adaptive design for mon-itoring clinical trials Biometrika 2003 90367ndash378

[12] Proschan M Hunsberger S Designed extension studies based on condi-tional power Biometrics 1995 511315ndash1324

[13] Bauer P Kohne K Evaluation of experiments with adaptive interimanalyses Biometrics 1994 501029-1041

[14] Posch M Bauer P Adaptive two stage designs and the conditional errorfunction Biom J 1999 41689-696

[15] Jennison C Turnbull BW Adaptive and nonadaptive group sequentialtests Biometrika 2006 931-21

[16] Jennison C Turnbull BW Efficient group sequential designs when thereare several effect sizes under consideration Stat Med 2006 25917-932

[17] Bartroff J Lai TL Efficient adaptive designs with mid-course samplesize adjustment in clinical trials Stat Med 2008 271593ndash1611

[18] Bartroff J Lai TL Generalized likelihood ratio statistics and uncertaintyadjustments in efficient adaptive design of clinical trials Sequent Anal2008 27254-276

[19] Lorden G Asymptotic efficiency of three-stage hypothesis tests AnnStat 1983 11129-140

[20] Bretz F Schmidli H Konig F Racine A Maurer W Confirmatory seam-less phase IIIII clinical trials with hypotheses selection at interim gen-eral concepts Biom J 2006 4623ndash634

[21] Schmidli H Bretz F Racine A Maurer W Confirmatory seamless phaseIIIII clinical trials with hypotheses selection at interim applicationsand practical considerations Biom J 2006 4635ndash643

[22] Simes RJ An improved Bonferroni procedure for multiple tests of sig-nificance Biometrika 1986 73751ndash754

[23] Stallard N Todd S Sequential designs for phase III clinical trials incor-porating treatment selection Stat Med 2003 22689ndash703

28

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

[24] Wang Y Lan KKG Ouyang SP A group sequential procedure for in-terim treatment selection Stat Biopharm Res 2011 31ndash13

[25] Brannath W Zuber E Branson M Bretz F Gallo P Posch M Racine-Poon A Confirmatory adaptive designs with Bayesian decision tools fora targeted therapy in oncology Stat Med 2009 281445ndash1463

[26] Jenkins M Stone A Jennison C An adaptive seamless phase IIIIIdesign for oncology trials with subpopulation selection using correlatedsurvival endpoints Pharm Stat 2011 10347-356

[27] Berry DA Bayesian clinical trials Nature Rev Drug Disc 2006 527ndash36

[28] Berry SM Carlin BP Lee JJ Muller P Bayesian Adaptive Methods forClinical Trials Boca Raton FL Chapman amp HallCRC 2011

[29] Thompson W On the likelihood that one unknown probability exceedsanother in view of the evidence of two samples Biometrika 1933 25285ndash294

[30] Meuer WJ Lewis RJ Berry DA Adaptive clinical trials A partialremedy for the therapeutic misconception J Am Med Assoc 20123072377ndash2378

[31] Zhou X Liu S Kim ES Herbst RS Lee JL Bayesian adaptive design fortargeted therapy development in lung cancerndashA step toward personalizedmedicine Clin Trials 2008 5181ndash193

[32] Barker AD Sigman CC Kelloff GJ Hylton NM Berry DA EssermanLJ I-SPY 2 An adaptive breast cancer trial design in the setting ofneoadjuvant chemotherapy Clin Pharmacol Ther 2009 8697ndash100

[33] Berry DA et al Adjuvant chemotherapy in older women with early-stagebreast cancer N Engl J Med 2009 3602055ndash2065

[34] Food and Drug Administration Center for Drug Evaluation andRe-search Guidelines for Industry Adaptive Design Clinical Tri-alsfor Drugs and Biologics 2010 httpwwwfdagovdownloads

DrugsGuidanceComplianceRegulatoryInformationGuidances

UCM201790pdf

29

[35] Committee for Medicinal Products for Human Use Reection paper onmethodological issues in conrmatory clinical trials planned with anadaptive design Eur Med Agency 2007 httpwwwemeaeuropaeupdfshumanewp245902enadoptedpdf

[36] Gallo P Anderson K Chuang-Stein C Dragalin V Gaydos B KramsM Pinheiro J Viewpoints on the FDA draft adaptive designs guidancefrom the PhRMA working group J Biopharm Stat 2010 20 1115ndash1124

[37] Brannath W Burger HU Glimm E Stallard N VandemeulebroeckeM Wassmer G Comments on the Draft Guidance on ldquoAdaptive de-sign clinical trials for drugs and biologicsrdquo of the US Food and DrugAdministration J Biopharm Stat 2010 20 1125ndash1131

[38] Chuang-Stein C Beltangady M FDA Draft Guidance on adaptive de-sign clinical trials Pfizerrsquos perspective J Biopharm Stat 201020 1143ndash1149

[39] Wittes J Comments on the FDA Draft Guidance on adaptive designsJ Biopharm Stat 2010 20 1166ndash1170

[40] Liu Q Chi YH Understanding the FDA Guidance on adaptive designshistorical legal and statistical perspectives J Biopharm Stat 2010 201178ndash1219

[41] Cook T DeMets DL Review of draft FDA adaptive design guidance JBiopharm Stat 2010 20 1132ndash1142

[42] Emerson SS Fleming TR Adaptive methods telling ldquothe rest of thestoryrdquo J Biopharm Stat 2010 20 1150ndash1165

[43] Cheng B Chow SC On flexibility of adaptive designs and criteria forchoosing a good onemdasha discussion of FDA Draft Guidance J BiopharmStat 2010 20 1171ndash1177

[44] Wang SJ Perspectives on the use of adaptive designs in clinical trialsPart I Statistical considerations and issues J Biopharm Stat 2010 201090ndash1097

[45] BendaN Brannath W Bretz F Burger HU Friede T Maurer W WangSU Perspectives on the use of adaptive designs in clinical trials PartII Panel discussion J Biopharm Stat 2010 20 1098ndash1112

30

[46] Wang SJ Editorial Adaptive designs Appealing in development oftherapeutics and where do controversies lie J Biopharm Stat 2010 201083ndash1087

[47] Chow S-C A note on special articles on adaptive clinical trial designsJ Biopharm Stat 2010 201088ndash1089

[48] Cox DR Commentary The likelihood paradigm for statistical evidenceby Richard Royall In The Nature of Scientific Evidence StatisticalPhilosophical and Empirical Considerations (Tapper ML Lele SR eds)138ndash140 Univ Chicago Press 2004

[49] Armitage P Discussion of the paper by Jennison and Turnbull J RoyStat Soc Ser B 1989 51334ndash335

[50] Goodwin GC Sin KS Adaptive Filtering Prediction and Control Mi-neola NY Dover 2009

[51] Lai TL Ying Z Recursive identification and adaptive prediction in linearstochastic systems SIAM J Contr Optim 1991 291061ndash1090

[52] Lai TL Zhu G Adaptive prediction in nonlinear autoregressive modelsand control systems Stat Sinica 1991 1309ndash334

[53] Lai TL Robbins H Asymptotically efficient adaptive allocation rulesAdv Appl Math 1985 6 4ndash22

[54] Lai TL Adaptive treatment allocation and the multi-armed bandit prob-lem Ann Stat 1987 15 1091ndash1114

[55] Lai TL Liao OY-W Kim DW Group sequential designs for develop-ing and testing biomarker-guided personalized therapies in comparativeeffectiveness research Cont Clin Trials 2013 36651ndash663

[56] Bartroff J Lai TL Shih MC Sequential Experimentation in ClinicalTrials Design and Analysis New York Springer 2013

[57] Lai TL Information bounds certainty equivalence and learning inasymptotically efficient adaptive control of time-invariant stochasticsystems In Topics in Stochastic Systems Modelling Estimation andAdaptive Control (Gerencser L Caines PE eds) 335ndash368 New YorkSpringer 1991

31

[58] Lai TL Lavori PW Liao OY Adaptive choice of patient subgroup forcomparing two treatments Con Clin Trials 2014 39191ndash200

[59] dela Pena V Lai TL Shao QM Self-normalized process Limit theoryand statistical applications New York Springer 2009

[60] Efron B Bootstrap methods Another look at the jackknife Ann Stat1979 71ndash26

[61] Efron B Better bootstrap confidence intervals J Amer Stat Assoc 198782171ndash185

[62] Chuang CS Lai TL Hybrid resampling methods for confidence intervals(with Discussion) Stat Sinica 2000 101ndash50

[63] Lai TL Li W Confidence intervals in group sequential trials with ran-dom group sizes and applications to survival analysis Biometrika 200693641ndash654

[64] Lai TL Shih MC Su Z Tests and confidence intervals for secondaryendpoints in sequential clinical trials Biometrika 2009 96903ndash915

[65] Kim et al The BATTLE trial Personalizing therapy for lung cancerCancer Discovery 2011 144ndash53

[66] He P Lai TL Zheng S Design of clinical trials with failure-time end-points and interim analyses An update after 15 years Con Clin Trials2015 this issue

[67] Berkhemer OA Fransen PS Beumer D van den Berg LA Lingsma HFYoo AJ et al A randomized trial of intraarterial treatment for acuteischemic stroke N Engl J Med 2015 372 11ndash20

[68] Campbell BC Mitchell PJ Kleinig TJ Dewey HM Churilov L Yassi Net al Endovascular therapy for ischemic stroke with perfusion-imagingselection N Engl J Med 2015 3721009ndash18

[69] Goyal M Demchuk AM Menon BK Eesa M Rempel JL Thornton J etal Randomized assessment of rapid endovascular treatment of ischemicstroke N Engl J Med 2015 3721019-1030

32

[70] Food and Drug Administration Innovativestagnation Challengeand opportunity in the critical path to new medical products FDAreport 2004 httpwwwfdagovocinitiativescriticalpath

whitepaperhtml

[71] Berry DA Adaptive clinical trials The promise and the caution J ClinOncol 2011 606ndash609

[72] Lai TL Liao OY-W Zhu RZ Adaptation in clinical development plansand adaptive clinical trials Stat amp Its Interface 2012 5431ndash442

33

[58] Lai TL Lavori PW Liao OY Adaptive choice of patient subgroup forcomparing two treatments Con Clin Trials 2014 39191ndash200

[59] dela Pena V Lai TL Shao QM Self-normalized process Limit theoryand statistical applications New York Springer 2009

[60] Efron B Bootstrap methods Another look at the jackknife Ann Stat1979 71ndash26

[61] Efron B Better bootstrap confidence intervals J Amer Stat Assoc 198782171ndash185

[62] Chuang CS Lai TL Hybrid resampling methods for confidence intervals(with Discussion) Stat Sinica 2000 101ndash50

[63] Lai TL Li W Confidence intervals in group sequential trials with ran-dom group sizes and applications to survival analysis Biometrika 200693641ndash654

[64] Lai TL Shih MC Su Z Tests and confidence intervals for secondaryendpoints in sequential clinical trials Biometrika 2009 96903ndash915

[65] Kim et al The BATTLE trial Personalizing therapy for lung cancerCancer Discovery 2011 144ndash53

[66] He P Lai TL Zheng S Design of clinical trials with failure-time end-points and interim analyses An update after 15 years Con Clin Trials2015 this issue

[67] Berkhemer OA Fransen PS Beumer D van den Berg LA Lingsma HFYoo AJ et al A randomized trial of intraarterial treatment for acuteischemic stroke N Engl J Med 2015 372 11ndash20

[68] Campbell BC Mitchell PJ Kleinig TJ Dewey HM Churilov L Yassi Net al Endovascular therapy for ischemic stroke with perfusion-imagingselection N Engl J Med 2015 3721009ndash18

[69] Goyal M Demchuk AM Menon BK Eesa M Rempel JL Thornton J etal Randomized assessment of rapid endovascular treatment of ischemicstroke N Engl J Med 2015 3721019-1030

32

[70] Food and Drug Administration Innovativestagnation Challengeand opportunity in the critical path to new medical products FDAreport 2004 httpwwwfdagovocinitiativescriticalpath

whitepaperhtml

[71] Berry DA Adaptive clinical trials The promise and the caution J ClinOncol 2011 606ndash609

[72] Lai TL Liao OY-W Zhu RZ Adaptation in clinical development plansand adaptive clinical trials Stat amp Its Interface 2012 5431ndash442

33

[70] Food and Drug Administration Innovativestagnation Challengeand opportunity in the critical path to new medical products FDAreport 2004 httpwwwfdagovocinitiativescriticalpath

whitepaperhtml

[71] Berry DA Adaptive clinical trials The promise and the caution J ClinOncol 2011 606ndash609

[72] Lai TL Liao OY-W Zhu RZ Adaptation in clinical development plansand adaptive clinical trials Stat amp Its Interface 2012 5431ndash442

33