Incorporating patient-preference evidence into regulatory ......indicating their relative value for...

10
Incorporating patient-preference evidence into regulatory decision making Martin P. Ho Juan Marcos Gonzalez Herbert P. Lerner Carolyn Y. Neuland Joyce M. Whang Michelle McMurry-Heath A. Brett Hauber Telba Irony Received: 5 September 2014 / Accepted: 9 November 2014 Ó Springer Science+Business Media New York (outside the USA) 2015 Abstract Background Patients have a unique role in deciding what treatments should be available for them and regulatory agencies should take their preferences into account when making treatment approval decisions. This is the first study designed to obtain quantitative patient-preference evidence to inform regulatory approval decisions by the Food and Drug Administration Center for Devices and Radiological Health. Methods Five-hundred and forty United States adults with body mass index (BMI) C30 kg/m 2 evaluated trade- offs among effectiveness, safety, and other attributes of weight-loss devices in a scientific survey. Discrete-choice experiments were used to quantify the importance of safety, effectiveness, and other attributes of weight-loss devices to obese respondents. A tool based on these mea- sures is being used to inform benefit-risk assessments for premarket approval of medical devices. Results Respondent choices yielded preference scores indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate the minimum weight loss acceptable by a patient to receive a device with a given risk profile and the maximum mortality risk tolerable in exchange for a given weight loss. For example, to accept a device with 0.01 % mortality risk, a risk tolerant patient will require about 10 % total body weight loss lasting 5 years. Conclusions Patient preference evidence was used make regulatory decision making more patient-centered. In addition, we captured the heterogeneity of patient prefer- ences allowing market approval of effective devices for risk tolerant patients. CDRH is using the study tool to define minimum clinical effectiveness to evaluate new weight-loss devices. The methods presented can be applied to a wide variety of medical products. This study supports the ongoing development of a guidance document on incorporating patient preferences into medical-device pre- market approval decisions. Electronic supplementary material The online version of this article (doi:10.1007/s00464-014-4044-2) contains supplementary material, which is available to authorized users. M. P. Ho Á H. P. Lerner Á C. Y. Neuland Á J. M. Whang Á M. McMurry-Heath Á T. Irony (&) Center for Devices and Radiological Health, U.S. Food and Drug Administration, 10903 New Hampshire Ave, Building 66, Room 2232, Silver Spring, MD 20993-0002, USA e-mail: [email protected] M. P. Ho e-mail: [email protected] H. P. Lerner e-mail: [email protected] C. Y. Neuland e-mail: [email protected] J. M. Whang e-mail: [email protected] M. McMurry-Heath e-mail: [email protected] J. M. Gonzalez Á A. Brett Hauber RTI Health Solutions, Durham, USA e-mail: [email protected] A. Brett Hauber e-mail: [email protected] M. McMurry-Heath FaegreBD Consulting, Washington, USA 123 Surg Endosc DOI 10.1007/s00464-014-4044-2 and Other Interventional Techniques

Transcript of Incorporating patient-preference evidence into regulatory ......indicating their relative value for...

Page 1: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

Incorporating patient-preference evidence into regulatorydecision making

Martin P. Ho • Juan Marcos Gonzalez • Herbert P. Lerner •

Carolyn Y. Neuland • Joyce M. Whang • Michelle McMurry-Heath •

A. Brett Hauber • Telba Irony

Received: 5 September 2014 / Accepted: 9 November 2014

� Springer Science+Business Media New York (outside the USA) 2015

Abstract

Background Patients have a unique role in deciding what

treatments should be available for them and regulatory

agencies should take their preferences into account when

making treatment approval decisions. This is the first study

designed to obtain quantitative patient-preference evidence

to inform regulatory approval decisions by the Food and

Drug Administration Center for Devices and Radiological

Health.

Methods Five-hundred and forty United States adults

with body mass index (BMI) C30 kg/m2 evaluated trade-

offs among effectiveness, safety, and other attributes of

weight-loss devices in a scientific survey. Discrete-choice

experiments were used to quantify the importance of

safety, effectiveness, and other attributes of weight-loss

devices to obese respondents. A tool based on these mea-

sures is being used to inform benefit-risk assessments for

premarket approval of medical devices.

Results Respondent choices yielded preference scores

indicating their relative value for attributes of weight-loss

devices in this study. We developed a tool to estimate the

minimum weight loss acceptable by a patient to receive a

device with a given risk profile and the maximum mortality

risk tolerable in exchange for a given weight loss. For

example, to accept a device with 0.01 % mortality risk, a

risk tolerant patient will require about 10 % total body

weight loss lasting 5 years.

Conclusions Patient preference evidence was used make

regulatory decision making more patient-centered. In

addition, we captured the heterogeneity of patient prefer-

ences allowing market approval of effective devices for

risk tolerant patients. CDRH is using the study tool to

define minimum clinical effectiveness to evaluate new

weight-loss devices. The methods presented can be applied

to a wide variety of medical products. This study supports

the ongoing development of a guidance document on

incorporating patient preferences into medical-device pre-

market approval decisions.Electronic supplementary material The online version of thisarticle (doi:10.1007/s00464-014-4044-2) contains supplementarymaterial, which is available to authorized users.

M. P. Ho � H. P. Lerner � C. Y. Neuland �J. M. Whang � M. McMurry-Heath � T. Irony (&)

Center for Devices and Radiological Health, U.S. Food and Drug

Administration, 10903 New Hampshire Ave, Building 66, Room

2232, Silver Spring, MD 20993-0002, USA

e-mail: [email protected]

M. P. Ho

e-mail: [email protected]

H. P. Lerner

e-mail: [email protected]

C. Y. Neuland

e-mail: [email protected]

J. M. Whang

e-mail: [email protected]

M. McMurry-Heath

e-mail: [email protected]

J. M. Gonzalez � A. Brett Hauber

RTI Health Solutions, Durham, USA

e-mail: [email protected]

A. Brett Hauber

e-mail: [email protected]

M. McMurry-Heath

FaegreBD Consulting, Washington, USA

123

Surg Endosc

DOI 10.1007/s00464-014-4044-2

and Other Interventional Techniques

Page 2: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

Keywords Patient preferences � Weight-loss devices �Obesity treatment � FDA � Benefit-risk assessment �Regulatory-approval decisions

In the past, the United States (US) Food and Drug

Administration (FDA) did not have a scientific way to

consider a representative opinion of patients when weigh-

ing the benefits and risks of a medical product to approve it

to be marketed in the United States [1, 2]. Since the release

of its guidance document on benefit-risk determinations in

2012, the FDA Center for Devices and Radiological Health

(CDRH) has made clear that patient preference is an

important factor to consider when evaluating medical

devices for market approval. The guidance explicitly states

that

Risk tolerance will vary among patients, and this will

affect individual patient decisions as to whether the

risks are acceptable in exchange for a probable ben-

efit. …FDA would consider evidence relating to

patients’ perspective of what constitutes a meaningful

benefit.

For example, when evaluating weight-loss devices for

market approval, CDRH seeks to identify appropriate

levels of benefit that outweigh different levels of risk [3].

Patients have a unique role in deciding what treatments

should be available for them. Their perspectives about the

value of benefits and the impact of risks of medical treat-

ments are essential because only patients live with their

medical conditions and consequences of the choices they

make for their own care. Moreover, their perspectives can

be different from those of regulators and care providers. As

the principle of patient-centered heath care has been widely

accepted, regulatory authorities are very interested in

obtaining input on patient perspectives on benefits and

tolerance for risk for making regulatory evaluations,

knowing that unless a treatment is approved, it is not an

option for patients regardless of their risk tolerance [4–6].

This study is the first step in demonstrating how to

implement this idea and offers a quantitative approach to

bridging the gap between what CDRH reviewers and

patients regard as acceptable benefit-risk tradeoffs for

weight-loss devices. Although numerous efforts of regu-

latory agencies have been made to actively reach out and

listen to the opinion of patients with various conditions, the

response received has been qualitative in nature. While

these contributions can be valuable for other purposes, they

cannot adequately address the approval decisions because

they lack measurability, representativeness, accuracy, and

inclusiveness. In contrast, this study provides quantitative

evidence of patient benefit-risk tradeoff preference that

includes a wide spectrum of patients. It quantifies patients’

perspectives on benefits, providing the minimum benefit

they expect from a weight-loss device to tolerate a given

level of risk and other device attributes. Patient tolerance

for risk is quantified as the maximum device-related mor-

tality risk patients are willing to tolerate for a given weight

loss and other device attributes.

Because patients’ perspectives on benefits and risks may

be very heterogeneous within a patient group, CDRH needs

to consider the whole spectrum of patient preferences.

Some patients in the top quartile of the risk-tolerance dis-

tribution could be ‘‘early adopters,’’ meaning they will

tolerate higher risks to gain quick access to innovative

treatments. CDRH will consider approving a medical

device that demonstrates meaningful benefits even though

its benefit-risk profile would be acceptable only to a subset

of patients who are risk-tolerant. Such a device’s Indication

for Use will explain that the benefit-risk profile may be

suitable to only a subgroup of patients who should consult

with their care providers in a context of shared medical

decision making.

The objectives of this study were to (1) apply well-

established methods for quantifying the relative importance

of effectiveness, safety, and other attributes of weight-loss

devices to patients; (2) derive measures of patients’ pref-

erences to inform benefit-risk assessments; and (3) incor-

porate such measures into regulatory decision making.

This is the first patient-preference study designed by

regulatory reviewers and used to inform regulatory deci-

sion making. The results are providing direction for

designing clinical trials for premarket approval, informing

benefit-risk assessments once the trial results are being

evaluated, and guiding post-approval decisions. The con-

ceptual framework and quantitative methods can be gen-

eralized to a wide variety of medical treatments and are

particularly relevant when patients and regulators face

difficult decisions when weighing potential treatment

benefits against serious adverse-event risks.

Materials and methods

Design overview

Choice-experiment surveys ask respondents to choose the

most-preferred alternative from a set of two or more con-

structed virtual alternatives in a series of questions. Sta-

tistical analysis of the pattern of these choices reveals the

implicit relative importance of the attributes of the health

intervention that influence respondents’ choices among the

alternatives offered. The result is an estimate of the per-

ceived value of an intervention as a weighted sum of the

intervention attributes, where the weights reflect the mean

Surg Endosc

123

Page 3: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

relative importance of each factor. In this study we call

these weights ‘‘preference scores.’’

Setting and participants

A Web-enabled, cross-sectional survey was administered

between September and November 2012 to a stratified sample

of English-speaking US residents with a current or previous

BMI of at least 30 kg/m2 who were willing to lose weight. The

sample was drawn from the online GfK KnowledgePanel�,

divided equally among three groups: 30 B BMI\35,

35 B BMI\40, and 40 B BMI.

Respondents who did not meet the BMI inclusion cri-

teria were screened out. Respondents who did not answer

any question or who always picked the same alternative

were excluded from the sample. Because we compared the

available choices with a no-device alternative, data that did

not include an answer to the follow-up device-acceptance

question or for which the answer was ‘‘Don’t know/not

sure’’ were excluded.

The consent form, survey instrument, and data-collec-

tion protocol were approved by the RTI International

Office of Research Protection and Ethics and the Office of

Information and Regulatory Affairs in the US Office of

Management and Budget.

Survey instrument

Survey content was developed based on published clinical

research on obesity, regulatory knowledge of weight-loss

device technology, and interviews with obese individuals.

Eight weight-loss device attributes were selected based on

knowledge about devices that are likely to be reviewed by

CDRH in the near future. The attributes were of primary

concern to physicians, regulators, and obese subjects con-

sulted during survey development: (1) total body weight loss

(TBWL) as percentage of current weight, (2) duration of

weight loss, (3) duration of mild-to-moderate side effects,

(4) mortality risk, (5) chance of a side effect requiring hos-

pitalization, (6) recommended dietary restrictions, (7)

reduction in chance of comorbidity or reduction in pre-

scription dosage for existing comorbidity, and (8) type of

surgery. Each attribute had three to five levels. The levels of

the continuous attributes covered a wide range of values to

allow for interpolation of estimates, whereas the levels of the

categorical attributes included several plausible categories.

Clinical-outcome attributes were described as average

results for each device profile. Respondents were asked to

assume that all costs were covered by insurance. All the

device attributes and levels used in the study are listed in

Appendix A in Supplementary material.

The study design included several internal-validity tests,

with particular attention to verifying that respondents

understood basic risk concepts. Before answering the

choice questions, respondents received a risk tutorial and

answered a question to verify their understanding.

The choice questions employed a standard format for

choice-experiment surveys [7–9]. Each respondent evalu-

ated eight pairs of virtual weight-loss device profiles and

were asked ‘‘Which weight-loss device do you think is

better for people like you?’’ Figure 1 is an example choice

question. Respondents were asked in a separate follow-up

question whether they personally would accept the better

device if it were available or they would prefer no device.

Device profiles in the choice questions varied according

to a predetermined experimental design with known sta-

tistical properties. An experimental-design algorithm yiel-

ded 120 choice questions [10–12]. Good-practice

guidelines recommend 8–12 questions per respondent to

limit measurement error due to fatigue. Because of the

complexity of a choice task that includes a probabilistic

outcome, the design was divided into 15 survey versions of

8 questions each, and respondents were randomized to one

of the versions. More information on the final experimental

design is included in Appendix B in Supplementary

material.

A draft of the survey instrument was pretested in

90-min, face-to-face, semistructured interviews with nine

obese subjects in accordance with US Office of Manage-

ment and Budget regulations. Following final ethics and

regulatory review, the Web-enabled instrument was further

field-tested by drawing 86 respondents from the GfK Web

panel before its full release.

Results

Data quality

Of 1,057 panel members randomly drawn from GfK’s

KnowledgePanel�, a large Web panel that matches the

demographics of the general US population, 710 responded

to the invitation and 568 qualified for the survey: 1.2 % did

not answer any tradeoff question, 1.9 % always picked the

same alternative, and 14.3 % did not answer the follow-up

device-acceptance question. After these exclusions, 540

respondents were available for the final analysis, yielding a

qualification rate of 80.0 % and a final-stage completion

rate of 67.2 %. Additional details on response rates are

provided in Appendix C in Supplementary material.

Results of internal validity tests indicated the data sat-

isfied high quality standards. For example, only 34

respondents (6.3 %) failed a quiz question following the

risk tutorial. Additional details on internal validity are

provided in Appendices D and E in Supplementary

material.

Surg Endosc

123

Page 4: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

The demographic characteristics of respondents who

were included and excluded in the analysis were similar.

Although the sample was stratified by BMI, the analysis

sample on average was about 2 years older and about 8.5 %

heavier than the general US obese population. Additional

details on comparisons among included respondents,

excluded respondents, and the general obese population are

provided in Appendix F in Supplementary material.

Fig. 1 Example tradeoff

question

Surg Endosc

123

Page 5: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

Preference-score estimates

The analysis provides estimates of the preference scores

that best explain the pattern of observed choices in the data.

Additional details on parameter estimates are provided in

Appendix G in Supplementary material.

Figure 2 shows the preference-score estimates for all

attribute levels. They are on a scale from -10 (least pre-

ferred), to 10 (most preferred), where -10 is the perceived

value of 5 % mortality risk. The difference between the

best and worst preference scores for each attribute indicates

how influential that attribute is in explaining device choi-

ces. Mortality risk was the most important attribute, fol-

lowed by weight loss, weight-loss duration, and side-effect

duration. However, no side effect requiring hospitalization

and a 5 % risk of a side effect requiring hospitalization

with no surgery were considered much less important.

Table 1 shows the estimated preference scores for the

attributes and their levels, indicating their relative impor-

tance. For example, the importance of an increase in

mortality risk from 0 to 1 % can be quantified by the

preference-score difference between these two levels; it

was about -3.5 (6.5 - 10). In comparison, the importance

of a 30 % TBWL was about 4.3 (4.3 - 0). Therefore, if

other device attributes are kept the same, 30 % TBWL

more than compensates for having to bear a 1 % risk of

death from the device.

Benefit-risk tradeoff comparisons

In this study, we used the estimated preference scores to

calculate the minimum acceptable benefit (MinB), i.e.,

minimum weight loss respondents expect from a device to

tolerate a specific level of risk and other device attributes.

Fig. 2 Relative-importance estimates for attributes of weight-loss

devices preference parameters were rescaled relative to 10, corre-

sponding to the absolute value of the most important outcome—5 %

mortality risk. Attributes with connecting lines were modelled as

nonlinear Box–Cox transformed continuous variables. Other attri-

butes were modelled as categorical variables. Vertical bars around

preference scores denote 95 % confidence intervals. Benefits have

positive scores and harms have negative scores. Differences between

the best and worst level in each attribute indicate relative attribute

importance over the level ranges in the study design. Attributes are

sorted from most important benefit to most important harm. Mortality

risk is the most important outcome or feature, followed by weight

loss, weight-loss duration, and side-effect duration. The relative

importance of a 1 % increase in risk is -3.5, in comparison with the

relative importance of a 30 % total body weight loss of ?4.3

Surg Endosc

123

Page 6: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

We also calculated the maximum acceptable risk (MaxR),

i.e., maximum device-related mortality risk respondents are

willing to tolerate for a given weight loss and other device

attributes.

The statistical analysis accounted for heterogeneity in

patients’ preferences and allowed quantitative segmenta-

tion of obese respondents according to their risk tolerance.

The MinB and MaxR estimates in the middle 50 % and the

upper 25 % of the sample indicate the quantitative benefit-

risk tradeoff preferences of average patients and risk-tol-

erant patients, respectively.

Table 2 compares the overall acceptability of three

devices relative to the no-device option indicated by MinB

and MaxR for the gastric-band and for Devices A and B,

which are worse and better than the gastric band, respec-

tively. Preference-score estimates predict that only 5.0 %

of respondents would judge Device A better than the no-

device alternative, while 27.5 % of respondents would

judge Device B better than the no-device alternative.

Devices with less attractive attributes require either larger

weight loss or smaller mortality risks in compensation. For

example, the gastric band offers only 13 % TBWL, but the

Table 1 Estimates of preference scores by attributes and levels

Attribute Level Preference score (SE)

Average amount of weight loss (TBWL) 30 % ?4.3 (0.52)

20 % ?2.0 (0.11)

10 % ?0.6 (0.15)

5 % ?0.2 (0.23)

0 % Reference level

Weight loss duration 60 months ?4.3 (0.47)

12 months ?2.0 (0.01)

6 months ?1.4 (0.1)

0 months Reference level

Side effect duration 0 months Reference level

1 months -1.0 (0.11)

12 months -2.0 (0.09)

60 months -3.2 (0.31)

Chance of side effects requiring hospitalization None Reference level

5 % chance of hospitalization, no surgery -0.2 (0.39)

20 % chance of hospitalization, no surgery -0.5 (0.35)

5 % chance of hospitalization, with surgery -0.6 (0.36)

Dietary restrictions Eat 1/4 cup of food at a time Reference level

Wait 4 h between eating -0.1 (0.29)

Can’t eat sweets or foods that are hard to digest -2.2 (0.33)

Average reduction in dose of prescription drugs for comorbiditya Eliminate need/risk ?3.2 (0.37)

50 % dose/risk ?2.2 (0.29)

No change Reference level

Type of operation Laparoscopic surgery Reference level

Endoscopic surgery -0.5 (0.3)

Open surgery -2.5 (0.31)

Chance of dying from getting the device 0 % Reference level

1 % -3.5 (0.13)

3 % -7.1 (0.15)

5 % -10 (0.37)

Receiving a weight-loss device through open surgery versus endoscopic surgery had a preference-score difference of -2 (-2.5 minus -0.5).

Maintaining weight loss for 12 months versus no months had a preference-score difference of ?2 (2 - 0). Holding everything else constant,

receiving a device placed through open surgery instead of endoscopically requires at least 12 months of weight loss to compensate. The same

metric can be used to measure patient tolerance for risks of adverse events

TBWL total body weight lossa Average reduction in dose of prescription drugs for the current primary comorbid condition or chance of getting the most feared comorbid

condition at the lower weight

Surg Endosc

123

Page 7: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

middle 50 % of respondents require a mean TBWL of

more than 30 % given the attributes of the gastric band.

Similarly, the gastric band has a 1 % mortality risk while

the middle 50 % of respondents would tolerate only a

0.16 % risk for the attributes of the gastric band. On the

other hand, for risk-tolerant early adopters, the mean

minimum acceptable TBWL is only 13.0 % and the mean

maximum acceptable risk is 7.1 %, which compare favor-

ably to the gastric band’s profile. Overall, estimates imply

that only 11.6 % of respondents would judge the gastric

band better than no device, which is consistent with the fact

that only a small percentage of obese patients have chosen

gastric-band surgery [13, 14].

The tool

One of the main issues faced by regulatory reviewers when

designing clinical studies and analyzing their results is

determination of the ‘‘minimum clinical effectiveness’’ that

is sufficient to offset the risks and inconveniences posed by

the treatments under review. This value is used to size the

clinical studies and, when results are available, to decide

Table 2 Preference relative to a no-device alternative, minimum acceptable benefit, and maximum acceptable risk of a 243 pound respondent

for three weight-loss device profiles

Weight loss device profile % Respondents who

judged better than

a no-device

alternative

Risk

Tolerance

Minimum

acceptable

TBWLa

Maximum

acceptable

mortality riskb

Virtual device A 5.0 % Middle 50 % [30 %c 0.1 % (0.04–0.26)

Benefit: 5 % TBWL

Risk: 1 % chance of death Upper 25 % 15.1 %

(11.63–19.05)

3.8 % (1.85–7.77)

Type of surgery: laparoscopic surgery

Dietary restriction: eat 1/4 cup of food at a time

Weight-loss duration: 36 months

Minor side-effect duration: 36 months

Chance of hospitalization with surgery: 5 %

Comorbidity: no improvement

Gastric band 11.6 % Middle 50 % [30 %c 0.16 % (0.07–0.38)

Benefit: 13 % TBWL

Risk: 1 % chance of death Upper 25 % 13.0 % (9.76–16.90) 7.1 % (3.82–13.50)

Type of surgery: laparoscopic surgery

Dietary restriction: eat 1/4 cup of food at a time

Weight-loss duration: 5 years

Minor side-effect duration: 5 years

Chance of hospitalization for side effects requiring surgery:

5 %

Comorbidity: no improvement

Virtual device B 27.5 % Middle 50 % [30 %c 0.6 % (0.29–1.12)

Benefit: 20 % TBWL

Risk: 1 % chance of death Upper 25 % 12.8 % (9.85–16.45) 10.9 %

(6.54–18.78)Type of surgery: endoscopic surgery

Dietary restriction: eat 1/4 cup of food at a time

Weight-loss duration: 60 months

Minor side-effect duration: 12 months

Chance of hospitalization but no surgery: 5 %

Comorbidity: 50 % improvement

95 % confidence intervals in parentheses

TBWL total body weight lossa For indicated mortality risk and other device attributesb For indicated weight-loss benefit and other device attributesc Exceeds upper limit of weight-loss levels included in the study design

Surg Endosc

123

Page 8: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

whether the benefits of the treatment outweigh the risks for

market approval. There is an enormous amount of discus-

sion among device makers, regulators and experts about

this value.

To include patient preference input in these discussions,

we use the preference-score estimates to build a MaxR–

MinB calculator for weight-loss devices (‘‘the tool’’). The

MinB obtained from the tool is used to inform CDRH

reviewers in establishing the minimum clinical effective-

ness (weight loss) of devices under review to size clinical

studies. In addition, once the results of the study are

available, reviewers use the minimum clinical effective-

ness as a reference to evaluate the benefit of a device that is

being considered for premarket approval. Given the profile

of the weight-loss device, the tool can answer three dif-

ferent but related questions:

1. What percentage of subjects would prefer getting the

device over a no-device alternative?

2. What is the MaxR of an average subject or of an early

adopter for a device that provides a given weight loss?

3. What is the MinB of an average subject or of an early

adopter for a device that poses a given mortality risk?

Development of the tool went through several evalua-

tions and iterations with the regulatory reviewers to make it

usable, understandable, and relevant to the review work.

Examples of the tool’s appearance, input fields, and func-

tion are shown in Fig. 3.

Discussion

This is the first study designed to provide regulators with

quantitative data on patient preferences in support of reg-

ulatory benefit-risk tradeoff determinations. Previous

information on patient preferences largely has consisted of

qualitative, anecdotal testimony obtained in advisory-

committee proceedings and public meetings. In contrast,

this study provides quantitative data obtained in a con-

trolled experiment from a cross-sectional sample of US

obese respondents. FDA regulators and the investigators

jointly developed the survey instrument to ensure the study

design provided relevant tradeoff information for devices

likely to be reviewed by the agency in the foreseeable

future. Moreover, these attributes were important to obese

subjects, and their levels are clinically meaningful and

relevant to regulatory decision making.

Patients’ perspectives on benefits and tolerance for risks

are likely to be diverse within a patient population and, as a

whole, may also differ from clinicians’ perspectives. It is

important that a regulatory agency takes into account

variations in risk tolerance of the whole spectrum of pos-

sible users of a medical treatment because unless a

treatment is approved, it is not available to physicians and

patients, regardless of their risk tolerance. Patients who are

more tolerant of higher risks are likely to be ‘‘early

adopters,’’ meaning they could be willing to accept higher

risks to gain faster access to innovative treatments. Risk-

tolerant early adopters also are likely to participate in

clinical trials of a novel technology and play a critical role

in postapproval evaluations by accepting the novel tech-

nology once it is on market. If only risk-tolerant patients

would accept a certain device profile, the FDA might

consider approving such a device only for risk-tolerant

patients. Such indication for use will be explained in the

device label.

CDRH is using the MaxR–MinB calculator ‘‘the tool,’’

to inform reviewers in establishing the minimum clinical

effectiveness of weight-loss devices under review. Similar

decision-aid calculators can be developed for other medical

products. For a given medical product profile, the tool can

assist regulatory reviewers to determine the minimum

clinical effectiveness to be used in trial design and inter-

pretation of trial results. In addition, opinions among cli-

nicians and patients may be mixed when evaluating a new

medical product that offers an incremental improvement in

effectiveness but a slightly worse safety profile than

existing treatment options. Such medical products could be

preferred by some but rejected by others, and the FDA

should take this into account.

This study may have been subject to some technical

limitations. Choices among virtual weight-loss devices do

not have the same clinical and emotional consequences as

actual choices. However, clearly defined and validated

definitions of device attributes helped increase the reli-

ability of the study results by ensuring that respondents

were well informed. The potential for hypothetical bias was

further reduced by first eliciting judgments about which

combination of plausible attributes was better and subse-

quently eliciting the stated choice between device and no-

device alternatives.

The inclusion criteria relied on patient-reported height

and weight, not measured BMI. Assurance of anonymity

reduced incentives to understate weight and overstate

height. In fact, the mean self-reported weight in the study

cFig. 3 The MaxR–MinB Tool. 3.1 Using the tool to estimate

minimum acceptable benefit (25 %TBWL). 3.2 Using the tool to

estimate maximum acceptable risk (0.5 % mortality). As shown in 3.1and 3.2, the tool provides an input area at the top where the device

attributes, including mortality risk (0.05 %) for (3.1) and benefit

(15 % TBWL) for (3.2) are entered. The green cells in the center

display the estimated MinB (25 %) and MaxR (0.5 %) respectively.

Results can be shown for average patients (middle quartile of the

distribution) or for early adopters (upper quartile of the distribution)

and for different base weights (Color figure online)

Surg Endosc

123

Page 9: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

Surg Endosc

123

Page 10: Incorporating patient-preference evidence into regulatory ......indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate

sample was greater than the national average for the same

age cohort. Furthermore, we found no significant differ-

ences in preferences between respondents with BMI less

than or greater than 40.

Whereas the Web panel matches the demographics of

the general U.S. population, our stratified sample of obese

panel members does not need to match the demographics

of the general obese population. Nevertheless, we found

only a 2-year difference in mean age. Because we did not

find significant differences in preferences between

respondents by BMI, the 8.5 % difference in mean weight

between the study and national sample has no effect on

estimates.

Despite these limitations, the estimated preference

scores for treatment attributes and their levels provide a

quantitative solution to the problem of comparing different

outcomes in benefit-risk assessments. They also provide

information for making patient-centered, evidence-based

regulatory decisions. This study has demonstrated the

practical feasibility of eliciting and using evidence on

patient benefit-risk tradeoff preferences to inform regula-

tory decisions. The experience acquired from this study

serves as a proof of principle to support the ongoing

development of a CDRH guidance document on including

evidence on patient preferences in medical-device sub-

missions for premarket approval. We believe that the

approach used here is superior to making regulatory deci-

sions without an understanding of the patients’ values and

their heterogeneity. This study is a significant step in the

direction of incorporating patients’ benefit preferences and

risk tolerance into the existing evidence-based regulatory

process.

Acknowledgments We acknowledge Jeffrey Shuren, MD, JD for

his support of the study and insightful advice on the manuscript. We

also acknowledge Priya Venkataraman-Rao, MD; Megan Shackel-

ford, MS; Rebecca Nipper; Richard Kotz, MS; Kathleen Olvey; and

Martin Golding, MD, for their regulatory input in the development of

the survey instrument. We are grateful to the FDA CDRH Obesity

Devices Working Group for their comments on interpretation of study

results and their feedback on using the MinB–MaxR calculator in

regulatory reviews of weight-loss device submissions. The members

of that group include Jeffrey Cooper, DVM; Megan Shackelford, MS;

Irene Bacalocostantis, PhD; Brandan Reid, PhD; Martha Betz, PhD;

Elizabeth Katz, PhD; David Pudwill; Martin Golding, MD; Priya

Venkataraman-Rao, MD; and Benjamin Fisher, PhD. The acknowl-

edged persons above are of CDRH, FDA.

Disclosures Drs. Hauber and Gonzalez received compensation for

their work through a contract with the Center for Devices and

Radiological Health (CDRH) of the U.S. Food and Drug Adminis-

tration (FDA). Ho, Lerner, Neuland, Whang, McMurry-Heath, and

Irony have no conflicts of interest or financial ties to disclose.

References

1. http://www.fda.gov/downloads/MedicalDevices/DeviceR

egulationandGuidance/GuidanceDocuments/UCM296379.pdf.

Accessed 2 Sept 2014

2. Section 513(a) of the Federal Food, Drug, and Cosmetic Act [21

U.S.C. 360c(a)]

3. Lerner H, Whang J, Nipper R (2013) Benefit-risk paradigm for

clinical trial design of obesity devices: FDA proposal. Surg En-

dosc 27(3):702–707

4. Shuren J, Woodcock J (2013) Reviewing FDA’s implementation

of FDASIA. http://www.fda.gov/NewsEvents/Testimony/ucm374

544.htm. Accessed 2 Sept 2014

5. McMurry-Heath M (2013) FDA brings patients into the process.

http://blogs.fda.gov/fdavoice/index.php/2013/09/fda-brings-

patients-into-the-process/. Accessed 2 Sept 2014

6. http://www.ema.europa.eu/docs/en_GB/document_library/

Report/2013/10/WC500153276.pdf. Accessed 2 Sept 2014

7. Marshall D, Bridges JFP, Hauber B et al (2010) Conjoint analysis

applications in health: how are studies being designed and

reported? Patient 3(4):1178–1653

8. Bridges JFP, Hauber AB, Marshall D et al (2011) Conjoint

analysis applications in health—a checklist: a report of the IS-

POR good research practices for conjoint analysis task force.

Value Health 14(4):403–413

9. Johnson FR, Lancsar E, Marshall D et al (2013) Constructing

experimental designs for discrete-choice experiments: report of

the ISPOR conjoint analysis experimental design good research

practices task force. Value Health 16(1):3–13

10. Kuhfeld WF, Tobias RD, Garratt M (1994) Efficient experimental

design with marketing research applications. J Mark Res

31:545–557

11. Kuhfeld WF (2010) Marketing research methods in SAS. http://

support.sas.com/techsup/technote/mr2010.pdf. Accessed 2 Sept

2014

12. Ogden CL, Carroll MD, Kit BK, Flegal KM (2012) Prevalence of

obesity in the United States, 2009–2010. NCHS Data Brief No. 82

13. Nguyen NT, Masoomi H, Magno CP, Nguyen XMT, Laugenour

K, Lane J (2011) Trends in use of bariatric surgery, 2003–2008.

J Am Coll Surg 213(2):261–266

14. Callegaro M, DiSogra C (2008) Computing response metrics for

online panels. Public Opin Q 72(5):1008–1032

Surg Endosc

123