Recommendations of the Panelon Cost-Effectivenessin Health and MedicineMilton C. Weinstein, PhD; Joanna E. Siegel, ScD; Marthe R. Gold, MD, MPH; Mark S. Kamlet, PhD;Louise B. Russell, PhD; for the Panel on Cost-Effectiveness in Health and Medicine

Objective.\p=m-\Todevelop consensus-based recommendations for the conduct ofcost-effectiveness analysis (CEA). This article, the second in a 3-part series, de-scribes the basis for recommendations constituting the reference case analysis, theset of practices developed to guide CEAs that inform societal resource allocationdecisions, and the content of these recommendations.

Participants.\p=m-\ThePanel on Cost-Effectiveness in Health and Medicine, anonfederal panel with expertise in CEA, clinical medicine, ethics, and health out-$comes measurement, was convened by the US Public Health Service (PHS).

Evidence.\p=m-\Thepanel reviewed the theoretical foundations of CEA, currentpractices, and alternative methods used in analyses. Recommendations were de-veloped on the basis of theory where possible, but tempered by ethical and prag-matic considerations, as well as the needs of users.Consensus Process.\p=m-\Thepanel developed recommendations through 21/2

years of discussions. Comments on preliminary drafts prepared by panel workinggroups were solicited from federal government methodologists, health agency of-ficials, and academic methodologists.

Conclusions.\p=m-\Thepanel's methodological recommendations address(1) components belonging in the numerator and denominator of a cost\x=req-\effectiveness (C/E) ratio; (2) measuring resource use in the numerator of a C/Eratio; (3) valuing health consequences in the denominator of a C/E ratio; (4) esti-mating effectiveness of interventions; (5) incorporating time preference and dis-counting; and (6) handling uncertainty. Recommendations are subject to the "ruleof reason," balancing the burden engendered by a practice with its importance toa study. If researchers follow a standard set of methods in CEA, the quality andcomparability of studies, and their ultimate utility, can be much improved.

JAMA. 1996:276:1253-1258

COST-EFFECTIVENESS analysis(CEA) has emerged as a basic tool inthe evaluation of health care practices.

Despite widespread application, there re¬main disparities in the methods thatinvestigators employ.1 Some of these dis¬parities can be traced to a misunder¬standing of the principles ofCEA, whileothers reflect divergent views on keymethodological choices. For example, aninvestigator who fails to account for im¬portant negative side effects ofa therapyin estimating effectiveness is making aclear error, while investigators who in¬clude or exclude the financial costs oflost productivity that accompany ill¬ness are reflecting different views ofhowproductivity should be accounted for ina CEA.

The divergence ofmethods used to con-

duct CEA interferes with the ability ofdecision makers charged with resourceallocation to make appropriate compari¬sons of cost-effectiveness (C/E) ratiosacross programs. As described in the firstarticle of this series,2 this concern aboutlack of standardization has led the PanelonCost-Effectiveness in Health andMedi¬cine to develop a set of recommendationsfor the practice ofCEA that can serve as

a point of reference for investigators whoseek comparabilitywith other analyses inthe literature. The panel refers to this setof methodological practices as the refer¬ence case.

The reference case will not address alltypes of questions regarding the cost-effectiveness of interventions. In some

cases, depending on the goals ofthe analy¬sis, the author may prefer to highlight ananalysis based on a slightly different setof principles, or one based on quite dif¬ferent assumptions. In the interest ofcom¬

parability, however, we urge that the ref¬erence case set of assumptions andpractices be included in every CEA thatis designed to permit broad comparisonsacross interventions or thatmight be usedfor this purpose.

The recommendations outlined here,together with others that provide moredetailedmethodological guidance, are ex¬

panded in the full report of the panel.3While this article focuses on the refer¬ence case recommendations, we also de¬scribe a few recommendations that areintended to improve the conduct ofanaly¬ses but that are not explicitly incorpo¬rated within the reference case.

RATIONALES FORRECOMMENDATIONS

Reference case analyses are intendedto inform resource allocation decisionsand, as described in the first article of thisseries, are conducted from the societalperspective for this reason.2 Specific ree-

From the Departments of Health Policy and Man-agement (Dr Weinstein) and Maternal and Child Health(Dr Siegel), Harvard School of Public Health, Boston,Mass; Office of Disease Prevention and Health Promo-tion, US Public Health Service, Washington, DC (DrGold); Heinz School, Carnegie Mellon University, Pitts-$burgh, Pa (Dr Kamlet); and the Institute for Health,Health Care Policy, and Aging Research, Rutgers Uni-versity, New Brunswick, NJ (Dr Russell).

A complete list of the Panel on Cost-Effectiveness inHealth and Medicine membership and staff appears atthe end of this article.

Corresponding author: Marthe Gold, MD, MPH, Of-fice of Disease Prevention and Health Promotion, 200Independence Ave SW, Room 738G, Washington, DC20201.

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from

ommendations for conductingCEAs fromthe societal perspective were based on anumber of considerations, including eco¬nomic and decision theory; consistency inthe accountingofcosts and consequences;ethical concerns; pragmatic concerns; andneeds of users of analyses. In some in¬stances, where neither theory nor theseother considerations led to a clear choice,the panel recommended a conventionalpractice for the sake ofconsistency acrossstudies.

Where possible, recommendationswerebased on theoretical considerations in or¬der to provide a defensible, logical, andconsistent framework formethodologicalchoices. At one level, CEA can be basedsolely on the mathematical theory of"optimization."4 In that formulation, thedecision maker is free to select any ob¬jective to maximize (eg, life-years or qual¬ity-adjusted life-years [QALYs]) and tospecify the particular resource constraintunder which allocations must be made(eg, national health care costs). However,this framework alone provides no guid¬ance on key issues that arise from a so¬cietal perspective, such as which costs toinclude in the analysis, how to measure

costs, and whose values to incorporateinto the definition ofhealth consequences.Therefore, drawing from recent litera¬ture suggesting a link between CEA andwelfare economic theory,5 the panel re¬lied on economic theory for many of therecommendations for the reference case.In addition to economic theory, principlesofdecision theory were invoked to definethe basis for individual preferences.

Some recommendationswere dictatedby the need to maintain a logical account¬ing of costs and health effects. For ex¬

ample, theoretical considerations, alongwith a basic presumption about the defi¬nition of the C/E ratio,2 led to a cleardecision concerning the placement of allhealth effects in the denominator of theC/E ratio. Then, the "accounting" prin¬ciple that no cost or effect should becounted twice disallowed the inclusion ofhealth effects—even inmonetary form—in the numerator of the C/E ratio.

The implications of welfare economicswere often modified in the interest ofproducing recommendations that wereboth pragmatic and ethically acceptable.The practical need to obtain data onhealthoutcomes, utilization of services and unitcosts, andweights forhealth-related qual-ity-of-life (HRQL) states led to such com¬

promises. For example, while medicalprices are not an exact reflection of thetrue value of resources, pragmatism sug¬gests that prices be used to approximatecosts except where distortions are likelyto be significant and important to theanalysis.

Ethical considerations sometimes tem-

pered recommendations based on eco¬nomic theory or were decisive in choicesamongalternatives. Most fundamentally,the decision to use QALYs as the effec¬tiveness measure reflects the ethicalstance that QALYs accruing to differentpeople or at different stages in life shouldbe valued equally, even though welfareeconomics implies that health benefitsshould beweighted by willingness to pay.Where theory did not offer a clear

choice, the panel based some recommen¬dations simply on the need for a clearconvention to which analysts would ad¬here in the reference case. In some cases,the recommendation was somewhat ar¬

bitrary. For example, the choice ofa stan¬dard time discount rate, while guided bytheory and data, is fixed by the need fora standard practice.

Finally, needs of the potential users ofCEAs influenced several recommenda¬tions, playing a particularly great role inthe panel's recommendations for the re¬

porting ofCEAs as described in the thirdarticle of this series.6 They also enteredinto the recommendations regarding theevidence ofeffectiveness used in analysesand the treatment ofuncertainty. For ex¬ample, sensitivity analyses,which explorethe implications of alternative assump¬tions and data, are often recommended sothat decision makers can gain confidencein the conclusions of an analysis or iden¬tify areas for further investigation.While some of the reference case rec¬

ommendations address common errors inthe practice of CEA, many more repre¬sent the panel's view of the best amongseveral defensible choices. The task ofde¬veloping CEA standards is analogous tothe formulation of the consumer price in¬dex (CPI), which is used to adjust forinflation based on the prices of a typicalmarket basket ofgoods and services. Thechoice of what items go into that marketbasket, and how they are weighted, re¬flects judgments made by the Bureau ofLabor Statistics and its advisors. Legiti¬mate opposing views exist.7 However,there is an implied consensus that theCPI will be used so that industry, gov¬ernment, and consumers can have a sharedunderstanding of the inflation rate. In¬deed, flows of resources, such as the levelof Social Security payments, depend inpart on the CPI. While this country doesnot base policy directly on C/E ratios asit does with the CPI, it is important formany decision makers to be able to relyon a dependable yardstick for measuringcost-effectiveness of health services.

RECOMMENDATIONSThe panel's recommendations fall

largely into 8 categories: (1) the natureand limits of CEA and of the referencecase; (2) components belonging in the

numerator and the denominator of a C/Eratio; (3) measuring terms in the numera¬tor of a C/E ratio (costs); (4) valuing thehealth consequences in the denominatorof a C/E ratio; (5) estimating effective¬ness of interventions; (6) time preferenceand discounting; (7) handling uncertaintyinCEA; and (8) reporting guidelines. Thefirst group of recommendations, regard¬ing the nature and limits of CEA, hasbeen described in the first article of thisseries.2 The last group, regarding report¬ing guidelines, is the subject of the thirdarticle.6 Additional recommendations re¬

garding research to develop improveddata for CEA and improved methods aredescribed in the full report of the panel.8This article summarizes the remaining 6categories of recommendations.

Components Belonging in theNumerator and the Denominatorof a C/E Ratio

Cost-effectiveness analysis rests on theproposition that a decision maker wishesto select programs so as tomaximize somedesired objective subject to a resourceconstraint. In practice, CEA in healthcare has been based on the premise thathealth benefits are the objective that so¬cietal decision makers wish to maximize,subject to a constraint on health care re¬sources. This formulation leads directlyto the construction ofa C/E ratio in whichthe net expenditure of health care re¬sources (amonetarymeasure) goes in thenumerator and the net improvement inhealth (a nonmonetary measure) goes inthe denominator.

Unfortunately, however, this definitionis incomplete. It leaves open to questionwhether certain costs and consequencesshould be thought of as health care costsor savings (numerator), or health decre¬ments or improvements (denominator),and it completely ignores nonhealth costsand effects, such as those associated witheconomic productivity, the environment,or education. Therefore, ifanalyses are tohandle such issues consistently, the choicebetweennumerator (resource impact) anddenominator (HRQL impact) must fol¬low an established convention. In any case,the societal perspective dictates that allimportant impacts on human health andon resources must be included some¬

where, either in the numerator or thedenominator. With this principle inmind,the panel reached the following recom¬mendations regarding the distinction be¬tween costs and health consequences.By convention, the denominator of a

C/E ratio is reserved for the improve¬ment in health associated with an inter¬vention. Thus, effects of an interventionon length of life and on morbidity, includ¬ing the full value of HRQL to patients,should be incorporated in the denomina-

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from

tor. In order to avoid double counting,monetary values for lost life-years shouldnot be imputed in CEA and should not beincluded in the numerator of the C/E ra¬tio. For a reference case analysis, HRQLshould be captured by a measure that, ata minimum, implicitly incorporates theeffects of morbidity on productive timeand leisure. When instruments used tomeasure health states are silent concern¬ing the consideration of lost income, weassume that financial effects have beenconsidered by the respondent and that itis therefore not necessary to account forthese effects in the numerator. Prefer¬ences for health states ideally should beelicited using health statusmeasures thatexplicitly invite respondents to considerthe full range of impacts of the healthstatus change, including loss of incomeand leisure activities.

Currently, some instruments used tomeasure health states explicitly excludeconsideration of lost income. Moreover,methods thatmeasure changes in HRQLin monetary terms, such as contingentvaluation (willingness to pay), have alsobeen used. While these approaches are

technically valid, a CEA in which thesepractices were used would not constitutea reference case analysis.

The numerator of a C/E ratio captureschanges in resource use associated withan intervention. The major categories ofresource use that should be included arecosts of health care services; costs of pa¬tient time expended for the intervention;costs associated with caregiving (paid or

unpaid); other costs associated with ill¬ness, such as child care and travel ex¬

penses; economic costs borne by employ¬ers, other employees, and the rest ofsociety, including so-called friction costsassociated with absenteeismand employeeturnover8; and costs associated with non-health impacts of the intervention, suchas on the educational system, the criminaljustice system, or the environment.

The handling of patient time in CEApresents challenges and is the focus ofseveral reference case recommendations.Time spent by individuals seeking healthcare or undergoing an intervention is a

component of the intervention, and thusit should be valued in monetary termsand incorporated into the numerator ofthe C/E ratio. Time spent sick (morbiditytime) is part of the health outcome mea¬sured in the denominator of the CEA, asdescribed above. In some instances (eg,when recuperating from surgery), timecould be categorized either as morbiditytime (valued in the denominator) or as an

input to the intervention itself (costedout in the numerator); as a general rule,this time should be considered as mor¬

bidity time. These recommendations arenot based on any fundamental theoretical

consideration, but are made for consis¬tency across reference case analyses.Measuring Terms in the Numeratorof a C/E Ratio (Costs)A change in the use ofa resource caused

by a health intervention should be valuedat its opportunity cost, which is the valuethe resource could have produced ifitwerespent in its best available alternative use.In economics, this principle is the basisfor valuing resource flows in society.

Several implications arise from the op¬portunity cost principle. First, it is thedifference in resource use between anintervention and the intervention withwhich it is being compared that should beincluded in the numerator of the C/E ra¬tio. That is, costs should reflect the mar¬

ginal or incremental resources consumedor saved, rather than total resources.Fixed costs—costs unaffected by the levelof implementation of an intervention—should generally be excluded fromconsideration. However, resource costsshould be measured from a long-term per¬spective, which implies that many coststhat may be fixed in the short run (suchasmost ofwhat is usually considered over¬head in the financial accounts of hospi¬tals and other health care providers) arein fact variable in the long run and shouldbe included in CEAs.Direct measurement of opportunity

costs is difficult and often impossible. Tothe extent that market prices of health¬care inputs reflect opportunity costs, theyprovide an appropriate means for valuingchanges in resources. According to eco¬nomic theory, prices in competitive mar¬kets reflect opportunity costs ofresources.However, when prices do not adequatelyreflect opportunity costs because ofmar¬ket distortions, they should be adjustedappropriately. Examples of adjustmentscommonly used in CEAs include the useof ratios of cost to charge (RCCs) to ad¬just hospital prices, the use of manage¬ment accounting systems to estimatecosts, and the use ofthird-party paymentsto providers in lieu of fees to reflect pro¬vider opportunity costs. When substan¬tial bias is present in prices and adjust¬ment is not feasible, the panel recommendsthat more suitable proxies for opportu¬nity costs be considered, including the pos¬sibilityofconducting "microcosting" stud¬ies within provider organizations. (Suchstudies collect information on the range ofinputs to a service, such as the nursingcare, supplies, and ancillary services con¬

stituting a day of hospital care.)Costs should be measured in constant

dollars, that is, in the dollars of a fixedyear. When the original data are for dif¬ferent years, the effect of price inflationmust be removed, either by inflating thedata from an earlier year to the chosen

year or by deflating the data from a lateryear. Depending onwhether the resources

being valued are more representative ofgoods and services in the economy at largeor in the medical care sector, either theCPI or its medical care component(s) issuitable for inflation adjustment in CEAs.Transferpayments (suchas cash trans¬

fers from taxpayers to welfare recipients)associated with a health intervention re¬distribute resources from one individualto another. While administrative costs as¬sociated with such transfers could be in¬cluded in the numerator of a C/E ratio,the transfers themselves should not besince, by definition, their impact on thetransferor and the recipient cancel outfrom the societal perspective.

Time costs for individuals in the laborforce should generally be valued by thewage rate as an acceptable measure ofopportunity cost of time. The referencecase recommendation is to use wages cor¬

responding to the age and gender com¬

position of the target population. How¬ever, group-specific wages may influencethe conclusions of an analysis in ethicallyproblematic ways. For example, a policy¬maker might object to having the wagedifferential between men and women re¬flected in the results of a CEA. In theseinstances, sensitivity analyses should beconducted to explore the specific natureof this influence. Because of such ethicalconcerns, and because of practical prob¬lems in obtaining data on wages by char¬acteristics other than age and gender, thepanel does not recommend using wagesspecific to target groups defined by race,ethnicity, or other characteristics.

Wage ratesgenerally do not adequatelyreflect the value of time for persons en¬

gaged primarily in leisure—such as re¬tired persons—or in activities for whichthey are not compensated—such as house¬hold activities. Average age- and gender-specific wages among persons in the la¬bor force may be applied to approximatethe opportunity cost of time for personsof similar age and gender not in the laborforce.

Should health care costs that resultsolely from the fact that a successfullytreated patient lives longer be attributedto the health intervention? Which futurecosts should be included in a CEA? Forexample, a cost-effective analysis of an-tihypertensive therapy found that exclud¬ing noncardiovascular disease costs in fu¬ture years would reduce the C/E ratio by5% to 20%, with the greatest impacts onthe ratios for youngerpopulation groups.9To clarify the issues, we define 5 catego¬ries of induced costs that may or may notbe germane in a CEA. These are (1) costsfor intervention-related diseases incurredin years of life that would have been livedanyway; (2) costs for unrelated diseases

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from

that are incurred in years oflife thatwouldhave been lived anyway; (3) health carecosts for related diseases that ensue inyears of life added (or subtracted) as aresult of the intervention, (4) health carecosts for unrelated diseases that occur inyears of life added (or subtracted) by theintervention, and (5) nonhealth care coststypified by commodities such as food andshelter that occur in years of life added(or subtracted) by the intervention.

The handling of some of these catego¬ries of costs is uncontroversial. Costs ofrelated diseases in the original life spanclearly must be included in the analysis.For example, costs and savings associ¬ated with treatment of strokes and myo-cardial infarctions must be included inanalyses ofhypertension programs. Costsof treating adverse effects of treatmentmust be included as well. On the otherhand, because unrelated health and non-health care costs occurring throughoutthe expected years lived without the in¬tervention would cancel from the incre¬mental cost calculation in the numeratorof the C/E ratio, these may be omittedfrom the analysis. It may actually be pref¬erable to omit these unrelated costs be¬cause their measurement may add to er¬ror in the estimation of costs with andwithout the intervention.

Costs for intervention-related diseasesthat occur in added years of life are typi¬cally included in CEAs. For example, if afatal myocardial infarction is delayed 5years by a coronary bypass operation ora cholesterol-lowering regimen, the costsof treating coronary events ensuingthroughout the 5 years should be, andusually are, included. Similarly, costs ofan ongoing therapy throughout addedyears of life, such as lifelong antihyper-tensive treatment and its medication sideeffects, are always included.

Costs of diseases unrelated to the in¬tervention and ensuingas a result ofaddedyears of life have been the source ofmoredebate.4,5,10"12 Difficulties with the choiceto include or exclude them are illustratedby the example of a cholesterol-loweringintervention. The analyst might decide toexclude all unrelated costs occurring inyears of life gained because of the pro¬gram. In this case, costs of illnesses suchas arthritis and Alzheimer disease wouldbe excluded. However, age-specific "back¬ground" costs ofcoronary heart disease—that is, the level of disease that wouldoccur among people who are not candi¬dates for the intervention—are also "un¬related" to the intervention and shouldalso be excluded. To neglect to do sowouldprovide an uneven playing field for com¬

parisons of interventions affecting differ¬ent diseases: life-prolongingheart diseaseinterventions would be encumbered withall future costs of heart disease even

though they only target an excess risk,while suicide prevention programswouldnot. To avoid the practical and conceptualproblems in disengaging the "related" and"unrelated" elements ofcosts for "related"diseases, it would be preferable to in¬clude all these costs. However, this choicewould impose a burden on the analystfrequently not warranted by the impor¬tance of future costs.

Because there are unresolved theoreti¬cal and empirical questions and becausehealth care costs in added years of life are

typically small compared with the othercosts in an analysis, the panel concludedthat the reference case may either in¬clude or exclude these costs. Wheneverthe investigator has reason to believe thatinclusion or exclusion offuture health carecosts may make a significant differenceto the analysis, a sensitivity analysisshould be performed to assess the effecton the C/E ratio.We now consider nonhealth care costs

in added life-years. Although there is no

precedent in CEA for including thesecosts, one could reasonably argue that ifhealth care costs in added years can beincluded, future expenditures on food,clothing, and shelter should also be in¬cluded. The theoretical answer is that thenet economic burden of survivors on therest of society (consumption minus pro¬ductivity) should indeed be included as acost. However, if these nonhealth carecosts are truly "unrelated," then theirconsistent inclusion or exclusion wouldonly add or subtract a constant from theC/E ratio.5 Whether nonhealth care costsare in fact "unrelated," or at least ap¬proximately so, is an unresolved empiri¬cal question. Nonetheless, on the assump¬tion that these costs can reasonably beconsidered to be unrelated and to avoidplacing an unnecessaryburden on the ana¬

lyst, the panel does not recommend in¬cluding future nonhealth care costs in ref¬erence case analyses.Valuing the Health Consequencesin the Denominator of a C/E RatioAs discussed in the first article of this

series,2 a reference case analysis shouldmeasure health effectiveness in terms ofQALYs. These QALYs incorporatechanges in survival and changes inHRQLby weighting years of Ufe to reflect thevalue of the HRQL during each year.13,14In order to be consistent with the QALYconstruct, the quality weights must bemeasured by or transformed into the in¬terval scale on which optimal health hasa value of 1 and death has a value of 0.

The weights used in QALYs should bebased on a health-state classification sys¬tem that reflects health-related domains(attributes) that are important for the par¬ticular analysis. In order to qualify as a

reference case analysis, the CEA shoulduse a generic health-state classification,that is, a classification that applies broadlyacross diseases and conditions. Disease-specific health-state classifications are ap¬propriate for a reference case analysis pro¬vided that they are designed to bemappedonto or embedded within a generic sys¬tem. Some examples of commonly usedhealth-state classification systems thatmay be suitable for CEA include theHealth Utilities Index,14,15 the EuroQol,16the Quality ofWell-Being Scale,17 and theYears of Healthy Life measure.18 TheHealth Utilities Index, for example, con¬sists of8 domains (vision, hearing, speech,dexterity,mobility, cognition, emotion, andpain), each of which is classified into ei¬ther 5 or 6 levels. Each combination oflevels is assigned a weight, using a for¬mulabased onmultiattribute utility theoryand a community preference survey.13,14

The weights used in QALYs should bebased on preferences for health states. Ina reference case analysis, these weightsshould be based on community prefer¬ences, rather than those ofpatients, pro¬viders, or investigators. The rationale forcommunity preferences has been de¬scribed in the first article of this series.2Health status scales that are not prefer¬ence weighted, such as the Medical Out¬comes Study Short-Form Health Survey(SF-36),19 are not suitable for CEA intheir present form. Use of patient pref¬erences to value health states is accept¬able in a reference case analysis onlywhenadequate information is unavailable re¬

garding community preferences.The weights assigned to health states

should be interval scaled; that is, themethod ofmeasurement should be one inwhich the ratio of differences betweenvalues is meaningful. (By analogy, theFahrenheit, Celsius, and Kelvin scales areequivalent and appropriate measures oftemperature, because the intervals be¬tween degrees reflect meaningful differ¬ences in temperature.) According to de¬cision theory, preferenceweights obtainedfrom standard gamble questions and, un¬der certain conditions, time trade-offques¬tions satisfy the interval property.At thesame time, psychometric research sug¬gests that rating scales can produce in¬terval data. These claims are mutuallyinconsistent, since there is apparently nota linear relationship between rating scalesand standard gambles or time trade¬offs.20,21It remains an open question whether

standard gambles, time trade-offs, rat¬ing scales, or other measures such as

person trade-offs22 produce the closestapproximation to the type of interval-scaled weights needed for QALYs. Forexample, some research suggests thatrespondents may introduce distortions

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from

into responses to utility questions suchas the standard gamble, compromisingtheir theoretical desirability. Therefore,the panel does not recommend onesource of weights over the others; forpurposes of the reference case, prefer¬ence weights can come from measure¬ment systems that rely on any of thesetechniques. The discrepancies associatedwith different measurement strategiespose a problem for standardization thatwill be important to address in futureresearch.

To date, most CEAs using QALYshave assumed that a year of life gainedby an intervention is valued at 1.0QALY.In fact, people are rarely in the state ofoptimal health that a full QALY implies;the use of a value of 1.0 for years that lifeis prolonged will therefore overstate anintervention's effectiveness and under¬estimate the true C/E ratio. The panelrecommends that, when calculatingQALYs gained from a life-extending in¬tervention, estimates of age- and sex-

specific HRQL should be applied to theyears of extended life—even if the inter¬vention itselfhas no effect on HRQL. Simi¬larly, when estimating QALYs gained byameliorating disease symptoms, a re¬turn to average rather than optimal HRQLshould be credited. It should be noted thatthis use of average quality of life in ref¬erence case analyses means that studiesusing ratios of cost per year (unad¬justed) of life saved will not be compa¬rable to reference case results.

Sociodemographic characteristics, suchas age, sex, or race, are associated withHRQL. When the QALYs produced byan intervention vary as a result of thesesociodemographic differences, referencecase results are affected in ways that maybe ethically problematic. For example, anintervention that extends the lives of 80-year-olds may appear less cost-effectivethan an equally effective intervention ap¬plied to 20-year-olds, not only becausefewer years are gained, but also becausethe average quality of those years is less.In these instances, sensitivity analysesshould be conducted to indicate explic¬itly how results are affected.

Estimating Effectivenessof Interventions

The quality and validity of a CEA de¬pend crucially on the quality of the un¬

derlying data that describe the effective¬ness of interventions and the course ofillness without intervention. Data maybe obtained from primary data collectionefforts specifically intended to inform theCEA or from secondary data sources. Theappropriateness ofvarious sources ofdatawill depend on the purpose ofaCEA. Theconsequences of misestimation of cost-effectiveness may be regarded as more

serious by some decision makers than byothers. For example, the Food and DrugAdministration might desire a greaterlevel of certainty in distinguishing thecost-effectiveness of very similar drugsfor the purpose of reviewing marketingclaims than a formulary manager mightdemand in adopting a new drug.For the purpose of a reference case

analysis, acceptable data for estimationof effectiveness may come from a varietyof sources: randomized controlled trials,observational studies, uncontrolled ex¬

periments, or descriptive series. The ana¬

lyst should select outcome probabilitiesfrom the best-designed (and least-biased)sources that are relevant to the questionand population under study. There areoften trade-offs between the internal va¬lidity of data (optimized in randomizedtrials) and their external validity in ac¬tual practice. Meta-analysis and other syn¬thesis methods can be used when no singlestudy has sufficient power to detect ef¬fects or when studies produce conflictingresults. Expert judgment should be usedonly as placeholders where no adequateempirical data exist, or when the param¬eter of interest plays only a minor role inthe analysis.

Modeling is a valid and necessary sci¬entific procedure for estimating effec¬tiveness for CEA. Typically, data fromrandomized trials are combined with ob¬servational data and public health sta¬tistics in models that are used to esti¬mate changes in life expectancy andquality-adjusted life expectancy. Mod¬els may incorporate features such as lo¬gistic regression to estimate incidenceof disease or death contingent on riskfactors; Bayesian analysis to estimateposttest probabilities of disease fromdata on sensitivity, specificity, andprevalence; and life-table analysis to es¬timate life expectancy from survivalcurves. Models include population andcohort models, deterministic and proba¬bilistic models, decision analysis andstate-transition models. Because of lim¬ited time horizons and selected studypopulations in clinical studies, failure touse models to extrapolate from primarydata can lead to greater error than themodels themselves would introduce.Models should be used as complementsto, not substitutes for, direct primary or

secondary empirical evaluation of effec¬tiveness. Readers are referred to thefull report of the panel for more discus¬sion of the roles of particular types ofdata and models in CEA.3

Time Preference and DiscountingInterest rates reflect people's prefer¬

ence forhavingmoney and material goodssooner rather than later. Similarly, people

value health outcomes that occur in dif¬ferent time periods differently. In CEA,time preference for resources is reflectedby discounting future costs to presentvalue. Discounting the value of future ex¬

penditures requires that health effects ex¬

perienced in the future also be discountedat the same rate. This conclusion is basedon the observation that people have op¬portunities to exchangemoney for health,and vice versa, throughout their Uves. Fail¬ure to discount health effects will lead toinconsistent choices over time; for ex¬

ample, it will appear that delaying invest¬ments will always result in a program'sbecomingmore cost-effective. For this rea¬son and based on other evidence and con¬siderations outlined in its full report,3 thepanel recommends that costs and healthoutcomes occurring during different timeperiods should be discounted to their pres¬ent value and that they should be dis¬counted at the same rate.Although a wide variety of discount

rates are used in the literature and can bedefended, a convention is needed toachieve consistency across analyses. Theo¬retical considerations suggest that thereal (inflation-adjusted) discount rateshould be based on time preference, thedifference in value people assign to eventsoccurring in the present vs the future.Further, economic theory suggests thattime preference is reflected in the rate ofreturn on riskless, long-term securities.Empirical evidence is consistent with thisrate's being in the vicinity of 3% per an¬num (net of inflation). Direct evidence ontime preferences for health outcomes isalso consistent with a discount rate of3%.

The panel therefore recommends theuse of a real, 3% discount rate in thereference case. Before discounting, allcosts should be adjusted for inflation. Be¬cause many published CEAs have used adiscount rate of5%, future analyses shouldinclude sensitivity analysis using 5% aswell as other rates in the range of 0% to7%. The discount rate should be reviewedand possibly revised periodically, to re¬flect important changes in economic con¬ditions. To ensure that analyses will re¬main comparable, however, both 3% and5% should continue to be used for at leastthe next 10 years.

Handling Uncertainty in CEACost-effectiveness analyses are subject

to uncertainty with regard to estimatesof effectiveness, the course of illness,HRQL consequences and preferences,and health care utilization and costs. Us¬ers of analysis need information on thedegree to which CEA conclusions mightchange with changes in assumptions orvalues.

Sensitivity analysis is an appropriatetool with which to respond to this need.

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from

The simplest method, which should beused in all CEAs, is a univariate (1-way)sensitivity analysis, in which estimates orassumptions are changed one at a time.These establish where uncertainty or lackofagreement about some key parameter'svalue could have substantial impact onthe conclusion of a CEA. They suggestareas where efforts to obtain additionaldata might be warranted in terms of im¬pact on decisions and, conversely, areaswhere additional precision would be un¬

likely to change results. For example, theestimated cost-effectiveness of thrombo-lytic therapy in older patients with sus¬

pected myocardial infarction was shownto be stable when the efficacy of therapy,the prevalence of myocardial infarction,or the incidence ofstrokewas varied, thussuggesting that further research to evalu¬ate the risks and benefits ofstreptokinasein this age group was not warranted.23

One-way sensitivity analyses under¬state the overall uncertainty in the C/Eratio; therefore, analysts should also con¬duct multivariate (multiway) sensitivityanalyses, in which several estimates or

assumptions are changed at the same

time, for important parameters. If pos¬sible, a reasonable confidence interval orcredible interval for the C/E ratio shouldbe estimated based either on statisticalmethods or on simulation. The value ofmultivariate sensitivity analysis is great¬est when there is reason to believe thatestimation errors for key parameters are

correlated, for example, if studies thatoverestimated the effectiveness ofthrom-bolysis also tended to underestimate the

associated risk of stroke. Several meth¬ods for performing such statistical analy¬ses and simulations are described in thepanel's full report.3CONCLUSION

The panel recognizes that many of therecommendedmethodological approachesare not broadly practiced at present, andthere may be gaps in the availability ofdata to satisfy the criteria for the refer¬ence case. For example, there is no broadconsensus on which health-state classifi¬cation systems are suitable for CEA, andvalues of community-based weights forsome systems are not publicly available.The ability toweight years oflife by popu¬lation averages of community preferenceweights is limited by the lack of appro¬priate (age- and sex-specific) populationdata for some systems. In the area ofcosts, there are no well-accepted meth¬ods for determining time costs for indi¬viduals outside the labor force, and fewgood-quality data on resource use, reflect¬ing costs rather than charges and clearlyapplicable to the populations under study,are readily available to analysts.

The intention ofthese recommendationsis tomove the field ofCEA closer to stan¬dardization in the near term where pos¬sible and to identify desirable practicewhere optimal methods are not currentlyfeasible. All ofthese recommendations forthe reference case, and for CEAs in gen¬eral, are subject to a "rule of reason."When a parameter estimate or an ele¬ment of the analysis is unlikely to have an

appreciable effect on the result, then it

may be acceptable to use shortcuts to ob¬tain them; expert opinion may be used toassess them; or they may be excludedfrom the analysis altogether. Examplesmay include theweights assigned to short-term and mild impairments of HRQL,costs of unrelated health care in addedyears of life, or the incidence or costs ofminor side effects of treatments. The ruleof reason applies if the cost of obtainingmore precise estimates of the parameterin question would exceed the value ofachievingmore precision in the final cost-effectiveness result. However, the bur¬den is on the analyst to justify suboptimalmethods of parameter estimation or ex¬clusion of effects from an analysis.Ifresearchers endeavor to follow a stan¬

dard set ofmethods in CEA and to obtainthe required inputs for their studies,muchwill have been accomplished toward im¬proving the utility of this form of analy¬sis. It is hoped that the recommendationscontained here will stimulate rapidprogress toward availability of the nec¬

essary data and tools, so that the practiceof CEA can soon become as establishedas many other forms of scientific inquiry.

The Panel on Cost-Effectiveness in Health andMedicine membership and staff: Norman Daniels,PhD; Dennis G. Fryback, PhD; Alan M. Garber,MD, PhD; Marthe R. Gold, MD, MPH; David C.Hadorn, MD; Mark S. Kamlet, PhD; JosephLipscomb, PhD; Bryan R. Luce, PhD; Jeanne S.Mandelblatt, MD, MPH; Willard G. Manning, Jr,PhD; Donald L. Patrick, PhD; Louise B. Russell,PhD; Joanna E. Siegel, ScD; George W. Torrance,PhD; Milton C. Weinstein, PhD.

We thank Kristine I. McCoy, MPH, for coordina¬tion, research, and editorial assistance in associationwith this project.

References1. Udvarhelyi S, Colditz GA, Rai A, Epstein AM.Cost-effectiveness and cost-benefit analyses in themedical literature: are the methods being used cor-

rectly? Ann Intern Med. 1992;116:238-244.2. Russell LB, Gold MR, Siegel JE, Daniels N,Weinstein MC, for the Panel on Cost-Effectivenessin Health and Medicine. The role of cost-effective-ness analysis in health and medicine. JAMA. 1996;276:1172-1177.3. Gold MR, Siegel JE, Russell LB, Weinstein MC,eds. Cost-Effectiveness in Health and Medicine.New York, NY: Oxford University Press; 1996.4. Weinstein MC, Stason WB. Foundations of cost\x=req-\effectiveness analysis for health and medical prac-tices. N Engl J Med. 1977;296:716-721.5. Garber AM, Phelps CE. Economic foundationsof cost-effectiveness analysis. J Health Econ. Inpress.6. Siegel JE, Weinstein MC, Russell LB, Gold MR,for the Panel on Cost-Effectiveness in Health andMedicine. Recommendations for reporting cost-ef-fectiveness analyses. JAMA. In press.7. Newhouse JP. Medical care costs: how muchwelfare loss? J Econ Perspect. 1992;6:3-21.8. Koopmanschap MA, Rutten FFH, van IneveldBM, van Roijen L. The friction cost method for

measuring indirect costs of disease. JHealth Econ.1995;14:171-189.9. Weinstein MC, Stason WB. Hypertension: APolicy Perspective. Cambridge, Mass: Harvard Uni-versity Press; 1976.10. Drummond MF, Stoddart GL, Torrance GW.Methods for the Economic Evaluation of HealthCare Programmes. Oxford, England: Oxford Uni-versity Press; 1987.11. Russell LB. Is Prevention Better Than Cure?Washington, DC: Brookings Institution; 1986.12. Meltzer DO. Accounting for future costs in medi-cal cost-effectiveness analysis.JHealth Econ. In press.13. Torrance GW, Feeny D. Utilities and quality-$adjusted life years. Int J Technol Assess HealthCare. 1989;5:559-575.14. Patrick DL, Erickson P. Health Status andHealth Policy: AllocatingResources toHealth Care.New York, NY: Oxford University Press; 1993.15. Torrance GW, Boyle MH, Horwood SP. Appli-cation of multiattribute utility theory to measuresocial preferences for health states. Operations Res.1982;30:1043-1069.16. EuroQol Group. EuroQol: a new facility for themeasurement ofhealth-related quality of life. HealthPolicy. 1990;16:199.

17. Kaplan RM, Anderson JP. A general healthpolicy model: update and applications. Health ServRes. 1988;23:203-235.18. Erickson P, Wilson R, Shannon I. Years ofHealthy Life. Hyattsville, Md: Centers for DiseaseControl and Prevention, National Center for HealthStatistics; 1995. Dept of Health and Human Ser-vices publication (PHS) 95-1237.19. Ware JE Jr, Sherbourne CD. The MOS 36-ItemShort-Form Health Survey, I: conceptual frameworkand item selection. Med Care. 1992;30:473-483.20. Torrance GW. Social preference for healthstates: an empirical evaluation of three measure-ment techniques. Socio-economic Plann Sci. 1976;10:129-136.21. Read JL, Quinn RJ, Berwick DM, FinebergHV, Weinstein MC. Preferences for health out-$comes: comparison of assessment methods. MedDecis Making. 1984;4:315-329.22. Nord E. Methods for quality adjustment of lifeyears. Soc Sci Med. 1992;34:559-569.23. Krumholz HM, Pasternak RC, Weinstein MC,et al. Cost effectiveness of thrombolytic therapywith streptokinase in elderly patients with sus-

pected acute myocardial infarction. N Engl J Med.1992;327:7-13.

at STANFORD Univ Med Center on August 4, 2011jama.ama-assn.orgDownloaded from