NUREG-1563

Branch Technical Position on theUse of Expert Elicitation in theHigh-Level RadioactiveWaste Program

Manuscript Completed: August 1996Date Published: November 1996

J. P. Kotra, M. P. Lee, N. A. Eisenberg, NRCA. R. De~ispelare, CNWRA

Center for Nuclear Waste Regulatory AnalysesSan Antonio, IX 78228-0510

Division of Waste ManagementOffice of Nuclear Material Safety and SafeguardsU.S. Nuclear Regulatory CommissionWashington, DC 20555-0001

AlESTRACT

The U.S. Department of Energy (DOE) isconducting a program of site characterization togather enough information, about the YuccaMountain (Nevada) site, to be able to evaluate thewaste isolation capabilities of a potential geologicrepository. Should the site be found suitable,DOE will apply to the U.S. Nuclear RegulatoryCommission for permission to construct and thenoperate a proposed geologic repository for thedisposal of spent nuclear fuel and other high-levelradioactive waste at Yucca Mountain. In decidingwhether to grant or deny DOE's licenseapplication for a geologic repository, NRC willclosely examine the facts and expert judgment setforth in any potential DOE license application.NRC expects that subjective judgments ofindividual experts and, in some cases, groups ofexperts, will be used by DOE to interpret dataobtained during site characterization and toaddress the many technical issues and inherentuncertainties associated with predicting theperformance of a repository system for thousandsof years. NRC has traditionally accepted, for

review, expert judgment to evaluate and interpretthe factual bases of license applications and isexpected to give appropriate consideration to thejudgments of DOE's experts regarding thegeologic repository. Such consideration, however,envisions DOE using expert judgments tocomplement and supplement other sources ofscientific and technical information, such as datacollection, analyses, and experimentation. In thisdocument, the NRC staff has set forth technicalpositions that: (1) provide general guidelines onthose circumstances that may warrant the use of aformal process for obtaining the judgments ofmore than one expert (i.e., expert elicitation); and(2) describe acceptable procedures for conductingexpert elicitation when formally elicited judgmentsare used to support a demonstration ofcompliance with NRC's geologic disposalregulation, currently set forth in 10 CFR Part 60.

In this NUREG, the staff also provides anexpanded definition of "peer review" over thatprovided earlier in NUJREG-1297.

iii NUREG-1563

CONTENTS

Page

ABSTRACT ...................................................................... iii

FOREWORD............................................................................. vii

ACKNOWLEDGMENTS .................................................................. ix

1 INTRODUCTION ...................................................................... 1

1.1 Background ........................................................................ 2

1.2 What are Expert Judgment, Expert Elicitation, and Peer Review? ........................ 3

1.2.1 Expert Judgment . ............................................................. 3

1.2.2 Expert Elicitation . ............................................................. 3

1.2.3 Peer Review .................................................................. 5

1.3 Selected Examples of NRC Use of Expert Elicitation ............................... .5

13.1 Severe Accident Risk Analysis ................................................. 5

1.3.2 Seismic Hazard Analysis ................. ..................................... 6

1.3.3 HLW 'Tchnical Analyses ................. ..................................... 7

1.4 Purpose of the BTP . ................................................................ 71.5 BTPs as Technical Guidance ..................... .................................... 9

2 REGULATORY FRAMEWORK ......................................................... 11

2.1 10 CFR Part 60 ..................................................................... 11

2.2 Commission Policy Statement Concerning the Use of PRA Methods in NuclearRegulatory Activities . ................................................................ 13

3 BRANCH TECHNICAL POSITIONS .................................................... 15

4 DISCUSSION ...................................................................... 19

5 REFERENCES ........................................................................ 31

FIGURE

1 Components in An Acceptable Expert Elicitation Process .................................. 16

v NUREG-1563

FOREWORD

This Branch Technical Position (BTP) wasdeveloped only for application to the high-levelwaste program, as part of the staff's IterativePerformance Assessment efforts. The technicalpositions contained here reflect staff experiencegained from both monitoring the U.S. Departmentof Energy's site characterization program atYucca Mountain, Nevada, and developing the U.S.Nuclear Regulatory Commission's independentregulatory capability.

Although there are several examples of the use ofexpert elicitation in a nuclear regulatory context,no formal Agency guidance on this subject exists.Thus, in developing this BT, the Division ofWaste Management (DWM) staff has also drawnfrom previous staff experience of other NRCprogram offices, in the use of expert elicitation. In

this regard, DWM staff has relied on certainAgency resource documents, such as: "RiskAssessment: A Survey of Characteristics,Applications, and Methods Used by FederalAgencies for Engineered Systems"; 'A Review ofNRC Staff Uses of Probabilistic Risk Assess-ment"; and "Recommendations for ProbabilisticSeismic Hazard Analysis: Guidance onUncertainty and Use of Experts," to helpformulate its position statements. Consequently,the reader will find that this BTP is largelyconsistent with these other resource documents, insubstance.

Subsequent to the finalization of this BTP, thestaff may elect to develop guidance on the use ofexpert judgment in other areas of nuclearregulatory regulation.

vii NUREG-1563

ACKNOWLEDGMENTS

The authors gratefully acknowledge substantialcontributions from the staffs of the Center forNuclear Waste Regulatory Analyses (CNWRA),the Division of Waste Management (DWM), theOffice of the General Counsel, the Office ofNuclear Reactor Regulation (NRR), the Office ofNuclear Regulatory Research (RES), and theOffice for Analysis and Evaluation of OperationalData (AEOD), which have been incorporated intothe text of this Branch Technical Position. In this'regard, the following staff deserve specialrecognition for their comments and/or reviews:

Robert G. Baca/CNWRAJoseph A. Murphy/RESEJ. Bonano/Beta Corporation InternationalJohn D. Randall/RESMichael C. Cheok/NRRCharles E. Rossi/AEOD

Anthony H. Hsia/NRRPhyllis A. Sobel/DWMAK. IbrahimlDWMJohn S. Trapp/DWMChristiana H. Lui/DWMSandra L. Wastler/DWMSteven E. Mayi/AEOD-Michael F. Veber/DWMTimothy J. McCartin/DWMErnst G. Zurflueh/RESKeith I. McConnell/DWM

Thanks also to George A. Apostolakis, Depart-ment of Nuclear Enginebring/MassachusettsInstitute of Technology, under a grant from RES,for his review comments. Ellen Kraus providededitorial guidance.

L5; NUREG-1563

APPENDICES

Page

A GLOSSARY ........................................ A-1

B AN ELICITATION ON FUrtURE CLIMATE: LESSONS LEARNED .B-1

C FINAL COMMISSION POLICY STATEMENT ON THE USE OF PRAMETHODS IN NUCLEAR REGULATORY ACTIVITIES .C-1

D DISPOSITION OF PUBLIC COMMENTS ON FEBRUARY 28, 1996,DRAFT BRANCH TECHNICAL POSITION ......................................... D-1

E STAFF VIEWS ON DOE'S 1995 PRINCIPLES AND GUIDELINES AND THEPOSSIBLE DISPOSITION OF SCA COMMENT 3 ................. E-1

F DISPOSITION OF ACNW` COMMENTS ................. F-1

NUREG-1563 x

1 INTRODUCTION -:

The U.S.'Nucleai Regulatory Commission'sregulations for the licensing of a geologicrepository for the disposal of spent nuclear fueland other high-level radioactive waste (HLWV) areintentionally non-prescriptive in that 10 CFR Part60 leaves to the U.S. Department of Energy.(DOE) the opportunity and responsibility todetermine how best to site and design a potentialgeologic repository that can meet the performanceobjectives contained in Subpart E of thoseregulations. DOE is conducting a program of sitecharacterization, at Yucca Mountain, Nevada.'Through this program, DOE is gatheringinformation on the characteristics of the YuccaMountain site, to evaluate the waste isolation

- capabilities of the proposed site, as it wouldperform in concert with DOE's repository design,in order to meet the performance objectives.DOE will need to interpret the geologic recordand demonstrate that the repository site anddesign will comply with explicit numericalperformance standards established by the U.S.Environmental Protection Agency (EPA) andadopted by NRC.1

For the Commission's part, if it is to reach anaffirmative licensing decision, the Commissionwill need to confirm that the numericalperformance standards have been met and must

.satisfy itself that DOE's analyses of the'site anddesign are sufficiently convincing, that'their.limitations are well understood, and that DOEhas demonstrated that its analyses have madeappropriate allowance for the time period,hazards;and uncertainties involved.

Nearly every aspect of site characterization andperformance assessment will involve'significantuncertainties. The primary method to evaluate,'and perhaps reduce, these uncertainties' should becollection of sufficient data and informationduring site characterization. However, factors

Ibe staff recognizes that revised EPA standards, specific to theYucca Mountain site, required by the Energy Policy Act of 1992(Public Law 102486) must be based on and consistent with recentfindings and recommendations of the National Academy of Sciences(NAS-see National Research Council, 1995a) Once EPA promul-gates regulations establishing its final standards, NRC will modify itsrequirements at 10 CFR Part 60 to conform to the new standards.Notwithstanding these forthcoming revisions to both EPA's;standards and NRC's conforming regulations, the staff presumesthat there will continue to be a requirement for some kind of aquantitative performance assessment to estimate the long-term,pet5-closure performance of the overall repository system.,

such as temporal and spatial variations in thedata, the possibility for multiple interpretations ofthe same data; and the absence of validatedtheories for predicting the performance of a.repository for thousands 'of years, will make itnecessary to complement and supplement thedata obtained during site characterization withthe interpretations and subjective judgments oftechnical experts (i.e., expert judgments-seedefinition in Section 1.2.1).

Expert judgment is ubiquitous in almost everyscientific or technical endeavor. As important asits role may be, however, for the purposes'of theHLW program, the subjective judgment of expertsshould be distinguished from both measured dataor technical 'calculations based on'acceptedscientific laws and principles. It should be viewedas an alternative, and employed when other meansof obtaining requisite data or information havebeei thoroughly considered and it has beenconcluded that such means are not practical toimplement. Thus, expert judgment, informal aswell as formally elicited, may be used by DOE inits 'demonstrations of compliance with NRC'sgeologic disposal regulation. Moreover, the NRCstaff will continue to accept, for review, thosecompliance demonstrations and other analysesemploying inf6rmal expert judgment and expertelicitation.

With this notion in mind, current NRC policy is toencourage the use of probabilistic risk assessment(PRA) state-of-the-art'technology and methods asa compleiment to the deterministic approach innuclear regulatory activities (NRC, 1995; 60 FR42622)2 and in keeping with the Commission'sparamount responsibility to protect public healthand safety. Expert judgment may provide anessential part of the information used'in PRAs (orperformance assessments,- in the case of waste.management' systems),'and may also be used indeterministic analyses. Consistent with theCommission's' policy, the NRC staff has developedthis' Branch Technical Position (BTP) to identifyacceptable procedures for the formal elicitation of2EPA's 1985 HLW standards (50 FR 38066) adopted a probabilistic

perspective when making compliance determinations. Because ofthe uncertainties inherent in the geologic disposal of nuclear waste,it is anticipated that a probabilistic treatment of the performance ofthe waste disposal system will continue to be the regulatoryapproach.

1 1 ~~~~~NUREG-1563

such judgments in the area of HLW The staffbelieves that PRA methods, such as described inthis guidance, can be applied to the HLWprogram in a manner that would contributesignificantly to the necessary confidence that ageologic repository could be licensed withoutundue risk to the health and safety of the public.

1.1 Background

A geologic repository is a complex system, thefuture performance of which must be predictedover many thousands of years. Because of thenature of the task and the limitations of scientificunderstanding in many pertinerit technical fields,DOE (and its contractors) will use expert judgmentthroughout site characterization ad later, in thepreparation of a license application, to construct apotential geologic repository for HLW. DOE andits contractors will also use expert judgment in theidentification and screening of events andscenarios; development and selection of modelsthat describe the geology and hydrology of therepository system; assessment of modelparameters; collection of data; assessment ofvolcanic and seismic hazard potential; and forstrategic decision-making, about the repository'sdesign, that could affect its long-termperformance (e.g., DOE, 1986; Dennis, 1991;Seismic Methods Peer Review Panel,- 1991; TestPrioritization Task Force, 1991; DOE, 1991b;Barnard et al., 1992; Andrews et al.; 1994, Wilsonet al., 1994; and Schenker et al., 1995).

Although acquisition and analysis of physical datashould be the primary manner in which licensinginformation is collected, many considerations maypreclude the collection of such informationnecessary for licensing. As with all complextechnical analyses, "professional" judgment,usually informal and implicit, is used routinely bythe scientists, engineers, and technical programmanagers who contribute to the repositoryprogram, to supplement and interpret thisinformation, indeed, even to determine how toobtain the data or perform the analyses. The staffbelieves that its ability to evaluate a potentialSLW license application will, in large measure,depend on the transparency with which data arecollected, analyzed, and interpreted, andsafety-related decisions are made. Therefore, thestaff believes that it is important for all programparticipants to have a common understanding of

the general circumstances under which it may beworthwhile to obtain and apply expert judgmentsin a more formal manner, and of the appropriatemethods for doing so.

In reviewing DOE's Site Characterization Plan(SCP-see DOE, 1988), the staff identifiedconcerns related to DOE's proposed use of expertjudgment as part of a potential licenseapplication. In its Site Characterization Analysis(SCA), the staff noted that DOE's SCP relied tooheavily on the elicitation of expert judgment as asubstitute for quantitative data and analyses(NRC, 1989; pp. 4-8-4-10). Subsequently, theNRC staff has criticized specific DOE uses ofexpert elicitation in the Calico Hills Risk/BenefitAnalysis (DOE, 1991a) and the Early SiteSuitability Study (Science ApplicationsInternational Corporation, 1992). The NRC stafffound fault with both the manner in which theseelicitations were conducted and the way in whichthe elicitation results were used to make sitecharacterization decisions (see Linehan, 1990 and1991; and Holonich, 1992). As with other types ofinformation, expert judgment can be mis-interpreted, misrepresented, and misused. TheNRC staff is particularly concerned with thepotential for over-reliance on expert judgment asa basis for decision-making, as well as itspotential misuse as an inappropriate justificationto avoid gathering additional objective data.Since 1990, the Nuclear Waste Technical ReviewBoard (NWTRB) and the NAS have alsoaddressed DOE's plans to use expert judgment.Both the NWIRB and the NAS, independently,have expressed concerns with these plans and, inparticular, with how DOE addresses the potentialfor "bias" and "conflicts of interest" whenconducting expert eicitations (see NWTRB(1990a, p. 21; 1990b, p. 26; 1991, pp. 29-30; 1994,pp. 31-35) and National Research Council (1990,p. 24)).3 For example, DOE has been criticized forrelying almost entirely on its own scientific expertsand contractors, with little or no external peerreview.

More recently, DOE has announced a series ofinitiatives that would lead to the restructuring ofits geologic repository program, including sitecharacterization activities (see DOE, 1994 (pp.1-3-1-4) and 1995 (pp. 1-3-1-8)). The exact

3DOE's response to some of the Board's recommendations arereprintedinNTRB (1992 (pp. E-11-E-12) and 1995 (pp.H-20-H1-2 1)).

NUREG-1563 2

details of the re-structuring plan have not yet beendisclosed, nor have the impacts of recent budgetconstraints imposed on DOE's repository.program been fully evaluated in the 'context of thisplan. That being said, the staff generally ' 'understands that DOE will, both by intent andnecessity,'c6llect fewer data and, instead, relyextensively on bounding analyses to both form itsHLW programmatic decisions and to support apotential license application it would submitpursuant to NRC's geologic disposal regulation(see DOE, 1994; pp. 1-3-14).

1.2 What are Expert Judgment, ExpertElicitation, and Peer Review?

1.2.1 Expert Judgment

Expert judgment is infornation,4 provided by atechnical expert, in his or her subject matter areaof expertise, based on opinion, or on a beliefbased on reasoning. Questions are usually posedto experts because they cannot be'answered byother means. Expert judgments can be evaluationsof theories, models, or experiments, or theyjcan berecommendations for further research. Expert'judgments may also be opinions that can be"analyzed and interpreted, and used in subsequenttechnical assessments. Expert judgments can beeither qualitative or quantitative. Expertjudgments can also be judgments about uncertainquantities or judgments about value preferences.Expert judgment has also been called expertopinion, subjective judgment, expert forecast, bestestinate, educated guess, and, most recently expert

. knowledge (see Meyer and Booker, 1990; p. 3).'Regardless of how one defines it, expert judgmentultimately reflects the technical expert's evaluationand interpretation of some scientific knowledge

' base, to the extent that the knowledge' base exists.Moreover, expert judgment does-not createknowledge, rather it "synthesizes disparate andoften conflicting sources of information toproduce an integrated picture" (see Hora,' 1993).

The distinction between judgmental information(e.g., is there life on other planets ?) and morestraightforward, factual information (e.g., what is£ dr/x ?) is not sharp. The use of expert judgmentin technical and scientific work, including that

4 Expert judgment is sometimes referred to as 'data" (e.g., for pur-poses of aggregating the judgments of multiple experts).

routinely reviewed by the NRC staff, isubiquitous.' This'infonnal use of expert judgmentis implicit in the choice of mathematicalequations to describe a system, the methods usedfor testing, the interpretation of data, the methods'used to obtain numerical results, and otheraspects of analysis and testing.

Judicial and administrative proceedings frequentlyinvolve "expert witnesses" who, by virtue of theirtraining and/or experience, are judged capable ofproviding useful opinions or conclusions aboutcertain matters in issue. These experts performtheir roles under the rules governing the particularproceeding, whether it be a judicial or anadministrative setting. However, in this BTP, theword "expert" will be used more broadly and withreference to the "pre-licensing" and "licensingphases" (Johnson, 1994) of the HLW program. Inthe context of this BT, "experts" refer to thoseknowledgeable individuals, in engineering andscience, who, by the nature'6f their experienceand academic achievement,'can' speak to theunderstanding of certain scientific laws and.principles. This BTP makes no attempt to identifywho would be an "expert"' for.the purposes of ajudicial or an administrative proceeding.However, an "expert" used in an' elicitationprocess may also be used as an "expert" in ajudicial proceeding.

1.2.2 Expert Elicitation1.. .2 .x

Expert elicitation is a formal, highly structured,and well-documented process whereby expertjudgments,'usuallyof multiple experts, areobtained. Although informal expert judgmentinvolves 'only subject-matter experts, formal expertelicitations usually involve normative experts,generalists, and subject-matter experts (see -

Appendix A). The normative expert has trainingand experience in statistics, decision analysis, andprobability encoding; this expert's main functionis to structure the formal elicitation and train thesubject-matter experts in probability encoding.The generalist understands the context in whichthe results of the expert elicitation will be used,-guides the structure of the elicitation to producethe needed results, provides relevant informationand 'documentation to the subject-matter experts,

' and helps to train them. Often the generalist'sexpertise overlaps that of the subject-matterexperts or is in a closely-related, allied field. Forexample, a performance assessment expert could

3 NUREG-1563

be the generalist in an expert elicitation forclimate change. The subject-matter experts, ofcourse, provide the subjective judgments that areusually encoded as probabilities (e.g., aprobability distribution or a point estimate for anuncertain parameter).

In the context of the HLW program, expertelicitation may play several roles in DOE'sdecision-making process for a potential geologicrepository. Typically an elicitation is conducted toevaluate uncertainty. The uncertainty could beassociated with: the value of a parameter to beused in a model; the likelihood and frequency ofvarious future events; or the relative merits ofalternative conceptual models. In each of thesecases, the information regarding uncertaintywould be represented by encoding the subjectiveprobabilities from each subject-matter expert. Forexample, a few boreholes may yield data necessaryto calculate the permeability of a rock type in aparticular hydrogeologic unit. Given that thesedata represent a small sample from a highlyvariable population and given further thatundiscovered structures and features could greatlyinfluence the effective permeability of the unit,each expert might be asked to generate aprobability distribution for the effectivepermeability. of the hydrogeologic unit. Similarly,probabilities may be attached to the likelihood ofvarious futures, frequencies of various events, andvalidity of various models. These probabilitydistributions could be used as direct input toprobabilistic performance assessments.Alternatively,. discrete distributions describingalternative conceptual models could be used tocharacterize the outcomes of analyses using eachof the alternative concepts. In either event, theoutputs of these performance assessments areexpected to provide important inputs to DOE'songoing site characterization and designprocesses, as well as to indicate the relativeimportance of various programmatic activities forperformance assessment. In addition, these"iterative" assessments are expected to provideimportant feedback on the nature and importanceof improvements to be made in the analyticalbasis for the performance assessment. Ultimatelythe performance assessment, supported in part byexpert elicitation, is expected to be the centralfocus of any potential DOE license application,wherein the various lines of evidence (field data,

laboratory experiments, natural analogues,theoretical and semi-empirical analyses) aredrawn together in an effort to demonstratecompliance. Because the NRC staff has aninterest and oversight role5 in these variousaspects of performance assessment and theassociated expert elicitation(s), the staff hasdeveloped this guidance document.

Finally, it should be noted that NRC evaluation ofany potential DOE license application wouldinvolve an opportunity for a hearing. If a hearingis requested and if the Commission appoints anAtomic Safety and Licensing Board to preside atthe hearing, then that Licensing Board will beresponsible for conducting a hearing at whichevidence is taken on contested issues. TheLicensing Board has considerable liberty in theselection of those values or judgments it woulduse to reach any potential licensing decision.Although the Board's decision can be based onlyon evidence in the record, it can exercise widediscretion in the weight it attributes to anyparticular piece of evidence. Thus, for anyparticular issue or case, the staff cannot predictwhat weight, if any, the Licensing Board wouldattribute to expert opinion derived from an expertelicitation, as opposed to expert opinion obtainedby other means.

51n addition to the review of the site characterization activitiesspecified under 10 CFR 60.18, the Commission noted in its finalrule that it contemplated an ongoing review of information on siteinvestigation and site characterization, such as those with longlead-time procurement actions, so as to allow for the early identifi-cation and resolution of potential licensing issues. Moreover, NRC'sstrategic planning assumptions call for the early identification andresolution, to the extent practicable, at the staff level, before thereceipt of a potential license application for a geologic repository(see Johnson, 1994). The principal means for achieving this goal isthrough informal, pre-licensing consultation with DOE the State ofNevada, Indian Thbes, and affected units of local government. Thisapproach attempts to reduce the number of, and to better define,issues that will be litigated during a potential licensing hearing, byobtaining input and striving for consensus from the technicalcommunity, interested parties, or other targeted groups on suchissues.

In this regard, the staff has undertaken the development of thisBTP as a means for closure on acceptable procedures for con-ducting expert elicitation when formally elicited judgments are usedto support a demonstration of compliance with NRC's geologicdisposal regulations The staff believes that rigid adherence to thespecific steps proposed in the IP is not sought so much as the useof a consistent process that produces an accurate and properlydocumented assessment of the state of scientific uncertainty. More-over, the staff believes that effective implementation of a goodelicitation process cannot guarantee acceptance of the technicalconclusions; however, use of a flawed process or improper imple-mentation of a good process cannot help but cast serious doubt onthe quality of the conclusions.

NUREG-1563 4

1.23 Peer Review6

Much of scientific and engineering development issubjected to the normal review process of criticalevaluation by colleagues in various venues. Theseso-called peer reviews are typically documented,- critical reviews that evaluate the acceptability andadequacy of some particular form of originalresearch, performed by peers who are :independent of the work being reviewed. A peerreview can be conducted by obtaining inputseparately from a number of peers or byconvening a panel to conduct the review. Also,discussions among the panel members cangenerate useful information not available from aset of independent reviews. The most common...peer review process (i.e., pre-publication technicalreview of a scientific article) typically usesinformal expert judgment to evaluate scientificmethods and results. However, in principle, the -nature of peer review is sufficiently flexible that itsrigor and formality are commensurate with thestudy being reviewed. For example, the NAS isfrequently called on to review reports orconclusions as a group of technical experts (seeNational Research Council, 1995b and 1995c).Peer reviews can also be conducted using a formalprocess' to 'review the solution'of problems of highimportance. Formal'peer review ha'some of thesame basic attributes of the formal expertelicitation process (e.g., disclosure of potential"'conflicts of panelists, documentation behinddecision-making).'

The peers are recognized experts in the'domain ofinterest as evidenced by their scientificqualifications. The' peers may comment on thevalidity of the assumptions, the appropriatenessand limitations 'of the methodology andprocedures; the accuracy of the calculations, the'validity of the conclusions, and the uncertainty of

-the results and consequences of the work; Theymay also offer -alternative explanations of the!results and comment on the adequacy of theinformation and data used to obtain them.

The peer review process requires the expertjudgments of peers. However, it is important tonote that peer review as an expert judgmentprocess is different from' the formal elicitation ofexpert judgments in the context of this staff

6Tbis discussion is an cxpansion of the earlier definition of peerreview provided by Altnan et al (1988, p. 2).

position. The reference to expert judgment hereindenotes judgments, opinions, or informationprovided by subject-matter experts that give riseto or contribute to the generation of a scientificstance or solution to a given problem. Peer review,by contrast,'seeks judgments from subject-imatterexperts regarding the soundness and quality of an*existing or proposed scientific stance or solutionto a'problem. In this context, expert judgments'can be the subject of peer review. The admittedlysubtle differences between the elicitation of expertjudgment and independent peer reviewnotwithstanding, both processes contribute in apositive way to enhancing quality.

1.3 Selected Examples of NRC Use ofExpert Elicitation

In addition to reviewing DOE's site character-'ization activities, the NRC staff has drawn onexperience obtained in' other NRC regulatoryprograms (see PRA Working Group, 1994) andhas-be'en exploring, independently, the ways inwhich expert elicitation' may be applied in thegeologic repository program. Some of the staff'sprogram activities are summarized below.

13.1 Severe'Accident Risk Analysis

The formal use of expert elicitation in NRC PRAswas introduced, during the mid-1980s, with thedevelopment of NUREG-1150, an assessment ofsevere accident risk at five U.S. nuclear powerplants (NRC, 1990). Earlier, in 1975, NRC hadcompleted its first study of the probabilities andconsequences of severe reactor accidents at twocommercial nuclear power reactors. This work, forthe first time, used the analytical technique ofPRA for the study of core meltdown accidents(see NRC, 1975). After completion of these firstPRAs, NRC, industry groups, and the utilities

-initiated programs to improve PRA technology tomeasure and enhance nuclear power reactorsafety, and NRC gradually introduced rRAs intoits regulatory process.

In the late 1980s, the staff updated the 1975 PRAand in doing so, reassessed the 1975 estimates,using improved PRA techniques; the results ofthis reassessment were documented inNUREG- 1150. One of the major accomplish-ments of the NUREG-1150 study, whichcontinues to be one of the most sophisticated'

. 5 ~~~~~NUREG-1563

I

applications of PRA performed by the staff todate, was the adoption of a formal protocol toelicit expert judgment in areas of the risk studieswhere little or no operational data existed. Expertjudgment was needed to supplement and interpretthe available data, and to explicitly address theuncertainties in the analysis itself. The elicitationprocess relied on a formal set of procedures thatare described in detail by Gorham-Bergeron et al.(1986).

This approach was subsequently reviewed andmodified,7 based, in large part, onrecommendations made by Kouts et al. (1987) andKastenberg et al. (1988) after peer reviews of thefirst draft of NUREG-1150. Based on theserecommendations, the elicitation process for thefinal NUREG-1150 report was made more formaland rigorous by the identification of nine discreteprocess steps; these process steps are described inAppendix A of the final report.

More recently, expert elicitation was applied touncertainty assessment for two new probabilisticaccident consequence codes. In a joint effort withthe Commission of European Communities, theNRC Office of Nuclear Regulatory Researchconducted a formal expert elicitation of 16international experts to develop a library ofuncertainty distributions for selected consequenceparameters. Distributions of measurableatmospheric dispersion and depositionparameters were successfully elicited from 16international experts involved in the manyphenomenological areas of consequence analysis(see Harper et al, 1995). For the most part, theelicitation protocol used by the NRC/Commissionof European Communities followed the sameprinciples that guided the NUREG-1150 process.One noteworthy exception, however, was theexpectation in the later study that, should itbecome necessary (e.g., to support an independentpeer review), the subject-matter experts should bewilling to be identified with their elicitedprobabilities and the rationales for thoseprobabilities. In NUREG-1150, experts werepermitted complete anonymity if they so desired.

1.3.2 Seismic Hazard Analysis

Expert elicitation is also widely recognized asintegral to probabilistic seismic hazard analyses(see National Research Council, 1988). In themid-1980s, NRC sponsored a major study ofprobabilistic seismic hazard in the eastern UnitedStates conducted by Lawrence LivermoreNational Laboratory (LLNL-see Bernreuter etal., 1980-83, 1985, and 1989). This study of seismichazard at 69 reactor sites relied heavily on thejudgments of experts whose interpretations ofgeophysical, seismological, and geologic data wereindividually obtained using a formal elicitationprocess. In conjunction with funding the LLNLstudy, NRC recommended that the nuclear powerindustry perform an independent study to providea coordinated utility position on seismic hazardestimates. The Electric Power Research Institute(EPRI), with funding from a consortium ofnuclear power utilities, developed its ownelicitation methodology and applied it to hazardestimates for 56 of the 69 reactor sites examinedby LLNL (see Seismicity Owners Group/EPRI,1986).

In both the LLNL and EPRI methodologies,seismic hazard curves were developed for U.S.commercial nuclear power reactor sites east of theRocky Mountains, using expert judgment tointerpret the available data. Both the LLNL andthe EPRI programs were based on the premisethat available geologic data were not sufficient tofully predict seismic phenomena, and theyadopted the use of expert opinion to characterizethe uncertainties in the data. Although bothmethodologies used essentially the same sets ofdata, the two methodologies producedsignificantly different results. The differences inthe approaches and results were subsequentlyreviewed and critically evaluated (see Bernreuteret al., 1987). It was concluded that, in general, thediffering results could be attributed to howelicited information was aggregated in therespective elicitation processes (op cit., pp.254-258). LLNL has subsequently performed are-elicitation of the seismicity and ground motionexperts to improve its earlier estimates ofuncertainty in seismicity parameters and groundmotion models (see Sobel, 1994).

Most recently, in a separate effort jointlysponsored by NRC, DOE, and EPRI, a "SeniorSeismic Hazard Analysis Committee" hasdeveloped methodological guidance on how best

7Gorham-Bergeron et aL (1986) was available in draft form (asNUREG/ClR-4551 in the Public Document Room) as a comple-mentary report to NUREG-1150, when NUREG-1150 was firstissued in draft form, for public comment, in 1987. The final versionof NUREG/CR-4551 was published as Gorham ct aL (1993).

NUREG-1563 6

to perform a probabilistic seismic hazard analysis,with special emphasis on the formal process foreliciting expert opinion. Results of this projectwere published in late 1995 (see Senior SeismicHazard Analysis Committee (SSHAC), 1995) andare currently being reviewed by an NAScommittee.

Appendix A ("Geologic Siting and Design Criteriafor Nuclear Power Plants") to 10 CFR Part 100contains existing NRC siting and design policy;related to geological and seismological hazards forcommercial nuclear power reactors. Inconjunction with the Standard Review Plan for'nuclear power reactors and'other applicableregulatory guides, Appendix A'sets forth aregulatory fraimework that guides the NRC staff inits evaluation of the adequacy of an applicant'sinvestigations of geologic phenomena andproposed desigri parameters for nuclear powerreactors. Also, independent spent fuel storageinstallations, monitored retrieval storage systems,and mine-tailings'dams for uranium. processing'mills refer to Appendix A for guidance on faultingand seismic criteria. Recently, the Commissionproposed revisions to the requirements andapplication of Appendix A'to 10 CFR Part 100'(see NRC, 1994; 59 FR 52255). Under review aspart of this reassessment are recommendationsthat NRC's geological and seismologicalinvestigations and design criteria be modified toreflect better the current state of the art in theseareas. Although probabilistic seismic hazardassessments have been discussed previously inSafety Evaluation Reports for nuclear powerreactors, these revised requirements, as proposed,would explicitly recognize the use of probabilistictechniques, as'a valid means to assess uncertaintyassociated with the analysis of and'engineeringdesign for seismic phenomena. Specifically, theCommission has proposed the use of either of theprobabilistic methodologies independentlydeveloped by LLNL'or EPRI as aceptableapproaches to the evaluation of seismic hazard,uncertainty.' Because both methodologies rely,extensively on the formal elicitation of expertjudgment to assess uncertainty in seismic hazardestimates, this proposed rule; if promulgated infinal form,' would represent the Commission's firstexplicit regulatory recognition of the value offormal expert elicitation as a technique forassessing uncertainty.

1.33 HLW Technical Analyses

The process of formally eliciting the judgments'ofmultiple experts has also been studied andapplied in support of the staff's independentperformance assessment activities. For example,Bonano et al. (1990) discussed the state of the artof formal expert elicitation and its possible'application to HLW performance assessments.DeWispelare et al. (1993) subsequently 'aplied theformal process of elicitation to the prediction offuture climate, with associated parameterdistributions, at Yucca Mountain, and to theestimation of corresponding probabilities'ofoccurrence. As part of the NRC staff'sindependent Iterative Performance Assessmentefforts to develop a performance assessmentreview capability, the staff has relied on informalelicitations to identify and screen scenarios (seeCodell et al. (1992) and Wescott et al. (1995)). Astructured elicitation process was used by'the'staff to evaluate potential quantitative criteria toclarify the ".. substantially complete contain-ment requirement" (10 CFR 60.113(a)(1)(i)(A)-see Tchoepe and Abramson, 1992). Broader

'.applications of expert judgment were exanined byDeWispelare et al. (1994) who identified situationswhere the use of expert judgment might beappropriate in the HLW program.'Fro nthisreview, the staff was able to verify' that lessonslearned from'these direct experiences hadrelevance for broader. applications. In pursuingthese reviews of prior Agency experience withexpert elicitation, as well as its own elicitationactivities, the NRC staff has acquired a betterunderstanding of the critical issues associatedwith the development and use of expert elicitation(and, more generally,-expert judgment).'

1.4 Purpose of the BTPThe NRC staff recognizes that expert judgment isimplicit in all scieritific inquiry and engineeringendeavors, and is generally applied in an informalmanner. It is the purpose of this BTP to:(1) provide general guidelines on thosecircumstances that may. warrant the use of aformal process for obtaining the judgments ofmore than one expert (i.e., expert elicitation); and(2) describe'acceptable procedures for conductingexpert elicitation when formally elicited judgmentsare used to support a demonstration ofcompliance with NRC's geologic repositorydisposal regulations. Included in this BTP is a

7 NUREG-1563

I

recommended procedure for selecting experts,structuring a formal elicitation, and documentingthe elicitation process.

The NRC staff believes that formal elicitationprocedures, used prudently and appropriately, canhelp ensure that expert judgments are wvell-documented and that the technical reasoning usedto reach those judgments is openly displayed forreview. If conducted optimally, formal elicitationcan reveal a wide range of scientific and technicalinterpretations, thereby exposing (and possiblyquantifying) the uncertainties in estimatesconcerning repository siting, design, andperformance attributable to limitations in thestate of technical knowledge. Formal proceduresmay also help groups of experts resolvedifferences in their estimates by providing acommon scale of measurement and a commonvocabulary for expressing their judgments.

In preparing this BTP, the NRC staff has drawnon the specific recommendations of DeWispelareand Bonano (1995), as well as earlier, moregeneral recommendations in Winkler et al. (1992)and Bonano et al. (1990). It should also be notedthat this BTP has attempted to incorporate, asappropriate, a number of NWIRB and NASrecommendations with regard to the use of expertelicitation as a formal decision-aidingmethodology, including the treatment of "bias."(See Section 4 for a discussion of staffconsideration of specific NWTRB and NASrecommendations.) Moreover, this BTP has notattempted to prescribe the specific technicalissues for which expert judgment should (orshould not) be applied. The staff has viewed suchdeterminations to be the prerogative of DOE.However, with respect to performance assessment,the staff is considering the potential need forfuture guidance to identify those specific aspectsof a performance assessment for which theapplication of expert judgment may or may not beappropriate.

The positions and discussions in this BTP arebased on the premise that, under appropriatecircumstances, it is acceptable to supplement dataand analyses with the opinions of experts as partof the support for demonstrating compliance withNRC's geologic disposal regulation, and that, insome cases, these opinions are best obtainedusing a formal and well-documented process. ThisBTP gives specific guidance by which DOE may

determine if formal expert elicitation would beuseful and provides guidelines for an acceptableprocess for obtaining it. Section 2 summarizes theprincipal regulatory requirements andconsiderations that relate to this topic. The staff'stechnical position statements are listed in Section3, and Section 4 provides a discussion of thesupporting rationale behind each statement ofposition.

Definitions of key terms used in the BTP areprovided as Appendix A. Summarized inAppendix B are lessons learned from a recentexpert elicitation exercise performed by the NRCstaff and its contractor, the Center for NuclearWaste Regulatory Analyses (CNWRA). AppendixC contains the final Commission Policy Statementwith regard to the use of PRA methods in itsregulatory activities, including HLW. The staff'sresponse to the public comments received on anearlier draft of the BTP, noticed in the FederalRegister on February 28, 1996 (61 FR 7568), arecontained in Appendix D. Appendix E containsthe staff's views with regard to a possible courseof resolution for NRC SCA Comment 3. Lastly,Appendix F contains the staff's response tocomments received from the Advisory Committeeon Nuclear Waste on the final draft of the BTP

The NRC staff recognizes that DOE has theflexibility to determine whether the costs andbenefits of performing an expert elicitation areadvantageous compared with the costs andbenefits of performing theoretical analyses andlorgathering additional field and experimental data.That being said, however, the use of expertelicitation should not be considered as anacceptable substitute for traditional analysesbased on adequate field or experimental data,when such data are reasonably available orobtainable, or the analyses are practicable toperform. Nor can the use of a formal elicitationprocess, even when conducted in a mannerconsistent with guidance provided in this BT?,guarantee that specific technical conclusions willbe accepted and adopted by the staff, a LicensingBoard, the Commission itself, or any other partyto a potential HLW licensing proceeding. Rigidadherence to a sound elicitation process, in and ofitself, in no way guarantees that the resultingjudgments will be sufficient to meet theapplicant's burden of proof regarding thesubstantive issues addressed by the elicitation.Nonetheless, expert judgments obtained through

NUREG-1563 8

an evidently flawed or poorly documented processwill weaken their ability to support demonstra-tions of compliance.

It should also be noted that nothing in this BTPprecludes the use of expert judgment obtainedthrough informal means by DOE, in thepreparation of a potential license application. Ashas been the case in previous regulatory activities,the staff will accept for review the results offormal or informal judgment so long as the'rationale associated with the judgment isadequate, transparent, and sufficientlydocumented.

1.5 BTPs as Technical Guidance

BTPs are issued to describe, and make availableto the public, methods acceptable to the NRCstaff, for implementing specific parts of theCommission's regulations, and to provideregulatory guidance to regulated entities such as

DOE. BTPs are not substitutes for regulations,and compliance with them is not required.Methods and solutions differing from those setout in the BTP will be acceptable if they provide asufficient basis for the findings requisite to theissuance of a permit or license by theCommission.

This BTP constitutes informal pre-licensingactivity between the NRC staff and a prospectiveapplicant under 10 CFR 2.101(a)(1) and is notpart of a proceeding under the Atomic EnergyAct (Public Law 83-703), as amended. Nothing inthis BTP constitutes a commitment to issue anyauthorization or license, nor in any way affects theauthority of the Commission, the Atomic Safetyand Licensing Boards, other presiding officers, orthe Director, in any such proceeding.

Published BTPs will be revised, as appropriate, toaccommodate comments and to reflect newinformation and experience.

9 NUREG-1563

2 REGULATORY FRAMEWORK

'The following discussion describes the currentregulatory framework and Commission policy thatunderpin the staff's technical positions presentedin this BTP.

2.1 10 CFR Part 608

As noted earlier, the Commission's regulationsfound at 10 CFR Part 60 address the licensing ofa mined geologic repository for the disposal ofHLW Subpart E of those regulations sets forthspecific performance objectives, along with anumber of general siting and design criteria.Section 60.112 of Subpart E currently establishescompliance with EPA standards for the disposalof HLW (The previously applicable EPAstandards would have limited the release ofradioactive material to the accessible environment(weighted by a factor approximately proportionalto radiotoxicity, and integrated over a period oftime) as the overall system performance objectivefor the geologic repository after permanentclosure.)

With regard to 10 CFR 60.112, Section60.101(a)(2) states that:

While these performance objectives andcriteria are generally stated in unqualifiedterms, it is not expected that completeassurance that they will be met can bepresented. A reasonable assurance, on thebasis'of the record before the Commission,that the objectives and criteria will be met isthe general standard that is required. For§60.112, and other portions of this subpartthat impose objectives and criteria forrepository performance over long times intothe future, there will inevitably be greateruncertainties. Proof of the futureperformance of engineered barrier systemsand the geologic setting over time periods ofmany hundreds or many thousands of yearsis not to be had in the ordinary sense of theword. For such long-term objectives and -

BAs noted earlier, the need for future revision to 10 CFR Part 60 isunder consideration. Should 10 CFR Part 60 be revised, thelanguage of the citations quoted in this section of the BTP wouldneed to be re-examined, and if necessary, be revised accordingly. Itis not expected, however, that these revisions would require anychange to the staffs technical positions set out here.

criteria, what is required is reasonableassurance, making allowances for the timeperiod, hazards, and'uncertainties involved,that the outcome will be in conformance withthose objectives and criteria. Demonstrationof compliance with such objectives andcriteria will involve the use of data fromaccelerated tests and predictive models thatare supported by such measures as field andlaboratory data and natural analog studies.

In a subsequent proposal to conform the 10 CFRPart 60 regulations to then extant EPA standardsfor management and disposal of HLW, theCommission further elaborated on what wasnecessary for a satisfactory demonstration ofcompliance (NRC, 1986; 51FR 22288):

Demonstration of compliance with the- performance objectives of §60.112 will also

involve predicting the likelihood andconsequences of events and processes thatmay disturb the repository. Such predictionsmay involve complex computational models,analytical theories and prevalent expertjudgment. Substantial uncertainties are likelyto be encountered and sole reliance onnumerical predictions to determinecompliance may not be appropriate. Inreaching a'determination of reasonableassurance, the Commission may supplementnumerical analyses with qualitative judgmentsincluding, for example, consideration of thedegree of diversity or redundance amongmultiple barriers of a specific repository.

A primary consideration of a decision toauthorize construction of a repository will bewhether the site and design comply with theperformance-objectives and criteria contained inSubpart E. DOE must interpret the geologic'record and provide a demonstration that therepository site and'design will comply with explicitnumerical performance standards. As noted in theregulation, there will be substantial andunavoidable uhcertainties9 in predicting thelong-term performance of a geologic repository.

9Uncertainties may include, but not be limited to the: (a) identifica-tion of basic phenomena and their potential effects on repositoryperformance; (b) development and validation of models to describethese phenomena; (c) accuracy of available data; and (d) calcu-lational uncertainties.

11 NUREG-1563

Conclusions as to the performance of the geologicrepository and of particular barriers over longperiods of time, by necessity, will be based largelyon inference, as it will not be possible to carry outtest programs of sufficient duration or thatsimulate the full range of potential conditionsexpected over the period of regulatory concern(see NRC, 1983; 48 FR 28204). Given theseuncertainties, it will be necessary for DOE toadopt a variety of design features, developsophisticated models, perform tests, acquire data,and undertake other measures to be able todemonstrate that the performance objectives willbe met.

For its part, in reaching a potential constructionauthorization decision, the Commission has statedthat ". . . a reasonable assurance, on the basis ofthe record before the Commission, that theobjectives and criteria will be met is the generalstandard that is required." (10 CFR 60.101(a)(2))To reach a "reasonable assurance" finding, theCommission has said it will need to be able to doat least two things (48 FR 28201). It must firstdetermine that DOE has demonstratedcompliance with the numerical performancestandards, and, second, it must satisfy itself thatDOE's analysis of the site and design issufficiently convincing, that its limitations arewell-understood, and that DOE has demonstratedthat its analyses have made appropriate allowancefor the time period, hazards, and uncertaintiesinvolved.10

Confidence in the adequacy of DOE's data,analyses, and other items and activities associatedwith the repository program will be enhanced tothe extent that they are obtained through a qualityassurance (QA) program consistent with SubpartG of 10 CFR Part 60. In this regard, the staff hasacknowledged that external peer reviews may beused as part of the QA actions necessary toprovide confidence in the work submitted. TheNRC staff has provided guidance on how thesepeer reviews would be conducted inNUREG-1297 (see Altman et al., 1988).

One of the greatest challenges facing theCommission in making the determinationsnecessary for the licensing of a potential HLW

1 For a more detailed discussion of the Commission's views on the'reasonable assurance" concept, in the context of the geologicrepository regulation, see NRC (1983 and 1986).

repository will be to assess;the validity of DOE'streatment of uncertainty. Various methods may beused (e.g., probability distributions, conservative"bounding" analyses), and the Commission willevaluate quantitative and nonquantitativearguments to assess their application. Forpurposes of demonstrating compliance with the10 CFR Part 60 performance objectives, theCommission has acknowledged that the treatmentof uncertainty in DOE's compliancedemonstrations is expected to ". . . rely heavily on[the use of some form of] expert judgment. . . inthe selection of an appropriate [uncertaintyreduction] method and for the application of thattechnique .... " (NRC, 1986; 51 FR 22292)11 Thisis consistent with previous licensing experiencewhere the NRC Licensing Board has admittedexpert opinion evidence by one expert or by apanel of experts in the analysis of risks at nuclearpover plants where little or no data exist or theavailable data are unreliable.

As with other NRC licensing actions, a LicensingBoard's decision to grant or deny a license for aproposed repository will be based on submittedevidence that is a combination of fact andopinion. The subjective judgments of individualexperts and, in some cases, groups of experts,would likely be presented to interpret DOE's dataand analyses, and also to address the technicalissues presented in the hearing. Such assessmentsmay be not only quantitative but qualitative aswell. In certain instances, it may not be possibleto develop a "correct" estimate of some event orprocess. Rather, it may only be possible todevelop a "range" of estimates. Thus, both DOEand NRC may have no choice but to present theinformed judgment of qualified experts.

That being said, however, in its review of DOE'sstatutory SCP (10 CFR 60.16), the NRC staffnoted its concerns with DOE's site character-ization programs, specifically calling into questionDOE's potential over-reliance on the use of expertjudgment to supply the necessary information inany potential license application.12

tin the context of the containment requirements (10 CFR 60.112),EPA also noted that"... sole reliance on [the] numericalprediction of [geologic repository performance) may not beappropriate. . [an ethe implementing agencies may choose tosupplement such predictions vith non-quantitative judgments"(EPA, 1985; 50 FR 38088).

12See the staff's SCA Comment 3 (in NRC, 1989). This open itemhas not been resolved at this time.

NUREG-1563 12

2.2 Commission Policy StatementConcerning the Use of PRAMethods in Nuclear RegulatoryActivities

NRC has generally regulated the use of nuclearmaterial based on deterministic approaches andused probabilistic approaches only in certainspecialized areas. However, after the Three MileIsland incident, NRC has increasingly used PRAtechniques to augment the traditionalnon-probabilistic methods of analyzing nuclearsafety. PRA techniques have been appliedsuccessfully in several regulatory areas (see DOEet al., 1992) and have proven to be a valuablecomplement to engineering approaches.

Recently, Commission policies and regulationshave been based, in part, on PRA methods andinsights (NRC, 1995; 60 FR 42623). In light ofthese developments, the Commission issued anoverall policy statement on the expanded use ofPRA in nuclear regulatory activities, including thearea of nuclear waste disposal (60 FR 42622).Among other things, the Commission PolicyStatement (see Appendix C) called for the use of"... state-of-the-art... [PRA] methods ... ." in amanner that complements the deterministicapproach. (60 FR 42628) (According to a recentstaff survey, such PRA methods include, amongother things, methods to obtain and processexpert judgment (see PRA Working Group, 1994).)For the purposes of the regulatory actions in thewaste management program, the Commissionnoted, in the "Statement of Considerations for thefinal Policy Statement," that it .... agrees with

[public] comments regarding [the treatment] ofuncertainties in projecting repository performance... [through] the use of technical expert judgment... ." (60 FR 42627)

To ensure evenness and consistency in the staff'sfuture uses of PRA methods, NRC recentlyestablished a PRA Working Group to: (a) evaluatethe staff's current uses of PRA; (b) identify areasfor improvement in that use; and (c) recommendthe tenets of some basic principles and guidancefor application in the future. In the area of expertelicitation, the PRA Working Group made anumber of general recommendations on formaltechniques for obtaining, evaluating, andprocessing expert judgment (PRA WorkingGroup, 1994; pp. C-129-C-148). To ensure thequality and reproducibility of the elicitedinformation, the Working Group recommendedthat any formal elicitation procedure contain thefollowing process steps (op cit., pp. C-130-C-135):

* Selecting and defining technical issues.

* Selecting experts.

* Organizing assessments.

* Preparing for the elicitation.

* Processing expert judgment.

* Documenting.

The technical positions cited in Section 3 areconsistent with these recommended process steps.

13 NUREG-1563

3 BRANCH TECHNICAL POSITIONS

In view of the aforementioned policyconsiderations and statements of regulatoryconsideration underpinning 10 CFR Part 60, thestaff has adopted the following technical positionsconcerning the use of expert elicitation indemonstrating compliance with the geologicrepository disposal regulations. (As a supplementto the technical positions here, Appendix Aprovides definitions for certain key terms.)

(1) In matters important to the demonstration ofcompliance, the use of formal expertelicitation should be considered whenever oneor more of the followingconditions exist:

(a) Empirical data are not reasonablyobtainable, or the analyses are notpractical to perform;

(b) Uncertainties are large and significant toa demonstration of compliance;

: . More than one conceptual model canexplain, and be consistent with, theavailable data; or

provided to them, and the form of thejudgments that will be 'required.

Step No. 2-Selection of ExpertsBefore selection of the subject-matter experts,whose judgments will be elicited, two othertypes of experts should be recruited- thenormative expert and the generalist. Becausethese types of experts may influence theoutcome of the elicitation by the manner inwhich judgments are elicited, analyzed, orused, care should be taken in their selectionto ensure that they can perform in anobjective and impartial manner. Workingtogether, the normative experts andgeneralists generate and apply specificcriteria for the selection' of the subject-matterexperts.

The subject-matter experts selected forelicitation should be individuals who: (a)possess the necessary knowledge andexpertise;13 (b) have demonstrated theirability to apply their knowledge and expertise;(c) represent a broad diversity of independentopinion and approaches for addressing thetopic(s) in question; (d) are willing to beidentified publicly with their judgments; and(e) are willing to publicly disclose all potentialconflicts of interest.

The criteria used to select the various expertsof the elicitation team should be documented.

Step No. 3-Refinement of Issues and ProblemDecompositionThe generalists and normative experts shouldwork with'the subject-matter experts todecompose the broad objectives of theelicitation by clearly and precisely specifyingmore focused and simpler sub-issues.

Step No. 4-Assembly and Dissemination ofBasic InformationAssembly of background information shouldbe initially conducted by the generalists and

13 Vsth regard to Item (a), it would be useful for members of theexpert panel to possess at least some rudimentary knowledge ofboth decision-making theory and statistics. However, the possessionor the lack of this knowledge should not be used as a selectioncriterion.

(d) Technical judgments are required to'assess whether bounding assumptions orcalculations are appropriatelyconservative.

- (2) (a) When formally eliciting expert judgment,the applicant should use a consistent andsystematic procedure that will ensure that theresults obtained accurately reflect what isknown and not known about the topic inquestion. The components in an acceptable

-- elicitation process are described below and -are illustrated in Figure 1. Although writtenlargely for the elicitation of individualsubject-matter experts, the same approachcan be applied to a panel (or a team) ofsubject-matter experts.

Step No. .1-Definition of ObjectivesThe objectives of the elicitation should bedefined explicitly and in a manner thatreflects a clear understanding of how thejudgments obtained will be used. Theexplication of these objectives should thenguide the choice of experts, the information

15 15 ~ ~ ~ ~ .NUREG-1563

Tvni0

Step No. II

DEFINITIONOF #

OBJECTIVES

I

a

_ ,~~~~~~~

:

:

,......................

Stop No. 2

SELECTIONOF -

EXPERTS

Step No. 3v .

REFINEMENTOF

ISSUES

! Stop No. 4

ASSEMBLY ANDDISSEMINATION

OF BASICINFORMATION

.... ...........

a

a

Step No. 5 Step No. 6 Step No. 7 Step No. 8

PRE-| ELICITATION POST-O DSPARATEeEUICITATION X OF ELICITATION bVIEWS AND

TRAINING JUDGMENTS FEEDBACK AGGREGATION_ _ A~~~~~~~~~~~~----. OFJUDG MENTS

.6

Step No. 9P

| = ~~D 0 C U M E N T A T I NX

* E.g., te data, evidence, or knowledge base used by the subjct-watexpertsto arrive at their wspective judgments.

Figure 1 Components in an acceptable expert elicitation process. (Numbers next to the processblocks correspond to the technical position statements described in the text.Dashed lines indicate a feedback loop.)

normative experts. As the elicitation processproceeds, the subject-matter experts may beable to recommend additional sources ofinformation. Bias in the selection of thisbackground material should be avoided suchthat a full range of views is represented andthe necessary data and information areprovided in a uniform, balanced, and timelyfashion to all subject-matter experts.

Step No. 5-Pre-Elicitation TrainingIndividual (or teams of) subject-matterexperts should be provided training beforethe elicitations to: (a) familiarize them withthe subject matter (including the necessarybackground information on why theelicitation is being performed and how theresults will be used); (b) familiarize them withthe elicitation process; (c) educate them inboth' uncertainty and probability encodingand the expression of their judgments, usingsubjective probability; (d) provide thempractice in formally articulating theirjudgments as well as explicitly identifyingtheir associated assumptions and rationale;and (e) educate them with regard to possiblebiases that could be present and influencetheir judgments.

Step No. 6-Elicitation of JudgmentsThe individual elicitation session with eachsubject-matter expert (or teams ofsubject-matter experts) should be held in aprivate setting conducive to uninterrupteddiscussion. The generalists and normativeexperts should be in attendance for thecomplete session with each subject-matterexpert. At the start of the session for eachsubject-matter expert, the normative expertshould summarize the issues to be coveredand outline the logistics of the elicitation. Alldefinitions and assumptions agreed to by thegroup during pre-elicitation meetings shouldbe reviewed. All subject-matter expertsshould be queried in a uniform manner andasked to provide specific answers toquestions about the issues considered and thereasoning behind their responses. Careshould be taken to ensure that the requiredinformation is obtained and that it isinternally consistent. Responses of allsubject-matter experts should be documented

thoroughly with one or more of the' following:written notes, transcription, and audio orvideo tape.

Step No. 7-Post-Elicitation FeedbackEach subject-matter expert (or teams ofsubject-matter experts) should be providedfeedback from the elicitation team on theresults of his or her elicitation as soon aspractical after the elicitation sessions arecompleted. Each expert should be queried asto the need for revision or clarification of hisor her respective judgments based on thatfeedback. As is the case for all the elicitedjudgments, the rationale for any revisionsshould be documented scrupulously.

Step No.' 8-Aggregation of Judgments(Including Treatment of Disparate ievs)Whatever aggregation method is employed,the individual expert's opinions must bepreserved, documented, and provided to theNRC staff. Transparency in the aggregationprocess will render these judgments,including disparate views or outliers, 4 usefulfor subsequent analyses. If disparate

'judgments are aggregated or combined, theapplicant should: (a) provide some rationalefor the specific aggregation techniquesemployed and provide documentationsufficient to trace the impact of the individualexpert's judgment on the consolidatedjudgment; and (b) show what effect, if any,the disparate views would have on designand/or performance.

When widely disparate opinions arise, extraeffort should be taken to documentthoroughly the bases for the differing views.Subject-matter experts with differing viewsshould be asked to comment on opposingviews during and/or after their individualelicitations. Should the disparity in viewspersist, then each of the significantly varyingviews should be provided as output of theelicitation so that it may be incorporateddirectly into technical analyses andperformance assessments, or used torepresent the extremes in a sensitivityanalysis.

1As used in this guidance, outliers" refers to those opinions whichlie apart from the views or expected (average) views af otherexperts.

17 NUREG-1563

-

Step No. 9-Documentation

Proper documentation of a formal expertelicitation should indicate what was done,why, and by whom. The resulting judgmentsshould be clearly described along with thereasoning supporting these judgments. Thespecific issues addressed by the elicitationshould be precisely defined. Unambiguousdefinitions of all specific terms should beprovided and any assumptions used in theelicitation should be explicitly stated. Thejudgments, as they are stated by eachsubject-matter expert, should be provided,accompanied by the logic and information onwhich they are based. Any calculations thatthe experts considered important indetermining judgments or models usedshould be recorded and all literature used,whether public or restricted, should beproperly referenced. Proper documentationshould clearly distinguish between thatinformation provided directly by eachsubject-matter expert and any subsequentprocessing of that information, such assmoothing, interpolation, extrapolation, or

aggregation of the judgments of differentexperts.

(b) The approach described above envisionsthat all of these process steps would be partof a procedure for an expert elicitation. Ifpreferred, some of these steps can becombined as long as all of the elements of theprocess are addressed.

If one or more of the process steps areomitted from the recommended procedure,the staff may need additional information forits consideration before accepting the resultsof an elicitation for its review and evaluation.

(3) If information from an expert elicitation is tobe submitted in support of a license applica-tion, and if additional data or informationbecomes available, subsequent to the com-pletion of the elicitation, which could changeopinions or judgments obtained in the formalelicitation, the results of the elicitation shouldbe re-examined and updated, as appropriate.In addition to the information requestedabove, documentation should include adetailed description of the updating process.

NUREG-1563 18

4 DISCUSSION

The technical positions outlined in Section 3 aremotivated by several primary purposes.

First, as noted in Section'1.1, the NRC staff hastraditionally accepted, for review, the appropriatejudgments of technical experts regarding the basisof a license application. In this regard, theCommission has already acknowledged that itexpects that expert judgment will play an,important role in any potential geologic repositorylicensing proceeding (see NRC, 1986; 51'FR22292). Thus,' the staff believes that these technicalpositions are consistent with both previousCommission practice and Commissionexpectations on how'the geologic repositorydisposal regulations will be implemented.

In defining its technical positions, the staff hasattempted to draw from previous Commissionexperience with the application of formally elicitedexpert judgment in a regulatory context. The mostprominent examples of the Commission's use ofthis process can be found in the area ofcommercial nuclear reactor regulation. Asdiscussed in Section 1.1, the techniques of expertelicitation have been applied heretofore, in NRC'sregulatory program, to the assessment of- 'uncertainty associated with seismic hazards andfaulting parameters, and to the assessment of riskfrom severe accidents.

Second, the NRC staff recognizes that, in certaininstances, the collection'of data is not feasible andthat expert judgment must be used to complementexisting information, in order to supportdemonstrations of compliance with the.;requirements of 10 CFR Part 60. Topics'wherestaff expects that expert judgment will be broughtto bear are discussed in more detail in Bonano etal. (1990) and DeWispelare et al. (1994). Amongthese are: scenario formulation, development, andprobability estimation; development of andconfidence building for computational models;parameter estimation; and identification of whereadditional data and information gathering arenecessary and where they may be impractical. Asimportant as its role may be,' however, expertjudgment should be distinguished from bothmeasured data'or technical calculations based onaccepted scientific laws and principles. It shouldbe viewed as an alternative, and employed when

other means'of obtaining requisite data orinformation have been thoroughly considered andit has been concluded that such means are notpractical to implement.

,in evaluating the practicality of obtaining theneeded information (or data), DOE is expected toconsider cost, schedule, resource availability, andother programmatic factors. DOE will need toevaluate the programmatic costs and risks of-using expert elicitation as an alternative to moreobjective data-gathering methods, and thelikelihood that the elicitation itself may introducegreater uncertainty into the demonstration ofcompliance than will the use of objectiveinformation.

Lastly, when expert judgments are used to supporta demonstration of compliance, whether they areformally elicited or not, sufficient documentationshould exist to allow external examination of whatthe judgments were, how the judgments werearrived at (their basis), how the judgments wereused, and why'the judgments were used instead ofobtaining objective information (e.g., obtainingthe needed data). Such documentation supports abroader understanding and acceptance of theexpert judgment. For expert judgment to bedefensible under external peer review, the basisfor the judgment should be well-documented.Peers must be able to trace origins of specificjudgments from initial assumptions through.integration of results and conclusions. Thereshould be no gaps in the documentation, so thatan evaluation can be based on a thoroughunderstanding of the work presented. In addition,the availability of such documentation makespossible technical discussions in terms ofunderlying principles rather than just theindividual outcomes. Documentation should alsoinclude the justification for using expert judgmentto complement other data and analyses inreducing the residual uncertainty.

All assumptions 'about the technical issues thatwere used to develop the expert judgments shouldbe identified. The judgments 'as they are stated byeach individual subject-matter expert should beprovided in the documentation. Any calculationsthat a subject-matter expert considered importantin forming judgments, or models used, should bedocumented.' All literature used, whether public or

19 NTREG-1563

restricted, should be properly referenced. Anysubsequent manipulation of the individualexpert's judgments, such as aggregation orcombining with other opinions or data, should bedocumented thoroughly.

In closing, it should be noted that this NUREGdiscusses the rationale for and the potentialbenefits derived from the use of expert judgmentin a number of places. Without intending to limitthose discussions, the following summary isprovided. Section 1 describes the ubiquitousnature of expert judgment in scientific andtechnical activities. Section L2.1 provides adescription of typical outputs of an expertelicitation and the potential uses of the expertjudgments in the evaluation of repositoryperformance assessment. Section 13 providesactual examples of the NRC use of expertjudgments, including those used in PRA. Section 3provides the technical position on conditionswarranting the consideration of a formal expertelicitation. Later in this section, the staffelaborates on these conditions and discuss thebenefits derived from obtaining expert judgments.Finally, Appendix A provides a discussion ofexpert judgment from a definitional perspectiveand Appendix B provides results and lessonslearned from the NRC-sponsored formal climateexpert elicitation.

The following discussion parallels the list oftechnical positions given in Section 3.

(1) Conditions That May Warrant Considerationof a Formal. Process of Expert Elicitation

So-called "professional" or "engineering"judgment is exercised by scientists, engineers, andtechnical program managers routinely. More oftenthan not, this is done informally and in anon-explicit manner. However, as noted in Section1, the staff believes that confidence in DOE'sHLW program will be enhanced by thetransparency of the decision-making process.(This may also enhance public confidence, aswell.) Thus, it may be appropriate for DOE to"formalize" the way in which certain decisions onissues or problems are made, as noted below.

One way in which the necessary transparency canbe achieved in an expert elicitation process is touse a structured procedure, based on provendecision analysis methods, to gather the necessary

judgments from recognized subject-matterexperts. The use of such formal methods, asdiscussed in Technical Position No. 2 anddepicted in the figure shown previously, promotesthe accurate, consistent, and efficient collectionand processing of the expert judgments.Moreover, the use of a formalized processincreases the scrutability of the resultingjudgments and enhances the communication ofthe results.

The question thus arises when or under whatcircumstances might it be appropriate toundertake formal methods of expert elicitation inthe HLW program. As discussed in Section 3, thestaff believes that formal methods of expertelicitation may be of the greatest value to theprogram and should be considered in thefollowing situations: (a) empirical data are notreasonably obtainable or the analyses are notpractical to perform; (b) uncertainties are largeand significant to a demonstration of compliance;(c) more than one conceptual model can explain,and be consistent with, the available data; or(d) technical judgments are required to assesswhether bounding assumptions or calculations areappropriately conservative, as discussed below.

Precise criteria for determining when an expertelicitation is to be undertaken are not presentedhere. Programmatic concerns such as timing, cost,and compliance demonstration will have a majorimpact on determining whether an expertelicitation or some other form of informationgathering should be used. For example, expertelicitations may be more costly than a particulartype of data-gathering investigation, but they mayprovide results in a more timely manner.Programmatic concerns dominate the choices of:(a) gathering additional field or laboratory data;(b) undertaking additional theoretical analyses;(c) using expert elicitation; or (d) altering thecompliance demonstration strategy, to lessen oreliminate the need to resolve a particular issue.Therefore, the intent of this technical position isto allow DOE the maximum flexibility in choosingan approach, as long as an effectivedemonstration of compliance with the regulationscan be made.

* Empirical data are not reasonably obtainable orthe analyses are not practical to perform-Insome cases, data directly relevant to aproblem, question, or issue may be lacking orincomplete (e.g., do not cover the entire

NUREG-1563 20

period of regulatory interest), or are notreasonably obtainable or practical toperform. Under such circumstances, existingdata may be supplemented with expertjudgments. These judgments may be obtainedusing a formal elicitation process. Examplesof situations where field data are notreasonably or practicably obtainable include:(i) the site characteristics important to wasteisolation would be irreversibly compromisedby extensive data collection in such a way.that could potentially disqualify the site;(ii) it is infeasible or impossible to collectdata over the temporal or spatial scales.appropriate to adequately address aparticular issue;'and (iii) the cost of collectingthe comprehensive suite of data may beprohibitive.

Uncertainties are large and significant to ademonstration of compliance-Because of theextremely long period of regulatory concern, acombination of experimental methods,studies of natural analogues, andmathematical models'will be used in thetechnical and performance assessmentsnecessary to support any potential licenseapplication. Mathematical models areexpected to be the primary tools forestimating the long-term future performanceof the repository. Identification of externalconditions to which the'repository will besubjected during the period of regulatoryconcern is an essential requirement forapplying the mathematical models. Theseexternal conditions include evolvingtectonism, volcanism, seismicity, climate,hydrology, geochemistry, and other physicalprocesses that may affect repositoryperformance. Because of the complexinteractions of these processes, as well as thetemporal and spatial scales involved, morethan one credible interpretation of existingdata is frequently possible.

As a result of these practical limitations, afterevery reasonable effort has been made to -reduce the uncertainties affecting repositorysiting, design, and performance, largeuncertainties will probably still persist. Howthe applicant intends to examine and resolvespecific residual uncertainties has not yetbeen identified. However, regardless of the

specific approach adopted to accommodatethese uncertainties, there is no doubt thatexpert judgments will be pervasive (Fehringerand Coplan, 1992). Where such judgmentshave a direct and significant bearing oncompliance, formal elicitation methods maybe appropriate.

* More than one conceptual model can explain,and he consistent with the availabledata-Conceptual modeling of a HLWdisposal site is based on a combination of theapplication of fundamental physical andchemical principles and data interpretation.Data interpretation and conceptual modeldevelopment rely extensively on expertjudgments and it is not uncommon formultiple or alternative conceptual models to

-emerge that are consistent with available- data. This is particularly true when the data

available are limited and amenable tomultiple, equally valid interpretations.

Because conceptual models provide theunderpinning for the development of themathematical models and computer codes'that will be used in the quantitative estimatesof performance measures, the selection orrejection of a conceptual model could have aconsiderable impact on the results ofcomputational analyses: These judgments,when made by DOE to support itscalculations in a potential license application,will be subject to considerable'scrutiny by theNRC staff (Park et al., 1994), and thereforeare appropriately derived from a formalelicitation process.

* Technicaljudgments are required to assesswhether bounding assumptions or calculationsare appropriately conservative-Bounding'assumptions or calculations are used as atechnical approach to providing scientificallybased estimates when'the level of uncertaintyis very high,when the subject matter iscomplex, and when approximations aresufficient to resolve the issue at hand. If suchassumptions and calculations are used as abasis to terminate'or curtail further datacollection or analyses, judgments must bemade, and justified, that such estimates aresufficiently conservative for their intendedapplication. For issues critical to compliance

- 21 21 ~~~~NUREG-1563

-

demonstration, formal elicitation of thesejudgments and justifications may be of value.

(2) A Consistent and Systematic Process forElicitation Should Be Applied

If expert elicitations are to render accuraterepresentations of the legitimate range ofscientifically supportable interpretations amongthe informed technical community, they should beconducted using a structured procedure, based onproven decision analysis methods. A structured,thoroughly documented procedure allows reviewersto reconstruct the logic and events involved in theelicitation and use of expert judgment. Use ofsuch a protocol establishes and maintains thequality of the process in much the same way asuse of a QA program for field data collection.Adherence to such a protocol is also critical insupporting internal and external reviews of theresulting judgments and to foster confidence inthe integrity of the process. That being said,however, scrupulous adherence to a soundelicitation process, in and of itself, in no wayguarantees that the resulting judgments will besufficient to meet the applicant's burden of proofregarding the substantive issues addressed by theelicitation. Nonetheless, expert judgmentsobtained through an evidently flawed or poorlydocumented elicitation process might ultimatelyundermine the credibility of any demonstrationsof compliance supported by those judgments.

The procedure set forth in Section 3 in summaryform, and repeated below, has elements of severalother protocols-see, for example, NRC-NtREG/CR-5411 (Bonano et al., 1990) andNUREG-1150 (NRC, 1990); EPRI (Coppersmithet al., 1993); the Sandia National Laboratories(Trauth et al., 1994); and the CNWRA(DeWispelare et al., 1993 and 1994). It is providedhere, but not with the intent that it be rigidlyapplied in every instance where judgments areelicited. Instead, it should be viewed as a generalframework for a formal elicitation acceptable tothe NRC staff. The applicability of any of theproposed process steps and thus the degree towhich the overall process is implemented shouldbe evaluated in each separate elicitation, and theprocess may be customized or revised, as needed,for the elicitation of interest.

In addition, although the process steps are listedin numerical order, it is not necessary that the

individual steps be performed in the exactsequence presented. In fact, it is expected thatseveral of these process steps will proceed or canbe initiated concurrently, subject to repeatediterations and opportunities for feedback from thesubject-matter experts. This may be especiallytrue for Step Nos. 2, 3, and 4, which are depictedas parallel process steps with feed-backs, asshown in the preceding figure. What this figureshows, for example, is that once the subject-matter experts are identified (in Step No. 2), theycan help to better define the objective of theelicitation (Step No. 1) and thus aid in thedecomposition of the elicitation issue into itsconstituent parts (Step No. 3). Moreover, thesubject-matter experts can also aid in theidentification of additional information that couldfacilitate the elicitation (Step No. 4).15

Step No. 1-Definition of ObjectivesPerhaps the most important step in anyelicitation process is the precise definition ofits ultimate objective(s). Proper definition ofthe objective(s) calls for an understanding ofhow the judgments will be used in subsequentanalyses. This understanding should directthe overall content of the elicitation bydefining the nature of the expertise thatshould be brought to bear (Step No. 2); theassumptions and information that will beprovided to the appropriate subject-matterexperts (Step Nos. 3 and 4); and the form ofthe judgments that should result (Step No. 6).This is a critical step that should includeinput from generalists who are familiar withthe overall project and the specificinformation needs and intended uses.

Step No. 2-Selection of Experts7jpes of Experts: Three types of experts areordinarily recruited to participate in a formalelicitation: generalists, normative experts, andsubject-matter experts (the last of whom areusually referred to as "the experts").Generalists and normative experts arerecruited first and together comprise theelicitation team, which is responsible for

15Alternatively, before deciding on the final panel of subject-matterexperts (Step No. 1), in contacting candidate experts, it might beuseful to solicit their input on refining the general problem (StepNo. 3) that was previously formulated bv the generalists and thenormative experts. In this way, a much arger knowledge base isavailable to fine-tune the issues that will ultimately be addressed bythe final group of subject-matter experts. Further, final refinementof the sub-issues (and questions-Step No. 3) should be performedultimately by the selected panel of subject-matter experts.

NUREG-1563 22

organizing, conducting, and documenting theelicitation process.

Generalists are individuals knowledgeableabout various overall, and one or morespecific aspects related to sitecharacterization, repository design, andperformance assessment. Typically,generalists have substantive knowledge in onediscipline (e.g., hydrology, geology, materialscience, transport phenomena, etc.) and asolid general understanding of the technicalaspects 'of the problem.

Normative experts have training in probabilitytheory, psychology, and decision analysis.They assist the generalists and subject-matterexperts in articulating their professionaljudgments and thought processes in a formsuitable for input into a particular technicalassessment.

Subject-matter experts are the experts fromwhom judgments are elicited. These areindividuals who are at the forefront of aspecialty relevant to geologic waste disposal,and are recognized by their peers'asauthorities because of their sustained andsignificant research on the topic.

In selecting each of the three types of experts,especially the subject matter experts, it maybe useful to seek qualified nominations fromoutside sources, or recognized peers in thefield. This would include, for example: theNational Academies of Sciences andEfgifiering; academia; recognized -professiofial'societies (e.g., Sigma Xi, TheGeological Society of America, The AmericanSociety of Civil Engineers), Nationallaboratories; knowledgeable Federal Agenciesand International Organizations; privateindustry; State development and regulatingbodies; representative public interest groups;;and interested stakeholders. (Examination ofthe frequency of citations in the scientificliterature may also be of help in this regard.)The elicitation team members, and theirrespective assignments, would then beselected from the list of nominees (forexample, see Appendix A in DeWispelare etal., 1993).

Selection Criteria: Among its many tasks, theelicitation team generates and applies criteriafor the selection of the subject-matter experts.For this reason, and because the elicitationteam members may influence the outcome ofthe elicitation by the manner in whichjudgments are elicited, analyzed, or used,special care should be taken in the selectionof generalists and normative experts to ensurethat they can perform in an objective andimpartial manner. Although the selectioncriteria discussed below focus primarily onthose attributes necessary to establish thesuitability and substantive'knowledge'ofprospective subject-matter experts, the samecritiria-should be applied to the selection ofgeneralists and normative experts insofar asthey relate to their expertise, experience, andability to cany out their respective roles. Tis isparticularly true as it relates to the criterionconcerning the appearance of bias or conflict ofinterest owing to the influence the generalistand normative expert can have on the outcomeof any potential elicitation, as discussed below.

As stated in the technical position, the panelof experts selected for elicitation shouldcomprise individuals who: (a) possess thenecessary knowledge and expertise; (b) havedemonstrated their ability to apply theirknowledge and expertise; (c) represent abroad diversity of independent opinion.andapproaches for addressing the topic(s) inquestion; (d) are willing to be identifiedpublicly with their judgments; and (e) are

: willing to identify, for the record, anypotential conflicts of interest.

- Technical knowledge, expertise, and theability to address the topic of the elicitationby a subject-matter candidate can beestablished through examination of his or hereducational background, professionalexperience (including research and consultingactivities in related problems or studies),publication record, previous experience as apeer reviewer for the work of others,membership and leadership positions inprofessional societies, and awards and otherindications of peer recognition.

A subject-matter expert's ability to apply hisor her substantive knowledge to the task athand can be determined by examining theexpert's record of published research and

23 NUREG-1563

participation in consulting activities onrelated problems or studies, priorparticipation in other expert elicitations, aswell as experience as peer reviewer of thework of others. Although difficult to quantify,an expert's flexibility of thought and ability toobjectively consider evidence that challengeshis or her own conventional wisdom, as wellas an ability to explain complex topics inclear and straightforward terms, is also ofvalue for a successful elicitation.

In selecting subject-matter experts, it isimportant to identify candidates who havethe requisite credentials, including a varietyof backgrounds and experiences, to ensurethat the full range of legitimate opinions on aparticular scientific topic is represented andis incorporated into the study. Particular careshould be taken to avoid selecting expertswith similar educational backgrounds andexperiences, because this increases thelikelihood that they will invoke similarassumptions and approaches to arrive attheir respective judgments. Dependenceamong subject-matter experts (so-called"expert dependence"-see Clemen andWinkler (1985); and Chhibber andApostolakis (1993)) can significantly diminishthe diversity of opinion. In this regard, careshould be taken to select experts who havehad training at different academicinstitutions and, to the extent possible,represent a diversity of scholarly approaches.

Conflict of Interest: The credibility of thejudgments from any formal elicitation will beincreased when the subject-matter expertshave fewer conflicts of interest in the areas ofinstitutional influences, financial orprofessional gain, or promotion of a social orpolitical agenda, to the extent practicable. Ifsome subject-matter experts are encumberedby such conflicts of interest, an attemptshould be made to balance the influences byother panelists. The NWTRB has commentedfrequently on this point and hasrecommended using balanced panels ofexperts, incorporating experts from outsideDOE and its contractors, who haveindependent and varying perspectives on anissue. This is important to building credibilityinto the judgments of the experts (NWIRB,1990a, 1990b, 199L 1993, and 1995).

Perhaps the most frequent conflict-of-interestconcern is that of "set bias" by virtue of whoemploys the experts. The issue here, forexample, is whether the subject-matterexperts derive employment or income fromorganizations charged with conducting theoverall performance assessment or with theconstruction and licensing of the repository.Other potential conflicts of interest mayinvolve subject-matter experts' close-workingrelationships with individuals involved inrepository characterization or development.Subject-matter experts should be asked toprovide written statements of any potential orpotentially perceived conflicts of interest,each of which should be made a part of therecord of the elicitation.

The staff position does not assume, however,that any individual with a perceived or realconflict of interest will permit this conflict toinfluence his or her professional judgments.Furthermore, the staff does not wish toexclude crucial information from an expertelicitation simply because a knowledgeableindividual has a potential conflict of interest.The HLW program is a small technicalprogram in many respects and in certain keysubject disciplines, there are relatively fewexperts. The population of known experts hasprobably been involved with one or more ofthe interested/affected parties, and thereforesome conflicts of interest may beunavoidable.

Consequently, an elicitation should bedesigned such that the knowledge andreasoning of experts with potential conflicts,along with the fact and nature of thosepotential conflicts, should be made availableto all subject-matter experts. Conveningbalanced elicitation panels and disclosures ofpossible conflicts are appropriate measuresto counter the appearance of a potentialconflict of interest.

Also, credibility of the elicitation can beenhanced when nominations for candidatesubject-matter experts can come fromorganizations such as professional andacademic societies, peers in the field, orreviews of the scientific literature. Thesubject-matter experts should be selectedfrom the list of nominees, using explicitcriteria, and the entire selection process

NUREG-1563 24

should be systematic and thoroughly.documented.

Finally, this staff position is not intended topreclude the ability of DOE to use its ownstaff or its contractor's staff in any elicitationpanel. NRC has traditionally considered, forreview, the technical analyses prepared by thelicensee (and its contractors) and willcontinue to do so.

Step No. 3-Refinement of Issues and ProblemDecompositionAfter definition of the objective of theelicitation (Step No. 1), perhaps the next mostimportant process step in an elicitation wouldbe decomposition of the problem into conciseand distinct questions. (As noted earlier, thisstep may be initiated concurrently with theselection of the subject-matter experts.)Generalists and normative experts comprisingthe elicitation team may make a preliminaryattempt to identify key sub-issues and todecompose the problem. Problemdecomposition in elicitation refers tobreaking down issues to provide for easierand less complex assessments that can berecombined into a probability distribution orutility function for the quantity of interest.The recombination is usually accomplished'using one or more mathematical models thatexpress the value of interest as amathematical function of componentquantities. -

Definition of the issues should be precise;clarity of the issues is important for theelicitation design. The issues can range fromgeneral to specific and from simple tocomplex. Conventionally, complex problemsare broken up into smaller and simplercomponents to facilitate the solution of theproblem. The basic tenet of problemdecomposition is that the solution of thesmaller components is more tractable thanthat of the'entire problem. Problem'decomposition related to the elicitation ofexpert judgments is advocated by many (seeChhibber et al. (1992); and Hora 'and Iman(1989), among others) as the vehicle toincrease the likelihood that the judgments arefocused on issues with which'the ` 'subject-matter experts are thoroughly

familiar. If possible, issue definition andproblem decomposition should first placeemphasis on making explicit the subject-matter expert's direct knowledge, based onexperience and evidence, and second on hisor her ability to process or encode thisknowledge into probability estimates (as inthe approach advocated by Kaplan (1992).The normative expert, drawing on his or herexpertise, may assist the subject-matterexperts in the framing'of key questions orsub-issues in such a way as to minimize theintroduction of certain types of bias. Forexample, it is known that soliciting input onthe extremes of a probability distribution canhelp reduce the tendency for experts to"anchor" on a central value and, thereby,reduce bias from over-confidence (a form ofcognitive bias).

A preliminary statement of the issues andassumptions may be prepared by theelicitation team, presented to the subject-matter experts, and then, later, refined, based

-on their feedback. Alternatively, theelicitation team can work with the subject-matter experts in'structured interactions todevelop the ehtire list of issues andassumptions. In either case,' it is importantthat the subject-matter experts have ampleopportunity to provide input to theformulation of technical questions and thedecomposition of the problem'into moretractable sub-issues. Structured interactionsamong the subject-matter experts to discussand decompose the problem enhance thelikelihood that all share a commonunderstanding of the problem, relevantsub-issues, and appropriate boundaryconditions that are used to define theproblem.

By having the subject-matter experts agree ona common decomposition of a particularproblem, the views of the various experts onthe sub-issues can be compared in aconsistent fashion. Ad hoc or arbitrarydecomposition of a problem, by each expert,may not permit such comparisons.

Step No. 4-Assembly and Dissemination ofBasic InformationWhen subject-matter experts are asked toprovide judgments, they are being asked to

25 25 ~~~~~NUTREG-1563

analyze information provided to them about agiven question, issue, or problem. The factthat expert judgments are being sought is anindication that there is either a lack ofdirectly relevant information or major gaps inthe available information about the questionor problem being addressed, and moreimportantly, that obtaining such informationor closing critical gaps, using experimental orother traditional approaches, is not feasible.The goal of the elicitation, therefore, is toidentify the true state of uncertainty withinthe scientific community-be it aleatory(stochastic) or epistemic (state-of-knowledge)uncertainties (see SSHAC, 1995; pp. 13-14).

Accordingly, it is reasonable that, as a startingpoint, the sponsors of the elicitation shouldassist the elicitation team by assembling apreliminary body of basic information,"'which, in the view of the sponsors, isnecessary to allow the subject-matter expertsto arrive at their judgments. It is important torecognize, however, that the judgments of thesubject-matter experts may be influenced bythe type of information they receive, and themanner in which that information ispresented. Information that covers the fullrange of views (i.e., information that isall-inclusive) on the subject of the elicitationshould be provided. Biasing may beintroduced at this very influential point, andcredibility of the elicitation could be reduced,.if a suitably broad range of availableinformation on a particular issue or sub-issueis not made available.

As the subject-matter experts interact toidentify issues and decompose the problem(Step No. 3), they may identify additionaldata and information they will need to assessthe relevant sub-issues. Therefore, feedbackfrom the subject-matter experts andsuggestions of additional sources ofinformation should be solicited before theelicitations. It is important that data andinformation identified as necessary by thesubject-matter experts are provided in auniform, balanced, and timely fashion to all

16e.g., knowledge base-can be in the form of data, evidence, models,analyses, parameters, or statistics

members of the expert panel as they preparefor the elicitations.

Step No. 5-Pre-Elicitation TrainingBecause many subject-matter experts areunfamiliar with the purpose and mechanicsof formal expert elicitation, it is importantthat sufficient training be provided by theelicitation team to adequately prepare themfor the elicitation of their technicaljudgments. This training should:(a) familiarize them with the elicitationprocess; (b) educate them in uncertaintyencoding and the expression of theirjudgments using subjective probability;(c) provide them practice in formallyarticulating their judgments as well asexplicitly identifying their associatedassumptions and rationale; and (d) educatethem with regard to possible biases thatcould be present and influence theirjudgments.

Before conducting an elicitation, thesubject-matter experts should have a clearunderstanding of the objective of theelicitation, the reason for obtaining theirjudgments through a formal process, and themanner in which their judgments will beused. In most expert elicitations, thesubject-matter experts are specialists withextensive knowledge and experience in ahighly refined, but perhaps very narrow fieldof study. If, for example, the purpose of theelicitation is to obtain probability judgmentsthat are to serve as input to a performanceassessment, it is important that thesubject-matter experts receive some basicoverview training in performance assessmentto provide a context for how their specificexpert judgments and any associatedmathematical functions or distributions willbe used.

As a designated member (or members) of theelicitation team explains the purpose andprocess of the elicitation, it should beemphasized that the goal is not consensus,but rather a realistic description of the truestate of scientific knowledge and uncertaintyabout the subject area in question. 'Painingshould consist of familiarizing thesubject-matter experts with the elicitationprocess; motivating them to provide

NUREG-1563 26

judgments; giving them practice in formallyexpressing their judgments'as well as theassumptions and rationale'for the judgments(especially uncertainty encoding); andeducating them about possible biases thatcould be present and influence thejudgments.

One important aspectof training is theconduct of rehearsal sessions to assist thesubject-'matter experts in becoming familiarand comfortable with offering judgments andthe corresponding underlying reasoning andassumptions. Training helps ensure that thejudgments-represent the subject-matterexperts actual state of knowledge about theproblem of interest.

The most common method of encodinguncertainty associated with an expert'sjudgment is through elicitation of a'subjectiveprobability distribution. When subject-matterexperts are asked to express degrees of beliefin terms of subjective probabilities, they mustbecome familiar with the techniques that willallow them to perform this task. Rehearsalsessions provide'the means for thesubject-matter experts to practice theapplication of these techniques.

Familiarizing the subject-matter experts withpossible biases that maybe present andinfluence their opinions is another reason forconducting elicitation training. There are twogeneral classes of bias: motivational andcognitive. Motivational biases occur because asubject-matter expert has a vested interest in'an issue and consciously or'unconsciouslydistorts his or her judgment. Exarples ofcircumstances that might contribute tomotivational bias were noted'under'Step No.'2 (above). Cognitive biases occur because of afailure to process, aggregate, or integrate theavailable data and information: Motivationalbiases can generally be reduced, or at leastmitigated, by careful selection of thesubject-matter experts and by sensitizing the'subject-matter experts to the motivationalbias' potential for influence. Cognitive biasesare more difficult to address, and to date, the'best approach 'to 'deal with them is to becomefamiliar with and practice the application ofdebiasing techniques during elicitationtraining sessions.

-In addition to their expertise, it should alsobe noted that subject-matter experts,themselves, are likely to bring preconceivednotions (e.g., cognitive biases) to thequestions or problems they are addressing.These are' most often the result of many yearsof involvement in research in the specific areain which their judgments are sought-one ofthe main reasons why'specific individuals arerecruited in the first place. It is inevitablethat these notions or biases will significantlyinfluence the expert's judgments. Because oftheir extensive experience, subject-matterexperts may tend to focus quickly onsolutions for problems of a similar naturewithout closely examining the manner inwhich the current problem may differ frompast experiences and the impact thosedifferences should have on their judgments ofthe problem at hand. Slovic (1991) concludedthat most individuals have great difficulty inaccepting information that does not confirmprior experience, and generally tend todiscard information that does not fit withintheir reasoning paradigms. Morgan andHenrion (1990) suggest that subject-matterexperts need to be presented with, andrequired to consider, information thatchallenges 'their conventional wisdom. Theseinvestigators state that subject-matter expertsare most likely to exhibit over-confidence andbias when they fail to examine informationthat supports a point of view different fromtheir own. If, instead,'subject-matter expertsexplicitly consider and address informationthat contradicts their prior judgments,Morgan and Henrion argue thatover-confidence biases are reduced andoverall judgments tend to improve. For thesereasons, normative experts should challengethe subject-matter experts to consider afreshconflicting information, especially when newinformation becomes available that couldresult in a re-evaluation of earlier judgments.

Step N. 6-Elicitation of Judgments"The actual elicitation of the judgments is theclimax of the process for the individualsubject-matter experts'(6r teams' ofsubject-matter experts-i.e., all activitiespreceding the elicitation were aimed atpreparing for it). The elicitation must betailored to the specific question or issue athand, the type of judgments required (e.g.,

27 NUREG-1563

identification of events and phenomena,subjective probabilities, probabilitydistributions, etc.); the resources available forthe elicitation; the availability of thesubject-matter experts; and the individualprofessional preferences (e.g., the schedulingand length of elicitation sessions, format forconducting the elicitations, etc.); Thesefactors may influence the methods andtechniques employed in the elicitation.Regardless of the manner in which theelicitation is carried out, reviewers should beable to discern not only the judgmentsthemselves, but also the reasons,assumptions, approaches, and informationthat each of the subject-matter experts used.A dry run or rehearsal of the elicitation isvery beneficial to familiarize the elicitationteam with the procedure. This rehearsalshould occur in advance of the individualelicitation sessions.

Step No. 7-Post-Elicitation FeedbackThe subject-matter expert (or teams ofsubject-matter experts) should be givenindividual feedback from the elicitation teamon the results of the elicitation as soon aspractical after the elicitation sessions arecompleted. In particular, they should beprovided with numerical, graphical, and/orother useful representations of theirjudgments. The subject-matter experts shouldbe allowed to revise their judgments based onthe feedback. However, the rationale for anyrevisions should be carefully documented.

The elicitation team should seek confirmationof the conclusions from each subject-matterexpert, but guard against attempting to forceconsensus or influence their outcome, duringthe individual feedback session, if disparateopinions exist. Finally, this step also allowsthe elicitation team members the opportunityto verify data codification and check forencoding errors.

Step No. 8-Aggregation of Judgments(Including Treatment of Disparate Views)The preceding steps make no assumptionswith respect to whether individual expertjudgments (or teams of judgments) ratherthan combined or aggregated results areultimately needed. The advantages of

focusing on the individual judgments, up tothis point, include traceability of specificjudgments to an individual and the voicing ofdiverse views and perspectives that increasethe likelihood that the actual state ofknowledge on a given problem or issue hasbeen captured. The use of multiplesubject-matter experts as a means to capturethe existing diversity of opinion about theanswer to a given question or the solution ofa problem of interest gives rise to multiplejudgments-typically at least one differentjudgment for each subject-matter expert. Inthose cases where a similar or the sameconclusion is arrived at from different paths,the confidence in this conclusion is obviouslyenhanced.

To render differing judgments from multiplesubject-matter experts useful for certainpractical analyses and assessments, or fordiscrete input to a performance assessmentcalculation, it may be necessary to aggregate,or combine, in some fashion, the individualjudgments. Two general approaches tocombining expert judgments are commonlyidentified as behavioral and mechanicalaggregation. Behavioral aggregation usuallyentails the bringing together of thesubject-matter experts to discuss andcombine their judgments. Such interactionsallow the thinking, logic, and experience baseof the different experts to be exchanged. Thismay bring about some reconciliation ofdifferences and result in a single consensusrepresentation of the state of knowledge, or itmay minimize the differences among experts.At a minimum such interactions shouldreduce the potential for unintentionaldisagreement. The behavioral approach ismost beneficial when subject-matter expertshave basic differences in fundamentalassumptions, on which their judgments arebased, which have not been made explicit.Interactions among the differing experts canthereby illuminate these assumptions andmay lead to a more thorough understandingand documentation of conflicting approaches.

Mechanical aggregation techniques (alsoknown as analytical aggregation) consist oflogic, and formulas consistent with that logic,that have been developed by normativeexperts for combining individual judgments.

NUREG-1563 28

Individual judgments are combinedmathematically such that the sum of theweighted individual judgments is normalized(e.g.,' equal to one). Among the obviousadvantages of mechanical combination areease of use,' amenability to extensivesensitivity analyses, and the fact thatindividual experts need have no influence onthe judgments of other experts, afterelicitation. The most common andstraightforward mechanical aggregation is asimple average that assigns equal weights(parity) to the judgment of each expert.However, differential weighting techniqueshave also been used'to account for relativeexpertise or experience of individual experts(see PRA Working Group, 1994; pp.C-139-C-142).

A third approach to aggregation is one''recently discussed by SSHAC (1995), wherebyb6th the behavioral (judgrniental)'and'mechanistic schemes are "blended" orcombined. In this approach, themathematical models used to producepreliminary elicitation results are shown tothe experts, along with an explanation of the

'assumptions used to construct the models.The experts are subsequently 'asked if theywish' to revise their judgments, given theknowledge about the consequences of theresults, and they thus develop a "consensus"representation or aggregate distributionbehaviorally based on their revisedjudgments.

In the stepwise approach to elicitation, thevalue of structured, iterative, interactionsamong the subject-matter experts isrecognized. The staff believes thatinteractions, among the experts, properlystructured to permit exchange of reasoning,data, and assumptions, may outweigh thepotential disadvantage previously ascribed toa behavioral approach, namely that someexperts may be dominated or "forced" tosuppress their ideas and contribute to anartificial consensus. Should interactionamong the experts, after the individualelicitations, result in any changes ofjudgments by the individual experts (as in themanner cited by SSHAC (1995), above (e.g.,repeating Step Nos. 6-8 after some initialintegration has been achieved)), the

'descriptions and implications of the changesshould be included in updatedrepresentations of the individual experts'states of knowledge. If such interactions leadto a commonly held representation of thestate of knowledge, then the representation ofeach individual should also reflect therepresentation for the group. Morecommonly, however, residual differencesbetween the individual experts will persist,and these can then be'combined usingmechanical techniques.

It is not the intent of the NRC staff toprescribe any particular algorithm oraggregation technique, or require aggregationitself. Choice of an appropriate method ormethods of aggregation may be highlyissue-specific. That being said, however, thestaff believes that adherence to a step-wiseelicitation process similar to that identified inthis technical position will tend to fosterconditions where equal weighting ofindividual judgments'may be mostappropriate. Regardless of which aggregationtechniques are ultimately selected, however,sufficient documentation must be provided totrace the impact of each individualsubject-matter opert's judgment on theconsolidated position. It cannot beemphasized enough that, because of thereviewer's potential need to examine anindividual expert's judgments and reasoningbases, the professional judgment of eachsubject-matter expert must be explicitlydocumented as opposed to that of a personwho is a panel spokesman or facilitator. Aunanimous, consensus, or summary opinionwithout such documentation, will generallynot be suitable to support a compliancedemonstration, for purposes of licensing.

Step No. 9-DocumentationAn essential element of a formal elicitationprocess is thorough documentation of allaspects of the process, the judgmentsacquired, and the rationale and basis for thejudgments. The reasons for documenting theuse of expert judgment for technical problemsare derived from the following objectives:(a) to improve decision-making associatedwith public policy; (b) to enhancecommunication; (c) to facilitate peer review,appraisal, and acceptance; (d) to recognize

29 29 ~~~~NUREG-1563

and minimize biases in expert judgment;(e) to indicate the current state of knowledgeabout important technical and scientificmatters; and (f) to provide a basis forupdating that knowledge.

Since credibility and acceptability areobjectives of any expert elicitation associatedwith controversial issues in the licensing andpublic acceptance of a repository, assuggested by the NWTRB (1990a), anorganized, thoroughly documented procedureallows reviewers to reconstruct the logic andevents involved in the elicitation and use ofexpert judgment The availability of suchdocumentation supports a broaderunderstanding and acceptance of what wasundertaken. In addition, it makes possibletechnical discussions in terms of underlyingprinciples rather than just the individualoutcomes. Comprehension of the elicitationresults and the utility of their. use will begreatly enhanced by the use of a uniform andconsistent reporting format for documentingthe elicitation of each subject-matter expert.

Documentation is a continuous task thatbegins as soon as an issue is identified as acandidate for expert judgment elicitation.Precise and complete documentation ispivotal to the success of the elicitation (andultimately, acceptance of the results). Forexample, the documentation should include a.discussion of all steps in the elicitationprocedure. Each step should be describedand the results presented. Moreover, thedocumentation should also reflect what

specific information was used or relied on bythe experts (e.g., rationale) to reach aparticular judgment (reached in Step No. 4).As emphasized above, the results should beprovided for each subject-matter expert aswell as for any aggregated judgments.

(3) Elicited Judgments Should Be Updated asWarranted

When new data or information becomes availablebefore license application submittal, it couldpotentially change a DOE position with regard tothe design and perhaps the performance of thegeologic repository. To the extent practicable, anypotential license application should address thesignificance and impact that any new informationmight have on the validity of all previouslyexisting data and elicited judgments used. If theimpacts are determined to be significant, then thedata and expert judgments should be updated toincorporate the new data or information, as theinformation becomes available. Of course, the newinformation may resolve the issue by providingthe objective data needed and thus obviate theneed for a new elicitation.

The methods of updating the expert judgmentsrange from the use of Bayes' Theorem, forstatistical updating, to conducting another set ofindividual elicitations for the same or a differentset of experts. Whichever method is used forincorporating the new data or information intothe existing expert opinions, it should bethoroughly documented to provide a transparentview of the updating process and resultingjudgments.

NUREG-1563 30

-

5 REFERENCES

Altman, WD., et al., "Peer Review for High-LevelNuclear Waste Repositories," U.S. NuclearRegulatory Commission, Generic TechnicalPosition, NUREG-1297, February 1988.

Andrews, R.W, T.F. Dale, and J.A. McNeish,"Total-System Performance Assessment-1993:An Evaluation of the Potential Yucca MountainRepository," Las Vegas, Nevada, INTERA, Inc.,March 1994. [Prepared for the U.S. Departmentof Energy/Yucca Mountain Site CharacterizationProject.]

Barnard, R.W, et al., "TSPA: 1991 An InitialTotal-System Performance Assessment for YuccaMountain," Albuquerque, New Mexico, SandiaNational Laboratories, SAND91-2795, September1992. [Prepared for the U.S. Department ofEnergy/Yucca Mountain Site CharacterizationProject.]

Bernreuter, D.L., C. Minichino, and the TERACorporation, "Seismic Hazard Analysis," U.S.Nuclear Regulatory Commission, NUREG/CR-1582, 5 vols., 1980-83. [Prepared by theLawrence Livermore National Laboratory and theTERA Corporation.]

Bernreuter, D.L., et al., "Seismic HazardCharacterization of the Eastern United States,"Livermore, California, Lawrence LivermoreNational Laboratory, UCID-20421, 2 vols., April1985. [Prepared for the U.S. Nuclear RegulatoryCommission.]

Bernreuter, D.L., J.B. Savy, and R.W Mensing,"Seismic Hazard Characterization of the EasternUnited States: Comparative Evaluation of theLLNL and EPRI Studies," U.S. NuclearRegulatory Commission, NUREG/CR-4885, May1987. [Prepared by the Lawrence LivermoreNational Laboratory.]

Bernreuter, D.L., et al., "Seismic HazardCharacterization of 69 Nuclear Power Plant SitesEast of the Rocky Mountains," U.S. NuclearRegulatory Commission, NUREGICR-5250,January 1989. [Prepared by the LawrenceLivermore National Laboratory.]

Bonano, E.J., et al., "Elicitation and Use of ExpertJudgment in Performance Assessment of High-Level Radioactive Waste Repositories," U.S.Nuclear Regulatory Commission, NUREG/CR-5411, May 1990. [Prepared by the SandiaNational Laboratories.]

Clemen, R.T, and R.L. Winkler, "Limits for thePrecision and Value of Information fromDependent Sources," Journal of OperationsResearch, 33:427-442 [1985].

Chhibber, S., G. Apostolakis, and D. Okrent, 'ATaxonomy of Issues Related to the Use of ExpertJudgments in Probabilistic Safety Studies,"Reliability Engineering and System Safety, 38:2745[1992].

Chhibber, S. and G. Apostolakis, "SomeApproximations Useful to the Use of DependentInformation Sources," Reliability Engineering andSystem Safety, 42:67-86 [1993].

Code of Federal Regulations, "Disposal ofHigh-Level Radioactive Wastes in GeologicRepositories," Part 60, Chapter I, Title 10,"Energy."

Code of Federal Regulations, 'Appendix A to Part100-General Design Criteria for Nuclear PowerPlants," in "Reactor Site Criteria," Part 100,Chapter I, Title 10, "Energy."

Codell, R.B., et al., "Initial Demonstration of theNRC's Capability to Conduct a PerformanceAssessment for a High-Level Waste Repository,"U.S. Nuclear Regulatory Commission,NUREG-1327, May 1992.

Coppersmith, KJ., R.C. Perman, and R.R.Youngs, "Earthquakes and Tectonics ExpertJudgment Elicitation Project," Palo Alto,California, Electric Power Research Institute,EPRI TR-102000, February 1993. [Prepared byGeomatrix Consultants, Inc.]

Dennis, A.W (ed.), "Exploratory Studies Facility[ESF] Alternatives Study: Final Report,"Albuquerque, New Mexico, Sandia NationalLaboratories, SAND91-0025, 2 vols, September1991. [Prepared for the U.S. Department ofEnergy. Also see A.L. Stevens and L.S. Costin,

31 NUREG-1563

Findings of the ESF Alternatives Study,. -.Albuquerque, New Mexico, Sandia NationalLaboratories, SAND90-3232, March 1991.]

DeWispelare, A.R., et al., "Expert Elicitation ofFuture Climate in the Yucca Mountain Vicinity-Iterative Performance Assessment Phase 2.5," SanAntonio, Texas, Center forNuclear WasteRegulatory Analyses, CNWRA 93-016, August-1993. [Prepared for the U.S. Nuclear RegulatoryCommission.]

DeWispelare, A.R., et al, "Background Report onthe Use and Elicitation of Expert Judgment," SanAntonio, Texas, Center for Nuclear WasteRegulatory Analyses, CNWRA 94-019, September1994. [Prepared for the U.S. Nuclear RegulatoryCommission.]

DeWispelare, A.R., and EJ. Bonano, "Input tothe Draft Staff Technical Position on Elicitation ofExpert Judgment," San Antonio, Texas, Center forNuclear Waste Regulatory Analyses, CNWRA95-006, March 1995. [Prepared for the U.S.Nuclear Regulatory Commission.]

Fehringer, D. J., and S.M. Coplan, "Uncertainty inRegulatory Decision-Making," in American -Nuclear Society/American Society of CivilEngineers, High-Level Radioactive Waste Manage-ment: Proceedings of the Third InternationalConference, April 12-16, 1992, Las Vegas, Nevada,1:106-109 [1992].

Gorham-Bergeron, E.D., et al., "Evaluation of'Severe Accident Risks: Methodology for theAccident Progression, Source Term, Consequence,Risk Integration, and Uncertainty Analyses," U.S.Nuclear Regulatory Commission'NUREGfCR-4551 (Draft Revision 1), Vol. 1 [1986]..[Prepared by the Sandia National Laboratories.]

Gorham, E.D., et al., "Evaluation of SevereAccident Risks: Methodology for the ,Containment; Source Term; Consequence, and'Risk Integration Analyses,".U.S NuclearRegulatory Commission, NUREG/CR-4551(SAND8S6-1309), Vol. -1, Revision 1, December1993. [Prepared by the Sandia NationalLaboratories.]

Harper, .T, et at, "Probabilistic AccidentConsequence Uncertainty Analysis; A JointReport Prepared by U.S. Nuclear Regulatory

- Commission and Commission of EuropeanCommunities, U.S. Nuclear RegulatoryCommission, NUREG/CR-6244, 3 vols., January1995. [Prepared by the Sandia NationalLaboratories].

Holonich, JJ., Office of Nuclear Material Safetyand Safeguards/Division of High-Level WasteManagement, letter to J.P. Roberts, U.S.Department of Energy/Office of CivilianRadioactive Waste Management [Subject: "Reviewof Report of Early Site Suitability Evaluation ofthe Potential Repository Site at Yucca Mountain,Nevada"], U.S. Nuclear Regulatory Commission,July 22, 1992.

Hora, S.C., and R.L. Iman, "Expert Opinion inRisk Analysis: The NUREG-1L50 Methodology,"Nuclear Science and Engineering, 102 323-331[1989].

Hora, S.C., 'Acquisition of Expert Judgment:Examples from Risk Assessment," Journal ofEnergy Engineering, 118:136-148 [1993]. :

Johnson, R.L, "Overall Review Strategy for theNuclear Regulatory Commission's High-LevelWaste Repository Program," U.S. NuclearRegulatory Commission, NUREG-1495,November 1994.

Kaplan, S., "Expert Information Versus ExpertOpinions: Another Approach to the Problem ofEliciting/Combining!Using Expert Knowledge inPRA," Journal of Reliability Engineering andSystem Safety, 35:61-72 [1992].

Kastenberg, WE., et a., Findings of the PeerReview Panel on the Draft Reactor RiskDocument-NUREG-1150," Livermore, CA,Lawrence Livermore National Laboratory,UCID-21346,- May 1988. [Prepared for the U.S.

* Nuclear Regulatory Commission asNUREG/CR-5113.]

Kouts, H., et al., "Methodology for UncertaintyEstimation in NUREG-1150 [Draft]: Conclusionsof a Review Panel," Upton, New York, Brook-haven National Laboratory, BNL-NUREG-52119, December 1987. [Prepared for the U.S.Nuclear Regulatory Comrnission asNUREG/CR-5000.]

Linehan, JJ., Office of Nuclear Material Safetyand Safeguards/Division of High-Level Waste

NUREG-1563 32

Management, letter to R. Stein, U.S. Departmentof Energy/Office of Civilian Radioactive WasteManagement [Subject: "U.S. Department ofEnergy Plans for Conducting the Calico HillsRisk/Benefit Analysis"], U.S. Nuclear RegulatoryCommission, January 5, 1990.

Linehan, JJ., Office of Nuclear Material Safetyand Safeguards/Division of High-Level WasteManagement, letter to D.E. Shelor, U.S.Department of Energy/Office of CivilianRadioactive Waste Management [Subject"Transmittal of Meeting Summary for the U.S.Nuclear Regulatory Commission-U.S.Department of Energy Meeting on the CalicoHills Risk/Benefit Analysis and the ExploratoryShaft Facility Alternatives Study"], U.S. NuclearRegulatory Commission, August 27, 99L

Meyer, M.A. and J.M. Booker, "Eliciting andAnalyzing Expert Judgment: A Practical Guide,"U.S. Nuclear Regulatory Commission,NUREG/CR-5424, January 1990. [Prepared bythe Los Alamos National Laboratory.]

Morgan, M.G., and M. Henrion, Uncertainty: AGuide to Dealing with Uncertainty in QuantitativeRisk and PolicyAnalysis, Cambridge,Massachusetts, Cambridge University Press, 1990.

National Research Council, "Probabilistic SeismicHazard Analysis," Washington, D.C., NationalAcademy Press, Commission on PhysicalSciences, Mathematics, and Resources, 1988.

National Research Council, "Rethinking High-Level Radioactive Waste Disposal: A PositionStatement of the Board on Radioactive WasteManagement," Washington, D.C., NationalAcademy Press, Commission on Geosciences,Environment, and Resources, July 1990.

National Research Council, "Technical Bases forYucca Mountain Standards," Washington, D.C.,National Academy Press, Commission onGeosciences, Environment, and Resources, July1995a.

National Research Council, "Ward Valley: AnExamination of Seven Issues in Earth Sciencesand Ecology," Washington, D.C., NationalAcademy Press, Commission on Geosciences,Environment, and Resources, [May] 1995b.

National Research Council, "Review of U.S.Department of Energy Technical Basis Report forSurface Characteristics, Preclosure Hydrology,and Erosion," Washington, D.C., NationalAcademy Press, Commission on Geosciences,Environment, and Resources, [December] 1995c.

Nuclear Waste Technical Review Board, FirstReport to the US. Congress and the U.S. Secretaryof Energy, U.S. Government Printing Office,March 1990a.

Nuclear Waste Technical Review Board, SecondReport to the US. Congress and the US. Secretaryof Energy, U.S. Government Printing Office,November 1990b.

Nuclear Waste Technical Review Board, FourthReport to the US. Congress and the US. Secretaryof Energy, U.S. Government Printing Office,December 1991.

Nuclear Waste Technical Review Board, FifthReport to the US. Congress and the US. Secretaryof Energy, U.S. Government Printing Office, June1992.

Nuclear Waste Technical Review Board, Report tothe US. Congress and the US. Secretary of EnergyJanuary to December 1993, U.S. GovernmentPrinting Office, May 1994. [Tenth in a series ofreports]

Nuclear Waste Technical Review Board, Report tothe US. Congress and the US. Secretary of Energy:1994 Findings and Recommendations, U.S.Government Printing Office, March 1995.[Eleventh in a series of reports]

Park, J.R., et al., "The Nuclear RegulatoryCommission Strategic Plan for PostclosurePerformance Assessment Activities for theHigh-Level Waste Geologic Repository: DraftReport," San Antonio, Texas, Center for NuclearWaste Regulatory Analyses, Letter Report, March1994. [Prepared for the U.S. Nuclear RegulatoryCommission.]

PRA Working Group, 'A Review of NRC StaffUses of Probabilistic Risk Assessment," U.S.Nuclear Regulatory Commission, NUREG-1489,March 1994.

Science Applications International Corporation,"Report of Early Site Suitability Evaluation of the

33 NUREG-1563

Potential Site at Yucca Mountain, Nevada," LasVegas, Nevada, 2 vols., SAIC-91/8000O1, January1992. [Prepared for the U.S. Department ofEnergy/Yucca Mountain Site CharacterizationProject.]

Seismic Methods Peer Review Panel, "FinalRecommendations of the Peer Review Panel forthe Use of Seismic Methods for CharacterizingYucca Mountain and Vicinity," Report submittedto the U.S. Department of Energy/Office ofCivilian Radioactive Waste/Yucca Mountain SiteCharacterization Project, January 22, 1991.

Seismicity Owners Group/Electric PowerResearch Institute, "Seismic Hazard Methodologyfor the Central and Eastern United States [FinalReport]," Palo Alto, California, EPRI NP-4726, 10vols., July 1986. [Prepared by Risk Engineer-ing, Inc., Woodward-Clyde Consultants, Inc.,Geomatrix Consultants, Inc., and the CYGNACorporation.]

Senior Seismic Hazard Analysis Committee,"Recommendations for Probabilistic SeismicHazard Analysis: Guidance on Uncertainty andUse of Expert," Livermore, California, LawrenceLivermore National Laboratory,UCRLID-122160, 2 vols., August 1995.[Sponsored by the U.S. Department of Energy,the U.S. Nuclear Regulatory Commission, and theElectric Power Research Institute. Also expectedto be published as NUREG/CR-6372.]

Schenker, A.R., et al., "Stochastic HydrogeologicUnits and Hydrogeologic Properties Developmentfor Total-System Performance Assessments,"Albuquerque, New Mexico, Sandia NationalLaboratory, SAND94-0244, September 1995.[Prepared for the U.S. Department of Energy/Yucca Mountain Site Characterization Project.]

Slovic, P., "Beyond Numbers: A BroaderPerspective on Risk Perception and RiskCommunication," in D.G. Mayo and R.D.Hollander (eds.), Acceptable Evidence. Scienceand Values in Risk Management, New York,Oxford University Press, 1991, pp. 48-65.

Sobel, PA.,- "Revised Livermore Seismic HazardEstimates for Sixty-Nine Nuclear Power Plant:Sites East of the Rocky Mountains FinalReport," U.S. Nuclear Regulatory Commission,NUREG-1488, April 1994.

lbst Prioritization Thsk Force, "Testing Prioritiesat Yucca Mountain: Recommended Early Tests toDetect Potentially Unsuitable Conditions for aNuclear Waste Repository," U.S. Department ofEnergy/Office of Civilian Radioactive WasteManagement, 2 vols., March 1, 1991.

Trauth, KM., S.C. Hora, and R.V. Guzowski,'Expert Judgment on Markers to DeterInadvertent Human Intrusion into the WasteIsolation Pilot Plant," Albuquerque, New Mexico,Sandia National Laboratories, SAND92-1382,November 1993.

Tschoepe,' E., Iml, and L.R. Abramson,"'Substantially Complete Containment'.Elicitation Report," San Antonio, Texas, Centerfor Nuclear Waste Regulatory Analyses, CNWRA92-016, August 1992. [Prepared for the U.S.Nuclear Regulatory Commission.]

U.S. Environmental Protection Agency,"Environmental Standards for the Management ofSpent Nuclear Fuel, High-Level and THansuranicWastes [Final Rule]," Federal Register, Vol. 50,No. 182, September 19, 1985, pp. 38066 - 38089.

U.S. Department of Energy, 'A MultiattributeUtility Analysis of Sites Nominated forCharacterization for the First Radioactive WasteRepository-A Decision-Aiding Methodology,"Office of Civilian Radioactive WasteManagement, DOE/RW 0074, May 1986.

U.S. Department of Energy, Chapter 8,Section 8.1, Rationale," in "Site CharacterizationPlan, Yucca Mountain Site, Nevada Research andDevelopment Area, Nevada," Office of CivilianRadioactive Waste Management, NevadaOperations Office/Yucca Mountain Project Office,Nevada, DOE/RW-0199, Vol. IV, Part B,December 1988.

U.S. Department of Energy, "Record Memoran-dum: Risk/Benefit Analysis of AlternativeStrategies for Characterizing the Calico Hills Unitat Yucca-Mountain (Revision 0), Office of CivilianRadioactive Waste Management, 2 vols.,YMP/91-6, January 1991a.

U.S. Department of Energy, "Unsaturated ZoneHydrology Peer' Review Record Memorandum,"Office of Civilian Radioactive Waste :Management/Yucca Mountain Project Office,June 24, 1991b.

NUREG-1563 34

U.S. Department of Energy, et al., "Risk Assess-ment: A Survey of Characteristics, Applications,and Methods Used by Federal Agencies forEngineered Systems (Submitted to the FederalCoordinating Council for Science, Engineering,and Technology, Ad Hoc Working Group on RiskAssessment)," Washington, D.C., U.S. NuclearRegulatory Commission, November 1992.[Contributions from eight Federal agencies,including NRC.]

U.S. Department of Energy, "Site Character-ization Progress Report: Yucca Mountain,Nevada, October 1, 1993-March 31, 1994(No. 10)," Office of Civilian Radioactive WasteManagement, DOE/RW-0450, October 1994.

U.S. Department of Energy, "Site Character-ization Progress Report: Yucca Mountain,Nevada, April 1, 1994-September 30, 1995(No. 11)," Office of Civilian Radioactive WasteManagement, DOE/RW-0463, March 1995.

U.S. Nuclear Regulatory Commission, "ReactorSafety Study: An Assessment of Accident Risks inU.S. Commercial Nuclear Power Plants,"WASH-1500 (NUREG-75/014), October 1975.

U.S. Nuclear Regulatory Commission, 'Disposalof High-Level Radioactive Wastes in GeologicRepositories [Final Rule]," Federal Register,Vol. 48, No. 120, June 21, 1983, pp. 28194-28229.

U.S. Nuclear Regulatory Commission, "Disposalof High-Level Radioactive Wastes in GeologicRepositories [Conforming Amendments]," FederalRegister, Vol. 51, No. 118, June 19, 1986, pp. 22286-22301.

U.S. Nuclear Regulatory Commission, "NRC StaffSite Characterization Analysis of the Departmentof Energy's Site Characterization Plan, YuccaMountain, Nevada," Office of Nuclear MaterialSafety and Safeguards, NUREG-1347, August1989.

U.S. Nuclear Regulatory Commission, "SevereAccident Risks: An Assessment for Five U.S.

Nuclear Power Plants-Final Summary Report,"Office of Nuclear Regulatory Research,NUREG-1150, 3 vols., December 1990.

U.S Nuclear Regulatory Commission, "ReactorSite Criteria Including Seismic and EarthquakeEngineering Criteria for Nuclear Power Plantsand Proposed Denial of Petition from FreeEnvironment, Inc., et al. [Proposed Rule andProposed Denial of Petition]," Federal Register,Vol. 59, No. 199, October 17, 1994, pp. 52255 -52273.

U.S. Nuclear Regulatory Commission, "Use ofProbabilistic Risk Assessment Methods inNuclear Regulatory Activities [Final PolicyStatement]," Federal Register, Vol. 60, No. 158,August 16, 1995, pp. 42622 - 42630.

U.S. Nuclear Regulatory Commission,'Availability of Draft Branch Technical Positionon the Use of Expert Elicitation in the High-LevelWaste Program [Notice of Availability]," FederalRegister, Vol. 61, No. 40, February 28, 1996, pp.7568 - 7569.

Wescott, R.G., et al. (eds.), "NRC IterativePerformance Assessment Phase 2: Development ofCapabilities for Review of a Performance Assess-ment for a High-Level Waste Repository," U.S.Nuclear Regulatory Commission, NUREG-1464,October 1995.

Wilson, M.L., et al., "Total-System PerformanceAssessment for Yucca Mountain-SNL SecondIteration (TSPA-1993)," Albuquerque, NewMexico, Sandia National Laboratory, SAND93-2675, 2 vols., April 1994. [Prepared for the U.S.Department of Energy/Yucca Mountain SiteCharacterization Project.]

Winkler, R.L., S.C. Hora, and R.G. Baca, "TheQuality of Expert Probabilities Obtained throughFormal Elicitation Techniques," San Antonio,Texas, Center for Nuclear Waste RegulatoryAnalyses, Letter Report, September 1, 1992.[Prepared for the U.S. Nuclear RegulatoryCommission.]

35 NUREG-1563

----

APPENDIX AGLOSSARY

As used in this guidance:

'Aggregation of judgments" refers to thecombining of the individual elicited judgments ofmore than one subject-matter expert to produce asingle judgment, point estimate, range, oruncertainty distribution.

"Behavioral aggregation methods" use personalinteractions among the subject-matter expertsemploying consensus techniques to combinemultiple individual judgments.

"Cognitive bias" occurs when a subject-matterexpert fails to process, include, or integrate theavailable data or information available.

"Debiasing techniques" refers to training thesubject-matter experts in dealing with andreducing the effects of cognitive biases on theelicitation results. Examples of these techniquesare: (i) familiarity and practice with the elicitationtask; and (ii) awareness of the biases throughpersonal experiences, making use of feedback.

"Elicitation team" refers to the group ofgeneralists and normative experts conducting andfacilitating the elicitation. This group may also bereferred to as the project team or panel.

"Expert judgment" refers to the data or informa-tion provided by a subject-matter expert. It is thesubject-matter experts opinion or belief based onreasoning. Expert judgments can be evaluations oftheories, models, experiments, or recommenda-tions for further research. Expert judgments mayalso be opinion that can be analyzed andinterpreted and can be used in performanceassessment and other technical models. Expertjudgments can be either qualitative or quantita-tive. Expert judgments can also be judgmentsabout uncertain quantities or judgments aboutvalue preferences. A subject- matter expert mayprovide a probability distribution or a pointestimate for an uncertain parameter.

"Expert elicitation" is a formal, highly structured,and well-documented process for obtaining thejudgments of multiple experts.

"Generalist" is an individual with substantialtechnical background in one or more of thedisciplines needed to solve the problem ofinterest-but whose understanding of the problemtypically spans beyond the particular discipline-and is well-versed on how the judgments will beused in the solution of the problem. Generalistsmay be selected from the project staff and theywork with the normative experts in the conduct ofthe elicitation. They serve several roles: (i) pro-pose the problem decomposition; (ii) prepare theissue definition; (iii) provide assistance to thesubject-matter experts by explaining how theirjudgments will be used; and (iv) together with thenormative experts and with input of the subject-matter experts, orchestrate the final presentationand, where appropriate, aggregation of the elicitedjudgments.

"Mechanical aggregation methods" rely onanalytic formulae (such as weighted averaging) tocombine multiple individual judgments.

"Motivational bias" occurs when a subject-matterexpert has a vested interest in an issue, institution,political agenda, or personal relationship, andwhen that vested interest consciously orunconsciously acts to distort his or her judgment.

"Normative expert" refers to individuals with asound theoretical and conceptual knowledge ofprobability and practical experience in theelicitation of judgments from individuals.Normative experts are well-versed in thepsychological and cognitive processes thatsubject-matter experts follow in the analysis ofinformation to produce the desired judgments.

'"eer review" is frequently described as a form ofexpert judgment; however, in the context of thisBTP, peer review does not fall within the afore-mentioned definition of expert judgment. The staffrecognizes a subtle, yet fundamental, differencebetween peer review and expert judgment as usedhere. The former refers to judgments providedxegarding the soundness and quality of an existingsolution to a given problem, whereas the latterrefers to judgments which, themselves, give rise toor contribute to a scientific stance or solution. Inthis context, the staff expects expert judgments tobe elicited and used in a manner that would allow

A-1 NUREG-1563

-

the elicitation and the judgments to be the subjectof peer review.

"Subject-matter expert" is the individual fromwhom the expert judgment will be elicited. Thesubject-matter expert is an individual recognizedby his or her peers as an authority in a specific

subject matter or topic. Subject-matter expertstypically gain recognition as such because ofsignificant and sustained research in the subjectmatter or topic, and their knowledge is believedby others to represent the current state of the artin that subject or topic.

NUREG-1563 A-2

APPENDIX BAN ELICITATION ON FUTURE CLIMATE: LESSONS LEARNED

B-1 Introduction

In 1992-93, the Center for Nuclear WasteRegulatory Analyses (CNWRA) conducted anelicitation, to familiarize the U.S. NuclearRegulatory Commission staff with a state-of-the-art formal expert judgment elicitation. This workwas performed as part of the staff's IterativePerformance Assessment efforts to develop anindependent license application review capability(see Appendix D in Johnson, 1994). The subject ofthe elicitation was the future climate in the YuccaMountain vicinity. The details of this formalelicitation itself, as well as the results, aredescribed in DeWispelare et al. (1993).

B-2 Rationale

A formal elicitation was selected for this subject,for the following reasons:

(1) The state of climate science and modelingdoes not support accurate sub-regionallong-term projections based on historic orcurrent meteorologic data. Expert judgmentis a way to integrate and supplement theoutput of general circulation models.

(2) There are great uncertainties associated withpredictions of future climate at YuccaMountain. This is especially evident when oneconsiders that climate varied during theQuaternary Period (ranging from pluvial toarid conditions), and that the influences ofsuch variations on infiltration can potentiallydominate the predicted performance of ageologic repository for high-level waste.

(3) Because of the state of science in long-termclimate prediction, there exist a number ofconceptual approaches. These range fromgeneral circulation models conditioned withcombinations of past and present meteoro-logic data, to energy-balance models based oncurrent physical data, to empirical historicdata used to establish past conditions.

(4) The published record contains a variety ofdata and opinion that establishes variousbounding limits, some of which have beeninterpreted to be conservative.

Several lessons were garnered, from the climateelicitation, that have relevance for the process ofexpert elicitation, generally:

* The ability to understand and compare thejudgments of each participant in an expertelicitation is far easier when uniformprocedures are used to elicit judgments anddocument results. The generalists allcommented that the clarity and logic that thesubject-matter experts provided in thejudgments and supporting rationale wereattributable to their overall expertise.

* It is possible to have a defensible process forselection of subject-matter experts. Toaddress concerns regarding the lack of biasor independence in the selection of thesubject-matter experts, a documented processof peer nomination and selection can beconducted even within a relatively short timeschedule.

* Debiasing training of the subject-matterexperts is essential to a smooth elicitation.Most subject-matter experts have had onlylimited or no experience at producingconsistent subjective probability distribu-tions. All the subject-matter experts agreedwith the normative experts and generaliststhat this training was essential to the process.

* The elicitation team had considerabledifficulty in constructing a behaviorally-basedaggregation because of the variation in theindividual judgments and the subject-matterexperts conviction regarding their judgments.After an attempt at behavioral aggregation, amechanical aggregation of the subject-matterexperts judgments was attempted. Thisapproach was faster and easier to implementthan a behavioral aggregation. When it isnecessary to aggregate the individualjudgments of the subject-matter experts afterthey have been elicited, it is efficient to use a

B-1 NUREG-1563

mechanical aggregation scheme. This schemecan be easily documented to providetraceability.

* It is valuable to have the subject-matterexperts visit the site. The subject-matterexperts indicated that experience gainedthrough the site visit was important to theirpreparing for the elicitation, and facilitatingthe post-elicitation validation process. Thesubject-matter experts noted that by visitingthe site, they were able to place the data andresearch assembled for the elicitation into thecontext of the physical setting at YuccaMountain, thus leading to what thesubject-matter experts regarded as a morerealistic interpretation of this information.

* Individual documentation is critical to asuccessful elicitation. The documentationconsisted of two parts. First, a short paper,which formed the scientific basis for aparticular judgment, served as the referencefor understanding the technical reasoningexpressed in the elicitation. The second partconsisted of the elicitation team's docu-mentation of the elicited judgments (e.g.,

probability distributions associated with theclimate variables) and accompanying specificreasoning resulting from each expert'selicitation session, to ensure that the rationaleused by each subject-matter expert waswell-understood and expressed consistently.Video taping of each elicitation sessionhelped the team to check session notes andserved as a permanent record of each session.

B-3 References

DeWispelare, A.R., et al., "Expert Elicitation ofFuture Climate in the Yucca Mountain Vicinity-Iterative Performance Assessment Phase 25," SanAntonio, Texas, Center for Nuclear WasteRegulatory Analyses, CNWRA 93-016, August1993.

Johnson, R.L., 'Appendix D: NRC Post-ClosurePerformance Assessment Strategy for aHigh-Level Nuclear Waste Repository" in "OverallReview Strategy for the Nuclear RegulatoryCommission's High-Level Waste RepositoryProgram," U.S. Nuclear Regulatory Commission,NUREG-1495, November 1994.

NUREG-1563 B-2

APPENDIX CFINAL COMMISSION POLICYSTATEMENT ON THE USE OF PRA METHODS

IN NUCLEAR REGULATORY ACTIVITIES

C-1 Introduction

The following statement presents the policy thatthe U.S. Nuclear Regulatory Commission willadopt in the use of probabilistic risk assessment(PRA) methods in nuclear regulatory matters.This policy was developed because the Com-mission believed that the potential applications ofPRA methodology could improve public healthand safety decision-making while promotingstability and efficiency in the regulatory processand reducing unnecessary burdens on licensees.After a public workshop, the Policy Statement waspublished in draft form in the Federal Register(NRC, 1994; 59 FR 63389). On receipt andconsideration of public comments, it waspublished in final form (see NRC, 1995; 60 FR42622).

C-2 The Commission Policy (at 60 FR42628)

1. The use of PRA technology should beincreased in all regulatory matters to theextent supported by the state of the art inPRA methods and data and in a manner thatcomplements NRC's deterministic approachand supports NRC's traditional defense-in-depth philosophy.

2. PRA and associated analyses (e.g., sensitivitystudies, uncertainty analyses, and importancemeasures) should be used in regulatorymatters, where practical within the bounds ofthe state of the art, to reduce the unnecessaryconservatism associated with currentregulatory requirements, regulatory guides,license commitments, and staff practices.

Where appropriate, PRA should be used tosupport the proposal for additional regula-tory requirements in accordance with 10 CFR50.109 (Backfit Rule"). Appropriate proce-dures for including PRA in the process forchanging regulatory requirements should bedeveloped and followed. It is, of course,understood that the intent of this policy isthat existing rules and regulations shall becomplied with unless these rules andregulations are revised.

3. PRA evaluations in support of regulatorydecisions should be as realistic as prac-ticable, and appropriate supporting datashould be publicly available for review.

4. The Commission's safety goals for nuclearpower plants and subsidiary numericalobjectives are to be used with appropriateconsideration of uncertainties in makingregulatory judgments on the need forproposing and backfitting new genericrequirements on nuclear power plantlicensees.

C-3 References

U.S. Nuclear Regulatory Commission, "Use ofProbabilistic Risk Assessment Methods inNuclear Regulatory Activities [Proposed PolicyStatement]," Federal Register, Vol. 59, No. 235,November 8, 1994, pp. 63389-63391.

U.S. Nuclear Regulatory Commission, "Use ofProbabilistic Risk Assessment Methods inNuclear Regulatory Activities [Final PolicyStatement]," Federal Register, Vol. 60, No. 158,August 16, 1995, pp. 42622-42630.

C-1 NUREG-1563

APPENDIX D -DISPOSITION OF PUBLIC COMMENTS 1 ON FEBRUARY 28, 1996, DRAFT

BRANCH TECHNICAL POSITION (61 FR 7568)

State of Nevada

The following comments were submitted by theState of Nevada (see Loux, 1996):

The Agency'for Nuclear Projects/NuclearWaste Project Office has reviewed thesubject Draft Branch 'Tchnical Position(BTP) and finds that'it is, in general,favorably responsive to concerns transmittedin our letter of July 25, 1995 (see Loux, 1995),regarding the U.S. Department of Energy's(DOE's) June 5, 1995, Principles andGuidelines for Form"al Use of Exjqert Judgmentby the Yucca Mountain Site Characterization

-Project Office (see Brocoum, 1995).

The BTP properly emphasizes' the need to'formalize and thoroughly document theprocess of preparing for and eliciting expertopiniofi, aggregating judgments, and iperforming follow-up, when appropriate.However, missingfrom the discussion isexplicit guidance that it is equally importantthat DOE's decision, and its basis, toemploy the expert elicitation process, bethoroughly documented and trans'parent to-future reviewers. Also;'the documentation of'the decision should be maintained'as part ofthe record of the'expert elicitation process.

It will be particularly important to know atthe time of license application review.

. whether cost was a primary consideration inthe decision to employ expert judgmentrather than performing theoretical analysesand/or gathering additional field andexperimental data, as the BTP suggests itmight be (pages 11 and 24). This isimportant because the Yucca MountainProject has' a continuing history of changingpriorities on field and lab data collectionand 'analysis that often is driven from year toyear by available funds. What may seem to'DOE to be practical and 'necessary one yearcan be deemed imiipractical the next year,:with the issue then set aside as if it were

IThe indented portions of this appendix quote the public comments.

closed and unnecessary to pursue. Anexample of this which may persist into alicense application is the question ofadditional field study to understand the highhydrologic gradient north of Yucca Moun-tain. At present, it appears that someinformal expert judgment has determinedthat it is not important to understand whythis condition exists, whereas in the past ithas been considered important by DOEwhen more money was available to theproject for surface-based testing.

The BTP states the four conditions underwhich the use' of expert elicitation should beconsidered (page 17 and elaborated on pages24-25):

(a); Empirical data are not reasonablyobtainable, or the analyses are not'practical to perform;

(b) -Uncertainties are large and significant toa demonstration of compliance;

(c) More than one conceptual model canexplain, and be consistent with, theavailable data; or

(d) Technical judgments are required toassess whether bounding assumptions orcalculations are appropriatelyconservative.

*Throughout the text of the BTP, the use ofexpert judgment is described in variousmanners that are only in some casesobviously consistent with the above list, e.g.,'an altemative'when other means of obtainingdata and information'are not practical toimplement; a means of reducing uncertainty;'an assessment of the state'of scientificuncertainty; a meais of exploring the state ofknowledge on a particular topic conple-menting and supplementing other sources of-sientific and technical information; etc. Itwould be helpful for the BTP to collect anddiscuss all of the various descriptions of thebeneficial use of expert judgment in one 'place in 'order for the reader to better

D-1 D-1 ~~~~NUREG-1563

interpret the Staff's apparently broader viewon when the use of expert elicitation mightbe appropriate, and to what purpose. Thisfurther discussion possibly could be addedto [Section] 4(1) as a supplement. It alsoappears that the final three items in theabove list of conditions are often simply asubset of the first condition. Each of thefinal three conditions can be the result of aDOE decision that further data collectionand analysis is not practical, but, in somecases such as the above example, the con-dition likely could be mitigated by additionaldata collection and analysis. The BTPrepeatedly admonishes against substitutingexpert judgment for traditional analyses, butin each case makes cost and practicality aprominent consideration. As the regulator,the NRC should be most concerned withsafety considerations, which can only bederived from a high-quality license appli-cation firmly grounded on data and analyses.As in every regulatory arena, it is theapplicant's responsibility to weigh costagainst quality in the preparation of itslicense application, and it is the regulator'sresponsibility to judge whether the productis adequate. The BTP gives helpful guidanceregarding how the regulator might view thequality of DOE's application, but, [we]believe, errs in emphasizing cost andpracticality on the part of the applicant as apotential measure of compromise indetermining the "reasonable assurance" thatthe applicant's safety case is adequatelyproven. The text of this BTP should berevised to reflect that the NRC's primaryregulatory role, and highest priority, is topromote safety. As it stands, the unfortunateimplication in this BTP is that cost andpracticality for the applicant are acceptablemeasures against which to weigh safety inthe regulatory proceeding.

And finally, despite DOE's view2 that muchof [Sections] 1 and 2 of the BTP is anunnecessary review of past experience withthe use of formal expert judgment exercises,r[he State] believes that it is a usefuldescription that adds basis to the staff'spositions as set out in [Section] 3 and further

2 Expre:ed during a joint NRC/DOEState of Nevada telephoneconference call on April 23, 1996.

discussed in [Section] 4. The use of expertjudgment in a licensing proceeding in thisunique case will remain a prominent issuethroughout, and it will be important tounderstand as thoroughly as possible thecurrent staff's basis for the guidance whichit provided to the applicant and otherparties prior to preparation of the licenseapplication. It must be remembered that, inthe licensing proceeding the product of thisguidance may or may not be foundacceptable, and the original basis for theguidance could be integral to that decision.

Response

With respect to the State of Nevada's first generalcomment, that missing from the guidance is somediscussion that it is important for DOE todocument its decisions, and their bases, when itemploys an expert elicitation process, the staff isfully aware of this issue and generally shares asomewhat related concern. In fact, DOE's use ofexpert judgment, during site characterization, wasfirst identified as an issue by the NRC staff in thecourse of its review of DOE's Site Characteri-zation Plan (SCP-see DOE, 1988). In its SiteCharacterization Analysis (SCA) Comment 3, thestaff expressed the concern that DOE might"... rely [too] heavily on the use of expertjudgments (e.g., expert elicitations) to supply thelicensing information or to substitute for quanti-tative analyses" (NRC, 1989; pp. 4-8-4-40).Because of NRC's regulatory interest andoversight role in the repository program, the staffhas consistently expressed the view that DOEneeds to document its decision-making record soas to allow for the identification and resolution, atthe staff level, of any potential licensing issues.The staff repeated its concerns later, during theconduct of DOE's site characterization program.

That being said, it is generally recognized thatDOE's repository program has evolvedsignificantly beyond that described in the 1988SCP As DOE prepares its future program plansfor site characterization and the repositoryViability Assessment, the staff believes that DOEunderstands NRC's overall intent to ensure thatthere is transparency in its (DOE's) decision-making process and fully expects DOE to providesufficient documentation to support its decisions,including those that relate to the use of formalexpert judgment. Consistent with DOFs own

NUREG-1563 DD-2

Principles and Guidelines for Formal, Use of ExpertJudgment by the Yucca Mountain Site Character-ization Project Office (hereafter referred to as'thePrinciples and Guidelines), the staff also expectsthat DOE would prepare the necessarydocumentation, as recommended by'the State inits comment. The staff believes that this point isapparent throughout Section 1 of the BTP and'acknowledged specifically elsewhere in the BTP(e.g., in Section 4, refer to the following locationsin the discussion: paragraph 2 (in the intro-duction); Technical Position 1, paragraph 1; andTechnical Position 2, Step No. 9, paragraph 1).Therefore, the staff does not agree with the Statethat it is necessary to provide further guidance orto include additional information in the BTP.

NWith respect to the State of Nevada's secondgeneral comment, ". . . to collect and discuss all ofthe various descriptions of the beneficial use ofexpert judgment in one place. . .'," the staff has noobjection to including'additional information. Thestaff has added the following'paragraph to theintroduction of Section 4 of the BTP to addressthe State's request:

In closing, it should be noted that thisNUREG discusses the rationale for' and thepotential benefits derived from the use ofexpert judgment'in a number of places.Without intending to limit those discussions,the following summary is provided. Section 1describes the ubiquitous nature of expertjudgment in scientific and technicalactivities. Section 1.2.1 provides adescription of typical outputs of an expertelicitation and the potential uses of theexpert judgments in the evaluation ofrepository performance assessment. Section1.3'provides actual examples of the NRC useof expert judgments, including those used inPRA. Section 3 provides the' technicalposition on conditions warranting theconsideration of a formal expert elicitation.Later in this section, the staff elaborates onthese conditions and discuss the benefitsderived from obtaining expert judgments.Finally, Appendix A provides a discussion ofexpert judgment from a definitionalperspective and Appendix B provides resultsand lessons learned from the

NRC-sponsored formal climate expertelicitation..

In its third general comment, the State of Nevadaobserved that the ". . . implication in this BTP isthat:cost and practicality for the applicant (DOE)are acceptable measures against which to weighsafety'in the regulatory proceeding... ." The staffdoes not share this view of the implications of theBTP. For example, paragraph 2 of Section 1.4 ofthe BTP states that:

.. . under appropriate circumstances, it isacceptable to supplement data and analyseswith the opinions of experts as part of thesupport for demonstrating compliance withNRC's geologic disposal regulation, and that,under certain circumstances, these opinionscan be obtained using a formal andwell-documented process...."

In making this statement, in recognition thatcompliance with NRC regulations is the measurefor judging safety in an NRC licensing proceeding,the staff also provided additional caveats (high-lighted) to the BTP (see Section 1.4, paragraph 5):

The NRC staff recognizes that DOE has theflexibility to determine whether the costs andbenefits of performing an expert elicitation areadvantageous when compared with'the costsand benefits ofperforming theoretical analysesandlor gathering additionalfield andexperimental data. That being said, however,the use of expert elicitation should not beconsidered as an acceptable substitute fortraditional analyses based on adequate fieldor experimental data, when such data arereasonably available or obtainable, or theanalyses are practicable to perform. Nor canthe use of a formal elicitation process, evenwhen conducted in a manner consistent withguidance provided in this BTPE guarantee thatspecific technical conclusions will be acceptedand adopted by the staff a Licensing Board,the Commission itself or any other party to apotential HLWlicensing proceeding. Rigidadherence to a sound elicitation process, inand of itself, in'no way guarantees that theresulting judgments will be sufficient to meetthe applicant's burden of proof regarding thesubstantive issues addressed by theelicitation. Nonetheless, expert judgmentsobtained through an evidently flawed orpoorly documented process will weaken their

D-3 NUREG-1563

poorly documented process will weaken theirability to support demonstrations ofcompliance.

It should also be noted that nothing in thisBTP precludes the use of expert judgmentobtained through informal means by DOE,in the preparation of a potential licenseapplication. As has been the case in previousregulatory activities, the staff will accept forreview the results of formal or informaljudgment so long as the rationale associatedwith the judgment is adequate, transparent,and sufficiently documented.

However, in order to avoid any ambiguity on thissubject, the staff has revised the first sentence ofthe fifth paragraph of the "Introduction" toSection 1 of the BTP to emphasize that NRC'sprimary regulatory role and highest priority is toensure public health and safety; the revision willread as follows:

"With this notion in mind, current NRCpolicy is to encourage the use ofprobabilistic risk assessment (PRA)state-of-the-art technology and methods as acomplement to the deterministic approach innuclear regulatory activities (NRC, 1995; 60FR 42622)4 and in keeping with theCommission's paramount responsibility toprotect public health and safety...."

2EPA's 1985 HLW standards (50 FR 38066) adopted aprobabilistic perspective when making compliancedeterminations Because of the uncertainties inherent in thegeologic disposal of nuclear waste, it is antiipated that aprobabilistic treatment of the performance of the wastedisposal system will continue to be the regulatory approach.

In providing this clarification, the staff wishes tofinally note that it is DOE's ultimate prerogativeand responsibility to adopt a strategy fordemonstrating compliance with NRC'sregulations. The intent of this NUREG, therefore,is to allow DOE the maximum flexibility inchoosing an approach, as long as an effectivedemonstration of compliance with the regulationscan be made.

In its final comment, the State of Nevada disputesDOE's views regarding the value of or the needfor the background information contained inSections 1 ("Introduction") and 2 ("RegulatoryFramework") of the BTR The staff believes that

this information provides a helpful context for thetechnical positions themselves insofar as itsummarizes the history and considerations thatrelate to guidance on this particular subject. Theinformation may be of value, for example, to amember of the public and thereby furthers NRC'sinterest in conducting its regulatory activities insuch a way that those activities are under-standable to all interested parties. (See NRC(1996) for a discussion of the Commission's"openness policy" with respect to how theCommission conducts its regulatory activities.)

U.S. Department of Energy

The following comments were submitted by DOE(see Brocoum, 1996):

The NRC's "Branch Technical Position onthe Use of Expert Elicitation in theHigh-Level Radioactive Waste Program"provides guidance to develop a structuredprocess to conduct formal expert elicitations.DOE and NRC seem to have consistentpositions on the general steps that areappropriate for these structured exercises.DOE has identified no substantivedisagreements with respect to the processthe NRC has outlined for elicitation and itsassociated documentation. DOE followedeach of the nine steps specified in the NRC'sprocess while conducting its recentlycompleted "Probabilistic Volcanic HazardAnalysis for Yucca Mountain, Nevada"(Geomatrix Consultants/TRW Environ-mental Safety Systems, Inc., 1996).

DOE's only major comment on the BTPcenters on the possibility that the NRC mayoffer additional guidance on the use ofexpert elicitation in the area of performanceassessment. The Department believes thatadditional guidance is unnecessary.

As stated in DOE's June 1, 1995, letter toNRC (Brocoum, 1995), DOE's elicitationprocess will identify and document the basisfor any judgment, and this basis couldinclude both site-specific informationdeveloped during site characterization(including qualitative, descriptive, andquantitative analytical information) as well

NUREG-1563 Do

as information from other relevant or similarsettings.

Response

With regard to DOE's first overall comment, that".. . DOE and NRC seem' to have consistentpositions on the general steps [for the formalelicitation of expert judgments]. . ." and that DOEhas ". . . identified no substantive disagreementswith respect to the process the NRC has outlinedfor elicitation and its associated documen-tation... ," the staff welcomes DOE's view. Thestaff also believes that it may be possible toresolve, at the staff level, the staff's earlierconcerns 'expressed in SCA' Comment 3 (e.g.,criteria for the use of expert elicitations areneeded in light of the requirement for a licenseapplication to be as complete as possible givenreasonably available information). (See thedis'cussion in Appendix E for more on the staff'sreasoning in this area.)

The staff is mindful, however, that despite thisapparent convergence in' thinking, 'the instructioncontained in DOE's Principles and Guidelines isnot identical to the guidance contained in thisBT? DOE should view this BIT as the primaryguidance, in this area, although as with otherNRC staff guidance, it may choose to pursuealternative approaches, at its own risk. Inaddition, consistent with the BTP, the staff hasidentified a number of specific concerns withDOE's Principles and Guidelines documentationitself, which appear, along with a possible path totheir resolution, in Section E-2 of this document.

With regard to DOE's second overall comment,that additional guidance is not necessary in thearea of performance assessment, the'staff will takethis comment into account in due course but hasmade no decision as yet on this option. In view ofthe potential that new performance' measures maybe established for a potential repository at YuccaMountain, pursuant to 'the Energy Policy Act of1992, and consistent with the'findings andrecommendations of the recent National Academyof Sciences (NAS) report (see National ResearchCouncil, 1995),3 the staff wishes to provide for the

31n addition to the recent NAS recommendations the staff notes thatthere are pending legislative proposals that could affect therrgulation of HIP at Yucca 'Mountain. See summary in the 1994Fndins and Recommendations of the Nuclear Vaste ¶IchnicalReview Board (NWTRB-see NWTF.B, 1995; pp. 46-48).

possibility that site-specific standards maynecessitate the development of additionalimplementing guidance. The staff, therefore,continues to leave open this option, as noted inSection 1.4, paragraph 3 of the BIT (e.g., that it

J"...is considering the potential need for futureguidance to identify those specific aspects of aperformance assessment for which the applicationof expert judgment may or may not beappropriate.-. ."). Currently, the staff has no plansfor, or resources devoted to, preparing anyadditional guidance in this area. If the staffdecides that additional guidance should beconsidered, it will explain its reasons and offer theopportunity for comment by DOE (and others).

General Comments (Prescribed Use ofElicitations in the Area of PerformanceAssessment)

In Section 1.4 of the BTP, NRC is appropriately'silent about the specific technical issues for whichexpert elicitation should or should not be applied,except in the area of performance assessment.DOE would like clarification of why the staffbelieves they should consider additional guidance(page 11)". . to identify those specific' aspects ofa performance assessment for which theapplication of expert judgment may or may not beappropriate." DOE believes that once the NRCstaff have set out the process, as described in theBTP, it is the applicant's prerogative to decide ifand how its use is advantageous to supportarguments for licensing.

DOE believes that the BTT may suggest genericcircumstances when use of expert elicitation isappropriate, but that it is not appropriate toprescribe categories or topical requirements forthese exercises.' DOE will have to balance manyfactors in selecting the topic and scope for expertelicitations. DOE is currently evaluating theadvisability of conducting expert elicitations inseveral areas supporting development of its TotalSystem Performance Assessment (ISPA) for theViability Assessment,' including: scenario analysis

"and associated estimates of probability,parameter'uncertainty and bounding caseidentification, and certain aspects of processmodel abstraction'and conceptual model evalu-ation. DOE's plans currently do not include anelicitation on the totality of the TSPA submittalforming the basis of a license application. DOE

D-5 NUREG-1563

expects to discuss our future plans in this areawith the NRC when our planning has matured.

DOE notes that development of prescriptiveguidance for performance assessment is inconflict with the BTP description (page 11) whichstates ". . . under appropriate circumstances, it isacceptable to supplement data and analyses withopinions of experts as part of the support fordemonstrating compliance... ." Decisions as towhen or whether to conduct an elicitationexercise, the identification of specific issues ortopics, or other "appropriate circumstances" aremanagement prerogatives of the DOE. DOE isconcerned with the potential for prescriptivedefinitions or circumstances that might limitmanagement's alternatives or options to useexpert elicitations as part of the technical basisfor our compliance argument(s). DOE intendsthat any use of expert elicitations in the area ofperformance assessment be consistent with itsPrinciples and Guidelines.

Response

In this general comment, DOE has requestedclarification regarding why the staff believes itmight consider additional guidance"... toidentify those specific aspects of a performanceassessment for which the application of expertjudgment may or may not be appropriate... ." Asnoted above, in response to DOE's second overallcomment, it is premature to decide whether and,if so, what additional guidance is appropriate orto attempt to predict those specific aspects of aperformance assessment for which the applicationof expert judgment may or may not be appro-priate. If the staff decides that additionalguidance should be considered, it will explain itsreasons and offer the opportunity for comment byDOE (and others).

tangential or marginal issues. Section 2.2 of theBTP is a good example.

1. Section 1 ("Introduction"): The"Introduction" section is somewhat confusingbecause it is not clear whether the NRC'sintent is to provide guidance on the genericuse of expert elicitation (i.e., in any repositoryprogram) or restrict the guidance to theYucca Mountain Project. It is also not clearto which part of the NRC this BTP applies.

DOE suggestions: Include a statement of"scope" to describe the NRC's intentions indeveloping the BTP generic or project-specific guidance. A suitable description ofscope appears to be available (with minormodifications) in Section 4, Discussion, onpages 22 and 23.

Also, include a reference to the Division ofWaste Management, or the Branch(es) withinit that are responsible for preparation of theBTP. This information is not obvious andonly now occurs in the "Foreword."

Response

The staff believes that the clarification requestedby this comment already exists within the body ofthe BTP, as discussed below.

With regard to DOE's first comment, regardingthe applicability of the BTP to NRC's regulatoryprograms, it should be noted that this BTPapplies to the HLW program. There are a numberof references to that effect, both explicitly as wellas implicitly throughout the BT. For example, seethe "Foreword," 'Abstract," and "Introduction"(5th paragraph). Because this guidance wasdeveloped for application to the HLW program, itis the staff's view that it would apply, generically,to any potential repository licensed by NRC.

As regards DOE's.second comment, as indicatedon both the cover page and in the "Foreword" ofthe guidance, the BTP was prepared by Divisionof Waste Management (DNVM) staff (Kotra, Lee,and Eisenberg) with assistance from its technicalassistance contractor, the Center for NuclearWaste Regulatory Analyses-CNWRA(DeWispelare). As indicated, these authorsincorporated the many useful comments andsuggestions of other DWM and CNWRA staffmembers, representatives of the Office of the

Specific Comments

The specific comments below (Nos. 1-5) refer toformat and content issues or subject headings thatmay be prescribed for BTPs. DOE's theme inthese comments is that, early on and for somelength, they tend to focus the document on a lookback instead of forward. DOE's general concernis that together Sections 1 and 2 tend to cloud thepoints of agreement between the two agencies inSections 3 and 4 by including and discussing

NUREG-1563 D-6

General Counsel, and members of the staff fromthe Office of Nuclear Reactor Regulation, theOffice of Nuclear Regulatory Research, and theOffice for Analysis and Evaluation of OperationalData. (These individuals are identified in the'Acknowledgments" section of the BTP.) Lastly,the final technical positions reflect discussionswith and recommendations from NRC's AdvisoryCommittee for Nuclear Waste (ACNW), which aredocumented in Appendix F. As noted in the"Foreword," separately and at a later time, thestaff may expand and refine this guidance forapplication in other areas of NRC regulation.

2. Section 1.1: The Background" ection isdistracting. To revisit and critique DOE'spast uses of expert judgment or priorelicitations recapitulates a comment recordalready made by the NRC on the DOE's pastefforts and documentation. It distracts fromthe BIT's purpose in establishing;expectations on how these exercises areperformed in the future.

DOE suggestion: Delete this section orcondense significantly.

establish process expectations for futureelicitation applications.

DOE suggestion:'Condense significantly ormove the material in Section 15 either tofront material such as a "Foreword," or anAppendix.

Response

In conformance with a standard format for allDWM staff guidance documents, the informationin question, in Section 1.5, is typically containedin staff technical positions, and is intended tocommunicate the staff's overall goals in issuingregulatory guidance.

4. Section 2.1 ("Regulatory Framework-10 CFR Part 60"): The last paragraph in thesection is' a good example of raising issuesthat are not relevant to the purpose of theBTP, i.e., establishing process expectations forexpert elicitations. It refers to a pastcomment and response dialog on the DOE'sSCP.

DOE suggestion: Delete the paragraph.

Response

The staff does not share DOE's view that the"Background" section (Section 1) of the BTPdistracts from the technical position statements.'Although the information is historical in nature,the staff believes this information provides auseful context for the technical positionsthemselves insofar as it summarizes the history,reasoning, and staff considerations that go into adecision to issue guidance on a particular subject.As noted in its response to the State of Nevada(above), the staff believes that this informationcould provide a helpful context for the 'iechnicalpositions themselves insofar as it summarizes thehistory and considerations that relate to guidanceon this particular subject.'The info'ination"maybe of value, for example, to a-member of the public and to this extent, furthers NRC's interestin conducting its regulatory activities in such a"way that those activities are understandable to allinterested parties.

3. Section 15: The "Branch Technical Positionsas Technical Guidance" section provides littlevalue toward the purpose of.theguidance, to

Response

The staff does not share DOE's view concerningthe relevance of the paragraph in question to theBTP. As a result of the staff's review of DOE'sstatutory SCP, several subsequent reviews of DOEsite characterization activities, and recommen-dations of certain advisory bodies-the ACNWand the NWIRB (all of which are summarized inSection 1.1), the staff decided to develop guidancein this area.

5. Section 2.2 ("Commission Policy StatementConcerning the Use of PRA Methods inNuclear Regulatory Activities"): This sectionhas little apparent relevance to the develop-ment of guidance for the use of expertelicitation, and only the last sentence in theentire section states the essence of the sectionthat could bear the purpose of the BT. The"Regulatory Framework" section, in general,is distracting and does not appear to berelevant to the purpose of the BTP as processguidance.

DOE suggestion: Delete Section 2.24 move thelast sentence [with edits], "The technical

D-7 NUREG-1563

positions cited in Section 3 are consistentwith the recommended process steps [fromthe NRC's staff's Probabilistic RiskAssessment Working Group."] to anappropriate place in Section 3. Considercondensing Section 2 and especially Section1 significantly.

Response

The staff does not agree with DOE's comment todelete the material contained in Section 2.2 of theBTP. In the staff's view, a brief discussion of theCommission's Policy Statement, concerning the useof probabilistic risk assessment (PRA) methods innuclear regulatory activities (see NRC, 1995; 60FR 42623), contributes useful information to theBTP that places it in a broader regulatory context.The staff believes that the Commission's decisionregarding "emerging" PRA technologies in thePolicy Statement is supportive of the BTP.

6. Section 3 ("Branch Technical Positions")-page 17, Technical Position 2(a), line 2: Theinclusion of the word "defensible" in thedescription of the procedure includes acriterion whose definition is arbitrary until itcan be determined whether the procedurewas in fact successfully or unsuccessfullydefended. Furthermore, "defensibility" or the"need to defend" are managementconsiderations which are inappropriate topicsfor the guidance.

7. Technical Position No. 3 in Sections 3 and4-pages 21 and 36, respectively: The timeframe implied in the description on page 21is not clear. One interpretation wouldindicate that expert judgment, expertelicitation, and peer review are intended to besnap-shots in time of the experts' views of theissue in question based on the informationavailable at the time the exercise wasconducted. An alternative interpretationwould imply that the guidance envisions anelicitation process that is iterative and mightnever be completed if additional data werecontinuously available.

On page 36, the expectation for re-examiningand updating the results of past elicitationsto new, relevant data needs clarification. It isthe DOE's prerogative and responsibility toensure that the materials submitted to theNRC for a licensing action are current.When new information becomes available,DOE will evaluate and document itsrelationship to the assumptions and rangevariation established in prior elicitations. Itis not necessary, however, for the BTP toestablish expectations for how an elicitationis re-evaluated in light of new information,beyond the means of being well documented.

DOE suggestion. Use wording that does notimply a process-specific means ofre-examining the results of a prior elicitationin light of new data.

There are similar references to "defensible"in Section 4 ("Discussion"), p. 25, Item (2)and p. 29, 3rd paragraph.

DOE suggestion: Revise these references to"defensible" to describe thoroughlydocumented processes.

Response

The staff's intent behind the use of the word"defensible" in Technical Position 2(a) (andelsewhere) was to reference the obligation of thepotential applicant (i.e., DOE) to demonstrate theacceptability of both the elicitation process andthe outcome of that process. With thatunderstanding in mind, the staff has no objectionto making the requested change.

Response

NRC expects that a potential license applicationfor a geologic repository to be complete aspossible in light of the information that isreasonably available at the time of docketing (10CFR 60.24(a)). As a potential applicant, the intentof this technical position was to remind DOE ofthe staff's expectation on this matter. However, toavoid any additional confusion in this area, thestaff has revised Technical Position No. 3 to readas follows:

If information from an expert elicitation is tobe submitted in support of a licenseapplication, and if additional data orinformation becomes available, subsequentto the completion of the elicitation, whichcould change opinions or judgments

NUREG-1563 D-8

obtained in the formal elicitation, the resultsof the elicitation should be reexamined andupdated, as appropriate. In addition'to theinformation requested above, documentationshould include a detailed 'description of theupdating process.

Consistent with these changes, the staff has alsorevised'the second paragraph of the rationale forthis technical position (in Section 4) to read asfollows:

The methods'of updating the expertjudgments'range from the use of Bayes'Theorem, for statistical updating, toconducting another set of individualelicitations for the same or a'different set ofexperts. Whichever method is used forincorporating the new data or informationinto the existing expert opinions, it should bethoroughly documented to provide a'transparent view of the updating process andresulting judgments.

8. :Section 4 ("Discussion" behind TechnicalPosition No. 2)-page 26, 2nd full paragraph,lines' 6-9: The staff's expectations with regardto the use of subject-matter experts, to betterdefine the objective of the elicitation, are notclear.Using subject-matter experts to definethe'objectives of the elicitation on 'which they

- have been asked to participate,'represents apotential to create coiflicts of interest-especially financial and pfofessional conflicts

* ' of interest.'Discussion of the'sameconsideration on page 30 is clear and does

- not contain the apparent ambiguities foundin the discussion on page 26.

DOE suggestion: Delete the sentence "Whatthis figure shows,' into' its constituent parts(Step No.'3)" to remove this ambiguity andpotential inconsistency that could create aconflict of interest. ' ':-' i

Response

The staff does not agree with DOE's character-ization'of this issue. It is the staff's view that thereis no ambiguity regarding the sentence'in question(introductory discussion in Section 4 behind'Technical Position No. 2-page 26). The staff isexpressing the 'view in this section of the BTP thatsubject-matter experts, working with the

generalist, can help to better define the overallobjectives of any particular elicitation. Althoughnot expressed in the BTP itself, it is the staff'sview that the normative expert and generalist,working with the elicitation sponsor (i.e., DOE),reserve the right to limit the scope and extent ofany proposed elicitation given, of course, contrary

'(documented) advice from the subject-matterexperts themselves.

9. Appendix C: This Appendix is relevant onlyto Section 2.2 of the draft Branch TechnicalPosition.

DOE suggestion: Delete Appendix C ifSection 2.2 is deleted, as suggested.

Response

The staff's response to this comment is addressedunder DOE specific comment no. 5.

U.S. Nuclear Waste Technical Review Board

The following comments Were submitted by theNWIRB (see Cantlon, 1996):

The Board has had a long-standing interestin the use of formal expert judgment byDOE as it characterizes the potentialrepository site at Yucca Mountain andmoves toward a possible application toconstruct a permanent undergroundrepository there. The Board has addressed'the need for DOE to develop soundelicitation methodologies, to involve outsideexperts in any formal elicitation conducted,and to resolve possible conflicts with theNRC well before the submission of anylicense application.

The BTP lays out in a thoughtful and well-argued manner the key issues involved incarrying out a successful and reliable formalelicitation. For the most part,' the BTP hasincorporated the best'curint thinking ofdecision analysts who have examined this

'area'as well as appropriate lessons fromprevious NRC experience. The BTP correctlyrecognizes that DOE ultimately bears theburden of convicing a Licensing Board, andprobably'others as well, that its use of expert

' judgment' on a" 'parti6ular issue has properly

NUREG-1563

characterized the relevant uncertainties andthat their magnitude is, in fact, acceptable.

There are, however, some areas and issueswhere the Board feels greater specificity inthe BTP might be helpful.

* Based on its own analyses, does the NRC[staff] believe that there are technicalissues that are so critical to demonstrat-ing the safety of a repository system thattheir resolution should be based almostexclusively on primary data, minimizingthe reliance on expert judgment? If so,what are they?

* Has the NRC staff developed views orguidance as to when "it is infeasible orimpossible to collect data" and whattypes of management challenges wouldsupport a determination that "datacollection [has] become "prohibitivelyexpensive" (p. 24)?

* Are there any circumstances where theNRC staff might not accept the results ofa DOE elicitation that was conducted inaccordance with the process outlined inthe BTP? If so, what are they?

* Can some guidance be offered to DOEon the conditions and circumstances thatjustify departure from equal weighting ofexperts' judgments?

NRC might wish at some point to explore,with DOE, the related question of howbiases of experts might be minimized whentheir judgment is rendered informally,although such an effort could be outside ofthe scope of the BTP.

Response

With respect to the NNVTRB's first comment, itshould be noted that it is DOE's responsibility, inthe first instance, to formulate a strategy fordemonstrating compliance; the staff is, therefore,reluctant to constrain DOE by identifyingtechnical issues that must be resolved almostexclusively by the use of primary data, rather thanexpert elicitation. For issues DOE elevates tocritical importance, DOE should make anadequate, but decisive, case. The responsibility

will then fall to the staff to evaluate theacceptability of DOE's analysis, in the context ofa complete license application. DOE will have todecide how much reliance to place on variousfeatures and components of the repository systemin order to demonstrate compliance with the total-system performance objective. The staff shouldnot attempt to predict, in the BTP, whether DOEmay choose to take credit at all for certainpotentially helpful components or features (e.g.,cladding of spent fuel as an additional barrierisolating the waste). The repository system ishighly complex and nonlinear; modeling therepository system is based on limited data.Determination of critical issues is difficult andmight be subject to change under theseconditions, even presupposing that DOE has afirm strategy for demonstrating compliance. Someimportant issues of repository performance (e.g.,the validity of long-term predictions), mustnecessarily be based on scant primary data. Forall these reasons, the staff is not inclined tospecify issues, in this BTP, requiring resolutionalmost exclusively using primary data. Thesepoints are either made already in the responses topublic comments or may be found in Section 1 ofthe BTP.

With respect to the NWTRB's second comment,concerning possible guidance on when it might be". . . infeasible or impossible to collect data . .or when the cost of data collection might be"prohibitive," the staff tends to think that thismatter, like the first NWTRB comment, is betterleft for later consideration, in the context of atopical report or a completed license application,for example. However, the staff believes that it isimpossible to collect some data, such as directconfirmation of long-term predictions ofradionuclide migration at the site. Nevertheless,the staff has not developed guidance on theseissues. Furthermore, DOE, the party responsiblefor collecting the necessary site characterizationdata, as the potential applicant, has not requestedfurther guidance at this time. Moreover, as theparty responsible for preparing a potential licenseapplication, it might be more appropriate to deferthis question to DOE. In this regard, the staffnotes in Section 1.4 ("Purpose of the BTP"-seeparagraph 6) of the BTP that:

"DOE has the flexibility to determinewhether the costs and benefits of performingan expert elicitation are advantageous

NUREG-1563 D-10

compared to the costs and benefits ofperforming theoretical analyses and/orgathering additional field and experimentaldata."4

In response to the NWTRB's third comment,concerning .. . circumstances where the NRCstaff might not accept the results of a DOE*elicitation that was conducted in accordance withthe process outlined in the BTP. . . ," the staff'spreference is to be cautious on what adherence tothe BTP can provide and not to predict where itcan fail. Hypothetically, though, circumstancescould arise in which the staff might not accept theresults of an elicitation. For example, onecircumstance might be the existence ofsubsequent analyses or opinions that contradictthe conclusions reached by an elicitation. Inanticipation that such situations cannot becompletely discounted in the future, the staffacknowledged this possibility (see Section 1.4 ofthe BTP). However, the staff believes that -

following the guidance set forth in the BTPenhances the chances of the acceptance of theelicitation and its results.

In response t the NWTRB's final commentregarding conditions or circumstances that mightjustify departure from equal weighting of experts'judgments, although tere may be somecircumstances in which unequal weighting mightbe appropriate, the staff prefers not to speculateon what those conditions or. circumstances mightbe. The staff believes that the response to theNWTRB's question is more appropriatelyaddressed by DOE, again, as the practitioner.Whatever weighting factors are used, DOE shoulddocument its rationale for selecting them.However, the reader is referred to-DeWispelare etal. (1994) and the Senior Seismic Hazard AnalysisCommittee (1995) for discussion and examples ofhow this issue has been dealt with in specificapplications.

* Lastly, with respect to the NWTRB's recommen-dation, that NRC and DOE explore the relatedquestion of how biases of experts might beminimized when their judgment is rendered

4As noted in the staff's response to the State of Nevada's thirdgeneral comment, this BTP is conistent with current NRC policyvwhich encourages the use of PRA state-ofthe-art technology andmethods as a con Wiement to the deterministic approach in nuclearregulatory activities and is also in keeping with the Commission'sparamount responsibilit to protect public health and safety.

informally, the staff agrees with the NWTRB thatthis issue is beyond the scope of the BTP andshould be addressed separately.

References

Brocoum, S.J., U.S. Department of Energy/Officeof Civilian Radioactive Waste Management, letterto JJ. Holonich, Office of Nuclear Material Safetyand Safeguards/Division of Waste Management[Subject: " 'Principles and Guidelines for FormalUse of Expert Judgment by the Yucca MountainSite Characterization Project Office' andResolution of Site Characterization AnalysisComment 3"], Yucca Mountain SiteCharacterization Office, June 1, 1995.

Brocoum, S.J., U.S. Department of Energy/Officeof Civilian Radioactive Waste Management, letterto Secretary, Docketing and Services Branch/U.S.Nuclear Regulatory Commission [Subject: "U.S.Department of Energy Comments on [the] U.S.Nuclear Regulatory Commission Draft Branch'Technical Position on the Formal Use of ExpertElicitation in the High-Level Waste Program"],Yucca Mountain Site Characterization Office,May 13, 1996.

Cantlon, J.E., U.S. Nuclear Waste TechnicalReview Board, letter to Secretary, Docketing andServices Branch/U.S. Nuclear RegulatoryCommission [Subject: "NWTRB Comments onthe NRC's Draft BTP on the Formal Use ofExpert Elicitation in the HLW Program"],Arlington, Virginia, May 10, 1996.

DeWispelare, A.R., et al., "Background Report onthe Use and Elicitation of Expert Judgment," SanAntonio,`Texas, Center for Nuclear WasteRegulatory Analyses, CNWRA 94-019, September1994. [Prepared for the U.S. Nuclear RegulatoryCommission.]

Geomatrix Consultants/TRW EnvironmentalSafety Systems, Inc., "Probabilistic VolcanicHazard Analysis for Yucca Mountain, Nevada,"San Francisco, California, Document No.BAOOOOOOO-1717-2200-00082 (Rev. 0), June 1996.[Prepared for the U.S. Department ofEnergy/Office of Civilian Radioactive WasteManagement.]

Holonich, JJ., Office of Nuclear Material Safetyand Safeguards/Division of Waste Management,letter to R.R. Loux, State of Nevada/Agency for

D-11 D1NUREG-1563

.

Nuclear Projects [Subject: "State of NevadaComments on U.S. Department of EnergyActivities Related to the Use of Expert Elicitationin the High-Level Waste Program"], U.S. NuclearRegulatory Commission, August 22, 1995.

Loux, R.R., State of Nevada/Agency for NuclearProjects, letter to JJ. Holonich, Office of NuclearMaterial Safety and Safeguards/Division of WasteManagement [Subject: "'Principles andGuidelines for Formal Use of Expert Judgment bythe Yucca Mountain Site Characterization ProjectOffice' and Resolution of Site CharacterizationAnalysis Comment 3"], Carson City, NuclearWaste Project Office, July 25, 1995.

Loux, R.R., State of Nevada/Agency for NuclearProjects, letter to J.H. Austin, Office of NuclearMaterial Safety and Safeguards/Division of WasteManagement [Subject: "State of NevadaComments on 'Draft Branch Technical Position onthe Formal Use of Expert Elicitation in theHigh-Level Waste Program' (February 1996)"],Carson City, Nuclear Waste Project Office,May 15, 1996.

National Research Council, "Technical Bases forYucca Mountain Standards," Washington, D.C.,National Academy Press, Commission onGeosciences, Environment, and Resources, July1995.

Senior Seismic Hazard Analysis Committee,"Recommendations for Probabilistic SeismicHazard Analysis: Guidance on Uncertainty andUse of Experts," Livermore, California, LawrenceLivermore National Laboratory, UCRLID-122160, 2 vols., August 1995. [Sponsored by theU.S. Department of Energv, the U.S. NuclearRegulatory Commission, and the Electric PowerResearch Institute. Also expected to be publishedas NUREG/CR-6372.]

U.S. Department of Energy, "Site Character-ization Plan, Yucca Mountain Site, NevadaResearch and Development Area, Nevada," Officeof Civilian Radioactive Waste Management,Nevada Operations Office/Yucca MountainProject Office, Nevada, DOE/RW-0199, 9 vols.,December 1988.

U.S. Environmental Protection Agency,"Environmental Standards for the Management ofSpent Nuclear Fuel, High-Level and TransuranicWastes [Final Rule]," Federal Register, Vol. 50, No.182, September 19, 1985, pp. 38066 - 38089.

U.S. Nuclear Regulatory Commission, "NRC StaffSite Characterization Analysis of the Departmentof Energy's Site Characterization Plan, YuccaMountain, Nevada," Office of Nuclear MaterialSafety and Safeguards, NUREG-1347, August1989.

U.S. Nuclear Regulatory Commission, "Use ofProbabilistic Risk Assessment Methods inNuclear Regulatory Activities [Final PolicyStatement]," Federal Register, Vol. 60, No. 158,August 16, 1995, pp. 42622 - 42630.

U.S. Nuclear Regulatory Commission, "PublicInvolvement in the Nuclear Regulatory Process,"NUREG/BR-0215, 1996.

U.S. Nuclear Regulatory Commission,'Availability of Draft Branch Technical Positionon the Use of Expert Elicitation in the High-LevelWaste Program [Notice of Availability]," FederalRegister, Vol. 61, No. 40, February 28, 1996, pp.7568 - 7569.

Wescott, R.G., et al. (eds.), "NRC IterativePerformance Assessment Phase 2: Development ofCapabilities for Review of a PerformanceAssessment for a High-Level Waste Repository,"U.S. Nuclear Regulatory Commission,NUREG-1464, October 1995.

NUREG-1563 D1D-12

APPENDIX ESTAFF VIEWS ON DOE'S 1995 PRINCIPLES AND GUIDELINES AND THE

POSSIBLE DISPOSITION OF SCA COMMENT 3

E-1 Introduction

The staff's original intent, as indicated incorrespondence to both the U.S. Department ofEnergy (DOE) and the State of Nevada (seerespectively, Austin (1995) and Holonich (1995)),was to comment on both DOE's Principles andGuidelines for Formal Use of Expert Judgment bythe Yucca Mountain Site Characterization ProjectOffice (Broc6um, 1995) (hereafter referred to asthe rinciples and Guidelines) and, in that comment, to reflect its consideration of the State'sreview (Loux, 1995), thereof, after a Fall 1995 U.S.Nuclear Regulatory Commission/DOE technicalexchange. However, the proposed technicalexchange was cancelled, at DOE's request. In lightof the canciellation, the staff decided to defer -comment until after public comments had beenreceived on the draft Branch Technical Position(BTP).

In its comments on the draft BTh, DOE indicatedthat it is now in "substantial agreement'? with theNRC staff's technical positions on the formal useof expert elicitation in the high-level wasteprogram (see Brocoum, 1996). For its part, the

'State commented that the draft BTP was"favorably responsive" to its earlier concerns (seeLoux, 1995). Therefore, the staff is inclined tobelieve that with publication of the BTP, thestaff's original intention, to comment on the DOEPrinciples and Guidelines, has been overtaken byevents (i.e., issuance and acceptanceof the NRCguidance). e

With these thoughts in mind, the NRC staff offers* the following comments and describes a possible

path to resolution, at the staff level, of theparticular Site Characterization Analysis (SCA)open item, SCA Comment 3, to which this issueapplies.

E2- -DOE's Principles and Guidelines

To address the concerns raised by the staff in itsSCA Coinment 3, the staff made tworecommendations to DOE (NRC, 1989; p. 4-10):

" State the criteria for the formal use of expertjudgment to ensure that objective, quantita-tive analyses based on empirical data areused in preference to expert elicitation,wherever possible.

* Modify the Site Characterization Plan, in anearly update, to assure that the requisite datawill be available."

Consistent with the staff's first iecommendation,.. . to state criteria for the formal use of expert

judgment. . ," DOE issued its 1995 Principles andGuidelines and in doing so, has argued that theinformation contained in it provides the necessarycriteria. Along with the information and directioncontained in the QualityAssurance RequirementsDocument (QARD), DOE suggests that its 1995Principles and Guidelines would be adequate toresolve, at the staff level, this particular openitem.

The staff has reviewed DOE's Principles andGuidelines and has a number of concerns. Thefirst is that DOE's QARD addresses only the-subject of "peer review" (see DOE, 1995; Section2.2.9) and does not treat the issue of elicitedexpert judgments (either f6rmally or informally).Second, language in the Principles and Guidelines,in many places, appears to confuse the conceptsof "expert judgment," "expert elicitation," and"peer review," concepts- that are, in the staff'sopinion, distinct. Because the staff believes thereis frequent confusion in the use and application ofthese terms, it decided to provide the necessaryclarification in its own BTP, including expandingthe definition of "peer review" over that providedearlier in Altman et al. (1988). These threesubjects are distinct and should be addressedseparately. A-third deficiency identified by thestaff in the Principles and Guidelines document isthat it contains no substantive discussionregarding potential procedures per se that wouldbe used to conduct a formal elicitation. If DOEintends to rely on its Principles and Guidelines asinstruction to Department management and staff,the NRC staff believes that a "how-to" statement,such as that contained in the recommendednine-step process in Section 3 of the BT? is

E1 E-1 ~~~~NUREG-1563

needed. Lastly, the staff believes that DOE'sPrinciples and Guidelines needs to acknowledge oraddress, as the staff did in the BTP (see Section4), how DOE management and staff would dealwith the potential for conflict of interest whenconducting a formal elicitation.

With respect to the'second staff SCA recommen-dation .. . to modify the Site CharacterizationPlan (SCP-DOE, 1988), in an early update, toassure that the requisite data will be available... ," DOE has not modified the SCP, as firstrecommended. However, it is generally recognizednow that DOE's repository program has evolvedbeyond that which was earlier described in the1988 SCP, for a variety of reasons. As DOEprepares its future program plans for sitecharacterization and the repository ViabilityAssessment, the staff fully expects DOE toprovide sufficient documentation to support itsdecisions, including those that relate to where itmight rely on formal use of expert judgment.Therefore, until the Department has had anopportunity to revise and update its sitecharacterization plan, it may be appropriate forthe staff to consider retraction of this earlierrecommendation, at this time.

E-3 State of Nevada July 1995Comments on DOE'S Principlesand Guidelines

As noted above, and subject to specific commentsand recommendations, the State has indicatedthat the draft BTP is "favorably responsive" to itsearlier concerns expressed to the staff concerningDOE's Principles and Guidelines (Loux, 1995).Further, as described below, DOE is being askedto revise its Principles and Guidelines, consistentwith the BTP. Thus, in light of this, the staffbelieves that further comment on the State's July1995 letter would serve no useful purpose.However, the staff does wish to point out, as itdid in its initial response to the State on thesecomments, that the views of DeWispelare et al.(1994) as well as the views of all other contractorreports (e.g., Bonano et a!. (1990) and SeniorSeismic Hazard Analysis Committee (1995)) arethe views of those authors and do not, necessarily,reflect the views on regulatory positions of theNRC staff. To the extent that these or any othercontractor reports are cited in this BTP, the onlystaff endorsement these documents receive is

limited to the particular points for which they arereferenced. Moreover, as regards the BTP'sreference to or consistency with any otherdocuments, the reader is reminded that for thepurposes of any potential license application, thebenchmark that will be used by the staff to judgethe acceptability of the process of any formalelicitation is the staff's technical positionsdescribed in this NUREG.

E-4 Staff Recommendation on theDisposition of SCA Comment 3

In light of the aforementioned comments andobservations, and because DOE is in "substantialagreement" with the NRC staff's technicalpositions in the BTP, the staff believes that thereis a sufficient basis to recommend that SCAComment 3 be closed, at the staff level, althoughon somewhat different grounds from thosesuggested by DOE in its 1996 letter to the staff(see Brocoum, 1996). However, in making thisrecommendation, the staff believes that DOE willneed to agree to the following course of action,with the attendant commitments or, an equivalentcourse of action.

1. The 1995 Principles and Guidelines should berevised to reflect DOE's acceptance of thestaff's BTP. Moreover, DOE's Principles andGuidelines should be revised to address therecommended changes to format and content,as noted by the staff in Section E-2, above.To summarize, these would include:

* Correction and clarification regardingDOE's use of the terms "expertjudgment," "expert elicitation," and"peer review."

* Substantive discussion regarding thespecific procedures per se that theDepartment and its contractors wouldfollow when conducting a formalelicitation.

* Direction to DOE management and staffregarding how to address the potentialfor conflict of interest when conducting aformal elicitation.

2. The current version of the QARD (DOE,1995) should be revised to include adiscussion of the treatment of "formal" expert

NUREG-1563 E-2

elicitation comparable to the discussionwhich already exists for "peer reviews."

3. DOE decisions on the need to use formalexpert elicitations should be transparent.DOE's Principles and Guidelines should berevised to ensure that its management andstaff prepare the necessary documentation topermit tracking of such decision-making.

E5 References

Altman, WD., et aL, "Peer Review for High-LevelNuclear Waste Repositories," U.S. NuclearRegulatory Commission, Generic TechnicalPosition, NUREG-1297, February 1988.

Austin, J.H., Office of Nuclear Material Safetyand Safeguards/Division of Waste Management,letter to SJ. Brocoum, U.S. Department ofEnergy/Office of Civilian Radioactive WasteManagement [Subject: "Use of Expert Elicitationin the High-Level Waste Program and Status ofSite Characterization Analysis Comment No. 3"],U.S. Nuclear Regulatory Commission, July 21,1995.

Bonano, EJ., et al., "Elicitation and Use of ExpertJudgment in Performance Assessment of High-Level Radioactive Waste Repositories," U.S.Nuclear Regulatory Commission,NUREG/CR-5411, May 1990. [Prepared by theSandia National Laboratories.]

Brocoum, SJ., U.S. Department of Energy/Officeof Civilian Radioactive Waste Management, letterto J.J. Holonich, Office of Nuclear Material Safetyand Safeguards/Division of Waste Management[Subject: "'Principles and Guidelines for FormalUse of Expert Judgment by the Yucca MountainSite Characterization Project Office' andResolution of Site Characterization AnalysisComment 3"], Yucca Mountain SiteCharacterization Office, June 1, 1995.

Brocoum, SJ., U.S. Department of Energy/Officeof Civilian Radioactive Waste Management, letterto Secretary, Docketing and Services Branch/U.S.Nuclear Regulatory Commission [Subject: "U.S.Department of Energy Comments on [the] U.S.Nuclear Regulatory Commission Draft BranchTechnical Position on the Formal Use of ExpertElicitation in the High-Level Waste Program"],

Yucca Mountain Site Characterization Office,May 13, 1996.

DeWispelare, A.R., et al., "Background Report onthe Use and Elicitation of Expert Judgment," SanAntonio, Texas, Center for Nuclear WasteRegulatory Analyses, CNWRA 94-019, September1994. [Prepared for the U.S. Nuclear RegulatoryCommission.]

Holonich, J.J., Office of Nuclear Material Safetyand Safeguards/Division of Waste Management,letter to R.R. Loux, State of Nevada/Agency forNuclear Projects [Subject: "State of NevadaComments on U.S. Department of EnergyActivities Related to the Use of Expert Elicitationin the High-Level Waste Program"], U.S. NuclearRegulatory Commission, August 22, 1995.

Loux, R.R., State of Nevada/Agency for NuclearProjects, letter to JJ. Holonich, Office of NuclearMaterial Safety and Safeguards/Division of WasteManagement [Subject: "'Principles andGuidelines for Formal Use of Expert Judgment bythe Yucca Mountain Site Characterization ProjectOffice' and Resolution of Site CharacterizationAnalysis Comment 3"], Carson City, NuclearWaste Project Office, July 25, 1995.

Senior Seismic Hazard Analysis Committee,"Recommendations for Probabilistic SeismicHazard Analysis: Guidance on Uncertainty andUse of Experts," Livermore, California, LawrenceLivermore National Laboratory, UCRL-ID-122160, 2 vols., August 1995. [Sponsored by theU.S. Department of Energy, the U.S. NuclearRegulatory Commission, and the Electric PowerResearch Institute. Also expected to be publishedas NUREG/CR-6372.]

U.S. Department of Energy, "Site Character-ization Plan, Yucca Mountain Site, NevadaResearch and Development Area, Nevada," Officeof Civilian Radioactive Waste Management,Nevada Operations Office/Yucca MountainProject Office, Nevada, DOEJRNV-0199, 9 vols.,December 1988.

U.S. Department of Energy, "Quality AssuranceRequirements and Description (for the CivilianRadioactive Waste Management Program),"Washington, D.C., Office of Civilian RadioactiveWaste Management, DOE/RW/0333P (Rev. 5),October 1995.

E-3 NUREG-1563

U.S. Nuclear Regulatory Commission, "NRC StaffSite Characterization Analysis of the Departmentof Energy's Site Characterization Plan, Yucca

Mountain, Nevada," Office of Nuclear MaterialSafety and Safeguards, NUREG-1347, August1989.

NUREG-1563 E-4

APPENDIX FDISPOSITION OF ACNW COMMENTS

After receipt and analysis' of public comments onthe February 1996 draft Branch 'Rchnical Position(BlP), the staff briefed the Advisory Committeeon Nuclear 'Waste (ACNW) on the staff'sdisposition of public comments and any proposedrevisions to the BTP based on those publiccomrnents. As a result of that briefings datedAugust 22, 1996, the ACNW submitted thefollowing' commentsl to the Commission.

Recommendation

This BTP provides important guidance to the(potential) applicant (the U.S. Department ofEnergy-DOE), affected units of government, andinterested parties, on the use of formally elicitedexpert judgment. The ACNW anticipates that theBTP will be immediately useful to the NRC staff,for example, in its evaluations of (or commentson) DOE's Probabilistic Volcanic HazardAnalysis for Yucca Mountain, Nevada"(Geomatrix Consultants/TRW EnvironmentalSafety Systems, Inc., 1996) and later comments onDOE's (ongoing) probabilistic seismic hazardassessments In the long term, the BTP willprovide valuable guidance to the DOE in thepreparation of its license application and to otherparties carrying out expert elicitations inconnection with the facility licensing process.

We wish to commend the NRC staff forcompleting the final draft BTT, which is desirablybrief and nonprescriptive. The applicant is left toits own creativeness on how to handle suchimportant issues as probabilities, methods ofaggregating uncertainties, data updating, and the

t1he indented ortions of this appndix quote the ACNWcomnments. Moreover, the staff hs responded o minor comments

* ~on the BTP made by the ACNW, as noted in the Transcript for the85th Meet of the ACNW-August 21-23. 1996.

2It should be noted that during the August 22. 1996, briefing of theACNW, DOE indicated that it contemplated a series of additionalexpert elicitations as part of its total-systemn performance assess-ment efforts. In an October 9, 1996, public meeting with theNuclear Waste Thchnical Review Board, DOE tentatively identifiedthe general subject areas for these additional focused elicitalionsas: unsaturated zone hydrology, waste package canisterdegradation, Thermal hydrology, waste form dissolution, andsaturated zone hydrology.

final form of the results. The Committee stronglyrecommends the prompt completion andpublication of the final draft BTP.

Residual Concerns

Although the ACNW welcomes and supports thesubject draft BTP, we have several residualconcems'regarding the use of formally elicitedexpert judgment in the decision-making process.The Committee does not intend that theseconcerns delay publication of the draft BT? Werealize that these concdrns could be addressed bya variety of means outside the BTP, includingworkshops,' letters, NUREGs, technical exchanges,and so on. These concerns are discussed below.The Committee believes that these residualconcerns should not delay the prompt publicationof the BTP. The ACNW looks forward to workingwith the staff to address these concerns throughother avenues.

1. Subject-Matter Experts

The Committee believes that the nominationprocess for selecting subject-matter expertsshould include organizations such as theNational Academies of Sciences (NAS) andEngineering, private industry, State develop-ment and regulating bodies, and representa-tive public interest groups.

We also believe that the process of formulat--ing the problem to be solved, the issues to beaddressed, and the detailed questions to beanswered should take place, primarily, beforeand during the process of selecting experts.The Committee's suggested approach is thatbefore deciding on the final panel of experts,a much larger number of experts be con-tacted and their input be elicited on refiningthe general problem that has been formulatedby the generalists and the normative experts.In this way, a much larger knowledge base isavailable to fine tune the issues, and theopportunity exists for a very effective groupof experts to evolve that will eventually makeup the panel. Further refinement of the issues

F-1 F-i ~ ~ ~ ~ ~ .. NUREG-1563

and questions should be performed by theselected panel of subject-matter experts.

Response

Regarding the first portion of the ACNW'scomment, the staff's intent behind Step No. 2("Selection of Experts") of Technical Position No.2 ('A Consistent and Systematic Process forElicitation Should be Applied") of the BTP was toencourage the enlistment of qualified individualsrepresenting or affiliated with the organizations/entities identified by the ACNW in its comment.To make the staff's intent clearer, the first twosentences of the fifth paragraph of the "Discuss-ion" section (Section 4) behind Technical PositionNo. 2/Step No. 2 have been modified as follows:

In selecting each of the three types of experts,especially the subject-matter experts, it maybe useful to seek qualified nominations fromoutside sources, or recognized peers in thefield. This would include, for example: theNational Academies of Sciences andEngineering; academia; recognizedprofessional societies (e.g., Sigma Xi, TheGeological Society of America, the AmericanSociety of Civil Engineers); Nationallaboratories; knowledgeable Federal Agenciesand International Organizations; privateindustry; State development and regulatingbodies; representative public interest groups;and interested stakeholders.

Apropos the second portion of the ACNW'scomment, regarding the timing of final subject-matter expert selection and problem formulation(Step No. 3-"Identification of Issues andProblem Decomposition" of Technical PositionNo. 2), the staff believes that the BTP iscompatible with the ACNW's views on this issue.Although the BTP may not state so in the sameway the ACNW has, the staff believes that theBTP recognizes that it may be appropriate to"iterate" on problem formulation (Step No. 3)before subject-matter expert selection (Step No. 1)is finalized, as noted in the third introductoryparagraph (in Section 4, "Discussion") toTechnical Position No. 2:

In addition, although the process steps arelisted in numerical order, it is not necessarythat the individual steps be performed in the

exact sequence presented. In fact, it isexpected that several of these process stepswill proceed or can be initiated concurrently,subject to repeated iterations and oppor-tunities for feedback from the subject-matterexperts. This may be especially true for StepNos. 2, 3, and 4, which are depicted asparallel process steps with feed-backs, asshown in the preceding figure. What thisfigure shows, for example, is that once thesubject-matter experts are identified (in StepNo. 2), they can help to better define theobjective of the elicitation (Step No. 1) andthus aid in the decomposition of theelicitation issue into its constituent parts(Step No. 3). Moreover, the subject-matterexperts can also aid in the identification ofadditional information that could facilitatethe elicitation (Step No. 4).

However, the staff's intent here may not be clearto all readers. Accordingly, the staff has addedthe following footnote to the paragraph inquestion:

Alternatively, before deciding on the finalpanel of subject-matter experts (Step No. 1),in contacting candidate experts, it might beuseful to solicit their input on refining thegeneral problem (Step No. 3) that waspreviously formulated by the generalists andthe normative experts. In this way, a muchlarger knowledge base is available tofine-tune the issues that will ultimately beaddressed by the final group of subject-matter experts. Further, final refinement ofthe sub-issues (and questions-Step No. 3)should be performed ultimately by theselected panel of subject-matter experts.

Moreover, the staff has revised the title of StepNo. 3-"Identification of Issues and ProblemDecomposition" of Technical Position No. 2 toread as "Refinement of Issues and ProblemDecomposition" to better reflect the staffs intentin this particular process step.

2. Aggregation of Results

The Committee believes that the results fromexpert elicitation should clearly display theuncertainties in the chosen performancemeasures for a particular issue. Therefore,the aggregation of the results of the expertpanel should also be clear in terms of the

NUREG-1563- F-L

uncertainties in the individual judgments ofthe panel members and the method ofaggregation and integration of bottom-lineresults that include the quantification ofuncertainties. This'property of the elicitationprocess becomes' especially important to theregulators in the consideration of multipleelicitations covering similar or identicalissues. The scientific process considers a fullrange of alternatives on the basis of thetechnical knowledge base of each and theassociated reasoning processes, all of whichshould be exposed in the decision-makingprocess. This documentation will facilitate theregulator's ability to discriminate betweendifferent alternatives on the basis of theevidence presented.

In this regard, major guidance Would comefrom an illustration of the aggregationprocess that embraces the notion ofcombining and integrating probabilitydistributions. The idea would not be toprescribe a process but rather to illustrate ingraphical and analytical terms'an example ofwhat is meant by the aggregation process.'Itis believed that such an aid Would greatlyfacilitate and add meaning to-the use ofprobability methods in the licensing processin general, and in expert elicitation inparticular.

Responsee

The staff agrees with this comment and notes thatthe BTP is not inconsistent with the ACNW'sviews. The BTP emphasizes the importance of anindividual expert's opinion. In this regard, theBT?, as part of the "Documentation" process step(Step No. 9 of Technical Position No. 2), notes-that the documentation associated with any.'formal elicitation'should be adequate insofar as itaccurately reflects the subject-matter expert'sopinion but also captures the rationale for thatopinion. This would include the prelirinary'andconstituent opinions thatare used'to form a finalopinion. If, during the feedback process (Step No.7), a subject-matter expert modifies his' or heropinion, then the staff believes that both theoriginal (Step No. 6) and modified opinions (StepNo. 6), along with the rationale for any change,should be included as part of the elicitationrecord (Step No. 9).

Accordingly, given the underlying "transparency"theme of the BT, the staff expects that thedocumentation of the aggregation method used(Step No. 8), whatever that aggregation methodmight be, must adequately record, among otherthings, the individual subject-matter expert'sopinions, and their attendant effect on results ofthe elicitation itself, so that they are alsotraceable, within 'the aggregate. As noted in'theBTP, the staff expressed no opinion regardingwhat type of aggregation should be used in a'formal elicitation because this issue is consideredbeyond the scope of this guidance. Furthermore,providing an example on this aspect of an expertelicitation is inconsistent with the level of detailsought in this'particular guidance document.However, the Center for Nucleair WasteRegulatory Analyses' climate elicitation(DeWispelare el al., 1993) does provide agraphical example illustrating one type ofmechanical aggregation. NUREG-1489 (PRAWorking Group, 1994; pp. C-139-C-142) alsoprovides some detailed discussion regarding themathematical aggregation of expert's judgments.

3. Interpretation of the Results

-The Committee wishes to emphasize that as aresult of the flexibility of the process, theapplicant should not conclude that followingthe guidance implies automatic acceptance ofthe results. The results, and the detailedbases thereof, are the desired outcome of theelicitation process. The credibility of the.results has to be principally based on theindividual's reasoning process, the method ofaggregation, and the supporting knowledgebase, including the use of specific datawherever possible.

Response

'The staff agrees with this comment and notes thatthe BTP, in particular, the sixth and seventhparagraphs in Purpose of the BTP" (Section 1.4),

* address this issue.

4. a Application of Expert Elicitation

Although the Committee was pleased that theBTP was not overly prescriptive on thematter of how to conduct expert elicitations,there is a need for additional guidance oncandidate issues for application. A discussion

F-3 NUREG-1563

- -

of appropriate applications would illustratethe limitations and the general intent of theprocess.

Response

As noted in the staff's response to DOE's secondoverall comment (in Appendix D), on the need foradditional guidance in the area of performanceassessment, the staff will take this comment intoaccount in due course but has made no decisionas yet on this option. In view of the possibilitythat new performance measures may beestablished for a potential repository at YuccaMountain, pursuant to the Energy Policy Act of1992, and consistent with the findings andrecommendations of the recent NAS report (seeNational Research Council, 1995), the staff wishesto provide for the possibility that site-specificstandards may necessitate the development ofadditional implementing guidance. The staff,therefore, continues to leave open this option, asnoted in Section 14, paragraph 3 of the BTP (e.g.,that it ". . . is considering the potential need forfuture guidance to identify those specific aspectsof a performance assessment for which theapplication of expert judgment may or may not beappropriate... ."). Currently, the staff has noplans for, or resources devoted to, preparing anyadditional guidance in this area. If the staffdecides that additional guidance should beconsidered, it will explain its reasons and offer theopportunity for comment by the ACNW (andothers).

Observation

Additionally, the Committee believes that theCommission, consistent with its PolicyStatement on probabilistic risk assessment(see NRC, 1995; 60 FR 42623), may wish toexamine the decision-making process to takegreater advantage of results developedthrough state-of-the-art expert elicitations.For example, there may be an impact on theadmissibility for testimony of a validelicitation resulting from the unavailability ofone or more subject matter experts. Although

there are legal arguments for the need for a''sponsoring witness," such an individual maynot be able to represent, as his or her own,the full range of the technical argumentscontained in the original elicitation.

Response

This comment concerns Commission policy andadjudicatory issues that are beyond the scope ofthe BTP.

References

DeWispelare, A.R., et al., "Expert Elicitation ofFuture Climate in the Yucca Mountain Vicinity-Iterative Performance Assessment Phase 2.5," SanAntonio, Texas, Center for Nuclear WasteRegulatory Analyses, CNWRA 93-016, August1993. [Prepared for the U.S. Nuclear RegulatoryCommission.]

Geomatrix Consultants/TRW EnvironmentalSafety Systems, Inc., "Probabilistic VolcanicHazard Analysis for Yucca Mountain, Nevada,"San Francisco, California, Document No.BAOOOOOOO-1717-2200-00082 (Rev. 0), June 1996.[Prepared for the U.S. Department ofEnergy/Office of Civilian Radioactive WasteManagement.]

National Research Council, "Technical Bases forYucca Mountain Standards," Washington, D.C.,National Academy Press, Commission onGeosciences, Environment, and Resources, July1995.

PRA Working Group, "A Review of NRC StaffUses of Probabilistic Risk Assessment," U.S.Nuclear Regulatory Commission, NUREG-1489,March 1994.

U.S. Nuclear Regulatory Commission, "Use ofProbabilistic Risk Assessment Methods inNuclear Regulatory Activities [Final PolicyStatement]," Federal Register, Vol. 60, No. 158,August 16, 1995, pp. 42622 - 42630.

NUREG-1563 F-4

NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER(2-89) (Assigned by NRC, Add Vol.,NRCM 1102, Supp., Rev., and Addendum Num-3201, 3202 BIBLIOGRAPHIC DATA SHEET bars If any.)

(See nstructions on the reverse)

2. TITLE AND SUBTITLENUREG-1563

Branch Tchnical Position on the Use of Expert Elicitation in the 3. DATE REPORT PUBLISHED

High-Level Radioactive Waste Program MONTH I YEAR

November 19964. FIN OR GRANT NUMBER

5. AUTHOR(S) 6. TYPE OF REPORT

J. P. Kotra*, M. P. Lee*, N. A. Eisenberg*, and A. R. DeWispelare** Technical7. PERIOD COVERED (Inclusive Dates)

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (it NRC, provide Division, Office or Region, U.S. Nuclear Regulatory Commission, andmaling address; I contractor, provide name and maling address.)

Division of Waste Management **Center for Nuclear Waste Regulatory AnalysesOffice of Nuclear Material Safety and Safeguards 6220 Culebra RoadWashington, DC 20555-0001 San Antonio, TX 78228-0510

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (it NRC, type Sarne as above'; If contractor, provide NRC Division, Office or Region,U.S. Nuclear Regulatory Commission, and mailIng address.)

Same as Item 8, above

10. SUPPLEVENTARY NOTES

11. ABSTRACT (200 words or less)

The U.S. Nuclear Regulatory Commission staff expects that subjective judgments of individual experts and, in somecases, groups of experts, will be used by the U.S. Department of Energy (DOE) to interpret data obtained during sitecharacterization and to address the many technical issues and inherent uncertainties associated with predicting the per-formance of a repository system for thousands of years. NRC has traditionally accepted, for review, expert judgment toevaluate and interpret the factual bases of license applications and is expected to give appropriate consideration to thejudgments of DOE's experts regarding the geologic repository. Such consideration, however, envisions DOE using ex-pert judgments to complement and supplement other sources of scientific and technical information, such as data col-lection, analyses, and experimentation. In this document, the NRC staff has set forth technical positions that: (1) pro-vide general guidelines on those circumstances that may warrant the use of a formal process for obtaining the judg-ments of more than one expert; and (2) describe acceptable procedures for conducting expert elicitation when formallyelicited judgments are used to support a demonstration of compliance with NRC's geologic disposal regulation, cur-rently set forth in 10 CFR Part 60.

12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers In locating the report.) 13. AVAILABILITY STATEMENT

Unlimited14. SECURITY CLASSIFICATION

(This Page)

Expert elicitation Peer review UnclassifiedExpert judgment Performance assessment (This Report)

Geologic repository Probabilistic risk assessment UnclassifiedHigh-level waste Yucca Mountain 1s. NUMBER OF PAGES

16. PRICE

NRC FORM 335 (2-89)

Federal Recycling Program