Validation of RISMC Models

Nam Dinh, Linyu Lin, Paridhi Athe

Multiphysics Model Validation WorkshopRaleigh, NC -- June 27-29, 2017

7/24/2017 1

Acknowledgement

The Integrated Research Project (IRP)

“Development and Application of a Data-Driven Methodology for Validation of Risk-Informed Safety Margin Characterization Models”

is supported by the U.S. Department of Energy’s Nuclear Energy University Program and managed by the Light Water Reactor Sustainability Program’s RISMC R&D Pathway.

7/24/2017 2

IRP Team

• Steve Prescott (INL)• Cristian Rabiti (INL)• Robert Youngblood (INL)• Weiju Ren (ORNL)• Ram Sampath (CPL)• Jeffrey Lane (Zachry)• Robert Sewell (RSA)• Anh Bui (NCSU)

• Maria Avramova (NCSU)• John Baugh (NCSU)• Igor Bolotnov (NCSU)• Nam Dinh (NCSU)• Abhinav Gupta (NCSU)• Hany Abdel-Khalik (Purdue)• Philippe Bardet (GWU)• Xiaodong Sun (U-Mich)

* Curtis Smith (INL, LWRS)

Contents

• Background – CSAU/EMDAP– PCMM– RISMC– RISMC models

• IRP Approach – Technical components– PCMQ

T-H model validation: Model Characteristics

• Designed to predict a broad range of nuclear (reactor coolant) system T-H behaviors– Various subsystems, components– Steady state regime, transients and accident scenarios

• Multi-fluid (multi-field) averaged conservation treatment (equations)– (Physics-based) Reduced Order Modeling– Use “closure relations” (or constitutive laws) to substitute for

information/ interactions lost in “averaging” • Large number of correlations,

– Each describes a particular physics/phenomenon/effect/ interaction– Each obtained under certain experimental conditions

• Low-dimensionality formulation (1D; 1.5D)– RELAP5, TRACE, CTF

• Exception GOTHIC (0D, 1D, 2D, 3D for CTH) and future STH code

System T-H model validation

• Legacy code development and assessment over decades– 1960-1980: RELAP series; TRAC– Since mid 90s: RELAP5.3D; TRACE– Large-scale test programs (e.g., LOFT, Semiscale, APEX)

• US NRC CSAU Code Scaling, Applicability and Uncertainty– PIRT Phenomena Identification and Ranking Table

• US NRC Reg.Guide 1.203 “Transient and Accident Analysis”– EMDAP: Evaluation Model Development and Assessment Process

• PIRT guided Assessment base (SET, IET)• Sensitivity and uncertainty analysis

• BEPU: Best Estimate Plus Uncertainty

System T-H model validation: Existing Approaches

• US NRC CSAU and EMDAP– High level, methodological (heuristic)– Acceptance criteria not defined– Limited implementation in practice– Little attention on code verification and solution verification

• Focused on validation– To a large extent, MET and IET calibration and benchmarks– Sequential calibration– “Viewgraph norm” comparison

• “User effects”– Nodalization effect

• “Discretization” error? “Averaging” error?

• Uncertainty Quantification– Lack of UQ-grade data – Deficient techniques on Data assimilation/ UQ for T-H complex models

Code Scaling, Applicability, and Uncertainty (CSAU)

7/24/2017CSAU, “Code Scaling, Applicability, and Uncertainty”, US NRC, NUREG/CR-5249 (Boyack et al.), 1989. US

NRC Regulatory Guide 1.157 (“Best-Estimate Calculations of ECCS Performance”, 1989). 8

• Specify scenario, System conditionIdentify and rank phenomena (PIRT)

• Select frozen code, provide necessary documentation

Element 1: Requirement and Code Capability

• Establish assessment matrix for validation (set of integral and separate effect test)• Define Nodalization• Determine accuracy, scale effects, bias & uncertainty

Element 2: Assessment and ranging of parameters

• Determine impact of input parameter uncertaintyCombine bias and uncertainty + additional margin

Element 3: Sensitivity and uncertainty analysis

Systematic process for Complexity resolution and

uncertainty quantification

Westinghouse Non-Proprietary Class 3 © 2014 Westinghouse Electric Company LLC. All Rights Reserved.

Evaluation Model Development and Assessment Process- EMDAP

Six Basic Principles (Elements) ofEvaluation Model Development andAssessment Process (EMDAP) arepresented in Transient and Accident Analysis Methods (USNRC, Regulatory Guide 1.203, 2005).

Scaling Analysis is a part of EMDAP and Transient and Accident Analysis Methods.

9

Regulatory Guide 1.203, “Transient and AccidentAnalysis Methods,” USNRC,December 2005

Evaluation Model Development and Assessment Process (EMDAP)

7/24/2017EMDAP, " Regulatory Guide 1.203: Transient and Accident Analysis Methods," US Nuclear Regulatory

Commission, Office of Nuclear Regulatory Research, 2005. 10

• Specify scenario, System condition, figure of merit• Identify, system, components, geometries, fields and process to be modeled

Identify and rank phenomena and process (PIRT)

Element 1: Establish Requirement for Evaluation Model Capability

• Identify Separate Effect Tests (SETs) and Integral Effect Tests (IET) • Evaluate IET distortions and SET scale up capability• Determine experiment uncertainty

Element 2: Develop Assessment Base

• Establish EM structure• Develop and incorporate closure models

Element 3: Develop Evaluation model (EM)

• Determine model pedigree and applicability • Assess model fidelity and accuracy• Assess scalability of models and integral calculations• Determine EM biases and uncertainty

Element 4: Assess Evaluation Model Adequacy

Emphasis on Hierarchical

system analysisAnd

Scaling analysis

• Probabilistic Risk Analysis– For each component, the failure criterion is firstly characterized by comparing the load and capacity,

where the load is calculated deterministically by system code. The safety margin is then defined by the ratio (or the difference) of an applied capacity to the load. i.e. Load/Capacity (or Load-Capacity)

• Risk Analysis for External Hazards– Hazard analysis– Plant-system and structure response analysis– Evaluation of fragility and vulnerability of components– Plant system and sequence analysis– Consequence analysis

11

Loss of Preferred Power Event Tree

Fragility curvesHazard intensity

curve

• Risk-Informed Safety Margins Characterization– Risk-Informed Safety Margins Characterization

(RISMC) Pathway is to “conduct research and development for advanced methods and tools to support NPP safety assessments and management”

– Load and capacity distributions are determined by advanced simulation tools

• RISMC Analysis for External Hazards of Flooding

12

Load Capacity

INL, 2016

Science and Practical Challenges

Risk = ∑ Probability * Consequence = ∑ P * C i

[Small] [Large]

i

Uncertainty

Sequences

Uncertainty in Characterization of Rare Events, High Hazards

Experience, Culture

Knowledge, Capability (R&D)

Decision Making, Regulation

$$$ (ALARA)

Risk, Safety and Licensing: From Perception to Practice and Public Acceptance

What ArePossible UpsetConditions?

How Do They Occur?

How Often DoThey Occur?

What AreConsequences?

Imagination, Precaution,

Operating Experience

Comprehensive Safety Analysis vs. Surrogate Licensing Criteria

PRA: Probabilistic Risk Assessment

DSA: Deterministic Safety Analysis (Mechanistic)

Experiments, Simulations

How plant aging affects these four questions?

How confident are we in the answers to these questions (UQ)?

16

Safety Figure of Merit

a number

a number

PDF

LoadCapacity

“A decision made without taking uncertainty into account is barely worth calling a decision”,

R.E. Wilson (1985)

CapacityLoad

?

AgingPower uprate, Increasing Fuel burnups

Higher frequencies of IEsNew IEs and sequencesCascading failures

Potential Developments

Risk-Informed Safety Margin Characterization (RISMC)

Safety Figure of Merit

The REMDAP of SPH (Flood Simulation) and STH in flooding-induced extended SBO RISMC applications

System T-H model validation: RISMC Requirements

• RISMC requires a large number of simulations– Computationally efficient– (Yet) Sufficiently accurate

• Realistic scenarios– Sufficiently accurate representation of safety-significant SSCs– Realistic boundary conditions for sub-domain analysis

• Realistic simulations– Bounded numerical errors– Bounded model parameter errors– No significant model form biases

• Predictive capability maturity evaluated to be Level 1, preferred Level 2– See PCMM definition of levels

• Uncertainty quantification– Uncertainty sources identified, evaluated, and propagated to FOM/ QOI

System T-H model validation: Data-Driven

• Data requirements– Use high-resolution validation-grade data

• From relevant computational and physical experiments– Covering key phenomena identified in PIRT

• Sufficient coverage of parameter range to ensure prototypic flow patterns

• Data status– Limited amount of data, primarily from METs and

IETs• Scaling (applicability) is subjective

– Potential use of high-resolution CFD/ CMFD • Including DNS, LES, and RANS

RISMC Models

V&V-UQ Methods & Tools

RISMC Applications Validation DataIntegrated

Research

Key attributes of the IRP approach

• Risk-informed approaches for collection and characterization of evidence,

• Decision-oriented framework for quantification of “validity” (or “maturity”), with a probabilistic criterion for adequate level of validation.

• Techniques for physics-based simulation-aided scaling, uncertainty and sensitivity analysis (to compute and optimize the relevance and value of experiments),

• Methods for data-driven model development and assessment process that effectively utilizes data from physical and numerical experiments;

Key Ideas on Validation

• Validation of RISMC Tool = – RISMC Tool is to support a practical decision

• Decision is always about managing a risk– Risk is rooted in uncertainty

– Decision-Oriented Risk-Informed, S/UA-aided Validation• (S/UA = sensitivity/uncertainty analysis)

• Validation = Predictive Capability Maturity– Maturity for a Decision

• Reliability / Robustness of Decision

• Validation– Accuracy assessment– Adequacy assessment

Key Ideas on Validation Data

• “Data Realism”– Data Quality/Relevancy/ Applicability

• Always deficient;• New data (massive)• Multiscale heterogeneous data

• Data generation– Advanced diagnostics experiments– High-fidelity numerical simulation

• Data-driven – Data assimilation

William L. Oberkampf, Martin Pilch, and Timothy G. Trucano, SAND2007-5948

Predictive Capability Maturity Model for Computational Modeling and Simulation

Stringency increases with potential impact of decision (consequences) to be made with

using findings from simulation

Risk-informed Validation

Decision Model

Elements of the Framework forPredictive Capability Maturity Quantification

(PCMQ)

7/24/2017 27

Model CoverageCMP = PM/PP

PCMQ: How good is our validation?

PP

PIRT-basedPhenomena

Pyramid

PM

Code System Model

Pyramid

PE

ValidationExperiment

Pyramid

PCMQ = CMP *CME*CEP

7/24/2017

Elements of the framework

Preprocessing for the framework development Specification of issue, simulation

tool, decision objective Specification of scenario, FOM Phenomena resolution PIRT

Structural Knowledge representation Hierarchical pyramids and Goal

Structuring notation (GSN)

Classification and Characterization of Evidence Direct evidence Indirect evidence (Process quality

assurance) Data characterization

• Scaling analysis

Formulation of the decision model Structural decision model Maturity quantification

• Bayesian Network• Fuzzy logic & fuzzy

inference system• Evidential reasoning

Phenomena knowledge base, model base and data base for validation

GSN modules

II. STRUCTURAL KNOWLEDGE REPRESENTATION

VII. REFINEMENT

GSN Modules

III. CLASSIFICATION AND CHARACTERIZATION OF EVIDENCES

V. EVALUATION

VI. APPRAISAL

ACT

IV. DECISION MODEL

I. PREPROCESSING REQUIREMENT FOR THE FRAMEWORK DEVELOPMENT

Computable Network

Structural decision model

GSN module

Structural Knowledge Representation- Implementation Requirements

Clarity and Traceabilityo Detailed structured information representationo Multi-level information abstraction

Indicators for knowledge, importance, status of work Ability to manage and organize large chunks of information

Tool/technique employed Goal Structuring Notation (Adelard’s Assurance and safety case environment, ASCE 4.2)o Goal Structuring Notation (GSN) is an argument modeling technique that was

primarily developed to represent assurance arguments for safety caseso “How does argumentation helps in structuring knowledge?”

7/24/2017 30

Overview of Goal structure Notation

7/24/2017 31

Maturity Quantification

Step 1: Formulate the decision model using GSNo Provide criteria, sub-criteria, evidence incorporation, and dependency relations

Step 2: Transforming the GSN based decision model into a computable network

Step 2: Quantitative Confidence assessmento Decision model consists of two sets of information / data: (i) Subjective data

based on expert opinion, and (ii) objective data based on evidenceo Assimilate these two sets of information to provide quantitative maturity

assessment

7/24/2017 32

Case study : Maturity Quantification ( Fuzzy Logic & Fuzzy Inference System)

7/24/2017 33

Assumptions Measurement uncertainty is negligibleValidation results(VR) are evaluated on the basis of “Bias”VR=1-BiasData applicability (DA) is determined by scaling analysisDA=1-SD, SD scale distortion

Maturity quantification for this example would imply a quantitative evaluation of the claim G1, i.e. “Code x is suitable for predicting the application of interest.”

34

Fuzzy Logic

If (DA is MF1) and (VR is MF2) then (CA is MF3)

Deg

ree

of m

embe

rshi

pD

egre

e of

mem

bers

hip

Deg

ree

of m

embe

rshi

p

Validation Results (VR)

Code adequacy (CA)

Data Applicability (DA)

MF1

MF3

MF2

CA

DA VR

CA

VR DA

Indirect evidence (Process quality assurance factors)

PQA of Phenomenology pyramid based on Process factors:o Past Use What is confidence in the past use of the techniques used for phenomena resolution (i.e.

PIRT process : Knowledge and importance grading)?o Peer review What is the confidence that adequate peer review of the PIRT has been carried out? o Definition What is the confidence that phenomenon have been clearly defined/described in the context

of the Application of interest?o Standard Procedure What is the confidence that the standard procedure as specified in the EMDAP

process has been followed?

All factors can be evaluated based on the confidence expressed as:No opinion / Low / Medium / High

7/24/2017 35

Process

• Past Use• Peer review• Definitions• Standard procedure

Personnel

• Past Knowledge• Domain Expirience• Independence• Competency

Tool/Techniques

• Bound qualification• Standard qualification

Case Study: Representation of Phenomenology Pyramid (PIRT) using GSN

Process Quality Assurance of Phenomenology pyramid Development of structured Phenomenology Pyramid Example: CASL Challenge Problem CRUD Induced Power Shift (CIPS)

7/24/2017 36

Phenomenology pyramid for CIPS (showing only Top goal and sub-goals)

7/24/2017 37

All nodes corresponding to each Sub tree (G 1.1, G1.2, G 1.3 and G 1.4 ) has beencollapsed for clarity and visibility.“[+]” sign indicated that the node has further expansion.ASCE 4.2 facilitates expansion/collapse of any part of the network in the GSN tree.

Coupling is not considered in this decomposition but can be incorporated as a separate goal at a later stage (see assumption A1a)

Multi-level information abstraction

7/24/2017 38

Multi-level information abstraction

7/24/2017 39

Solution node contains evidences in the form ofexcerpt or link from research paper, CASLreports or other documentation, e.g., Sn1.1.2.2 contains the evidence that explains howCRUD removal due to mechanical contractionin Clad can impact the FOM 3 (Axial offset).In this way, supporting evidence for a completebranch ( G 1, G 1.1, G 1.1.2, G 1.1.2.2) canbe incorporated in the GSN tree

Furtherdecomposed in

Decomposition of Goal G 1.1, Governing phenomena in Fuel Performance

Concluding Remarks (for PCMM)

PCMQ supports implementation of PCMM by providing a structured approach for knowledge representation, information abstraction, integration and evidence incorporation

It provide support for structural organization and classification of evidence for maturity assessment

Subjective information based on expert knowledge can be incorporated in the quantification process using Bayesian inference / fuzzy logic.

PCMQ can provide a formal approach for code review that may help in expediting the code licensing process

7/24/2017 40