(NASA J.harben P.reese) Implementing Strategies for Risk Mitigation (NCSLI-2011)

8/12/2019 (NASA J.harben P.reese) Implementing Strategies for Risk Mitigation (NCSLI-2011)

1/25

2011 NCSL International Workshop and Symposium

Implementing Strategies for Risk Mitigation

In the Modern Calibration Laboratory

Speaker: Jonathan Harben

The Bionetics Corporation

NASAKennedy Space Center1046 South Patrick Drive; M/S: ISC-6175

Patrick AFB, FL 32925Phone: (321) 494-7907 Fax: (321) 494-5253

E-mail:[email protected]

Authors: Jonathan Harben, Paul Reese

Abstract

Many strategies for risk mitigation have been employed in calibration laboratories. A modern

look at these concepts is presented in terms of compliance to ANSI/NCSL and ISO standards.

Specifically, the practical application of various techniques to manage false accept risk in a highproduction calibration lab is reviewed. Understanding the factors affecting the probability of

incorrect Pass/Fail decisions reveals aspects which can be exploited and leveraged, reducing theeffort required for compliance. Test Uncertainty Ratio (TUR) and End Of Period Reliability

(EOPR) are directly related to risk, but one or both of these parameters may be impractical or

impossible to quantify and manage in daily practice for every measurement process. TUR and

EOPR are explained in terms of Z540.3 and these parameters are combined to form basicstrategies for compliance. These concepts are investigated as well as their mathematical

boundary conditions to gain a comprehensive understanding of practical, efficient risk

mitigation. Bench level techniques alone may be impracticable and/or insufficient to assess thequality of a calibration program en masse, as it relates to measurement decision risk. Program

level techniques can estimate or limit the false accept risk for future decisions, prior to obtainingany specific measurement result. Working practices and principals are presented that allow a

modern calibration laboratory to meet the demand of customers and manage risk formultifunction instrumentation while maintaining compliance to national and international

standards.

1. Background

In the simplest terms, Measurement & Test Equipment (M&TE) owners submit an instrument to

the calibration laboratory and want to know, Is my equipment good or bad? During a

compliance test, M&TE is evaluated using laboratory standards to determine if it is performing

as expected. This performance is compared within some pre-established specifications or

tolerance limits requested by the end-user or customer. Often, these specifications are themanufacturers published accuracy

1[1] specifications. As customers, they are asking for an In-

tolerance or Out-of-tolerance decision to be made. On the surface, this might appear to be a

relatively straightforward request. But exactly what level of assurance is the customer receiving

1The term accuracyis used throughout this paper to facilitate the classical concept of uncertainty for a broad

audience. It is acknowledged that the VIM [1] defines accuracy as qualitative term, not quantitative, and that

numerical values should not be associated with it
mailto:[email protected]:[email protected]:[email protected]:[email protected]


2/25


when statements of compliance are issued? How good are those statements of compliance? Is

simply reporting measurement uncertainty enough? What is the risk that a statement of

compliance is wrong? While alluded to in many international standards documents, this conceptis directly addressed in ANSI/NCSL Z540.3-2006 [2].

Since its publication, Z540.3 sub-clause 5.3b has, understandably, received a disproportionateamount of attention compared with other sections in the standard [3, 4, 5]. This sectionrepresents a significant change compared to its predecessor, Z540-1 [6]. Section 5.3b has come

to be known by many simply as The 2 % Rule and addresses calibrations involving

compliance tests, also called conformance tests, tolerance tests, or verification tests. It states:

Where calibrations provide for verification that measurement quantities are within specified

tolerances, the probability that incorrect acceptance decisions (false accept) will result from

calibration tests shall not exceed 2% and shall be documented. Where it is not practicable toestimate this probability, the test uncertainty ratio shall be equal to or greater than 4:1.

Many different inferences may be derived from these two seemingly innocuous statementsabove. Familiarity with ISO-17025 [7] shows that, by comparison with Z540.3, subject matter

relating to compliance testing is relatively sparse as that standard is primarily focused on

reporting uncertainties with measurement results, similar to Z540.3 section 5.3a. Perhaps the

most significant reference to compliance testing in ISO-17025 is found in section 5.10.4.2Calibration Certificates which requires that, When statements of compliance are made, the

uncertainty of measurement shall be taken into account. However, practically no guidance on

possible methods which could be implemented to take the measurement uncertainty into accountis provided. The American Association of Laboratory Accreditation (A2LA) further clarifies the

requirements associated with this concept in R205 [8]:

when parameters are certified to be within specified tolerance, the associated uncertainty of themeasurement result is properly taken into account with respect to the tolerance by a documented

procedure or policy established and implemented by the laboratory that defines the decision

rules used by the laboratory for declaring in or out of tolerance conditions2.

Simply reporting the uncertainty along with a measurement result may not sufficiently satisfy

customer requirements where compliance tests are desired. Without a quantifiable control limitsuch as false accept risk, this type of reporting may impart unknown risks to the customer.

Z540.3 addresses this concept directly.

2. Taking the Uncertainty Into Account

What does it mean to take the uncertainty into accountand why it is necessary? For an intuitiveinterpretation, refer to Figure 1. During a bench level compliance test, what are the decision rulesif uncertainty is taken into account? How might an item be declared to be In-tolerance when, in

reality, it is actually Out-of-tolerance? During the calibration, the UUT (Unit Under Test) might

2The default decision rule is found in ILAC-G8:1996 [9], Guidelines on Assessment and Reporting of Compliance

with Specification, section 2.5. With agreement from the customer, other decision rules may be used as provided for

in this section of the Requirements


3/25


legitimately be observed to be In-tolerance. However, the observationcould be misleading; in

fact, it could be wrong.

Figure 1. Five Possible Bench Level Calibration Scenarios

In scenario #1, a reading on a laboratory standard voltmeter of 9.98 V can confidently lead to anIn-tolerance decision (Pass) for this 10 V UUT source with negligible risk. This is true due to

sufficiently small uncertainty in the measurement process and the proximity of the measured

value to the tolerance limit. Likewise, a non-compliance decision (Fail) resulting from scenario

#5 can also be made with high confidence, as the measured value of 9.83 V is clearly Out-of-tolerance with little ambiguity. However, in scenarios #2, #3, and #4, this is not the case; a

Pass/Fail decision involves significant risk of being incorrect.

It is understood that all measurements are only estimates of the truevalue of the measurand; this

true value is clouded somewhat by the uncertainty of the measurement process. In scenarios, #2,

#3, & #4, this uncertainty provides the possibility for the true value of the measurand to lie either

Inor Outof the UUT tolerance limits. The truth is not precisely known. Consider scenario #3,where the UUT was observed at 9.90 V, exactly at the lower allowable tolerance limit. Undersuch conditions, there is a 50 % probability that either an In-tolerance or Out-of-tolerance

decision will be incorrect, barring any other information3. In fact, even with the best available

standards with the lowest possible uncertainty, the probability of being incorrect will remain at50 % in scenario #3 regardless of how small the uncertainty is. It is easy to see that a Pass/Fail

compliance decision carries some risk of being incorrect. This concept of bench level risk is

addressed in several documents [9, 10, 11, 12].

3Bayesian analysis can result in false accept risk other than 50 % in such instances, where the a prioriin-tolerance

probability (EOPR) of the UUT is known in addition to the measurement result and uncertainty. Such analysis is

beyond the scope of this paper.

9.92 V

9.90 V

9.87 V

9.98 V

9.83 V

9.75 V

9.80 V

9.85 V

9.90 V

9.95 V

10.00 V

10.05 V

10.10 V

10.15 V

10.20 V

#0 #1 #2 #3 #4 #5

Some Possible Measurement Results of a 10 V Source (UUT)

Upper Tolerance Limit

Lower Tolerance Limit

Expanded Uncertainty of theMeasurement Process:0.05 V

UUT Set Voltage = 10.00 V


4/25


It may be surprising to find that simple analysis of the individual measurement results presented

in the aforementioned scenarios is not directly consistent with the intent of The 2 % rule in

Z540.3, although it still has some application. To this point, the discussion has dealt exclusivelywith bench level analysis of measurement decision risk. That is, risk was predicated only on

knowledge of the relationship between the UUT tolerance, the uncertainty of the measurement

process, and the observed measurement result made on-the-bench; after the measurementresult had been obtained. But computation of false accept risk (FAR, also known as theProbability of False Accept, PFA, or consumer risk), for strict compliance with the 2 % rule in

Z540.3, does not depend on any particular measurement result nor its proximity to a given UUT

tolerance limit. The 2 % rule in Z540.3 addresses the risk prior to any particular singlemeasurement result being obtained, at the program level. To foster an understating of both

bench level and program level false accept risk among interested parties from a multitude of

backgrounds, this paper provides a more conceptualperspective of the intent underlying the 2 %

rule and its relationship to TUR and EOPR4.

3. The Answer to Two Different Questions

False accept risk describes the overall probability of false acceptance when Pass/Fail decisionsare made. False accept risk can be interpreted and analyzed at two different levels; the bench

level and the program level [4]. Both risk levels are described in ASME Technical ReportB89.7.4.1-2005 [13]. This report refers to bench level risk mitigation as controlling the quality

of individualworkpieces, while program level risk strategies are described as controlling the

averagequality of workpieces. These two approaches result in two answers to two differentquestions, but they are related. In essence, it is not necessarily a matter of rightand wrong, but

rather an appropriate answer to an appropriate question to meet a desired quality objective. It is

ambiguity in the question itself that may lead to different assumptions regarding the meaning offalse accept risk. Many international documents speak only to the bench level interpretation of

risk, requiring an actual measurement result to be available [9, 10, 11, 12]. However, Z540.3

was intended to address risk at the program level [14].

When Z504.3 requires that, the probability that incorrect acceptance decisions (false accept)

will result from calibration tests shall not exceed 2%...,it may not be immediately evident which

view point is being addressed, the bench level or the program level. However, the strategies forimplementing such a requirement are quite different depending upon the level at which the

requirement applies. The implications of this were significant enough to prompt NASA to

request interpretive guidance from the NCSLI 174 Standards Writing Committee [15]. It was

affirmed that the 2 % false accept requirement applies to a population of like calibrationsessions orlike measurement processes[14]. As such, Z540.3 section 5.3b does not directly

address the probability of false accept to any single, discrete measurement result or individual

workpiece and supports the program level view of risk prior to, and independent of, any actualmeasurement result.

4The subject of measurement decision risk includes not only the probability of false-accept (PFA), but the

Probability of Correct Accept (PCA), Probability of False Reject (PFR) and the Probability of Correct Reject (PCR).

While false-rejects can have significant economic impact to the calibration lab, the discussion in this paper is

primarily limited to the direct requirements of Z540.3 (5.3b), i.e. false accept risk.


5/25


What exactly does this mean and how might risk be computed prior to any particular single

measurement result being obtained? In statistical terms, the 2 % rule refers to the unconditional

probability of false acceptance. Consider the outcomes of a collection or population of singlemeasurement results obtained over time. Each individual result does, by some small amount,

comprise the statistics which describe a population of like calibration sessions or like

measurement processes. Each measurement result (a Pass or Fail) contributes to the End OfPeriod Reliability (EOPR) for a particular test-point, e.g. the 10 V, 1 kHz, AC voltage test-pointon a population of Acme model 123 digital multimeters. This EOPR data can help predict the

probability of making a false accept decision for future tests. Reliability data for the M&TE

model and manufacturer levelcan be used to conservatively estimate the reliability of the M&TEtest point. This is addressed in complianceMethod 2of the Z540.3 Handbook [16].

In terms of Z540.3, false accept risk describes the overall or average probability of false

acceptance decisions to the calibration program at large, for all of the Pass/Fail decisions that aremade at a particular test point. It does not represent risk associated with any particular individual

measurement result of any unique instrument. The 2 % rule speaks to the following question:

Given a historical collection of Pass/Fail decisions at a particular test-point for a population oflike-instruments (i.e. EOPR is known), what is the probability that an incorrect acceptance

decision will result during an upcoming test? Notice that no information is provided on any

particular measurement result. The question is being asked before the scheduled measurement is

ever made. This can be answered as long as previous EOPR data on the UUT population isavailable and the measurement uncertainty (and thus TUR) is known. In certain circumstances,

it is also possible to comply with the 2 % rule by bounding or limiting false accept risk by using

either:

EOPR data withoutknowledge of the measurement uncertainty.

TUR withoutknowledge of EOPR data.

To understand how this is possible, a closer look at the relationship between false accept risk,

EOPR, and TUR is helpful.

4. End of Period Reliability (EOPR)

EOPR is the probability of a UUT test-point being In-tolerance at the end of its normal

calibration interval. It is sometimes known as In-tolerance probability and is derived from

previous calibration events, based directly on measured values. In its simplest form, EOPR canbe defined as

as-recieved

If it is known that a significant number of previous measurements for a population of UUTs werevery close to their tolerance limits as-received, would this affect the false accept risk for an

upcoming measurement? Yes; there would indeed be a higher false acceptance risk for the

planned (upcoming) measurement if it was known ahead of time that the measurement about tobe made was likely to lie near one of the UUTs tolerance limits. Consider Figure 2: Two

different model UUT voltage sources are scheduled for calibration. One is a model-A and the


6/25


other is a model-B. The five previous calibrations on model-As have shown these units to be

highly reliable; see Group-A. Most often, they are found to be well within their tolerance limits

and easily comply with their specifications. Model-As are gems in the parlance of the cal lab.In contrast, previous calibrations of model Bshave shown them to be dogs. Model-Bs hardly

ever meet their specifications; see Group-B. Of the last five calibrations, two model-Bswere

recorded as being Out-of-tolerance and one of them was barely-in. Making an Inor Outoftolerance decision will be a precarious judgment-call and there is a higher probability of makinga false accept decision compared to the model-A.

Figure 2. Previous Historical Measurement Data can Influence Future False Accept Risk.

In Figure 3, imagine the blue dot representing the measurement result is not shown on the chartyet. If it was known ahead of time that this dot (the upcoming measurement result) was likely to

be near the tolerance limit when the measurement finally does occur, a false accept would indeed

be more likely. An elevated false accept risk would also be present if it was known ahead oftime that the measurement was likely to lie slightly outsidethe UUTs tolerance limits.

Again, if there was a high probability that, based on previous measurement data, the

measurement result was likely to be in close proximity to the upper or lower tolerance limit, it

can easily be seen that the upcoming measurement would carry a larger false accept risk than ifthe measurement was likely to lie near the nominal value (10 V). The critically important point

is this: If the historical reliability data indicates that In-tolerance probability (EOPR) of the UUT

is poor (up to a point5), the false accept risk is elevated.

5Graphs of EOPR vs. false-accept risk can reveal a perceived decrease in false-accept risk as the EOPR drops below

certain levels. This is due to the large number of out-of-tolerance conditions that lie far outside (away from) the

UUT tolerance limits. This is discussed later in this paper.

9.75 V

9.80 V

9.85 V

9.90 V

9.95 V

10.00 V

10.05 V

10.10 V

10.15 V

10.20 V

10.25 V

#0 #1 #2 #3 #4 #5

Previous Measurement Results for Two Groups of Instruments

Upper Tolerance Limit


Expanded Uncertainty ofthe Measurement Process:0.05 V


Group - A Group - B


7/25


Figure 3. The Possibility of a False Accept for a Measurement Result

The previous scenarios assumed some familiarity with populations of similar instruments which

are periodically resubmitted for calibration. But how can EOPR be reconciled when viewedfrom a new calibration laboratorys perspective? What if no historical reliability or EOPR data

exists for UUTs? Can a new laboratory even open its doors for business on day-one and still

meet the 2 % false accept requirement of Z540.3 without EOPR data? The answer to thisquestion isyes.However, the new lab must employ bench level techniques or other appropriatetechniques (boundary condition methods, guardbanding, etc.). Such methods are described later

in this paper. This same logic would apply to a well-established calibration laboratory, in

business for years, which receives a new, totally unique instrument for calibration the first time.If no similar item exists, no historical data can be analyzed to determine EOPR and compute

false accept risk; therefore other appropriate techniques and/or bench level methods must be

employed.

If EOPR data or In-tolerance probability is important for compliance with the 2 % rule, how

gooddoes the estimate of EOPR data have to be before program level methods can be used to

address false accept risk for a population of instruments? Undoubtedly, this will be the subjectof many discussions and papers to follow. Sharing or exchanging EOPR data between differentlaboratories has even been proposed with varying opinions. Acceptance of this is generally

dependent upon the consistency of the calibration procedure used and the laboratory standards

employed. The rules used to establish EOPR data can be somewhat subjective (e.g. how manysamples are available, are first-time calibrations counted, are broken instruments included, are

late calibrations included, etc.). Other considerations include the validity of aggregating or

pooling like-instruments together by groupings and various classifications (e.g. model number).

9.92 V

9.85 V

9.87 V

9.89 V

9.91 V

9.93 V

9.95 V

9.97 V

9.99 V

10.01 V

10.03 V

10.05 V

#0 #1 #2 #3 #4 #5Measured (Observed) Voltage = 9.92 V

UUT Voltage Source: Calibration Scenario


A False Accept Occurs ifthe True Valueof the UUTis outside its tolerancelimit


Expanded Uncertainty ofthe Measurement Process


8/25


5. Test Uncertainty Ratio

It has been illustrated that EOPR can greatly affect the false accept risk of calibration processes.

However, the concept of Test Uncertainty Ratio (TUR) is likely to be more familiar to the

broader audience as a metric of quality for calibrations. From the preceding examples, it is

evident that a lower uncertainty generally reduces the likelihood of a false accept decision.

Historically, TUR was thought of by many as a comparison of the quality (accuracyspecification) of a UUT to the quality (uncertainty) of a laboratorys reference standards and

processes. If a low TUR was computed, it often meant that laboratory standards were notgoodenoughto confidently make acceptance decisions during compliance tests, or that the UUT was

toogood (accuracy spec was too tight). TUR has been viewed as the uncertainty or tolerance of

the UUT in the numerator divided by the uncertainties of the laboratorys measurementstandard(s) in the denominator [17]. A high ratio such as >4 to 1 was thought to be a quality

metric indicative of a robust calibration process.

The roots of TUR were born out of the Navys Production Quality Division during the 1950s inan attempt at limiting incorrect acceptance decisions. The origins of the ubiquitous 4 to 1 TUR

have been summarized by Dr. Howard Castrup of Integrated Sciences Group [18]. Early work inmeasurement decision risk by the Navys Jerry Hayes and Stan Crandon concluded:

if the measuring device had three times the accuracy of the UUT, the consumer's risk was

around 1.06 %. This assumes that the only error in the measurement process is the bias in the

measuring device. It also assumes a 95% in-tolerance probability for both the measuring deviceand the UUT, i.e., that the specs for both devices are roughly 2-sigma specs They decided to

pad the 3:1 ratio a bit to accommodate additional measurement process errors and to cover

cases where the in-tolerance probabilities would be less than 95%. They felt that a 4:1 ratiowould be sufficient.

In those early days, the above assumptions were necessary to ease the burden of rigorouscomputational requirements of measurement risk analysis. Since that time, manufacturersaccuracy specifications have often been loosely inferred to represent 2 sigma or 95 %

confidence for many implementations of TUR, unless otherwise stated. This is synonymous

with assuming that all UUTs will meet their published specifications 95 % of the time (i.e.EOPR will be 95 %). Even if the calibration personnel did not realize they were making this

assumption when computing TUR, they were directly relying on such an assumption to gain any

utility out of the 4 to 1 TUR. For a simple TUR calculation to provide meaningful risk

mitigation, the assumptions stated above are critically important to the success of this simpleratio tool. As previously demonstrated, the In-tolerance probability or EOPR doeshave a direct

affect on the false accept risk for populations of tests. Is the EOPR for all M&TE really 95 %?

That is, are all specs provided by manufacturers really representative of two standarddeviations of the true product distribution? If not, the time-honored 4 to 1 TUR will not providethe expected level of protection for the consumer and false accept risk could be much higher than

expected. Calibration managers and quality personnel are well aware that allM&TE does not

have an EOPR of 95 %. It is often much lower.

Both the 4 to 1 TUR rule and the 2 % rule do not require information regarding individual

measurement results at the bench level nor how close these results might be to a particular


9/25


tolerance limit. The measurement result of any particular test is not used for the computation of

the false accept risk intended by Z540.3, nor is a measurement result necessary for maintaining

TUR above 4 to 1. Both areprogram levelrisk mitigation efforts.

While the spirit of Z540.3 is to move away from the reliance on TUR altogether, its use is still

permitted if adherence to the 2 % rule is deemed impracticable. Use of the TUR isdiscouraged due to the many assumptions it relies on for controlling risk. However, given thatthe computation of false accept risk requires the somewhat laborious collection of EOPR data,

the use of TUR might be perceived as an easy way for labs to circumvent the 2 % rule. The

catch is the definition, or more poignantly the re-definition, of TUR. Section 3.11 in Z540.3defines Test Uncertainty Ratio as:

The ratio of the span of the tolerance of a measurement quantity subject to calibration, to twice

the 95% expanded uncertainty of the measurement process used for calibration.

At first, this may not appear to be significantly different than other definitions currently in place

in many laboratories. One might notice the numerator, associated with the UUT, specifies thespan of the tolerance. If the tolerance specification of the UUT is two-sided and documented as

a plus-or-minus () tolerance, the entirespanof the tolerance must be included. However, this is

countered by the requirement to multiply the 95 % expanded uncertainty of the measurement

process in the denominator by a factor of 2. The confidence level associated with the UUTtolerance specifications in the numerator is undefined. This quandary is really nothing new.

Assumptions of the level of confidence associated with the UUT (numerator) have been made for

decades.

However, there is a very distinct difference between the TUR as defined in Z540.3 and some

widely accepted historical definitions. This difference centers on the components of the

denominator of the TUR. In Z540.3, the uncertainty in the denominator is very specificallydefined as the "uncertainty of the measurement processused in calibration." This definition has

broader implications than historical definitions by including elements of the UUT performance

(e.g., resolution, and process repeatability) in the denominator. These components of uncertaintymust be included in the 95 % expanded uncertainty of the measurement process, as used in the

TUR denominator. Many labs have long assumed that the uncertainty of the measurement

process, as it relates to the denominator of TUR, should encompass all aspects of the laboratorystandards, environmental effects, measurement processes, etc., but not aspects of the UUT.

Essentially, the TUR denominator was to reflect the capabilityof the laboratory to make highly

accurate measurements. The concept of capability is important here, as the capability of a

laboratory to make accurate measurements was sometimes viewed in the abstract sense, isolated,removed, and independent of any aspects of the UUT. However, this is not the case with the

definition of expanded measurement uncertainty in the denominator of the Z540.3 TUR. All

influences, which affect the result of a measurement, should be included in the expanded

uncertainty. This includes the resolution, noise, and other possible contributions of the UUTwhich can affect the laboratorys ability to accurately perform a measurement on a particular

device. This was reiterated to NASA in another response from the NCSLI 174 Standards Writing

Committee [19].


10/25


The new definition of TUR is meant to serve as a single simplistic metric or barometer to

evaluate the plausibility of a proposed compliance test with regard to mitigating false accept risk.

No distinction is made as to where the risk originates - the UUT or the laboratory standard(s).A low TUR does not necessarily imply that the laboratory standards are not good enough. It

may indicate, however, that the measurement cannot be made without significant false accept

risk due to limitations of the UUT itself (noise, resolution, etc.). Such might be the case wherethe accuracy specification of a device is equal to its resolution or noise floor. This can prevent areliable Pass/Fail decision from being made.

When computing TUR, where specifications have been published at confidence levels otherthan95 %, laboratories have sometimes attempted to convert these specifications into 2 sigma specs

before dividing by the expanded uncertainty(2) of the measurement process. Or, equivalently,

UUT specs were converted to 1 sigma specs for division by the standard uncertainty (1) of

the measurement process. Either way, this was believed (and still is by some) to provide a more

useful apples-to-apples ratio for the TUR. Efforts to develop an equivalentor normalizedTestUncertainty Ratio have been documented by several authors [18, 20, 21, 22]. In many cases, the

EOPR (confidence level or In-tolerance probability) for a UUT is the single most influentialfactor affecting the probability of false accept for a population of instruments. Simply making anassumption (i.e. 95% confidence level) regarding this important reliability metric devalues the

usefulness of TUR calculations. In these situations, the validity or utility of risk mitigation is

greatly diminished. Its integrity then depends primarily on the level of effort, honesty, andcompleteness demonstrated by the manufacturer that assigned the accuracy specifications to the

equipment. Were the accuracy specifications arrived at objectively by proper statistical analysis

of a population of instruments? Is the calibration interval for these specifications provided? Is

the calibration period reasonable? Are the tolerance limits conservative and highly reliable; orwere they arrived at by a marketing department or advertising group driven by other

motivations? Unless the TUR numerator and denominator represent similar confidence levels,

this simplistic ratio alone is of little value.

6. Understanding False Accept Risk

Both EOPR and TUR have been shown to influence false accept risk. A closer scrutiny of the

relationship between EOPR, TUR, and false accept risk can reveal some intriguing aspects that

are of great value to the modern calibration laboratory. Investigating the dependency of falseaccept risk on EOPR and TUR requires some mathematical prowess, but is well worth the effort

involved. Details of the calculus operations are deferred to other papers which give an excellent

treatment of the subject matter. Several papers have already been published which expound uponthe mathematics behind the Z540.3 risk requirement at the program level [3, 4, 23, 24, 25].

These publications and many others build upon the seminal works on measurement decision risk

by Eagle, Grubbs, Coon, & Hayes [26, 27, 28] and should be considered required reading forprospective organizations considering adoption of Z540.3.

As this discussion is intended to be more conceptual in nature, a brief overview of the

fundamental principles as they relate to conformance decisions is prudent. As stated earlier,

M&TE tolerance limits are often set by the manufacturers published accuracy specifications,establishing the maximum permissible error limits that the end-user expects the device to comply

with during the calibration interval. The device may be declared In-tolerance by the calibration


11/25


laboratory if the UUT is observed to have a calibration result that is within the tolerancelimitsL. This can be written as . The observed calibration result is relatedto the actual or truedevice under test error and the measurement process error by theequation .

Errors (such as and ), as well as measurement observations (such as ), are statisticalquantities represented by random variables and characterized by probability density functions.These distributions represent the relative likelihood of any specific error ( and ) ormeasurement observation () actually occurring. The key concept is that the observation is only an approximation of reality. The smaller the measurement uncertainty, the morevalid the approximation. Most often in metrology, these distributions are of the Gaussian form or

normal distributionand are described by two parameters, a mean or average, and a standarddeviation. The standard deviation is a measure of the variability or spread in the values fromthe mean. In simplified cases, it is often assumed that the meanof all the possible error values

will be zero, unless their evidence to the contrary (i.e. a bias exists). Real-world measurements

are a function of both the UUT performance and the measurement uncertainty,

where . The relative likelihood of all possible measurement results isrepresented by the two dimensional surface area created by the joint probability distribution

given by Figure 4 graphically represents the concept ofprobability density of measurement and assumes that measurement uncertainty and the UUT

distribution follow a normal or Gaussian probability density function.

Figure 4. Graphical Representation of the Probability Density of Measurement

The shape and the angle of Figure 4 changes as a function of TUR and EOPR. It is worth noting

that the quantity is the parameter being sought after when a calibration is performed, but is what is obtained via the measurement process. At anytime, the precise value of isalways unknown due to the possibility of measurement process errors described by


12/25


uncertainty. Therefore it is never possible to determine exactly. Computing an actualnumeric value for the probability (PFA or PFR) involves integrating a joint probability density

function over the appropriate two dimensional surface area defined by the limits stated below.To compute probability, the input variables can be simplified to TUR and EOPR and the

mathematics represented symbolically as in Figure 5.

Figure 5. Conceptual Dependency of PFA on TUR and EOPR

Incorrect (false) acceptance decisions are made where the condition exists that:

and

The UUT is actually OUT of tolerance

and

The UUT is observed to be IN tolerance

Likewise incorrect (false) rejectdecisions are made where the condition exists that:

and

The UUT is actually OUT of tolerance

and

The UUT is observed to be IN tolerance

Integration over the entire joint probability region will yield a value of 1 as would be expected.

In the ideal case, if the measurement uncertainty was zero, the probability of measurement errors

occurring would be zero. The measurement observations would then reflect the behavior ofthe UUT perfectly and the distribution of possible measurement results would be limited to the

distribution of actual UUT errors. That is, would equal and the graph in Figure

UUT is truly OUT of tolerance

and

UUT is observed to be IN tolerance

UUT is truly IN tolerance

and

UUT is observed to be OUT of tolerance


13/25


4 would collapse to a straight line at a 45 degree angle, where the angle is given by

In this case, However, since real-world measurements are always hindered

by the probability of measurement errors, observations do not always reflect reality and false

accept risk results. The case of false accept and false reject is represented graphically in Figure 6.

Figure 6. Graphical Representation of False Accept and False Reject Decisions

7. Efficient Risk Mitigation

Early practices in metrology such as 4 to 1 TUR were based on assumptions and were necessaryto ease the burden of rigorous time consuming computational requirements of measurement risk

analysis. The limitations of early computing devices no longer exist and any modern computer is

capable of computing PFA using common mathematical tools such as MathCad, Excel

, or

commercially available risk analysis software. However, the collection, management, and

logistics of risk analysis can be quite cumbersome and time consuming, especially for

multifunction instrumentation.

To comply with Z540.3 (5.3b), the PFA must not exceed 2 %. However, computing an actualnumeric value for PFA is not necessarily required in order to comply with the 2 % rule. To

understand how this is possible, the boundary conditions of PFA can be investigated by varyingthe TUR and EOPR over a wide range of values and observing the resultant PFA. This is best

illustrated by a three dimensional surface plot, where the x and y axis represent TUR and EOPR,

and the height of the surface on the z-axis represents PFA. See Figures 7 & 8.

9.80 V

9.85 V

9.90 V

9.95 V

10.00 V

10.05 V

10.10 V

10.15 V

#0 #1 #2 #3 #4 #5

Voltage

Calibration Scenarios

Possible Measurement Results of a 10 V Source (UUT)

Measured Value (Observed)

UUT Actual (True)

Upper Tolerance Limit (+L )

Lower Tolerance Limit (-L)

UUT Nominal Voltage

euuteobs

estd

False RejectFalse Accept

euut eobs

estd

eobs= euut+ estd


14/25


15/25


This surface plot combines both aspects affecting false accept risk into one visual representation

that can further illustrate the relationship between the variables TUR and EOPR. One curiousobservation is that the PFA can neverbe greater than 13.6 % for anycombination of TUR and

EOPR. That is, allcalibration processes result in a PFA of less than 13.6 %, regardless of how

low the TUR is and how low the EOPR is. No calibration scenario can be proposed which willresult in a PFA greater than 13.6 %. This maximum value of 13.6 % PFA results when the TURis approximately 0.3 to 1 and the EOPR is 41 %. Any change, higher or lower, for either the

TUR or EOPR will result in a PFA lower than 13.6 %.

One particularly useful observation is that, for allvalues of EOPR, the PFA never exceeds 2 %

when the TUR is above 4.6 to 1. That is, regardless of what the actual EOPR might be for a

UUT, the PFA has an upper boundary condition of 2 % as long as the TUR is greater than 4.6 to

1. Notice in Figure 8 that the darkest blue region of the PFA surface is always below 2 %. Evenif the TUR axis in the above graph is extended to infinity, the darkest blue PFA region would

continue to fall below the 2 % threshold. Calibration laboratory managers will find this an

efficient risk mitigation technique for compliance with Z540.3. This fact can eliminate theburden of collecting, analyzing, and managing EOPR data in circumstances where the TUR has

been evaluated and shown to be greater than 4.6 to 1.

This concept can further be illustrated if the perspective (viewing angle) of the above surfaceplot in Figure 8 is rotated. This allows the two dimensional maximum outer-envelopeto be easily

viewed. With this perspective, PFA can be plotted only as a function of TUR (Figure 9). In this

instance, the worst-case EOPR is used whereby the maximum PFA is produced for each TUR.

Figure 9. Worst Case False Accept Risk vs. TUR

0 %

2 %

4 %

6 %

8 %

10 %

12 %

14 %

16 %

0 1 2 3 4 5 6 7

ProbabilityofFalseAccept(Risk)

Test Uncertainty Ratio (TUR)

Max Risk vs TUR(Assumes Worst-Case EOPR for a given TUR)

False Accept Risk is

always below 2 % for

TUR 4.6 to 1


16/25


There is no EOPR which would yield a PFA above 2 % for TURs greater than 4.6 to 1.

Therefore, whatever the EOPR is for items exhibiting TUR above 4.6 to 1, it is adequate and

need not be investigated to ensure that PFA is less than 2 %. The left-hand side of the graph inFigure 9 might not appear intuitive at first. Why would the PFA suddenly decrease as the TUR

drops below 0.3 to 1 and approaches zero? While a full explanation is beyond the scope of this

paper, the answer lies in the number of items rejected (falsely or otherwise) when extremely lowTUR exists.

Another benefit of examining the boundary conditions of the surface plot can be realized by

noting that the PFA is always below 2 % where the true EOPR is greater than 95 %. This is trueregardless of how low the TUR is. Even cases with extremely low TURs (even below 1:1) will

always produce a PFA less than 2 % where the true EOPR exceeds 95 %. Again, if the

perspective of the PFA surface plot in Figure 8 is properly rotated, a two dimensional outer-

envelope is produced whereby PFA can be plotted only as a function of EOPR (Figure 10). Inthis case, the worst-case TUR is used, maximizing the PFA. This results in a graph of PFA that is

a function only of the EOPR, with each instantaneous point computed at the TUR which

maximizes the PFA. In other words, a worst-case TUR has been assumed at each and everypoint on the curve below. This curve represents the absolute worst possible PFA for any given

EOPR and knowledge of the TUR is not required.

Figure 10. Worst Case False Accept Risk vs. EOPR

0%

2%

4%

6%

8%

10%

12%

14%

16%

0 % 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 %


True End Of Period Reliability (EOPR)

Max Risk vs EOPR(Assumes Worst-Case TUR for a given EOPR)

False Accept Risk is


true EOPR 95 %


17/25


Figure 11. EOPR DataThree Cases (Poor, Moderate, Excellent)

As was the case of low TUR, a similar phenomenon is noted on the left-hand side of the graph in

Figure 10; the maximum PFA decreases for true EOPR values below 41 %. As EOPRapproaches zero on the left side, most of the UUT values lie far outside of the tolerance limits.

When the values are not in close proximity to the tolerance limits, the risk of falsely accepting an

item is low. Likewise on the right-hand side of the graph, where the EOPR is very good (near

100 %), the false accept risk is low. Both ends of the graph represent areas of low PFA becausemost of the UUT values have historically been found to lie far away from the tolerance limits,

either significantly Out-of-tolerance (left side), or significantly In-tolerance (right side). ThePFA is highest, in the middle of the graph, where EOPR is only moderately poor and much of thedata is found near the tolerance limits. Refer to Figure 11.

8. True vs. Observed EOPR

Until now, this discussion has been limited to the concept of true EOPR. This caveat deservesfurther attention. The idea of a trueEOPR implies that an immaculate value for reliability exists,

which has not been influenced by any non-ideal factors. However, as with all empirical data,

this is not the case. In the calibration laboratory, reliability data is collected from a history of

real-worldobservations or measurements. These observations of UUTs aremade using actual

equipment often expensive reference standards with very low uncertainty under controlledconditions. But even the best available standards have some finite uncertainty and the UUT

itself often contributes noise and other undesirable effects. All of this uncertainty impinges on

the integrity of compliance decisions, manifesting in observedreliability data that is not a truereflection of reality. The measurement uncertainty contaminates the EOPR data to some

degree, calling into question its validity. The observedEOPR is never a completely accurate

representation of the trueEOPR.

Region 3

EOPR = 100 %

Region 2

EOPR = 50 %Region 1

EOPR = 20 %

9.75 V

9.80 V

9.85 V

9.90 V

9.95 V

10.00 V

10.05 V

10.10 V

10.15 V

10.20 V

Voltage

Low Risk High Risk Low Risk

EOPR Data

UUT As Received Condition

L

-L


18/25


The difference between the observed and true EOPR becomes more pronounced as the

measurement uncertainty increases (i.e. TUR drops). A low TUR can result in a significant

deviation between what is observedand what is trueregarding the reliability data. This concepthas been presented in some depth previously [23, 29, 30, 31]. The reported or observed EOPR

from calibration history includes all influences from the measurement process. In this case, the

standard deviation of the observed distribution is given by whereand are derived from statistically independent events. The corrected or true standarddeviation can be approximated by removing the effect of measurement uncertainty and solving

for where is the true distribution width represented by standarddeviation.

The above equation shows that the standard deviation of the observed EOPR data is alwaysworse (higher) than the true EOPR data. That is, the reliability history maintained by a

laboratory will always cause the UUT data to appear further dispersed than what is actually true.

This results in an 89 % observedEOPR boundary condition where the PFA is less than 2 % for

allpossible values of TUR6. See Figure 12 below.

Figure 12. PFA Assumes Worst Case TUR for TrueEOPR and ObservedEOPR.

If measurement uncertainty is thought of as noise, and the EOPR is the measurand, then the

observed data will have greater variability or scatter than what the truevalue of the EOPR really

6When correcting EOPR under certain conditions, low TUR values can result in imaginary values for uut. This can

occur where uutand stdare not statistically independent and/or the levels of confidence associated with stdand/or

uut have been misrepresented.

0 %

2 %

4 %

6 %

8 %

10 %

12 %

14 %

16 %

0 % 20 % 40 % 60 % 80 % 100 %


End of Period Reliability

Max False Accept Risk vs EOPR

(Assumes Worst-Case TUR for a given EOPR)

Observed

TRUE

False Accept Risk


observed EOPR 89 %


19/25


is. Measurement uncertainty always hinders the quest for accurate data; it never helps. It should

be noted that the true value of a single data point may indeed be higher or lower than the

measured value, due to uncertainty. For any single instance, it is never known whether themeasurement uncertainty contributed an additive error or negative error. Therefore, it is not

possible to remove the effect of measurement uncertainty from a singlemeasurement result. But

EOPR data is a historical collection of many Pass/Fail compliance decisions which can berepresented by a normal probability distribution with a standard deviation . Sometimes themeasurement uncertainty will contribute positive errors and sometimes it will contributenegative errors. If the mean of these errors is assumed to be zero, the effect of measurementuncertainty on a population of EOPR data can be removed as previously shown. The inverse

normal function is used to estimate from observed EOPR data [32].

where -1

represents the inverse normal distribution.

EOPR is a numerical quantity arrived at by statistical means applied to empirical data analogous to a Type A estimate in the language of the GUM [33]. The data comes from repeated

measurements made over time rather than accepting manufacturers claims at face value -

analogous to Type B or heuristic estimates. However, the influence of the measurement processis always present. This method of removing measurement uncertainty from the EOPR data is a

best estimate of the truereality or reliability which is sought through measurement.

There are many other factors that affect EOPR. For items which are routinely re-submitted to the

calibration lab, shortening or lengthening the calibration interval will affect EOPR. Laboratories

which are presently ISO-17025 compliant may not currently have the mechanisms in place to

determine and manage calibration intervals or adjustment policies. Such a policy has a directinfluence on the probability of false acceptance for the population of instruments. A

laboratorys adjustment policy can also directly affect the false accept risk to individual

instruments and to the population at large. Laboratories which require the technician to adjust oralign an instrument exhibiting an error, which exceeds some established percentage of its

allowable tolerance, can control false accept risk at the bench level and the program level. An

adjustment policy affects the false accept risk to the population of instruments by virtue ofaffecting the In-tolerance probability or EOPR of all similar instruments.

To summarize, the 2 % PFA maximum boundary condition, formed by either 4.6 to 1 TUR or

89 % observed EOPR, can greatly reduce the effort and labor required for the modern calibration

laboratory in managing false accept risk for a significant portion of the M&TE submitted forcalibration. For some labs, obtaining EOPR data might be the most burdensome task, while

TUR could be the most difficult parameter for other labs to produce. The PFA boundary

condition can be leveraged from either perspective, providing benefit to practically alllaboratories. However, there will still be instances where the TUR is lower than 4.6 to 1 and the

observed EOPR is less than 89 %. In these instances, it is still possible for the PFA to be less

than 2 %. In these cases, a full PFA calculation is required to compute risk to show the 2 %


20/25


requirement has not been exceeded. However, other techniques can be employed to ensure that

the PFA is held below 2 %.

9. Guardbanding

During a compliance decision, it is sometimes helpful to establish acceptance limits A at the

time-of-test that are more stringent (tighter) than the manufacturers tolerance limits L.Acceptance limits are often called guardband limits or test-limits. These time-of-test constraints

are only imposed during the calibration process when making compliance decisions in order toreduce the risk of a false acceptance. It is only necessary to implement acceptance limits A,

which differ from the tolerance limits L, when the false accept risk is higher than desired or as

part of a program to keep risk below a specified level. Acceptance limits may be chosen tomitigate risk at the bench level or program level. PFA calculations may be used to establish

acceptance limits based on the mandated risk requirements. In most instances, where guard

bands are applied, the tolerance limits are temporarily tightened or reduced to create

acceptance limits needed to meet a PFA goal. The subject of guardbanding is extensive andnovel approaches exists for computing and establishing acceptance limits to mitigate risk, even

where EOPR data is not available [25]. However, in the simplified case of no guardbanding, theacceptance limitsAare set equal to the tolerance limitsL(A =L).

The Z540.3 Handbook references six possible methods to achieve compliance with the 2 % rule

[16]. One particularly useful method employing a guardbanding technique is described in

Method 6 of the Handbook [16, 25]. Strictly speaking, this method does not requireEOPR datato be available because the method relies on using worst-caseEOPR, computed for a specified

TUR value. Using this approach, a guardband multiplier is computed as a function of TUR. The

acceptance limits are expressed as follows: , where is the newly establishedacceptance limits, is the original tolerance limits, is the expanded measurement processuncertainty, and is multiplying factor that yields a risk of a specified maximum target. Figure13 plots guardband multipliers for varying levels of risk. The risk level for Z540.3 is specifiedat 2% but could be a different value depending on the agreement with the customer. waspreviously calculated by Dobbert [25] by fitting a line though the data points that mitigate risk to

a level of 2 % and is given by the following simplified formula.

It can be seen that the line is a good fit for the condition where . The intent was tokeep the equation simple for ease of use but cover the appropriate range of TUR values that

make physical sense. It has been shown in this paper that for .To verify this we can set and solve for TUR and we indeed find that is aboundary condition. It is worth noting that, for . Thisimplies that a calibration lab could actually increase the acceptance limits Abeyond the UUT

tolerancesLand still be compliant with the 2 % rule. While not a normal operating procedure

for most calibration laboratories, setting guard band limits outside the UUT tolerance limits isconceivably possible while maintaining compliance with the program level risk requirement of

Z540.3. In fact, laboratory policies often require items to be adjusted back to nominal for

observed errors greater that a specified portion of their allowable tolerance limitL.


21/25


Figure 13. Guardband Multiplier for Acceptable Risk Limits as a Function of TUR

10. Conclusion and Summary

Organizations must determine if risk is to be controlled for individual workpieces at the benchlevel, or mitigated for the population of items at the program level

7. Where ISO-17025 was not

specific in methodologies to account for the uncertainty of measurement, Z540.3 provides for

detailed treatment of the uncertainty during compliance testing and much more. Z540.3 points to

ISO-17025 as suitable for the core competency of calibration labs, but it also levies therequirements of section 5.3. For the calibration program, it places an upper limit on the

probability associated with the risk of incorrectly accepting measurements as In-tolerance during

compliance decisions.

ISO-17025 contains no specific provision for utilizing a TUR as a method to account for the

uncertainty, while Z540.3 does allow the limited use of a 4 to 1 TUR. However, under Z540.3, a4 to 1 TUR is provided onlyas a secondary alternative to the preferred method of limiting PFA

to less than 2 % and, even then, only when calculating PFA is impracticable. However, the

indiscriminate blanket use of 4 to 1 TUR, in lieu of calculating and limiting the PFA, is notacceptable for compliance with Z540.3. It must first be demonstrated that the more rigorous,

primary method of limiting false accept risk to


22/25


requirement and the Z540.3 TUR requirement would also pose significant challenges to the

transition process. Compliance with the 2 % rule in Z540.3 must be accomplished by calculating

PFA and/or limiting its probability to less than 2 %.

Computation of PFA requires the use of double integral calculus formulas of joint probability

density functions, the solutions of which are not trivial. The input variables to these formulascan be reduced to EOPR and TUR. While expanded uncertainties (and possibly TURs) may bewell documented for most labs which are already in compliance with ISO-17025, it is EOPR that

is of pivotal importance to organizations considering the adoption of Z540.3. For the purposes

of complying with the 2 % rule in Z540.3, properly calculatingPFA requires the availability ofEOPR data.

While some calibrations laboratories currently maintain historical EOPR data, these metrics are

almost universally retained at the equipment-level (i.e. did the M&TE instrument as a whole,Pass or Fail?). For multi-parameter, multi-test-point instruments, a single failure of one test-

point represents a Fail at the instrument-level. Historical data at many laboratories simply

indicates whether the instrument Passed or Failed; not which specific test-point was Out-of-tolerance. Therefore, EOPR history data or metrics are not readily available at the test-point-

level for the vast majority of calibration laboratories. This can, however, be used as a

conservative estimate of EOPR for PFA evaluation. Even for laboratories which maintain a

comprehensive database of all measurement data ever acquired for all M&TE calibrated at thetest-point-level, these results are stored in a plethora of different manners and systems including

handwritten, hardcopy, paper datasheets. Obtaining and/or extracting this data for the

computation of PFA represent an enormous logistical challenge with a correspondingly highcost. Moreover, if laboratories do not already posses this EOPR history, many years of

calibration events may be required to obtain the reliability data necessary for the calculation of

PFA. Therefore, efficient mitigation strategies are required.

There are six methods listed in the Z540.3 Handbook for complying with the 2 % false accept

risk requirement [16]. It should be noted that this paper has specifically focused on some efficient

approaches to this objective. This does not, in any way, negate the use of other methods nordoes it imply that the ones discussed here are necessarily the best methods for any particular

laboratory or program. It presents some efficient methods for taking the uncertainty into

account and mitigating risk, even where a numeric value for the measurement uncertainty orEOPR might be unknown. Basic strategies for handling risk without rigorous computation of

PFA are:

Analyze EOPR data. This will most likely be done at the instrument-level, as opposed tothe test-point level, depending on data collection methods. If the observed EOPR data

meets the required level of 89 %, then the 2 % PFA rule has been satisfied.

If this is not the case, then further analysis is needed. TUR must be determined at eachtest point. If the analysis reveals that the TUR is greater than 4.6 to 1, no further action isnecessary and the 2 % PFA rule has been met.

If neither the EOPR nor TUR threshold is met, a Method #6 guardband can be applied.


23/25


Compliance with the 2 % rule can be accomplished by either calculating PFA and/or limiting its

probability to less than 2% by the methods presented above. If these methods are not sufficient,

other strategies must be used. Beyond guardbanding, alternative methods of mitigating PFA areavailable [16].

Determine if the acceptable tolerance can be expanded and still meet customerrequirement. This is sometimes known as aLimited Calibrationto de-ratedspecifications.

Determine if the measurement uncertainty of the calibration process can be improvedperhaps by changing calibration equipment, calibration method, etc. Shortening thecalibration interval may improve EOPR and reduces risk over time.

However a poorly specified UUT, which is being tested for compliance to overly optimistic

specifications, presents a serious problem for the calibration lab and for the end-user of M&TE.In some circumstances, shortening the calibration interval, implementing conservative guard-

bands, and applying rigorous laboratory adjustment policies cannot render a dog into a gem.Occasionally, no amount of effort or action on the part of the calibration laboratory can force a

UUT to comply with unrealistic expectations of accuracy and performance. Contacting themanufacturer with this evidence may result in the issuance of revised/amended specifications of

a more realistic nature.

Assumptions, approximations, estimations, and uncertainty have always been part of metrology

and will continue to be present for all of time; there are few guarantees or absolutes. Practically

no process can guarantee that instruments will provide the desired accuracy, or will functionwithin their assigned tolerances during any particular application or use. However, through a

well managed calibration process, confidencecan be attained that an instrument will perform as

expected and within limits. Quantification of this confidence can be realized via analysis ofuncertainty, EOPR, and false accept risk. Reducing the number of assumptions and improvingthe estimations involved during calibration can provide higher levels of confidence, reduced risk,

and improved quality. Historical reliance on several key assumptions has limited the

effectiveness of time-honored practices to manage risk. Z540.3 provides the organizationalframework to advance the state of metrology beyond these conventional limitations and efficient

methods exists for compliance with this standard.

Acknowledgement

The authors would like to thank the many people who contributed to an understanding of the

subject matter presented here. Specifically, the contributions of Perry King (Bionetics), ScottMimbs (NASA), and Jim Wachter (Millennium Engineering and Integration) at Kennedy Space

Center were invaluable. Several graphics were generated using PTCs MathCad

14. Where

numerical methods were more appropriate, Microsoft Excelwas used incorporating VBA

functions developed by Dr. Dennis Jackson of the Naval Surface Warfare Center in Corona, CA.


24/25


References

1. JCGM 200:2008 (VIM).International vocabulary of metrologyBasic and general conceptsand associated terms (VIM). Joint Committee for Guides in Metrology - Working Group 2, 2008.

2. ANSI/NCSL Z540.3:2006.Requirements for the Calibration of Measuring and Test

Equipment. American National Standards Institute / NCSL International, 2006.3. D. Deaver, J. Somppi.A Study of and Recommendations for Applying the False Acceptance

Risk Specification of Z540.3. Proceedings of the Measurement Science Conference, AnaheimCA, 2010.

4. H. Castrup, Integrated Sciences Group.Risk Analysis Methods for Complying with Z540.3.

Proceedings of the NCSL International Workshop and Symposium, St. Paul Minnesota, 2007.

5. M. Dobbert.A Pragmatic Method for Pass/Fail Conformance Reporting that Complies with

ANSI Z540.3, ISO 17025, and ILAC-G8. Proceedings of the NCSL International Workshop and

Symposium, Providence RI, 2009.

6. ANSI/NCSL Z540.1 1994 (R2002). Calibration & Measurement & Test Equipment - GeneralRequirements. American National Standards Institute / National Conference of Standards

Laboratories, ISBN: 9781584640035, 2002

7. ISO/IEC 17025:2005(E). General requirements for the competence of testing and calibration

laboratories. International Organization for Standardization / International Electrotechnical

Commission, 2005.

8. A2LA:R205. Specific Requirements: Calibration Laboratory Accreditation Program.American Association for Laboratory Accreditation, 10 May 2011.

9. ILAC-G8:1996. Guidelines on Assesment and Reporting of Compliance with Specification

(based on measurements and tests in a laboratory). International Laboratory Accreditation

Cooperation, 1996 (Revised in 2009)

10. UKAS:M3003. The Expression of Uncertainty and Confidence in Measurement (Appendix

M). United Kingdom Accreditation Service, 2nd Edition, Jan 2007.

11. ASME B89.7.3.1-2001. Guidelines for Decision Rules: Considering MeasurementUncertainty in Determining Conformance to Specifications. American Society of Mechanical

Engineers, ISBN: 0791827429, Jan 2001

12. ISO-14253-1:1998(E). Geometrical Product Specifications (GPS) - Inspection by

measurement of workpieces and measuring equipment - Part 1: Decision Rules for proving

conformance or non-conformance with specifications. International Organization forStandardization, First Edition, 15 November 1998.

13. ASME B89.7.4.1-2005 (Technical Report).Measurement Uncertainty Conformance Testing:Risk Analysis. American Society of Mechanical Engineers, 03 February 2006.

14. NCSLI.Response to NASA Interpretation Request (IR2).NCSL International, 174 Standards

Writing Committee, 06 March 2008.

15. S. Mimbs.Request for Interpretation; Correspondence Letter.NASA - John F. Kennedy

Space Center (SA-G), 30 July 2007.


25/25

16. ANSI/NCSL Z540.3 Handbook.Handbook for the Application of ANSI/NCSL Z540.3:2006 -

Requirements for the Calibration of Measuring and Test Equipment. American National

Standards Institute / NCSL International, 2009.

17. J. Bucher (Editor). The Metrology Handbook. American Society for Quality, Measurement

Quality Division, ASQ Quality Press, ISBN 0-87389-620-3, 2004.

18. H. Castrup, Integrated Sciences Group.A Note on the Accuracy Ratio Requirements, 2006.

19. NCSLI.Response to NASA Interpretation Request (IR1).NCSL International, 174 StandardsWriting Committee, 06 March 2008.

20. M. Nicholas, L. Anderson. Guardbanding Using Automated Calibration Software.

Proceedings of the NCSL International Workshop and Symposium, Salt Lake City UT, 2004.

21. Fluke Corporation. Calibration: Philosopy in Practice. ISBN: 978-0963865007, May 1994.

22. T. Skwircznski. Uncertainty of the calibrating instrument, confidence in the measurement

process and the relation between them. International Organization of Legal Metrology , OIML

Bulletin, Vol XLII, No.3, July 2001.23. NASA-HDBK-8739.19-4.Estimation and Evaluation of Measurement Decision Risk, NASA

Measurement Quality Assurance HandbookANNEX 4. July 2010.

24. M. Dobbert. Understanding Measurement Decision Risk. Proceedings of the NCSLInternational Workshop and Symposium, St. Paul Minnesota, 2007.

25. M. Dobbert.A Guard Band Strategy for Managing False Accept Risk. Proceedings of the

NCSL International Workshop and Symposium, Orlando FL, 2008, .

26. A. Eagle.A Method for Handling Error in Testing and Measuring. Industrial Quality

Control, March 1954.

27. F. Grubbs, H. Coon. On Setting Test Limits Relative to Specification Limits. IndustrialQuality Control, March 1954.

28. J. Hayes, Technical Memorandum No. 63-106.Factors Affecting Measuring Reliability. U.S.Naval Ordnance Laboratory, Measurements Reliability Branch, Production Quality Division,

Missle Evaluation Dept, 24 October 1955.

29. S. Mimbs.Measurement Decision Risk - The Importance of Definitions. Proceedings of the

Measurement Science Conference, Anaheim, CA, 2008.

30. J. Ferling. The Role of Accuracy Ratios in Test and Measurement Processes. Proceedings of

the Measurement Science Conference Proceedings, Long Beach CA, 1984.

31. S. Mimbs. Using Reliability to Meet Z540.3s 2% Rule. Proceedings of the NCSLInternational Workshop and Symposium, National Harbor MD, 2011.

32. H. Castrup, Integrated Sciences Group.Analytical Metrology SPC Methods for ATEImplementation. Proceedings of the NCSL Workshop and Symposium, Albuquerque NM, 1991.

33. ISO/IEC Guide 98-3:2008. Uncertainty of measurement -- Part 3: Guide to the expression of

uncertainty in measurement (GUM:1995). International Organization for Standardization /

International Electrotechnical Commission, 2008.

(NASA J.harben P.reese) Implementing Strategies for Risk Mitigation (NCSLI-2011)

Documents

Transcript of (NASA J.harben P.reese) Implementing Strategies for Risk Mitigation (NCSLI-2011)