Meteorological Measurement Error Analysis based on ANSI/ANS-3.11 (2005) Kenneth G. Wastrack...

Meteorological Measurement Error Analysis

based on ANSI/ANS-3.11 (2005)

Kenneth G. WastrackKenneth G. Wastrack

Tennessee Valley AuthorityTennessee Valley Authority

NUMUG 2005NUMUG 2005

ANSI/ANS-3.11 (2005) Error AnalysiANSI/ANS-3.11 (2005) Error Analysiss

22

IntroductionIntroduction

Presentation TopicsPresentation Topics

Error Calculation HistoryError Calculation History ANSI/ANS-3.11 (2005) Method for Calculating ANSI/ANS-3.11 (2005) Method for Calculating

AccuracyAccuracy Example CalculationExample Calculation

Accuracy is the primary indicator of meteorological is the primary indicator of meteorological system performance. Observed accuracy must be system performance. Observed accuracy must be calculated in a consistent manner to permit calculated in a consistent manner to permit comparison with specified values. One of the key comparison with specified values. One of the key changes in ANSI/ANS-3.11 (2005) addresses this changes in ANSI/ANS-3.11 (2005) addresses this issue.issue.


33

Error Calculation HistoryError Calculation History

Section 4, "Instrument Accuracy," states plus or minus Section 4, "Instrument Accuracy," states plus or minus (±) accuracy values for different meteorological (±) accuracy values for different meteorological variables, but does not indicate if the accuracy applies variables, but does not indicate if the accuracy applies just to the sensors or to the entire data channel.just to the sensors or to the entire data channel.

No calculation methodology is defined.

Safety Guide 1.23, "Onsite Meteorological Programs," 1972


44


Section 3, “System Accuracy”:Section 3, “System Accuracy”: System accuracy refers to the composite channel System accuracy refers to the composite channel

accuracy.accuracy. System accuracy defined as the root sum of the squares.System accuracy defined as the root sum of the squares. For time-averaged values, random errors may be For time-averaged values, random errors may be

decreased by considering the number of samples.decreased by considering the number of samples. The system accuracies similar to Safety Guide 1.23 (1972).The system accuracies similar to Safety Guide 1.23 (1972).

NRC defines calculation methodology.

Proposed Revision 1 to Regulatory Guide 1.23, "Meteorological Measurement Programs for Nuclear Power Plants," ~1981 (never issued)(never issued)


55


Section 6.1, “System Accuracy”:Section 6.1, “System Accuracy”: System accuracy refers to the composite accuracy. System accuracy refers to the composite accuracy. System accuracy defined as the root sum of the System accuracy defined as the root sum of the

squares (squares (references “Brooks and Caruthers” methodsreferences “Brooks and Caruthers” methods). ). For time-averaged values, random errors may be For time-averaged values, random errors may be

decreased by considering the number of samples.decreased by considering the number of samples. The system accuracies are similar to those in RG 1.23 The system accuracies are similar to those in RG 1.23

(1972) and Proposed Revision 1 to RG 1.23 (1981).(1972) and Proposed Revision 1 to RG 1.23 (1981).

ANSI/ANS-2.5 (1984) fills gap in guidance.

ANSI/ANS-2.5 (1984), "Standard for Determining Meteorological Information at Nuclear Power Sites," 1984


66


Not a guidance document, but provides insight into Not a guidance document, but provides insight into how ANSI/ANS-2.5 (1984) guidance was interpreted how ANSI/ANS-2.5 (1984) guidance was interpreted and used.and used.

Section 3, “Potential Sources of Error”:Section 3, “Potential Sources of Error”: System accuracy refers to the composite accuracy. System accuracy refers to the composite accuracy. Distinguishes between “bias” and “random” errors.Distinguishes between “bias” and “random” errors.

Illustrates implementation of ANSI/ANS-2.5 (1984) and provides example calculations.

NUMUG Presentation, "A Methodology for Calculating Meteorological Channel Accuracies," by Brad Harvey, 1999


77


Section 7.1, “System Accuracy”:Section 7.1, “System Accuracy”: Calculation methodology is described as "root-mean-Calculation methodology is described as "root-mean-

square" (RMS) rather that "root sum of the squares" square" (RMS) rather that "root sum of the squares" (RSS).(RSS).

Does not clearly state that the calculation methodology Does not clearly state that the calculation methodology has changed--has changed--many users don’t realize the many users don’t realize the calculation methodology has changedcalculation methodology has changed. .

No longer distinguishes between instantaneous and No longer distinguishes between instantaneous and time-averaged values.time-averaged values.

Not certain changes were really intended.

ANSI/ANS-3.11 (2000), "Determining Meteorological Information at Nuclear Facilities," 2000


88


Root-mean-square (RMS)

RMS =RMS =

Root sum of the squares (RSS)

RSS = (rRSS = (r11))22+ (r+ (r22))2 2 +. . . + (r+. . . + (rxx))22

ANSI/ANS-3.11 (2000), "Determining Meteorological Information at Nuclear Facilities," 2000 (continued) (continued)

222

21 xrrrRSS

(r(r11))22+ (r+ (r22))2 2 +. . . + (r+. . . + (rxx))22

xx

r1, r2 . . . rx are random error components.

x is the number of components.


99


Section 7.1, “System Accuracy” has been revised and Section 7.1, “System Accuracy” has been revised and Exhibit 1, “Method for Calculating System Accuracy” has Exhibit 1, “Method for Calculating System Accuracy” has been added.been added.

ANSI/ANS-3.11 (2005) is intended as an ANSI/ANS-3.11 (2005) is intended as an officialofficial source source of guidance. of guidance.

ANSI/ANS-3.11 (2005) defines what is required by ANSI/ANS-3.11 (2005) defines what is required by including the actual equations and step-by-step including the actual equations and step-by-step instructions to ensure a consistent approach.instructions to ensure a consistent approach.

ANSI/ANS-3.11 (2005) reverts back to RSS. ANSI/ANS-3.11 (2005) reverts back to RSS. ANSI/ANS-3.11 (2005) distinguishes between bias errors ANSI/ANS-3.11 (2005) distinguishes between bias errors

and random errors.and random errors.

ANSI/ANS-3.11 (2005), "Determining Meteorological Information at Nuclear Facilities," 2005


1010


Section 7.1 was principally written by Ken Wastrack.Section 7.1 was principally written by Ken Wastrack. Significant input from Paul Fransiloi (SAIC), Brad Significant input from Paul Fransiloi (SAIC), Brad

Harvey (NRC), and Matt Parker (Westinghouse Harvey (NRC), and Matt Parker (Westinghouse Savannah River).Savannah River).

Reviewed by about 30 members of the ANS-3.11 Reviewed by about 30 members of the ANS-3.11 working group.working group.

Comments from Stan Krivo (EPA) and Walt Schalk Comments from Stan Krivo (EPA) and Walt Schalk (NOAA).(NOAA).



1111


Section 7.1 - System AccuracySection 7.1 - System Accuracy

Accuracy values shall reflect the performance of the total system, and Accuracy values shall reflect the performance of the total system, and shall be based on the more stringent of the individual facility shall be based on the more stringent of the individual facility requirements or the minimum system accuracy and resolution requirements or the minimum system accuracy and resolution requirements given in requirements given in Table 1Table 1 (which provides values for a monitoring (which provides values for a monitoring system using typical tower-mounted sensors and digital data processing system using typical tower-mounted sensors and digital data processing systems).systems).

System accuracy should be estimated by performing system calibrations, System accuracy should be estimated by performing system calibrations, or by calculating the overall accuracy based on the system's individual or by calculating the overall accuracy based on the system's individual components. Accuracy tests involve configuring the system near to components. Accuracy tests involve configuring the system near to normal operation, exposing the system to multiple known operating normal operation, exposing the system to multiple known operating conditions representative of normal operation, and observing results. conditions representative of normal operation, and observing results. Data channels may be separated into sequential components, as long as Data channels may be separated into sequential components, as long as results from each component are directly used as input for the next results from each component are directly used as input for the next component in sequence. component in sequence. Exhibit 1Exhibit 1 provides a method that should be used provides a method that should be used to calculate system accuracy from individual component accuracy values.to calculate system accuracy from individual component accuracy values.



1212

Method for Calculating System Method for Calculating System AccuracyAccuracy

[from ANSI/ANS-3.11 (2005) Exhibit 1][from ANSI/ANS-3.11 (2005) Exhibit 1]

1.1. Identify the individual components that Identify the individual components that contribute to system accuracy.contribute to system accuracy. SensorSensor Installation error (e.g., sensor alignment)Installation error (e.g., sensor alignment) Operational error (e.g., solar heating of temp. Operational error (e.g., solar heating of temp.

sensors)sensors) Data processing equipmentData processing equipment Computer (e.g., conversion equations)Computer (e.g., conversion equations) CalibrationsCalibrations etc.etc.

SA


1313



2.2. Classify the error type for each component.Classify the error type for each component. Bias errors (b (b11, b, b22, . . . b, . . . bxx).).

Bias (or systematic) errors consistently affect Bias (or systematic) errors consistently affect the system accuracy in a known manner.the system accuracy in a known manner.

For example: Solar heating only For example: Solar heating only increasesincreases apparent air temperatures during daytime.apparent air temperatures during daytime.

Random errors (r (r11, r, r22, . . . r, . . . rxx).).

Random errors are independent and can Random errors are independent and can fluctuate within the range between the fluctuate within the range between the extreme maximum and minimum values.extreme maximum and minimum values.


1414


[from ANSI/ANS-3.11 (2005) Exhibit 1][from ANSI/ANS-3.11 (2005) Exhibit 1]3.3. Estimate the values of the component errors Estimate the values of the component errors

based on engineering analysis, vendor based on engineering analysis, vendor specifications, accuracy tests, or operational specifications, accuracy tests, or operational experience.experience. If a bias applies to only a portion of the sampling If a bias applies to only a portion of the sampling

period, multiple calculations of the system accuracy, period, multiple calculations of the system accuracy, using different bias values, are necessary. using different bias values, are necessary.

The random errors of the individual components The random errors of the individual components should represent ±2should represent ±2 values or 2 times the standard values or 2 times the standard deviation of errors based on component testing deviation of errors based on component testing (95.5%). Unless otherwise stated, manufacturer's (95.5%). Unless otherwise stated, manufacturer's data for random errors can normally be assumed to data for random errors can normally be assumed to be ±2be ±2 values. values.


1515


[from ANSI/ANS-3.11 (2005) Exhibit 1][from ANSI/ANS-3.11 (2005) Exhibit 1]4.4. Perform time-average adjustments for each Perform time-average adjustments for each

random error component.random error component.a = r/( n)a = r/( n)

Where:Where: r is the unadjusted random error r is the unadjusted random error

component.component. n is the number of samples.n is the number of samples. a is the adjusted random error component.a is the adjusted random error component.

Note:Note: For instantaneous values (where n=1), a = For instantaneous values (where n=1), a = r.r.


1616



5.5. Calculate "root sum of the squares" (RSS) Calculate "root sum of the squares" (RSS) for the adjusted random error components.for the adjusted random error components.

RSS = (RSS = (aa11))22+ + ((aa22))2 2 +. . . + +. . . + ((aaxx))22

Where:Where: aa11, a, a22, . . . a, . . . axx are adjusted random error are adjusted random error

components.components.

x is the number of components.x is the number of components.


1717


[from ANSI/ANS-3.11 (2005) Exhibit 1][from ANSI/ANS-3.11 (2005) Exhibit 1]6.6. Add bias errors to obtain system accuracy Add bias errors to obtain system accuracy

((SASA).).

SASA = RSS + b = RSS + b11 + b + b22 +. . . + b+. . . + bxx

Where:Where:bb11, b, b22, . . . b, . . . bxx are bias error components. are bias error components.

x is the number of components.x is the number of components.

Note:Note: Repeat as necessary with different bias values to determine extreme values.


1818



7.7. Compare the extreme system accuracy (Compare the extreme system accuracy (SASA) ) values with applicable requirements to values with applicable requirements to evaluate system performance.evaluate system performance.• Table 1 in ANSI/ANS-3.11 (2005) for most cases.Table 1 in ANSI/ANS-3.11 (2005) for most cases.• Table 1 values are both system (channel) Table 1 values are both system (channel)

accuracy and sensor accuracy values.accuracy and sensor accuracy values.o System accuracy encompasses all channel components System accuracy encompasses all channel components

impacting system accuracy (sensors, data processing impacting system accuracy (sensors, data processing equipment, computer, calibrations, etc).equipment, computer, calibrations, etc).

o Sensor accuracy applies to the manufacturer’s Sensor accuracy applies to the manufacturer’s instrument specification.instrument specification.


1919

Example CalculationExample Calculation[Wind Speed – Ultrasonic Sensor][Wind Speed – Ultrasonic Sensor]

1.1. Identify Sources of Error.Identify Sources of Error.• SensorSensor

• Input – Laboratory Standard Input – Laboratory Standard • Input – Transfer StandardInput – Transfer Standard• Input – Test Position (placement in wind Input – Test Position (placement in wind

tunnel)tunnel)• Input – Comparison ApparatusInput – Comparison Apparatus• Output – ToleranceOutput – Tolerance

• Sampling (placement in field)Sampling (placement in field)• Signal Conditioning and Data LoggerSignal Conditioning and Data Logger

• Final rounding of hourly average valueFinal rounding of hourly average value


2020


2.2. Classify Error Type.Classify Error Type.• SensorSensor

• [Input] Sensor placement is [Input] Sensor placement is biasbias• [Input] Other sensor components are [Input] Other sensor components are randomrandom• [Output] Tolerance is [Output] Tolerance is randomrandom

• Sampling (placement) is Sampling (placement) is randomrandom• Final rounding is Final rounding is randomrandom


2121


3.3. Estimate the values of the component errors.Estimate the values of the component errors.[0-100 mph range][0-100 mph range]• Input – Laboratory Standard Input – Laboratory Standard [± 0.20 mph][± 0.20 mph]• Input – Transfer Standard Input – Transfer Standard [± 0.10 mph][± 0.10 mph]• Input – Test Position Input – Test Position [± 0.02 mph][± 0.02 mph]• Input – Comparison Apparatus Input – Comparison Apparatus [± 0.08 mph][± 0.08 mph]• Output – Tolerance Output – Tolerance [± 0.30 mph][± 0.30 mph]

• Sampling Sampling [± 0.04 mph][± 0.04 mph]• Final rounding of hourly average value Final rounding of hourly average value [± 0.05 [± 0.05

mph]mph]


2222


4.4. [Sensor only] Perform time-average [Sensor only] Perform time-average adjustments for each random error adjustments for each random error component.component.

• Only 1 sample is used for calibrationOnly 1 sample is used for calibration

• Adjusted values equal initial valuesAdjusted values equal initial values


2323


5.5. [Sensor only] Calculate "root sum of the [Sensor only] Calculate "root sum of the squares" (RSS) for the adjusted random squares" (RSS) for the adjusted random error components.error components.

RSS = (0.20)RSS = (0.20)22+ (0.10)+ (0.10)22+(0.08)+(0.08)22+ (0.30)+ (0.30)22 mphmph

RSS = RSS = ±0.38 mph±0.38 mph


2424

Example CalculationsExample Calculations[Wind Speed – Ultrasonic Sensor][Wind Speed – Ultrasonic Sensor]

6.6. [Sensor only] Add bias errors to obtain [Sensor only] Add bias errors to obtain system accuracy (system accuracy (SASA).).

SASA = RSS + b = RSS + b

SASA = 0.38 + 0.02 = 0.40 mph = 0.38 + 0.02 = 0.40 mph

SASA = 0.38 – 0.02 = 0.36 mph = 0.38 – 0.02 = 0.36 mph

SASA = = ±± 0.40 mph0.40 mph


2525


4.4. [Field Measurements] Perform time-average [Field Measurements] Perform time-average adjustments for each random error adjustments for each random error component.component.

• Sensor output is random (time-averaged)Sensor output is random (time-averaged)

0.03 = 0.40/( 180) mph0.03 = 0.40/( 180) mph• Sampling is random (time-averaged)Sampling is random (time-averaged)

0.00 = 0.04/( 180) mph0.00 = 0.04/( 180) mph• Rounding is random (1 sample)Rounding is random (1 sample)

0.05 = 0.05/( 1) mph0.05 = 0.05/( 1) mph


2626


5.5. [Field Measurements] Calculate "root sum [Field Measurements] Calculate "root sum of the squares" (RSS) for the adjusted of the squares" (RSS) for the adjusted random error components.random error components.

RSS = (0.03)RSS = (0.03)22+ (0.00)+ (0.00)22 +(0.05) +(0.05)22 mph mph

RSS = RSS = ±0.06 mph±0.06 mph


2727


6.6. [Field Measurements] Add bias errors to [Field Measurements] Add bias errors to obtain system accuracy (obtain system accuracy (SASA).).

• No bias term:No bias term:

SASA = RSS + b = RSS + b

SASA = 0.06 + 0.00 = 0.06 mph = 0.06 + 0.00 = 0.06 mph

SASA = = ±± 0.06 mph0.06 mph


2828


7.7. Compare the extreme system accuracy (Compare the extreme system accuracy (SASA) ) values with applicable requirements to values with applicable requirements to evaluate system performance.evaluate system performance.• Table 1 specifies WS accuracy as:Table 1 specifies WS accuracy as:

0.2 m/s +5% of observation0.2 m/s +5% of observation == 0.45 mph at 0 0.45 mph at 0 mphmph

5.45 mph at 100 5.45 mph at 100 mphmph

• Applies to both sensor and channel accuracies.Applies to both sensor and channel accuracies.• Sensor accuracy of Sensor accuracy of ±± 0.40 mph 0.40 mph and channel and channel

accuracy ofaccuracy of ± ± 0.06 mph0.06 mph both meet specification. both meet specification.

Measurements satisfy ANSI/ANS-3.11 (2005).Measurements satisfy ANSI/ANS-3.11 (2005).


2929

Final CommentsFinal Comments

ANSI/ANS-3.11 (2005) provides specific ANSI/ANS-3.11 (2005) provides specific guidance for calculating meteorological guidance for calculating meteorological measurements accuracy.measurements accuracy.

The ANSI/ANS-3.11 (2005) methodology is The ANSI/ANS-3.11 (2005) methodology is consistent with past practices--prior to consistent with past practices--prior to ANSI/ANS-3.11 (2000), so calculations should be ANSI/ANS-3.11 (2000), so calculations should be comparable with historical information.comparable with historical information.

Hopefully, ANSI/ANS-3.11 (2005) will be Hopefully, ANSI/ANS-3.11 (2005) will be adopted as an adopted as an officialofficial source of guidance source of guidance regarding meteorological measurements regarding meteorological measurements accuracy.accuracy.

Meteorological Measurement Error Analysis based on ANSI/ANS-3.11 (2005) Kenneth G. Wastrack...

Documents

Transcript of Meteorological Measurement Error Analysis based on ANSI/ANS-3.11 (2005) Kenneth G. Wastrack...