module 6 1

Module 6: Uncertainty

Don’t just calculate—first think about sources of error, and don’t

double-count errors

module 6 2

Two Sources of ErrorTwo Sources of Error

Sampling – How/where/when/who makes the

measurementsPopulation

– Actual variability in what you are measuring

module 6 3

Measurement Measurement error specific to…error specific to…

– Operator– Instrument– Lab– Procedure– Standard– Time (day of week, year, season)– Measurement level (harder to measure

at low concentrations)

module 6 4

Population ErrorPopulation Error

Ideally, estimate some aspect of homogenous “clump” of air, water, people

If population is totally homogenous, only one measurement is necessary

The more variability in the population, the more measurements you need

module 6 5

Minimizing the effect of Minimizing the effect of population uncertaintypopulation uncertainty

Careful sampling plan, designed to include measurements from all “over” the distribution

Sampling plan to measure smallest “homogenous” parts of environment as possible

Careful adherence to identical procedures

module 6 6

QC measurements QC measurements designed to…designed to…

Identify where errors occur Quantify errors (difference from “reality”) Save $ by improving program Produce estimates of how certain your

conclusions can be… …therefore allowing decisions based on

what you really “can” know

module 6 7

PM QC ResultsPM QC Results

Collocated Flow rate checks done with routine

standard Flow rate checks done with an external

standard PEP intercomparisons of external

instrument and lab What to do with each?

module 6 8

How EPA Summarizes QCHow EPA Summarizes QC

First, estimate uncertainty for each site Use collocated results to calculate

confidence interval for precision (CV) Start with RPD (diff/mean) Always use same pair and order

module 6 9

See P&B DASC with PM DataSee P&B DASC with PM Data

module 6 10

PM2.5 Precision Estimate (40 PM2.5 Precision Estimate (40 CFR 58 App. A eq’n 11)CFR 58 App. A eq’n 11)

module 6 11

90% confidence limit for 90% confidence limit for precision = 7.7%precision = 7.7%

Average over quarter = x microg/m3 +- 8% (with 90% confidence, from precision error alone)

Can use this as part of overall uncertainty estimate

Combine with bias estimates from flow rate and PEP audits

module 6 12

To estimate bias…To estimate bias…

Use PEP audit results, if available Use any comparisons that are independent

as possible Use DASC PM2.5 Bias (Current PEP) tab Calculates upper and lower 90%

confidence intervals

module 6 13

UCL and LCLUCL and LCL(upper and lower confidence (upper and lower confidence

level)level)

module 6 14

What does this mean?What does this mean?

UCL is ~ 10% LCL is ~ -10% Uncertainty of bias

about 10% Average bias of 7%

could really be 7.7, or about 8%

module 6 15

Combining precision and bias?Combining precision and bias?

For rough estimate: square root of sum of squares

Start with d=diff/mean for all QC checks

Calc STDEV of each set of d’s Square each STDEV Add squares Take square root, see if it makes sense!

module 6 16

Precision for qrtr 1 of 2003Precision for qrtr 1 of 2003

Collocated pairs, so

PRECISION estimate (if A

is not consistently higher/lower

than B)

module 6 17

BiasBias for 2003, based on for 2003, based on independent checksindependent checks

module 6 18

Square Root of Square Root of Sum of SquaresSum of Squares

module 6 19

Presenting UncertaintyPresenting Uncertainty

Use error bars or upper, lower lines in graph

module 6 20

Uncertainty for Gaseous Uncertainty for Gaseous MethodsMethods

Simpler than PM RPD between known and measured for

automated and manual checks Estimates validated with results of

independent audits QC checks produce estimates that include

both precision and bias error

module 6 21

Summarizing Uncertainty:Summarizing Uncertainty:COMMON SENSE first!COMMON SENSE first!

“Highest” estimate or worst-case calculation from results of independent audits (that encompass both precision and bias)

Uncertainty estimate should encompass (already include the error from) your internal assessments, so do not double-count results