PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources -...
-
Upload
rebecca-sherman -
Category
Documents
-
view
215 -
download
1
Transcript of PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources -...
![Page 1: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/1.jpg)
PM Model Performance Goals and Criteria
James W. BoylanGeorgia Department of Natural Resources
- VISTAS
National RPO Modeling MeetingDenver, CO
May 26, 2004
![Page 2: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/2.jpg)
Outline
• Standard Bias and Error Calculations
• Proposed PM Model Performance Goals and Criteria
• Evaluation of Eight PM Modeling Studies Using Proposed Goals and Criteria
• Discussion: Should EPA recommend PM Model Performance Goals and Criteria in PM Modeling Guidance Document?
![Page 3: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/3.jpg)
PM Model Evaluations
• Air Quality Modeling and Ambient Measurements are two different ways to estimate actual ambient concentrations of pollutants in the atmosphere
• Both modeling and measurements have some degree of uncertainty– Measurements should not be considered the
absolute truth– Large differences between monitoring networks
due to sampling and analysis techniques
• Normalized bias and error calculations should not be normalized by observations, but rather the average of the model and the observations.
![Page 4: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/4.jpg)
Performance Metrics Equation
Mean Bias (g/m3)
Mean Error (g/m3)
Mean Normalized Bias (%) (-100% to +) Mean Normalized Error (%) (0% to +)
Normalized Mean Bias (%) (-100% to +) Normalized Mean Error (%) (0% to +)
Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)
N
iom CC
NMB
1
1
N
i mo
om
CC
CC
NMFE
1
2
1
N
i mo
om
CCCC
NMFB
1
2
1
N
io
N
iom
C
CCNME
1
1
N
io
N
iom
C
CCNMB
1
1
N
i o
om
C
CC
NMNE
1
1
N
i o
om
C
CC
NMNB
1
1
N
iom CC
NME
1
1
![Page 5: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/5.jpg)
Performance Metrics
• Mean Normalized Bias and Error – Usually associated with observation-based minimum
threshold• Some components of PM can be very small making it
difficult to set a reasonable minimum threshold value without excluding a majority of the data points
– Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small
• A few data points can dominate the metric
– Overestimations are weighted more than equivalent underestimations
– Assumes observations are absolute truth
![Page 6: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/6.jpg)
Performance Metrics
• Normalized Mean Bias and Error – Biased towards overestimations– Assumes observations are absolute truth
• Mean Fractional Bias and Error– Bounds maximum bias and error – Symmetric: gives equal weight to underestimations
and overestimations– Normalized by average of observation and model
![Page 7: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/7.jpg)
Example Calculations
• Mean Normalized Bias and Error– Most biased and least useful of the three metrics
• Normalized Mean Bias and Error • Mean Fractional Bias and Error
– Least biased and most useful of the three metrics
Model g/m3
)
Obs. g/m3
)
MBg/m3)
NMB (%)
MNB (%)
MFB (%)
ME g/m3
)
NME (%)
MNE (%)
MFE (%)
0.05 1.0 -0.95 -95 -180.95 +0.95 +95 +180.95
1.0 0.05 +0.95 +1900 +180.95 +0.95 +1900 +180.95
1.0 0.01 +0.99 +9900 +196.04 +0.99 +9900 +196.04
0.04 0.05 -0.01 -20 -22.22 +0.01 +20 +22.22
0.5225 0.2775 +0.245 +88.3 +2921.3 +43.5 0.725 261.3 2978.8 145.0
![Page 8: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/8.jpg)
PM Goals and Criteria• Performance Goals: Level of accuracy that
is considered to be close to the best a model can be expected to achieve.
• Performance Criteria: Level of accuracy that is considered to be acceptable for regulatory applications.
• It has been suggested that we need different performance goals and criteria for:– Different Species– Different Seasons– Different Parts of the Country– 20% Haziest and 20% Cleanest Days
• Answer: performance goals and criteria that vary as a function of concentration
![Page 9: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/9.jpg)
PM Modeling Studies Used for Performance Benchmarks
• SAMI (GT)– July 1995 (URM/IMPROVE/variable grid)– July 1991 (URM /IMPROVE /variable grid)– May 1995 (URM /IMPROVE /variable grid)– May 1993 (URM /IMPROVE /variable grid)– March 1993 (URM /IMPROVE /variable grid)– February 1994 (URM /IMPROVE /variable grid)
• VISTAS (UCR/AG/Environ)– July 1999 (CMAQ/IMPROVE/36 km)– July 1999 (CMAQ /IMPROVE/12 km)– July 2001 (CMAQ /IMPROVE/36 km)– July 2001 (CMAQ /IMPROVE/12 km)– January 2002 (CMAQ /IMPROVE/36 km)– January 2002 (CMAQ /IMPROVE/12 km)
![Page 10: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/10.jpg)
PM Modeling Studies Used for Performance Benchmarks
• WRAP 309 (UCR/CEP/Environ)– January 1996 (CMAQ/IMPROVE/36 km)– February 1996 (CMAQ/IMPROVE/36 km)– March 1996 (CMAQ/IMPROVE/36 km)– April 1996 (CMAQ/IMPROVE/36 km)– May 1996 (CMAQ/IMPROVE/36 km)– June 1996 (CMAQ/IMPROVE/36 km)– July 1996 (CMAQ/IMPROVE/36 km)– August 1996 (CMAQ/IMPROVE/36 km)– September 1996 (CMAQ/IMPROVE/36 km)– October 1996 (CMAQ/IMPROVE/36 km)– November 1996 (CMAQ/IMPROVE/36 km)– December 1996 (CMAQ/IMPROVE/36 km)
![Page 11: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/11.jpg)
PM Modeling Studies Used for Performance Benchmarks
• WRAP 308 (UCR/CEP/Environ)– Summer 2002 (CMAQ/IMPROVE/36 km/WRAP)– Summer 2002 (CMAQ/IMPROVE/36 km/US)– Winter 2002 (CMAQ/IMPROVE/36 km/WRAP)– Winter 2002 (CMAQ/IMPROVE/36 km/US)
• EPA (Clear Skies)– Fall 1996 (REMSAD/IMPROVE/36 km)– Spring 1996 (REMSAD/IMPROVE/36 km)– Summer 1996 (REMSAD/IMPROVE/36 km)– Winter 1996 (REMSAD/IMPROVE/36 km)
![Page 12: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/12.jpg)
PM Modeling Studies Used for Performance Benchmarks
• MANE-VU (GT)– July 2001 (CMAQ/IMPROVE/36 km)– July 2001 (CMAQ/SEARCH/36 km)– January 2002 (CMAQ/IMPROVE/36 km)– January 2002 (CMAQ/ SEARCH /36 km)
• Midwest RPO– August 1999 (CMAQ/IMPROVE/36 km)– August 1999 (CAMx/IMPROVE/36 km)– August 1999 (REMSAD/IMPROVE/36 km)– January 2000 (CMAQ/IMPROVE/36 km)– January 2000(CAMx/IMPROVE/36 km)– January 2000(REMSAD/IMPROVE/36 km)
![Page 13: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/13.jpg)
PM Modeling Studies Used for Performance Benchmarks
• EPRI (AER/TVA/Environ)– July 1999 (CMAQ/IMPROVE/32 km)– July 1999 (CMAQ/IMPROVE/8 km)– July 1999 (MADRID/IMPROVE/32 km)– July 1999 (MADRID /IMPROVE/8 km)– July 1999 (CAMx/IMPROVE/32 km)
![Page 14: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/14.jpg)
Mean Fractional Error
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
![Page 15: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/15.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
Mean Fractional Bias
![Page 16: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/16.jpg)
Proposed PM Goals and Criteria
• Based on MFE and MFB calculations• Vary as a function of species
concentrations– Goals: MFE +50% and MFB ±30%– Criteria: MFE +75% and MFB ±60%– Less abundant species should have less stringent
performance goals and criteria
• Continuous functions with the features of:– Asymptotically approaching proposed goals and
criteria when the mean of the observed and modeled concentrations are greater than 2.5 g/m3
– Approaching +200% MFE and ±200% MFB when the mean of the observed and modeled concentrations are extremely small
![Page 17: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/17.jpg)
Proposed Goals and Criteria
• Proposed PM Performance Goals
• Proposed PM Performance Criteria
301703/5.0
)(5.0
mg
CC mo
eMFB
501503/75.0
)(5.0
mg
CC mo
eMFE
751253/75.0
)(5.0
mg
CC mo
eMFE
601403/5.0
)(5.0
mg
CC mo
eMFB
![Page 18: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/18.jpg)
MFE Goals and Criteria
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Goal
Criteria
Zone 1
Zone 3
Zone 2
![Page 19: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/19.jpg)
MFB Goals and Criteria
-200
-150
-100
-50
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
(+) Goal
(-) Goal
(+) Criteria
(-) CriteriaZone 1
Zone 3
Zone 3
Zone 2
Zone 2
![Page 20: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/20.jpg)
Model Performance Zones
• Zone I – Good Model Performance– Level I Diagnostic Evaluation (Minimal)
• Zone II– Average Model Performance– Level II Diagnostic Evaluation (Standard)
• Zone III– Poor Model Performance– Level III Diagnostic Evaluation (Extended)
and Sensitivity Testing
![Page 21: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/21.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
Mean Fractional Error
![Page 22: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/22.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Mean Fractional Bias
![Page 23: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/23.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0 10.0 12.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Goal
Criteria
Sulfate Mean Fractional Error
![Page 24: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/24.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0 10.0 12.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Sulfate Mean Fractional Bias
![Page 25: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/25.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Nitrate
Goal
Criteria
Nitrate Mean Fractional Error
![Page 26: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/26.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Nitrate
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Nitrate Mean Fractional Bias
![Page 27: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/27.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Ammonium
Ammonium Bi
Goal
Criteria
Ammonium Mean Fractional Error
![Page 28: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/28.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Ammonium
Ammonium Bi
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Ammonium Mean Fractional Bias
![Page 29: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/29.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0 6.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Organics
Goal
Criteria
Organics Mean Fractional Error
![Page 30: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/30.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0 5.0 6.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Organics
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Organics Mean Fractional Bias
![Page 31: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/31.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
EC
Goal
Criteria
EC Mean Fractional Error
![Page 32: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/32.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
EC
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
EC Mean Fractional Bias
![Page 33: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/33.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Soils
Goal
Criteria
Soils Mean Fractional Error
![Page 34: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/34.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 1.0 2.0 3.0 4.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Soils
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
Soils Mean Fractional Bias
![Page 35: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/35.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
PM2.5
Goal
Criteria
PM2.5 Mean Fractional Error
![Page 36: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/36.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
PM2.5
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
PM2.5 Mean Fractional Bias
![Page 37: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/37.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0 30.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
PM10
Goal
Criteria
PM10 Mean Fractional Error
![Page 38: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/38.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0 30.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
PM10
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
PM10 Mean Fractional Bias
![Page 39: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/39.jpg)
Combined Modeling Studies
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
CM
Goal
Criteria
CM Mean Fractional Error
![Page 40: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/40.jpg)
Combined Modeling Studies
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
CM Mean Fractional Bias
![Page 41: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/41.jpg)
SAMI: May 1995
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
SAMI Mean Fractional Error
![Page 42: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/42.jpg)
SAMI: May 1995
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
SAMI Mean Fractional Bias
![Page 43: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/43.jpg)
SAMI: 6 Episodes
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0 30.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
SAMI Mean Fractional Error
![Page 44: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/44.jpg)
SAMI: 6 Episodes
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 5.0 10.0 15.0 20.0 25.0 30.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Ammonium Bi
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
SAMI Mean Fractional Bias
![Page 45: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/45.jpg)
EPA Mean Fractional Error
EPA: 4 Seasons (1996)
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Organics
EC
Soils
PM2.5
Goal
Criteria
![Page 46: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/46.jpg)
EPA Mean Fractional Bias
EPA: 4 Seasons (1996)
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Organics
EC
Soils
PM2.5
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
![Page 47: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/47.jpg)
VISTAS: 3 Episodes (2 Grids)
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
VISTAS Mean Fractional Error
![Page 48: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/48.jpg)
VISTAS: 3 Episodes (2 Grids)
-200
-150
-100
-50
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
VISTAS Mean Fractional Bias
![Page 49: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/49.jpg)
MANE-VU Mean Fractional Error
MANE-VU: 2 Episodes (IMPROVE + SEARCH)
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
PM2.5
Goal
Criteria
![Page 50: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/50.jpg)
MANE-VU Mean Fractional Bias
MANE-VU: 2 Episodes (IMPROVE + SEARCH)
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 4.0 8.0 12.0 16.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
PM2.5
(+) Bias Goal
(-) Bias Goal
(+) Criteria
(-) Criteria
![Page 51: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/51.jpg)
WRAP (309) Mean Fractional Error
WRAP: 12 Months (1996)
0
50
100
150
200
0.0 2.0 4.0 6.0 8.0 10.0 12.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
![Page 52: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/52.jpg)
WRAP (309) Mean Fractional Bias
WRAP: 12 Months (1996)
-200
-150
-100
-50
0
50
100
150
200
0.0 2.0 4.0 6.0 8.0 10.0 12.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
![Page 53: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/53.jpg)
WRAP (308) Mean Fractional Error
WRAP: 2 Seasons (WRAP & US Domains)
0.00
50.00
100.00
150.00
200.00
0.0 3.0 6.0 9.0 12.0 15.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
![Page 54: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/54.jpg)
WRAP (308) Mean Fractional Bias
WRAP: 2 Seasons (WRAP & US Domains)
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 3.0 6.0 9.0 12.0 15.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
![Page 55: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/55.jpg)
MRPO Mean Fractional Error
MRPO: 2 Episodes (CMAQ/CAMx/REMSAD)
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0 10.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
CM
Goal
Criteria
![Page 56: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/56.jpg)
MRPO Mean Fractional Bias
MRPO: 2 Episodes (CMAQ/CAMx/REMSAD)
-200.00
-150.00
-100.00
-50.00
0.00
50.00
100.00
150.00
200.00
0.0 2.0 4.0 6.0 8.0 10.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
CM
(+) Bias Goal
(-) Bias Goal
(+) Criteria
(-) Criteria
![Page 57: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/57.jpg)
EPRI Mean Fractional Error
EPRI: July 1999 (3 Models)
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
Goal
Criteria
![Page 58: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/58.jpg)
EPRI Mean Fractional Bias
EPRI: July 1999 (3 Models)
-200
-150
-100
-500
50
100
150
200
0.0 5.0 10.0 15.0 20.0
Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Sulfate
Nitrate
Ammonium
Organics
EC
Soils
PM2.5
PM10
CM
(+) Goal
(-) Goal
(+) Criteria
(-) Criteria
![Page 59: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/59.jpg)
Concluding Remarks• Performance evaluation should be done on
an episode-by-episode basis or on a month-by-month basis for annual modeling.
• Recommended performance goals and criteria should be used to help identify areas that can be improved upon in future modeling – Failure to meet proposed performance criteria
should not necessarily prohibit the modeling from being used for regulatory purposes • Need to perform extended diagnostic evaluation
and sensitivity tests to address poor performance
![Page 60: PM Model Performance Goals and Criteria James W. Boylan Georgia Department of Natural Resources - VISTAS National RPO Modeling Meeting Denver, CO May 26,](https://reader036.fdocuments.us/reader036/viewer/2022062801/56649e425503460f94b34abd/html5/thumbnails/60.jpg)
Concluding Remarks (cont.)• As models mature, performance goals can be
made more restrictive by simply adjusting the coefficients in the performance goals and criteria equations (MFE and MFB)
• Performance goals and criteria for measurements with longer averaging times (e.g., weekly) should be more restrictive and those with a shorter averaging times (e.g., hourly) should be less restrictive.
• Discussion Questions:– Should EPA recommend PM model performance goals
and criteria in PM Modeling Guidance Document?– Is there a need for performance goals for gaseous
precursors and/or wet deposition species?