Development and Application ofParallel Plume-in-Grid Models

Prakash Karamchandani, Krish Vijayaraghavan, Shu-Yun Chen and Christian Seigneur

AER, San Ramon, CA

7th Annual CMAS ConferenceOctober 6–8, 2008

Chapel Hill, NC

Limitations of Traditional Grid Modeling

• Horizontal resolution of a few kilometers to tens of kilometers cannot resolve sub-grid scale effects such as transport and chemistry of point source emissions

• Artificial dilution of stack emissions

• Unrealistic near-stack plume concentrations

• Incorrect representation of plume chemistry

• Incorrect representation of plume transport

Plume Size vs Grid Size (from Godowitch, 2004)

Plume Chemistry & Relevance to Ozone and PM Modeling

Early Plume Dispersion

NO/NO2/O3 chemistry1

2Mid-range Plume Dispersion

Reduced VOC/NOx/O3 chemistry — acid formation from OH and NO3/N2O5 chemistry

Long-range Plume Dispersion

3

Full VOC/NOx/O3 chemistry — acid and O3 formation

Plume-in-Grid (PiG) Modeling

• Plume-in-Grid (PiG) approach provides a sub-grid scale representation of stack plumes

• Addresses inability of 3-D grid models to correctly simulate atmospheric fate of stack emissions

• Approach consists of embedding a reactive puff model within the grid model

• The initial transport and chemistry of point source emissions are treated with the puff model

• When puff sizes are comparable to grid model resolution, the puffs are merged with the grid and subsequent calculations are done with the grid model

AMSTERDAM

• Advanced Modeling System for Transport, Emissions, Reactions & Deposition of Atmospheric Matter

• Suite of models based on CMAQ v 4.6, October 2006 release and with alternate options

• Options:

– MADRID treatment for PM: Model of Aerosol Dynamics, Reaction, Ionization and Dissolution

– APT: Advanced Plume Treatment with embedded plume model SCICHEM (state-of-the science treatment of stack plumes at the sub-grid scale)

• APT can be used with either the MADRID treatment for PM or the CMAQ treatment (currently available for AERO3 option)

AMSTERDAM Components

CMAQ v. 4.6

MADRID PM TreatmentCMAQ-MADRID

SCICHEM-AERO3PM Treatment based on EPA CMAQ

SCICHEM-MADRIDPM Treatment based on CMAQ-MADRID

CMAQ-MADRID-APTCMAQ-AERO3-APT

SCICHEM

• Three-dimensional puff-based model

• Second-order closure approach for plume dispersion

• Puff splitting and merging

• Treatment of plume overlaps

• Optional treatment of building downwash

• Optional treatment of turbulent chemistry

• PM, gas-phase and aqueous-phase chemistry treatments consistent with host model

Model Applications/Evaluations

• North-eastern U.S. with two nested grid (12 km & 4 km) domains (NARSTO)

– 5 day episode (O3 only)– 30 point sources simulated explicitly– PiG model about 1.5 times slower than grid

model

• Central California (4 km resolution)

– 3 day episode (O3 only)– 10 point sources simulated explicitly– PiG model about 1.2 times slower than grid

model

Model Applications/Evaluations (continued)

• Eastern U.S. (VISTAS 12 km domain)

– 2 month simulations for O3 and PM– 14 point sources simulated explicitly– PiG model about 1.25 times slower than grid

model

• Southeastern U.S. (ALGA 12 km domain)– Annual simulations

– O3, PM, Hg and nitrogen deposition– 40 point sources simulated explicitly– PiG model about 1.8 times slower than grid model

Typical Results

• The PiG treatment has a strong effect on model predictions of surface O3 titration (near large NOx point sources), as well as O3, sulfate and nitrate formation downwind of large NOx and SO2 point sources

• A purely gridded approach typically overestimates PM production downwind of large NOx point sources because it overestimates SO2 to sulfate and NOx to nitrate conversion rates near the stack

• Overall model performance statistics are almost identical between the grid-only and PiG treatments, but observed plume events are better captured in the PiG approach than in the purely gridded approach

Power-Plant Contributions to 24-hr Average Sulfate Concentrations

CMAQ-MADRID CMAQ-MADRID-APT

Change in Power-Plant Contributions (%) to PM2.5 Sulfate Concentrations When a PiG Approach is

Used

Conversion of Power Plant SO2 Emissions

Domain-wide mass-budget analysis performed for SO2 and sulfate attributable to power plant emissions

Sulfate to Total Sulfur Ratios (%)

Emissions CMAQ-MADRID CMAQ-MADRID-APT

January 2.25 17.8 15.6

July 2.35 75.5 67.4

Approximate SO2 Conversion (%)

CMAQ-MADRID CMAQ-MADRID-APT Change

January 15.9 13.7 -14%

July 74.9 66.6 -11%

Spatial Distribution of Total Nitrogen Deposition

Change in annual dry + wet deposition flux due to power plant NOx controls

CMAQ-MADRID CMAQ-MADRID-APT

APT: Less oxidation of NOx to HNO3 => Less dry deposition near the plant

Maximum reduction in deposition flux

0.85 kg/ha 0.42 kg/ha

PiG Modeling Constraints

• Can be computationally expensive if a large number of point sources are treated with the puff model – computational requirements increase by a factor of two to three for 50 to 100 sources

• Point sources have to be selected carefully to limit the number of sources treated

• To obtain results in a reasonable amount of time, annual simulations are usually conducted by dividing the calendar year into quarters and simulating each quarter on different processors or machines

• Parallel version of code can address these constraints

Parallelization of APT

• Parallelization of stand-alone version of puff model (SCICHEM)

• Adaptation of parallel stand-alone SCICHEM to parallel plume-in-grid version

• Development of appropriate parallel interfaces between parallel CMAQ-MADRID/CMAQ-AERO3 and parallel plume-in-grid version of SCICHEM

Parallelization of Host Models

• CMAQ and CMAQ-MADRID

– Based on Message Passing Interface (MPI)

– Horizontal domain decomposition

Parallelization of SCICHEM

• SCICHEM

– Puff decomposition for chemistry step

– Needs access to entire 3-D grid

– Uses MPI

Parallel Interface

• Gathers 3-D sub-domain concentrations from the various processors to create a global 3-D concentration array

• Provides global 3-D concentration array to SCICHEM

• After the SCICHEM time step, the modified global 3-D concentration array is scattered to the 3-D sub-domain concentration arrays

Model Interaction Diagram

Domain, grid informationGeophysical dataMeteorological dataDeposition velocities

ParallelCMAQ-MADRID/CMAQ-AERO3

ParallelSCICHEM

Emissions,IC/BC

Outputconcentrations,

Deposition,Puff diagnostics

Outputpuff

information

Pointsource

emissions

Mergepuffs

I/OAPI

I/OAPI

I/OAPI

I/OAPI

StandardSCICHEM

output

Puff diagnostics

ControlFile

ParallelInterface

Current Status

• Development of parallel version of AMSTERDAM completed in 2008

• On a 4-processor machine, the parallel version is about 2.5 times faster than the single-processor version

• On-going project to apply the model to the central and eastern United States at 12 km resolution and to evaluate it with available data

– Over 150 point sources explicitly treated with APT

– Annual actual and typical simulations for 2002

– Future year emission scenarios

– Other emission sensitivity scenarios

Ongoing Application of Parallel PiG Model

• 12 km grid resolution

• 243 x 246 x 19 grid cells

• Over 150 PiG sources

Acknowledgements

• Funding for model development, including parallelization, and model application:– EPRI

• Collaboration on parallelization:– L-3 Communications Titan Group

• Parallelization insights:– Dr. David Wong, EPA