Update on the CCA Groundwater Simulation Framework: the BOCCA

Experience

Bruce Palmer, Yilin Fang, Vidhya Gurumoorthi, James Fort, Tim Scheibe

Computational Sciences and Mathematics DivisionPacific Northwest National Laboratory

Richland, WA 99352

Progress on Smoothed Particle Hydrodynamics (SPH) Framework

Implementation of component to model transport of a passive contaminantRefinement of existing components to enhance scalability to an modularityPorted to Cray XT4 platform (Franklin)Large runs of 7-14 million SPH particlesIncorporated SPH applications into SALSSA workflow frameworkMoved entire SPH framework into BOCCA

3

SPH Framework

4

Progress on Subsurface Transport Over Multiple Phases (STOMP) Framework

Developed an interface for a grid component using an unstructured grid syntaxImplemented grid component using GA for a structured gridModified STOMP kernel to use grid component

STOMP Framework

STOMP component Grid component

Input

Grid PortGrid Port

PhysicsSolvers

Time Integrator

Output

GA GA

Data Mapping

GA

MPI

Chemistry

7

STOMP/CCA Simulations

Simulation of transport of contaminant through an array of monitoring wells at the Hanford IFRC site, using the component version of STOMP

Current Status

A complete framework for performing Smoothed Particle Hydrodynamics simulations exists

Validation of SPH hydrodynamic flows in 3D is underway using the SALSSA workflow environmentInitial simulations of tracer transport in 3D porous media are in progressPorted to Chinook and Franklin computersDevelopment of additional chemistry modules will be starting soon

The underlying grid in STOMP has been separated out as a standalone component

Grid component supports an unstructured syntaxAll references to a structured grid in STOMP have been eliminated except the outputCollaboration underway with ITAPS project to develop an unstructured grid componentWork is underway to split out solvers and output into separate components

Platforms Used

ChinookRunning SPH with HPMPI and OpenMPI.

MVAPICH seems to have problems with dangling ports. This is a problem for SPH but not for STOMP

FranklinRunning with static builds (kudos to Jim Kohl)

Using the BOCCA Developer Environment

./$SPH_BOCCA_HOME create_sphbocca env_gcc_3.4.csh env_gwbocca.csh sph-orig-impls/ sph-bocca/

create_sphbocca#==========================================# Creating ports:#==========================================bocca create port rdcomp.ReadInputPort --import-sidl=$WORKDIR/sph-orig-impls/ports/sidl/rdcomp.ReadInputPort.sidl#==========================================# Creating Components:#==========================================bocca create component drivers.SPHDriver --import-sidl=$WORKDIR/sph-orig-impls/components/sidl/drivers.SPHDriver.sidl --import-impl="drivers.SPHDriver:$WORKDIR/sph-orig-impls/components/drivers.SPHDriver“--go=GoPort --uses=rdcomp.ReadInputPort:ReadInputPort--uses=rdprtcl.ReadPrtclPort:ReadPrtclPort--uses=wrtprtcl.ExportXYZPort:ExportXYZPort--uses=lgrng.LagrangePort:LagrangePort--uses=stepper.StepPort:StepPort--uses=dtmgr.DataPort:DataPort --language=f90

create_sphbocca (cont)#===============================================# Copy the top level make files#=============================================== cd $WORKDIR/sph-orig-implscp make.vars.user $SPH_HOME/ #===============================================# Copy files in individual component directories#=============================================== cd $WORKDIR/sph-orig-impls/components/drivers.SPHDriver/cp make.vars.user common.fh $SPH_HOME/components/drivers.SPHDriver/

Observations on BOCCA

Having a repository for files (sph-orig-impls) and a separate working directory (sph-bocca) is confusing and error proneWe could REALLY use a “duplicate” capability inside BOCCA. This would take an existing component, make a copy of it and give it a new name. Not having this function is a major impediment to getting new developers on board.The contractor build was not producing compilable F90 files after importing. Don’t know what the status of this bug is.

Observations on BOCCA (cont)

The only way to build components is to go into the components directory and hit “make”. You cannot go into the directory for individual components and build them one by one nor can you go into the component directory and go “make individual.component” and get a single component to build. Having the make system compile every component, or at least check every component for changes, takes a long time if you have many components

We could use the ability to create static builds as a standard feature

We are having trouble changing the home machine that the sph-origs-impls directory is located on using the contractor build

Miscellaneous Issues

It would be useful to be able to call component methods from inside the component using the SIDL interfaceNeed mechanisms for provenance capture

Version of components

Version of CCA and configuration information

Components utilized in application and their linkages

Acknowledgements

Funding for this project was provided by DOE’s Office of Advanced Scientific Computing Research under the Scientic Discovery through Advanced Computing programComputer time on Chinook supplied by the Environmental and Molecular Sciences Laboratory at Pacific Northwest National Laboratory through its Science Theme programComputer time on Franklin supplied by the National Energy Research Scientific Computing Center