Support for Global Cloud Resolving Model Simulations

VACET All-Hands MeetingIEEE Vis 2008

Support for Global Cloud Resolving Model Simulations

VACET All-Hands MeetingIEEE Vis 2008

PRABHAT

SciDAC/INCITE PI: Dave Randall (CSU) 2008 INCITE award at NERSC

Multi-institution collaboration CSU researchers developing the climate

simulation PNNL researchers developing the Data

API VACET researchers helping w/ Data

model, Parallel IO and Vis

Global atmospheric circulation model non-hydrostatic dynamical core cloud microphysics, turbulence, radiation coupled to land-surface model

3km grid-cell spacing Planned runs on 20K+ franklin cores Planned 1:1 simulated:wall_clock time Data throughput

~1TB/hr 2GB/s aggregate IO performance requirement ~500M cells*100 layers, 50 variables

Identify a data model for GCRM code Enable simulation code to do efficient

parallel IO Visualization and Analysis on

extremely large icosahedral meshes

1st Phase: NetCDF-3 metadata conventions Icosahedral mesh variables/layout

▪ cell/corner/edge/face centered ▪ satisfy VisIt plugin requirements

2nd Phase NetCDF-4 (HDF5)

Need sustained 2+GB/s collective IO to single shared file

Current franklin performance ~0.3-1.2 GB/s (for single file)

Large number of IOR tests examining factors IO patterns lustre striping blocking/transfer sizes #IO nodes, #OSTs

Issues identified with Cray's mpich implementation Not sufficiently aware of OST striping

Modifying HDF5 to align/pad to OST stripes

Current best strategy is to use 2-phase IO (use aggregator nodes) #IO nodes = #OSTs Match file transfer size to stripe width Simulate ‘file-per-process’ IO pattern

VisIt plugin developed to load icosahedral NetCDF mesh data

Preliminary custom visualizations rendered

Online tutorials and startup instructions published at:

http://vis.lbl.gov/~prabhat/Incite19/

Further work w/ Cray on optimizing mpi-io

Finalize data model cell/corner/edge/face centered data validate NetCDF-4

VisIt plugin enhancements parallelization incorporate morton-ordering

Colorado State Univ Dave Randall, Ross Heikes

PNNL Karen Schuchardt, Bruce Palmer,

Annette Koonz LBNL

Janet Jacobsen, Wes Bethel Mark Howison Katie Antypas, John Shalf, Shane Canon

VisIt team

VisIt Update (Hank’s part)

Project is doing very well There is an upcoming slide on

partnering Much more like an open source

project SVN repo at LBL through SciDAC

Outreach Center Mailing lists at ORNL ~200 people on users lists Visitusers.org Wiki Wiki for developers and users Tutorials (at SciDAC conf, Snowbird,

PPPL)

Likely upcoming features in the next year

SciDAC stuff: Poincare plots Integration of FastBit VisTrails (?)

Non-SciDAC stuff: Qt4 Parallel Python Hopefully more for in-situ, multi-res, and

time-varying data. + a lot more…

VisIt has a rapidly growing developer community.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.ASC / LLNLEric Brugger (PL)Kathleen Bonnell

Hank ChildsCyrus Harrison

Mark MillerBrad Whitlock

SciDAC/LBLGunther Weber

Prabhat

SciDAC/LLNLHank Childs

Brad Whitlock

SciDAC/ORNLSean Ahern

Jeremy MeredithDavid Pugmire

SciDAC/UCDavisChristoph GarthOliver RuebelEduard Dienes

SciDAC/UUtahJosh Stratton

GNEP/LLNLDavid BremerHank Childs

AWEMatt Wheeler

Paul Selby

CEAFabien Vivodtzev

ORNL InstitutionalSean Ahern

Jamison DanielJeremy MeredithDavid Pugmire

LBL InstitutionalGunther Weber

PrabhatINL Institutional

Jim Galbraith

ANL InstitutionalRandy Hudson

LDRD/LLNLDaniel LaneyHank Childs

Oil $ to LLNLEric Brugger

UCDavis: ResearchJohn Anderson

UNH: ResearchTom Fogal

ITT (Industry)Matt Fago

• + more partnerships coming soon (NSF?)• + contributions from people not in the repository