RegCM-ROMS Tutorial: Coupling...
Transcript of RegCM-ROMS Tutorial: Coupling...
RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu ICTP (International Center for Theoretical Physics) Earth System Physics Section
Outline • Information about coupling and ESMF
• Installation and setup
• Coupled model (RegCM+ROMS) • Design, Limitations and Pros/Cons • Installation • Configuration • Running and Restarting • Debugging
• Future Plans
- Outline
- Information about ESMF (1/2)
• Why ESMF?
Different ways for online coupling • Creating subroutinized sub-components (like sub-grid
parameterizations or current version of the CLM coupling) • Most of the existing code use this approach. It is not generic
solution and not flexible. Think about changing version of CLM in current version of RegCM model.
• Using existing communication protocols such as (MPI, InterComm) • Implementation is not easy. It is not generic and efficient solution.
• Using coupling libraries or frameworks (i.e. ESMF, MCT) • Flexible, efficient, scalable and generic solution. It needs time to
convert existing code into components. • Using a coupler (OASIS, MPCCI etc.)
• Flexible, efficient, scalable and generic solution. It also needs time to implement into existing codes.
Increasing needs to such tools like ESMF, MCT and OASIS !!! The current models evolve to a multi-component Earth System Models (ESM).
- Information about ESMF (2/2)
• Basics
ESMF • The complexity of the regular tasks to create a coupled modeling
system (synchronization of the model components, exchanging coupling fields among modeling components, interpolation between deferent grids etc.) can be simplified.
• ESMF can perform different interpolation types (bilinear, conservative etc.) over exchange fields (i.e. sea surface temperature, heat and momentum fluxes) in a parallel way without using external grid weight file generated by SCRIP.
• Supports both C/C++ or Fortran programming language. • Highly optimized and portable (the library is tested regularly across
different computing platforms, compilers (and also versions), mpi implementations etc.) library
• Scalable beyond few thousand processor (if you can feed the CPUs) • Supports different technologies such as parallel IO, XML, web services
(like OpenMI).
- Installation of ESMF (1/3)
• Steps to install ESMF library …
Get ESMF library source code • Latest public release 5.2.0rp2
http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_5_2_0rp2/reg/ESMF_Framework_Reg.html
• Fill simple form and download the source code • Installation information for different platforms can be found in
http://www.earthsystemmodeling.org/download/platforms/ Install third-party libraries • NetCDF • Parallel NetCDF (required for coupled model debugging)
http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/Download • Xerces XML Parser (required for exporting component, field level
metadata in XML format) http://xerces.apache.org/xerces-c/ Install ESMF and Setup Environment Variables • Third party libraries must be installed first !!! • Same compiler and MPI implementation must be used !!!
- Installation of ESMF (2/3)
• Example installation environment on ARGO
third-party libraries
ESMF library related definitions
Install both optimized and debug versions
Used for RegCM+ROMS
installation
- Installation of ESMF (3/3)
• Commands
To install optimized version • mkdir ~/progs; cd progs • tar -zxvf esmf_5_2_0rp2_src.tar.gz; mv esmf esmf_5_2_0rp2 • cd esmf_5_2_0rp2 • make info (it shows the configuration. important for solving installation
problems) • make >& make_O.log • make install To install debug version • make clean • export ESMF_BOPT=g (for tcsh setenv ESMF_BOPT g) • make >&make_g.log • make install Notes To test the installation, the ESMF test suite can be installed
- Design (1/5)
• General structure (1)
Ocean / Lake
Psurf, T2m, Q2m, Rain Swrad, Lwrad, U10m, V10m SST Hice
• ATM àSurface atmospheric conditions, heat fluxes and rain. The atmospheric variables is used to calculate fluxes in the ROMS.
• OCN à SST and Ice Thickness (if the ice model exist). The SST is used to calculate sensible heat flux, evaporation, ground temperature etc. Ice Thickness is used to change land type and updates the evaporation flux over ice
• The interaction or coupling time step (in second) can be defined using RegCM namelist file
- Design (2/5)
• General structure (2)
Init
Run
0 1 Number of PETs: N
Number of PETs: N
Total Numberof PETs: 2N
N-1
N N+1 2N-1
Init
Run
Finalize
Finalize
Export States
Export States
12
11
15
16 17
18112
111
Import States
Import States
Backward
Forward
Initialize
Run
Finalize
Initialize
Run
Finalize13
14
RegCMAtmosphere Model
ROMSOcean Model
110
19
COUPLER: Online Interpolation,Vector Rotation,
Unit Conversion
MPI Comm. World
COUPLER
ATM to OCN
OCN to ATM
- Design (3/5)
• General structure (2)
MTX ZENG LAKE UPDCPLFWD
T = 0
T = Tcpl
BATS
Time Axis
send oceanvariables
(SST, ICE etc.) to RegCM
OCNCPLBCK
Time Axis
ATM
updateforcing
variablesin ROMS
updateforcing
variablesin ROMS
update SST and ICE
in RegCM
- Design (4/5)
• Design details (1)
Supports • It can be used together with Zeng Ocean and one-dimensional Lake
Model. • ESMF online re-gridding capability and conservative type interpolation
is used for flux variables (bilinear type interpolation for others) • Works with ROMS 3.5 tagged version and also ROMS-Ice branch. • Different number of processor can be assigned to gridded components
(coupler component uses all processors) Limitations • It only interacts with BATS (no support CLM or BAND) • The ocean model still needs forcing files (RegCM model updates them
when coupled mode is activated). • Current version only supports Gregorian calendar • Ocean model must be configured to write daily restart file and this
adds extra overhead into coupled model run. • There is a bug in ROMS (PERFECT_RESTART+LcycleRST == F)
- Design (5/5)
• Design details (2) Pros • The version of ocean component (ROMS) can be changed easily by
porting patch (it may also requires minor change in the RegCM component code). This is tested before is tested before, 3.4 à 3.5 à 3.5_ice.
• The coupled model provides better representation of oceans, lakes and inland waters (If the ROMS model is tuned well).
• New components (like CLM) can be added by modifying coupler interface.
Cons • It needs experience also in ocean modeling, which the configuration of
realistic application is not straightforward (create ocean model grid, initial, boundary and forcing files).
• Adds additional computational and messaging overhead into the model • User needs to tune the ocean model like atmospheric model. This
must be done in standalone mode first !!!
- Installation (1/3)
• Applying patch to ROMS and install Patch includes following modifications • Updates forcing variables with RegCM provided ones • Enables to use two namelist (*.in for RegCM and ROMS) file in the job
submission step • Fixes caldate subroutine in (ROMS/Utils/get_date.F90). • The changes are activated by new pre-processor flag (#define
REGCM_COUPLING) which must be set in *.h header file. Without this option ./build.sh compiles original version of ROMS.
Applying patch Creating and installing new ROMS case (before compiling RegCM !!!)
- Installation (2/3)
• Configure and install RegCM
options for coupling
./configure --help
Example command to configure RegCM in coupled model mode
- Installation (3/3)
• Additional information about installation
• User may need to install newer version of autoconf (>2.68) and automake (>1.11.1) tools to run bootstrap.sh script.
• Script --with-esmf option looks for ESMF_LIB environment variable. • The ROMS Build path must be given as an absolute path into
configure script using --with-roms option !!! • After ./configure step, the RegCM code can be compiled with
conventional way using make install command. • The ROMS ice branch is automatically identified by configure script
and required code is added into RegCM (it just check the seaice.o file in the ROMS Build directory).
• The patched version of ROMS (with REGCM_COUPLING option) and also ESMF library must be installed before RegCM installation !!!
• The ROMS does not need ESMF library. So, do not install it with existing ESMF_COUPLING and/or ESMF_LIB header definitions. It is also gives error with these options because it is not fully implemented yet.
- Configuration (1/2)
• New parameters (*.in)
• physicsparam • iocncpl
• 0 – no coupling (default) • 1 – activates ROMS coupling
• cplparam • cpldt – controls the coupling time step (in second, 21600) • cpldbglevel – specify the debug level for coupled model. (1)
• 0 – no debugging • 1 – only informative print statement (array and pointer
dimensions etc.). • 2 – previous option + write grid information in VTK (The
Visualization Toolkit) format which can be visualized by using such tools like Visit.
• 3 – previous option + write exchange fields in NetCDF format • 4 – previous option + write exchange fields in ASCII format
- Configuration (2/2)
• New parameters (regcm.rc)
• The new namelist file (regcm.rc) is created to pass parameters to the coupled model (sample file is in the Testing/CPL_FILES)
• In the current implementation, the new parameter file is used just to set number of processor for each gridded (M, N) component.
• If the file does not exist, then the total number of processor (M+N) divided to two and assigned to each gridded component (M, N). In this case N+M can be divided evenly or not.
• We plan to move all the parameters (coupling time step, debug level etc.) to the new parameter file (regcm.rc).
• The example for the PETs parameter: • PETs: 32 28 • In this case atmospheric model (RegCM) will use 32 (M)
processor and ocean model (ROMS) will use 28 (N) and the coupler component uses all the available CPUs (32+28 = 60) to perform interpolation between grids.
• The M must be evenly divided to the jx and placed under case directory like regcm.in
- Running (1/4)
• Job submission under PBS regcm.job
path of ESMF shared libraries
definition of namelist files …
• The same ESMF path must be defined with ./configuration step. • The ESMF must be compiled with same version of the compiler and
ESMF implementation • The total number of processor must be consistent. In the example script
• regcm.rc (M+N = 32+28 = 60 = 12*5 in regcm.job) • cas.in (NtileI*NtileJ = 4*7 = N = 28)
- Running (2/4)
• Restarting (1)
Requirements • User must configure ROMS to write restart data each day without
cycling. To do that set NRST to 86400/DT and LcycleRST to F. • The date of the records in the restart file might be checked by CDO.
cdo sinfo ocean_rst.nc. Steps in ROMS side • In namelist file, set NRREC to -1 (indicates that the run is restart) • Split desired date from ocean_rst.nc file (this date must be same with
the date of latest RegCM restart file - SAV). • Point created file as a restart file in namelist file. To do that set
ININAME as output/ocean_rst_splited.nc (or point the correct directory).
Steps in RegCM side • Set ifrest as .true. • Set mdate1 as a date of the latest SAV file • Run RegCM model
- Running (3/4)
• Restarting (2) Notes • Restating of the coupled model can be little bit confusing • Both model must have restart information in same date • RegCM normally creates restart file at the beginning of the each month
but you can not define monthly time interval for ROMS restart files. As a result we choose to write ROMS restart each day. By this way, ROMS will have restart information at the beginning of the month.
• This can be potential performance bottleneck and waste of disk usage • In theory, it is possible to set date for RegCM model to write
intermediate weekly restart file (TMPSAV) by using savfrq parameter. In this case, ROMS model can be configured to write restart file with the same interval (i.e. weekly). It is not tested !!! There could be time lag between restart files created by RegCM and ROMS.
• To simplify the current restart mechanism, we created a restart script (Tools/CPL_FILES/restart.sh) that automatize the steps in the previous slide.
- Running (4/4)
• Restarting (3) restart.sh
definition of RegCM and ROMS namelist files …
Required modules (CDO and NCO).
• User may need to modify this script. • It uses Module (user environment manager) to setup paths for NCO
and CDO tools. It is written in bash shell and uses awk and sed. • User need to check the RegCM and ROMS namelist files after using
this script. The backup is created for namelist and restart file.
- Debugging (1/2)
• VTK (The Visualization Toolkit) and Visit
• Used to see the grid representation of each gridded component
• Created for each grid stencil • Files with *.vtk extension for each
processor can be visualize in Visit • https://wci.llnl.gov/codes/visit/
- Debugging (2/2)
• Notes
If you have problem with the coupled model, check following before doing anything … • Check grid definitions using Visit (see previous page …) • Check import and export fields (can be written in NetCDF and ASCII
format). The ATM/OCN export files must match with OCN/ATM import files. The debug level of the coupled model must be >2.
• Use debug version of ESMF library (compiled with ESMF_BOPT=g). The ESMF can give some information in the PET* files.
• Use debug version of the model components. The ROMS can be compiled in debug mode by modifying ./build.sh script (set USE_DEBUG as on). The RegCM can be compiled in debug mode by running configure script with --enable-debug option. In the stdout file you can see the actual line number of the source file that triggers the error.
• Check input files and the configuration of the model components (RegCM and ROMS).
- Future Plans
• More work … • Add support for identical mesh/grid.
• No need to interpolation between grid. • The possible candidate to test this configuration is the BAND
version of the RegCM. • The tool to create ROMS grid from RegCM domain file is almost
ready. It is written in NCL. • Move all coupled model configuration options to regcm.rc • Add support for boundary smoothing. It is important for the ocean or
sea centric applications (i.e. Mediterranean Sea, Indian Ocean). • Add support for selective interpolation types for each variable. User
can change from conservative type re-gridding to bilinear or others. • Improve the air-sea interaction interface. In the current implementation,
ocean send SST and ICE thickness to atmospheric model. The aerosol related variables, surface roughness etc. can be added into list.
• Benchmarking is needed to see the actual performance of the model. It also helps to find possible performance bottleneck sources of the current implementation.
- News
• RegCM is in the component list of the ESMF J
http://www.earthsystemmodeling.org/components/