WWOSC 2014, Montreal, August 16-21 2014 A review of technologies and algorithms used to couple Earth...
-
Upload
frederick-small -
Category
Documents
-
view
216 -
download
1
Transcript of WWOSC 2014, Montreal, August 16-21 2014 A review of technologies and algorithms used to couple Earth...
WWOSC 2014, Montreal, August 16-21 2014
A review of technologies and algorithms
used to couple Earth System model components
S. Valcke - CERFACS
WWOSC 2014, Montreal, August 16-21 2014
Outline
Introduction
1.Two main technical approaches to coupling in climate
2.Current coupling technologies used in climate modelling• ESMF• CESM/cpl7• FMS• OASIS
3. Few ocean-atmosphere (+ ice, land, wave) coupling algorithms:
European ESMs, Env Canada, CESM
Conclusions
WWOSC 2014, Montreal, August 16-21 2014
Why couple ocean and atmosphere (and sea-ice and land and …) models? Of course, to treat the Earth System globally
What does “coupling of codes” imply? Exchange and transform information at the code interface Manage the execution and synchronization of the codes
What are the constraints? Physical constraints: e.g. energy conservation at the interfaces Numerical stability of the algorithm Start from existing and independently developed codes Global performance and load balancing issues are crucial Computing platform and OS characteristics
Introduction
WWOSC 2014, Montreal, August 16-21 2014
Outline
Introduction
1.Two main technical approaches to coupling
2.Current coupling technologies used in climate modelling• ESMF• CESM/cpl7• FMS• OASIS
3. Few ocean-atmosphere (+ ice, land, wave) coupling algorithms:
European ESMs, Env Canada, CESM
Conclusions
WWOSC 2014, Montreal, August 16-21 2014
1. Coupling framework integrated approach
Split code into elemental units at least init/run/finalize
Write or use coupling units
Adapt data structure and calling interface Use the framework to build a
new hierarchical merged code
prog1_u1 prog2_u1
coupling
prog1_u2 prog1_u3
couplingprog2_u2
program prog1…end prog1
prog1_u1
prog1_u2
prog1_u3
program prog2…end prog2
prog2_u1
prog2_u2
efficient sequential and concurrent components
existing codes
(easy)
1. Two main approaches to coupling
Ex: with built-in driving layer: CESM, FMS without pre-defined driving layer: ESMF
WWOSC 2014, Montreal, August 16-21 2014
2. Coupler or coupling library approach
program prog2…call cpl_recv (data2, …)end
program prog1…call cpl_send (data1, …)end
cou
ple
r
couplingconfiguration
cou
ple
r
existing codes
concurrent coupling (parallelism)
possible waste of resources if sequential execution of the components is enforced by coupling algorithm
multi-executable: may be harder to manage for the OS
efficient
Ex: OASIS, TDT, Open-Palm, GOSSIP
1. Two main approaches to coupling
WWOSC 2014, Montreal, August 16-21 2014
Outline
Introduction
1.Two main technical approaches to coupling
2.Current coupling technologies used in climate modelling
• ESMF• Cpl7/CESM• FMS• OASIS
3. Few ocean-atmosphere (+ ice, land, wave) coupling algorithms:
European ESMs, Env Canada, CESM
Conclusions
WWOSC 2014, Montreal, August 16-21 2014
Earth System Modeling Framework
2. Current coupling technologies - ESMF
• NUOPC layer: US NWP centers conventions and templates for better interoperability
Open source software for building climate and weather applications based on components developed in different modeling centers
• Multi-agency governance (NSF, NASA, DoD, NOAA) with many partners
• 12 different modelling systems, about 85 different components
component = well-defined interface + coherent function
• Gridded Components: scientific code • Coupler Components: data transformation
user builds a new single code as hierarchy of nested components support for multiple executables and web services
• Component-based design:
Goddard Space Flight Center GEOS-5 model
WWOSC 2014, Montreal, August 16-21 2014
1. Define Gridded Components : slip code into init, run and finalize methods
2. Wrap native data structures into ESMF data structure
3. Write Coupler Components
4. Schedule components and exchange data
5. Execute the application
subroutine myOceanRun (.. , impState, expState, clock, …) type(ESMF_State) :: impState
subroutine oceanToAtmCpl (.. ,) call ESMF_FieldRedist(oceanField, atmField, …)
…call ESMF_GridCompRun(oceanComp, …)call ESMF_CplCompRun (oceanToAtmCpl, …)call ESMF_GridCompRun(atmComp, …)
ESMF “Superstructure”: coupling tools and component wrappers with standard interfaces
2. Current coupling technologies - ESMF
ESMF “Infrastructure” :• calendar management; message logging, regridding weight generation (can be used offline) data regridding & communication
Hill et al., Comput. Sci. Eng., 2004
WWOSC 2014, Montreal, August 16-21 2014
Cpl7 for CCSM4 and CESM1
2. Current coupling technologies – Cpl7
Software with top-level driver and coupler component for flexible assembling of atmosphere, ocean, land and sea ice into one executable
Developed by the NCAR Earth System Laboratory, uses Argonne Nat Lab MCT for data regridding and exchange
From multiple concurrent executables (cpl6) to one executable: time flow easier to understand, easier to debug
interface compatibility for ESMF-compliant components
Ported to IBM p6, Cray XT4/XT5, BGP, Linux Clusters, SGI
WWOSC 2014, Montreal, August 16-21 2014
2. Current coupling technologies – Cpl7
Varying layouts on computing resources via external configuration (metadata) :
processors
tim
e
CPL (regridding, merging) 0.5 sec
CLM (land) 1.4 sec
CICE (ice) 5.0 sec
CAM (atm) 6.2 sec
Driver (controls time evolution)
POP (ocean) 8.2 sec
Sequential layout Hybrid layout
CPL 0.7 sec
CICE 8.4 sec POP 14.9 sec
Driver
CLM 2.3 sec
CAM 11.2 sec
7.7 sec
processors
tim
e
Hybrid layout
CPL 0.7 sec
CICE 8.5 sec POP 19.1 sec
Driver
CLM 2.3 sec
CAM 11.2 sec
5.2 sec
processors
tim
e
1.0 sec
Scaling evaluated on up to 10 000 processors: Craig et al., Int. J. High Perform. C,
2012
WWOSC 2014, Montreal, August 16-21 2014
The Flexible Modeling System (FMS)
• Active for over a decade at GFDL; developed in F90
• FMS shown to be scalable with up to O(10000) pes
Software to assemble a climate model with domain-specific “slots” for atmosphere, ocean, ocean surface including sea ice and land surface
FMS Superstructure
FMS Infrastructure
User code
• FMS “Superstructure”: • Domain-specific coupling layer (“stubs” (no component), or “data” also possible)• Components “wrapped” in FMS-specific data structures and procedure calls• Single executable with serial or concurrent execution of components• Regridding, redistribution, or direct (hard-coded) exchanges between components• Includes data assimilation
2. Current coupling technologies – FMS
WWOSC 2014, Montreal, August 16-21 2014
• Interface fluxes must be globally conserved atmosphere water-land fractions adjusted to fit ocean sea-land mask quantities are transferred from the parent grids to the exchange grid,
where fluxes are computed; they are then averaged on the receiving grid
2. Current coupling technologies – FMS
• Exchanges consistent with physical processes occurring near the surface
nn TAT 1
atmosphere
land
Implicit calculation of vertical diffusive fluxes over the whole column
Up-down sweep for tridiagonal matrix resolution through the exchange grid
Atm
Land
Exchange
FMS “Superstructure” obeys specific geophysical constraints
WWOSC 2014, Montreal, August 16-21 2014
2. Current coupling technologies – OASIS
Communication and regridding library to exchange data between independent models with minimal level of interference in the codes (external configuration through namelist-like file)
• Initialization: call oasis_init(...)• Grid definition: call oasis_write_grid (...)• Local partition definition: call oasis_def_partition (...)
Ap
pli
cati
on P
rog
Inte
rfac
e
• Coupling field exchange: in model time stepping loop
call oasis_put (…, time, var_array. …) call oasis_get (…, time, var_array, …)• user external configuration: => source or target (end-point communication)
=> effective coupling frequency=> transformations and regridding
• Developed by CERFACS since 1991 with CNRS since 2005 and many others• Written in F90 and C; open source license (LGPL)• Last OASIS3-MCT version based on MCT • Large community of users: ~35 climate modelling groups world-wide
WWOSC 2014, Montreal, August 16-21 2014
• Toy coupled model: coupling exchanges between NEMO ORCA025 grid (1021x1442) and Gaussian Reduced T799 grid (843 000)
• Bullx Curie thin nodes (Intel® procs Sandy Bridge EP; IFort 12.1.7.256, Bullx MPI 1.1.16.5)
• IBM MareNostrum3 (Intel Sandy Bridge processors, Intel MPI 4.1.0.024)
Coupling overhead for one-year long simulation with one 1 coupling exchange every hour in each direction between codes with O(1 M) grid points running on 4000 cores/component:
~60 seconds for initialisation, ~30 seconds for data exchange
2. coupling technologies – OASIS
WWOSC 2014, Montreal, August 16-21 2014
Outline
Introduction
1.Two main technical approaches to coupling
2.Current coupling technologies used in climate modelling• ESMF• Cpl7/CESM• FMS• OASIS
3. Few ocean-atmosphere (+ ice, land, wave) coupling algorithms:
European ESMs, Env Canada, CESM
Conclusions
WWOSC 2014, Montreal, August 16-21 2014
OCN +
ICE
ATMn-1 n
TO
TI
AlbO
AlbI
U/V/WO+I
Few coupling algorithms – EU ESMs
Taux/y/z O+I
LH+SH+LWO
LH+SH+LWI
DQ/DTSWO
SWI
EV+SB-R-SNWnd10mWndmod
RunoffCalving
n+1
All fluxes calculated in the atmosphere-surface model For most: Double ocean-ice flux calculation in the atmosphere For some: sea-land fraction in atm defined by interp of ocean mask For most: addition of a runoff model ECMWF: wave-model in addition
CNRM-CM5, IPSL-CM5, EC-Earth, CMCC-ESM, MPI-ESM, MetOffice GC2.0
WWOSC 2014, Montreal, August 16-21 2014
NEMO+
LIM
GEM
1 2 3
TO/I
SHAO/AI
LHAO/AI
TauxAO/AI
TauyAO/AI
Ice FrachI
hS
SWA
LWA
PR TA
UA
VA
qA
PA
P0
SHAO/AI (TA, TO/I, UA, ZM, ZH)LHAO (qA, qO, TA, TO, UA, ZM, ZH)LHAI (qA, qI, TA, TI, UA, ZM, ZH)Taux/yAO/AI (U/VA, TA, TO/I, U/VO/I, ZM, ZH)
SWA (ALO/I –not from O/I)LWA ( TO
4, TI4 )
Environment Canada
1
4
2 3 4
WWOSC 2014, Montreal, August 16-21 2014
Ocn
Atmn-1
Ice
n
Lnd
Cpl accatm/ocn
flx
merge
atm/ocn flx
merge
acc
Atm : SWd, LWd, rain, snow -> Lnd, Ice, Ocn (via acc)
Lnd, Ice, Cpl : LWu, LH, SH, EV, taux/y albedo -> Atm, Ocn (via atm/ocn flx)Ice : heat & salt fluxes, SWpen -> Ocn (via acc) + Runoff, Glacier, …
Few coupling algorithms – CESM
WWOSC 2014, Montreal, August 16-21 2014
Different technical coupling approaches used in climate modellingValcke et al, 2012: Coupling technologies for Earth System Modelling, GeoscMD.
• External coupler and/or coupling library (e.g. OASIS): easiest solution to couple independent codes but some drawbacks
• Integrated approach: split original codes and rebuild a new code (e.g. ESMF)
more efficient in many cases but puts more constraints on the components
The “best” coupling tool is not uniquely defined; it depends on: • ready to change/adapt your original codes• efficiency you want/need to achieve, etc.
Conclusions
Diverse approaches for flux calculation at the component interfaces:• Dictated by science? By numerical constraints?• Ad-hoc developments given code constraints?• We need to better understand the impact into ESMs
Series of Coupling Technology Workshops: • Toulouse Dec 2010• Boulder Feb 2013: “Int Working Committee on Coupling Technologies”
launched • Next: Manchester 20-22 April 2015
WWOSC 2014, Montreal, August 16-21 2014
The end
WWOSC 2014, Montreal, August 16-21 2014
NEMO+
LIM2
IFS
1
TO/I
UO/I
VO/I
Ice frac
UA
VA
SWA
LH+SH+LWPREV
WAM
TauxTauyStokes driftTurb energy
Stokes driftRoughness
2
TO/I
UO/I
VO/I
Ice frac
UA
VA
SWA
LH+SH+LWPREV
TauxTauyStokes driftTurb energy
Stokes driftRoughness
Few coupling algorithms – ECMWF
WAM: wave-model One-executable with sequential execution of components
WWOSC 2014, Montreal, August 16-21 2014
TA
hA
θ
φT
hI
Tf
ψ
Ψ = hA (θ – TA) ; Ψ variable d’état; θ CI
T =Tf –φ/hI ; T variable d’état; φ CI
T(n) =Tf –(hA/hI) θ(n) + (hA/hI) TA
converge si hA/hI <1 ; diverge sinon
Classiquement, hI = 4 W/m2/K ; hA/hI ~5couplage atmosphère-glace diverge
Pour l’océan ho~∞, donc hA/ho ~0couplage atmosphère-ocean stable
WWOSC 2014, Montreal, August 16-21 2014
1*- Sensible heat flux
6*- Evaporation + int. energy [+ Qlat]
3x- Fresh water flux4x- Salt flux7x- Mass of snow and ice
1- Temperature at sea-ice base3- Highest level temperature (SST)4- Ocean radiative temp.8- Absorbed solar radiation (in 1st layer)
5- Surface ocean current7- Surface height 7
1- Continental runoff + internal Energy 8
1*- Surf. Temp2*- Surf. Roughness3*- Displacement height4x- Surface velocity
5
1- Rainfall + int. energy2- Snowfall + int. energy
3- Incoming solar radiat.4- Solar zenith angle5- Fraction of diffuse solar radiation6- Downward infrared radiation7- Sensitivity of atmos temp. & humidity to surf. fluxes
1 1
2*- Surf. emissivity 3*- Albedo, direct4*- Albedo, diffuse5*- Surf. radiative temp.
2
2
Land surface model
Ocean model
Atmosphere model
Ocean surface module
Sea ice model (wave model)
Surface layer turbulence
6
1x- Non solar heat flux2x- Solar radiation
7*- Wind stress
5x- Wind stress6x- U^3
8- Subgrid fractions
Note on subgrid fractiondependance:<>x- Sea Ice categories (incl. open ocean)<>*- Sea Ice or Land Surf. categories
8- Subgrid fractions2- Salinity at sea-ice base6- Sea surface salinity
1*- Cd 42*- Ce 43*- Ch
1- Surface pressure2-4 Air temperature, humidity and wind5- Mean scalar wind speed6- Height of these variables 3
3
Few coupling algorithms – PRISM
WWOSC 2014, Montreal, August 16-21 2014
Typed Data Transfert (Potsdam Institute for Climate – PIK)
•library to transfer data between programs in a platform and language independent way with different protocols (Unix Socket, I/O from/to files, MPI) -> heterogeneous coupling
Current coupling technologies
Open-PALM (CERFACS - ONERA)•Tool originally designed for creating data assimilation suites•Dynamic driver and CWIPI coupling library (ONERA) performing parallel data exchanges, parallel calculation of weights, parallel remapping•Multiphysic coupling for ~40 different applications mainly in France
Bespoke Framework Generator (U. Manchester):
•Generator of wrapper code around components to build a coupled model with a chosen coupling technology (curr. OASIS, MPI, argument passing) based on external metadata for flexible model composition and deployment
•Used for 20 configurations in CIAS (Comm Integrated Assessment System)www.cs.manchester.ac.uk/cnc/ projects/bfg
Model Coupling Toolkit (Argonne National Lab):
•library and related datatypes for parallel field exchanges, parallel data transformation for creating parallel couplers / coupled models
•used in CESM/Cpl7 and OASIS3-MCT www.mcs.anl.gov/research/projects/mct/
WWOSC 2014, Montreal, August 16-21 2014
Ocn
Atm1
Ice
2
Lnd
Cpl accatm/ocn
flx
merge
Atm : SWd, LWd, rain, snow -> Lnd, Ice, Ocn (via acc)
Lnd, Ice, Cpl : LWu, LH, SH, EV, taux/y albedo -> Atm, Ocn (via atm/ocn flx)Ice : heat & salt fluxes, SWpen -> Ocn+ Runoff, Glacier, …
accatm/ocn
flx
merge
Few coupling algorithms – CESM
WWOSC 2014, Montreal, August 16-21 2014
I.3 Time sequence of exchanges
Atm
SLT
OSM
Oce
7
5
6sea ice t
Frequency of coupling exchanges:
F7 = F6 < F5 = F3 = F1 = F4 = F2
slow fast
3
1
4+3
Comp.Fluxes
2
atm t (implicit)
3
1
4+3
25
3
1
4+3
2
3
1
4+3
2
7
deep ocean t
Part I - On a revised ocean-atmosphere physical coupling interface
5 5
6
WWOSC 2014, Montreal, August 16-21 2014
Separation of fast ocean + sea ice surface processes involving heat, water and momentum exchanges with the atmosphere from slower deeper ocean processes.
Calculation of fluxes at the resolution of the surface (would be
non-physical to regrid the turbulent exchange coefficients Cd, Ce,
Ch).
Implicit calculation of energy fluxes from the base of the sea-ice to the top of the atmosphere.
Part I - On a revised ocean-atmosphere physical coupling interface
Comments and conclusions
• Increased modularity with SLT and OS modules.
• SLT runs on finer grid and computes surface turbulent coefficient.
• OS computes radiation and turbulent fluxes.