Copernicus Atmosphere Monitoring Service

CAMS General Assembly, Athens, 14-16 June 2016

Anna Agusti Panareda, Samuel Remy, Vincent Huijnen, J.J-Morcrette, Olaf Stein, Joaquim Arteta, Simon Chabrillat, Johannes Flemming & Angela Benedetti, Antje Inness, Sebastien Massart, Richard Engelen as well as all contributors to IFS and C-IFS

C-IFS: How are developments integrated

2

IFS : Integrated Forecasting System of ECMWF

A very good NWP forecast and data assimilation model

3

10th anniversary of IFS1997

IFS : Integrated Forecasting System of ECMWF

A complex model system for forecast and assimilation

4

Adding composition to IFS : Composition -IFS

• In GEMS project:• Coupled system IFS-MOZART for chemistry

• GHG and aerosol on-line (integrated) in

the IFS• MACC I-III: chemistry on-line in IFS

• Chemistry - IFS (2009)• Renamed to Composition –IFS: all

composition aspects • Composition – IFS : global production system

in CAMS at ECMWF

5

Integration of chemistry & aerosol modules in IFS

Dynamics & Physics

Chemistry

ctm

Dynamics & Physics

Transport & Chemistry

oasis4

oasis4

oasis4

IFS IFS CTM

Feedback Flow

Coupled SystemFeedback: slowFlexibility: high

Integrated System Feedback: fast Flexibility: low

Coupled SystemIFS- MOZART3 / TM5

C-IFSOn-line Integration

Flemming et al. 2009

Flexible but

very un-efficient

Fast, consistent

but higher

coding effort

6

Composition – IFS : multiple schemes

Composition –IFS

MOZART chemistry

Cariolle Strat. O3

CO2 & CH4

GLOMAP aerosol

MOCAGE chemistry

CAMS Procurement

Open IFS Interface

BASCOE stratospheric

chemistry

TM5 (CB05) chemistry

MACC (LMDz) aerosol

BMS Strat. O3

MACC III heritage

7

Benefits for CAMS using C-IFS• IFS is the best NWP model on the planet • IFS is a very efficient global model

• Operational IFS resolution is currently 9 km globally

• CAMS o-suite resolution is 40 km globally • IFS data assimilation (4D-VAR, ENS) used for

composition• Using 4D-Var algorithm (Ensemmble DA)• Infra structure to process assimilated

observations

8

Benefits of high resolution modelMid-tropospheric CH4 [ppb] at 450 hPa

Low resolution FC (80 km, L60) High resolution FC (16 km, L137)

Anna Agusti-Panareda

9

Challenges to use C- IFS for CAMS • Adaptation of data assimilation system to

specifics of composition field and observations

• IFS advection does not formally conserve mass • Global mass fixers implemented

• Link CAMS development with ongoing IFS development• 2-3 new cycles each year• Reproducibility of older cycles• IFS coding standards

10

Towards better integration between C-IFS Components • Between Chemistry, Aerosols and GHG

modules• Secondary aerosol formation based on

chemistry• Photolysis and surface chemistry

modulation by aerosol • Unified modelling of methane in Chemistry

and GHG• Code harmonisation

• Composition on NWP (and back !!) • Aerosol in radiation • Ozone in radiation• Land surface and fluxes (emissions and

deposition)

11

CAMS ozone fields in IFS radiation scheme I

12

CAMS ozone fields in IFS radiation scheme II

New CAMS Ozone climatology

used in next IFS cycle

13

How are C-IFS developments by CAMS partners integrated …

14

IFS - coding rules http://intra.ecmwf.int/publications/cms/ge

t/ifs/4663

REAL(KIND=JPRB),INTENT(IN) ::

PRR(KLON,NREAC)REAL(KIND=JPRB),INTENT(IN) ::

PRJ(KLON,NPHOTO)

DO JL=KIDIA,KFDIA ZP1=PRJ(JL,jbno3)*PY(JL,ino3)

…… ENDDO

not a coding rule but advised for efficiency

15

Code efficiencyUse Profiling to find bottle necks

Profiling information for program='/fws2/lb/work/rd/disr/g99u/2014120100/gfc/tmp.g99u_fc_fcgroup1.model.1.32453/ifsMASTER', proc#3:No. of instrumented routines called : 1254Instrumentation started : 20160502 143312Instrumentation ended : 20160503 035310Instrumentation overhead: 35.69%Memory usage : 1449 MBytes (heap), 1452 MBytes (rss), 0 MBytes (stack), 0 (paging)Total CPU-time is 98250.51 sec on proc#3, 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6) (32 procs, 2 threads)Thread#1: 55730.71 sec (56.72%), 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6)Thread#2: 42519.80 sec (43.28%), 0 MFlops (ops#0*10^6), 0 MIPS (ops#0*10^6)

# % Time Cumul Self Total # of calls MIPS MFlops Div-% Routine@<thread-id> (Size; Size/sec; Size/call; MinSize; MaxSize) (self) (sec) (sec) (sec)

1 13.13 12900.110 12900.110 12900.290 563 0 0 0.0 >MPL-TRGTOL_COMMS ( 803)@1 2 8.53 21278.660 8378.550 16104.600 56683200 0 0 0.0 *UKCA_DCOFF_PAR_AV_K@1 3 8.41 29536.700 8258.040 15986.450 55751976 0 0 0.0 *UKCA_VGRAV_AV_K@2 4 8.37 37761.020 8224.320 16033.160 56683200 0 0 0.0 UKCA_VGRAV_AV_K@1 5 8.33 45946.390 8185.370 15861.140 55751976 0 0 0.0 UKCA_DCOFF_AR_AV_K@2 6 4.73 50594.410 4648.020 37276.250 4048800 0 0 0.0 *UKCA_DDEPAER_INCL_SEDI@1 7 4.58 55095.040 4500.630 36813.330 3982284 0 0 0.0 UKCA_DDEPAER_INCL_SEDI@2

S.Remy, C-IFS GLOMAP profiling

Usage of resources per routine call

16

CAMSpartner

CAMSECMWF

IFS team RD ECMWF

CAMS FDECMWF

RD research dep.FD forecast dep.

e-suite

o-suite

CAMSVAL

How are C-IFS developments integrated ?

Up to 1 year from development to

o-suiteimplementation

17

Thank you!ευχαριστώ

Tower of WindsA meteorological monument nearby

with a CAMS theme:Skiron (NW) distributes the ashes

18

How are C-IFS developments integrated … 1. Contributing partner (or ECMWF):

• Testing (Test A) of individual model development • Delivery to ECMWF/CAMS

2. CAMS-ECMWF Section:• Integrate development in CAMS branch • Testing (Test B) of all integrated model improvements• Submit to ECMWF RD IFS section for ECMWF cycle upgrade

3. ECMWF RD IFS group• Merge new cycle from all ECMWF contributions

4. Forecast Department Copernicus section: • Run experimental CAMS suite (e-suite) and tested by VAL

5. Forecast Department Copernicus section:• Run operational CAMS suite (o-suite)

• Each of the steps can take 1-3 month so that it takes up to a year month from model update to implementation in o-suite

• Time line of ECMWF cycle upgrades will be announced to CAMS partners well in advance

19

Computational Cost C-IFS

16 km 40 km 80 km