NEMOVAR-LEFE Workshop 22/3 2007.

Slide 1

Slide 1

Current status of NEMOVAR

Kristian Mogensen

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 2

Slide 2

Outline of talk:Why do we need NEMOVAR?

- Isn’t OPAVAR good enough?Goals of the NEMOVAR projectImplementation planWhere are we nowOutstanding issues

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 3

Slide 3

Why do we need NEMOVAR? Anthony Weaver et al from CERFACS has been developing a

variational data assimilation system for OPA version 8.2 (OPAVAR).- Incremental approach

- Supports 3D-VAR (FGAT) and 4D-VAR

- Has been used with the ORCA2 grid and the TDH tropical pacific area grid

- Written mostly in the OPA 8.2 coding style (Fortran-77) with a few extensions

Dynamic memory in a few routines- No distributed memory (MPI) parallelization, only shared memory

(OpenMP) parallelization The OpenMP scaling is not spectacular Scaling to higher resolution than ORCA2 problematic due to

memory constraints (>26 GB needed for ORCA1 needed) OPA 8.2 is not actively developed anymore All work within the OPA developers team are focussed on the new

NEMO version of the OPA model

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 4

Slide 4

ECMWF/CERFACS goals for the NEMOVAR project

Short term (in ~2 years) goal- To have a 3D-VAR system based on NEMO- Support distributed memory parallelization

Possible also support shared memory parallelization- Support for different ORCA configurations

We do not worry about limited area versions of NEMO- Support for profiles and altimeter observations and SST products

It should easy to add a new observation type- Multi-incremental with different resolution in the inner loops

compared to the outer loops Not a trivial task

Long term goal- A full 4D-VAR system with all of the above properties

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 5

Slide 5

Splitting the variational problem into outer and inner loops:1. In the initial outer loop the following is done:

1. Compute misfit of observations relative to background2. Store initial trajectory for the inner loop

2. In the inner loops the following is done:1. Minimize the incremental cost function using an iterative

procedure to produce an increment

3. In the subsequent outer loop the following is done:1. Update trajectory with the increment2. Update misfit of observations

3. Update misfit of background

4. IF ( iloop < noutloop ) GOTO 2

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 6

Slide 6

Implementation plan: overviewWe have defined the following plan:

- Phase 1: Split the existing OPAVAR Fortran code into separate executables for inner and outer loops

- Phase 2: Develop an MPP implementation of the observation operators in the outer loop using NEMO

- Phase 3: Develop a hybrid system with NEMO in the outer loop and OPAVAR in the inner loop

- Phase 4: Develop an MPP implementation of the 3D-VAR with NEMO in both inner and outer loops

- Phase 5: Develop an MPP implementation of the full 4D-VAR with NEMO in both inner and outer loops

Phase 1 and 2 can done in parallelBy phase 4 we will have archived our short term

goalBy phase 5 we will have archived our long term goal

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 7

Slide 7

Implementation plan: phase 1 Split the existing OPAVAR Fortran code into

separate executables for inner and outer loops- This is needed for phase 3- Will allow scientific developments to use OPAVAR to continue

work in parallel with the NEMOVAR work We don’t want this to stop while we do the technical work Not all options of OPAVAR will be migrated to NEMOVAR

- Supports both 3D-VAR and 4D-VAR and combinations of both for multiple inner loops

- Will verify the approach of using different executables in the inner and the outer loop Similar to what is done for the IFS 4D-VAR system

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 8

Slide 8

Implementation plan: phase 2Adding observation operators to NEMO

- Import interpolation routines from OPAVAR- Distribute the observations according to NEMO domain

decomposition initially Load imbalance in the observation operators?

- Initially we will focus on the following observations: Profiles (XBT’s, Argos etc.) SLA data SST data

- Easy extendable to other observationsCan be used for other assimilation schemesThe observation operators can be used for diagnostics to

compare a model run with observations

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 9

Slide 9

Implementation plan: phase 3Develop a hybrid system with NEMO in the outer loop

and OPAVAR in the inner loop- Useful to verify the implementation of the NEMO outer loop- Since the NEMO inner loop is the “hard” part of the migration to

NEMO this system will be useful for scientific developments for some time

Writing of model trajectories in NEMO for input to the OPA inner loop will have to be coded

Reading of increments for applying the them to the non-linear states will have to be coded as well

The later 2 items are relevant for phase 4We will probably not worry too much about MPP for

phase 3Can in principle run both 3D-VAR and 4D-VAR

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 10

Slide 10

Implementation plan: phase 4Develop an MPP implementation of the 3D-VAR with

NEMO in both inner and outer loops- Parallelization of the control vector will be done based on

ECMWF IFS experiences A Fortran-90 derived type (control_vectors) is used to

defined the distributed memory layout of the control vector Fortran-90 operations like assignment and dot product are

overloaded for this type and all message parsing is done in the overloaded functions

Flexible code for non MPP developers- The minimization of the cost function is going to be based on

work done at CERFACS- Initially we will assume the same resolution in the outer and inner

loop Later we will consider different resolution in the outer and

the inner loop

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 11

Slide 11

Implementation plan: phase 5

The full 4D-VAR system is dependent on the existence of the tangent linear and adjoint of the NEMO model

We will get the status for this in the next talkWe aim to have the code of phase 4 flexible enough

that adding 4D-VAR as an option is easy once the TL/AD is available

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 12

Slide 12

Where are we nowPhase 1 (the splitting of OPAVAR) has been

completed.Phase 2 (observation operators in NEMO) is well

under way- It has been done for profile observations- Work on including altimeter is in progress- Work SST data will start shortly

Work on phase 3 (hybrid system) is just about to start

Work on phase 4 (3D-VAR NEMO) is being discussed and is like to start Q2 2007

NEMOVAR-LEFE Workshop 22/3 2007.

Slide 13

Slide 13

Some outstanding issues for discussionWe have not yet considered quality control of the

observations:- Some checks could be done between the first outer loop and the

first minimization- Other checks could be done before the first outer loop

How to best implement the change of resolutions between outer and inner loops:- Straight interpolation?- Something more advanced which better preserves the physical

quantities of the ocean fields?