Status of Meta-Files, ADMT 13 Hyderabad, India

Esmee van Wijk, Nov 2012CSIRO Marine and Atmospheric Research, Australia

Photo from Katy Hill

Metadata format version 2.4

Number of defined missions, not number of profiles

Metadata variables – 2.4.3 Float characteristicsIMEI numbers

• Security risk• Remove from meta file?• Or restrict numbers even further?

TC: YES to remove (not needed in metafiles).But AIC needs it for tracking (authorization, private use)

New variables• PLATFORM_FAMILY (SRT)• PLATFORM_TYPE (SRT)

• Standard reference tables developed by TC (SRT)

Metadata variables – 2.4.3 Float characteristics

STRPLATFORM_FAMILYPLATFORM_TYPEPLATFORM_MAKERPLATFORM_TYPE_KEYSTANDARD_FORMAT_ID

and also applied to GTS tables (TESAC/BUFR).

PLATFORM_FAMILY PLATFORM_TYPE PLATFORM_MAKER PLATFORM_TYPE_KEY STANDARD_FORMAT_IDFLOAT PALACE WRC 000 000000 to 000999FLOAT APEX WRC, TWR 001 001000 to 001999FLOAT EM-APEX WRC, TWR 005 005000 to 005999FLOAT_DEEP APEX_D TWR 020

FLOAT PROVOR-MT METOCEAN 100 100000 to 100999

FLOAT PROVOR MARTEC 101 101000 to 101999FLOAT ARVOR NKE 102FLOAT PROVOR-II NKE 103 etc …

FLOAT_COASTAL ARVOR_C NKE 110FLOAT_DEEP ARVOR_D NKE 120

FLOAT SOLO SIO-IDG 200FLOAT SOLO-W WHOI 201FLOAT SOLO-II SIO-IDG 202FLOAT S2A MRV 203FLOAT_DEEP SOLO_D SIO-IDG 220

FLOAT NINJA TSK 300FLOAT_DEEP NINJA_D TSK 320

FLOAT NEMO OPTIMARE 400

FLOAT NAVIS-A SBE 5002)

FLOAT NOVA METOCEAN 600

PROFILER_TETHERED ITP WHOI 910

Combinations exist in reality (POPS)FLOAT_TETHERED PROVOR-MT METOCEAN 100FLOAT_TETHERED NOVA METOCEAN 600FLOAT_TETHERED NEMO OPTIMARE 400

1) STR Needs agreement

2) STANDARD_FORMAT_ID detailed as far as possible (on progress on-line)

3) link to each WMO:Suggestion to fill up a new table for global update for 9000 floats (AIC, DACs, GDACs) including key new metadata (new tab, blank on-line)

http://tinyurl.com/cweskt6

Metadata variables – 2.4.3. Float characteristics

New variables• FLOAT_SERIAL_NO replaces INST_REFERENCE variable• should only be filled with the serial number of the float i.e. 1679 NOT sbe1679 or APEX1679 etc.•Attention PROVOR serial is “complex” (typos)

• FIRMWARE_VERSION• MANUAL_VERSION• STANDARD_FORMAT_ID• DAC_FORMAT_ID

• important for identifying float format and enabling cross-reference to online manuals• important information required for trajectory analyses

Metadata variables – 2.4.3 Float characteristics

New variables• BATTERY_TYPE (SRT)• BATTERY_PACKS (SRT)• CONTROLLER_BOARD_TYPE_PRIMARY (SRT)• CONTROLLER_BOARD_TYPE_SECONDARY (SRT)• CONTROLLER_BOARD_SERIAL_NO _PRIMARY• CONTROLLER_BOARD_SERIAL_NO _SECONDARY

• Standard reference tables developed by TC where possible

Metadata variables – 2.4.3 Float characteristics

New variables

• Standard reference tables developed by TC where possible (SRT)

• SPECIAL_FEATURES• SAMPLING_MODE (SRT)• REDEPLOYED• FLOAT_OWNER• OPERATING_INSTITUTION• CUSTOMISATION • ARGO_GROUP (SRT)

Redefine descriptions – i.e. core Argo currently too specific, e.g. 2000 profile, 1500 park? - equivalent Argo? – Argo floats that are not considered part of the core Argo mission, e.g. navy, special funding, no delayed mode activity, accelerated profiling etc.

Do we want additional categories..? Currently, we have Argo equivalent, Bio Argo, Core Argo categories that are well known within the Argo community..

Do we want others..?• coastal.. (issues - floats may be deployed in a coastal region but drift out..)• marginal seas..• boundary current..• under ice.. (only floats with under ice algorithm or hardware..? Or those floats

deployed above 60˚N and below 60˚S ?

Mandatory Meta-data Parameters

Have we missed anything?

Mandatory Meta-data Parameters con.

2.4.5. Configuration Parameters

• For each configuration parameter the name and value of the parameter are recorded

• Users can identify which mission belongs to which cycle by looking at the: CONFIGURATION_MISSION_NUMBER (N_CYCLE) variable in the trajectory file• Do we also want CONFIGURATION_MISSION_NUMBER (N_PROF) in the profile file?

2.4.5. Configuration Parameters

• Historically, input was not standardised, not all information required was reported and fields were not populated correctly.

• Clearly there is a need to standardise configuration parameters.• Scheme adopted is similar to the technical files.• All configuration parameters are identified by a CONFIG_*

prefix.• Configuration parameters are float settings, not measurements

reported by the float.• DACS were asked to check their float types and provide

feedback before this meeting.

Configuration Parameter Names

Configuration Variable NameExplanationMandatory/Optional Comment/Action Float Type

CONFIG_ArgosTransmissionRepetitionPeriod_SECONDS

transmission repetition rate, metadata for most floats - may be variable for two-way communication

M if Argos float, O for other floats, i.e. SOLO

  ARGOS, SOLO

CONFIG_AscentToSurfaceTimeOut_DecimalHour

maximum time a float will try to reach its target depth (surface) before aborting this step

M if exists?, O for others

  APEX, APEX APF9, NINJA, SOLO, SOLOII

CONFIG_CycleTime_DecimalHour For APEX and ARVOR floats this is the total duration of one cycle, usually 240 hours (10 days). For SOLO floats this is the total duration of one cycle, assuming that all float operations reach their full time-out intervals before moving to the next float stage. Typically the actual cycle time will be shorter than this value.

M, if applies   APEX, APEX APF9, NINJA, NEMO(Iridium), PROVOR, ARVOR, POPS, SOLO, SOLOII, Isopycnal Park APEX

CONFIG_DescentToMaxPresTimeOut_DecimalHour

maximum amount of time (in hours) allowed for the float to descend to Profile Pressure

M if exists?, O for others

this variable should cover 'DELAI parameter required for tajectory processing

APEX, APEX APF9, SOLO, SOLOII

CONFIG_DescentToParkTimeOut_DecimalHour

maximum time a float will try to reach its target depth (park) before aborting this step

M if exists?, O for others

  APEX, APEX APF9, SOLO, SOLOII

CONFIG_Direction_LOGICAL Ascending = 1, Descending = 2 M APEX, APEX APF9, NINJA, NEMO(Iridium), PROVOR, ARVOR, POPS, SOLO, SOLOII, Isopycnal Park APEX

CONFIG_DownTime_HOURS preset duration of the down time (start of descent to start of ascent) - changed from : CONFIG_DownTimeInterval_HOURS and CONFIG_DownTimeOut

M if exists?, O important for trajectory files APEX, NINJA

• Reference table 18 in Argo User’s Manual (version 2.4 March 30th 2012) • Available from http://www.argodatamgt.org/Documentation

• New configuration variables will need to be approved and added to the table before they are accepted, to ensure names remain standard – similar to the technical file approach.

• Requests for new names should be sent to: argo-dm-chairman@jcommops.org

• Configuration parameter values are stored as numerals• Any parameters with logical or string input will require an equivalent numeric

code to be added to the “Explanation” section of the parameter names table.• Currently, only 3 parameters are affected:

CONFIG_Direction_LOGICAL (Ascending = 1, Descending = 2)CONFIG_MeasureBattery_LOGICAL (Yes = 1, No = 0)CONFIG_TriProfileOption_LOGICAL (Yes = 1, No = 0)

• Units are irrelevant i.e. CONFIG_MissionPreludeTime can have units of _HOURS or _MINUTES or _HHMMSS, do not need a new parameter in each case

Configuration Parameters

• Typically, an Argo float configuration is valid for the life of the float (one configuration) CONFIG_MISSION_NUMBER = 0 (launch or pre-deployment info) and 1 (basic mission)

• For floats with multiple configurations, the configuration from the first cycle is set to 1. Each subsequent configuration change has a new mission number, i.e. from 1 to N.

• Argo best practice is a minimum of configuration missions. – i.e. if there is a change to configuration parameters that does not mirror a previous change then use a new

mission number.– If the configuration parameters change but mirror a previous mission, then that mission number should be

re-used (in complex cases where mission changes are unclear – then a new mission number can be used.

Mission Parameters

CONFIG_MISSION_NUMBER

• Each time a new mission number is added, the metafile will need to be rewritten.

MISSION_NUMBER 0 1 2 … …

CONFIG_PistonPositionPressureActivation_COUNT

100 100 100

CONFIG_ParkPressure_dBAR 1000 1500

CONFIG_ProfilePressure_dBAR 2000 2000

CONFIG_Direction_STRING 1* 2*

CONFIG_AscentToSurfaceTimeout_DecimalHour

3 3

CONFIG_ParkPistonPosition_COUNT 113 75

CONFIG_MeasureBattery_LOGICAL 0 ^ 0 ^

…

CONFIG_PARAMETER_NAME(N_CONF_PARAM)

Mission Settings(N_MISSIONS, N_CONF_PARAM)

*1=ascending, 2=descending, ^0=no, 1=yes

• All variables from Mission 1 must be repeated in subsequent missions even if they do not change.• If new variables are added in later missions, previous missions will need to be rewritten.

Floats with Multiple Missions

* N_CONF_PARAM = int value * N_MISSIONS = <unlimited>

Data Formats

We need a way to describe the float data format..

• A way to link the data manual and the ANDRO decoding information to a standard format ID for each float.

• This is important information for the trajectory processing and for decoding the raw data (delays!)

• Input needs standardisation.

• For some of the APEX APF9 floats there is no data format number on the manual – CSIRO has made up their own numbers, we need to standardise these.

• PROVOR float manuals also have no numbers and have two different kinds of format, i.e. layer or chords.

• need a copy of each version of the data manual for all float types, scanned and made available on the AIC or/and ADMT websites.

CSIRO: http://www.cmar.csiro.au/argo/dmqc/html/Australian_float_manuals.html

Float Data Formats

FIRMWARE_VERSION

MANUAL_VERSIONDAC_FORMAT_ID

JMA: http://argo.kishou.go.jp/document/float_type.html

SIO: http://sio-argo.ucsd.edu/manuals.html

Float Data Formats

DAC_FORMAT_ID ?

MANUAL_VERSION ? DAC_FORMAT_ID ?

• Some floats are returning version numbers e.g. SOLOII i.e. FIRMWARE_VERSION

• Progress with Online Formats…• INCOIS: Done – publically available?• Other DACS…??? estimated finish date, impediments?

• Require feedback from DACS as to what they are populating in their DAC_FORMAT_ID, FIRMWARE_VERSION and MANUAL_VERSION fields ?

• Working with the ANDRO team and Argo TC on a table linking the online float documentation, e.g. the DAC_FORMAT_ID field to the ANDRO decoder format ID and a proposed new STANDARD_FORMAT_ID field.

• Issues: ANDRO float decoders only exist for Argos floats where:- There is a description of the format (i.e. in a user’s manual)- There is original Argos data- ANDRO has implemented a decoder- Float data was transmitted before Jan 1st 2010- The float made a contribution to ANDRO (POPS are excluded)

• Iridium Apex floats have not been decoded, only Iridium Provors

Float Data Formats

FLOAT TYPE(1 = PROVOR)

2 = APEX3=SOLO WHOI4 = SOLO SIO

5 = NINJA)

STANDARD_FORMAT_ID

(at the moment we are using the

ANDRO Decoder id)

DAC_FORMAT_ID

(CSIRO)

WMO ID # example

2 63 43 19011192 15 11 59003322 64 28 59003392 65 1010 (01/31/08) 19011282 42 1010 (06/26/08) 59016752 9 1 59000262 66 1002 19011342 43 1003 59016492 67 1003 59032272 39 32 59008412 68 44 59011762 34 1001 59016362 69 31 59003452 70 35 59011652 -1 1004 59016772 71 0 535461 6 1 (provor) 59000361 7 1 (provor) 5900865

Float Data Formats• Proposed table linking DAC_FORMAT _ID to a STANDARD_FORMAT_ID

• Table hosted at AIC by Argo TC

• STANDARD_FORMAT_ID – should we use ANDRO decoder ID or do we need a new universal number ?i.e. "PPPNNN“, where PPP = a unique platform id for APEX, PROVOR etc and NNN = a made-up standard data id #

• manufacturers/operators to declare each new format at AIC and circulate to DACs (FAR BEFORE DEPLOYMENT)

•Operators could then register this information at float deployment notification step (via AIC web form), and include it in metafiles

• ANDRO Decoder’s do not exist for float data post Jan 1st 2010 – how do we fill this gap?

…_TYPE …_KEY STANDARD_FORMAT_ID ANDRO_ID DAC_IDSOLO 200 200001 SSI #67 SOLO_TS1

SOLO 200 200002

SSI #65

(only SB5.20 9 6 2000 floats)

SSI #66

(except SB5.20 9 6 2000 floats) SOLO_TS2

SOLO 200 200003 SSI #63 SOLO_TS9SOLO 200 200004 SSI #64 SOLO_TS9SOLO 200 200005 noneSOLO 200 200010 SSI #60, SSI #61 SOLO_TS12SOLO 200 200011 SSI #60, SSI #61 SOLO_TS12SOLO 200 200012 SSI #62 SOLO_TS12

Float Data FormatsSOLO Example

Work to be refined gradually ...

ALL DACS, PIS, manufactuers to participate !!!!

Standard Reference Tables

We need to agree on a set of new metadata reference tables..• PLATFORM_FAMILY (in manual)• PLATFORM_TYPE (in manual)• PLATFORM_MAKER (in manual)• SENSOR_MAKER (in manual)• SENSOR_MODEL (in manual)• SENSOR_UNITS (in manual)• SAMPLING_MODE (in manual)• CONTROLLER_BOARD_TYPE (in manual)• BATTERY_PACK (in manual)• BATTERY_PACKS (define approved abbreviations for battery types, in manual)• STANDARD_FORMAT_ID/DAC_FORMAT_ID/ANDRO_DECODER_ID – table linking all 3 will be hosted at AIC.• ARGO_GROUP (already exists, needs refinement)

Actions: On-line tables for drafting (see e.g. http://tinyurl.com/cweskt6 )TC will call community for contributions

Action Items

Action Item 36 – Configuration Parameter Names

Action Item 37 – Manual Updates completed (few small recent additional changes)

Action Item 38 – new format scheme devised after ADMT12 in Korea, DACS should now be able to generate metafiles correctly

Action Item 39 – decoder format to online documentation

Action Item 44 – Resubmit oxygen data in agreed format