C. Huc/CNES, D. Boucon/CNES-SILOGIC Specification for the Formal Definition and Transfer Phase of a...

C. Huc/CNES, D. Boucon/CNES-SILOGIC

Specification for Specification for the Formal Definition and Transfer Phase the Formal Definition and Transfer Phase

of a Producer-Archive Interfaceof a Producer-Archive Interface

CCSDS Meeting, 02/11/04 Specification for the Formal Definition and Transfer Phase 2

ContentsContents

Introduction• Context• Objectives• General framework• Example

General principles for implementing Formal Definition and Transfer Phases

• From Formal Phase to Transfer Phase• Model of Objects to be Transferred (MOT)• SIPs• Sequencing constraints

Other steps

Conclusion


IntroductionIntroductionContextContext

Community MethodologicStandard

OAIS RMPAIMAS

Generic Méthodologic Standard

Specialisation rules

Specification for Formal Definition/Transfer Phases

Specialisation rules

Effective specialisation

Community Specification forFormal Definition/Transfer Phases

Effective specialisation


Provide a standard method to

• formally define the digital information objects to be transferred by a Producer to an Archive

• effectively transfer these objects in the form of SIPs

IntroductionIntroductionObjectivesObjectives


Preliminary Phase

Formal Definition Phase

Transfer Phase

ValidationPhase

IntroductionIntroductionGeneral framework: PAIMAS PhasesGeneral framework: PAIMAS Phases

Initial contacts, feasibility study, draft of the SIPs, Preliminary Agreement

Precise definition of the digital objects, SIP design, Submission Agreement

Actual transfer of the SIPsActual validation processing ofthe SIPs by the Archive and Required follow-up action with the Producer

•Organisational sub-phase•Formal definition sub-phase Complete and precise definition of the data to be transferred and the way it is organised in the form of a SIP Complementary points (contractual and legal aspects, …)•Submission Agreement

• implementing tests sub-phase• performing the transfer sub-phase (technical means –XFDU …)


Preliminary Phase


Transfer Phase

ValidationPhase

IntroductionIntroductionGeneral framework: PAIMAS PhasesGeneral framework: PAIMAS Phases

Initial contacts, feasibility study, draft of the SIPs, Preliminary Agreement

Precise definition of the digital objects, SIP design, Submission Agreement

Actual transfer of the SIPsActual validation processing ofthe SIPs by the Archive and Required follow-up action with the Producer

•Organisational sub-phase•Formal definition sub-phase Complete and precise definition of the data to be transferred and the way they are organised in the form of a SIP Complementary points (contractual and legal aspects, …)•Submission Agreement

• implementing tests sub-phase• performing the transfer sub-phase (technical means –XFDU …)


How can we define an object (and have a sufficiently precise description without creating the objects)?

How can we define the organisation between the objects (« Model of Objects to be Transferred » )?

How can we define a complete packaging description for an object and a group of objects?

How can we adapt the general framework to the domain specificities and project needs?

How can we specify the constraints (granularity and sequencing) applying to these objects?

How can we have a unique visual representation of the objects to transfer and their constraints?

IntroductionIntroductionGeneral framework: needs to meetGeneral framework: needs to meet


IntroductionIntroductionExampleExample

WIND

WAVES Experiment

WIND WAVES TNR L2 Data set

WIND WAVES TNR L2 Data (1..N)

Complementary Data Objects

EAST Description (1) Document (1)

Space Physics domain

•WIND mission•WAVES experiment •WIND WAVES TNR L2 data set•Complementary Data Objects:

EAST syntactic descriptor Mission description document


ContentsContents




Other steps

Conclusion


General principlesGeneral principlesFrom Formal Phase to Transfer PhaseFrom Formal Phase to Transfer Phase


Transfer Phase

Model of Objects to be Transferred:

SIPs definition

Transferred Objects in SIPs

MOT

Des

crip

tor

Sl i

p

Object Description

SIPs

Type of Object

Relations in the Model

Transfer information

Object

From Formal Phase to Transfer PhaseFrom Formal Phase to Transfer Phase


Transfer Phase

Standard Archive Project

Generic Descriptor Models

Archive Projectspecific Descriptor Models

Instantiated Descriptors

MOTGeneric SlipModels

Archive Projetspecific SlipModels

Delivered SIPs

Standardspecializationfor a Domain

DomainGeneric Descriptor Models

DomainGenericSlip Models

SIPsDefinition

InstantiatedSlips

Digital Objects


Model of Objects to be Transferred (MOT)

Gives a complete and overall view of all digital objects to be transferred: described by Descriptors.

Descriptor : This is an information unit for describing a set of characteristics for a given Data Object.

A Descriptor may come in the form of a model; in this case, it is used to identify the object category which can then be described as well as the attributes for this description.

The Descriptor may have the state of an instance; in this case, it describes an object instance and supplies the attribute values defined in the Descriptor model.

General principlesGeneral principlesModel of Objects to be Transferred: the DescriptorsModel of Objects to be Transferred: the Descriptors


Typology of information categories

Data ObjectsComplementary Data ObjectsCollectionsComplementary Collections

General principlesGeneral principlesModel of Objects to be TransferredModel of Objects to be Transferred




Transfer Phase


SIPs definition


MOT

Des

crip

tor

Sl i

p

Type of Object



Object Description

Object

SIPs

Data Objects, Complementary Data ObjectsCollections, Complementary Collections




Transfer Phase


4 Generic Descriptor Models

MOT

Des

crip

tor

Sl i

p

Type of Object



Object Description

Object

SIPs


SIPs definition




Name of the information object category

Name of the corresponding Descriptor model

Data Object DO_DESCRIPTOR

Complementary Data Object COMPLEMENTARY_DO_DESCRIPTOR

Collection COLLECTION_DESCRIPTOR

Complementary Collection COMPLEMENTARY_COLLECTION_ DESCRIPTOR


Descriptor Model: attributes (meaning, syntax, defined occurrence) for:

– Identifying the type of object described.– Defining the relations between the object described and the other

Model objects.– Describing the object content.– Giving a name to the Slip Model to be delivered.




DO_DESCRIPTORGeneric Model of a Data Object Descriptor

Attribute Meaning Attribute occurrence

DESCRIPTOR_ID Identifier of Data Object Descriptor 1..1

OBJECT_OCCURRENCE Number of Data Objects described by this DESCRIPTOR_ID

1..N

LABEL Extensive name of the Data Object 1..1

PARENT_COLLECTION Identifier of Collection Descriptor to which this object belongs

1..1

CONTENT Explanatory text specifying the content and main characteristics of the object (e.g. magnetic field, etc.)

0..1

SIZE Estimated volume of an object and unit 0..1

APPLICABLE_SLIP Name of slip model to be applied 1..1

DESCRIPTION_STANDARD Enumerated list (EAST, DED, etc.). Used to specify the data description standard to be applied.

0..N

METADATA_MODEL Enumerated list (XML Schema-name, etc). Used to specify the ‘catalogue’ data characterising the object: for instance in the form of an XML document specified by an XML Schema

0..N

OBJECT_FORMAT Format of objects in the collection (PNG, PDF, CDF, Flat Binary, Flat ASCII, …)

0..1




Transfer Phase


DESCRIPTOR_IDOBJECT_OCCURRENCELABEL

PARENT_COLLECTIONRELATION_DO_CDO

CONTENTFORMATSIZEMETADA_MODEL

APPLICABLE_SLIP SLIP_MODEL_ID


Des

crip

tor

Sl i

p

Type of Object



Object Description

Object


SIPs definition





Transfer Phase


4 Specialised Descriptor Models MOT

Des

crip

tor

Sl i

p

Type of Object



Object Description

Object Instantiated Descriptor Models


SIPs definition




General principlesGeneral principlesModel of Objects to be Transferred: Descriptor Model specialisationModel of Objects to be Transferred: Descriptor Model specialisation

Generic Model Domain Model Producer-Archive Project Model

•Addition of an attribute•Elimination of an attribute•Modification of existing attributes (content, occurrence, name of the attribute to adapt to the project terminology, syntax, …)


General principlesGeneral principlesModel of Objects to be Transferred: exampleModel of Objects to be Transferred: example

DO_DESCRIPTORSpecialized Model of a Data Object Descriptor for Space Physics


DESCRIPTOR_ID Identifier of Data Object Descriptor 1..1

OBJECT_OCCURRENCE Number of Data Objects described by this DESCRIPTOR_ID

1..N

LABEL Extensive name of the Data Object 1..1

PARENT_COLLECTION Identifier of Collection Descriptor to which this object belongs

1..1

CONTENT Explanatory text specifying the content and main characteristics of the object (e.g. magnetic field, etc.)

0..1

SIZE Estimated volume of an object and unit 1..1

APPLICABLE_SLIP Name of slip model to be applied 1..1

DESCRIPTION_STANDARD Enumerated list (EAST, DED, etc.). Used to specify the data description standard to be applied.

0..2

METADATA_MODEL Enumerated list (XML Schema-name, etc). Used to specify the ‘catalogue’ data characterising the object: for instance in the form of an XML document specified by an XML Schema

1..1

OBJECT_FORMAT Format of objects in the collection (PNG, PDF, CDF, Flat Binary, Flat ASCII, …)

1..1


Most crucial task during the Formal Definition Phase definition of the actual objects to be transferred:

– Identifying all of the project data Collections,– Organising these Collections in a hierarchical tree structure,– Defining a Descriptor instance for each of the Collections,– Defining a Data Object Descriptor for the objects in each Collection,

– …

Building the MOT

General principlesGeneral principlesModel of Objects to be Transferred: Descriptor instancesModel of Objects to be Transferred: Descriptor instances



Example of Descriptor instance for a level 2 Data Object of WAVES experiment on board WIND satellite (Space Physics)

Attribute Value

DESCRIPTOR_ID WIND_WAVES_TNR_L2_DATA

OBJECT_OCCURRENCE N

LABEL Daily file of Thermal Noise receiver data of WAVES experiment, WIND mission

PARENT_COLLECTION WIND_WAVES_TNR_L2

CONTENT Each data file contains:- Date and time- data of the Automatic gain control - high resolution spectra for each frequency band observed. The unit used for

the spectra is the microV2/Hz Each file covers one day of data from 0 to 24 hours.

SIZE 10..40 megabytes

APPLICABLE_SLIP TNR_L2_DO

DESCRIPTION_STANDARD EAST, DEDSL

METADATA_MODEL CDPP_DO.xsd

OBJECT_FORMAT Flat_binary



ROOT

WIND_WAVES

WIND_WAVES_TNR_L2METADATA_STANDARD = DIF

WIND_WAVES_TNR_L2_DATA (N)METADATA_MODEL = CDPP_DO.xsd

WIND_WAVES_CC

EAST_DESCRIPTION (1) DOCUMENTATION (1)

COLLECTION_ID

COMPLEMENTARY_COLLECTION_ID

COLLECTION_DESCRIPTOR

COLLECTION_DESCRIPTOR

COMPLEMENTARY_COLLECTION_DESCRIPTOR

DO_DESCRIPTORCOMPLEMENTARY_DO_DESCRIPTOR COMPLEMENTARY_DO_DESCRIPTOR

Descriptors to be created during Formal Definition Phase


Slip: A Data Object that conveys information related to another Data Object which is transferred between two entities. This information will allow the addressee to identify, locate and if needed process the various transferred bit sequences which compose this Data Object.

The Slip is the packaging of the transferred Object.The Slip is derived by a Model referenced by the APPLICABLE_SLIP (object

Descriptors).Several types of Slips depending on the type of the objects.Slip Models are defined during the Formal Definition Phase and instantiated

during the Transfer Phase.

General principlesGeneral principlesModel of Objects to be Transferred: delivery SlipModel of Objects to be Transferred: delivery Slip




Transfer Phase

Model of Objects to be Transferred: Data Objects, Complementary Data ObjectsCollections, Complementary Collections

SIPs definition







Des

crip

tor

Sl i

p

Type of Object



Object Description

Object

SIPs

1 Generic Slip Model



Name of information objects class

Name of corresponding Slip Model

Data Object TRANSFERRED_OBJECT_SLIP

Complementary Data Object

Collection

Complementary Collection



TRANSFERRED_OBJECT_SLIPGeneric Slip Model


SLIP_MODEL_ID Slip type identifier. This attribute has a constant value in the model. 1..1

OBJECT_ID Object identifier (Data Object, Complementary Data Object, Collections) 1..1

DESCRIPTOR_ID Identifier of the Object Descriptor to which the object refers. This attribute is used to locate this object in the MOT.

1..1

TRANSFERRED_DO_FILES Data file(s) which make up this object. This attribute is of the TRANSFERRED_FILES_DESCRIPTION type.

0..N

LAST_OBJECT Indicator specifying that it is the last object (among the Data Objects, Complementary Data Objects or Collections corresponding to the same MOT node).

0..1

TRANSFERRED_METADATA_FILES Metadata file(s) accompanying this object. This attribute is of the TRANSFERRED_FILES_DESCRIPTION type.

0..N



TRANSFERRED_FILES_DESCRIPTIONComposite attribute for file transfer


TRANSFERRED_FILE Name of the file to be transferred 1..N

DO_EXTRACTION_SOFTWARE Name of software used to reconstitute the Data Object from different files

0..1

DO_EXTERNAL_REFERENCE External reference of files constituting this Data Object if these files are not included during transfer but deposited in a shared space.

0..N




Transfer Phase

Model of Objects to be Transferred: Data Objects, Complementary Data ObjectsCollections, Complementary Collections

SIPs definition







Des

crip

tor

Sl i

p

Type of Object



Object Description

Object

Specialisation

SIPs

OBJECT_IDTRANSFERRED_DO_FILESLAST_OBJECTTRANSFERRED_METADATA_FILES


Instantiated Slips and


General principlesGeneral principlesModel of Objects to be Transferred: Delivery SlipModel of Objects to be Transferred: Delivery Slip

Generic Model Domain Model Producer-Archive Project Model

•Addition of an attribute•Elimination of an attribute•Modification of existing attributes (change of name occurrence, …)



Example of Slip Model for a Data Object for Space PhysicsSPACE_PHYSICS_SLIP


SLIP_MODEL_ID TNR_L2_DO 1..1

OBJECT_ID Object identifier 1..1

DESCRIPTOR_ID Identifier of the Object Descriptor to which the object refers. This attribute is used to locate this object in the MOT.

1..1

TRANSFERRED_DO_FILES Data file(s) which make up this object. This attribute is of the TRANSFERRED_FILES_DESCRIPTION type.

0..N

LAST_OBJECT Indicator specifying that it is the last object of the Collection 0..1

START_DATE Start date for the Data Object 1..1

STOP_DATE End date for the Data Object 1..1


The Slip provides information for reconstituting the Data Object from the different data containers used for the transfer.

Each Slip instance refers to 1 object Descriptor identifier.

General principlesGeneral principlesModel of Objects to be Transferred: Slip Model instancesModel of Objects to be Transferred: Slip Model instances



Example of instance of Slip for a Data Object for Space PhysicsSPACE_PHYSICS_SLIP

Attribute Value

SLIP_MODEL_ID TNR_L2_DO

OBJECT_ID Wind_waves_tnr_l2_20040610


TRANSFERRED_DO_FILES TRANSFERRED_FILE = Wind_waves_tnr_l2_20040610.datTRANSFERRED_FILE = Wind_waves_tnr_l2_20040610.xml

LAST_OBJECT FALSE

START_DATE 2004/06/10-00 :00 :00

STOP_DATE 2004/06/10-23 :59 :59


General principlesGeneral principlesModel of Objects to be Transferred: example of Model representationModel of Objects to be Transferred: example of Model representation

ROOT

WIND_WAVES

WIND_WAVES_TNR_L2METADATA_STANDARD=DIF

WIND_WAVES_TNR_L2_DATA (N)METADATA_MODEL=CDPP_DO.xsl

WIND_WAVES_CC

EAST_DESCRIPTION (1) DOCUMENTATION (1)

Object Descriptor ID

SPACE_PHYSICS_SLIPWind_waves_tnr_l2_aaammjj.datWind_waves_tnr_l2_aaammjj.xml

EAST_SLIPWind_waves_tnr_l2.east

EXPERIMENT_DESCRIPTION_SLIPWind_waves_22000_0387.pdf

SPACE_PHYSICS_COLLECTION_SLIPWind_waves_tnr.dif

N

Files to be transferred


ContentsContents




Other steps

Conclusion


Group of elementary packages in a package to be transferred (objects of the same or different types).

Defined during the Formal Definition Phase:– Rules for grouping objects.– SIP categories for the project

General principlesGeneral principlesSIPsSIPs


General principlesGeneral principlesSIPsSIPs

SIP_DESCRIPTORGeneric SIP Model


SIP_MODEL_ID SIP type identifier 1..1

DESCRIPTOR_ID Object Descriptor identifier contained in the SIP.

1..N

SIP1: Collection WIND_WAVES_TNR_L2 SIP2: Complementary Objects EAST file and document SIP3: Data Objects

Example of SIP3 Descriptor instance

Attribute Value

SIP_MODEL_ID SIP3



A given digital object may have to be transmitted before or after another.

Constraint group: set of objects related to each other by a set of dependent sequencing constraints.

General principlesGeneral principlesSequencing ConstraintsSequencing Constraints

Attribute name Meaning Value syntax Occurrence

TIME_CONSTRAINT_GROUP define the groups identifier 0..1

SERIAL_NUMBER_IN_CONSTRAINT define the constraints within the same group

integer 0..1


General principlesGeneral principlesSequencing ConstraintsSequencing Constraints

SIP1: Collection WIND_WAVES_TNR_L2SIP2: Complementary Objects EAST file and document: is delivered before any delivery of SIP3.SIP3: Data Objects

Example of SIP3 instance

Attribute Value

SIP_MODEL_ID SIP3

SIP_CONTENT WIND_WAVES_TNR_L2_DATA

TIME_CONSTRAINT_GROUP Group_1

SERIAL_NUMBER_IN_CONSTRAINT 2

Example of SIP2 instance

Attribute Value

SIP_MODEL_ID SIP2

DESCRIPTOR_ID EAST_DESCRIPTION

DESCRIPTOR_ID DOCUMENTATION

TIME_CONSTRAINT_GROUP Group_1

SERIAL_NUMBER_IN_CONSTRAINT 1

1

2


General principlesGeneral principlesOverall viewOverall view


Transfer Phase


4 Specialised Descriptor Models MOT

Des

crip

tor

Sl i

p

Type of Object



Object Description

Object


SIPs definition



Specialisation

SIPs

OBJECT_IDTRANSFERRED_DO_FILESLAST_OBJECTTRANSFERRED_METADATA_FILES


Instantiated Slips and

APPLICABLE_SLIP

Instantiated Descriptor Models


General principlesGeneral principlesOverall viewOverall view

Generic Descriptor Model

Generic Slip Model

Specialised Descriptor Model

Specialised Slip Model

Instantiated Descriptor Instantiated Slip

Attribute = DESCRIPTOR_ID

……

Attribute = APPLICABLE_SLIP

Attribute = SLIP_MODEL_IDAttribute = OBJECT_IDAttribute = DESCRIPTOR_ID……Attribute = TRANSFERRED_FILE

Attribute = DESCRIPTOR_ID

……

Attribute = APPLICABLE_SLIP

DESCRIPTOR_ID = WIND_WAVES_TNR_L2_DATA

……

APPLICABLE_SLIP = TNR_L2_DO

Attribute = SLIP_MODEL_ID = TNR_L2_DO

Attribute = OBJECT_IDAttribute = DESCRIPTOR_ID……Attribute = TRANSFERRED_FILE

SLIP_MODEL_ID = TNR_L2_DO

OBJECT_ID = Wind_waves_tnr_l2_20040610

DESCRIPTOR_ID = WIND_WAVES_TNR_L2_DATA

……TRANSFERRED_FILE = Wind_waves_tnr_l2_20040610.dat

Wind_waves_tnr_l2_20040610.xml

Wind_waves_tnr_l2_20040610


ContentsContents




Other steps

Conclusion


Formal Definition Phase detailed specification

Transfer Phase specification

Managing modifications

Tools

Other stepsOther steps

To be developed


How can we define an object (and have a sufficiently precise description without creating the objects)?

– Descriptor and attributes

How can we define the organisation between the objects (« Model of Objects to be Transferred « )?

– Descriptors and relation definitions between objects

How can we define a complete packaging description for an object and a group of objects?

– Slips and SIPs

How can we adapt the general framework to the domain specificities and project needs?

– Specialisation process

How can we specify the constraints (granularity and sequencing) applying to these objects?

– TIME_CONSTRAINT_GROUP, SERIAL_NUMBER_IN_CONSTRAINT attributes

How can we have a unique visual representation of the objects to deliver and their constraints?

– To be discussed

ConclusionConclusionNeeds to meet: some answersNeeds to meet: some answers


ConclusionConclusion

Reach an Agreement on– Terminology,– Concepts (Descriptors, Slips, SIPs, sequencing

constraints),– Process,

Before any implementation phase.

C. Huc/CNES, D. Boucon/CNES-SILOGIC Specification for the Formal Definition and Transfer Phase of a...

Documents

Transcript of C. Huc/CNES, D. Boucon/CNES-SILOGIC Specification for the Formal Definition and Transfer Phase of a...