CSTS-File Transfer service discussions (2)
description
Transcript of CSTS-File Transfer service discussions (2)
19/05/2011 CSTS File transfer service discussions
CSTS-File Transfer service discussions (2)
CNES position
19/05/2011 CSTS File transfer service discussions
• File Content Description– Define manifest files identifying
– File content (to the level that the recipient needs to know)– File format?
• Processing Instructions– Combine with content description?
• Pull Model, Push Model, or both?• [Mouy Francois-Xavier] No needs identified yet for push model (but still a little bit of work to do)
• Subscription and automatic push? [Mouy Francois-Xavier] for push model • Subscription and notification of new data? [Mouy Francois-Xavier] how send notifications if
provider not bind. Bind return could notify a list of avaliable data
• File Management Services (see dedicated slide)• File Directory / Catalogue Servcies• File Management [Mouy Francois-Xavier] for push model
– Create / Delete– Move– Local copy
• Directory Management [Mouy Francois-Xavier] for push model – Create / Delete / Copy
• Storage Control [Mouy Francois-Xavier] needs to be configurable – Duration of Storage– Maximum Volume (per client?)
19/05/2011 CSTS File transfer service discussions
• Security (see dedicated slide)• Off the shelf FT protocols support security but must be configured
and depending on features used may need supporting infrastructure• Firewall issues• Access Control
– At what level? (mission, individual accounts, …) [Mouy Francois-Xavier] Mission level seem to be the right level
– Access Control Groups – More complex schemes
• What level of privacy / access control is needed? [Mouy Francois-Xavier] We need to be able to provide this kind of features before security person's asks – E.g. can user X see that data are stored for user Y? [Mouy Francois-
Xavier] he shouldn't ;-) • Key management issues• Others?
19/05/2011 CSTS File transfer service discussions
Reading Martin initial thoughts.– Protocol choices are wide. If we use one or other
protocol, it comes with all this complexity (advantages, drawbacks).
– Some features of standards COTS are not useful in our context, but you have to tune and secure those features anyway.
– Most of the difficulty come with the push model (security, behavior), the pull model is very near the existing offline model.
Initial CNES position:– Firstly, we thought we only need the pull model which
is quite simple than the push model. – Furthermore, File transfer function is already done by
other ways. (ftp servers in parallel)– Therefore, why redo it ?
19/05/2011 CSTS File transfer service discussions
• However, Experience show us that deploying a file transfer feature with common COTS is not quite easy that it is on paper (e.g. with ftp: passive form, active , dynamic port ranges, firewalls, user management, … ).
• Using CSTS could simplify the integration because the network security aspects are already done. A part of the reporting too (notifications).
• CNES think CSTS could be a good « Secured Pipe » to the Ground center.– A binded instance is maintained (Heartbeat)– The Start/Stop mechanisms could be useful to drive
providers behavior– We are able to use CSTS services via a high level of
firewalls system
19/05/2011 CSTS File transfer service discussions
Candidate standards (from initial thoughts)
• File Transfer Standards– FTP, FTPS, SFTP (SSH based FT)– HTTP / HTTPS ?– Others?
• CCSDS Standards– XFDU ?– Others?
19/05/2011 CSTS File transfer service discussions
XFDU (xml formatted data unit)
• GOAL : organizing transferred data in an Open Archive environment (OAIS)
A container (zip file) with
a descripition of its content in a manifest file.
The logical description points to the physical content which can be either local or distant
19/05/2011 CSTS File transfer service discussions
XFDU description
• The Information Package Map contains the logical view of the package. It’s a hierarchical xml tree representing the content of the package. Each leaf of the tree is a Content Unit and can be referred to from the other parts of the package (i.e. information is linked by the Content Unit reference ID).
• The Data Object Section contains all the physical information needed to get the file objects out.
• The Metadata Section records all of the metadata for all items in the package.
• The Behavior Section associates executable code with the content of the package.
19/05/2011 CSTS File transfer service discussions
XFDU manifest structure
The structure map is a hierarchical view of logical references which are pointing to physical data, metadata and specific behaviors.
19/05/2011 CSTS File transfer service discussions
XFDU : benefits
• Unique Identification of the data ( ContentUnit ID)
• Interoperability by using an XML schema description
• Hierarchical structure of the information• Data physical access (DataObject) separated
from the logical concepts (ContentUnit)• Metadata associated with the ContentUnit• Specific processes associated with the
ContentUnit (BehaviorObject)
19/05/2011 CSTS File transfer service discussions
XFDU : drawbacks
XFDU developed for approaching needs but not totally the sames (multi-parts, files, cdrom)
• Few implementations due to standard complexity.:– NASA XFDU library, ESA SAFE’s archive for
ENVISAT data, CNES’s PAIMAS
• Behavior function not really validated
19/05/2011 CSTS File transfer service discussions
CSTS-XFDU : conclusion
• The data organization has been studied earlier by the MOIMS/IPR group. XFDU is an answer to csts concerns with the following conditions: – Needs of an XFDU « light » to answer the
requirements.– Needs to study the behavior statement.
.
19/05/2011 CSTS File transfer service discussions
CSTS-XFDU : conclusion
• XFDU (simplified one) Could be a good system to integrate the file transfer feature.
• The behavior side of XFDU could be a response to File transfer and more …
• CNES thinks it could be a possible cooperation and some people are interested to study those aspects …
19/05/2011 CSTS File transfer service discussions
19/05/2011 CSTS File transfer service discussions
Some use cases at CNES
• Distant management of stations– Management by xfdu of TM archived in station– TM playback via CSTS-offline service
• Inter-facilities scheduler– Distant scripts used by the scheduler managed by
xfdu– Launch via services CSTS– Re-use of CNES well-known scheduler GUI
• …
19/05/2011 CSTS File transfer service discussions
Let’s take an example….
XFDU over CSTS for playing recorded telemetry
19/05/2011 CSTS File transfer service discussions
Telemetry transfert and play
• A facility wants to transfert simulated telemetry to a station and to be able to play it back later.
• The user build an xfdu package (X1) with the TM file and two scripts (play and rewind)
• All the configuration and security concerns to be solved by the management layer.
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
bind
STEP 1
CONNEXION
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
bind
START PUT X1
STEP 2
XFDU TRANSFERT
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
INSTALL
STEP 3
XFDU AUTOMATICINSTALL
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
START LIST
START LIST X1
PLAY, REW
X1
INSTALL
STEP 4
The user lists all available xfdus
and their behaviors
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
START LIST
START LIST X1
START RUN X1 PLAY
TM 1
PLAY, REW
X1
INSTALL
STEP 5
The user plays the TM1 telemetry back