Post on 12-Jan-2016
description
Dominik Stokłosa
Poznań Supercomputing and Networking Center, Supercomputing Department
INGRID 2008 Lacco Ameno, Island of Ischia , ITALY, April 9-11
Workflow management in Remote Instrumentation Infrastructure – based on e-VLBI experiences
Introduction to the e-VLBIIntroduction to the e-VLBI
PSNC in EXPReS - FABRIC PSNC in EXPReS - FABRIC
Grid – VLBI design & technologyGrid – VLBI design & technology
SummarySummary
Outline
Introduction to the e-VLBIIntroduction to the e-VLBI
VLBI is a technique, in which physically independent and widely
separated radio telescopes observe the same region of sky
simultaneously, in order to generate very high-resolution
continuum and spectral-line images of cosmic radio sources
Telescopes are usually separated by thousands of kilometres
Data from each telescope are digitally sampled and stored locally,
using high-capacity magnetic tape systems and magnetic disk-array
systems
Data are sent and correlated at the central point (JIVE)
The total flow of data into the central processor is approximately 10-
100 Terabytes per single observation, after processing this is reduced
to 10-100 Gbytes.
Introduction to EXPReSIntroduction to EXPReS
EXPReS – the objective is to create a production-level
“electronic” VLBI (e-VLBI) service, in which the radio telescopes
are reliably connected to the central data processor at JIVE via a
high-speed optical-fibre communication network.
Project Details
Three years, started March 2006
International collaboration
Funded at 3.9 million EUR
FP6, Contract #026642
PSNC in EXPReSPSNC in EXPReS
EXPReS – a Real-time e-VLBI Radio Telescope
- JRA1: Future Arrays of Broadband Radio-Telescopes
on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management
- Grid Routing
Creating solution for incorporating Grid resources for
distributed correlation using existing infrastructure.
Once upon a time (1)Everything was slow Telescopes collected data on tapes Sent via postal mail
Hard drive arrays slightly improved the situation
The entire cycle could easily require 6 months or more
Once upon a time (2)
Hardware correlator; the EVN MkIV data correlator at JIVE dedicated, purpose designed/built hardware a super computer; ~100 T ops/sec
Today / In the near future
Data can be transferred over the network
Each stage of the process can be speeded up
GRID resources
Software correlator
e-VLBI - electronic VLBI
e-VLBI System - architecture
e-VLBI System – control flow
Workflow Manager module (WFM)
CCF – correlator configuration file
WFM – phase 1
Definition of radio telescopes – automatically based on theobservation schedule
WFM – phase 2
Definition of file servers (each file sever is responsible for capturing data from RT)
WFM – phase 3
Definition of correlation nodes and data flows between components
WFM – properties
Definition of resource properties
WFM (1)
The WFM application has several purposes:
•Workflow creation – a user is given a list of building blocks. Each block represents the available resource. It is the user’s responsibility to design and construct the experiment’s workflow from the available building blocks.
•Workflow submission – the scenario prepared in the previous step can be submitted to the system.
•Workflow management and monitoring – allows to manage the executed workflow.
WFM (2)
Obrazek z WFM -> przeplywy danych
Link definition
Type definition
Radio telescope
File server
Correlator
Properties definition
Dynamic forms
Parameters stored in Json
format
PSNC in EXPReSPSNC in EXPReS
EXPReS – a Real-time e-VLBI Radio Telescope
- JRA1: Future Arrays of Broadband Radio-Telescopes
on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management
- Grid Routing
Creating solution for incorporating Grid resources for
distributed correlation using existing infrastructure.
Thank you for your attention
http://www.expres-eu.org/d.stoklosa@man.poznan.pl
EXPReS is made possible through the support of the EC, 6th FP, Contract #026642