Workflow sharing and integration services by the ER-flow project [email protected] on behalf...
-
Upload
norma-poole -
Category
Documents
-
view
212 -
download
0
Transcript of Workflow sharing and integration services by the ER-flow project [email protected] on behalf...
![Page 1: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/1.jpg)
Workflow sharing and integration services by the ER-flow project
on behalf of the ER-flow consortium
EGI Community Forum, Manchester, 8-12 April, 2013
![Page 2: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/2.jpg)
Outline• Motivations
• SHIWA ER-flow
• Technology services• SHIWA Simulation Platform• Integrated workflow engines
• Example• Integrating Moteur & WS-PGRADE workflows
• User support services
2
![Page 3: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/3.jpg)
MotivationIsolation by technology
– Many different workflow systems exists We expect no change– Technological choice isolates users and user communities– Technological choice selects DCI/middleware (WS, gLite,
Globus, …)
Benefits of the SHIWA Simulation Platform– Share your own workflow, re-use workflows of others– Create and execute ‘meta-workflows’: built from smaller
workflows that use different workflow languages/technologies– Combine workflows and DCIs
SHIWA ER-flow– SHIWA: FP7 R&D project. Created the SHIWA Simulation Platform (2010-2012)– ER-flow: FP7 support action project. Disseminates the SHIWA technology (2012-
2014).3
![Page 4: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/4.jpg)
Generic use-cases
DCI #1DCI #2
CE SECE SE
DCI – distributed computing infrastructure; CE/SE – computing/storage element
![Page 5: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/5.jpg)
Generic use-cases
DCI #1DCI #2
CE SECE SE
combine WFs
find and re-use
with own data
on own CE?
DCI – distributed computing infrastructure; CE/SE – computing/storage element
![Page 6: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/6.jpg)
ER-flow project
6
University of Westminster UoW United KingdomMagyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete
MTA-SZTAKI Hungary
Centre National de la Recherche Scientifique CNRS FranceStichting European Grid Initiative EGI.eu The NetherlandsAcademic Medical Center of the University of Amsterdam AMC The Netherlands
Technische Universität Dresden TUD GermanyLudwig-Maximilians-Universität München LMU GermanyUniversity College London UCL United KIngdomTrinity College Daudublin TCD IrelandIstituto Nazionale di Astrofisica INAF Italy
Technology providers + User support:MTA-SZTAKI, UoW, CNRS, EGI.eu (through the NGIs)Research Communities:Astro-physics INAFComputational Chemistry LMU + TUDHeliophysics TCD + UCLLife Science AMC+ WE ARE LOOKING FOR NEW COMMUNITIESDuration:September 2012 – August 2014
![Page 7: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/7.jpg)
Aims, services, actors
7
Aims:• Provide a platform for research communities that enables
seamless execution of workflows of different systems• Investigate data interoperability issues in the workflow domain
and propose solutions
Services:• SHIWA Simulation Platform• Dissemination and outreach• Training for workflow developers, workflow users
Key actors:• Workflow engine developers• Workflow developers• Researchers
![Page 8: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/8.jpg)
ER-flow technology:SHIWA Simulation Platform
8
![Page 9: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/9.jpg)
Access to various workflow engines and DCIs where these WF applications can be run
CloudsLocal clusters
Supercomputers
Desktop grids(BOINC, Condor, etc.)
Cluster grids(EGI, OSG, …)
Supercomputer grids
(PRACE, XSEDE)
Grid systems
Distributed Computing Infrastructures (DCIs)
What does the developer of a (meta-)workflow need?
SHIWA Repository
Access to a large set of ready-to-run scientific workflow applications
SHIWA Portal
A portal/desktop to integrate, parameterize and run these applications
Web services
+ Workflow engines
![Page 10: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/10.jpg)
SHIWA Repository
Facilitates publishing and sharing workflows Supports:• Abstract workflows with
multiple implementations of over 10 workflow systems
• Concrete workflows with execution specific data
Available:• From the SHIWA Portalhttp://ssp.shiwa-workflow.eu• Standalone interface:http://repo.shiwa-workflow.eu
![Page 11: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/11.jpg)
Example: Create meta-workflow from Moteur and WS-PGRADE workflows
SHIWA Repository
SHIWA portal
DCI 1gLite
DCI 2Cloud
DCI nARC
12
Access to Moteur WF
Connected workflow engines
Moteur WF engine
![Page 12: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/12.jpg)
An example: SHIWA solution for LINGA
Sub-WorkflowsManagement
13
Multi-Workflow
![Page 13: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/13.jpg)
Computational Chemistry Applications in ER-flow
14
workflow system middleware VO
GROMACS Molecular Dynamic application
Energy Minimisation WS-PGRADE gLiTE/UniCore MoSGrid VO
Equilibration WS-PGRADE gLiTE/UniCore MoSGrid VO
Single TPR WS-PGRADE gLiTE/UniCore MoSGrid VO
CADDSuite docking application
Docking with ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VO
Docking without ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VO
NWChem Quantum Chemistry/Molecular Dynamics application
Geometry optimisation WS-PGRADE gLiTE/UniCore MoSGrid VO
Spectroscopic analysis WS-PGRADE gLiTE/UniCore MoSGrid VO
Parameter sweep WS-PGRADE gLiTE/UniCore MoSGrid VO
Transition state analysis WS-PGRADE gLiTE/UniCore MoSGrid VO
High throughput WS-PGRADE gLiTE/UniCore MoSGrid VO
Separate talks later today about Astro & Heliophysics workflows!
![Page 14: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/14.jpg)
ER-flow user support• SHIWA User Forum
– www.erflow.eu/shiwa-user-forum • Workshops:
– Workflow Application Porting Workshop, London 19-22 March 2013 http://www.erflow.eu/19-22-march-2013-application-porting-workshop
15
![Page 15: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/15.jpg)
Conclusions• Scientific workflow landscape is fragmented
– Many workflow system with specialised & small user base
– Many workflows with specialised & small user base
• SHIWA Simulation Platform enables integration– Within disciplines– Across disciplines
• ER-flow project provides user support– Consultancy & Workshops– User forum– Open for new communities
16
![Page 16: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,](https://reader038.fdocuments.us/reader038/viewer/2022110403/56649e615503460f94b5c6d4/html5/thumbnails/16.jpg)
Questions?
17
www.erflow.eu/shiwa-user-forum
www.erflow.eu