Workflow sharing and integration services by the ER-flow project [email protected] on behalf...

16
Workflow sharing and integration services by the ER-flow project [email protected] on behalf of the ER-flow consortium EGI Community Forum, Manchester, 8-12 April, 2013

Transcript of Workflow sharing and integration services by the ER-flow project [email protected] on behalf...

Page 1: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Workflow sharing and integration services by the ER-flow project

[email protected]

on behalf of the ER-flow consortium

EGI Community Forum, Manchester, 8-12 April, 2013

Page 2: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Outline• Motivations

• SHIWA ER-flow

• Technology services• SHIWA Simulation Platform• Integrated workflow engines

• Example• Integrating Moteur & WS-PGRADE workflows

• User support services

2

Page 3: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

MotivationIsolation by technology

– Many different workflow systems exists We expect no change– Technological choice isolates users and user communities– Technological choice selects DCI/middleware (WS, gLite,

Globus, …)

Benefits of the SHIWA Simulation Platform– Share your own workflow, re-use workflows of others– Create and execute ‘meta-workflows’: built from smaller

workflows that use different workflow languages/technologies– Combine workflows and DCIs

SHIWA ER-flow– SHIWA: FP7 R&D project. Created the SHIWA Simulation Platform (2010-2012)– ER-flow: FP7 support action project. Disseminates the SHIWA technology (2012-

2014).3

Page 4: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Generic use-cases

DCI #1DCI #2

CE SECE SE

DCI – distributed computing infrastructure; CE/SE – computing/storage element

Page 5: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Generic use-cases

DCI #1DCI #2

CE SECE SE

combine WFs

find and re-use

with own data

on own CE?

DCI – distributed computing infrastructure; CE/SE – computing/storage element

Page 6: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

ER-flow project

6

University of Westminster UoW United KingdomMagyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete

MTA-SZTAKI Hungary

Centre National de la Recherche Scientifique CNRS FranceStichting European Grid Initiative EGI.eu The NetherlandsAcademic Medical Center of the University of Amsterdam AMC The Netherlands

Technische Universität Dresden TUD GermanyLudwig-Maximilians-Universität München LMU GermanyUniversity College London UCL United KIngdomTrinity College Daudublin TCD IrelandIstituto Nazionale di Astrofisica INAF Italy

Technology providers + User support:MTA-SZTAKI, UoW, CNRS, EGI.eu (through the NGIs)Research Communities:Astro-physics INAFComputational Chemistry LMU + TUDHeliophysics TCD + UCLLife Science AMC+ WE ARE LOOKING FOR NEW COMMUNITIESDuration:September 2012 – August 2014

Page 7: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Aims, services, actors

7

Aims:• Provide a platform for research communities that enables

seamless execution of workflows of different systems• Investigate data interoperability issues in the workflow domain

and propose solutions

Services:• SHIWA Simulation Platform• Dissemination and outreach• Training for workflow developers, workflow users

Key actors:• Workflow engine developers• Workflow developers• Researchers

Page 8: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

ER-flow technology:SHIWA Simulation Platform

8

Page 9: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Access to various workflow engines and DCIs where these WF applications can be run

CloudsLocal clusters

Supercomputers

Desktop grids(BOINC, Condor, etc.)

Cluster grids(EGI, OSG, …)

Supercomputer grids

(PRACE, XSEDE)

Grid systems

Distributed Computing Infrastructures (DCIs)

What does the developer of a (meta-)workflow need?

SHIWA Repository

Access to a large set of ready-to-run scientific workflow applications

SHIWA Portal

A portal/desktop to integrate, parameterize and run these applications

Web services

+ Workflow engines

Page 10: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

SHIWA Repository

Facilitates publishing and sharing workflows Supports:• Abstract workflows with

multiple implementations of over 10 workflow systems

• Concrete workflows with execution specific data

Available:• From the SHIWA Portalhttp://ssp.shiwa-workflow.eu• Standalone interface:http://repo.shiwa-workflow.eu

Page 11: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Example: Create meta-workflow from Moteur and WS-PGRADE workflows

SHIWA Repository

SHIWA portal

DCI 1gLite

DCI 2Cloud

DCI nARC

12

Access to Moteur WF

Connected workflow engines

Moteur WF engine

Page 12: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

An example: SHIWA solution for LINGA

Sub-WorkflowsManagement

13

Multi-Workflow

Page 13: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Computational Chemistry Applications in ER-flow

14

workflow system middleware VO

GROMACS Molecular Dynamic application

Energy Minimisation WS-PGRADE gLiTE/UniCore MoSGrid VO

Equilibration WS-PGRADE gLiTE/UniCore MoSGrid VO

Single TPR WS-PGRADE gLiTE/UniCore MoSGrid VO

CADDSuite docking application

Docking with ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VO

Docking without ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VO

NWChem Quantum Chemistry/Molecular Dynamics application

Geometry optimisation WS-PGRADE gLiTE/UniCore MoSGrid VO

Spectroscopic analysis WS-PGRADE gLiTE/UniCore MoSGrid VO

Parameter sweep WS-PGRADE gLiTE/UniCore MoSGrid VO

Transition state analysis WS-PGRADE gLiTE/UniCore MoSGrid VO

High throughput WS-PGRADE gLiTE/UniCore MoSGrid VO

Separate talks later today about Astro & Heliophysics workflows!

Page 14: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

ER-flow user support• SHIWA User Forum

– www.erflow.eu/shiwa-user-forum • Workshops:

– Workflow Application Porting Workshop, London 19-22 March 2013 http://www.erflow.eu/19-22-march-2013-application-porting-workshop

15

Page 15: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Conclusions• Scientific workflow landscape is fragmented

– Many workflow system with specialised & small user base

– Many workflows with specialised & small user base

• SHIWA Simulation Platform enables integration– Within disciplines– Across disciplines

• ER-flow project provides user support– Consultancy & Workshops– User forum– Open for new communities

16

Page 16: Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester,

Questions?

17

www.erflow.eu/shiwa-user-forum

www.erflow.eu