A PPARC funded project Workflow in Astrogrid Jeff Lusted Dept Physics and Astronomy University of...

A PPARC funded project

Workflow in Astrogrid

Jeff Lusted

Dept Physics and Astronomy

University of Leicester

15 Dec 2004 Workflow in Astrogrid 1

Part One...

A Quick Tour of Workflow Architecture


What is the purpose of Workflow?

To accomplish a complex piece of work:

It takes more than one step to achieve.

The workflow is executed asynchronously.

Each step is executed asynchronously.


A Skeleton Workflow<workflow name="ThreeStepWorkflow" >

<sequence>

<Activity name="Step One" >

<tool name="query-6df">

...

</tool>

</Activity>

<Activity name="Step Two" >

<tool name="query-2Mass" >

...

</tool>

</Activity>

<Activity name="Step Three" >

<tool name="federator" >

...

</tool>

</Activity>

</sequence>

</workflow>


Expansion of Step One...

<Activity name="Step one" >

<description>...</description>

<tool name="query-6df" interface="adql">

<input>

<parameter name="Format" indirect="false">

<value >VOTABLE</value>

</parameter>

<parameter name="Query" indirect="true">

<value >ivo://...#jl99/query/6dfQuery.xml</value>

</parameter>

</input>

<output>

<parameter name="Result" indirect="true">

<value >ivo://...#jl99/votable/6dfQueryResults.vot</value>

</parameter>

</output>

</tool>

</Activity>


The View from 10,000 Metres

A workflows exist at:

Design Time

Execution Time

*** When submitted for execution, a workflow becomes a job. ***


Design Time – From 5000 Metres

A Design:

Is held as an XML file in MySpace.

The Design process:

Is managed by the Workflow Editor in the Portal.

Is aided by holding metadata in the Astrogrid Registry.


Execution Time – From 5000 Metres

Job Execution System:

Central controlling element.

Interprets the workflow design.

Manages the flow of control between Steps.

Common Execution Architecture:

Manages the local environment for each Step executed.

Manages the flow of data required by each Step.


Overview of Workflow Architecture


Part Two...

Workflow as a Programming Language


Back to Design...

Steps and Tools

Sequences and Flows.

Variables and Scopes.

Loops and Conditional Execution.

Scripting and Script Objects.


Sequence – Sequential Execution <sequence> <Activity name="Step One" > <tool name="query-6df"> ... </tool> </Activity> <Activity name="Step Two" > <tool name="query-2Mass" > ... </tool> </Activity> <Activity name="Step Three" > <tool name="federator" > ... </tool> </Activity> </sequence>


Flow – Parallel Execution <sequence> <flow> <Activity name="Step One" > <tool name="query-6df"> ... </tool> </Activity> <Activity name="Step Two" > <tool name="query-2Mass" > ... </tool> </Activity> </flow> <Activity name="Step Three" > <tool name="federator" > ... </tool> </Activity> </sequence>


Setting Variables

Setting...

Using variables to construct a filename...

<set var="namestub" value="${votable}" /><set var="filecount" value="${1}"/>

<output> <parameter name="result" indirect="true"> <value>${namestub}-${filecount + 1}</value> </parameter></output>


Conditional Execution

<set var="x" value="${1}" />...<if test="${x > 0}"> <then> <sequence> ... </sequence> </then> <else> ... </else></if>


While Loop

<while test="${results == null || results.size() < 1}">

<step result-var="results"> ... </step>

</while>


For Loop

<for var="x" items="${1...10}>

<step result-var="results"> ... </step>

</for>


Scripting

Scripting Expressions

Script

<set var="x" value="${1}" /><if test="${x > 0}"> ...</if>

<script> <body> print("Step produced " + ${count} + "votables"); </body></script>


Scripting Example plus Script Objects

<step result-var="results"> ...</step><set var="urlList" /><script> <body> if (results.size() != 1) { jes.error("didn't produce expected number of results"); } else { votable = results.get('votable'); parser = new XmlParser(); nodes = parser.parseText(votable); urlList = nodes.depthFirst() .findAll{it.name() == 'STREAM'} .collect{it.value()}.flatten(); print(urlList); } </body></script>


Conclusion – Astrogrid Workflow Capabilities

Distributed workflow executing on an asynchronous basis

Highly programmable...

... Workflows from the simple to the complex

Can integrate astronomical tools


Contacts.

Work Group Leader: Jeff Lusted [email protected]

Workflow and JES: Noel Winstanley [email protected]

Common Execution Architecture: Paul Harrison [email protected]

Workflow Editor: Phil Nicolson [email protected]

www.astrogrid.org

A PPARC funded project Workflow in Astrogrid Jeff Lusted Dept Physics and Astronomy University of...

Documents

Transcript of A PPARC funded project Workflow in Astrogrid Jeff Lusted Dept Physics and Astronomy University of...