A Chemistry-Inspired Workflow Management System for Scientific Applications on Clouds
-
Upload
hector-fernandez -
Category
Technology
-
view
472 -
download
1
description
Transcript of A Chemistry-Inspired Workflow Management System for Scientific Applications on Clouds
A Chemistry-Inspired Workflow ManagementSystem for Scientific Applications in Clouds
Hector Fernandez, Cedric Tedeschi and Thierry Priol 00 MOIS 2011
7th IEEE International Conference on e–ScienceStockholm 2011
Context
• Scientific applications developed as workflows demanding more computational power. Demand for deployment on Grids or Clouds.
• Scientific workflow management systems (WMS): Implicit parallelism. Data-driven coordination. Support for the execution on Grids.
• Examples of Scientific WMS: Taverna, Pegasus, Triana and Kepler.
• Requirements of next generation Scientific WMS:• Management of high degree of parallelism and distribution.
• No single point of failure.
• Scalability.• Dynamicity.
2
Intr
od
uct
ion
Objectives
3
• Ensure a workflow execution:• Decentralized.• Loosely coupled (coordination mechanism).• Dynamic.• Autonomous.
“Nature-inspired metaphors have been shown to be of high interest for service coordination.”
[Viroli et al., 2009].
➔ Evaluate the viability of a nature-inspired scientific workflow system.
Intr
od
uct
ion
Chemical Programing Model (I)
4
• A program can be seen as a chemical solution:• Data: “floating” molecules in the solution.• Computation: chemical reactions between the molecules.
• Implicit parallelism and autonomy of reactions until inertia.• Expression of dynamicity.
• Data structure: Multiset (blackboard).• Containing all data molecules.• Reaction rules re-writing the multiset.
• Languages:• Gamma (Pioneered model) [Banâtre et al.,1990].• HOCL ( High-Order model) [Radenac, 2007].
Ch
emic
al P
rog
ram
ing
par
adig
m
Chemical Programing Model (II)
5
• Example:• A reaction rules is written
replace-one P by M if C
where P is a pattern which matches the required molecule, C is the reaction condition and M the result of the reaction.
Ch
emic
al P
rog
ram
ing
par
adig
m
6
HOCL-based Workflow System
Chemical Coordination: Workflow Definition
7
• Express all data and control dependencies (reaction rules and molecules).
• Molecular composition to express the logic of a workflow.
Ch
emic
al C
oo
rdin
ati o
n M
od
el
MULTISET
Chemical Coordination: Generic Rules
8
• Independent from any chemical workflow representation.• Used by chemical engines.
• Common tasks during a workflow execution:• Service invocation rule.
• Control and data transfer rule.Ch
emic
al C
oo
rdin
ati o
n M
od
el
Chemical Coordination: Workflow Patterns
9
• Control flow can be expressed using some generic rules.
• Molecular composition of composed generic rules, reactions triggering reactions.
• More patterns: parallel split, synchronization, exclusive choice, synchronization merge, cancel activity or simple merge.
Ch
emic
al C
oo
rdin
ati o
n M
od
el
Discriminator pattern
10
Architectures
• Coordination mechanism built upon HOCL.
• Two possible architectures for our workflow system:• Centralized.• Decentralized.
Centralized Architecture
11
• Central node coordinates all data and control flow between the Web services.• A chemical encapsulation per Web service participating in the workflow.• Multiset as storage space containing the workflow definition.• Chemical engine processing the content of the multiset.
Ch
emic
al W
or k
flo
w S
yste
m
Decentralized Architecture (I)
12
• Nodes communicating through a shared address space.• Persistent.• Fault-tolerant.
• Workflow executed in parts corresponding with each Web service.• Data and control transfer through this shared space.• Each node is co-responsible of the execution.
Ch
emic
al W
or k
flo
w S
yste
m
Decentralized Architecture (III)
13
• Multiset, dynamic and decentralized coordination mechanism.• Acts as a shared address space containing both control and data flows.• ChWSes communicate through the multiset. (reading and writing)• Physically distributed over ChWSes storage spaces.
Ch
emic
al W
or k
flo
w S
yste
m
14
Implementation
Centralized Prototype
15
• Service caller • Interface with all the concrete Wses.• Implemented based on Daios framework.
• HOCL Interpreter • Central engine.
• Multiset • Workflow definition.• Processed by the HOCL Interpreter.
Ch
emic
al W
or k
flo
w S
yste
m
Decentralized Prototype
16
• Chemical Web Services (ChWS):
• Service caller Interface with one concrete WS.
• Local Multiset Temporary store space.
• HOCL Interpreter Local workflow engine.
• JMS publisher/subscriber Communication module with the Multiset.
• Multiset:
• Storage space containing the whole workflow.
• Similarities with tuplespaces.
• JMS publisher/subscriber Communication module with the ChWSes.
Ch
emic
al W
or k
flo
w S
yste
m
17
Experiments
Experiments (I)
18
• Objective: Establish the viability of our chemical workflow engine in comparison with four WMS.
• Four workflow engines:
• Kepler 2.0.
• Taverna Workbench 2.2.0.
• Centralized prototype (HOCL Cen.).
• Decentralized prototype (HOCL Dec.).
• Real scenarios:
• Cardiovascular image analysis workflow (CardiacAnalysis) [7].
• Astronomical image mosaics workflow (Montage) [8].
• Bio-informatics workflow (BlastReport) [9].
• Experiments conducted on the French research infrastructure Grid'5000.
Per
form
ance
Res
ult
s
CardiacAnalysis Montage BlastReport
Num. services 6 27 5
Data exchanged High Low Medium
Coord. Complex High Medium Low
Experiments (II)
Per
form
ance
Res
ult
sP
erfo
rman
ce R
esu
lts
Experiments (II)
Per
form
ance
Res
ult
sP
erfo
rman
ce R
esu
lts
Results
21
Per
form
ance
Res
ult
s
Centralized Experiment
22
Data and computation intensive workflows.• Size and processing time increment.
Centralized coordination better for workflows with reduced computation.
Per
form
ance
Res
ult
s
Decentralized Experiment
23
Reduced computation workflows• Slightly increment of time (network latency).
Data and computation-intensive workflows show the benefits of a decentralized coordination.
Per
form
ance
Res
ult
s
Conclusion
24
• Chemical model is well featured for decentralized workflow execution. Proof of concept of the chemical workflow system.
• Our proposal: High-level decentralized coordination mechanism.
• Decentralized Architecture: Chemical web services working as local engines. Multiset as shared communication space. A High-order chemical language for workflows.
• Concepts for decentralized coordination.• Control and data driven.
Su
mm
a ry
On-going Work
25
• Implementation of a distributed multiset.
• Workflow scheduling in Federated Clouds using the chemical model.
• Modelling Agile Service Networks using the chemical choreography coordination model.
Questions ?
26
THANKS !
27