ORNL/IU Workshop on Computational Frameworks in Fusion Oak Ridge, TN, USA, 9 th 2005 Defining,...
-
Upload
clifford-walsh -
Category
Documents
-
view
215 -
download
0
Transcript of ORNL/IU Workshop on Computational Frameworks in Fusion Oak Ridge, TN, USA, 9 th 2005 Defining,...
ORNL/IU Workshop on Computational Frameworks in FusionOak Ridge, TN, USA, 9th 2005
Defining, Implementing, Executing and Deploying
a Parallel Component Model
Christian Pérez
joint work with Thierry Priol, Hinde Bouziane, Alexandre Denis, Sébastien Lacour & André RibesPARIS Research Team, INRIA/IRISA, Rennes, France
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Conclusion
Application in hydrogeology: Saltwater intrusion
flow : velocity and pressure function of the densityDensity function of salt concentration
Salt transport : by convection (velocity) and diffusion
Coupled physical models One model = one software Saltwater intrusion
flow / transport Reactive transport
transport / chemistry
Hydrogrid project,supported by theFrench ACI-GRID
Numerical coupling in saltwater intrusion
Flow Transport
time
t = 0
t = t
control
iterative scheme at each timestep
Code coupling into a Grid?
Visualization
Homogeneous cluster
SAN
SAN
Homogeneous cluster
LAN
WAN
Supercomputer
Flowcode
Transport code
Controller
velocity
scalarsscalars
concentration
Code coupling into a Grid?
Visualization
Homogeneous cluster
SAN
SAN
Homogeneous cluster
LAN
WAN
Supercomputer
Flowcode
Transport code
Controller
velocity
scalarsscalars
concentration
Programming Environment( Virtualization)
Requested Features Support of legacy codes
Multi-languages, multi-OS,multi-processors
Support of parallel codes MPI, PVM, OpenMP, ...
Support for network transparency Support for deployment
“Single-provider” application “Multi-provider” application
Software component!
Transport
Flow
Distributed components: OMG CCM A distributed component-oriented
model An architecture for defining
components and their interaction Interaction implemented through
input/output interfaces Synchronous and asynchronous
communication A runtime environment based on the
notion of containers (lifecycle, security, transaction, persistent, events)
Multi-languages/OS/ORBs/vendors, … With a packaging/deployment
model A packaging technology for deploying
binary multi-lingual executables Could be deployed and run on several
distributed nodes simultaneously
MyComponent
Component interface
Facets
Eventsources
Eventsinks
Attributes
Receptacles
OF
FE
RE
D
RE
QU
IRE
D
Component-based application
Courtesy of P. Merle, LIFL, France
Component and Parallelism
How to embed a parallel code into a component?
Parallel Code A Parallel Code B
MPI PVM
Parallel Component
HPCComponent
A
HPCComponent
B
What the application designer should see…
… and how it must be implemented !
// Comp. A // Comp. B
Distributed Component Model
Caller Receiver
// Caller Codeo->factorize(m);
Distributed component (a process)
// Receiver Codevoid serv::factorize(const Matrix mat) { ... ;}
SPMD Parallel Component Model (1)
DataRedistribution
Caller Parallel Receiver
Distributed component (a process)
// SPMD Receiver codevoid factorize(const DMatrix mat) { .... MPI_Bcast(...) ... ;}
Parallel Distributed Component
// Caller Codeo->factorize(m);
SPMD Parallel Component Model (2)
DataRedistribution
Parallel Caller Receiver
// Receiver Codevoid factoriser(const Matrice mat) { ... ;}
Distributed component (a process) Parallel Distributed Component
// SPMD Caller Codeo->factorize(m);
Parallel ReceiverParallel Caller
// SPMD Caller Codeo->factorize(m);
// SPMD Receiver Code SPMDvoid factorizer(const Matrix mat) { .... MPI_Bcast(...) ... ;}
DataRedistribution
Distributed component (a process) Parallel Distributed Component
SPMD Parallel Component Model (3)
GridCCM: Parallel CORBA Component
Extends the CORBA Component Model No modification of CCM specification Portable Extension of CCM descriptor (XML)
Parallel component Parallelism is a non-functional
property of a component It is an implementation issue
Collection of sequential component SPMD execution model
Support of distributed arguments API for data redistribution API for communication scheduling w.r.t. network
properties Support of parallel exceptions
Parallel Comp.
SPMDProc.SPMDProc.SPMDProc.SPMDProc.SPMDProc.
GridCCM Component Example
component CoPa { provides IExample to_client; uses Itfaces2 to_server; };
interface IExample { void op(in Matrix mat); };
Component: CoPaPort: to_clientName: IExample.op Type: ParallelArgument1: Basic_BC[*, bloc]ReturnArgument: noReduction
XML
Non functional property ofa component implementation
Comp. A-0
SPMDProc.
Comp. A-4Comp. A-0
SPMDProc.
Comp. A-3Comp. A-0
SPMDProc.
Comp. A-2Comp. A-0
SPMDProc.
Comp. A-1
Comp. A-0
SPMDProc.
Comp. A-0to_client to_server
IDL3
IDL2
Component: CoPaPort: to_clientName: IExample.op Type: ParallelArgument1: BasicBC[*, bloc]ReturnArgument: noReduction
GridCCM: Code Generation
User IDL
Standard Stub
Standard Skel.
Internal IDL
Standard IDL compiler
Distribution (XML)
Parallel IDL compiler
Parallel CORBA
Stub & Skel.
Interface Iexample { ... };component CoPa { provides IExample to_client; uses Itfaces2 to_server;};
Component ModelHeterogeneity
Parallelism
GridCCM Architecture Overview
Object Request Broker
CORBA stub/skeleton
CommunicationLibrary ( MPI)
Application
Client view management- Data distribution description
Communication management- Communication matrix computation- Communication scheduling computation- Communication execution
Redistribution Library 1
CommunicationLibrary
GridCCM runtime
SchedulingLibrary
Experiments
Synchronous call Parameter: processor number
Identity data redistribution
Parallel ReceiverParallel Caller
MPIMPI
Latency SAN
Myrinet-200/GM Follows MPI barrier
times Pt-2-Pt overhead
CORBA : 25 µs PaCO++ : 35 µs
WAN French VTHD
Network 2 Gbit/s ~ 1000 Km long
Negligible overhead 10 µs << 8 ms
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8
Nombre de nœuds par objet parallèle
PaCO++
MPI
Latence en microsecondes
Nodes milliseconds
1x1 8.01
2x2 8.04
4x4 8.08
8x8 8.22
16x16 8.40
Bandwidth SAN
Myrinet-2000/GM Aggregation
4 nodes :600 MB/s 4x150 MB/s
WAN French VTHD Network 1 Gbit/s switch
Saturate for n > 12 Sustaining bandwidth
Using a Scheduling Algorille
algorithmINRIA/LORIA
But need to knownetwork properties
0
100
200
300
400
500
600
700
1 10 100 1000 10000 100000 1000000 10000000 1E+08
Taille des messages (octet)
1 nœud par objet parallèle
2 nœuds par objet parallèle
4 nœuds par objet parallèle
Bande passante agrégée (Mo/s)
0
20
40
60
80
100
120
6 7 8 9 10 11 12 13 14 15 16
Nombre de nœuds par objet parallèle
Sans ordonnancement
Avec ordonnacement
Bande passante agrégée (Mo/s)
Relationship between Models
GridCCM: a parallel distributed component model Extension of CCM Based on PaCO++
PaCO++: a parallel distributed object model Extension of CCM
CORBA ObjectModel
CORBA ComponentModel (CCM)
Parallel ObjectModel (PaCO++)
Parallel ComponentModel (GridCCM)
Parallel Extension Parallel Extension
ComponentExtension
ComponentExtension
Status PaCO
Obsolete, not maintain any longer PaCO++
Available on the Web (version 0.2) LGPL/GPL licence, cf http://www.irisa.fr/paris/Padico
MPI support Data Redistribution
Identity, 1D Bloc-cyclic, “EPSN” Still going on
GridCCM Not yet available, only prototype version Being developed
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Status Conclusion
Deploy the same code using different networking resources
SAN
SAN
LAN
WAN
process
processusprocessus
processusprocessusprocess
processusprocessusprocessus
processusprocess
Configuration 1
process
processusprocessus
processusprocessusprocess
processusprocessusprocessus
processusprocess
Configuration 2
WAN
SAN
SAN
High performance middleware
Middlewareover a WAN
The middleware adapt itself to the available network
Mapping Objects onto Resources
Handling network issues: PadicoTM
Network heterogeneity Several networks, Non IP network, parallel stream, online
compression, firewalls Decouple communication API from actual
network Enables different access to the networks Transparent network access High Performance provided
to middleware systems Based on PM2
Consequences:• Control the mapping of a
middleware to a network• Add special adapters
TCP/IP Myrinet SCI ...
AdOC
ParallelStream
PadicoTM
MPI CORBA SOAP JXTA
SSL
0
50
100
150
200
250
1 100 10000 1000000 100000000Message si ze (bytes)
CORBA/ Myr i net - 2000
MPI / Myr i net - 2000
J ava/ Myr i net - 2000CORBA/ SCI
MPI / SCI
TCP/ Et her net - 100
Bandwi dth (MB/ s)
Bandwidth Myrinet-2000 : 240MB/s SCI : 86MB/s 96% of the hardware
bandwidth for CORBA,MPI and Java
Latency CORBA/Myrinet-2000: 18 µs MPI/Myrinet-2000: 11 µs Java/Myrinet-2000: 40 µs CORBA/Ethernet: 150 µs
PadicoTM-enabled Middleware bandwidth over Myrinet-2000 and SCI
PadicoTM-enabled Middleware Perfs
WAN PadicoTM Performance Parallel streams
VTHD Network (Rennes-Nice) Without : 8 MB/s (64 Mb/s) With (4 streams): 12 MB/s (96 Mb/s) For all runtimes
Overhead negligible
Adaptive Compression (AdOC) ADSL link 608 kbit/s Without compression : 580 kbit/s With compression : 800 to 1200
kbit/s For all runtimes
Overhead negligible The VTHD network
PadicoTM Status Middleware and methods supported by PadicoTM :
MPI mpich-1.1.2 & mpich-1.2.4 (ch_mad)
CORBA omniORB omniORB-3.0.2 through omniORB-
4.0.6 MicoCCM mico-2.3.4 through mico-2.3.11 others Iona ORBacus-4 and Orbix/E
SOAP gSOAP-2.1 through 2.6 JVM kaffe-1.0.6 HLA Certi-3.0 (Onera) DSM Mome (Y. Jégou) ICE ICE-1.0 & 1.1 (ZeroC Inc.) JXTA JXTA-C (CVS cersion)
PadicoTM 0.1.5 available 0.3 release coming very soon
http://www.irisa.fr/paris/Padicotm/
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Status Conclusion
How to Deploy my Applicationon Grid Resources?
Visualization
Homogeneous cluster
SAN
SAN
Homogeneous cluster
LAN
WAN
Supercomputer
Flowcode
Transport code
Controller
velocity
scalarsscalars
concentration
Grid Access Middleware
Low-level services Globus Toolkit, Unicore, Legion, Condor-G
Resource management (GRAM) Launch processes on remote resources
Data management (GASS) File transfer (GridFTP, RFT), replica
management (RLS) Information service (MDS) Security (GSI)
Complementary Solutions
CCM(component-based
distributed programming model)
Globus(grid access middleware)
Programming and deployment model
YES NO
Application execution infrastructure
NO YES
CCM
Globus
Manual Deployment
Discover available grid resources Select grid resources for execution
OS, architecture compatibility Map the application onto the selected
resources MPI processes Components
Select compatible compiled executables Upload and install executables, stage input
files in Launch processes on remote computers Set configuration parameters of the application
Components' attributes Network topology information
Too complex!
Automatic Deployment Automatic
Resource discovery Execution node selection File installation Process launch Application configuration
Hide application complexity
Hide grid complexity
stop reading your e-mails!
Grid
Identification of the Stepsof Automatic Deployment
MPI Application Description
CCM Application Description
Resource Description
Generic Application Description
Control Parameters
Deployment Planning
Deployment Plan Execution
Application Configuration
Stat
ic
Applic
atio
ns
Deployme
nt Tool
Generic Application Description
How to deploy a parallel component? Using a parallel technology (MPI) Using a distributed technology (CCM)
Deployment planner: complex to write Implement it once and re-use it
Generic Application Description
Translator From specific to generic application description Straightforward to write
CCM Appl. Descriptio
n
MPI Appl. Descriptio
n
GridCCM Appl.
Description
CCM Appl. Descriptio
n
MPI Appl. Descriptio
n
GridCCM Appl.
Description
CCM Planner
MPI Planner
GridCCM Planner
Plan Execution
Plan Execution
Plan Execution
Generic Application Description
Deployment Planner
Deployment Plan Execution
ADAGE Automatic Deployment of Applications in a Grid
Environment http://www.irisa.fr/paris/ADAGE/
Already implemented MPI and CCM application description Network topology description Generic application description Simple control parameters, simple planner Deployment plan execution (SSH, Globus)
Not implemented yet GridCCM application description and configuration Integrate more planners (GRAAL, LIP, Lyon)
Conclusion Software component is a promising technology for
handling code & grid complexity Padico = GridCCM + PadicoTM + ADAGE (today
) http://www.irisa.fr/paris/Padico
GridCCM : extends CCM Parallel component, scalable communication achieved
Data redistribution / communication scheduling API Approach applicable to other component/service models
PadicoTM: an open integration framework for communication middleware and runtimes Decouple middleware from actual networking technology Supports several middleware at the same time
Adage: Automatic Deployment of Applications in a Grid Environment Application/resource constraints, user goal, and mapping
algorithm needed , a European NoE
42 partners from 30 different countries
Reactive transport in Hydrogeology
u
Convection
uDispersion
Gaz-liquid exchange
Aqueous Réactions
Dissolution
Precipitation
Sorption
Relargage
Biology
Numerical coupling in reactive transport
Chemistryon the
N species
Transport N
time
t = 0
t = t
control
Transport 1
Iterative scheme at each timestep
N species
…..
Component Modelfor reactive transport
Transport component
Chemistry component
concentrations
Controller component
Transport component
Transport component
Transport component
Transport component
N components