INRIA Background & ActiveEon’s Mission 2. ProActive : Programming Scheduling Resourcing
description
Transcript of INRIA Background & ActiveEon’s Mission 2. ProActive : Programming Scheduling Resourcing
1. INRIA Background & ActiveEon’s Mission
2. ProActive:ProgrammingSchedulingResourcing
3. Use Cases: GenomicsFinance
D. Caromel, C. Delbé
Accelerate onWindows Desktops, HPCS, Clouds
with ProActive Parallel SuiteSpeed: Application + Management: Productivity
1. Background & Mission
2
3
INRIA OASIS Team
3
A Research team, about 35 persons 2004: First ProActive User Group 2009, April: ProActive 4.1, Distributed & Parallel:
From Multi-cores to Enterprise GRIDs & Clouds
4
ActiveEon and INRIA synergy
Customer DrivenDevelopments
ResearchContracts
HighAdded Value Services
Customers Partners
Located in Sophia Antipolis, between Nice and Cannes, Visitors Welcome!
The best of research
and industry!
5
ActiveEon Company Overview (1)
Co-developing, Support for ProActive Parallel Suite Worldwide Customers: Fr, UK, USA
5
Selected Partners:
6
ActiveEon Company Overview (2)
Open Source Software Editor Born of INRIA in 2007
Core mission: Scale Beyond Limits
Co-developsProActive Parallel Suite®
Provides high level services for ProActive
Ten years experience in development of parallel computing
2. ProActive Parallel Suite
7
8
ProActive Parallel Suite
Java ParallelToolkit
Multi-Platform Job Scheduler
ResourceManager
Strong Features:Java Parallel Programming + Legacy + Integration + Portability: Windows, Linux, Mac +Versatility: Desktops, Cluster, Grid, Clouds = Perfect Flexibility
Used in Production everyday:50 Cores 300 Cores early 2010
9
ProActive Parallel SuiteThree fully compatible modules
Programming Scheduling
Resource Management
Clutch Power: Solid Building Blocks for Flexible Solutions
ResourcingScheduling
10
ProActive Programming
10
1111
Groups: Broadcast and Scatter (aka MPI)
11
JVM
JVM
JVM
JVM
agcg
ag.bar(cg); // broadcast cgProActive.setScatterGroup(cg);ag.bar(cg); // scatter cg
c1 c2 c3c1 c2 c3
c1 c2 c3c1 c2 c3c1 c2 c3
c1 c2 c3
s
c1 c2 c3
s
Broadcast is the default behavior Use a group as parameter, Scattered depends on rankings
1212
Dynamic Dispatch Group
12
JVM
JVM
JVM
JVM
agcg
c1c2c3
c4c5
c6c7
c8c0c9c1
c2c3
c4c5
c6c7
c8c0c9
c1c2c3
c4c5
c6c7
c8c0c9
Slowest
Fastest
ag.bar(cg);
13
Optimizing GUI: IC2D
13
14
Optimizing GUI: IC2D
14
15
Optimizing GUI: IC2D
15
1616
ProActive Scheduling
16
17
ProActive Scheduling Big Picture
RESOURCES
Multi-platform Graphical Client (RCP)
File-based or LDAP authentication Static Workflow Job Scheduling, Native and
Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…
Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,…
ProActive Deployment capabilities: Desktops, Clusters, Clouds,…
ProActiveScheduler
ProActiveResource Manager
18
Job
TaskFlow Example : Picture Denoising
Split
Denoise DenoiseDenoiseDenoise
Merge
• with selection on native executable availability (ImageMagik, GREYstoration)• Multi-platform selection and command generation
• with file transfer
1919
ProActive Resourcing
19
20
Desktops + Clusters + Clouds
20
21
Versatility: Desktops, Grids, Clouds (EC2),Multi-Core, VMs (Hyper-V, Xen, VMware, Kvm)
ProActiveSchedulerProActive
Resource Manager
Dedicated resources
LSF
Static Policy
Amazon EC2
EC2
Dynamic Workload Policy
Desktops
SSH
Timing Policy 12/24
22
RESOURCING User Interface
22
2 mn Video:Scheduler, Resource Manager
23
3. Use Cases
24
Bio-Tech: Genomic Sequencing
25
2626
Resources set up
16nodes Cluster
Desktops
CloudsAmazon EC2
SOLID machine from
Nodes can be
dynamically added!
2727
Scalability:Internal SOLID Cluster to INRIA Cluster The distributed version with ProActive of Mapreads has been tested on the INRIA cluster
with two settings: the Reads file is split in either 30 or 10 slices
Use case: matching 31 millions sequences with the human genome (M=2, L=25)
Reference point with 16 cores(same as in SOLiD machine)
4 Time faster from 20 to 100Speed Up of
80 / Th. Sequential50 Hours 35 Minutes
28
Benchmark: Internal vs. EC2 cloud only
Execution time (min)
Cost (US$)
Standard PBS config 300 NA
ProActive Amazon EC2 340 20 US$
For only $3,2/hour, the EC2 setup has nearly the same performances as the local SOLiD cluster
29
ProActive in Finance with Price-It Excel
3030
Use case: Price-It and ProActive Price-It®
C++ library developed by Pricing Partners Pricing solution dedicated to highly complex
financial derivatives
Accelerated Price-It® A collaboration between Pricing Partners and ActiveEon An implementation based on ProActive Scheduling
Specification and constraints Accelerate Price-It® Excel product
▫ Built on Price-It® library, this product integrates an interface with Excel for input data management and results display
Focus on highly parallelizable Greek computation Operating system: Windows
3131
Price-it Sequential Computing
Price-it®library
Price-it®Computation
DLL
C/C++
Before distribution
3232
Price-it Accelerated by ProActive
Scheduler
Worker Worker
Price-it®Computation
DLL
Price-it®Computation
DLL
C/C++
Price-it®library
Price-it®Computation
DLL
Using distributionSmooth integration of legacy code!
3333
Accelerated Price-It performances Use case: Bermuda Vanilla, Model American MC Test conditions
One computation is split in 130 tasks that are distributed Each task uses 300ko
Sequential Distributed
More than 3 times faster with only 4 nodes!
4 nodes 5 nodes 6 nodes 7 nodes 8 nodes 9 nodes
Even 6 times faster with 9 nodes!
3434
Windows HPCS Benchmarks
35
Overall User Domain and Use Cases
Use Cases:
35
Finance
Life Science and BioTech
Engineering: Accelerating Matlab® and Scilab
IT
Numerical SPMD
Wrap Up
36
37
ProActive Parallel Suite
Java ParallelToolkit
Multi-Platform Job Scheduler
ResourceManager
Flexibility Clutch Power
Portability: Windows, Linux, Mac
Versatility: Desktops, Grids, Clouds
Free Professional Open Source SoftwareCommunity and Enterprise Editions
Choose the Edition you needCommunity EnterpriseUsers support Professional support
www.activeeon.com/downloads
Extra Material
38
39
ActiveEon: Mission, Services, ProductMission: Accelerate your applications:
Development, Deployment, Execution Optimize your Hardware Infrastructure utilization Easy and Accessible Parallel Computing & Clouds
High-End Services & Support: Training, Consulting, Integration, Development Technical Support and Subscription Partnerships (ISVs, VARs for OEM)
Open Source Products:
4040
ProActive : Active objects
40
A
Proxy
Java Object
A ag = newActive (“A”, […], VirtualNode)V v1 = ag.foo (param);V v2 = ag.bar (param);...v1.bar(); //Wait-By-Necessity
V
Wait-By-Necessity is a
Dataflow Synchronization
JVM
A
JVM
Active Object
Future Object Request
Req. Queue
Thread
v1v2 ag
WBN!