Live tweeting highlights - Il business plan (Salvatore Torrisi e Andrea Scotti)
Catania Grid & Cloud Engine Mario Torrisi ([email protected]) Istituto Nazionale di Fisica...
-
Upload
alvin-herder -
Category
Documents
-
view
221 -
download
4
Transcript of Catania Grid & Cloud Engine Mario Torrisi ([email protected]) Istituto Nazionale di Fisica...
![Page 1: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/1.jpg)
Catania Grid & Cloud EngineMario Torrisi ([email protected])
Istituto Nazionale di Fisica Nucleare – Sezione di Catania
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
![Page 2: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/2.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
2
Outline The Catania Grid & Cloud Engine
Job Engine Architecture Features
Code Samples Portlet for job management
MyJobs mi-hostname-portlet mi-parallel-app-portlet
References Summary and conclusion
Catania - 12 Marzo 2014
![Page 3: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/3.jpg)
The Catania Science Gateway model
AdministratorPower UserBasic User
Grid/Cloud/Local middleware supported so far
....... Science
Gatew
ay
App. 1 App. 2App.
N
Embedded Applications
Users from different
organisations having different
roles and privileges
Standard-based (SAGA) middleware-independent
Grid Engine
![Page 4: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/4.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
4
The Catania Grid & Cloud Engine
Catania - 12 Marzo 2014
Grid/Cloud Engine
UsersTracking
DB
Science GW Interface
SAGA/JSAGA API
Job EngineData Engine UsersTrack &Monit.
ScienceGW 1
ScienceGW 2
ScienceGW 3
Grid/Cloud/Local MWs
Liferay Portlets
eTokenServer
NewModified New Modified NewModified
![Page 5: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/5.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
5
Job Engine
The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure;
It is compliant with the OGF SAGA standard; It is optimized to be used in a Web Portal running
an application server (e.g. Glassfish, Tomcat,…) based on J2EE;
It could be used also in stand-alone mode; JSAGA is the SAGA implementation adopted.
Catania - 12 Marzo 2014
![Page 6: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/6.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
6
A Simple API for Grid Applications (SAGA)
SAGA is a family of related standards to define application programming interface (API) for common distributed computing functionality
It is independent of the details of the underlying infrastructure (e.g., the middleware);
SAGA is an OGF specification: http://www.gridforum.org/documents/GFD.90.pdf
SAGA is composed by:
SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);
SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);
N.B. SAGA defines a standard Catania - 12 Marzo 2014
We then need animplementation!
![Page 7: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/7.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
7
JSAGA
JSAGA is a Java implementation of SAGA developed at CCIN2P3;
JSAGA: Enables uniform data and job management
across different grid infrastructures/middleware;
Makes extensions easy: adaptor interfaces are designed to minimize coding effort for integrating support of new technologies/middleware;
Is OS indenpendent: most of the provided adaptors are written in full Java.
Catania - 12 Marzo 2014
![Page 8: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/8.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
8
Job Engine Architecture
Catania - 12 Marzo 2014
USERSTRACKING
DB
JobsSubmission
JobsCheck status/
Get output
WT
DC
Is
Worker Threads for Job Submission
WT
Worker Threads forJob Check Status
MO
NIT
OR
ING
M
OD
ULE
Jobs Queue
WT WT
WT WT WT
WT
WT WT
USER TRACK & MONITOR
![Page 9: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/9.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
9
Job Engine - Features
Catania - 12 Marzo 2014
![Page 10: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/10.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
10
Job Engine Middleware Independent
Catania - 12 Marzo 2014
JSAGA supports gLite, Globus,ARC, UNICORE, etc.
Adding new adaptors in JSAGAis a easy job
![Page 11: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/11.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
11
Job EngineEasiness
Allow to develop application able to submit jobs on the grid in a very short time;
A very intuitive API is exposed to the developers;
Support MPI applications; The developer has only to submit the job:
The Job Engine periodically check the job status;
When the Job is done, the job output is automatically downloaded by the Job Engine in the local machine.
Catania - 12 Marzo 2014
![Page 12: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/12.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
12
Job EngineScalability (1/2)
The Job Engine is able to manage a huge number of parallel job submissions fully exploiting the HW of the machine where it is installed;
It enqueues all the parallel requests received serving it according to the HW capabilities;
The Job Engine thread pools can be configured to optimally exploit the HW capabilities;
A burst of parallel job submissions cannot damage the Job Engine responsiveness thanks to the protection provided by the thread pool mechanism.
Catania - 12 Marzo 2014
![Page 13: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/13.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
13
Job EngineScalability (2/2)
Catania - 12 Marzo 2014
The answer time is linear;
Response time depends on the HW capabilities and thread pools configuration.
![Page 14: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/14.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
14
Job EnginePerformance
All the delays due to grid interactions are hidden to the final users: The Job Engine provide asynchronous functions for each
job management actions (submit, check status, download output, cancel);
Final users “feel” a response time equals to 0. The Job Engine is able to submit thousands of jobs in
a short time: The Job Engine submit jobs using a configurable thread
pool; The Job Engine is able to submit 10000 jobs in less than 1
hour with 50 threads in the thread pool; Increase the number of threads in the thread pool, can
improve the Job Engine performance.Catania - 12 Marzo 2014
![Page 15: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/15.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
15
Job EngineAccounting
A very powerful accounting system is included in the job engine;
It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy;
The following values are stored in the DB for each job submitted: Users; Job Submission timestamp; Job Done timestamp; Application submitted; Job ID; Proxy used; VO; Site where the job is running (e.g., CE for EMI-gLite).
Catania - 12 Marzo 2014
![Page 16: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/16.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
16
Job EngineFault tollerance
Job Engine implements an advanced mechanism to guarantee job submission: Developers can set an appropriate value of
“shallow retry”;
Automatic re-submission mechanism when a job is aborted: Hide every failure to the final users
Catania - 12 Marzo 2014
![Page 17: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/17.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
17
Job EngineCollection, Parametric & Workflow
Currently the Job Engine natively support: Collection jobs; Parametric jobs; N-1 workflows.
In the next releases we’ll add the support for DAG (Direct Acyclic Graph)
Catania - 12 Marzo 2014
![Page 18: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/18.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
18
Code ExampleJob submission
Catania - 12 Marzo 2014
String wmsList[] = {"wms://wms-4.dir.garr.it:7443/glite_wms_wmproxy_server“,"wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server"/};
Resource Manager List
GEJobDescription description = new GEJobDescription();description.setExecutable("/bin/sh");description.setArguments("hostname.sh");description.setInputFiles("/home/mario/Documenti/hostname.sh");description.setOutput(“Output.txt");description.setError(“Error.txt");
Job Description
JSagaJobSubmission tmpJSaga = new JSagaJobSubmission(description);tmpJSaga.useRobotProxy("etokenserver.ct.infn.it", "8082", "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit", true);tmpJSaga.setUserEmail("[email protected]");tmpJSaga.setResourceManagerList(wmsList);
tmpJSaga.submitJobAsync("mtorrisi", "193.206.208.183:8162", 1, "gLite Test job“);
Submit the Job
![Page 19: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/19.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
19
Code ExampleWorkflow N-1(1/4)
Catania - 12 Marzo 2014
JOB 1 JOB 2 JOB 4JOB 3 …
Final Job
OUTPUT
InfrastructureInfo infrastructures[] = new InfrastructureInfo[2];
String wmsList[] = { "wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server" };infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082“, "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit");
String globusList[] = {"wsgram://xn03.ctsf.cdacb.in:8443/GW"};Infrastructures[1] = new InfrastructureInfo("GARUDA","wsgram","", globusList, "etokenserver.ct.infn.it","8082","332576f78a4fe70a52048043e90cd11f","gridit","gridit");
Define Infrastructures
![Page 20: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/20.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
20
Code ExampleWorkflow N-1(2/4)
Catania - 12 Marzo 2014
for (int i = 0; i < 3; i++) {GEJobDescription description = new GEJobDescription();description.setExecutable("/bin/sh");switch (i) {case 0:
description.setArguments("hostname.sh");description.setInputFiles("/home/diego/test_wf/hostname.sh");break;
case 1:description.setArguments("ls.sh");description.setInputFiles("/home/diego/test_wf/ls.sh");break;
case 2:description.setArguments("pwd.sh");
description.setInputFiles("/home/diego/test_wf/pwd.sh");break;
}description.set…descriptions.add(description);
}
Define First Level Jobs
![Page 21: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/21.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
21
Code ExampleWorkflow N-1(3/4)
Catania - 12 Marzo 2014
GEJobDescription finalJobDescription = new GEJobDescription();finalJobDescription.setExecutable("/bin/sh");finalJobDescription.setArguments("ls.sh");String tmp = "";for(int i = 0; i < descriptions.size(); i++){
if(tmp.equals("")) tmp=descriptions.get(i).getOutput();else tmp+=","+descriptions.get(i).getOutput();
}finalJobDescription.setInputFiles(tmp + ",/home/diego/test_wf/ls.sh,/home/diego/test_wf/ifconfig.sh");finalJobDescription.setOutput("myOutput-FinalJob.txt");finalJobDescription.setError("myError-FinalJob.txt");
Define Final Job
![Page 22: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/22.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
22
Code ExampleWorkflow N-1(4/4)
Catania - 12 Marzo 2014
JobCollection wf = new WorkflowN1("scardaci", "Workflow N-1", "[email protected]", "/tmp", descriptions, finalJobDescription);JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission(wf);
tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.18:8162", 1);
Submit the Workflow
![Page 23: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/23.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
23
Grid & Cloud Engine javadoc
Catania - 12 Marzo 2014
http://grid.ct.infn.it/webinar-liferay/
![Page 24: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/24.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
24
Job Management portletMyJobs
Catania - 12 Marzo 2014
Active Jobs
Job Status and Output
Search on job description
Job description
![Page 25: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/25.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
25
Job Management portletmi-hostname-portlet
Catania - 12 Marzo 2014
![Page 26: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/26.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
26
Job Management portletmi-parallel-app-portlet
Catania - 12 Marzo 2014
![Page 27: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/27.jpg)
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
27
References
Catania - 12 Marzo 2014
The Catania Science Gateway Framework: http://www.catania-science-gateways.it/
Grid & Cloud Engine Javadoc: http://www.catania-science-gateways.it/training-material
A Simple API for Grid Applications (SAGA): http://www.gridforum.org/documents/GFD.90.pdf;
JSAGA: http://grid.in2p3.fr/jsaga/;
![Page 28: Catania Grid & Cloud Engine Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c785503460f9492dd48/html5/thumbnails/28.jpg)
Webinar on the DCH-RP e-Culture Science Gateway28
Thank you !
Catania - 24 June 2013
For more information on Catania Science Gateway framework please visit:
http://www.catania-science-gateways.it/