The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster...
Transcript of The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster...
![Page 1: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/1.jpg)
Workshop Open Modeller - Cluster
The use of Cluster and parallelism to support the execution of openModeller.
![Page 2: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/2.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiments;
− Results;
− Future Work;
![Page 3: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/3.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiments;
− Results;
− Future Work;
![Page 4: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/4.jpg)
Workshop Open Modeller - Cluster
� LAHPC (Laboratory of Architecture and High Performance Computing)
− Tools for parallel and distributed programming, respectively to shared memory machine and homogeneous and heterogeneous clusters,
− Performance analysis to parallel and distributed programs,
− Tools to parallel database,
− Memory management techniques,
− Parallel and distributed architectures;
− Grid computing.
![Page 5: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/5.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiments;
− Results;
− Future Work;
![Page 6: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/6.jpg)
Workshop Open Modeller - Cluster
� Cluster Hardware:
− 11 nodes: 2 quad intel Xeon 2.0 GHz and 8 GB ram memory;
− Network interconnection: Gigabit and Infiniband;
− Local File System and Network File System.
� Cluster Software:
− Operational System: SUSE Linux Enterprise 10, 64 bits, XFS;
− Compilers: C, C++ and Fortran;
− MPI: LAM, Voltaire, Intel;
− Condor job scheduler;
− Ganglia cluster monitor;
− Apache Server;
− Tomcat;
![Page 7: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/7.jpg)
Workshop Open Modeller - Cluster
![Page 8: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/8.jpg)
Workshop Open Modeller - Cluster
![Page 9: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/9.jpg)
Workshop Open Modeller - Cluster
![Page 10: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/10.jpg)
Workshop Open Modeller - Cluster
![Page 11: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/11.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiment;
− Results;
− Future Work;
![Page 12: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/12.jpg)
Workshop Open Modeller - Cluster
� Improve the performance of openModeller
− Parallelism;
− Cluster (node: multi-core architecture);
− Infiniband;
� Enable the use of cluster power processing on openModeller Desktop;
![Page 13: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/13.jpg)
Workshop Open Modeller - Cluster
� Condor:
− Specialized workload management system for compute-intensive jobs.
� LAM-MPI:
− LAM (Local Area Multicomputer) an open source implementation of the Message Passing Interface (MPI) standard. The MPI standard is the de facto industry standard for parallel applications.
� SOA (Service Oriented Architecture):
− Architectural model to enable the integration of loosed coupling systems.
![Page 14: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/14.jpg)
Workshop Open Modeller - Cluster
� Integration with openModeller desktop using web services;
� openModeller execution as a job:− openModeller Library build on Front Node (NFS);− Layers replicated in all the nodes;− Condor support;
� openModeller parallelization:− openModeller components parallelization using LAM;− Condor support;
![Page 15: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/15.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiment;
− Results;
− Future Work;
![Page 16: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/16.jpg)
Workshop Open Modeller – Cluster
![Page 17: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/17.jpg)
Workshop Open Modeller – Cluster
![Page 18: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/18.jpg)
Workshop Open Modeller – Cluster
![Page 19: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/19.jpg)
Workshop Open Modeller – Cluster
![Page 20: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/20.jpg)
Workshop Open Modeller – Cluster
![Page 21: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/21.jpg)
Workshop Open Modeller - Cluster
� Experiment:− Species: Furcata boliviana
− Layers:� WorldClim (bioclimáticos);
− Output Mask: /application/scratch/layers/WORLDCLIM/bioclimaticos/bio_10min/bio_1
− Algorithm: BioClim
− Format: GreyTiff100
− Average time 1 min 55 segundos. (serial)
− Average time 55 segundos. (parallel) 10 nodes;
![Page 22: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/22.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiment;
− Results;
− Future Work;
![Page 23: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/23.jpg)
Workshop Open Modeller - Cluster
� Experiments:
− Furcata Boliviana:
� Algorithms:
− Bioclim
− DG GARP
− GARP
− Envelope Score
− Support Vector Machines
− Environmental Distance
− Environmental Distance (Chebyshev)�
− Environmental Distance (Mahalanobys)�
� Executions:
− From 1 to 15 process running 5 times each.
![Page 24: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/24.jpg)
Workshop Open Modeller - Cluster
• Average Execution Time (om_project):
1 3 4 5 6 7 8 9 10 11 12 13 14 15
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
BioclimDG GARPGARPEnvelope ScoreSupport Vector M achinesEnvironmental DistanceEnvironmental Distance (Chebyshev)Environmental Distance (M ahalanobys)
![Page 25: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/25.jpg)
Workshop Open Modeller - Cluster
� Discussion:
− Degradation level;
− Network communication;
− Map Generation;
− Layer file reading.
![Page 26: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/26.jpg)
Workshop Open Modeller - Cluster
� Agenda:
− LAHPC;
− Cluster;
− Activities;
− Experiment;
− Results;
− Future Work;
![Page 27: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/27.jpg)
Workshop Open Modeller - Cluster
� Future Work:
− Performance evaluations;
− Port new algorithms to openModeller: AdaptGarp, Pgarp, ... ;
− Implement an authorization infra-structure based on credentials;
− Enforcement of web services invocation by known users identified by credentials;
![Page 28: The use of Cluster and parallelism to support the ...€¦ · Workshop Open Modeller - Cluster Condor: −Specialized workload management system for compute-intensive jobs. LAM-MPI:](https://reader035.fdocuments.us/reader035/viewer/2022071001/5fbd7be6b0a9b40dbc5d081f/html5/thumbnails/28.jpg)
Workshop Open Modeller - Cluster
� Questions ?