Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3...
Transcript of Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3...
![Page 1: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/1.jpg)
OPENCCQFederated High Performance Computing with
![Page 2: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/2.jpg)
Where to start with HPC in the Cloud?
![Page 3: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/3.jpg)
Here – Combine all the services yourself?
![Page 4: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/4.jpg)
Clus
ter i
s Rea
dy
In Minutes your Cluster is ready
![Page 5: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/5.jpg)
Collaboration
• Have the ability to create collaborations
• Invite other collaborators to CloudyCluster
• Initially can share Google Drive Folders
• Oauth and Shib / InCommonsupport for Web Authentication
![Page 6: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/6.jpg)
Self-Service Elastic HPC
Scheduler
CCQ
EFS S3
Auto-ScalingCompute
OrangeFSHPCParallelStorage
DDB
HPCJob
Login
WebDAViRODSGlobus
CreateafullyoperationalHPCClusterinminutes,completewith:• Storage: OrangeFSonEBS,S3,EFS
• Compute: JobDrivenElasticComputethroughCCQ
• Scheduler: withv.1.2Torque/Maui&SLURMwithCCQMetaScheduler
• HPCLibraries:Boost,Cuda Toolkit,Docker,FFTW,FLTK,GCC,Gengetopt,GRIB2,GSL,Hadoop,HDF5,ImageMagick,JasPer,NetCDF,NumPy,Octave,OpenCV,OpenMPI,PROJ,R,Rmpi,SciPy,SWIG,WGRIB,UDUNITS
• HPCSoftware:Ambertools,ANN,ATLAS,BLAS,Blast,Blender,Burrows-WheelerAligner,CESM,GROMACS,LAMMPS,NCAR,NCL,NCO,nwchem,OpenFoam,papi,paraview,QuantumEspresso,SAMtools,WRF
• YoucanalsoInstallyourowninacustomAMI• AllfromaneasytouseWebUIfrommobile,tabletordesktop• iRODS andXDMoD aretargetedforFallrelease.
Torque/MauiSlurm
![Page 7: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/7.jpg)
CCQ - Elastic HPC Dispatching
Scheduler
DynamoDB
Login Instance
Public Subnet
Submit the Job
Through CCQ
CCQ holds job determines and
launches instances needed
CCQ Sends the job to the
scheduler when ready
Scheduler launches the Job normally
If no jobs are in the queue for that
instance type near the billing hour, instances
are terminated
Compute Groups
SchedulerTorqueSlurm
![Page 8: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/8.jpg)
OpenCCQ Project (Federated HPC)On Prem to Cloud bursting using heuristics and hints to determine where the job goes; supporting multiple schedulers and environments
Job Routing• On Premise• Cloud• Nat. ResourcesData MovementJob to Data
![Page 9: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/9.jpg)
For $100 AWS Credit you can fill out the form at:http://www.cloudycluster.com/ccvisualhpc/
![Page 10: Federated High Performance Computing with OPENCCQ · Self-Service Elastic HPC Scheduler CCQ EFS S3 Auto-Scaling Compute OrangeFS HPC Parallel Storage DDB HPC Job Login WebDAV iRODS](https://reader033.fdocuments.us/reader033/viewer/2022042112/5e8e0d48e3acb128332a5bf8/html5/thumbnails/10.jpg)
Discussion and Questions?
Come see a demo at our table