Solving BIG problems with Open Source: Condor
-
Upload
antonio-sanz-alcober -
Category
Technology
-
view
2.650 -
download
8
description
Transcript of Solving BIG problems with Open Source: Condor
![Page 2: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/2.jpg)
2
![Page 3: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/3.jpg)
3
> Antonio Sanz
> I3A System Manager
> HERMES HPC cluster sysadmin
> @antoniosanzalc
![Page 4: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/4.jpg)
4
![Page 5: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/5.jpg)
5
Show / Know / Use
![Page 6: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/6.jpg)
6Problema inicial
3. Sistemas de gestión de colas : Condor
> Dr. Good
> Neurologist
> Alzheimer research
> Process 20000
brain image scans
(1h/image)
> A thousand times. Maybe two.
![Page 7: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/7.jpg)
7Problema inicial > Mrs. Nice
> Santa’s Logistic Officer
> Gift transportation
> Analize 6x10e7 possible load/reindeers/routes
(10min/analysis)
> Before Christmas!
![Page 8: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/8.jpg)
8
Hey … ! It’s a 64K one !
![Page 9: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/9.jpg)
9
Queue distribution systems
![Page 10: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/10.jpg)
10Condor Basics
3. Sistemas de gestión de colas : Condor
Single queue
![Page 11: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/11.jpg)
11
![Page 12: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/12.jpg)
12Condor Basics
Multiple queues
3. Sistemas de gestión de colas : Condor
![Page 13: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/13.jpg)
13
![Page 14: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/14.jpg)
14
Problem partitioning
![Page 15: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/15.jpg)
15Problem can be broken into independent pieces
![Page 16: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/16.jpg)
16Condor Basics
Oh Yeah!
![Page 17: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/17.jpg)
17Condor Basics
For loops are your best friends
3. Sistemas de gestión de colas : Condor
![Page 18: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/18.jpg)
18Condor Basics
3. Sistemas de gestión de colas : Condor
While loops …can sometimes be convinced
![Page 19: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/19.jpg)
19Condor Basics
Do it yourself !3. Sistemas de gestión de colas : Condor
![Page 20: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/20.jpg)
20Condor Basics
>
![Page 21: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/21.jpg)
21Condor Basics
Heterogeneous computing
![Page 22: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/22.jpg)
22
Resource harvesting
![Page 23: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/23.jpg)
23
Requirements
![Page 24: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/24.jpg)
24
Job Surveillance
![Page 25: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/25.jpg)
25Condor Basics
Fair use of resources
3. Sistemas de gestión de colas : Condor
![Page 26: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/26.jpg)
26
Checkpoints
![Page 27: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/27.jpg)
27Condor Basics
Nested jobs (DAG)
![Page 28: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/28.jpg)
28Condor Basics
Email Notifications
![Page 29: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/29.jpg)
29
Grid & Cloud Computing
![Page 30: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/30.jpg)
30Condor Basics
Flexibility
![Page 31: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/31.jpg)
31
… with Hadoop, MPI, OpenMP, GPU
![Page 32: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/32.jpg)
32Condor Basics
3. Sistemas de gestión de colas : Condor
![Page 33: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/33.jpg)
33
How Condor works
![Page 34: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/34.jpg)
34
Management
[Hello, Dave]
![Page 35: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/35.jpg)
35
Compute
![Page 36: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/36.jpg)
36Condor Basics
Job list � ClassAd
3. Sistemas de gestión de colas : Condor
![Page 37: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/37.jpg)
37
Resource list � ClassAd
![Page 38: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/38.jpg)
38
Matchmaking
![Page 39: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/39.jpg)
39Condor Basics
Priority Management
![Page 40: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/40.jpg)
40
Data
Transfer
![Page 41: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/41.jpg)
41Condor Basics
3. Sistemas de gestión de colas : Condor
Job running
![Page 42: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/42.jpg)
42
Job Monitoring
![Page 43: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/43.jpg)
43
Job End
![Page 44: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/44.jpg)
44
Example
![Page 45: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/45.jpg)
45
Hello, World !!
#!/bin/sh# I’m hola.shecho Hola mundo desde `hostname`
# # A Hello World .. In Condor!# # I’m hello.subUniverse = vanilla Executable = hola.shLog = hola.logOutput = hola.outError = hola.errQueue
![Page 46: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/46.jpg)
46Lanzar el cálculo
condor_submit
4. Condor Basics – Un cálculo fácil
![Page 47: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/47.jpg)
47Lanzar el cálculo
condor_q
![Page 48: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/48.jpg)
48
Something tastier…#!/bin/sh# I’m hello2.shOUTPUT=hello${1}.resultcat hello.input >> $OUTPUT cat echo Hello world, I’m job $1 here from
`hostname` > $OUTPUT
# Execute n times with different outputsUniverse = vanillaExecutable = hello2.shTransfer_input_files = hello.inputWhenToTransferOutput = ON_EXIT_OR_EVICTArguments = $(Process)Log = hello.logOutput = hello.outQueue 10
![Page 49: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/49.jpg)
49
Perfect Simulation
4. Condor Basics – Un cálculo fácil
![Page 50: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/50.jpg)
50
Extra Bonus
![Page 51: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/51.jpg)
51
Dynamic Partitioning
![Page 52: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/52.jpg)
52Condor Basics
Configurable Jobs
![Page 53: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/53.jpg)
53
Advanced Accounting
![Page 54: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/54.jpg)
54
Dynamic Checkpointing
![Page 55: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/55.jpg)
55Condor Basics
Hadoop Integration
3. Sistemas de gestión de colas : Condor
![Page 56: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/56.jpg)
56
Green Computing
![Page 57: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/57.jpg)
57Condor Basics
GPU Integration
![Page 58: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/58.jpg)
58
I3A & Condor
![Page 59: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/59.jpg)
59
Gaming IA
![Page 60: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/60.jpg)
60
MRI
Brain
Analysis
![Page 61: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/61.jpg)
61Communication
Systems
![Page 62: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/62.jpg)
62
Tissue Modelling
![Page 63: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/63.jpg)
63Condor Basics
3. Sistemas de gestión de colas : Condor
![Page 64: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/64.jpg)
64Condor Basics
> Conclusiones
3. Sistemas de gestión de colas : Condor
![Page 65: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/65.jpg)
65
Example
![Page 66: Solving BIG problems with Open Source: Condor](https://reader033.fdocuments.us/reader033/viewer/2022052303/554bd243b4c905706a8b4d69/html5/thumbnails/66.jpg)
66
Antonio Sanz
@antoniosanzalc
Slides here:
http://web.hermes.cps.unizar.es/doc/condor.pdf