1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo...
-
Upload
evan-graves -
Category
Documents
-
view
217 -
download
0
Transcript of 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo...
![Page 1: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/1.jpg)
1
SIMGRID_SchedulerA Simulated Scheduling System
for GRID Environment
Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONEDepartment of Electrical and Electronic Engineering, Politecnico di Bari
Domenico DIBARI, Eugenio NAPPI INFN Bari
![Page 2: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/2.jpg)
2
Why Simulate?• We haven’t a working GRID system yet• We can test scheduling algorithms without occupying physical resources• The simulated environment acts like the physical one should do (no
bugs and failures)• Simulation is faster (we don’t have to wait the real execution of the job)
GoalEvaluation of super-scheduling algorithms
![Page 3: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/3.jpg)
3
Identifying the Broker/Scheduler System
The Grid Architecture
WORKLOADMANAGEMENT
SYSTEM
GRID INFORMATION
SPACE
RESOURCES
USERS
![Page 4: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/4.jpg)
4
Identifying the Broker/Scheduler System
LOGGING & BOOKEEPING
GRIDINFORMATION
SERVICE
Workload Management System ArchitectureR
ES
OU
RC
E IN
FO
Job Status
Job Submit E
vent
Resource Info
Resource Info
Resource InfoRESOURCE
Resource InfoRESOURCE
JobSubmit
Job SpecificationsUSER
INTERFACE
Job specificationsspecialization
Job specifications
specializedJob S
tatus
BROKER SCHEDULER
Job Status JOB SUBMISSION SERVICES
Job S
tatus
RESOURCE
Ref: Integrating GRID tools to build a Computing resource broker: activities of DataGrid WP1
![Page 5: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/5.jpg)
5
Identifying the Broker/Scheduler System
Broker/Scheduler System Architecture
SCHEDULER
ResourceInfo
Job specifications
Job specifications+
Resource SpecificationsJob Stat
usBROKER
Job specifications
Job specifications
![Page 6: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/6.jpg)
6
Formalizing the Model
![Page 7: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/7.jpg)
7
Formalizing the Model: How it works
SCHEDULER
USER
SCHEDULED JOBS
LOCAL RESOURCES MANAGERS
BROKER
INFORMATION SPACE
MATCHING
DISPATCH
RESCHEDULE
SCHEDULE
![Page 8: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/8.jpg)
8
Formalizing the Model: Configuration Files
• There are 3 configuration files:– CL.INI (Computing Level configuration file)– RG.INI (Request Generator configuration file)– SG.INI (Simulator configuration file)
![Page 9: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/9.jpg)
9
Formalizing the Model:Computing and Storage
Resources
%number of computing element: 5
%maxtime,maxjobsinqueue,maxcount,totalnodes,freenodes,maxtotalmemory,maxsinglememory,refresh
#1
1000,5,10,5,5,64,256,5
#2
600,10,10,5,5, 64,256,5
#3
800,10,10,5,5, 64,256,5
#4
500,7,10,5,5, 64,256,5
#5
1000,10,10,5,5, 64,256,5
An Example - The Configuration File - CL.INI
![Page 10: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/10.jpg)
10
Formalizing the Model: UsersAn Example - The Configuration File - RG.INI
%Final Time [s]: 3000;
%Request Rate (probability of 1 request in 1 sec) [%] : 10;
%Max number of executions of the same job requested (maxcount) : 3;
%Min memory needed by job [Mb] (maxmaxmem) : 16;
%Max memory needed by job [Mb] (maxminmem) : 128;
%Max Duration Time for a job [s] (maxduration) : 500;
%Max Delay Time for a job [s] maxdelay : 100;
Job Specifications How we generate them
Number of executions of the same job Uniformly between 1 and maxcount
Minimun Memory Needed Uniformly between 1 and maxminmem
Maximun Memory Needed Uniformly between minmem and maxmaxmem
Duration (foreseen) Uniformly between 1 and maxduration
DelayTime Uniformly between 1 and maxdelay
![Page 11: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/11.jpg)
11
Formalizing the Model: Information Space
• It reports the state of the resources
• It is updated by resources every refresh seconds or by an event
![Page 12: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/12.jpg)
12
Formalizing the Model: Scheduler
• The scheduler calculates the index using the chosen algorithm and puts the user request in the buffer
An Example - The Configuration File - SG.INI
%Broker Rate (number of request managed by broker in 1 s) [s]: 4;
%Scheduling policy 0=EDF 1=FCFS 2=SJF: 0;
![Page 13: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/13.jpg)
13
Formalizing the Model: Broker
• The Broker gets the first request in the buffer• Queries the Information Space• Dispatch the Job• If it doesn’t find the suitable resource, it resubmits the request to the scheduler
![Page 14: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/14.jpg)
14
Metrics
Cumulative Value =
timedeadlinei
valueii i
ii
timeexecutiontotal
stoptimevalue )(
Cumulative value [0..100]
Missed deadlines [per cent]
System Usage [per cent]
Average Queue time [per cent of total time]
Average Execution time [per cent of total time]
![Page 15: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/15.jpg)
15
Status Report
• We have implemented 3 scheduling algorithms– FCFS First Come First Served
– EDF Earliest Deadline First
– SJF Shortest Job First
More algorithms will be added
• We have developed a first level simulation
• Brokering strategies need to be improved
![Page 16: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/16.jpg)
16
Technologies• UML is used to analyze and design the system
model and the simulation software
• C++ is used to develop software so that we can easily :
– reuse the same code to implement a real superscheduler
– use code developed by others (WP1)
• Telelogic Tau
![Page 17: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/17.jpg)
17
FCFS SIMULATION RESULTSCumulative value [0..100] : 79
Missed deadlines [per cent] : 23
System Usage [per cent] : 65
Average Queue time [per cent of total time]: 13
Average Execution time [per cent of total time]: 87
Final Time : 10895
0 2000 4000 6000 8000 10000 120000
2
4
6
8
10
time [s]
num
be
r o
f jo
bs
System Usage Requested jobs
0 2000 4000 6000 8000 10000 120000
10
20
30
40
50
time [s]
Num
be
r o
f jo
bs
Waiting Jobs
![Page 18: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/18.jpg)
18
EDF SIMULATION RESULTSCumulative value [0..100] : 85
Missed deadlines [per cent] : 16
System Usage [per cent] : 65
Average Queue time [per cent of total time]: 10
Average Execution time [per cent of total time]: 90
Final Time : 10998
0 2000 4000 6000 8000 10000 120000
2
4
6
8
10
time [s]
num
be
r o
f jo
bs
System Usage Requested jobs
0 2000 4000 6000 8000 10000 120000
10
20
30
40
time [s]
Num
be
r o
f jo
bs
Waiting Jobs
![Page 19: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/19.jpg)
19
SJF SIMULATION RESULTSCumulative value [0..100] : 85
Missed deadlines [per cent] : 15
System Usage [per cent] : 65
Average Queue time [per cent of total time]: 10
Average Execution time [per cent of total time]: 90
Final Time : 10913
0 2000 4000 6000 8000 10000 120000
2
4
6
8
10
time [s]
num
be
r o
f jo
bs
System Usage Requested jobs
0 2000 4000 6000 8000 10000 120000
10
20
30
40
time [s]
Num
be
r o
f jo
bs
Waiting Jobs
![Page 20: 1 SIMGRID_Scheduler A Simulated Scheduling System for GRID Environment Marcello CASTELLANO, Giacomo PISCITELLI and Nicola SIMEONE Department of Electrical.](https://reader035.fdocuments.us/reader035/viewer/2022062318/55154904550346c77d8b627b/html5/thumbnails/20.jpg)
20
Next Step• to consider the HEP computing model (Ex. Monarc project)
• to choose a class of applications (Ex: ALICE experiment apps)
• to determine an appropriate scheduling algorithm• to evaluate the right parameters
in order to design and develop a real
“Community Broker Scheduler”
using the DATAGRID tools available