Superscheduling and Resource Brokering Sven Groot (0024821)
-
Upload
august-marshall -
Category
Documents
-
view
218 -
download
0
Transcript of Superscheduling and Resource Brokering Sven Groot (0024821)
![Page 1: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/1.jpg)
Superscheduling and Resource Brokering
Sven Groot (0024821)
![Page 2: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/2.jpg)
Grid Information Service• Not all information available• Grid Information System
– Globus Monitoring and Discovery Service (MDS2)– Grid Monitoring Architecture (GMA)
• Common features– Organise sensors– Static vs. Dynamic data– Extensible– Agreed upon schema
![Page 3: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/3.jpg)
Stages of Grid Scheduling• Phase 1: Resource Discovery
– Authorization filtering– Application Requirement Definition– Minimal requirement filtering
![Page 4: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/4.jpg)
Stages of Grid Scheduling (2)• Phase 2: System Selection
– Dynamic information gathering– System Selection
![Page 5: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/5.jpg)
Stages of Grid Scheduling (3)• Phase 3: Job Execution
– Advance Reservation (optional)– Job Submission– Preparation Tasks– Monitoring Progress– Job Completion– Cleanup Tasks
![Page 6: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/6.jpg)
Application requirements
![Page 7: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/7.jpg)
Application Requirements (2)• General requirements
– Compute-related requirements– Data-related requirements– Network-related requirements
![Page 8: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/8.jpg)
Application Requirements (3)• Challenges
– Application deployment– Metacomputing– Predicting performance
• Theoretical prediction• History based prediction• Testcase-based prediction
– Adaptive brokering
![Page 9: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/9.jpg)
Application Requirements (4)• Related issues
– Application frameworks– Virtual Organizations– Security requirements– Accounting policies– User preferences
![Page 10: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/10.jpg)
Scheduling in GrADS• Scheduling phases
– Launch-time scheduling– Rescheduling– Meta-scheduling
![Page 11: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/11.jpg)
GrADS
![Page 12: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/12.jpg)
GrADS (2)• Focus applications
– ScaLAPACK– Cactus– FASTA– Iterative applications
• Jacobi method• Game of Life• Fish
![Page 13: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/13.jpg)
GrADS: Launch-time scheduling
![Page 14: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/14.jpg)
GrADS: Launch-time scheduling (2)
• Configurable Object Program – Application requirements definition
• AART• ClassAds• Redline
![Page 15: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/15.jpg)
ClassAds sample
![Page 16: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/16.jpg)
GrADS: Launch-time scheduling (3)
• Performance model– General method
• develop an analytic model for well-understood aspects of applicatio or system performance
• test the analytic model against achieved application performance
• develop empirical models for poorly-understood aspects of application or system behavior
– Some application specific methods– Implemented as shared libraries
![Page 17: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/17.jpg)
GrADS: Launch-time scheduling (4)
• Mapper– Maps data and/or tasks to resources– Different mapping methods
• Equal allocation• Time balancing• Data locality
![Page 18: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/18.jpg)
GrADS: Launch-time scheduling (5)
• Search procedure– General steps
• identify a large number of sets of resources that may be good platforms for the application
• use the application-specific mapper and performance model to generate a data map and predicted execution time for those resource sets
• select the resource set that results in the lowest predicted execution time
![Page 19: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/19.jpg)
GrADS: Launch-time scheduling (6)
• Resource-aware search
![Page 20: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/20.jpg)
GrADS: Launch-time scheduling (6)
• Simulated Annealing
![Page 21: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/21.jpg)
GrADS: Rescheduling• Additional complexities
– Lack of built-in mechanisms– Need to distinguish processors that are
running/not running the current process– Overheads can be high
![Page 22: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/22.jpg)
GrADS: Rescheduling (2)
![Page 23: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/23.jpg)
GrADS: Rescheduling (2)• Rescheduling methods
– Application migration– Process swapping
![Page 24: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/24.jpg)
GrADS: Metascheduling
![Page 25: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/25.jpg)
Grid Service Level Agreements
• Contract– Provide some capability– Perform some task
• Types of SLAs– Resource Service Level Agreements– Task Service Level Agreements– Binding Service Level Agreements
![Page 26: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/26.jpg)
Grid SLAs (2)
![Page 27: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/27.jpg)
Grid SLAs (3)• Motivating scenarios
– Community Scheduler Scenario
![Page 28: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/28.jpg)
Grid SLAs (4)• Motivating scenarios (cont’d)
– File transfer scenario
![Page 29: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/29.jpg)
Grid SLAs• Resource virtualization
![Page 30: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/30.jpg)
Multicriteria• Basic definitions
– Pareto Dominance– Pareto Optimality– Pareto-optimal set– Pareto Front
![Page 31: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/31.jpg)
Multicriteria (2)• Motivations
– Various stakeholders and their preferences– Job scheduling– Application-Level scheduling– Hard constraints and soft constraints
![Page 32: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/32.jpg)
Multicriteria (3)• Approach
– Criteria• Related to stakeholders• Related to entire system• Time criteria• Cost criteria• Resource utilization criteria
– Modeling preferences
![Page 33: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/33.jpg)
Multicriteria (4)• Selection method
– Rule-based system requirements• Expression of policies• Execution of different scheduling procedures• Adaptation to the environment• Selection of the best solution
– Multicriteria optimization
![Page 34: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/34.jpg)
![Page 35: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/35.jpg)
Example (cont’d)
![Page 36: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/36.jpg)
Example (cont’d)• Aggregate criteria
– End user satisfaction
– Resource Owner Satisfaction
– VO overall performance
![Page 37: Superscheduling and Resource Brokering Sven Groot (0024821)](https://reader035.fdocuments.us/reader035/viewer/2022062803/56649f1c5503460f94c329f3/html5/thumbnails/37.jpg)
Example (cont’d)