Weaving the World-Wide Grid [Marketplace]:

Rajkumar Buyya

Melbourne, Australiawww.buyya.com/ecogrid

WW Grid

“Economic Paradigm for Distributed Resource Management and Scheduling for Service-Oriented Computing”

2

3

Vision: Grid for Service Oriented Computing?

WW Grid

World Wide Grid!

Nimrod-G

4

Overview

A quick glance at Grid computing Resource Management challenges for next

generation Grid computing A Glance at Approaches to Grid computing. Grid Architecture for Computational

Economy Nimrod-G -- Grid Resource Broker Scheduling Experiments on

the World Wide Grid: both Real and Simulation

ConclusionsScheduling Economics

Grid

EconomyGrid

5

2100

2100 2100 2100 2100

2100 2100 2100 2100

Desktop SMPs or SuperComputers

LocalCluster

GlobalCluster/Grid

PERFORMANCE

Inter PlanetaryGrid!

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Galaxy

Administrative Barriers

EnterpriseCluster/Grid

?

Scalable HPC: Breaking Administrative Barriers & new challenges

6

Why SC? Large Scale Explorations need them—Killer Applications.

Solving grand challenge applications using modeling, simulation and analysis

Life Sciences

CAD/CAM

Aerospace

Military ApplicationsDigital Biology Military ApplicationsMilitary Applications

Internet & Ecommerce

7

What is Grid ?

A paradigm/infrastructure that allows sharing, selection, & aggregationof geographically distributed resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators.

[depending on their availability, capability, cost, and user QoS requirements]

for solving large-scale problems/applications. Thus enabling the creation of “virtual enterprises” (VEs)

Widearea

8

P2P/Grid Applications-Drivers

Distributed HPC (Supercomputing): Computational science.

High-Capacity/Throughput Computing: Large scale simulation/chip design & parameter studies.

Content Sharing (free or paid) Sharing digital contents among peers (e.g., Napster)

Remote software access/renting services: Application service provides (ASPs) & Web services.

Data-intensive computing: Drug Design, Particle Physics, Stock Prediction...

On-demand, realtime computing: Medical instrumentation & Mission Critical.

Collaborative Computing: Collaborative design, Data exploration, education.

Service Oriented Computing (SOC): Computing as Competitive Utility: New paradigm, new

industries, and new business.

9

Building and Using Grids require

Services that enable the execution of a job on a resource in different admistrative domain.

Security mechanisms that permit resources to be accessed only by authorized users.

App/Data Security (?) – A must for commercial users (protecting from GSPs/other users).

(New) programming tools that make our applications Grid Ready!.

Tools that can translate the requirements of an application/user into the requirements of computers, networks, and storage.

Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results.

Globus

Nimrod-G

Resource Management Challenges in

Grid Computing Environments

11

A Typical Grid Computing Environment

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

12

What users want ?Users in Grid Economy &

Strategy Grid Consumers

Execute jobs for solving varying problem size and complexity

Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost

Strategy: minimise expenses Grid Providers

Contribute (“idle”) resource for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity

Strategy: maximise return on investment

13

Sources of Complexity in Grid for Resource Management and Scheduling

Size (large number of nodes, providers, consumers) Heterogeneity of resources (PCs, Workstations, clusters, and

supercomputers, instruments, databases, software) Heterogeneity of fabric management systems (single system image OS,

queuing systems, etc.) Heterogeneity of fabric management polices Heterogeneity of application requirements (CPU, I/O, memory, and/or

network intensive) Heterogeneity in resource demand patterns (peak, off-peak, ...) Applications need different QoS at different times (time critical results). The

utility of experimental results varies from time to time. Geographical distribution of users & located different time zones Differing goals (producers and consumers have different objectives and

strategies) Unsecure and Unreliable environment

14

Need Grid tools for managing

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Development Tools

15

Traditional approaches to resource management & scheduling are NOT useful

for Grid ? They use centralised policy that need

complete state-information and common fabric management policy or decentralised consensus-

based policy. Due to too many heterogenous parameters in the Grid it is

impossible to define/get: system-wide performance matrix and common fabric management policy that is acceptable to all.

“Economic” paradigm proved as an effective institution in managing decentralization and heterogeneity that is present in human economies!

Hence, we propose/advocate the use of “computational economy” principles in the management of resources and scheduling computations on the Grid.

16

Benefits of Computational Economies

It provides a nice paradigm for managing self interested and self-regulating entities (resource owners and consumers)

Helps in regulating supply-and-demand for resources. Services can be priced in such a way that equilibrium is maintained.

User-centric / Utility driven: Value for money! Scalable:

No need of central coordinator (during negotiation) Resources(sellers) and also Users(buyers) can make their own decisions and try to

maximize utility and profit. Adaptable It helps in offering different QoS (quality of services) to different applications

depending the value users place on them. It improves the utilisation of resources It offers incentive for resource owners for being part of the grid! It offers incentive for resource consumers for being good citizens There is large body of proven Economic principles and techniques available, we can

easily leverage it.

17

New challenges of Computational Economy

Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …..

Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How I trade between timeframe & cost ? ….

Any tools, traders & brokers available to automate the process ?

18

mix-and-match

Object-oriented

Internet/partial-P2P

Network enabled Solvers

Market/Computational Economy

19

Building an Economy Grid(Next Generation Grid

Computing!)

To enable the creation and promotion of:Grid Marketplace (competitive)

ASPService Oriented Computing

. . .And let users focus on their own work (science, engineering, or commerce)!

20

Grid Node N

GRACE: A ReferenceGrid Architecture for Computational Economy

Grid Consumer

Pro

gra

mm

ing

En

viro

nm

ents

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Grid Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

21

Grid Node N

GRACE: A ReferenceGrid Architecture for Computational Economy

Grid User

Application

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Server(s)

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Grid Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

See PDPTA 2000 paper!

22

Economic Models

Price-based: Supply,demand,value, wealth of economic system

Commodity Market Model Posted Price Model Bargaining Model Tendering (Contract Net) Model Auction Model

English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate)

Proportional Resource Sharing Model Monopoly (one provider) and Oligopoly (few players)

consumers may not have any influence on prices. Bartering

Shareholder Model Partnership Model

See SPIE ITCom 2001 paper!: with Heinz Stockinger, CERN!

23

Cost Model

Without cost model any shared system becomes un-managable

Charge users more for remote facilities than their own

Choose cheaper resources before more expensive ones

Cost units (G$) may be Dollars Shares in global facility Stored in bank

24

Cost Matrix @ Grid site X

Non-uniform costing Encourages use of

local resources first Real accounting

system can control machine usage

11 33

22 11User 5User 5

Mach

ine 1

Mach

ine 1

User 1User 1

Mach

ine 5

Mach

ine 5

Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.)

Simple: price based on peaktime, offpeak, discount when less demand, ..

Nimrod-G:The Grid Resource Broker

Soft Deadline and Budget-based Economy Grid Resource Broker

for Parameter (Task Farming Applications) Processing on

Grids

26

A resource broker for managing, steering, and executing task farming (parameter sweep/SPMD model) applications on Grid based on deadline and computational economy.

Based on users’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability.

Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting

Nimrod/G : A Grid Resource Broker

27

Parametric Computing(What Users think of Nimrod

Power)

Multiple RunsSame ProgramMultiple Data Killer Application for the Grid!

ParametersAge Hair

23 CleanAge Hair

23 Clean23 Beard28 Goatee

Age Hair23 Clean23 Beard

Age Hair23 Clean23 Beard28 Goatee28 Clean

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean

-4000000 Too much

Courtesy: Anand Natrajan, University of Virginia

Magic Engine

28

Sample P-Sweep/Task Farming Applications

Sample P-Sweep/Task Farming Applications

Bioinformatics: Bioinformatics: Drug Design / Protein Drug Design / Protein

ModellingModelling

SensitivitySensitivityexperiments experiments

on smog formationon smog formation

Combinatorial Combinatorial Optimization:Optimization:

Meta-heuristic Meta-heuristic parameter estimationparameter estimation

Ecological Modelling: Ecological Modelling: Control Strategies Control Strategies

for Cattle Tickfor Cattle Tick

Electronic CAD: Electronic CAD: Field Programmable Field Programmable

Gate ArraysGate ArraysComputer Graphics: Computer Graphics: Ray TracingRay Tracing

High Energy High Energy Physics: Physics:

Searching for Searching for Rare EventsRare Events

Finance: Finance: Investment Risk AnalysisInvestment Risk Analysis

VLSI Design: VLSI Design: SPICE SimulationsSPICE Simulations

Aerospace: Aerospace: Wing DesignWing Design

Network SimulationNetwork SimulationAutomobile:Automobile:

Crash Simulation Crash Simulation

Data MiningData Mining

Civil Engineering:Civil Engineering:Building Design Building Design

astrophysics astrophysics

29

Distributed Drug Design: Data Intensive Computing on the

Grid

A Virtual Laboratory environment for “Molecular Docking for Drug Design” on the Grid.

It provides tools for screening millions of chemical compounds (molecules) in the Chemical DataBase (CDB) to identify those having potential use in drug design (acts as inhibitor).

In collaboration with: Kim Branson, Structural Biology,

Walter and Eliza Hall Institute (WEHI)

http://www.buyya.com/vlab

30

Docking Application Input data configuration file

score_ligand yesminimize_ligand yesmultiple_ligands norandom_seed 7anchor_search notorsion_drive yesclash_overlap 0.5conformation_cutoff_factor 3torsion_minimize yesmatch_receptor_sites norandom_search yes . . . . . . . . . . . .maximum_cycles 1ligand_atom_file S_1.mol2receptor_site_file ece.sphscore_grid_prefix ecevdw_definition_file parameter/vdw.defnchemical_definition_file parameter/chem.defnchemical_score_file parameter/chem_score.tblflex_definition_file parameter/flex.defnflex_drive_file parameter/flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2

Molecule to Molecule to be screenedbe screened

31

score_ligand $score_ligandminimize_ligand $minimize_ligandmultiple_ligands $multiple_ligandsrandom_seed $random_seedanchor_search $anchor_searchtorsion_drive $torsion_driveclash_overlap $clash_overlapconformation_cutoff_factor $conformation_cutoff_factortorsion_minimize $torsion_minimizematch_receptor_sites $match_receptor_sitesrandom_search $random_search . . . . . . . . . . . .maximum_cycles $maximum_cyclesligand_atom_file ${ligand_number}.mol2receptor_site_file $HOME/dock_inputs/${receptor_site_file}score_grid_prefix $HOME/dock_inputs/${score_grid_prefix}vdw_definition_file vdw.defnchemical_definition_file chem.defnchemical_score_file chem_score.tblflex_definition_file flex.defnflex_drive_file flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2

Parameterize Dock input file(use Nimrod Tools: GUI/language)

Molecule to be Molecule to be screenedscreened

32

parameter database_name label "database_name" text select oneof "aldrich" "maybridge" "maybridge_300" "asinex_egc" "asinex_epc" "asinex_pre" "available_chemicals_directory" "inter_bioscreen_s" "inter_bioscreen_n" "inter_bioscreen_n_300" "inter_bioscreen_n_500" "biomolecular_research_institute" "molecular_science" "molecular_diversity_preservation" "national_cancer_institute" "IGF_HITS" "aldrich_300" "molecular_science_500" "APP" "ECE" default "aldrich_300";

parameter CDB_SERVER text default "bezek.dstc.monash.edu.au";parameter CDB_PORT_NO text default "5001";parameter score_ligand text default "yes";parameter minimize_ligand text default "yes";parameter multiple_ligands text default "no";parameter random_seed integer default 7;parameter anchor_search text default "no";parameter torsion_drive text default "yes";parameter clash_overlap float default 0.5;parameter conformation_cutoff_factor integer default 5;parameter torsion_minimize text default "yes";parameter match_receptor_sites text default "no"; . . . . . . . . . . . .parameter maximum_cycles integer default 1;parameter receptor_site_file text default "ece.sph";parameter score_grid_prefix text default "ece";parameter ligand_number integer range from 1 to 2000 step 1;

Create Dock PlanFile1. Define parameters and their value

Molecules to be Molecules to be screenedscreened

33

task nodestart copy ./parameter/vdw.defn node:. copy ./parameter/chem.defn node:. copy ./parameter/chem_score.tbl node:. copy ./parameter/flex.defn node:. copy ./parameter/flex_drive.tbl node:. copy ./dock_inputs/get_molecule node:. copy ./dock_inputs/dock_base node:.endtasktask main node:substitute dock_base dock_run node:substitute get_molecule get_molecule_fetch node:execute sh ./get_molecule_fetch node:execute $HOME/bin/dock.$OS -i dock_run -o dock_out copy node:dock_out ./results/dock_out.$jobname copy node:dock_cnt.mol2 ./results/dock_cnt.mol2.$jobname copy node:dock_chm.mol2 ./results/dock_chm.mol2.$jobname copy node:dock_nrg.mol2 ./results/dock_nrg.mol2.$jobnameendtask

Create Dock PlanFile2. Define the task that each job needs to

do

34

Nimrod-G Broker Automating Distributed Processing

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

South

Compose, Submit, & Play!

35

Nimrod & Associated Family of Tools

P-sweep App. Composition: Nimrod/

EnfusionResource Management and Scheduling:

Nimrod-G BrokerDesign Optimisations:

Nimrod-OApp. Composition and Online Visualization:

Active SheetsGrid Simulation in Java:

GridSimDrug Design on Grid:

Virtual Lab

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

South

Remote Execution Server(on demand Nimrod Agent)

File Transfer Server

36

A Glance at Nimrod-G Broker

Grid Middleware

Nimrod/G Client Nimrod/G ClientNimrod/G Client

Grid Information Server(s)

Schedule Advisor

Trading Manager

Nimrod/G Engine

GridStore

Grid Explorer

GE GISTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

Globus, Legion, Condor, etc.

G

G

CL

Globus enabled node.Legion enabled node.

GL

Condor enabled node.

RM & TSRM & TS

C LSee HPCAsia 2000 paper!

37

Globus Legion

Fabric

Nimrod-G Broker

Nimrod-G ClientsP-Tools (GUI/Scripting)

(parameter_modeling)

Legacy Applications

P2P GTS

Farming Engine

Dispatcher & Actuators

Schedule Advisor

Trading Manager

Grid Explorer

Customised Apps(Active Sheet)

Monitoring and Steering Portals

Algorithm1

AlgorithmN

Middleware

. . .

Computers Storage Networks InstrumentsLocal Schedulers

G-Bank. . .

Agents

Resources

Programmable Entities Management

Jobs Tasks

. . .

AgentScheduler JobServer

PC/WS/Clusters Radio TelescopeCondor/LL/NQS . . .Database

Meta-Scheduler

Nimrod/G Grid Broker Architecture

Channels

. . .

Database

Condor GMD

IP hourglass!

Condor-AGlobus-A Legion-A P2P-A

38

A Nimrod/G Monitor

A Nimrod/G Monitor

CostCostDeadlineDeadline

Legion hosts

Globus Hosts

Bezek is in both Globus and Legion Domains

Arlington

Alexandria

Richmond

HamptonNorfolk

Virginia BeachChesapeakePortsmouth

Newport News

Roanoke

Ap p om a toxRive r

Ja m esRive r

Shena nd oa hRive r

Ra p p a ha nnoc kRive r

Potom a cRive r

VIRGINIA77

81

64

64

66

85

39

User Requirements: Deadline/Budget User Requirements: Deadline/Budget

40

Another User Interface:Active Sheet for Spreadsheet Processing on

Grid

NimrodNimrodProxyProxy

Nimrod/GNimrod/G

41

42

Nimrod/G Interactions

Grid InfoServer

ProcessServer

UserProcess

File accessFileServer

Grid Node

NimrodAgent

Compute NodeUser Node

GridDispatcher

Grid Trade Server

GridScheduler

Local Resource Manager

Nimrod-G Grid Broker

TaskFarmingEngine

Grid ToolsAnd

Applications

Do this in 30 min. for $10?

43

Discover Discover ResourcesResources

Distribute JobsDistribute Jobs

Establish Establish RatesRates

Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?

Evaluate & Evaluate & RescheduleReschedule

Discover Discover More More

ResourcesResources

Adaptive Scheduling Steps

Compose & Compose & ScheduleSchedule

44

Deadline and Budget Constrained Scheduling Algorithms

Algorithm/Strategy

Execution Time(Deadline, D)

Execution Cost(Budget, B)

Cost Opt Limited by D Minimize

Cost-Time Opt Minimize when possible

Minimize

Time Opt Minimize Limited by B

Conservative-Time Opt

Minimize Limited by B, but all unprocessed jobs have guaranteed minimum budget

Application Scheduling Experiments on the World-

Wide Grid

Task Farming Applications on

World Wide Grid

WW Grid

46

The World Wide Grid SitesWW Grid

EUROPE:ZIB/GermanyPC2/GermanyAEI/Germany Lecce/ItalyCNR/ItalyCalabria/ItalyPozman/PolandLund/SwedenCERN/SwissCUNI/Czech R.Vrije: Netherlands

EUROPE:ZIB/GermanyPC2/GermanyAEI/Germany Lecce/ItalyCNR/ItalyCalabria/ItalyPozman/PolandLund/SwedenCERN/SwissCUNI/Czech R.Vrije: Netherlands

ANL/ChicagoUSC-ISC/LA

UTK/TennesseeUVa/Virginia

Dartmouth/NHBU/Boston

UCSD/San Diego

ANL/ChicagoUSC-ISC/LA

UTK/TennesseeUVa/Virginia

Dartmouth/NHBU/Boston

UCSD/San Diego

Monash/MelbourneVPAC/Melbourne

Monash/MelbourneVPAC/MelbourneSantiago/Chile

Santiago/Chile

TI-Tech/TokyoETL/TsukubaAIST/Tsukuba

TI-Tech/TokyoETL/TsukubaAIST/Tsukuba

Cardiff/UKPortsmoth/UKManchester, UK

Cardiff/UKPortsmoth/UKManchester, UK

Kasetsart/BangkokKasetsart/Bangkok

SingaporeSingapore

47

World Wide Grid (WWG)WW Grid

Globus+LegionGRACE_TS

Australia

Monash U. : Cluster

VPAC: Alpha

Solaris WS

Nimrod/G

Globus +GRACE_TS

Europe

ZIB: T3E/OnyxAEI: Onyx Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCalabria: Cluster CERN: ClusterCUNI/CZ: OnyxPozman: SGI/SP2Vrije U: ClusterCardiff: Sun E6500Portsmouth: Linux PCManchester: O3K

Globus +GRACE_TS

Asia

Tokyo I-Tech.: Ultra WSAIST, Japan: Solaris ClusterKasetsart, Thai: ClusterNUS, Singapore: O2K

Globus/LegionGRACE_TS

North America

ANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux ClusterUD: Linux clusterUTK: Linux clusterUCSD: Linux PCsBU: SGI IRIX

Internet

Globus +GRACE_TS South America

Chile: Cluster

WW Grid

48

Experiment-1: Peak and Off-peak

Workload: 165 jobs, each need 5 minute of cpu time

Deadline: 1 hrs. and budget: 800,000 units

Strategy: Minimize Cost and meet the deadline

Execution Cost with cost optimisation AU Peaktime:471205 (G$) AU Offpeak time: 427155 (G$)

49

Resources Selected & Price/CPU-sec.

Resource Type & Size

Owner and Location

Grid services

Peaktime Cost (G$)

Offpeak cost

Linux cluster (60 nodes)

Monash, Australia

Globus/Condor 20 5

IBM SP2 (80 nodes)

ANL, Chicago, US

Globus/LL 5 10

Sun (8 nodes) ANL, Chicago, US

Globus/Fork 5 10

SGI (96 nodes) ANL, Chicago, US

Globus/Condor-G

15 15

SGI (10 nodes) ISI, LA, US Globus/Fork 10 20

50

Execution @ AU Peak Time

0

2

4

6

8

10

12

Time (minutes)

Jo

bs

Linux clus ter - Monash (20) Sun - ANL (5) SP2 - ANL (5) SGI - ANL (15) SGI - ISI (10)

51

Execution @ AU Offpeak Time

0

2

4

6

8

10

12

Time (minutes)

Jo

bs

Linux clus ter - Monash (5) Sun - ANL (10) SP2 - ANL (10) SGI - ANL (15) SGI - ISI (20)

52

Experiment-2 Setup

Workload: 165 jobs, each need 5 minute of CPU time

Deadline: 2 hrs. and budget: 396000 G$ Strategies: 1. Minimise cost 2. Minimise time Execution:

Optimise Cost: 115200 (G$) (finished in 2hrs.) Optimise Time: 237000 (G$) (finished in 1.25 hr.) In this experiment: Time-optimised scheduling run

costs double that of Cost-optimised. Users can now trade-off between Time Vs. Cost.

53

Resources Selected & Price/CPU-sec.

Resource & Location

Grid services & Fabric

Cost/CPU sec.or unit

No. of Jobs Executed

Time_Opt Cost_Opt.

Linux Cluster-Monash, Melbourne, Australia

Globus, GTS, Condor

2 64 153

Linux-Prosecco-CNR, Pisa, Italy

Globus, GTS, Fork 3 7 1

Linux-Barbera-CNR, Pisa, Italy

Globus, GTS, Fork 4 6 1

Solaris/Ultas2

TITech, Tokyo, Japan

Globus, GTS, Fork 3 9 1

SGI-ISI, LA, US Globus, GTS, Fork 8 37 5

Sun-ANL, Chicago,US Globus, GTS, Fork 7 42 4Total Experiment Cost (G$) 237000 115200

Time to Complete Exp. (Min.) 70 119

54

Resource Scheduling for DBC Time Optimization

0

2

4

6

8

10

12

Time (in Minute)

No.

of

Tas

ks i

n E

xecu

tion

Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR

Solaris /Ultas2-TITech SGI-ISI Sun-ANL

55

Resource Scheduling for DBC Cost Optimization

0

2

4

6

8

10

12

14

Time (in Minute)

No.

of

Tas

ks i

n E

xecu

tion

Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR

Solaris /Ultas2-TITech SGI-ISI Sun-ANL

56

Experiment-3 Setup: Using GridSim

Workload Synthesis: 200 jobs, each job processing requirement = 10K

MI or SPEC with random variation from 0-10%. Exploration of many scenarios:

Deadline: 100 to 3600 simulation time, step = 500 Budget: 500 to 22000 G$, step = 1000

DBC Strategies: Cost Optimisation Time Optimisation

Resources: Simulated WWG resources

57

Simulated WWG Resources

Resource Name in

Simulation

Simulated Resource Characteristics

Vendor, Resource Type, Node OS, No of PEs

Equivalent Resource in

Worldwide Grid

(Hostname, Location)

A PE SPEC/ MIPS Rating

Resource Manager

Type

Price

(G$/PE time unit)

MIPS per G$

R0 Compaq, AlphaServer,

CPU, OSF1, 4

grendel.vpac.org,

VPAC, Melb, Australia

515 Time-shared 8 64.37

R1 Sun, Ultra, Solaris, 4 hpc420.hpcc.jp,

AIST, Tokyo, Japan 377 Time-shared 4 94.25

R2 Sun, Ultra, Solaris, 4 hpc420-1.hpcc.jp,

AIST, Tokyo, Japan 377 Time-shared 3 125.66

R3 Sun, Ultra, Solaris, 2 hpc420-2.hpcc.jp,

AIST, Tokyo, Japan 377 Time-shared 3 125.66

R4 Intel, Pentium/VC820,

Linux, 2 barbera.cnuce.cnr.it,

CNR, Pisa, Italy 380 Time-shared 2 190.0

R5 SGI, Origin 3200, IRIX, 6 onyx1.zib.de,

ZIB, Berlin, Germany 410 Time-shared 5 82.0

R6 SGI, Origin 3200, IRIX,

16 Onyx3.zib.de,

ZIB, Berlin, Germany 410 Time-shared 5 82.0

R7 SGI, Origin 3200, IRIX,

16

mat.ruk.cuni.cz, Charles U., Prague,

Czech Republic 410 Space-shared 4 102.5

R8 Intel, Pentium/VC820,

Linux, 2

marge.csm.port.ac.uk,

Portsmouth, UK 380 Time-shared 1 380.0

R9 SGI, Origin 3200, IRIX, 4

(accessible) green.cfs.ac.uk, Manchester, UK

410 Time-shared 6 68.33

R10 Sun, Ultra, Solaris, 8, pitcairn.mcs.anl.gov, ANL, Chicago, USA

377 Time-shared 3 125.66

58

DBC Cost Optimisation

5000

9000

13000

17000

21000

10

0

60

0

11

00

16

00

21

00

26

00

31

00

36

00

0

20

40

60

80

100

120

140

160

180

200

Gridlets

Budget

Deadline

100

600

1100

1600

2100

2600

3100

3600

5000

9000

13000

17000

21000

10

0

60

0

11

00

16

00

21

00

26

00

31

00

36

00

0

500

1000

1500

2000

2500

3000

3500

4000

Deadline Time Utilised

Budget

Deadline

100

600

1100

1600

2100

2600

3100

3600

50

00

70

00

90

00

11

00

0

13

00

0

15

00

0

17

00

0

19

00

0

21

00

0

10

0

11

00

21

00

31

00

0

5000

10000

15000

20000

25000

Budget Spent

Budget

Deadline

100

600

1100

1600

2100

2600

3100

3600

59

DBC Time Optimisation

5000

8000 11

000 1400

0

1700

0

2000

0

100

600

1100

1600

2100

2600

3100

3600

0

20

40

60

80

100

120

140

160

180

200

Gidlets Completed

Budget

Deadline

100

600

1100

1600

2100

2600

3100

3600

Time Optimise

5000

7000

9000

1100

0

1300

0

1500

0

1700

0

1900

0

2100

0

100

1100 21

00 3100

0

500

1000

1500

2000

2500

3000

3500

4000

Time Utilised

BudgetDeadline

100

600

1100

1600

2100

2600

3100

3600

Time-Optimise

5000

9000

13000

17000

21000

100

600

1100

1600

2100

2600

3100

3600

0

5000

10000

15000

20000

25000

Budget Spent

Budget

Deadline

100

600

1100

1600

2100

2600

3100

3600

Time Optimise

60

Comparison: D = 3100, B varied

0

500

1000

1500

2000

2500

3000

3500

5000

7000

9000

1100

0

1300

0

1500

0

1700

0

1900

0

2100

0

Budget

Dea

dli

ne

Sp

ent

Cost Optimisation

Time Optimisation

Deadline = 3100 Completion Time

0

5000

10000

15000

20000

25000

5000

7000

9000

1100

0

1300

0

1500

0

1700

0

1900

0

2100

0

Budget

Bu

dg

et S

pen

t

Cost Optimisation

Time Optimisation

Deadline = 3100 Processing Expenses

Time Opt

Execution Time vs. Budget

Execution Cost vs. Budget

Cost Opt

Conclude with a comparison to the Electrical Grid………..

Where we are ????

Courtesy: Domenico Laforenza

Alessandro Volta in Paris in 1801 inside French National Institute shows the battery

while in the presence of Napoleon I

Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence

University)

63

….and in the future, I imagine a WorldwidePower (Electrical) Grid …...

What ?!?!This is a mad man…

Oh, monDieu !

64

2002 - 1801 = 201 Years

65

Electric Grid Management and Delivery methodology is highly

advanced

Central Grid

Regional Grid

Regional Grid

Local Grid

Local Grid

Production Utility

Consumption

Whereas, our Computational Grid is in primitive/infancy state?

66

” I think there is a world market for about five computers.”

Thomas J. Watson Sr., IBM Founder, 1943

Can we Predict its Future ?

67

Summary and Conclusion

Grid Computing is emerging as a next generation computing platform for solving large scale problems through sharing of geographically distributed resources.

Resource management is a complex undertaking as systems need to be adaptive, scalable, competitive,…, and driven by QoS.

We proposed a framework based on “computational economies” for resource allocation and for regulating supply-and-demand for resources.

Scheduling experiments on the World Wide Grid demonstrate our Nimrod-G broker ability to dynamically lease services at runtime based on their quality, cost, and availability depending on consumers QoS requirements.

Easy to use tools for creating Grid applications are essential to attracting and getting application community on board.

The use of economic paradigm for resource management and scheduling is essential for pushing Grids into mainstream computing and weaving the World-Wide Grid Marketplace!

68

Download Software & Information

Nimrod & Parameteric Computing: http://www.csse.monash.edu.au/~davida/nimrod/

Economy Grid & Nimrod/G: http://www.buyya.com/ecogrid/

Virtual Laboratory Toolset for Drug Design: http://www.buyya.com/vlab/

Grid Simulation (GridSim) Toolkit (Java based): http://www.buyya.com/gridsim/

World Wide Grid (WWG) testbed: http://www.buyya.com/ecogrid/wwg/

Cluster and Grid Info Centres: www.buyya.com/cluster/ || www.gridcomputing.com

69

Selected GridSim Users!

70

Final Word?

Backup Slides

72

Further Information

Books: High Performance Cluster Computing, V1,

V2, R.Buyya (Ed), Prentice Hall, 1999. The GRID, I. Foster and C. Kesselman (Eds),

Morgan-Kaufmann, 1999. IEEE Task Force on Cluster Computing

http://www.ieeetfcc.org Global Grid Forum

www.gridforum.org

IEEE/ACM CCGrid’xy: www.ccgrid.org CCGrid 2002, Berlin: ccgrid2002.zib.de

Grid workshop - www.gridcomputing.org

73

Further Information

Cluster Computing Info Centre: http://www.buyya.com/cluster/

Grid Computing Info Centre: http://www.gridcomputing.com

IEEE DS Online - Grid Computing area:

http://computer.org/dsonline/gc

Compute Power Market Project http://www.ComputePower.com

74

75

Deadline and Budget-based Cost Minimization Scheduling

1. Sort resources by increasing cost.2. For each resource in order, assign as

many jobs as possible to the resource, without exceeding the deadline.

3. Repeat all steps until all jobs are processed.

76

Deadline and Budget Constraint (DBC) Time Minimization

Scheduling

1. For each resource, calculate the next completion time for an assigned job, taking into account previously assigned jobs.

2. Sort resources by next completion time.3. Assign one job to the first resource for

which the cost per job is less than the remaining budget per job.

4. Repeat all steps until all jobs are processed. (This is performed periodically or at each scheduling-event.)

77

DBC Conservative Time Min. Scheduling

1. Split resources by whether cost per job is less than budget per job.

2. For the cheaper resources, assign jobs in inverse proportion to the job completion time (e.g. a resource with completion time = 5 gets twice as many jobs as a resource with completion time = 10).

3. For the dearer resources, repeat all steps (with a recalculated budget per job) until all jobs are assigned.

4. [Schedule/Reschedule] Repeat all steps until all jobs are processed.

78

M - Resources, N - Jobs, D - deadline Note: Cost of any Ri is less than any of Ri+1 …. Or Rm

RL: Resource List need to be maintained in increasing order of cost Ct - Time when accessed (Time now) Ti - Job runtime (average) on Resource i (Ri) [updated periodically]

Ti is acts as a load profiling parameter. Ai - number of jobs assigned to Ri , where:

Ai = Min (No.Unassigned Jobs, No. Jobs Ri can complete by remaining deadline) No.UnAssignedJobsi = Diff( N, (A1+…+Ai-1)) JobsRi consume = RemainingTime (D- Ct) DIV Ti

ALG: Invoke Job Assignment() periodically until all jobs done. Job Assignment()/Reassignment():

Establish ( RL, Ct , Ti , Ai ) dynamically – Resource Discovery. For all resources (I = 1 to M) { Assign Ai Jobs to Ri , if required}

Deadline-based Cost-minimization Scheduling

79

What is Grid ?

An infrastructure that logically couples distributed resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators. and presents them as an integrated global resource

for solving large-scale problems. It enables the creation of virtual enterprise (VE) for

resource sharing and aggregation.

Widearea

data archives

80

Virtual Enterprise

A temporary alliance of enterprises or organizations that come together to share resources and skills, or competencies in order to better respond to business opportunities or challenges, and who cooperation is supported by computer

networks.

81

Many Testbeds ? & who pays ?,

who regulates supply and demand ?

GUSTO (decommissioned)

Legion Testbed

NASA IPG

World Wide Grid

WW Grid

82

Testbeds so far -- observations

Who contributed resources & why ? Volunteers: for fun, challenge, fame, charismatic apps, public

good like distributed.net & SETI@Home projects. Collaborators: sharing resources while developing new

technologies of common interest – Globus, Legion, Ninf, Ninf, MC Broker, Lecce GRB,... Unless you know lab. leaders, it is impossible to get access!

How long ? Short term: excitement is lost, too much of admin. Overhead

(Globus inst+), no incentive, policy change,… What we need ? Grid Marketplace!

Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.

83

Grid Open Trading Protocols

Get Connected

Call for Bid(DT)

Reply to Bid (DT)

Negotiate Deal(DT)

Confirm Deal(DT, Y/N)

….

Cancel Deal(DT)

Change Deal(DT)

Get Disconnected

Trade Manager

Trade Server

Pricing Rules

DT - Deal Template: - resource requirements (TM) - resource profile (TS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period

API

84

Layered Grid Architecture

Networked Resources across Organizations

Computers Networks Data Sources Scientific InstrumentsStorage Systems

Local Resource Managers

Operating Systems Queuing Systems Internet ProtocolsLibraries & App Kernels

Distributed Resources Coupling Services

Information QoSProcess

Development Environments and Tools

Languages/Compilers Libraries Debuggers Web tools

Resource Management, Selection, and Aggregation (BROKERS)

Applications and Portals

Prob. Solving Env.Scientific…CollaborationEngineering Web enabled Apps

Trading

…

…

…

…

FABRIC

APPLICATIONS

SECURITY LAYER

Security Data

CORE MIDDLEWARE

USER LEVEL MIDDLEWARE

Monitors

85

GridFabric

GridApps.

GridMiddleware

GridTools

Networked Resources across Organisations

Computers Clusters Data Sources Scientific InstrumentsStorage Systems

Local Resource Managers

Operating Systems Queuing Systems TCP/IP & UDP

…

Libraries & App Kernels …

Distributed Resources Coupling Services

Security Information … QoSProcess

Development Environments and Tools

Languages Libraries Debuggers … Web toolsResource BrokersMonitoring

Applications and Portals

Prob. Solving Env.Scientific …CollaborationEngineering Web enabled Apps

Resource Trading

Grid Components

Market Info

86

Economy Grid = Globus + GRACE

Applications

GRAM

Globus Security Interface (GSI)

Local Services

LSF

Condor GRD QBank

PBS

TCP

SolarisIrixLinux

UDP

High-level Services and Tools

Cactus MPI-G

Nimrod-G Broker

CC++

GASS GTSGARA

GridFabric

GridApps.

GridMiddleware

GridTools

GBankGMD

eCash

JVM

DUROC

Core Services

Science

Engineering Commerce Portals ActiveSheet……

……

MDS

Higher Level Resource Aggregators

Nimrod Parametric Language

87

Virtual Drug DesignA Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid

“Screen 2K molecules in 30min. for $10”

Grid Market Directory

ResourceBroker

Grid Info. Service

GTS

GTS

GTS

GTS

“Give me list PDBs sourcesOf type aldrich_300?”

“serv

ice co

st?”

(GTS - Grid Trade Server)

PDB2

“get mol.10 from pdb1 & screen it.”

Data Replica Catalogue

“service providers?”

GTS

PDB1

“mol.10 please?”

“mol.5 please?”

(RB maps suitable Grid nodes and Protein DataBank)

88

P-study Applications -- Characteristics

Code (Single Program: sequential or threaded)

High Resource Requirements Long-running Instances Numerous Instances (Multiple Data) High Computation-to-Communication

Ratio Embarrassingly/Pleasantly Parallel

89

Many Grid Projects & Initiatives

Australia Nimrod-G GridSim Virtual Lab Active Sheets DISCWorld ..new coming up

Europe UNICORE MOL UK eScience Poland MC Broker EU Data Grid EuroGrid MetaMPI Dutch DAS XW, JaWS and many more...

Japan Ninf DataFarm and many more...

USA Globus Legion OGSA Javelin AppLeS NASA IPG Condor-G Jxta NetSolve AccessGrid and many more...

Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, Parabon,….

Public Forums Global Grid Forum P2P Working Group IEEE TFCC Grid & CCGrid conferences

http://www.gridcomputing.com

90

Using Pure Globus/Legion commands

Do all yourself! (manually)

Total Cost:$???

91

Build Distributed Application & Scheduler

Build App case by case basisComplicated Construction

E.g., AppLeS/MPI based Total Cost:$???

92

Experiment-3 Setup

Workload: 200 jobs, each need 10 minute of CPU time

Deadline: 4 hrs. and budget: 250,000 G$ Strategies: 1. Minimise cost 2. Minimise time Execution:

Optimise Cost: 141,869 (G$) (finished in 150min./2.5hrs)

Optimise Time: 199,968 (G$) (finished in 250min.) In this experiment: Time-optimised scheduling run

costs double that of Cost-optimised. Users can now trade-off between Time Vs. Cost.

93

Number of Jobs Executed Organization &

Location

Vendor, Resource Type, # CPU, OS,

hostname

Grid Services,

Fabric, and Role

Price (G$ per

CPU sec.)

TimeOpt

CostOpt

Monash University, Melbourne, Australia

Sun: Ultra-1, 1 node, bezek.dstc.monash.edu.au

Globus, Nimrod-G, CDB Server, Fork (Master node)

-- -- --

VPAC, Melbourne, Australia

Compaq: Alpha, 4 CPU, OSF1, grendel.vpac.org

Globus, GTS, Fork (Worker node)

1 7 59

AIST, Tokyo, Japan Sun: Ultra-4, 4 nodes, Solaris, hpc420.hpcc.jp

Globus, GTS, Fork (Worker node)

2 14 2

AIST, Tokyo, Japan Sun: Ultra-4, 4 nodes, Solaris, hpc420-1.hpcc.jp

Globus, GTS, Fork (Worker node)

1 7 3

AIST, Tokyo, Japan Sun: Ultra-2, 2 nodes, Solaris, hpc420-2.hpcc.jp

Globus, GTS, Fork (Worker node)

1 8 50

University of Lecce, Italy

Compaq: Alpa cluster, OSF1, sierra0.unile.it

Globus, GTS, RMS (Worker node)

2 0 0

Institute of the Italian National Research Council, Pisa, Italy

Unknown: Dual CPU PC, Linux, barbera.cnuce.cnr.it

Globus, GTS, Fork (Worker node)

1 9 1

Institute of the Italian National Research Council, Pisa, Italy

Unknown: Dual CPU PC, Linux, novello.cnuce.cnr.it

Globus, GTS, Fork (Worker node)

1 0 0

Konrad-Zuse-Zentrum Berlin, Berlin, Germany

SGI: Onyx2K, IRIX, 6, onyx1.zib.de

Globus, GTS, Fork (Worker node)

2 38 5

Konrad-Zuse-Zentrum Berlin, Berlin, Germany

SGI: Onyx2K, IRIX, 16 onyx3.zib.de

Globus, GTS, Fork (Worker node)

3 32 7

Charles University, Prague, Czech Republic

SGI: Onyx2K, IRIX, mat.ruk.cuni.cz

Globus, GTS, Fork (Worker node)

2 20 11

University of Portsmouth, UK

Unknown: Dual CPU PC, Linux, marge.csm.port.ac.uk

Globus, GTS, Fork (Worker node)

1 1 25

University of Manchester, UK

SGI: Onyx3K, 512 node, IRIX, green.cfs.ac.uk

Globus, GTS, NQS, (Worker node)

2 15 12

Argonne National Lab, Chicago, USA

SGI: IRIX lemon.mcs.anl.gov

Globus, GTS, Fork (Worker node)

2 0 0

Argonne National Lab, Chicago, USA

Sun: Ultra –8, Solaris, 8, pitcairn.mcs.anl.gov

Globus, GTS, Fork (Worker node)

1 49 25

Total Experiment Cost (G$) 199968 141869

Time to Finish Expt. (Min.) 150 258

94

Jobs Completed for DBC Time Optimization

0

10

20

30

40

50

60

0.0

0

5.4

3

11

.75

17

.85

24

.06

29

.62

35

.00

40

.87

47

.12

53

.37

59

.75

65

.08

70

.48

76

.71

83

.87

90

.42

97

.25

10

3.7

9

11

0.4

1

11

6.4

1

12

2.5

9

12

8.6

9

13

4.4

4

14

0.2

2

14

5.6

7

Time (in min.)

No

. of

Jo

bs

Fin

ish

ed

grendel.vpac.org

hpc420.hpcc.jp

hpc420-1.hpcc.jp

hpc420-2.hpcc.jp

sierra0.unile.it

barbera.cnuce.cnr.it

novello.cnuce.cnr.it

onyx1.zib.de

onyx3.zib.de

mat.ruk.cuni.cz

marge.csm.port.ac.uk

green.cfs.ac.uk

lemon.mcs.anl.gov

pitcairn.mcs.anl.gov

95

Jobs Completed for DBC Cost Optimization

0

5000

10000

15000

20000

25000

30000

35000

400000

.00

15

.05

31

.40

49

.05

66

.22

82

.70

99

.38

11

5.8

2

13

3.0

9

14

9.7

3

16

6.5

1

18

2.4

6

19

9.8

7

21

5.8

4

23

0.9

5

24

6.0

8Time (in min.)

Bu

dg

et

Sp

en

t

grendel.vpac.org

hpc420.hpcc.jp

hpc420-1.hpcc.jp

hpc420-2.hpcc.jp

sierra0.unile.it

barbera.cnuce.cnr.it

novello.cnuce.cnr.it

onyx1.zib.de

onyx3.zib.de

mat.ruk.cuni.cz

marge.csm.port.ac.uk

green.cfs.ac.uk

lemon.mcs.anl.gov

pitcairn.mcs.anl.gov

96

Active Sheet:Microsoft Excel Spreadsheet Processing on Grid

NimrodNimrodProxyProxy

Nimrod-GNimrod-G

World-Wide Grid