Pegasus-a framework for planning for execution in grids Karan Vahi vahi@isi.edu USC Information...

Pegasus-a framework for planning for execution in grids

Karan Vahivahi@isi.edu

USC Information Sciences Institute

May 5th , 2004

2May 5th, 2004Karan Vahi, ISI vahi@isi.edu

People Involved

USC/ISI

Advanced Systems: Ewa Deelman, Carl Kesselmann, Gaurang Mehta, Mei-Hui Su, Gurmeet Singh, Karan Vahi.

Outline

Introduction To Planning DAX Pegasus Portal Demonstration

Planning in Grids

One has various alternatives out on the grid in terms of data and compute resources.

Planning – Select the best available resources and data sets,

and schedule them on to the grid to get the best possible execution time.

– Plan for the data movements between the sites

Recipe For Planning

Understand the request– Figure out what data product the request refers to, and

how to generate it from scratch.

Locations of data products– Final data product – Intermediate data products which can be used to generate

the final data product.

Location of Job executables

State of the Grid– Available processors, physical memory available, job

queue lengths etc.

Constituents of Planning

Domain Knowledge

ResourceInformation

Location Information

Planner

Plan submitted the grid

Terms (1)

Abstract Workflow (DAX)– Expressed in terms of logical entities– Specifies all logical files required to generate the

desired data product from scratch– Dependencies between the jobs– Analogous to build style dag

Concrete Workflow– Expressed in terms of physical entities– Specifies the location of the data and executables– Analogous to a make style dag

Outline

Introduction to Planning DAX Pegasus Portal Demonstration

The format for specifying the abstract workflow, that identifies the recipe for creating the final data product at a logical level.

In case of montage, the IPAC webservice ends up creating the dax for the user request.

Developed at University Of Chicago

DAX Example

<?xml version="1.0" encoding="UTF-8"?>   <adag xmlns="http://www.griphyn.org/chimera/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://www.griphyn.org/chimera/DAX http://www.griphyn.org/chimera/dax-1.6.xsd" count="1" index="0" name="black-diamond">

<filename file="f.a" link="input"/> <filename file="f.b" link="inout"/> <filename file="f.c" link="output"/>  <job id="ID000001" namespace="montage" name="preprocess" version="1.0" level = "2"> <argument>-a top -T60 -i <filename file="f.a"/> -o <filename file="f.b"/> </argument> <uses file="f.a" link="input" dontRegister="false" dontTransfer="false"/> <uses file="f.b" link="output" dontRegister="true" dontTransfer="true" temporaryHint="true"/> </job> <job id="ID000002" namespace="montage" name="analyze" version="1.0" level="1" > <argument>-a bottom -T60 -i <filename file="f.b"/> -o <filename file="f.c"/></argument> <uses file="f.b" link="input" dontRegister="false" dontTransfer="false"/> <uses file="f.c" link="output" dontRegister="false" dontTransfer="false"/> </job>

<child ref="ID000002"> <parent ref="ID000001"/> </child> </adag>

Outline

Introduction to Planning DAX Pegasus Demonstration Portal

Pegasus

A configurable system to map and execute complex workflows on the grids.– DAX Driven Configuration

– Metadata Driven Configuration

Can do full ahead planning or deferred planning to map the workflows.

Full Ahead Planning

At the time of submission of the workflow, you decide where you want to schedule the jobs in the workflow.

Allows you to perform certain optimizations by looking ahead for bottleneck jobs and then scheduling around them.

However, for large workflows the decision you

make at submission time may no longer be valid or optimum at the point the job is actually run.

Deferred Planning

Delay the decision of mapping the job to the site as late as possible.

Involves partitioning of the original dax into smaller daxes each of which refers to a partition on which Pegasus is run.

Construct a Mega DAG that ends up running pegasus automatically on the partition daxes, as each partition is ready to run.

High Level Block Diagram

GridGridGrid

workflow executor(DAGman)Execution

WorkflowPlanning

Globus ReplicaLocation Service

Globus Monitoringand Discovery

Service

Information andModels

Application Models

detector

Raw data

Replica LocationAvailableReources

Monito

ring in

Abstract Worfklow

Request Manager

Replica andResourceSelectorSubmission and

Monitoring System

WorkflowReduction

DataPublication

IPAC/JPL WebService

DataManagement

Replica Discovery

Pegasus needs to know where the input files for the workflow reside.

In Montage case, it should know where the fits files that are required for the mProject jobs reside.

Hence Pegasus needs to discover the files that are required for executing a particular abstract workflow.

LRCA LRCCLRCB

Each LRC sends periodic updates to the RLI

Each LRC is responsible for one pool

Pegasus

1) Pegasus queries RLI with the LFN

2) RLI returns the list of LRC’s that contain the desired mappings.

3) Pegasus queries each LRC in the list to get the PFN’s.

Figure (1) RLS Configuration for Pegasus

Interfacing to RLS done by Karan Vahi, Shishir

Alternate Replica Mechanisms

Replica Catalog– Pegasus supports the LDAP based Replica

Catalog

User defined mechanisms– Pegasus provides the flexibility for the user

to specify his own replica mechanism instead of RLS or Replica Catalog

– The user just has to implement the concerned interface

Design and Implementation done by Karan Vahi

Transformation Catalog

Pegasus needs to access a catalog to determine the pools where it can run a particular piece of code.

If a site does not have the executable, one should be able to ship the executable to the remote site.

Generic TC API for users to implement their own transformation catalog.

Current Implementations– File Based– Database Based

File based Transformation Catalog

Consists of a simple text file.– Contains Mappings of Logical Transformations to Physical

Transformations.

Format of the tc.data file#poolname logical tr physical tr envisi preprocess /usr/vds/bin/preprocess VDS_HOME=/usr/vds/;

All the physical transformations are absolute path names.

Environment string contains all the environment variables required in order for the transformation to run on the execution pool.

DB based Transformation Catalog

Presently ported on MySQL. Postgres to be tested.

Adds support for transformations, compiled for different architectures, OS, OS version and glibc combination, that would enable us to transfer transformation to remote sites if the executable does not reside there.

Supports multiple profile namespaces. At present using only the env namespace.

Supports multiple physical transformations for the same logical transformation,pool,type tuple.

Pool Configuration (1)

Pool Config is an XML file which contains information about various pools on which DAGs may execute.

Some of the information contained in the Pool Config file is– Specifies the various job-managers which are available on

the pool for the different types of condor universes.– Specifies the GridFtp storage servers associated with each

pool.– Specifies the Local Replica Catalogs where data residing in

the pool has to be cataloged.– Contains profiles like environment hints which are common

site wide.– Contains the working and storage directories to be used on

the pool.

Two Ways to construct the Pool Config File.– Monitoring and Discovery Service

– Local Pool Config File (Text Based)

Client tool to generate Pool Config File– The tool genpoolconfig is used to query the

MDS and/or the local pool config file/s to generate the XML Pool Config file.

This file is read by the information provider and published into MDS.

Formatgvds.pool.id : <POOL ID>gvds.pool.lrc : <LRC URL>gvds.pool.gridftp : <GSIFTP URL>@<GLOBUS VERSION>gvds.pool.gridftp : gsiftp://sukhna.isi.edu/nfs/asd2/gmehta@2.4.0gvds.pool.universe : <UNIVERSE>@<JOBMANAGER URL>@<

GLOBUS VERSION>gvds.pool.universe : transfer@columbus.isi.edu/jobmanager-

fork@2.2.4gvds.pool.gridlaunch : <Path to Kickstart executable>gvds.pool.workdir : <Path to Working Dir>gvds.pool.profile : <namespace>@<key>@<value>gvds.pool.profile : env@GLOBUS_LOCATION@/smarty/gt2.2.4gvds.pool.profile : vds@VDS_HOME@/nfs/asd2/gmehta/vds

DAX Driven Configuration(1)

Pegasus uses IPAC/JPL webservice as an abstract workflow generator

Pegasus takes in this abstract workflow and creates a concrete workflow by consulting the various grid services described before

(5) Full Abstract Dag

DAX Driven Configuration(2)IPAC/JPL Service

Transformation Catalog

Current State Generator

Request Manager

Concrete Planner

Abstract Dag Reduction

Abstract and Concrete Planner

VDL GeneratorSubmit File Generator

DAGMan Submission& Monitoring

Condor-G/ DAGMan

(1) Abstract Workflow(DAG)

(2) Abstract Dag

(3) Logical File Names(LFN’s)

(4) Physical File Names(PFN’s)

(6) Reduced Abstract DAG

(7) LogicalTransformations

(8) PhysicalTransformations and

Execution EnvironmentInformation

(9) Concrete Dag

(10) Concrete Dag

(11) DAGMan files

(13) DAG (14) Log files

(15) Monitoring

(16) Results

(12) DAGMan files

DAG Reduction

Abstract Dag Reduction– Pegasus queries the RLS with the LFN’s

referred to in the Abstract Workflow

– If data products are found to be already materialized, Pegasus reuses them and thus reduces the complexity of CW

KEYThe original node

Pull transfer node

Registration node

Push transfer node

Job g Job h

Job aJob c

Abstract Dag Reduction

Pegasus Queries the RLS and finds the

data products of jobs d,e,f already

materialized. Hence deletes those jobs

On applying the reduction algorithm additional jobs a,b,c

are deleted

Implemented by Karan Vahi

Pegasus adds replica nodes for each job that materializes data (g, h, i ).

These three nodes are for transferring the output files of the leaf job (f) to the output pool, since job f has been deleted by the Reduction Algorithm.

KEYThe original node

Pull transfer node

Registration node

Push transfer node

Node deleted by Reduction algo

Inter-pool transfer node

Job g Job h

Job aJob c

Concrete Planner (1)

Pegasus schedules job g,h on pool X and job i on pool Y. Hence adding an interpool transfer node

Pegasus adds transfer nodes for transferring the input files for the root nodes of the decomposed dag (job g)

Transient Files

Selective Transfer of output files– Data Sets generated by intermediate nodes in DAG

are huge– However, user maybe interested only in outputs of

selected jobs– Transfer of all the files could severely overload the

jobmanagers on the compute sites Need For Selective Transfer of Files

– For each file at the virtual data, user can specify whether it is transient or not.

– Pegasus bases it’s decision on whether to transfer the file or not on this.

Outline

Portal Architecture

Portal Demonstration

Outline

Demonstration

Run a small black diamond dag using both full ahead planning and deferred planning on the isi condor pool.

Show the various configuration files (tc.data and pool.config) and how to generate them (pool.config).

Generate the condor submit files.

Submit the condor dag to condor dagman.

Software Required!!

Submit Host– Condor DAGMAN (to submit the workflows on the grid).– Java 1.4 (to run Pegasus)– Globus 2.4 or higher– Globus RLS (the registration jobs run on the local host).– Xerces, ant , cog etc that come with the VDS distribution

Compute Sites (Machines in the pool)– Globus 2.4 or higher (gridftp server, g-u-c, MDS)– On one machine per pool, an lrc should be running. – Condor daemon running.– Various jobmanagers correctly configured.

TC File

Walk through the editing of TC file.

A command line client is also in the works that allows you to update, add and modify the entries in your transformation catalog regardless of the underlying implementation.

GenPoolConfig (Demo)

genpoolconfig is the client to generate the pool config file required by Pegasus.

It queries the MDS and/or a local pool config file (text based) and generates a XML file.

Am going to generate the pool config file from the text based configuration.

Usage : – genpoolconfig –Dvds.giis.host <MDS GIIS hostname> -

Dvds.giis.dn <MDS GIIS DN> --poolconfig <comma separated local pool config files> --output <pool config output>

gencdag

The Concrete planner takes the DAX produced by Chimera and converts into a set of condor dag and submit files.

Usage : gencdag –dax|--pdax <file> --p <list of execution pools> [--dir <dir for o/p files>] [--o <outputpool>] [--force]

You can specify more then one execution pools. Execution will take place on the pools on which the executable exists. If the executable exists on more then one pool then the pool on which the executable will run is selected randomly.

Output pool is the pool where you want all the output products to be transferred to. If not specified the materialized data stays on the execution pool

Mei’s Exploits

Mei has been running the montage code for the past one year, including some huge 6 and 10 degree dags (for the m16 cluster).

The 6 degree runs had about 13,000 compute jobs and the 10 degree run had about 40,000 compute jobs!!!

The final mosaic files can be downloaded from http://www.isi.edu/~griphyn/out_M16_10.fits

http://www.isi.edu/~griphyn/out_M16_6.fits

Questions?

Pegasus-a framework for planning for execution in grids Karan Vahi vahi@isi.edu USC Information...

Documents

Transcript of Pegasus-a framework for planning for execution in grids Karan Vahi vahi@isi.edu USC Information...

Pegasus- Advanced Features - SciTech · Pegasus- Advanced Features and Optimizations Griphyn-LIGO Meeting, Caltech July 20th, 2006 Karan Vahi, Ewa Deelman, Gaurang Mehta, Center for

Karan - Project Mangement

03 Bas Karan

Karan Project (Ins)

Karan Popli

Karan Automobile Honda

Jessica Karan Portfolio

karan singh

Highland Hardware Since 1978 - VaHi

Karan Project Final

karan nikon

Karan Fastfood (1)

Coofee Karan

Karan Goyal

Managing large-scale workflows with Pegasus Karan Vahi ( vahi@isi.edu)vahi@isi.edu Collaborative Computing Group USC Information Sciences Institute Funded.

Karan piyush

CoCoMo seminar Karan

The Virtual Data System - Information Sciences InstituteThe USC Information Sciences Institute: Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Mei-Hui Su, Karan Vahi, Jens

Karan Project

KARAN Inflation)