An Overview of Nephele and Pegasus

Presented  By:  Somnath  Mazumdar  somnath.mazumdar@ucdconnect.ie  

h8ps://www.csi.ucd.ie/users/somnath-­‐mazumdar    

Nephele: Efficient Parallel Data Processing in the Cloud

By: Daniel Warneke and Odej Kao Technische Universität Berlin

Berlin, Germany

Published in: Proceeding MTAGS '09 Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers ACM New York, NY, USA ©2009

Nephele: What for ???

•  General trends: At the beginning of processing resource allocation is done (static), no scope to extend/remove resources (dynamic) ….

•  Paper (Claims): “…first data processing framework to include the possibility of dynamically allocating/ deallocating different compute resources ….”

Known Issues in Cloud •  Cloud resources are dynamic and

heterogeneous •  Provisioning of resources on demand •  Cloud challenge: opaqueness

Nephele Architectural Overview

Jobs @ Nephele

q I. Steps to create a job(DAG): 1.  Write own code for task. 2.  Assign tasks to a vertex. 3.  Define communication path for

Job. q II. Add annotation to job

description. q III. Transform Job Graph

àExecution Graph  

An Execution Graph

Efficient Execution Graph creation depends on user input / job annotation description

 

Pegasus: A framework for mapping complex scientific workflows onto

distributed systems

By: Ewa Deelman, Gurmeet Singh, Mei-Hui Su, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Karan Vahia, G. Bruce Berriman, John Good, Anastasia

Laity, Joseph C. Jacob and Daniel S. Katz

University of Southern California Information Sciences Institute, CA, USA Infrared Processing and Analysis Center, Jet Propulsion Laboratory, CalTech USA

Published in: Journal Scientific Programming archive Volume 13 Issue 3, July 2005

IOS Press Amsterdam, The Netherlands

Before Starting….  ü  Workflow can capture the behavior of

application.(abstract & Concrete) ü  Workflows are abstract in the application-

level (describes application components and their dependencies)

ü  Simplifies application development process(+) ü  Concrete workflow describe resources that

would be used in execution of specific tasks.

Pegasus: What for..??

ü Describes: process of mapping from abstract to executable workflow can be automated.

ü Assumption: 1. Application is already represented in an abstract workflow form 2. Data does not specify particular

resources to be used. ü Scheduling horizon encompasses tasks that can be

sent to execution system. ü Mapping horizon indicates how far into the

workflow to map the tasks.

Horizons & Costs

Mapping depends on specific resources to execute specific tasks as well as data

locality.

Logical Overview of Pegasus

Failure Recovery In Pegasus

Thank  You