Capetown, South Africa– 04/10/2013

Riccardo Rotondo – Consortium GARR

(riccardo.rotondo@garr.it)

Science Gateways: one portal, many e-

Infrastructures andrelated services

Riccardo Rotondo 2

Outline

• Did HPC make the world flat?• Science Gateways

– Overview– CTSG Framework

• Services• Use case

• Training activities– Material available– Join us: Marketplace

October 8th, 2013

Riccardo Rotondo 3

Is the world flat?

October 8th, 2013

Riccardo Rotondo 4

The world is flat

October 8th, 2013

In 2006, Thomas Friedman's book The World is Flat portrayed a global population that was more borderless and interconnected than ever before.

Bibliographic informationISBN 1429923075, 9781429923071

5

Grid & Cloud Computing

Riccardo RotondoOctober 8th, 2013

Riccardo Rotondo 6

Users in the net

October 8th, 2013

Riccardo Rotondo 7

Some barriers limit grid

adoption(*)

October 8th, 2013

The eResearch2020 reporthttp://www.eresearch2020.eu/

Riccardo Rotondo 8

Did we learn from Grid ?

October 8th, 2013

Riccardo Rotondo 9

Reusability

October 8th, 2013

Riccardo Rotondo 10

Easy of use and access

October 8th, 2013

Riccardo Rotondo 11

Science Gateways

October 8th, 2013

“A Science Gateway is a community-developed set of tools, applications, and data that is integrated via a portal or a suite of applications, usually in a graphical user interface, that is further customized to meet the needs of a specific community.”

Teragrid/XSEDE

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013

Riccardo Rotondo 12

Science Gateway’s Requirements Authentication and Autorisation

SAML, LDAP

Application e-Infrastructure indipendent jSAGA, SAGA, OCCI

Standard Java Technology JSR 168/286

Web Technology Web CMS

Wiki, Blog, Messages Board, Vconf, Adobe Connect

Portal Framework Liferay portlet container Liferay Service Builder

October 8th, 2013

Riccardo Rotondo 13

Catania Science Gateway Framework architecture

October 8th, 2013

.......

Catania S

cience G

ateway

App. 1 App. 2 MyCloud

Embedded Services Administrator(s)ScientistsCloud tenants

Users belonging to Identity

Federations

Grid/Cloud Engine

(based on SAGA)

CLEVER Orchestrator

(based on OCCI)

Cloud #2 Cloud #n

Cloud #1

Single logical domain

HPCClusters

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013

Riccardo Rotondo 14

Catania Science Gateway Framerwork

• Authentication & Authorisation• Job Management• e-Infrastructure Service & Data

Service• Cloud

October 8th, 2013

Riccardo Rotondo 15

AuthN/AuthZ Schema

e-Infrastructures

Federation

1. Try to

login

3. Identity attributes

2. Forwarded to the IdP

4. Check

authoris

ations

Sync user r

oles

Retrieve e-Infrastructure credentials Science Gateway

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013

Riccardo Rotondo

Federated User

16October 8th, 2013

Science Gateway

Riccardo Rotondo

Social User

17October 8th, 2013

Science Gateway

Riccardo Rotondo 18

Enhancements of user management

October 8th, 2013

• Liferay supports several system to store users data, both local and remote

• Supporting the largest number of users in the easiest way

• A modular way to distinguish between different services and privileges is need

• Science Gateways stores users on an LDAP server• In the act of registration user data must be written

on the LDAP connected to Liferay• A portlet has been developed to perform this actions

Riccardo Rotondo 19

Authentication request on Africa GRID SGW

October 8th, 2013

Riccardo Rotondo 20

Mobile Authentication

Native apps

Web views

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013

Riccardo Rotondo 21

Some of the IdPs supported

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013

Riccardo Rotondo

Users currently registered(as of the end of 2012)

22

Users from 219 Organisations in 47 Countries

12 Science Gateways in production; others in preparation

October 8th, 2013

Riccardo Rotondo 23

Applications accessing grid services

October 8th, 2013

• 12 applications developed among 5 different countries and 3 continents (Europe, Latin America and Asia);

• 4 scientific domains:– Life Science;– Mathematic & Computer Science;– High Energy Physics;– Cultural Heritage.

Riccardo Rotondo 24

Integrated Services

October 8th, 2013

GRIDCLOUD

JSR 168/286

JSR 168/286

JSR 168/286

JSR 168/286

Catania Science Gateway Framework

Local Cluster

Riccardo Rotondo 25

CSGF components

October 8th, 2013

Grid Engine e-Infrastructure & Data Service

Cloud Services

JSAGAAdaptors

Riccardo Rotondo 26

Catania Science Gateway Framerwork

• Authentication & Authorisation• Job Management• e-Infrastructure Service & Data

Service• Cloud

October 8th, 2013

Riccardo Rotondo 27

A Simple API for Grid Applications (SAGA)

October 8th, 2013

SAGA is an API that provides the basic functionality required to build distributed applications, tools and frameworks;

It is independent of the details of the underlying infrastructure (e.g., the middleware);

SAGA is an OGF specification: http://www.gridforum.org/documents/GFD.90.pdf.

Riccardo Rotondo 28

A Simple API for Grid Applications (SAGA)

October 8th, 2013

SAGA is composed by:

SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);

SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);

SAGA defines a standard We then need animplementation!

Riccardo Rotondo 29

A Simple API for Grid Applications (SAGA)

October 8th, 2013

Several Implementations are available: A C++ and a Java implementation developed at

the Louisiana State University / CCT and Vrije Universiteit Amsterdam (http://saga.cct.lsu.edu);

A Java implementation developed at CCIN2P3 (http://grid.in2p3.fr/jsaga/);

A Python implementation based on those above.

Riccardo Rotondo 30

Job EngineMiddleware Independent

October 8th, 2013

JSAGA supports gLite, Globus,ARC, UNICORE, etc.

Riccardo Rotondo 31

Job Engine - Architecture

October 8th, 2013

WT

Worker Threads for Job Submission

WT

Worker Threads forJob Check Status

USERSTRACKING

DBM

ON

ITO

RIN

G

MO

DU

LE

Res

ourc

es

Jobs Queue

WT WT

WT WT WT

WT

WT WT

JobsSubmission

JobsCheck status/

Get output

Riccardo Rotondo 32

Job Engine at work

October 8th, 2013

1. Sign in

eTokenServer

User Track-ing DB

3. Proxy

request*

4. Proxy

transfer*

5. e-Infra Interactions

5. Tracking

2. Grid Request

6. Getting Results

* or equivalent e-Infra auth

Riccardo Rotondo 33

Job Submission

October 8th, 2013

Riccardo Rotondo 34

Job Submission

October 8th, 2013

Riccardo Rotondo 35

My Workspace – Active Job List

October 8th, 2013

Riccardo Rotondo 36

Accessing services from Facebook

October 8th, 2013

Riccardo Rotondo 37

Catania Science Gateway Framerwork

• Authentication & Authorisation• Job Management• e-Infrastructure Service & Data

Service• Cloud

October 8th, 2013

Riccardo Rotondo 38

Motivations

Um… isn’t your computer on fire

?

It’s ok, my files are stored in a safer place.

Image source:October 8th, 2013

Riccardo Rotondo 39

e-Infrastructure & Data Service

• We designed the architecture to provide a Service accessible from your application.

• Modular attachable resources available to the SG thanks to the straight coupled service that allows read/write data from several e-Infrastructures.

October 8th, 2013

Riccardo Rotondo 40

e-Infrastructure Service

• Four different kind of e-Infrastructure identified: – Local– Remote– Grid – Cloud

• Possibility of defining different Policy and Quota according e-infrastructures/users availability/needs

• Default and favorite e-Infrastructures selectable

• Disaster recovery and (cross) file replica.

October 8th, 2013

Riccardo Rotondo 41

e-Infrastructure Service • If you want to copy a file on a file

system you plug-in an external storage to it.

October 8th, 2013

HFS

ext4

NFS

FAT

Riccardo Rotondo 42

e-Infrastructure Service • On a SG now you are able to

plug-in an e-Infrastructure.

Local Storage

Remote Storage

Grid Storage

Cloud StorageOctober 8th, 2013

Riccardo Rotondo 43

Data Services

• Business logic related to file management totally independent from the underneath technology

• All the common features associated to file provided: create, copy, delete, move and sharing.

• For each features, we implemented a different Storage Interface that translates the single commands according to the technology

• All data are stored in different tables in order to let users organise files on a virtual file system.

October 8th, 2013

Riccardo Rotondo 44

Data Service

Local Storage

Remote Storage

Grid Storage

Cloud Storage

Different implementations

of the same storage interface

October 8th, 2013

Riccardo Rotondo 45

Upload workflow

1. Sign in

Database

5. File Upload

4. e-Infra found

7. Update

6. Upload on e-Infras

2. Upload request

? 3. e-Infra Discovery

Riccardo Rotondo 46

Data Management on variouse-Infrastructures

October 8th, 2013

Riccardo Rotondo 47

Catania Science Gateway Framework

• Authentication & Authorisation• Job Management• e-Infrastructure Service & Data

Service• Cloud

October 8th, 2013

Riccardo Rotondo 48

My Cloud (cloud interoperability based on OCCI Standard)

October 8th, 2013

For more information follow Roberto Barbera presentation

Riccardo Rotondo 49

Science Gateways deployed

12 SGs in production and others in development

Very easy and intuitive access procedure

User-driven developmentSurveys to propose applications are available in Italian and other languages

VRCs supported either by region or discipline

Africa Grid

agINFRA

CHAIN-REDS

COGITO-MED

DCH-RP e-Culture

DECIDE

EarthServer

EUMEDGRID

GARR

GISELA

IGI

KLIOS

VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013October 8th, 2013

Riccardo Rotondo 50October 8th, 2013

Training activities

Riccardo Rotondo 51

SG under construction

October 8th, 2013

Riccardo Rotondo 52

Web-course summary information

• Home page:– http://agenda.ct.infn.it/event/catania-sg-

course• Timetable (direct URL):

– http://agenda.ct.infn.it/conferenceOtherViews.py?view=standard&confId=942

• All presentations have been uploaded

• Links are provided both to watch and download recordings of all the webinars

October 8th, 2013

Riccardo Rotondo 53

Training material

• Training material page (short URL):– http://goo.gl/im7YG

• Available sections– SG plugin as portlets accessing e-

Infrastructure services;– Multi federation discovery service– Identity Provider installation an

configuration

October 8th, 2013

Riccardo Rotondo 54

Our code on sourceforge

• The Catania Science Gateway Framework on Sourceforge:– http://sourceforge.net/projects/ctsciencegtwys/

• Wiki• Tickets• Discussion• Code is open source, feel free to join and

contribute• For more info contact to sg-licence@ct.infn.it

October 8th, 2013

Riccardo Rotondo 55

A unique entry point

• The Catania Science Gateway Framework on the web:– http://www.catania-science-gateways.it

October 8th, 2013

Riccardo Rotondo 56

1. 2.

3.

4.

5.

Users/VRCs

Ex.: Africa Grid Science Gateway

SG Dev.

The Science Gateway Market Place

October 8th, 2013

Riccardo Rotondo 57

The list of available applications

October 8th, 2013

Riccardo Rotondo 58

To propose new applications

• http://www.catania-science-gateways.it/survey

October 8th, 2013

Riccardo Rotondo 59

To port new applications• Register as a Catania Science Gateway Developer:

– http://www.catania-science-gateways.it/port-app

• Sign In and book an application «to be ported»

• Once your booking is accepted, you have 30 days to create the needed portlet(s) for the application (the sooner the better)– After that time the access to our VPN will be denied

Once you are registered you can also port the applications YOU proposed

October 8th, 2013

Riccardo Rotondo 60

Conclusion

• Science Gateways permit easy federated access to e-Infrastructure services in a standards-oriented way

• We are developing a modular framework to increase the number of applications supported within our portals

• All functionalities are designed according users needs who give us inspiration and contributors … the next could be you!October 8th, 2013

Riccardo Rotondo 61

Questions ?

October 8th, 2013