GRID computing Frascati 14/10/2004 [email protected] .

GRID computing Frascati 14/10/2004 [email protected] www.telegrid.enea.it

www.enea.it


26 April 2005 Johon Cabot University

GRID Computing: esperienze realizzative ed elementi architetturali.

Outline• ENEA-GRID Infrastructure and Resources• Case studies: numerical simulations and support to

experimental facilities• Connection with other GRID projects• GRID paradigm & ENEA-GRID

GRID COMPUTINGReal Case GRID At ENEA

Ing. S. Migliori ENEA INFO [Servizio Centralizzato Informatica e Reti], Roma


The Grid: The Web on Steroidshttp://

http://

Web: Uniform access to HTML documents

Grid: Flexible, high-perf access to all significant resources

Ian Foster

On-demand creation of powerful virtual computing systems

Sensor nets

Data archives

Computers

Softwarecatalogs

Colleagues


TELEMICROSCOPY & GRID - BASED COMPUTINGTELEMICROSCOPY & GRID - BASED COMPUTINGREMOTE ACCESS FOR DATA ACQUISITION AND ANALYSISREMOTE ACCESS FOR DATA ACQUISITION AND ANALYSIS

NETWORKNETWORK

DATA ACQUISITIONDATA ACQUISITION DATA ANALYSISDATA ANALYSISADVANCEDADVANCEDCOMPUTERCOMPUTERGRAPHICSGRAPHICS

QuickTime™ and a YUV420 codec decompressor are needed to see this picture.

Cell Centered Data Base Cell Centered Data Base ““CCDB”CCDB”

ByByMark EllismanMark Ellisman

IMAGINGIMAGINGINSTRUMENTSINSTRUMENTS COMPUTATIONALCOMPUTATIONAL

RESOURCESRESOURCES

MULTI-MULTI-SCALESCALEDATA-DATA-BASESBASES


Architecture --- > Function

Application

Fabric

Connectivity

Resource

Collective

User

Sensor netsData archives

Computers

Softwarecatalogs

Colleagues

?

Le Funzioni / Esigenze comandano sulla tecnologia


Network Exponentials

• Network vs. computer performance– Computer speed doubles every 18

months– Network speed doubles every 9

months– Difference = order of magnitude

per 5 years

• 1986 to 2000– Computers: x 500– Networks: x 340,000

• 2001 to 2010– Computers: x 60– Networks: x 4000

Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.

Ian Foster


Performance Extrapolation

N=1

N=500

Sum

1 GFlop/s

1 TFlop/s

1 PFlop/s

100 MFlop/s

100 GFlop/s

100 TFlop/s

10 GFlop/s

10 TFlop/s

10 PFlop/s

PFlop/scomputer

TFlop/sTo enter the list

Jac. Dongarra SC2002


Selected Major Grid ProjectsName URL & Sponsors Focus

Access Grid www.mcs.anl.gov/FL/accessgrid; DOE, NSF

Create & deploy group collaboration systems using commodity technologies

BlueGrid IBM Grid testbed linking IBM laboratories

DISCOM www.cs.sandia.gov/discomDOE Defense Programs

Create operational Grid providing access to resources at three U.S. DOE weapons laboratories

DOE Science Grid

sciencegrid.org

DOE Office of Science

Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities

Earth System Grid (ESG)

earthsystemgrid.orgDOE Office of Science

Delivery and analysis of large climate model datasets for the climate research community

European Union (EU) DataGrid

eu-datagrid.org

European Union

Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics

g

g

g

g

g

g

New

New

Ian Foster


Selected Major Grid ProjectsName URL/

SponsorFocus

TeraGrid teragrid.org

NSF

U.S. science infrastructure linking four major resource sites at 40 Gb/s

UK Grid Support Center

grid-support.ac.uk

U.K. eScience

Support center for Grid projects within the U.K.

Unicore BMBFT Technologies for remote access to supercomputers

g

g

New

New

Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS

See also www.gridforum.org

Ian Foster


ENEAItalian Agency for New Tecnologies, Energy and Environment. 12 research centers in Italy A Central Computer and Network Service (INFO)6 Computer CentresCasaccia, Frascati, Bologna,Trisaia, Portici, Brindisi

Multiplatform resources for serial & parallel computation and graphical post-processing.

Others computer resources in ENEA:departments & individuals

Centrodi Brindisi


ENEA computational resources are distributed over WAN, connected by GARR, the Italian Academic & Research Network

ENEA Network connection

ENEA-GARR 9 PoP, 4-32 Mbps

BrindisiBolognaCasacciaFrascatiPorticiTrisaiaPalermoPisaRoma Sede


ENEA-GRID has been developed since 1998 and now it is an integrated infrastructure:

• offering a production quality, service oriented system

• covering most of the computational resources belonging to ENEA INFO Scientific Computation Service, connected in a Wide Area Network.

• providing a unified user environment and an homogenous access method for all ENEA researchers irrespective of their location

• implementing tools to facilitate the integration of department and individual resources and support to experimental facilities

ENEA GRID


Services for research activity in ENEA

• Multiplatform Parallel systems:

• AIX, Linux (Alpha,x86,AMD64), IRIX, [Unicos (Cray)]

• Graphical simulation & code result post-processing:

• SGI + 3D immersive facilities

• Software resources:

• Commercial Codes: Fluent, Gambit, Abacus, Catia, Ansys

• Research codes: mcpn/x, eranos, fluka,…

• Elaboration environments: IDL, Matlab, Mathematica, SAS

• Windows applications

ENEA GRID Services


ENEA GRID Computational resources

Portici, Brindisi & Trisaia resources increased in the framework of TELEGRID Project

Frascati(6), Portici(4),Trisaia(4), Brindisi(4)

5018Windows

Frascati(1), Trisaia(13)8021Mac OS X

Trisaia(4), Casaccia(2), Bologna(2)108Solaris

Frascati(8), Casaccia(4), Portici(1),Trisaia(8), Brindisi(1), Bologna(5)

4026IRIX

Casaccia10080Linux Alpha

Frascati(28), Casaccia(54), Portici(4),Trisaia(4), Brindisi(4)

20096Linux x86 32/64

Frascati(128), Bologna(8), Portici(18)1000154AIX

Gflops#cpu OS


CitrixMetafram

e

ICA

WEB(ICA)

power3.frascati.enea.it

boquad.bologna.enea.it

dafne.casaccia.enea.it

Kleos.portici.enea.it

GEANT

infocal.trisia.enea.it

……...brindisi.enea.it

Portici

Brindisi

CitrixNfuse

new


AFS Geographical cross platform & File System

AIX SGI SUN HP Win

Linux

Linux Alpha

MAC

LSF

User programs and commercial codes

LSF multi-cluster as integrator

XTERM

Graphical User Interface

Application Portal (Metaframe/Nfuse)

ICAWEB

Qu

ality

of

serv

ice

Mon

itori

ng

, A

uto

mati

c r

ecovery

Accau

nti

ng

Load Leveler RMS

ENEA GRID


Software Conceptual Structure

AFS

Load Leveler LSF

Graphic User Interface LSF multi-cluster as integrator

Telnet

User programs & commercial code

ICAWEB

Cluster 1° ( Bologna)

AFS

Load Leveler LSF


Telnet


ICAWEB

Cluster 3° (Frascati)

AFS

Load Leveler LSF


Telnet


ICAWEB

Cluster 2° (Nord Rome)

AFS

Load Leveler LSF


Telnet


ICAWEB

Cluster 4° (Trisaia )

LSF

Graphic User Interface

Telnet


ICAWEB

Cluster 5° ( Portici)

File System

LSF

Graphic User Interface

ICAWEB

Cluster 6° ( Brindisi)

NEW

NEW


Gestione generale di una specifica richiesta al sistema

InterfacciaGrafica

(RICHIESTA)LSF

Cluster Configuratio

n(STATICO)

Stato delle Risorse

(TEMPO REALE)

Politica delle code

(STATICO)•Accoda la richiesta

•Assegna la risorsa

•Lancia il comando sul calcolatore selezionato

DATI(SW/Prog.)

File Server

Client AFS

Cache locale

•Risolve la piattaforma•Verifica i diritti di accesso•Mantiene la coerenza dei dati

AFS

Data Base Server


ENEA GRID Graphics and 3D resources(1)

C.R. Frascati C.R. Bologna

C.R. Casaccia

SGI systems are coupled with 3D immersive facilities together with 3D modelling and rendering software as CAVELIB, DV-Mockup, AVS


ENEA GRID Graphics and 3D resources(2)

C.R. BrindisiC.R. Trisaia

C.R. Portici

SGI systems are coupled with 3D immersive facilities together with 3D modelling and rendering software as CAVELIB, DV-Mockup, AVS


ENEA GRID ARCHITECTURE

A choice of mature components for reliability and easiness of support and maintenance:

Distributed File System: AFS

Job and resources manager: LSF Multicluster

Unified GUI access: Java and Citrix Technologies

Quality monitoring system: Patrol

Licence Servers

Integration with department and individual resources

Distributed File System: AFS for software and data

Licence pool sharing

Possible integration with other institutions


GRID usage, examples and some case studies

650 registered users

2 TB stored software/data out of 4 TB available in enea.it AFS cell

Examples and cases studies● Running FLUENT on ENEA GRID

● Support to department experimental research:

• Biotechnology

• Electron microscopy

• Nuclear fusion research


CitrixMetafram

e

ICA

WEB(ICA)

power3.frascati.enea.it

boquad.bologna.enea.it

dafne.casaccia.enea.it

Kleos.portici.enea.it

GEANT

infocal.trisia.enea.it

……...brindisi.enea.it

Portici

Brindisi

CitrixNfuse

new


CITRIX / NFUSE

ENEA GRID makes use of Citrix Metaframe to publish an application providing a unified GUI interface all the available resources

Application components:●Java (GUI)●shell scripts●EnginFrame


FLUENT (CFD) code batch submission

LSF Options-o filename.%J - output file-w “done(idjob)” - start after idjob-u e-mail userid - output by E-mail-b begintime - job start time….

A FLUENT case can be run from a dedicated GUI for submission


Monitoring job status by xlsbatch (LSF)

xlsbatch informationJobidUsersJob statusQueuesSubmission hostRun hostSubmission timeSubmission command…..

Job status:


Monitoring host status by xlsmon (LSF)

Status of sp3-1 host using xlsmon utility


3D CFD Simulation on Aircraft Axial Turbin Stage with Wake & 3D CFD Simulation on Aircraft Axial Turbin Stage with Wake & Vortex Shedding Analysis on Turbine Blades [FLUENT]Vortex Shedding Analysis on Turbine Blades [FLUENT]

Pressure traces on stator trailing edge

165000

170000

175000

180000

185000

190000

195000

pres

sure

(Pa)

isolated stator stage stator

Temperature & wakes interactions

Large Eddy Simulation (LES) provides high

resonable results on wake analysis and turbine stage performance.

Secondary flow prevision and 3D Path Lines analysis

Max, min and time average Cp for the vane

-2.5

-2

-1.5

-1

-0.5

0

0.5

-0.01 0 0.01 0.02 0.03 0.04

x (m)

Cp

Temperature contours on Stator blades & Cp experimental data comparison

Mesh: 1 million cellsCPU Time: 6000 hours

Elapsed Time: 525 hours on

12 IBM SP3 platforms


Pressure losses

0

500000

1000000

1500000

2000000

2500000

3000000

0 0.2 0.4 0.6 0.8 1 1.2

velocity (m/s)

Pres

sure

jum

p (P

a)

Serie1

Caratteristica parabolica dello smorzatore con valvola aperta fino a 3 mm

Punto di progetto (apertura valvola)

CFD Analysis of a fluid dumper [FLUENTCFD Analysis of a fluid dumper [FLUENT]]

Vortex stagnationVortex stagnation

Secondary fluxSecondary flux

Primary FluxPrimary Flux

Mesh: 600,000 cells CPU Time: 960 hours

Elapsed Time: 130 hours on 8 Linux platforms


p-patm da 0 a 1.55 s.

0

10

20

30

40

50

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5time (s)

p-p

atm

Real P-Patm

linea-1

linea-2

linea-3

Blast Wave Analysis for VEGA Programm

DMA-Uniroma1 – AviogroupRome 2003

0.00E+00

5.00E+03

1.00E+04

1.50E+04

2.00E+04

2.50E+04

3.00E+04

3.50E+04

4.00E+04

0.00E+00 2.00E+01 4.00E+01 6.00E+01


La simulazione con Cover:il campo di Temperatura

Grid: 1,5 million cellsCPU Time: 5000hours

Elapsed Time: 672 hours on

6 SGI platforms

(onyx2ced )


Uno sguardo verso il futuro: la parametrizzazione via WebUno sguardo verso il futuro: la parametrizzazione via Web

4.0

100.0V

ProfiloNACA xxx

STEPSDefinire la geometria e le condizioni al contorno.Generare la griglia di calcolo necessariaEseguire il calcoloRisultati del Post-processing (su Web o home/directory)


FLUENT performance on different platforms in ENEA GRID

Cpu time normalized to the case: Sp4-1, 2 CPUs

GHz#CPU

IRIX

0.48Onyx.

1.88bw

34C..03

Linux

1.38Pace

1.316Ostro

1.132Sp4-1

.37516Sp3-1

AIX

Scaling with #CPU & platform type


DB_1

CPUSENEA GRID

WEBICASSH

DNA Sequence system

( ABI Prism 3700)Trisaia

DB_3

DB_2

Electronic Microscope (Brindisi)

300 Kev (sept. 2004)

ENEA GRID and experimental facilities

Controlled Nuclear Fusion:

FTUFrascati Tokamak

UpgradeVideo Acquisition


ENEA GRID: FTU video acquisition data


Conceptual structure for TEMProgetto UTS Materiali

Electronic Microscope (Brindisi)

300 KeV (sept. 2004)

RemoteLocal

Operation

TEM Room

Local Operation

Remote ControlledWeb-cam

Remote User

Local DataAFS Data Integration

> 4 TByte

Mtaframe

AFS

Special Driver for TEM

Local GRID Cluster


ENEA-Geographic SUBMISSION PROCEDURE

Input FilesLIV1

List of pathInput File

LIV1

IDL “GOME” Program

Output FileLIV2

.jpg Image

Input FileLIV1

List of pathInput File

LIV1

IDL Program“GOME”

Output FileLIV2

.jpg Image Input File

LIV1

List of pathInput FileLIV1

Case-1

Case_2

Case n

WEB

User afs/home

ENEA GRID/DATAGRID Test case: GOME data analysis


ENEA GRID/DATAGRID : GOME run cases in ENEA


ENEA GRID/DATAGRID: GOME results on Web


ENEA GRID

ANSALDO(GE)

ENEA(ISPRA)

ORNL(USA)

CRIOTEC(TO)

• DATA sharing• Applications in ENEA GRID• Sharing Working Sessions (Citrix)

ENEA IGNITOR/Pellet Collaboration Activity


Collaborazioni

DMA ( Prof. ENRICO Scubba) Fluidodinamica e sviluppo di metodologie

dell’utilizzo di grid … Istituto Matemetica Castelnuovo (In fase di perfezionamento)


ENEA ENEA

Main activities & Main activities & resources for EGEEfor EGEE

•Integration of EGEE-GRID to ENEA-GRID;Integration of EGEE-GRID to ENEA-GRID;

•100 CPUs at 20% of time100 CPUs at 20% of time

DataGrid at

ESA\ESRIN

Globus

Based g

ateway

GigaBit-

Link

EGEE GRID

Globus Based gateway

Glo

bu

s B

ased

gate

way

ENEA-GRID

S-PACIGRID

Globus Based gateway

GRID.IT WP13 Chem-Grid

(MAT)


Windows services Unix/Linux Services

CITRIX: ENEA GRID Web Access

http://www.afs.enea.it


ENEA GRID & GRID PARADIGM“GRID concept: coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations”

Foster & Kesselman, The Anatomy of the Grid, 2001

ENEA GRID infrastructure has been designed to provide a multisite resource sharing inside ENEA, both to optimize the utilization of Central Computer Service (ENEA INFO) resources and to integrate the other resources available inside the organization.

VO concept can not be fully developed inside an unique organization but similarities can be found with Project Groups sharing specific resources and ENEA GRID components can provide support to this (AFS groups, LSF resources).

There are cases where ENEA GRID infrastructure has been used in support to the collaboration with other institutions (+ other GRID projects)


ENEA GRID architecture answers to many of the challenges posed by the “GRID Problem” : “unique authentication, authorization, resource access and resource discovery”, Foster & Kesselman 2001

ENEA GRID & GRID PARADIGM

The choice of mature components (LSF & Citrix - proprietary, AFS - opensource project since 2000) reduce the generality of ENEA GRID as a GRID model but improves greatly the reliability and the easiness of update & management: production quality infrastructure.

Sensor nets

Colleagues Data archives

ComputersSoftwarecatalogs

by Ian Foster


ENEA GRID EXPERIENCE

Some consideration can be drawn from the analysis of the operation of ENEA GRID infrastructure in the last years.

User acceptance of GRID working mode requires a new cultural approach to the access to computational resources and depends strongly on the service quality:

● “Value” of the accessed resources

● Efficient networking and system faults monitoring

● Support of user customization for GRID environment

The administration of a distributed & integrated resource system is a technical, cultural and organization challenge in a diversified structure as ENEA.


ITALIAN NATIONAL AGENCYFOR NEW TECNOLOGY, ENERGY

AND THE ENVIRONMENT


ENEA GRID improvements in progress

Increasing resources: AIX +32 SP nodes/Frascati, Cluster 16 AMD 64bit/Casaccia, Cluster 8 G5/Trisaia, 2 CPU AIX/Brindisi (TELEGRID Project).

Extending Windows services.

AFS optimization.

Studies in collaboration with CASPUR to manage the replication of large amount of data over WAN with better performances than AFS.

Participation in CASPUR storage technology tracking activity (SAN systems and file systems testing and evaluation, AFS implementation for parallel systems [IPP Garching]=> DEISA Grid Project)


Connection with other GRID Projects

Each GRID project is characterized by its own architectural design and middleware.

The focus of the partecipation of ENEA INFO in other GRID projects is GRID interoperability

Access to ENEA GRID through other GRID models should be compatible with ENEA GRID architecture: gateway implementation.

Current GRID projects:

● EGEE [Datagrid]

● FIRB/Grid.it : WP13, Computational Chemistry (CHEMGRID), Perugia Univ., ENEA Casaccia, et al. ENEA activity starting July 2004 at Frascati


ENEA GRID and DATAGRID PROJECT

ENEA has partecipated (unfunded) in DATAGRID project (ended 3/2004) through the collaboration with ESA, ESRIN Laboratory in Frascati.

A gateway bewteen DATAGRID infrastructure and ENEA-GRID has been set-up at ESRIN enabling the submission of jobs from DATAGRID to ENEA-GRID

The gateway has been tested running the analysis of the data from the GOME satellite (Ozone level) using IDL codes which can be easily run in a multiplatform environment.

The gateway provided a limited integration between the different GRID environment: EGEE should a attain a full integration


ENEA-GRID and EGEE Project

EGEE (Enabling GRID for e-science in Europe) is a project in the 6th EU framework program, with the mission to deliver production level GRID services [www.eu-egee.org].

ENEA is one of the funded partners in the project, inside the Italian partecipation which is coordinated by INFN.

EGEE project is starting (April 2004); a kick-off Meeting has been hold in Cork, April 2004.

The installation of a small testing environment (7 Linux boxes) with standard EGEE middleware is currently in progress at ENEA Frascati.

The development of a gateway to ENEA-GRID will follow.

GRID computing Frascati 14/10/2004 [email protected] .

Documents

Transcript of GRID computing Frascati 14/10/2004 [email protected] .