EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE The Future of EGEE and gLite Dieter...
-
Upload
estella-dickerson -
Category
Documents
-
view
214 -
download
0
Transcript of EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE The Future of EGEE and gLite Dieter...
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
The Future of EGEE and gLite
Dieter KranzlmüllerGUP – Institute of Graphics and Parallel ProcessingJoh. Kepler Univ. Linz, Austria
CE EGEE&SEEGRID-2 Summer School on Grid Appl.
08 July 2006Budapest, Hungary
The Future of EGEE and gLite 2
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Austrian Grid Initiative
GGF - Global Grid Forum
e-IRG - e-Infrastructure Reflection Group
EGEE – Enabling Grids for E-sciencECERN, Geneva, Switzerland
GUP – Institute of Graphics and Parallel ProcessingJoh. Kepler University Linz
The Future of EGEE and gLite 3
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Lost in Definitions?
Defining the “Grid”:• Access to (high performance) computing power• Distributed parallel computing• Improved resource utilization through resource sharing• Increased memory provision• Controlled access to distributed memory• Interconnection of arbitrary resources
(sensors, instruments, …)• Collaboration between users/resources• Higher abstraction layer above network services• Corresponding security • …
The Future of EGEE and gLite 4
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Defining the Grid
• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.
• This interconnection of users, resources, and services for jointly addressing dedicated tasks is called a virtual organization.
• Comparison between Grids and Networks:– Networks realize message exchange between endpoints– Grids realize services for the users higher level of abstraction
The Future of EGEE and gLite 5
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Defining the Grid
• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.
The Future of EGEE and gLite 6
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The EGEE Project
• EGEE– 1 April 2004 – 31 March 2006– 71 partners in 27 countries,
federated in regional Grids
• EGEE-II– 1 April 2006 – 31 March 2008– Expanded consortium
91 partners 11 Joint Research Units (48 partners)
– Exploitation of EGEE results– Emphasis on providing
production-level infrastructure increased support for applications interoperation with other
Grid infrastructures more involvement from Industry
The Future of EGEE and gLite 7
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Defining the Grid
• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.
Status of EGEE-II (as of July 8, 2006)
The Future of EGEE and gLite 8
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE Infrastructure
Country participating
in EGEE
Scale (June 2006):~ 200 sites in 40 countries
~ 20 000 CPUs
> 10 PB storage
> 20 000 concurrent jobs per day
> 60 Virtual Organizations
The Future of EGEE and gLite 9
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE Infrastructures
• Production service– Scaling up the infrastructure with resource centres around the globe– Stable, well-supported infrastructure, running only well-tested and
reliable middleware
• Pre-production service– Run in parallel with the production service (restricted nr of sites)– First deployment of new versions of the gLite middleware – Test-bed for applications and other external functionality
• T-Infrastructure (Training&Education)– Complete suite of Grid elements
and application (Testbed, CA, VO, monitoring, …)
– Everyone can register and use GILDA for training and testing
15 sites on 3 continents(all of them GÉANT sites)
The Future of EGEE and gLite 10
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Defining the Grid
• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.
Status of EGEE-II (as of July 8, 2006)
The Future of EGEE and gLite 11
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Production Grid Middleware
Key factors in EGEE Grid Middleware Development:
1. Strict software processUse industry standard software engineering methods– Software configuration management, version control, defect
tracking, automatic build system, …
2. Conservative approach in what software to useAvoid “cutting-edge” software– Deployment on over 100 sites cannot assume a homogenous
environment – middleware needs to work with many underlying software flavors
Avoid evolving standards– Evolving standards change quickly (and sometime significantly
cf. OGSI vs. WSRF) – impossible to keep pace on > 100 sites
Long (and te
dious) path
from prototypes to
production
The Future of EGEE and gLite 12
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE Middleware: gLite
• Exploit experience & existing components – VDT (Condor, Globus) – EDG/LCG– AliEn– …
• Develop a lightweight stack of EGEE generic middleware– Dynamic deployment– Pluggable components
• Focus is on re-engineering and hardening
• March 4, 2006: gLite 3.0
LCG-2
prototyping
prototyping
product
2004200420042004
20052005 product
gLite
20062006 gLite 3.0
The Future of EGEE and gLite 13
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Developing
• gLite 3.0 now available on production infrastructure
• After gLite 3.0:– Continuous release of single components
As needed by users and as made available by developers
– Major releases provide a “check-point” In general in coincidence with major application challenges
• Continuing development to– Bring components not yet included in release to maturity – Improve functionality– Increase robustness– Increase usability– Improve the compliance to international standards
The Future of EGEE and gLite 14
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Grid Interoperability
Leading role in building world-wide grids• Incubator for new Grid
projects world-wide
• Interoperation efforts– Bilateral: EGEE/OSG, EGEE/NDGF,
EGEE/NAREGI– Multilateral: Grid Interoperability Now
(GIN)
• Experiences and requirements fed back into standardization process (GGF)
• Strengthening contacts with industry
GIN
The Future of EGEE and gLite 15
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM
Slide Courtesy David Abramson
The Future of EGEE and gLite 16
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM,Lower Middleware
Upper Middleware & Tools
Bonds
Slide Courtesy David Abramson
The Future of EGEE and gLite 17
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Defining the Grid
• A Grid is the combination of networked resources and the corresponding Grid middleware, which provides Grid services for the user.
Status of EGEE-II (as of July 8, 2006)
The Future of EGEE and gLite 18
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE Applications
• >20 applications– High Energy Physics– Biomedicine– Earth Sciences – Computational Chemistry– Astronomy– Geo-Physics– Financial Simulation– Fusion
• Further applications in evaluation
Applications now moving from testing to routine and daily usage
The Future of EGEE and gLite 19
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
High Energy Physics
Large Hadron Collider (LHC):• One of the most powerful instruments
ever built to investigate matter• 4 Experiments: ALICE, ATLAS, CMS, LHCb • 27 km circumference tunnel• Due to start up in 2007
Mont Blanc(4810 m)
Downtown Geneva
The Future of EGEE and gLite 20
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Applications Example: WISDOM
• Grid-enabled drug discovery process for neglected diseases– In silico docking
compute probability that potential drugs dock with target protein
– To speed up and reduce cost to develop new drugs
• WISDOM (World-wide In Silico Docking On Malaria)– First biomedical data challenge – 46 million ligands docked in 6 weeks
Target proteins from malaria parasite Molecular docking applications:
Autodock and FlexX ~1 million virtual ligands selected
– 1TB of data produced – 1000 computers in 15 countries
Equivalent to 80 CPU years
• Significant results– Best hits to be re-ranked using Molecular Dynamics
New data challenge planned
for a
utumn 2006
The Future of EGEE and gLite 21
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Example: Avian flu
• Avian Flu H5N1– H5 and N1 = proteins on virus surface
• Biological goal of data challenge– Study in silico the impact of selected point
mutations on the efficiency of existing drugs – Find new potential drugs
• Data challenge parameters:– 5 Grid projects: Auvergrid, BioinfoGrid, EGEE,
Embrace, TWGrid– 1 docking software: autodock– 8 conformations of the target (N1)– 300 000 selected compounds >100 CPU years to dock all configurations
on all compounds
• Timescale: – First contacts established: 1 March 2006– Data Challenge kick-off: 1 April 2006– Duration: 4 weeks
N1H5
Credit: Y-T Wu
Credit: Y-T Wu
The Future of EGEE and gLite 22
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Example: Determining earthquake mechanisms
• Seismic software application determines epicentre, magnitude, mechanism
• Analysis of Indonesian earthquake (28 March 2005)– Seismic data within 12 hours after the earthquake– Solution found within 30 hours after earthquake occurred
10 times faster on the Grid than on local computers– Results
Not an aftershock of December 2004 earthquake Different location (different part of fault line further south) Different mechanism
Rapid analysis of earthquakes important for relief efforts
Peru, June 23, 2001Mw=8.4
Sumatra, March 28, 2005Mw=8.5
The Future of EGEE and gLite 24
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE-II Overview
The Future of EGEE and gLite 25
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE and Sustainability
BUT …
• How does EGEE compare to other computing infrastructures?– Number of infrastructure users?– Number of application domains?– Number of computing nodes?– Number of years in service?
• What would happen, if EGEE is turned off?• What happens after April 2008
(End of EGEE-II)?
The Future of EGEE and gLite 26
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The Future of Grids
• Increasing the number of infrastructure users by increasing awareness– Dissemination and outreach– Training and education
• Increasing the number of applications by improving application support and middleware functionality– High level grid middleware extensions
• Increasing the grid infrastructure– Incubating related projects– Ensuring interoperability between projects
• Protecting user investments– Towards a sustainable grid infrastructure
The Future of EGEE and gLite 27
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
User Information & Support
• More than 170 training events and summer schools across many countries– >3000 people trained
induction; application developer; advanced; retreats– Material archive online with ~250 presentations
• Public and technical websites • Dissemination material constantly evolving to expand
information and keep it up to date
• 4 conferences organized (~ 460 @ Pisa)
• Next conference: September 2006 in Geneva ~600 participants
The Future of EGEE and gLite 28
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Industry and EGEE-II
• Industry Task Force– Group of industry partners in the project– Links related industry projects (NESSI, BEinGRID, …)– Works with EGEE’s Technical Coordination Group
• Collaboration with CERN openlab project– IT industry partnerships for hardware and software
development
• EGEE Business Associates (EBA)– Companies sponsoring work on joint-interest subjects
• Industry Forum– Led by Industry to improve Grid take-up in Industry– Organises industry events and disseminates grid information
The Future of EGEE and gLite 29
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The Future of Grids
• Increasing the number of infrastructure users by increasing awareness– Dissemination and outreach– Training and education
• Increasing the number of applications by improving application support and middleware functionality– High level grid middleware extensions
• Increasing the grid infrastructure– Incubating related projects– Ensuring interoperability between projects
• Protecting user investments– Towards a sustainable grid infrastructure
The Future of EGEE and gLite 30
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM,Lower Middleware
Upper Middleware & Tools
Bonds
Slide Courtesy David Abramson
???
The Future of EGEE and gLite 31
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Portals on EGEE
P-Grade
Genius
The Future of EGEE and gLite 32
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
www.austriangrid.at
High Level Middleware Extensions
Example:• Understanding data through graphical representations
Scientific Visualization
• Approach: Grid Visualization Kernel (GVK)– Interactive visualization service in the grid– Integrated in existing visualization toolkits– Optimized transportation and processing
• Research project:– Interactivity on the Grid glogin– Batch-bypass technology– Easy to use/install– Secure
The Future of EGEE and gLite 33
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
glogin - Interactive Tunneling
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
WorkerNode
GatekeeperGatekeeper
on the Grid
ClientClient
gloginPoint ofContact
glogin’socket
InteractiveBidirectional Connection
TrafficForwarding
The Future of EGEE and gLite 34
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
glogin Shell
• Interactive access to grid nodes
• Authentication via grid certificates• Tunneling of arbitary traffic
The Future of EGEE and gLite 35
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Grid Visualization Kernel
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
Workernode
GridVisualization
Kernel
GridVisualization
Kernel
on the Grid
ClientClient
glogin
glogin’
vis.data
vis.data
vis.data W
OR
KIN
G!
inter-action
interaction interaction
The Future of EGEE and gLite 36
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Example: Biomedicine
• Parallel simulationof blood flowon the Grid
• Onlinevisualizationof simulationresults on thedesktop
• Interactivesteering ofsimulation
• Grid is„invisible“
Cooperation with University Amsterdam
The Future of EGEE and gLite 37
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Example: Flooding Crisis Support
• Simulation of floodingon the Grid
• Onlinevisualizationof simulationresults in theCAVE
• Interactivesteering ofsimulation
• Grid is„invisible“
Cooperation with Slowak Academy of Sciences
The Future of EGEE and gLite 38
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Scientific Visualization
Use your favourite device to connect to the Grid:
Sony PSP – PlayStation Portable
The Future of EGEE and gLite 39
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
MiddlewareGlobus GT4 CondorAPST
PlatformInfrastructure
Unix Windows JVM TCP/IP MPI .Net Runtime
Environmental Sciences
Life & Pharmaceutical
Sciences
ApplicationsGeo Sciences
Building Software for the Grid
VPN SSH
Courtesy IBM,Lower Middleware
Upper Middleware & Tools
Bonds
Slide Courtesy David Abramson
GENIUS, P-Grade, GVK, glogin, GVid, …
The Future of EGEE and gLite 40
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The Future of Grids
• Increasing the number of infrastructure users by increasing awareness– Dissemination and outreach– Training and education
• Increasing the number of applications by improving application support and middleware functionality– High level grid middleware extensions
• Increasing the grid infrastructure– Incubating related projects– Ensuring interoperability between projects
• Protecting user investments– Towards a sustainable grid infrastructure
The Future of EGEE and gLite 41
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Projects related to EGEE
EUGRIDGRID
The Future of EGEE and gLite 42
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Related Infrastructures
GIN
The Future of EGEE and gLite 43
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The Future of Grids
• Increasing the number of infrastructure users by increasing awareness– Dissemination and outreach– Training and education
• Increasing the number of applications by improving application support and middleware functionality– High level grid middleware extensions
• Increasing the grid infrastructure– Incubating related projects– Ensuring interoperability between projects
• Protecting user investments– Towards a sustainable grid infrastructure
The Future of EGEE and gLite 44
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Sustainability: Beyond EGEE-II
• Need to prepare for permanent Grid infrastructure– Maintain Europe’s leading position in global science Grids– Ensure a reliable and adaptive support for all sciences– Independent of project funding cycles– Modelled on success of GÉANT
Infrastructure managed centrally in collaboration with national bodies (in EGEE-II: JRUs)
The Future of EGEE and gLite 45
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Grids in Europe
• Great investment in developing Grid technology• Sample of National Grid projects:
– Austrian Grid Initiative– DutchGrid – France: Grid’5000– Germany: D-Grid; Unicore– Greece: HellasGrid– Grid Ireland – Italy: INFNGrid; GRID.IT– NorduGrid– Swiss Grid– UK e-Science: National Grid Service;
OMII; GridPP
• EGEE provides framework for national, regional and thematic Grids
The Future of EGEE and gLite 46
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Evolution
EGEE EGEE-IIEDG EGEE-III
European e-Infrastructure
Coordination
Testbeds Utility ServiceRoutine Usage
The Future of EGEE and gLite 47
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Summary
Grids represent a powerful new tool for science
Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago)
EGEE offers …• … a mechanism for linking together people, resources
and data of many scientific community• … a basic set of middleware for gridfying applications with
documentation, training and support• … regular forums for linking with grid experts, other
communities and industry
The Future of EGEE and gLite 48
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Summary
Success will lead to the adoption of grids as the main computing infrastructure for science
If we succeed then the potential return to international scientific communities will be enormous and open the path for commercial and industrial applications
The Future of EGEE and gLite 49
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
EGEE’06 Conference
• EGEE’06 – Capitalising on e-infrastructures – Demos– Related Projects– Industry– International community (UN organisations in Geneva etc.)
• 25-29 September 2006• Geneva, Switzerland
http://www.eu-egee.org/egee06