Post on 08-May-2015
The CrossGrid Project
Marcel Kunze, FZK
representing the X#-Collaboration
ACAT 2002 Moscow Marcel Kunze - FZK
Main Objectives
New category of Grid enabled applications Computing and data intensive Distributed Interactive, near real time response (a person in a loop) Layered
New programming tools
Grid more user friendly, secure and efficient
Interoperability with other Grids
Implementation of standards
ACAT 2002 Moscow Marcel Kunze - FZK
Poland:Cyfronet & INP CracowPSNC PoznanICM & IPJ Warsaw
Portugal:LIP Lisbon
Spain:CSIC SantanderValencia & RedIrisUAB BarcelonaUSC Santiago & CESGA
Ireland:TCD Dublin
Italy:DATAMAT
Netherlands:UvA Amsterdam
Germany:FZK KarlsruheTUM MunichUSTU Stuttgart
Slovakia:II SAS Bratislava
Greece:AlgosystemsDemo AthensAuTh Thessaloniki
Cyprus:UCY Nikosia
Austria:U.Linz
21 21 institutinstitutees s
11 11 countriescountries
CrossGrid Collaboration
ACAT 2002 Moscow Marcel Kunze - FZK
GRIDLAB
GRIA
EGSO
DATATAG
CROSSGRID
DATAGRID
Applications
GRIP EUROGRID
DAMIENMiddleware
& Tools
Underlying Infrastructures ScienceIndustry / business
- Links with European National efforts
- Links with US projects (GriPhyN, PPDG, iVDGL,…)
IST Grid Project Space
ACAT 2002 Moscow Marcel Kunze - FZK
Collaboration with other # Projects
Objective – Exchange of Information Software components
Partners DATAGRID DATATAG GRIDLAB EUROGRID and GRIP
GRIDSTART Participation in GGF
ACAT 2002 Moscow Marcel Kunze - FZK
Project Phases
M 1 - 3: requirements definition and merging
M 4 - 12: first development phase: design, 1st prototypes, refinement of requirements
M 13 -24: second development phase: integration of components, 2nd prototypes
M 25 -32: third development phase: complete integration, final code versions
M 33 -36: final phase: demonstration and documentation
ACAT 2002 Moscow Marcel Kunze - FZK
Network infrastructure, archivers, HPC/HPV systems,
Labour instruments, etc. Local domain services
Protocols, Authentication, Authorization, Access policy, Resource management, etc.
Remote Data Access
Optimization
Monitoring
Schedulers
Roaming Access
Portals
Interactive simulation and visualisation of a biomedical systemFlooding crisis team support
Distributed Data Analysis in High Energy PhysicsWeather forecast and air pollution modeling
FABRIC FABRIC INFRASTRUCTUREINFRASTRUCTURE
GridGridSERVICESSERVICES
TOOLSTOOLS
APPLICATIONSAPPLICATIONS
GLOBUS TOOLKIT , Condor-G, ...
DATAGRID SET OF TOOLS
Grid Visualization
KernelBenchmarks
Structure Overview
ACAT 2002 Moscow Marcel Kunze - FZK
CrossGrid Architecture
ApplicationsAnd Supporting
Tools
ApplicationsDevelopment
Support
Grid Common Services
Grid Visualisation
Kernel
Grid Visualisation
Kernel
DataMining on
Grid
DataMining on
Grid
Interactive Distributed
Data Access
Interactive Distributed
Data Access
Globus Replica
Manager
Globus Replica
Manager
Roaming Access
Roaming Access
Grid Resource
Management
Grid Resource
Management
Grid Monitoring
Grid Monitoring
MPICH-GMPICH-G
Distributed Data
Collection
Distributed Data
Collection
User Interaction Service
User Interaction Service
DataGridReplica
Manager
DataGridReplica
Manager
Datagrid Job
Manager
Datagrid Job
Manager
GRAMGRAM GSIGSIReplica CatalogReplica Catalog GASSGASSMDSMDS GridFTPGridFTPGlobus-IOGlobus-IO
Resource Manager
Resource Manager
CPUCPU
ResourceManagerResourceManager
Resource Manager
Resource Manager
Secondary Storage
Secondary Storage
ResourceManagerResourceManager
Scientific Instruments
(Medical Scaners, Satelites, Radars)
Scientific Instruments
(Medical Scaners, Satelites, Radars)
ResourceManagerResourceManager
DetectorLocal High
Level Trigger
DetectorLocal High
Level Trigger
ResourceManagerResourceManager
VR systems(Caves,immerse desks)
VR systems(Caves,immerse desks)
ResourceManagerResourceManager
Visualization tools
Visualization tools
Optimization of Data Access
Optimization of Data Access
Tertiary StorageTertiary Storage
Local Resources
BiomedicalApplicationBiomedicalApplication
PortalPortalPerformance
AnalysisPerformance
AnalysisMPI
VerificationMPI
VerificationMetrics and BenchmarksMetrics and Benchmarks
HEP High LevelTriggerHEP High
LevelTriggerFlood
ApplicationFlood
ApplicationHEP Interactive Distributed Data
Access Application
HEP Interactive Distributed Data
Access Application
HEP DataMining on Grid
Application
HEP DataMining on Grid
Application
WeatherForecast
application
WeatherForecast
application
ACAT 2002 Moscow Marcel Kunze - FZK
Layered Structure
Interactive and Data Intensive Applications (WP1) Interactive simulation and visualization of a biomedical system Flooding crisis team support Distributed data analysis in HEP Weather forecast and air pollution modeling
Grid Application Programming Environment (WP2)
MPI code debugging and verification Metrics and benchmarks Interactive and semiautomatic performance evaluation tools
Grid Visualization Kernel Data Mining
New CrossGrid Services (WP3)
Globus Middleware
Fabric Infrastructure (Testbed WP4)
DataGridGriPhyN
...
Services
HLA
Portals and roaming access Grid resource management Grid monitoring Optimization of data access
ACAT 2002 Moscow Marcel Kunze - FZK
Scope of Applications
Applications in health and environment Data federation, processing and interpretation in geographically
distributed locations Fast, interactive decision making
Interactive access to distributed Databases Super computers and High Performance Clusters Visualisation engines Medical scanners Environmental data input devices
ACAT 2002 Moscow Marcel Kunze - FZK
Application Requirements
High quality presentation
High frame rate
Intuitive interaction
Real-time response
Interactive algorithms
High performance computing and networking
Distributed resources and data
ACAT 2002 Moscow Marcel Kunze - FZK
Role of Network Latency
Communication delay and rendering delay are negligible
ACAT 2002 Moscow Marcel Kunze - FZK
CrossGrid Application Development (WP1)
Interactive simulation and visualisation of a biomedical system Grid-based system for pre-treatment planning in vascular
interventional and surgical procedures through real-time interactive simulation of vascular structure and flow.
Flooding crisis team supportDistributed interactive data analysis in HEP Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb)
Weather forecast and air pollution modelling Porting distributed/parallel codes on Grid Coupled Ocean/Atmosphere Mesoscale Prediction System STEM-II Air Pollution Code
ACAT 2002 Moscow Marcel Kunze - FZK
Interactive Simulation and Visualisation
of a Biomedical SystemGrid-based prototype system for treatment planning in vascular interventional and surgical procedures through near real-time interactive simulation of vascular structure and flow.
The system will consist of a distributed near real-time simulation environment, in which a user interacts in Virtual Reality (VR) and other interactive display environments.
A 3D model of the arteries, derived using medical imaging techniques, will serve as input to a simulation environment for blood flow calculations.
The user will be allowed to change the structure of the arteries, thus mimicking an interventional or surgical procedure.
ACAT 2002 Moscow Marcel Kunze - FZK
Current Situation
Observation
Diagnosis & Planning
Treatment
ACAT 2002 Moscow Marcel Kunze - FZK
Experimental Setup
ACAT 2002 Moscow Marcel Kunze - FZK
Angio w/ Fem-Fem &
Fem-Pop
AFB w/ E-S
Prox.Anast.
Angio w/Fem-Fem
AFB w/ E-E Prox.
Anast.
Preop
Alternatives
Simulation Based Planning and Treatment
ACAT 2002 Moscow Marcel Kunze - FZK
VR-Interaction
ACAT 2002 Moscow Marcel Kunze - FZK
Flood Crisis Prevention
Support system for establishment and operation of Virtual Organization for Flood Forecasting associating a set of individuals and institutions involved in flood prevention and protection.
The system will employ a Grid technology to seamlessly connect together the experts, data and computing resources needed for quick and correct flood management decisions.
The main component of the system will be a highly automated early warning system based on hydro-meteorological (snowmelt) rainfall-runoff simulations.
System will integrate the advanced communication techniques allowing the crisis management teams to consult the decisions with various experts. The experts will be able to run the simulations with changed parameters and analyze the impact.
ACAT 2002 Moscow Marcel Kunze - FZK
Storage systems
databases
surface automatic meteorological and hydrological stations
systems for acquisition and processing of satellite information
meteorological radars
External sources of informationGlobal and regional centers GTSEUMETSAT and NOAAHydrological services of other countries
Data sources
meteorological models
hydrological models
hydraulic models
HPC, HTC
Grid infrastructure
Flood crisis teams
meteorologistshydrologistshydraulic engineers
Usersriver authoritiesenergyinsurance companiesnavigation
mediapublic
Virtual Organization for Flood Forecasting
ACAT 2002 Moscow Marcel Kunze - FZK
Váh River Catchment Area: 19700km2, 1/3 of Slovakia
(Inflow point)
Nosice
Strečno
(Outflow point)
Pilot Site Catchment Area: 2500km2
(above Strečno: 5500km2)
Váh River Pilot Site
Flood Crisis Prevention
Water stages/discharges in the real time operating hydrological stations
Mapping of the flooded areas
ACAT 2002 Moscow Marcel Kunze - FZK
Flow +
water depth
s
Flood SimulationsResults
ACAT 2002 Moscow Marcel Kunze - FZK
Distributed Analysis in High Energy Physics
Challenging points Access to large distributed databases in the Grid Development of distributed data-mining techniques Definition of a layered application structure Integration of user-friendly interactive access
(based on PROOF)
Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb)
ACAT 2002 Moscow Marcel Kunze - FZK
PROOF
Local
Remote
Selection
Parameters
Procedure
Proc.C
Proc.C
Proc.C
Proc.C
Proc.C
PROOF
CPU
CPU
CPU
CPU
CPU
CPU
TagDB
RDB
DB1
DB4
DB5
DB6
DB3
DB2
ACAT 2002 Moscow Marcel Kunze - FZK
Weather Forecast and Air Pollution Modeling
Integration of distributed databases into Grid
Migration of data mining algorithms to Grid
Porting distributed atmospheric & wave models to Grid
Porting parallel codes for air quality models to Grid
Integration, testing and demonstration of the application in the testbed environment
ACAT 2002 Moscow Marcel Kunze - FZK
COAMPSCoupled Ocean/Atmosphere Mesoscale Prediction System:
Atmospheric Components
Complex Data Quality ControlAnalysis:
Multivariate Optimum Interpolation Analysis of Winds and Heights Univariate Analyses of Temperature and Moisture Optimum Interpolation Analysis of Sea Surface Temperature
Initialization: Variational Hydrostatic Constraint on Analysis Increments Digital Filter
Atmospheric Model: Numerics: Nonhydrostatic, Scheme C, Nested Grids, Sigma-z Physics: Convection, Explicit Moist Physics, Radiation, Surface Layer
Features: Globally Relocatable (5 Map Projections) User-Defined Grid Resolutions, Dimensions, and Number of Nested Grids 6 or 12 Hour Incremental Data Assimilation Cycle Can be Used for Idealized or Real-Time Applications Single Configuration Managed System for All Applications Operational at:
7 Areas, Twice Daily, using 81/27/9 km or 81/27 km grids Forecasts to 72 hours
Operational at all Navy Regional Centers (w/GUI Interface)
ACAT 2002 Moscow Marcel Kunze - FZK
Status Quo … Quo Vadis ?
Current state (briefly) Simulation done on a single system or local clusters Visualisation on a single system, locally
What we are going to achieve HPC, HTC, HPV in geographically distributed environment Improved interaction with the end user Near real time simulations Different visualisation equipments (adaptive according to the end-user needs), like PDA Workstations VR studio (e.g. CAVE)
ACAT 2002 Moscow Marcel Kunze - FZK
Grid Application Programming
Environment (WP2)
MPI code debugging and verification
Metrics and benchmarks
Interactive and semiautomatic performance evaluation tools
Specify, develop, integrate, test tools for HPC and HTC applications on the Grid
ACAT 2002 Moscow Marcel Kunze - FZK
New Grid Services and Tools (WP3)
Portals and roaming accessGrid resource managementGrid monitoringOptimisation of data access
Objectives To develop interactive compute- and data-intensive applications To develop user-friendly Grid environments To offer easy access to the applications and Grid To have reasonable trade-off between resource usage efficiency
and application speedup To support management issues while accessing resources
ACAT 2002 Moscow Marcel Kunze - FZK
Auth Thessaloniki
U v Amsterdam
FZK Karlsruhe
TCD Dublin
U A Barcelona
LIP Lisbon
CSIC Valencia CSIC Madrid
USC Santiago CSIC Santander
DEMO Athens UCY Nikosia
CYFRONET Cracow
II SAS Bratislava
PSNC Poznan
ICM & IPJ Warsaw
1515 sisitteess
International Testbed Organisation (WP4)
Testbed setup & incremental evolution Integration with DataGridInfrastructure supportVerification & quality control
ACAT 2002 Moscow Marcel Kunze - FZK
Summary
Layered structure of all X# applicationsReuse of SW from DataGrid and other # projectsGlobus as the bottom layer of the middlewareHeterogeneous computer and storage systemsDistributed development and testing of SW 12 partners in applications 14 partners in middleware 15 partners in testbeds
ACAT 2002 Moscow Marcel Kunze - FZK
1980s: Internet 1990s: Web 2000s: Grid
Where do we need to get to ?Applications to support an “e-society”
(“Cyber-Infrastructure”)An international Grid infrastructure which
hides the complexities from the users (“Invisible Computing”)
A powerful and flexible network infrastructure
Where do we need to invest ?Applications targeted at realistic problems in
“e-science”Prototypes of Grid infrastructuresMaintain and improve the GEANT network
Expression of Interest for EU FP6 program:“Enabling Grids and e-Science in Europe
(EGEE)”
Grid-enabled Applications
Prototype Grid Infrastructures
Gèant: World Class Networking
ACAT 2002 Moscow Marcel Kunze - FZK
GRIDLAB
GRIA
EGSO
DATATAG
CROSSGRID
DATAGRID
Applications
GRIP EUROGRID
DAMIENMiddleware
& Tools
Underlying Infrastructures ScienceIndustry / business
EGEE Project Space
Enabling Grids and E-Science in Europe (EGEE)
ACAT 2002 Moscow Marcel Kunze - FZK
First results of EGEE Brainstorming
ConsultingPrototypingDeployment
ConsultingTraining Courses
DisseminationForum
Tools and Service Development
Creation and support E- Science centres
Applications inOther Sciences
EI ROforum
S/W HardeningGLOBUS
EuroGrid, Gridlab etc.
Semantic GRI DDatabaseSecurity
European I nfrastructure
Integrated ProjectENABLI NG GRI DS ESCI ENCE EUROPE
EGEE
National eScienceCentres I ndustry
Applications
SMEs developingGrid-enabled Applications
I ndustry Outreach
ModulableTestbeds
R&D Agenda
Deployment withI T I ndustry
ScienceOutreach
C Jon
es,
D O
Will
iam
s, e
t al