The Open Science Grid OSG
-
Upload
mufutau-kane -
Category
Documents
-
view
44 -
download
0
description
Transcript of The Open Science Grid OSG
![Page 1: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/1.jpg)
The Open Science GridOSG
Ruth PordesFermilab
![Page 2: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/2.jpg)
2
What is OSG? A Consortium of people working together to
Interface Farms and Storage to a Grid and
Researchers using these resources by adapting their applications to run on the Grid and
Software developers providing middleware and
A project that provides the Operations, Support, Training and Help to make it effective.
![Page 3: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/3.jpg)
3
Who is OSG ?
Large global physics collaborations: US ATLAS, US CMS, LIGO, CDF, D0, STAR
Research collaborations such as Mariachi, GROW,
Grid technology groups: Condor, Globus, SRM, NMI
Many DOE Labs and DOE/NSF sponsored University IT facilities and
Partnerships with TeraGrid and EGEE, and Campus Grids such as TACC, GLOW etc.
![Page 4: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/4.jpg)
4
OSG Consortium
![Page 5: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/5.jpg)
5
When is OSG ? Grown from of grass-roots collaboration of
GriPhyN, iVDGL and PPDG participants in 2003.
5 years of funding starting ~9/2006 from DOE SciDAC-II and NSF MPS and OCI
Deliver to US LHC and LIGO scales in 2008 and 2009:— Need to routinely distribute data at 1-5 Gbps over 10-50 sites.— Need to routinely exceed 10,000 running jobs per client— Need to reach 99% success rate for 10,000 jobs per day submission under
heavy load
Active engagement effort centered at RENCI to include new sciences.
![Page 6: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/6.jpg)
6
The OSG Map Aug-2006
![Page 7: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/7.jpg)
7
OSG’s world is flat - a Grid of Grids - from Local to Global
Global ScienceCommunity Systems
e.g. FermiGrid, NWIC
Local Campus And RegionalGrids
NationalCyberInfrastructures
for Science
e.g. OSG - TeraGrid
e.g. CMS , D0
![Page 8: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/8.jpg)
8
From the Local (Campus) Grids to the Wide Area GridWithin an organization a Local Grid provides for
simplicity in sharing and efficiency in purchasing and administration.
However, researchers collaborate outside the bounds of a single campus.
With a uniform environment acting globally is just an extension of acting locally.
Open Science Grid focuses on interoperation of the local and the wide area.
![Page 9: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/9.jpg)
9
A resource can be accessed by a user via the campus, community or national grid.
A user can access a resource with a campus, community or national grid identity.
![Page 10: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/10.jpg)
10
Example Uses
High Energy Physics ATLAS - >15 Million proton collision events simulated at 10
minutes each CMS - >70 Million events simulated, reconstructed and analyzed
Biology - GADU Populates databases from search and analysis of similarities and
differences among thousands of publicly available genome and protein sequences and metabolic pathways.
Gravitational Wave Physics - LIGO Data grid Eases grid tools to ensure that 9 computing sites have a copy of the
interesting data, and researchers at 36 LSC institutions use the LDG to find the data they need."
Math research
Education Grid Summer Workshop teaches students to run jobs on OSG and
TeraGrid.
![Page 11: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/11.jpg)
11
Running (and monitored) “OSG jobs” in 06/06.
![Page 12: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/12.jpg)
12
Example GADU run in 04/06
![Page 13: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/13.jpg)
13
Integration Testing of the System
Multi-site Integration Grid tests new OSG Releases and Configurations.
Software Readiness and Validations occur before deployment on the Integration Grid.
Integration Grid SitesIntegration Grid Sites
![Page 14: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/14.jpg)
14
CMS - US part of a Global Community Grid
GermanyTaiwan UKItaly
Data & jobs moving locally, regionally & globally within CMS grid.
Transparently across grid boundaries from campus to global.
Florida
USA
CERN
Caltech
Wisconsin
UCSD
France
Purdue
MIT
UNL
OSG
EGEE
![Page 15: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/15.jpg)
15
How do People and Organizations Participate? VO Registers with with Operations Center
Signs VO Agreement
User registers with VO User added to VOMS of one or more VOs. VO responsible for users to sign AUP. VO responsible for VOMS service support.
Sites Register with the Operations Center Signs the Service Agreement. Agree on which VOs to support (striving for default admit)
VOs and Sites provide Support Center Contact and join Operations groups
![Page 16: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/16.jpg)
16
Community Documentation..
![Page 17: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/17.jpg)
17
![Page 18: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/18.jpg)
18
OSG Principles
Systems are Virtual Organization scoped -- groups of users working together with a shared, common environment.
Sites maintain control, authority and management of use of their Grid accessible resources.
Opportunistically available resources are beneficial. Priorities are governed by Policies (role based within a VO).
The distributed system is heterogeneous. Information must be provided to allow applications to know which resources they can successfully use.
![Page 19: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/19.jpg)
19
VO software stacks that Rely on the Virtual Data Toolkit
NSF Middleware Initiative (NMI): Condor, Globus, Myproxy
Virtual Data Toolkit (VDT) Common Services NMI + VOMS, CEMon (common EGEE
components), MonaLisa, Clarens, AuthZ
OSG Release Cache: VDT + Configuration, Validation, VO management
LHCServices & Framewor
k
LIGOData Grid
OSG VOFramework
Infr
ast
ruct
ure
Applic
ati
ons …Bio
Services &Framewor
k
![Page 20: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/20.jpg)
20
What is the VDT?
A collection of software Grid software (Condor, Globus and lots more) Virtual Data System (Origin of the name “VDT”) Utilities
An easy installation Goal: Push a button, everything just works Two methods:
Pacman: installs and configures it all RPM: installs some of the software, no configuration
A support infrastructure
![Page 21: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/21.jpg)
21
Who uses the VDT?
Open Science Grid
LIGO Data Grid
LCG LHC Computing Grid, from CERN
EGEE Enabling Grids for E-Science
![Page 22: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/22.jpg)
22
What software is in the VDT?
Security VOMS (VO membership) GUMS (local authorization) mkgridmap (local authorization) MyProxy (proxy management) GSI SSH CA CRL updater
Monitoring MonaLISA gLite CEMon
Accounting OSG Gratia
Job Management Condor (including Condor-G &
Condor-C) Globus GRAM
Data Management GridFTP (data transfer) RLS (replication location) DRM (storage management) Globus RFT
Information Services Globus MDS GLUE schema & providers
![Page 23: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/23.jpg)
23
Client tools Virtual Data System SRM clients (V1 and V2) UberFTP (GridFTP client)
Developer Tools PyGlobus PyGridWare
Testing NMI Build & Test VDT Tests
What software is in the VDT?
Support Apache Tomcat MySQL (with MyODBC) Non-standard Perl modules Wget Squid Logrotate Configuration Scripts
And More!
![Page 24: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/24.jpg)
24
Due diligence to Security
Risk assessment, planning, Service auditing and checking
Incident response, Awareness and Training, Configuration management,
User access Authentication and Revocation, Auditing and analysis. End to end trust in quality of code executed on remote CPU -signatures?
Identity and Authorization: Extended X509 Certificates OSG is a founding member of the US TAGPMA. DOEGrids provides script utilities for bulk requests of Host certs, CRL
checking etc. VOMS extended attributes and infrastructure for Role Based Access
Controls.
![Page 25: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/25.jpg)
25
Operations Model
Real support organizations
often play multiple roles
Lines represent communication paths and, in our model, agreements.
We have not progressed very far with agreements yet.
Gray shading indicates that OSG Operations composed of effort from all the support centers
![Page 26: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/26.jpg)
26
The OSG VO
A VO for individual researchers and users.
Managed by the OSG itself.
Learn how to use the Grid!
![Page 27: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/27.jpg)
27
In Summary…
A production grid is the product of a complex interplay of many forces:
Resource providers Users Software providers Hardware trends Commercial offerings Funding agencies Culture of all parties involved …
![Page 28: The Open Science Grid OSG](https://reader033.fdocuments.us/reader033/viewer/2022051416/56812f73550346895d94fb6a/html5/thumbnails/28.jpg)
28
Where do you learn more?
www.opensciencegrid.org
http://vdt.cs.wisc.edu
http://osg.ivdgl.org/twiki/bin/view