Ted Hesselroth Nordugrid 2007 September 24-28, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San...

Ted Hesselroth Nordugrid 2007 September 24-28, 2007

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

The Open Science Grid

Ted Hesselroth

Fermilab

Slide attribution: Ruth Pordes, Miron Livny, Frank

Wuerthwein, Paul Avery, Kent Blackburn,CNGrid



Map of OSG Sites

OSG is a grid organization funded by a SciDAC-2/NSF grant 30 million dollars over five years. 33 FTE. 77 compute and 15 storage elements



OSG Mission Statement

Practical support for end-to-end community systems in a

heterogeneous gobal environment to

Transform compute and data intensive science through

a national cyberinfrastructure that includes from the smallest to the largest organizations.



OSG Goals – Use of Existing Resources

● Enable scientists to use and share a greater % of

available compute cycles.

● Help scientists to use distributed systems, storage,

processors, and software with less effort.

● Enable more sharing and reuse of software and

reduce duplication of effort through providing effort in

integration and extensions.

● Establish “open-source” community working together

to communicate knowledge and experience.erience

and also overheads for new participants.



OSG - forming communities

Software DevelopersSites

Experiments (VOs)

(e.g Condor, Globus, SRM, …)

(e.g BNL, FNAL, LBNL, SLAC, LHC-T2s, DISUN, …)

OSG enables communityformation to solve compute and data intensive scientificproblems.

(USCMS, USATLAS, CDF, D0, LIGO, BioTech, NanoTech, …)

Coordinating role



Principal Science Drivers

● High energy and nuclear physics 100s of petabytes (LHC) 2008 Several petabytes 2005

● LIGO (gravity wave detector) 0.5 - several petabytes 2002

● Digital astronomy 10s of petabytes 2009 10s of terabytes 2001

● Other sciences coming forward Bioinformatics (10s of petabytes) Nanoscience Environmental Chemistry Applied mathematics Materials Science?

Data

gro

wth

Com

mu

nit

y g

row

th2007

2005

2003

2001

2009



The Evolution of the OSG

1999 2000 2001 2002 20052003 2004 2006 2007 2008 2009

PPDG

GriPhyN

iVDGL

Trillium Grid3 OSG(DOE)

(DOE+NSF)(NSF)

(NSF)

Campus, regional grids

LHC OpsLHC construction, preparation

LIGO operation LIGO preparation

European Grid + Worldwide LHC Computing Grid

DOE Science Grid (DOE)



VOs in OSGSelf-Operated Research VOs Campus GridsCollider Detector at Fermilab Georgetown University GridCompact Muon Solenoid Grid Laboratory of Wisconsin

Grid Research and Education Group of IowaD0 Experiment at Fermilab University of New York at Buffalo

Fermi National Accelerator Center

Geant 4 Software Toolkit Regional GridsNew York State Grid

Great Plains NetworkNorthwest Indiana Computational Grid

United States Atlas CollaborationSolenoidal Tracker at RHIC OSG-Operated VOs

EngagementOpen Science GridOSG Education ActivityOSG Monitoring and Operations

CompBioGrid*

Dark Energy Survey*Functional Magnetic Resonance Imaging*

Genome Analysis and Database Update*Internation Linear ColliderInteractions in Understanding the Universe Initiative

Distributed Organization for Scientific and Academic Research

Laser Interferometer Gravitational-Wave Observatory*nanoHUB Network for Compuational Nanotechnology*Sloan Digital Sky Survey*

Structural Biology Grid*Grid Resources for Advanced Science and Engineering*Mixed Apparatus for Radar Investigation... (mariachi)*

*=non-physics



Example Campus Grid:

Grid Laboratory of Wisconsin (GLOW)



Institutions

Sites Florida State University University of NebraskaKansas State University LBNL University of Chicago

University of Michigan University of Iowa University of Notre Dame U. of California RiversideAcademia Sinica Hampton University Penn State University U. of California San DiegoBrookhaven National Lab UERJ Brazil Oklahoma University University of FloridaBoston University Iowa State University SLAC U of Illinois ChicagoCinvestab, Mexico City Indiana University Purdue University University of New MexicoCaltech Rice University U. of Texas Arlington

Lousiana University Southern Methodist U. University of VirginiaUniversity of Sao Paolo U. of Wisconsin Madison

Florida International U. McGill University Wayne State University U. of Wisconsin MilwaukeeFermilab MIT Texas Tech University Vanderbilt University

Others contributing staff

Columbia University Cornell University U of Southern California UNC Chapel HillUIUC

University of Arkansas*

Lehigh University*Clemson University*Dartmouth University* Lousiana Tech*

Green=Contributing staff*=non-physics



China National Grid (CNGrid)

● 17 TFlops



● Many resources are owned or statically allocated to one user community.

The institutions which own resources typically have ongoing relationships with (a few) particular user communities (VOs)

The remainder of an organization’s available resources can be “used by everyone or anyone else”.

organizations can decide against supporting particular VOs. OSG staff are responsible for monitoring and, if needed,

managing this usage.

Our challenge is to maximize good - successful - output from the whole system.

Use of Existing Resources



● Increased usage of CPUs and infrastructure alone (ie cost of processing cycles) is not the persuading cost-benefit value.

The benefits come from reducing risk in and sharing support for large, complex systems which must be run for many years with

a short life-time workforce. Opportunity and flexibility to distribute load and address peak needs. Savings in effort for integration, system and software support. Maintainance of an experienced workforce in a common system Lowering the cost of entry to new contributors. Enabling of new computational opportunities to communities that would

not otherwise have access to such resources.

Benefits to Sites



The “don’t”s and “do”s of OSG

● The OSG Facility does not – “Own” any compute (processing, storage and communication) resources “Own” any middleware Fund any site or VO administration/operation personel

• The OSG Facility does –– Help sites join the OSG facility and enable effective guaranteed and

opportunistic usage of their resources (including data) by remote users– Help VOs join the OSG facility and enable effective guaranteed and

opportunistic harnessing of remote resources (including data) – Define interfaces which people can use.– Maintain and supports an integrated software stack that meets the needs

of the stakeholders of the OSG consortium– Reach out to non-HEP communities to help them use the OSG– Train new users, administrators, and software developers



What Can the OSG Offer? ● Middleware

Packaging Testing Support Security operations

● Organizational support The OSG Consortium (brings together the stakeholders) The OSG Facility (brings together resources and users)

● Technical Support Troubleshooting distributed computing technologies

● Extensions Software capbilities needed by OSG

● Engagment Consultation on OSG participation

● Instruction Workshops Documentation



OSG Project Effort

Roughly 2/3 of leadership positions filled from outside HEP !



Benefits to HEP thus far

● LHC– Middleware stack for the LHC distributed computing systems

of USATLAS and USCMS– Strong partner to negotiate technical and operational

problems with EGEE and Nordugrid.– Framework for integrating “Tier-3” resources.

• Tevatron and other FNAL based HEP– CDF: MC production on OSG– D0: reprocessing on OSG– Other HEP benefit via FNAL campus grid

● Other HEP starting to show interest as well.



● D0’s own resources are committed to the processing of newly acquired data and analysis of the processed datasets.

● In Nov ‘06 D0 asked to use 1500-2000 CPUs for 2-4 months for re-processing of an existing dataset (~500 million events) for science results for the summer conferences in July ‘07.

● The Executive Board estimated there were currently sufficient opportunistically available resources on OSG to meet the request; We also looked into the local storage and I/O needs.

D0 Reprocessing



D0 Reprocessing OSG Portion



LIGO: Search for Gravity Waves

● LIGO Grid

6 US sites

3 EU sites (UK & Germany)

* LHO, LLO: LIGO observatory sites* LSC: LIGO Scientific Collaboration

Cardiff

AEI/Golm •

Birmingham•



Sloan Digital Sky Survey: Mapping the Sky



Astronomy Experiences on the GridAstronomy Experiences on the Grid

● Experience tells us that Grid is more suitable for CPU Intensive Jobs …

achieve parallelism … more jobs… finish sooner

● Running locally would limit the number of jobs run simultaneously

● On OSG, can run several run-rerun and camcols within a run-rerun in parallel

● Current Workflow also will facilitate further analysis

Grid not very

happy

Ideal for Grid

Grid Match

10-15 per

day ?

800-1200

per day

Avg. Rate of Job

Completion

12 Kilobyte

s

2 Megabyt

es

Data Output/J

ob

9 Gigabyt

es

1 Megabyt

e

Data Input/Jo

b

180~50000Total No. of Jobs

NEO

Data&CPData&CPU U

IntensiveIntensive

Quasar

Spectra

CPU CPU IntensiveIntensive



Engagement

● Currently main stakeholders are from physics - US LHC experiments, LIGO, STAR experiment, the Tevatron Run II and Astrophysics experiments

● Active “engagement” effort to add new domains and resource providers to the OSG consortium –

Rosetta at Kulhman Laboratory Weather Research and Forecast (WRF) model nanoHub applications – BioMoca and nanoWire Chemistry at Harvard Molecular Mechanics (CHARMM)



Rosetta Protein Folding Application

● “What impressed me most was how quickly we were able to access the grid and start using it. We learned about it [at RENCI], and we were running jobs about two weeks later,” Brian Kuhlman, PI.

● 3,000 CPU hours per protein● CASP similar protein: 3 hours on the 114 teraflops IBM Blue Gene Watson machine



Genome Analysis and Database Update system

● Runs across TeraGrid and OSG. Uses the Virtual Data System (VDS) workflow & provenance.

● 3.1 million protein sequences, 93,000 jobs.● “During the last run in January (2006), GADU VO jobs had access to only about

8-10 OSG sites and were not authenticated by a large number of sites. With the help of the GOC, we are working on getting more sites to authenticate GADU jobs.” Dinanath Sulakhe, Argonne National Laboratory



Integrated Database

Integrated Database Includes: Parsed Sequence Data and

Annotation Data from Public web sources.

Results of different tools used for Analysis: Blast, Blocks, TMHMM, …

GADU using GridApplications executed on Grid as workflows

and results are stored in integrated Database.

GADU Performs: Acquisition: to acquire Genome Data

from a variety of publicly available databases and store temporarily on the file system.

Analysis: to run different publicly available tools and in-house tools on the Grid using Acquired data & data from Integrated database.

Storage: Store the parsed data acquired from public databases and parsed results of the tools and workflows used during analysis.

Bidirectional Data Flow

Public DatabasesGenomic databases available on the web.Eg: NCBI, PIR, KEGG, EMP, InterPro, etc.

Applications (Web Interfaces) Based on the Integrated Database

PUMA2Evolutionary Analysis of

Metabolism

ChiselProtein Function Analysis

Tool.

TARGETTargets for Structural analysis

of proteins.

PATHOSPathogenic DB for

Bio-defense research

PhyloblocksEvolutionary analysis of

protein families

TeraGrid OSG DOE SG

GNARE – Genome Analysis Research Environment

Services to Other Groups

•SEED(Data Acquisition)

•Shewanella Consortium

(Genome Analysis)Others..

Bioinformatics: GADU / GNARE



NanoHub

● BioMOCA (Biology Monte Carlo) transport Monte Carlo tool.

● Written at Network for Computational Nanotechnology

● PI: Umberto Ravaioli, UIUC

Ion transfer in artificial membranes

● Job run is 8-40 days



Network Collaboration

Internet 2

National Lambda Rail

Ultralight



OSG Activities● Facility

Software Operations Deployment Integration Troubleshooting Engagement

● Security● Education● Extensions

Middleware Improvement Workload Management Scalability Testing Tools and prototypes

● User Support● Admin



The Software Stack



What is the VDT?● A collection of software

Grid software: Condor, Globus and lots more Virtual Data System: Origin of the name “VDT” (toolkit) Utilities: Monitoring, Authorization, Configuration Built for >10 flavors/versions of Linux

● Automated Build and Test: Integration and regression testing.

0

5

10

15

20

25

30

35

40

45

Jan-02May-02Sep-02Jan-03May-03Sep-03Jan-04May-04Sep-04Jan-05May-05Sep-05Jan-06May-06Sep-06

Number of major components

VDT 1.1.x VDT 1.2.x VDT 1.3.x

VDT 1.0Globus 2.0bCondor-G 6.3.1

VDT 1.1.3, 1.1.4 & 1.1.5, pre-SC 2002

VDT 1.1.8Adopted by LCG

VDT 1.1.11Grid2003 VDT 1.2.0

VDT 1.3.0

VDT 1.3.9For OSG 0.4

VDT 1.3.11Current ReleaseMoving to OSG 0.6.0

VDT 1.3.6For OSG 0.2

●An easy installation: Push a button, everything just works. Quick update processes.

●Responsive to user needs: process to add new components based on

community needs.●A support infrastructure:

front line software support, triaging between users and software

providers for deeper issues.



How we get to a Production Software Stack

Input from stakeholders and OSG directors

VDT Release

OSG Integration Testbed Release

OSG Production Release

Test on OSG Validation Testbed



Troubleshooting

● GOC Tickets

Assigns responsible

Interoperability with EGEE

● Mailing Lists

● “Office Hours”



OSG Storage Activities

● Support for Storage Elements in OSG (4 FTE)

dCache

Bestman

● Validation

Tier2-level test stand

With UCSD Tier2

● Packaging

Installation scripts

Through VDT



OSG Storage Activities

● Support

Mailing list

● Tools

For site administrators

Will collect existing tools

● Extensions

Space reservation file cleaner

dCache logging

Ted Hesselroth Nordugrid 2007 September 24-28, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San...

Documents

Transcript of Ted Hesselroth Nordugrid 2007 September 24-28, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San...