Building and Managing an Open Grid
description
Transcript of Building and Managing an Open Grid
Building and Managingan Open Grid
Ian Foster
Argonne National Laboratory
University of Chicago
http://www.mcs.anl.gov/~foster
Keynote talk at 8th IFIP/IEEE Intl Symp. on Integrated Network Management, March 26, 2003
2
[email protected] ARGONNE CHICAGO
Grids andIntegrated Network Management
2003: Production use in science & industry for distributed resource & service integration– Grids as management problem: definitely
– Grid technologies as management solution?
1995: I-WAY– Integration of HPC
resources for advanced scientific applications
– Management not quite on the agenda
3
[email protected] ARGONNE CHICAGO
Overview
The Grid: why and what Building an open Grid
– Open standards
– Open source
– Open infrastructure Summary
4
[email protected] ARGONNE CHICAGO
The Grid:On-Demand Access to Electricity
Time
Qua
lity,
eco
nom
ies
of s
cale
5
[email protected] ARGONNE CHICAGO
By Analogy, A Computing Grid
Decouple production and consumption– Enable on-demand access
– Achieve economies of scale
– Enhance consumer flexibility
– Enable new devices On a variety of scales
– Department
– Campus
– Enterprise
– Internet
6
[email protected] ARGONNE CHICAGO
Not Exactly a New Idea …
“The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility.”– Fernando Corbato and Robert Fano , 1966
“We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.”– Len Kleinrock, 1967
7
[email protected] ARGONNE CHICAGO
But, Things are Different Now Networks are far faster (and cheaper)
– Faster than computer backplanes “Computing” is very different than pre-Net
– Our “computers” have already disintegrated
– E-commerce increases size of demand peaks
– Entirely new applications & social structures We’ve learned a few things about software
8
[email protected] ARGONNE CHICAGO
Computing isn’t Really Like Electricity
I import electricity but must export data “Computing” is not interchangeable but highly
heterogeneous: data, sensors, services, … This complicates things; but also means that the sum
can be greater than the parts – Real opportunity: Construct new capabilities dynamically
from distributed services Raises three fundamental questions
– Can I really achieve economies of scale?
– Can I achieve QoS across distributed services?
– Can I identify apps that exploit synergies?
9
[email protected] ARGONNE CHICAGO
Why the Grid?(1) Revolution in Science
Pre-Internet– Theorize &/or experiment, alone
or in small teams; publish paper Post-Internet
– Construct and mine large databases of observational or simulation data
– Develop simulations & analyses
– Access specialized devices remotely
– Exchange information within distributed multidisciplinary teams
10
[email protected] ARGONNE CHICAGO
Why the Grid?(2) Revolution in Business
Pre-Internet– Central data processing facility
Post-Internet– Enterprise computing is highly distributed,
heterogeneous, inter-enterprise (B2B)
– Business processes increasingly computing- & data-rich
– Outsourcing becomes feasible => service providers of various sorts
11
[email protected] ARGONNE CHICAGO
New OpportunitiesDemand New Technology
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
15
[email protected] ARGONNE CHICAGO
Building an Open Grid
OpenStandards
OpenSource
OpenInfrastructure
16
[email protected] ARGONNE CHICAGO
Building an Open Grid
OpenStandards
OpenSource
OpenInfrastructure
OpenGrid
17
[email protected] ARGONNE CHICAGO
Building an Open Grid
OpenStandards
OpenSource
OpenInfrastructure
OpenGrid
18
[email protected] ARGONNE CHICAGO
Grids and Open Standards
Incr
ease
d fu
nctio
nalit
y,st
anda
rdiz
atio
n
Time
Customsolutions
Open GridServices Arch
GGF: OGSI, …(+ OASIS, W3C)
Multiple implementations,including Globus Toolkit
Web services
Globus Toolkit
Defacto standardsGGF: GridFTP, GSI
X.509,LDAP,FTP, …
App-specificServices
19
[email protected] ARGONNE CHICAGO
Open Grid Services Architecture
Service-oriented architecture– Key to virtualization, discovery,
composition, local-remote transparency Leverage industry standards
– Internet, Web services Distributed service management
– A “component model for Web services” A framework for the definition of
composable, interoperable services
“The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002
20
[email protected] ARGONNE CHICAGO
Realizing a Service-Oriented Architecture: How Do I
Create, name, manage, discover services? Render resources, data, sensors as services? Negotiate service level agreements? Express & negotiate policy? Organize & manage service collections? Establish identity, negotiate authentication? Manage VO membership & communication? Compose services efficiently? Achieve interoperability?
21
[email protected] ARGONNE CHICAGO
Web Services
XML-based distributed computing technology Web service = a server process that exposes typed
ports to the network Described by the Web Services Definition Language, an
XML document that contains– Type of message(s) the service understands & types of
responses & exceptions it returns
– “Methods” bound together as “port types”
– Port types bound to protocols as “ports” A WSDL document completely defines a service and
how to access it
22
[email protected] ARGONNE CHICAGO
Open Grid Services Infrastructure
Implementation
Servicedata
element
Other standard interfaces:factory,
notification,collections
Hosting environment/runtime(“C”, J2EE, .NET, …)
Servicedata
element
Servicedata
element
GridService(required)
Dataaccess
Lifetime management• Explicit destruction• Soft-state lifetime
Introspection:• What port types?• What policy?• What state?
Client
Grid ServiceHandle
Grid ServiceReference
handleresolution
23
[email protected] ARGONNE CHICAGO
Open Grid Services Infrastructure
GWD-R (draft-ggf-ogsi- gridservice-23) Editors:Open Grid Services Infrastructure (OGSI) S. Tuecke, ANLhttp://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI
I. Foster, ANLJ. Frey, IBMS. Graham, IBMC. Kesselman, USC/ISID. Snelling, Fujitsu LabsP. Vanderbilt, NASAFebruary 17, 2003
Open Grid Services Infrastructure (OGSI)
24
[email protected] ARGONNE CHICAGO
The OGSA Platform
OGSI
TransportProtocolHosting EnvironmentHosting Environment
Host. Env. & Protocol Bindings
OGSA Platform services: registry,authorization, monitoring, data
access, etc., etc.
More specialized &domain-specific
services
Models for resources &
other entitiesO
therm
odels
Environment-specificprofiles
Domain-specificprofiles
OGSAPlatform
GWD-R (draft-ggf-ogsa-platform-3) Editors:Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicagohttp://www.ggf.org/ogsa-wg D. Gannon, Indiana U.
25
[email protected] ARGONNE CHICAGO
Example:Reliable File Transfer Service
Performance
Policy
Faults
servicedataelements
Pending
FileTransfer
InternalState
GridService
Notf’nSource
Policy
interfacesQuery &/orsubscribe
to service data
FaultMonitor
Perf.Monitor
Client Client Client
Request and manage file transfer operations
Data transfer operations
26
[email protected] ARGONNE CHICAGO
Open Grid Service Architecture:Next Steps
Technical specifications– Open Grid Services Infrastructure is complete
– Security, data access, Java binding, common resource models, etc., etc., in the pipeline
Implementations and compliant products– OGSA-based Globus Toolkit v3, pyGlobus, …
– IBM, Avaki, Platform, Sun, NEC, Oracle, … Rich set of service defns & implementations
– Time to start on OGSI-compliant services!
27
[email protected] ARGONNE CHICAGO
OGSA Manageability
Management is a high-priority OGSA effort– GGF Common Management Model (CMM) WG
– Ellen Stokes (IBM) co-chair Goal:
– Define standard schema and interfaces for a manageable resource (modeled as a service)
And also:– Allow existing models to be used & exposed
– Leverage CIM schema when applicable
– Define how CMM interfaces can integrate with higher-level interfaces (e.g., provisioning)
28
[email protected] ARGONNE CHICAGO
Common Management Model A manageable resource is a Grid service, thus
– Global resource names: Grid service handles
– State data modeling + access: SDEs
– Lifetime management
– Service Group for grouping resources
– Interface definition language: WSDL Plus additional schema & operations
– Standard manageable resource SDE schema
– Interfaces for extensible lifecycle and relationship management
BaseManageableResource interface
29
[email protected] ARGONNE CHICAGO
Base Manageable Port Types
GridService
Locate Relationship LifecycleModel
BaseManageableResource
ServiceGroupHandleResolver
OGSA port types
CRM port types
30
[email protected] ARGONNE CHICAGO
CMM Schema
WSDL (open content model) & XSD describe resource’s manageable attributes (as SDEs)
Models are CIM-based where applicable– Re-factored for service efficiency; appropriate for
higher-level management applications
– Additional XML attributes: change control, measuring, lifecycle
– Additional XML data types CIM as basis for schemas: but may modify
– E.g. use constructs from XML/XSD where similar ones exist in CIM
31
[email protected] ARGONNE CHICAGO
“Use CIM Models Where Applicable”
Class is port type, properties of class are port type service data, methods of class are port type operations– Some refactoring of classes -> portTypes
Express in WSDL/GSDL as grid service– Managed resource port type from which other
resource port types are derived
– Mix in the base GridService port type
– Mix in other CMM port types as needed (Identity, relationship, lifecycleState)
32
[email protected] ARGONNE CHICAGO
CMM Bindings and Operations
A resource service may have more than one binding to access that resource’s manageability information
A resource may support more than one service implementation– e.g., Native, SNMP, CIM, RMC, etc
A resource can still be accessed (outside OGSA) using its existing implementation, resource definition, and access interface, e.g., SNMP, CIM, etc.
33
[email protected] ARGONNE CHICAGO
Basic Manageability Port Types
Needed to manage the resource– lifecycleModel
– relationship
– locate Others defined
– From CIM schema, e.g., opererational operations (start/stop/resume/pause)
– By OGSO spec, e.g. grid service, notification
– By other OGSA components, e.g. policy, logging/metering
34
[email protected] ARGONNE CHICAGO
lifecycleModel Port Type:A Container for Lifecycle States
There may be multiple models, but only one for a given resource’s port type
Example: Get/set resource’s lifecycle state – down, starting, up, stopping, failed
– Each state has additional info, e.g.,> up state: idle, busy, degraded
Down Starting
UpStopping
Failed
35
[email protected] ARGONNE CHICAGO
Relationships & Dependencies
Relationships describe existence and type of connections between resource instances– Relationships are discovered through the
relationship port type and its relatedResource service data element
– Relationship port type allows a view of relationships as they are known by the resources at each end of the relationship
Set of predefined relationship types– Hosts, Contains, Federates, Aggregates, Uses,
Implements
36
[email protected] ARGONNE CHICAGO
Locate Port Type
Provided to help in the location of manageable resources of a given type
Contains two service data elements– serviceGroupType, port type that provides the
ServiceGroup function for manageable resources of this type
– searchProperty, zero or more service data elements that are likely to be used for searching for a manageable resource, and are thus worth caching in the manageable resource registry
37
[email protected] ARGONNE CHICAGO
Building an Open Grid
OpenStandards
OpenSource
OpenInfrastructure
OpenGrid
38
[email protected] ARGONNE CHICAGO
Open Source
Encourage adoption of standards by reducing barriers to entry– Overcome new technology Catch-22
Enable broad Grid technology ecosystem– Key to international cooperation
– Key to science-commerce partnership Jumpstart Grid industry and allow vendors to
focus on value-add– E.g., IBM, Avaki use GT3; Platform Globus
– Open source is industry friendly!
39
[email protected] ARGONNE CHICAGO
Globus Toolkit v3 (GT3)Open Source OGSA Technology
Implement core OGSI interfaces Support primary GT2 interfaces
– High degree of backward compatibility Multiple platforms & hosting environments
– J2EE, Java, C, .NET, Python New services
– SLA negotiation (GRAM-2), registry, replica location, community authorization, data, …
Growing external contributions & adoption
40
[email protected] ARGONNE CHICAGO
Globus Toolkit Contributors: GT2
Grid Packaging Technology (GPT) NCSA Persistent GRAM Jobmanager Condor GSI/Kerberos interchangeability Sandia Documentation NASA, NCSA Ports IBM, HP, Sun, SDSC, … MDS stress testing EU DataGrid Support IBM, Platform, UK eScience Testing and patches Many! Interoperable tools Many! $$ DARPA, DOE, NSF, NASA, Microsoft, EU
41
[email protected] ARGONNE CHICAGO
Globus Toolkit Contributors: GT3
Replica location service EU DataGrid Python hosting environment LBNL Data access & integration UK eScience Data mediation services SDSC Tooling, Xindice, JMS IBM ... ... ...
42
[email protected] ARGONNE CHICAGO
Open Source is More than Software!
Success of Grid concept demands effective community mechanisms for coordinated– R&D for core technologies
– Testing, packaging, documentation
– Support and training of user communities All three must become collaborative activities
– Based on standards
– Centered on a common code base
– Supported by appropriate tools
– United by appropriate processes & governance
43
[email protected] ARGONNE CHICAGO
For Example:Distributed Test Facility
(Proposed activity for NSF GRIDS Center: grids-center.org)
44
[email protected] ARGONNE CHICAGO
Building an Open Grid
OpenStandards
OpenSource
OpenInfrastructure
OpenGrid
45
[email protected] ARGONNE CHICAGO
The Gridas Enabler of 21st Century Science
Entirely new approaches to enquiry based on– Deep analysis of huge quantities of data
– Interdisciplinary collaboration
– Large-scale simulation
– Smart instrumentation Enabled by an infrastructure that enables
access to, and integration of, resources & services without regard for location
46
[email protected] ARGONNE CHICAGO
Grid Infrastructure Broadly deployed services in support of fundamental
collaborative activities– Formation & operation of virtual organizations
– Authentication, authorization, discovery, … Services, software, and policies enabling on-demand
access to critical resources– Computers, databases, networks, storage, software
services,… Operational support for 24x7 availability Integration with campus and commercial
infrastructures
47
[email protected] ARGONNE CHICAGO
Tier0/1 facility
Tier2 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
Other link
Tier3 facility
The Foundations are Being Laid
Cambridge
Newcastle
Edinburgh
Oxford
Glasgow
Manchester
Cardiff
Soton
London
Belfast
DL
RAL Hinxton
48
[email protected] ARGONNE CHICAGO
Summary
An open Grid technology has been created– Standards, software, infrastructure
– All three must be collaborative activities We are now scaling this technology base, to
– Meet the needs of the science community
– Achieve synergy between open & commercial, scientific & industrial contributors
Management is a crucial part of this effort– One might argue, “Grid = management”
– We need help from your community