Grid Computing & Web Services:A Natural Partnership

Ian Foster

Mathematics and Computer Science Division

Argonne National Laboratory

and

Department of Computer Science

The University of Chicago

Address at 2nd US-Hungarian Workshop on Cluster and Grid Computing, Budapest, Hungary, February 6, 2002

Dave Angulo

Department of Computer Science

The University of Chicago

and

Mathematics and Computer Science Division

Argonne National Laboratory

dangulo@cs.uchicago.edu University of Chicago

Partial Acknowledgements Open Grid Services Architecture work is performed

by– Ian Foster, Globus Co-PI @ Argonne/UofC– Carl Kesselman, Globus Co-PI @ USC/ISI– Steve Tuecke, Globus Toolkit Architect @ANL– Jeff Nick, Steve Graham, Jeff Frey @ IBM

Globus Toolkit R&D involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org)

Strong collaborations with many outstanding EU, UK, US Grid projects

Support from DOE, NASA, NSF, Microsoft

dangulo@cs.uchicago.edu University of Chicago

Globus Toolkit: Evaluation (1)

Good technical solutions for key problems, e.g.– Authentication and authorization

– Resource discovery and monitoring

– Reliable remote service invocation

– High-performance remote data access This + good engineering is enabling progress

– Good quality reference implementation, multi-language support, interfaces to many systems, large user base, industrial support

– Growing community code base built on tools

dangulo@cs.uchicago.edu University of Chicago

Globus Toolkit: Evaluation (2) Protocol deficiencies, e.g.

– Heterogeneous basis: HTTP, LDAP, FTP

– No standard means of error propagation Significant missing functionality, e.g.

– Databases, sensors, instruments

– Programming tools: workflow, …

– Virtualization of end systems (hosting envs.) Little work on total system properties, e.g.

– Dependability, end-to-end QoS, …

– Reasoning about system properties

dangulo@cs.uchicago.edu University of Chicago

“Web Services” Increasingly popular standards-based framework for

accessing network applications– W3C standardization; Microsoft, IBM, Sun, others

WSDL: Web Services Description Language– Interface Definition Language for Web services

SOAP: Simple Object Access Protocol– XML-based RPC protocol; common WSDL target

WS-Inspection– Conventions for locating service descriptions

UDDI: Universal Desc., Discovery, & Integration – Directory for Web services

dangulo@cs.uchicago.edu University of Chicago

Transient Service Instances “Web services” address discovery & invocation

of persistent services In Grids, must also support transient service

instances, created/destroyed dynamically– E.g., to manage eBusiness workflow, video

conference, or distributed data analysis Significant implications for how services are

managed, named, discovered, and used– In fact, much of our work is concerned with the

management of service instances

dangulo@cs.uchicago.edu University of Chicago

Open Grid Services Architecture Service orientation to virtualize resources From Web services:

– Standard interface definition mechanisms: multiple protocol bindings, multiple implementations, local/remote transparency

Building on Globus Toolkit:– The Grid service defines standard semantics for service

interactions– Factory, registry, and mapper services– Reliable and secure transport

Multiple hosting targets: J2EE, .NET, “C”, etc.

dangulo@cs.uchicago.edu University of Chicago

OGSA Service Model System comprises (a typically few) persistent services

& (potentially many) transient services All services adhere to specified Grid service interfaces

and behaviors– Reliable invocation, lifetime management, discovery,

authorization, notification, upgradeability, concurrency, manageability

Interfaces for managing Grid service instances– Factory, registry, mapper

Heavily leverage Globus Toolkit technology=> Reliable secure mgmt of distributed state

dangulo@cs.uchicago.edu University of Chicago

The Grid Service A (potentially transient) Web service with specified

interfaces & behaviors, including– Creation (Factory)

– Global naming (GSH) & references (GSR)

– Lifetime management

– Registration & Discovery

– Authorization

– Notification

– Concurrency

– Manageability

dangulo@cs.uchicago.edu University of Chicago

Factory

A Grid service with Factory interface can be requested to create a new Grid service instance– Reliable creation (once-and-only-once)

Create operation can be extended to accept Grid-service-specific creation parameters

Returns a Grid Service Handle (GSH)– A globally unique URL

– Uniquely identifies the instance for all time

– Based on name of a home mapper service

dangulo@cs.uchicago.edu University of Chicago

Mapper A GSH is a stable name for a Grid service, but does

not allow client to actually communicate with the Grid service

A Grid Service Reference (GSR) is a WSDL document that describes how to communicate with the Grid service– Contains protocol binding, network address, …

– May expire (I.e. GSR information may change) The Mapper interface allows a client to map from a

GSH to a GSR– http get on GSH also returns a GSR

dangulo@cs.uchicago.edu University of Chicago

Lifetime Management GS instances created by factory or manually;

destroyed explicitly or via soft state– Negotiation of initial lifetime with Factory

SoftStateDestruction interface supports– GetTerminationTime message for inquiry

>Notification interface also allows for lifetime notification

– SetTerminationTime message for keepalive Soft state lifetime management avoids

– Explicit client teardown of complex state

– Resource “leaks” in hosting environments ExplicitDestruction interface also available

dangulo@cs.uchicago.edu University of Chicago

Discovery A Grid service instance may maintain a set of

service information– XML fragments encapsulated in standard <name,

type, TTL-info> containers Discovery interface allows clients to query the

Grid service instance for this information– Query operation, plus supporting operations

> Extensible query language support

See also Notification interfaces– Allows notification of service existence and about

service information

dangulo@cs.uchicago.edu University of Chicago

Registry

The Registry interface may be used to discover a set of Grid service instances– Returns a WS-Inspection document containing the

GSHs of a set of Grid services

– Also returns policy associated with the set

– Also available through Discovery interface The RegistryManagement interface allows for

soft-state registration of a Grid service– A set of Grid services can periodically register their

GSHs into a registry service, to allow for discovery of services in that set

dangulo@cs.uchicago.edu University of Chicago

Authorization

Protocol binding handles authentication during invocation of Grid service operation– Gives service URI for authenticated subject

Grid service instance should apply authorization policy on all operations– May be site-, service-, instance-, etc., specific

OGSA defines standard interfaces for remote management of access control policy– OperationAuthorizationManagement

– SubjectEquivalency

dangulo@cs.uchicago.edu University of Chicago

Notification Interfaces NotificationSource for client subscription

– One or more notification generators> Generates notification message of a specific type

> Typed interest statements: E.g., Filters, topics, …

> Supports messaging services, 3rd party filter services, …

– Soft state subscription to a generator NotificationSink for asynchronous delivery of

notification messages A wide variety of uses are possible

– E.g. Dynamic discovery/registry services, monitoring, application error notification, …

dangulo@cs.uchicago.edu University of Chicago

Use of Web Services (1)

A Grid service interface is a WSDL portType A Grid service definition is a WSDL extension

(serviceType) containing:– A set of one or more portTypes supported by

the service

– portType & serviceType compatibility statements, to support upgradability

> For discovery of compatible services when interfaces are upgraded

– Implementation version information

dangulo@cs.uchicago.edu University of Chicago

Use of Web Services (2)

A GSR is a WSDL document with extensions:– Extension to service element to reference

serviceType

– Service element extensions to carry the GSH, and the expiration time of the GSR

A GSH is an URL, with the following properties:– Globally unique for all time

– http get on GSH + “.wsdl” returns GSR

– Can derive GSH to Mapper from it Registry returns WS-Inspection documents

dangulo@cs.uchicago.edu University of Chicago

Using OGSAto Construct Grid Environments

Factory RegistryService

FactoryH2R

Mapper

ServiceService Service ...

...

(a) Simple HostingEnvironment

Factory RegistryService

FactoryH2R

Mapper

ServiceService Service ...

...

F R

F M

SS S

F R

F M

SS S

(b) Virtual HostingEnvironment

E2EFactory

E2E Reg

E2E H2RMapper

...

F1

R

M

SS S

F2

R

M

SS S

E2E S E2E S E2E S

(c) Compound Services

In each case, Registry handle is effectively the uniquename for the virtual organization.

dangulo@cs.uchicago.edu University of Chicago

OGSA and the Globus Toolkit Technically, OGSA enables

– Refactoring of protocols (GRAM, MDS-2, etc.)—while preserving all GT concepts/features!

– Integration with hosting environments: simplifying components, distribution, etc.

– Greatly expanded standard service set Pragmatically, we are proceeding as follows

– Develop open source OGSA implementation> Globus Toolkit 3.0; supports Globus Toolkit 2.0 APIs

– Partnerships for service development

– Also expect commercial value-adds

dangulo@cs.uchicago.edu University of Chicago

Globus Toolkit Refactoring

Grid Security Infrastructure (GSI)– Used in Grid service network protocol bindings

Meta Directory Service 2 (MDS-2)– Native part of each Grid service:

> Discovery, Registry, RegistryManagement, Notification

Grid Resource Allocation & Mngt (GRAM)– Gatekeeper -> Factory for job mgr instances

GridFTP– Refactor control channel protocol

Other services refactored to used Grid services

dangulo@cs.uchicago.edu University of Chicago

Timeline

Summer 2002 – Alpha releases of high-level Grid Services

Late 2002, Early 2003 – Alpha release of new core Grid Services (MDS, GRAM, GridFTP)

dangulo@cs.uchicago.edu University of Chicago

Migration Paths Globus ToolkitTM evolutionary in nature

– Toolkit implementation may change– Underlying model of Grid Computing remains the

same– Capabilities of future Toolkits will be superset of

today’s Toolkit New implementations integrate better with existing

commodity technologies In cases of radical departure from current

implementations, migration paths will be provided– possibly maintain compatible APIs– possibly create gateways to today’s protocols

dangulo@cs.uchicago.edu University of Chicago

Summary:Evolution of Grid Technologies

Initial exploration (1996-1999; Globus 1.0)– Extensive appln experiments; core protocols

Data Grids (1999-??; Globus 2.0+)– Large-scale data management and analysis

Open Grid Services Architecture (2001-??, Globus 3.0)– Integration w/ Web services, hosting environments,

resource virtualization

– Databases, higher-level services Radically scalable systems (2003-??)

– Sensors, wireless, ubiquitous computing

dangulo@cs.uchicago.edu University of Chicago

Summary The Grid problem: Resource sharing & coordinated

problem solving in dynamic, multi-institutional virtual organizations

Grid architecture: Protocol, service definition for interoperability & resource sharing

Globus Toolkit a source of protocol and API definitions—and reference implementations– And many projects applying Grid concepts (& Globus

technologies) to important problems Open Grid Services Architecture represents (we

hope!) next step in evolution

dangulo@cs.uchicago.edu University of Chicago

For More Information The Globus Project™

– www.globus.org Grid architecture

– www.globus.org/research/papers/anatomy.pdf

Open Grid Services Architecture (soon)– www.globus.org/research/

papers/ogsa.pdf

– www.globus.org/research/papers/gsspec.pdf