State-Wide Collaborative Grid Computing Course

Barry WilkinsonDepartment of Computer Science

University of North Carolina at Charlotteand

Mark HollidayDepartment of Mathematics and Computer Science

Western Carolina University

Teaching and Learning with Technology Conference March 30th – April 1st, 2005, Raleigh, NC.

Participating Sites

Talk Outline

Brief introduction to Grid Computing.

Software Tools.

Grid Computing Course.

Grid Computing

Using geographically distributed interconnected computers for high performance computing and/or for resource sharing.

The grid virtualizes heterogeneous geographically disperse resources

From "Introduction to Grid Computing with Globus," IBM Redbooks

Virtual Organizations

Grid computing offers

potential of virtual organizations:– groups of people, both geographically and

organizationally distributed, working together on a problem, sharing computers AND other resources such as databases and experimental equipment.

Crosses multiple administrative domains.

Shared Resources

Can share much more than just computers:

Storage Sensors for experiments at particular sites Application Software Databases Network capacity, …

Interconnections and Protocols

Focus now on: using standard Internet

protocols and technology, i.e. HTTP, SOAP, web services, etc.,

Grid computing standards around web services.

Applications

Originally e-Science applications– Computational intensive

• Not necessarily one big problem but a problem that has to be solved repeatedly with different parameters.

– Data intensive.– Experimental collaborative projects

Now also e-Business applications to improve business models and practices.

History

Began in mid 1990’s with experiments using computers at geographically dispersed sites.

Seminal experiment – “I-way” experiment at 1995 Supercomputing conference (SC’95), using 17 sites across the US running:– 60+ applications.– Existing networks (10 networks).

Grid Networks and Projects

Numerous very high performance computing projects developed in late 1990’s and 2000’s.

Examples: USA TeraGrid, UK e-Science Grid, and many others.

TeraGrid

UK e-Science Grid

Software - Globus Project

Open source software toolkit developed for grid computing.

Roots in I-way experiment. Work started in 1996. Four versions developed to present time. Reference implementations of grid computing

standards. Defacto standard for grid computing.

GSI (Grid Security Infrastructure)– Grid security.

MDS (Monitoring and discovery Service)– Interface to system and service information.

GRAM (Grid Resource Allocation Manager) – Remote job submission and control.

GridFTP– Secure data transfer.

GlobusKey Components

From “Globus Toolkit 4 Tutorial,” MCNC Jan-Feb, 2005, Pawel Plaszczak and Bogdan Lobodzinski, Gridwise Technologies.

2. discover resource

3. submit job

4. transfer data

1. secure environmentGSI

GRAM

MDS

GridFTP

Globus Toolkit: Recent History

GT2 (2.4 released in 2002)– GRAM, MDS, GridFTP, GSI.

GT3 (3.2 released mid-2004): redesign– OGSA (Open Grid Service Architecture) - OGSI (Open

Grid Services Infrastructure) based.– Introduced “Grid services” as an extension of web

services.– OGSI now abandoned.

GT4 (release for April 2005): redesign– WSRF (Web service Resource Framework) based.– Grid standards merged with Web services.

Supercomputing 2003 Demonstration

We used Globus version 2.4 in a Supercomputing 2003 demo organized by the University of Melbourne.

21 countries involved, numerous sites.

A re-implementation based upon the Open Grid Service Architecture (OGSA) standard.

We used version 3.2 for the Fall 2004 course.

Underlying implementation of version 3.x used OGSI Open Grid Service Infrastructure), which was not embraced by the community.

Version 3

Version 4

Currently under development to be released early 2005.

OGSA kept but OGSI abandoned in favor of new implementation standards based around web services. (Version 3 used “extended” web services)

Web Services-Based Grid Computing

Grid Computing is now strongly based upon web services.

Large number of newly proposed grid computing standards:– WS-Resource Framework– WS-Addressing– etc., etc. …. .

Grid Computing Course (Fall 2004)

Originated from WCU on NCREN network. Broadcast to:– UNC-Wilmington– NC State University– UNC-Asheville– UNC-Greensboro– Appalachian State University– NC Central University– Cape Fear Community College– Elon University– UNC-Charlotte*

Instructors: – Barry Wilkinson and

Clayton Ferner (UNC-Wilmington) Several faculty helped at various sites 43 students

(*canceled)

Participating Sites

Level and Prerequisites

Listed as an undergraduate course but can be taken for graduate credit. (Graduate students to do more work.)

Preferably have programming skills in Java on a Linux system.

Topics Review of Internet technologies Introduction to grid computing Web services Grid services (OGSA) Security, Public Key Infrastructure Globus 3.2 Condor-G MPI and grid-enabled MPI UNC-W workflow editor and other GUI tools Grid computing applications

5 grid computing assignments. Software from NMI package.

Grid computing Virtual organizations, grid projects in the US and around the world, grid challenges.

Internet IP addresses, HTTP, URL, HTTP, Technologies XML, Telnet, FTP, SSL.Web Services Service-Oriented Architecture,

service registry, XML documents, XML schema, namespaces, SOAP, XML/SOAP examples, WSDL, WSDL to/from code.

Assignment 1 “Simple” Web service Java programming assignment. Tomcat environment, axis, JWS facility.

Weeks 1 - 3

Weeks 3 - 4

Grid Service Concepts, differences to Web services, stateful/ stateless/transient/non-transient, Open Grid Services Architecture (OGSA), OGSI, grid service factory, Web Services Resource Framework (WSRF)

Assignment 2 "Simple" grid service Java programming assignment. Globus 3.2 environment.Tools: ant.

Weeks 4 - 6

Security Secure connection, authorization requirements, symmetric and asymmetric key cyptography, non-repudiation, digital signatures, X 509 certificates, certificate authorities.

Globus: Basic structure, grid service container, service browser, Resource Allocation Manager, job submission with managed-job-globusrun, Grid Security Infrastructure, Globus certificates, simpleCA, proxies, creating a proxy,

Assignment 3 Submitting a Job to the Grid, GT3 mangaged-job-globusrun, job specified in RSL-2 (XML file).

Weeks 6 - 7

Resource Master Managed Job Factory

Management Service, Resource Specification Language (RSL and RSL-2). Information Directory Services, LDAP, resource discovery.

Schedulers and Condor, submit descriptionresource brokers file, DAGMan, checkpointing,

ClassAd, Condor-G.

Assignment 4 Submitting a Condor-G Job

Weeks 7 - 8High performance Grand challenge problems,computing (HPF) parallel computing, potential speed-

up, types of parallel computers.Parallel Techniques suitable for a Grid,Programming embarrassingly parallel

computations, Monte Carlo, parameter studies, sample "big" problems, gravitational N-body problem

 Cluster Computing Basic message passing techniques, programming models, synchronous message passing, asynchronous message passing, message tags, collective routines.

Weeks 8 - 9 

MPI Process creation, communicators, unsafe message passing, MPI routines, running MPI program on a cluster.

 Grid-enabled MPI MPI-G2 internals, mpirun command, RSL script

Assignment 5 Running a simple MPI-G2 program (not done Fall 2004).

Grid Portal Design

“A web-based application server enhanced with the necessary software to communicate to grid services and resources”

“Provides application scientist a customized view of software and hardware resources from a web browser”

Weeks 10 to 11

OGCEGrid

Workflow Technique

Functional decomposition - dividing problem into separate functions that take results from other functional units and pass on results onto functional units.

Interconnection patterns depends upon the problem.

Workflow - describes the flow of information between the units.

Weeks 11 to 12

Workflow Example - Climate Modeling

Atmospheric Atmospheric

Hydrology

Land Surface ModelOceanic Circulation

Atmospheric Model

ChemistryCirculation Model

Model

Ocean Model

Ocean Chemistry

heating rates

water vapor content, humidity , pressure,wind velocities, temperature

sea surf ace temperature

wind stress,heat flux,water flux

GridNexus Workflow Editor

Developed by UNC-Wilmington.

Introduced in course.

Assignment set.

www.gridnexus.org

GridNexus Workflowusing Grid Services

Guest Speakers Professor Daniel A. Reed, Chancellor's Eminent Professor, Vice

Chancellor for IT and CIO, UNC-Chapel Hill, Director of Institute for Renaissance Computing, University of North Carolina at Chapel Hill, Duke University, and NC State University:

– “Grid computing: 21st Century Challenges.” Dr. Wolfgang Gentzsch, Managing Director, MCNC Grid Computing

and Networking Services:– “Grid Computing in the Industry”

Chuck Kesler, Director, Grid Deployment and Data Center Services, MCNC:

– “Security Policy, Legal, and Regulatory Challenges in Grid Computing Environments”

Professor Ian Foster: Ian Foster, Argonne National Laboratory and University of Chicago:

– “The Grid: Beyond the Hype”Taped presentation (originally given at Duke University, Sept. 14th, 2004).

Weeks 14 to 15

Course Home Pagehttp://www.cs.wcu.edu/~abw/CS493F04

WebCT also used for quizzes.

Fall 2005 Grid Computing Course

To originate from UNC-Charlotte.

Again in collaboration with UNC-Wilmington.

Can be for undergraduate or graduate credit.

Acknowledgements

Partial support for this work was provided by the National Science Foundation and University of North Carolina, Office of the President.

1. National Science Foundation, “Introducing Grid Computing into the Undergraduate Curricula,” ref. DUE 0410667, PI: A. B. Wilkinson, $100,000, 2004-2006.

2. University of North Carolina Office of President, “A Consortium to Promote Computational Science and High Performance Computing,” PI: B. Kurtz (Appalachian State University) co-PI B. Wilkinson and others at various universities, total $650,000, 2004-2006.

3. University of North Carolina Office of President, “Fostering Undergraduate Research Partnerships through a Graphical User Environment for the North Carolina Computing Grid,” PI: R. Vetter (UNC-Wilmington), co-PI B. Wilkinson and others at various universities, total $557,634, 2004-2006.

Papers Since Fall 2004

• B. Wilkinson, M. Holliday, and C. Ferner, “Experiences in Teaching a Geographically Distributed Undergraduate Grid Computing Course,” Workshop on Collaborative and Learning Applications of Grid Technology and Grid Education, IEEE International Symposium on Cluster Computing and the Grid (CCGrid2005), Cardiff, UK, May 9 - 12, 2005, accepted.

• M. A. Holliday, B. Wilkinson, J. House, S. Daoud, and C. Ferner, “A Geographically-Distributed, Assignment-Structured Undergraduate Grid Computing Course,” SIGCSE 2005 Technical Symposium on Computer Science Education, St. Louis, Missouri, February 23 - 27, 2005.

Questions?