OGCE Overview: Services, Gadgets, and Tags

Marlon Pierce, Suresh Marru, Gregor von Laszewski, Mary Thomas, Maytal Dahan, Gopi Kandaswamy, Rion Dooley, and Wenjun Wu

Links for More Information Website: http://www.collab-ogce.org

News/RSS/Blog: http://collab-ogce.blogspot.com

Email: discuss@ogce.org

SVN Code Repository: http://ogce.svn.sourceforge.net/viewvc/ogce/ And get tar.gz’s

Source Forge: http://sourceforge.net/projects/ogce

Project Description Status

Grid Portal Packaged, one step installation of Grid portlets into Tomcat+Gridsphere

Stable, V2.4 for SC08

Axis Services One step install of Resource Discovery and Prediction Services. More services to come.

Dev. Release for SC08.

Workflow Suite Support for scientific workflows, based on LEAD tools. Suresh will describe next.

Dev. Release for E-Science

GTLAB JSF Tag Libraries for building Grid portlets and gadgets

Dev. Release for SC08

JavaScript COG(Cyberaide)

JavaScript libraries for building Grid gateways.

Dev. Preview for SC08

Gadget Container

Open Social Inspired container for iGoogle gadget

Dev. Preview for SC08

Incubator projects

Pylons portals, metascheduling services, bio-gadgets, etc.

SVN if you dare

Services, Workflow, GTLAB, Cyberaide, and Gadget container are part of an emerging stack

OGCE and Gateways We develop and package software for use by

Gateways.

A lot of this comes from active Gateways. GPIR: TeraGrid User Portal Workflow tools: LEAD Resource Discovery Service, File Browser Applet:

TGUPGridChem

Gateways

OGCESoftwar

e

GPIR for Facebook, iGoogle GPIR: GridPort Information

Repository, stores dynamic and static resource information for VOs

Facebook Mini TeraGrid system monitor and full

size system monitor Mini:

http://apps.facebook.com/mini_tgsysmon/

Full Size: http://apps.facebook.com/tgsysmon/

Google Mini System monitor for iGoogle page To add visit: http://www.google.com/ig/adde?

moduleurl=http://iola.tacc.utexas.edu:22080/gpir-app/minigpir.xml&nocache=1&source=imag

GPIR Facebook App

Open Life Science Gateway Bio-gadgets

Job Submission History gadget

Blast gadget

ClustalW gadget

OGCE Portal

Components for Science Portals

OGCE is founded on the principal that portals should be built out of reusable parts.

Key standard in our first phase: the JSR 168 portlet specification.

Portlets can run in multiple containersuPortal, Sakai, GridSphere, LifeRay, etc.

Allows us to build Grid specific components and deploy along side other goodies: Sakai collaboration tools, contributed portlets, etc.

Future: Open Social compliant Google Gadgets

Downloading and Installing OGCE

Portal installation is described in greater detail later in the tutorial.

Key point for current purposes: you can have a basic portal up and running in a few minutes.

Installation consists of Updating properties in pom.xml to match your Grid.Running “mvn clean install”That’s it

We’ll now look at some features.

OGCE GPIR portlet can interoperate with TeraGrid and your own GPIR

services.

Manage TeraGrid MyProxy credentials with the OGCE

ProxyManager portlets.

OGCE file management client portlets interact with TeraGrid

GridFTP servers.

General purpose batch and interactive job submission to GRAM, WS-GRAM is supported.

Dashboard Portlet

14

The dashboard portlet allows users to track jobs on the selected resource. The user can view either his own set of jobs or get information on all submitted jobs.

Queue forecasting portlets work with the NWS QBETS to predict wait times and deadlines.

PURSe portlets manage user requests for portal accounts and Grid credentials.

Condor and Condor-G

OGCE IFrame Portlet can be used to integrate external

sites.

Under the Hood: Building Components

Portlets can be built with our Apache Velocity bridge.“Template” or “steal this code” approach.

OGCE portlets use the Java COG Abstraction Layer to access Grid services.Provides a buffer over different Grid toolkits and

versions.Provides ability to compose graph-based

workflows.Tutorial in itself

We have also developed Grid Tag Libraries and Beans (GTLAB) to simplify this process.

Some OGCE ServicesNew things available for download

Resource Discovery Service

Extracted from the TeraGrid User Portal Contributed by Rion Dooley

Axis2 Web service

Integrates information from INCA, GPIR services. What machines are in my Grid^H^H^H^H Cloud? What are the service names? Which ones are actually running?

Works with TeraGrid deployed services, but you can use it with your own information service installations.

Resource Prediction Service (RPS)

• Predicts an optimal set of resources for running scientific applications– Predicts a set of resources on which the sum of data

transfer time, queue wait time and compute time for the application will be minimum

– Data transfer time: Uses input data size and NWS info

– Queue wait time: Uses QBETS info– Compute time: Uses application performance info

• Axis2 web service with a MySQL backend

• RPS is a subset of the Fault Tolerance and Recovery service used in LEAD and VGrADS

Packaging ServicesEverything builds with one command

mvn clean install

We use Apache Maven 2 for builds.Structured, modular builds and consistent layouts

Build a single module: mvn clean install –f somemod/pom.xml

We include Apache Tomcat in the download, so all you need is Java and MySQL.

Suitable also for the NMI build and test system.

Want the Javadoc? mvn javadoc:javadoc

Upcoming Services

XMC-CAT metadata cataloging service with JSDL sample implementation.See also our demo on Wednesday.

Swarm Web Service for mass job submission and management.See also Sangmi Pallickara’s talk Friday at 2:00http://escience2008.iu.edu/sessions/SWARM.shtml

GPIR Web Service for information on your Grid. This is being repackaged; older packaged version (but same code) is already available.

Workflow suite services will be integrated in the future.

GTLABTag Libraries for building gadgets and

portlets out of reusable parts.

User’s BrowserJavaScript

OGCE Gadget Container

iGoogle, Facebook

GTLAB Gadgets

TeraGrid Services Other ServicesRPS, RDS, GPIR

Services

GTLAB’s Relationship to Other Components

MyProxy Example<h:form id="form"> <h:outputText value="Username:"/> <h:inputText value="#{resource.username}" required="true"/>

<h:outputText value="Password: "/> <h:inputSecret value="#{resource.password}" required="true"/>

<h:outputText value="MyProxy Server: "/> <h:inputText value="#{resource.myproxyHostname}" required="true"/>

<o:submit id="submit" value="Submit" binding="#{builder.body}" action="#{builder.junkAction}" >

<o:multitask id="multi" persistent="true" taskname="#{resource.taskname}"> <o:myproxy id="mypr" hostname="#{resource.myproxyHostname}"

lifetime="2" password="#{resource.password}" port="7512"

username="#{resource.username}"/> </o:multitask> </o:submit> </h:form>

Executing Multiple Task<o:submit id="submit" value="Submit" binding="#{builder.body}"

action="#{builder.junkAction}" > <o:multitask id="multi" persistent="true"

taskname="#{resource.taskname}"> <o:jobsubmit id="js" myfaces="true"

arguments="#{resource.arguments}" executable="#{resource.executable}" hostname="#{resource.hostname}" provider="#{resource.provider}" stdout="#{resource.stdout}"/>

<o:filetransfer id="ft" myfaces="true" from="#{resource.from}"

to="#{resource.to}" /> <o:dependency id="dep" task="ft"

dependsOn="js"/> </o:multitask></o:submit>

<o:submit id="submit" value="Submit" binding="#{builder.body}" action="#{builder.junkAction}">

<o:multitask id="multi" persistent="true" taskname="#{resources.taskname”>

<o:rpsOptimal resource="task1-resources"

id="rpsoptimal-exec" myfaces="true"/>

</o:multitask> </o:submit>

RRRPS Client with GTLAB

Example with Dependency <o:submit id="submit” …> <o:multitask id="multi” …>

<o:rpsAddModel resource="task1-resources"

id="rpsAddModel-exec"

myfaces="true"/> <o:rpsManager resource="task1-

resources" id="rpsManager-

exec" myfaces="true"/>

<o:dependency id="deps-rpsstuff” task="rpsManager-exec" dependsOn="rpsAddModel-exec"/>

</o:multitask> </o:submit>

GTLAB FeaturesExtends Java Server Faces.o Tag components wrap major COG Abstraction Layer

featureso And Web Service clients, GridShib, etc.o Supports tag inter-dependencies.

Allows you to do standalone development.

Use JSF portlet bridge to convert into portlets No new coding, just add/modify XML config files and jars.

Or forget about portlets and o Convert into Google Gadgetso Develop as a Facebook applicationo Etc.

Installing GTLABDownload the release code from www.collab-ogce.org or

check out from SVN:o svn checkout

https://ogce.svn.sourceforge.net/svnroot/ogce/GTLABo SourceForge also supports tar.gz downloadso Tagged release in

http://ogce.svn.sourceforge.net/viewvc/ogce/tags/

Run “mvn clean install” in GTLAB directory.Download includes a web server.Start with “./startup.sh”, go to http://localhost:8080/GTLAB

What’s New with GTLAB?Completely revised dependency graph processing.

Removed dependence on COG graph processing (still use the COG)

We can now support graphs of any tag you care to write. Only support pipelines currently, not full DAGs

Redesigned to simplify writing new tags. Extensive use of factories. You no longer have to change core code to add a tag

Simplified process of creating new tags 4 Java files and 3 XML config files Now can be done in one (mvn) step

Many new tags

Levels of Abstraction in GTLAB code

You can use prepackaged gadgets/portlets.

You can develop new applications using existing tag libraries.

You can make new tags for your services

You can hack or steal the codeNote JSF is compatible with JSP, so you can

mix and match.

And you can embed JavaScriptYUI, Scriptaculous user interface goodies.Google and Facebook APIs.

Tag Description

MyProxy Gets a proxy credential (COG)

JobSubmit Runs remote commands (COG)

FileOperation Create, delete, list remote files and dirs (COG)

FileTransfer GridFTP downloads, 3rd party transfers (COG)

XSubmissionOutput

Format outputs of job and file operations

ResourceDiscovery

Clients for the RDS

ResourcePrediction

Clients for the RPS

MyCommunityProxy

Get and decorate TG Community credentials with Grid Shib

Community Log Log use of TG Community credentials

GTLAB Example Description

MyProxyExample Get a credential and store in OGCE global proxy store.

JobSubmitExamples

Various examples showing how to submit a grid job and get output.

Dashboard Examples

Runs showq on Big Red, Cobalt, and Mercury and formats output.

DirectoryBrowser Displays and downloads files on a remote host via GridFTP

MultipleTaskExample

Shows how to couple file transfers, operations, and

These will work as gadgets, portlets, or standalone applications

GTLAB Gadgets CodeBase

GTLAB PortletsCodeBase

Tomcat Web

Server

Tomcat Web Server

+ Portlet Container

iGoogleClient

Aggregator

Compile and deploy into server

Decorate with bridge and container jars, xml config files.

GTLAB gadgets can also run as portlets with no code changes by using the JSF portlet bridge.

Making GTLAB Google Gadgets

A Gadget is a standalone application that is integrated into your personal iGoogle display.They can run on your Web server.You can use SSL/HTTPS

Google Gadgets have two versionshttp: these use Google’s JavaScript APIsurl: these wrap external applications with IFrames.

Limitations:Not much real estate (but location, location,

location) Cookies manage state on gadgets from the same

server but we need a way to integrate multiple independent servers.

Example Gadget Config File <?xml version="1.0" encoding="UTF-8" ?> <Module> <ModulePrefs title="MyProxy Gadget Example"

scrolling="true" height="500"/>

<Content type="url" href="http://…/:8080/GTLAB/examples/

MyProxyExample.jsf"> </Content> </Module>

Save this as MyProxy.xml and place in a Web accessible place.

Other GadgetsProviders

Tomcat + GTLAB Gadgets

Grid and Web Services

(TeraGrid, OSG, etc)

Other GadgetsProviders

Social Network Services (Orkut,

LinkedIn,etc)

RSS Feed, Cloud, etc

Services

Gadget containers aggregate content from multiple providers. Content is aggregated on the client by the user. Nearly any web application can be a simple gadget (as Iframes)

Click “Add stuff” and then “Add feed or gadget”. Type the URL of your gadget.

GTLAB Applications as Google Gadgets: MOAB dashboard, remote directory browser, and proxy management.

Making GTLAB Gadgets into Portlets

Do all development in jsf_standalone.

When you are ready to convert:cd GTLAB/transitionmvn clean process-resourcescd GTLAB/portletsmvn clean install

You will need to manually edit portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/

That’s it.

Tomcat +

Portlets and Container

Grid and Web Services

(TeraGrid, OSG, etc)

Grid and Web Services

(TeraGrid, OSG, etc)

Grid and Web Services

(TeraGrid, OSG, etc)

HTML/HTTP

SOAP/HTTP

Common science gateway architecture. Aggregation is in the portlet container. Users have limited selections of components.

Various GTLAB applications deployed as portlets: Remote directory browsing, proxy management, and LoadLeveler queues.

What’s next for GTLAB?

OpenID and RSS client support

Many other client librariesGoogle, Facebook, Amazon

Support for more complicated graphs

JavaScript/AJAX actions as well as HTML FORM actions.

But really, we’re looking for interested developers and users.