SURAgrid Accounting StatusSeptember 2007

SURAgrid Accounting StatusSeptember 2007

Accounting WG - Phase 1

• Call for a working group at SURAgrid All-Hands March 07, to develop (at least) a recommendation for implementing accounting on SURAgrid

• Group formed with desire to be short-term but contribute a tangible immediately useful first step – Volunteers included: Victor Bolet, Steve Brandt, Mahantesh

Halappanvar, Steve Johnson, Shawn McKee, Jerry Perez, John-Paul Robinson, Bin Zhou

Accounting WG Phase 1

• Goals of initial SURAgrid accounting:– To develop and implement the capability to analyze

SURAgrid usage in support of setting implementation priorities, planning, and input to proposals.

– To monitor allocations, however these are prescribed by the resource owner at this time.

– Create a system that can be implemented with current resources but also incorporated into, or expanded by, more complex system later on if needed - also tracking well with standards and best practices in this area.

Phase 1 Implementation Plan

• Determine set of common data to be collected from SURAgrid resources

• Document desired data collection in the form of a SURAgrid Usage Record– Based on job Usage Record standard currently

progressing through the OGF User Record Working Group (http://forge.ggf.org/sf/projects/ur-wg).

• Enable data collection through local schedulers• Store data in standard format (XML) and central database

for access now, also later availability to another accounting package if one is selected.

Snapshot Status Phase 1 Work

• Some still to discuss but…SURAgrid Usage Record essentially complete!

• WG members piloting with all major local schedulers & documenting for others to follow

• Data from pilots is being stored in centralized database w/querying capability, hosted by ODU

• Will need to see if initial database & reporting package provide enough utility, can scale, etc. – “Accounting WG Phase 2” to address if not…

With Special Thanks to…

This was a real working working group and extra appreciation goes to the following SURAgridr’s for producing the components you see today:– Victor Bolet– Steve Brandt– Mahantesh Halappanvar– Steve Johnson– Shawn McKee – John-Paul Robinson

SURAgrid Usage Record

See handout…

Status of Local Scheduler Pilots

• PBS: Conversion to XML complete, script provided, incorporated into workflow

• LoadLeveler: Conversion to XML complete, script provided

• SGE: Basic data mapping defined, needs conversion to xml

• Condor: Conversion to XML complete, script provided, OSG Gratia project

• GridWay: Initial investigation suggests data available, needs code to extra from data store

• LSF?

Working Group Action Items

• GSU: work on Loadlever scripts with LSU

• ODU, TAMU, LSU: Loadlever and pbspro upload workflow

• TAMU: hammer out PBS to URWG conversion and will look at user mapping

• ODU, UAB, TAMI: will working on running 1.4.3 xindice (parallel development path with work-flow effort)

• UAB: SGE and Gridway acct data record generation, project page updates, move wiki to more formal home to facilitate future collaboration

• ALL: solidify project documentation with ideally step-by-step instructions

Status of User Documentation

• Currently in a Wiki, hosted by UAB

– http://webapp.lab.ac.uab.edu/projects/jpr/wiki/SURAgridAccounting

– Moving to UABgrid collaboration environment: http://projects.uabgrid.uab.edu/sg-accounting

• Working Group members still adding detail to this resource

• Still need to determine best flow for information

• Eventually move to other location or format?

– Can be hosted by UABgrid indefinitely

From PBS to Central Data Store

• Resource Manager: OpenPBS/PBS-Pro/Torque

• XML Conversion Tool: PBS-XML Accounting Toolkit

• XML Database: Xindice

• Report Generator: DarkSlide

Figure 1. A Basic Flow Chart for Collecting SURAgrid Usage Records.

PBS to XML

• Site: install PBS-XML Accounting Toolkit• Requires Java (1.4) and Ant• Export code from CVS @ Sourceforge• Build and Install RPM

% ant buildrpm

% rpm –ivh pbsxmlaccounting.rpm

• Set up user environment• Run command

• Central: install Darkslide and Xindice

Status of Central Data Collection

• Update from Mahantesh

Additional considerations

• Need to revisit the “for further discussion” fields in the SURAgrid Usage Record

• Need a way to map local user to grid user (dn)• On account naming conventions: Thinking at this time is to

leave it to sites and work on mapping.• It's up to local system to filter out non-suragrid users. Sites

could leave all accounting data in place and use SURAgrid summary to generate local reports. Might be a good way to generate a community around this resource.

Next Steps

• Need longer term Accounting WG for Phase 2– Insure and support implementation of SURAgrid Usage

Record– Evolve record as necessary– Evaluate use and utility of the basic system and

consider/select other options if needed– Participants:

• From the community: Mahantesh & Steve B. so far committed. Other volunteers?

• From SURA: Dali Wang, MFY if/as needed