Scripted Job Submission with Grisu Jython

NeSI Workshop for NZ eResearch 2011Aaron Hicks, Landcare Researchhicksa@landcareresearch.co.nz

Introduction

Welcome to my workshop, where we will explore scripted job submission with Grisu Jython

• Why script job submission?• Introducing Grisu Jython• Practical Exercises– Trivial HelloWorld– Submitting multiple jobs using lists– Using BatchJobObject to reduce overheads

Why script job submission?

“I’ve got a bad feeling about this...”- various, the Star Wars franchise

BeSTGRID Jobs

The Grisu clients are used to submit jobs to compute resources via the BeSTGRID Jobs service.

Usually with the Grisu Template Client:• Convenient GUI for job submission• File management• Templates for common applications• Detailed job settings can be exposed

Limitations of Grisu Template Cilent

The Grisu Template Client is not suitable for all kinds of jobs:

• Batches of many jobs• Automated job submission• Jobs that require local processing to prepare

job items• Jobs that require complex submission

parameters

Batches of Many Jobs

The BeSTGRID Jobs service can handle hundreds, or even thousands of job submissions, but doing this with the Grisu Template Client is mind numbingly repetitive and requires inhuman patience.

SOLUTION: Write a script or application that iterates through all the jobs, submits them, and then retrieves the output.

Automated Job Submission

The Grisu Template Client is a human interface and can not submit jobs without human interaction. Not useful for submitting jobs from another application or service

SOLUTION: Write a script or object that can submit a job from within an application, or can be called as a scheduled task.

Jobs Requiring Pre-processing

A job may require a number of steps to be performed before a job is even entered into the Grisu Template Client for submission. This may include chaining multiple jobs together.

SOLUTION: Write a script that performs pre-processing steps as part of the job submission process

Complex Job Parameters

A job may require specific parameters, such as:• Specific submission sites• Specific applications and versions• Specific selection and partitioning of resourcesWhich may be difficult to repeat reliably using a

human interfaceSOLUTION: Write a script that sets the

parameters exactly every time

All of the Above

Complex job submission may exhibit any and all of the features discussed previously, and the human interface provided by the Grisu Template Client becomes cumbersome and limits job submission.

These can all be solved by creating a script or application to make job submission automated, repeatable and reliable.

How?

There are a number of solutions from the Grisu Project:

• Grisu Framework: for creating Java applications using the Grisu libraries

• GriCli: a command line interface for Grisu• Grisu Jython: Python-like scripting using Grisu

http://github.com/grisu

Introducing Grisu Jython

“Everything should be made as simple as possible, but not simpler.”

Albert Einstein

Grisu Jython

Grisu Jython allows scripting using Jython, a Python-like dynamically typed language, allowing complex job submission scripts using tools that many researchers are already familiar with, without the overheads of Java, or not easily done with GriCli.

Python, and Jython, can be used very simply, are well established in the research community, and are known for ease of use for new users.

Why Jython?

Jython is Python implemented in Java, and though this seems cumbersome, it provides some advantages over a pure Python implementation:

• Java is ubiquitous, and the JVM is available on almost all platforms

• Jython allows access to native Java objects and methods to a Python-like language

• Jython can be bundled into .jar files for distribution, hence does not need installing

Using Grisu Jython

1. Download the grisu-jython.jar file from:http://code.ceres.auckland.ac.nz/downloads

2. Write your script in your favourite editor or IDE

3. Execute your script in the form of:

java -jar grisu-jython.jar myscript.py

Monitoring Jobs

The Grisu Template Client still has a use in Grisu Jython job submission scripts. It provides a convenient interface for monitoring and managing submitted jobs, and as a file manager between the local workstation and remote sites.

Download the Grisu Template Client from:http://code.ceres.auckland.ac.nz/downloads

Run the Grisu Template Client with:java -jar grisu-template-client.jar

Practical Exercises

“Work! Work!”- Orc Peons, WarCraft

Practical Exercises

Next comes the practical component of the workshop, where we will explore three use cases of Grisu Jython.

1. A trivial HelloWorld job2. Submission of multiple HelloWorld jobs using

the Python List objects3. Using the BatchJobObject to parallelise job

submission and minimise upload and download time and volume

Do you have a Grid identity?

Let’s sort that out now.

Get the examples

The example code is available on my personal GitHub repository:

http://github.com/aethylred/grisu-scripted-submission

This should soon be merged with the Grisu Examples repository:

http://github.com/grisu/examples

This can be retrieved via eGit within Eclipse

Trivial HelloWorld

Submit a job that echoes “Hello World” to the standard output.

• This example demonstrates the absolute minimum of code required to submit a job.

• Good for repurposing to run shell commands for testing or demonstrations.

• Requires specifying the job submission site

Processing jobs in a list

Using one script to run a HelloWorld script as a job multiple times.

• Attaching input files to jobs• Using loops to create multiple jobs in a list• Iterating through a list of jobs to:– Define and submit jobs– Wait for jobs to finish– Retrieve job outputs– Kill jobs and clean up

BaconCount

The Bacon Counter uses random text generated by the Bacon Ipsum site: http://baconipsum.com

The Bacon Counter is actually a simple dictionary matching script that counts the incidents of the words provided in the dictionary.txt file in each file in the input directory.

Using BatchJobObject

The BatchJobObject has additional features and methods that can upload files and submit jobs in parallel.

• Submission location Blacklists and Whitelists• De-duplicate file uploads by uploading files per

batch, rather than per job• Simplify staging files, submitting jobs, and killing job

within the batch fileBUT not perfect, job output has to be retrieved

individually using the BatchJobObject as list

Submitting Bacon

The submitbacon.py script :• Uploads and submits 5 jobs at a time• Uses the BatchJobObject to upload common

files (countbacon.py and dictonary.txt) only once

• Create jobs per file in a directory

Support for Grisu

Grisu JavaDocs on GitHub:http://grisu.github.com/grisu/javadoc

Known issues at the Grisu Project site on GitHubhttp://github.com/grisu/grisu-jython/issues

http://github.com/grisu/grisu/issuesNew issues should be submitted to NeSI support

with “[Grisu]” in the subjectsupport@nesi.org.nz

Finished

I hope you enjoyed the workshop.

Aaron Hickshicksa@landcareresearch.co.nz06 353 4954

PS. Yes, I wrote this presentation on the flight down