JAS – Distributed Data Analysis

Grid Enabled Analysis Workshop

Caltech - June 23-25, 2003

Contents

JAS2Historyclient-server modeJAS2 and the Grid

JAS3What’s newJAS3 and AIDAPlans for Gridification

JAS History

First version of JAS2 released in 2000.Incremental improvements released over time.

JAS2 History – Use Cases

With WIRED event display Online Monitoring

JAS2 History – Use Cases

Custom Applications Web Servlets

JAS Client-Server Mode

Java Compiler+

Debugger

GUI

Experiment Extensions

(Event Display)

User’sJava Code

Padded Cell

Data AnalysisEngine

DATA

Distributed Analysis System: Goals

Prototype for GRID enabled JAS analysisRun analysis on a farm of machines

Use multiple CPU’s in parallel for CPU-intensive analysisAccess multiple I/O channels for data-intensive analysis

Use standard JAS (Client) as if we are running a local JobGet interactive feedback

Create analysis modules (code)Control job executionView results (Plots/Histograms)

Access distributed datasets as if they were local datasets

Distributed Analysis System: Architecture

JASClient

JAS DataServer

JASClient

JASClient

…

Network

Users

CatalogServer

ControlServer

ControlServer

Network

JAS DataServer

JAS DataServer

…

JAS 2 – GRID interface (Tech-X)

JAS3 Overview

A completely new version of JASDesign based on Application Shell, into which many (optional)

modules can be pluggedHighly customizable for different application domains

– HEP/Astrophysics/Other– DST analysis/Online Monitoring/GRID analysis– Experiment/User specific modules

Modules can be updated independently of shell– Possible to release bug fixes fast

Includes support for programming in many languagesScripting: Python, Pnuts, Dynamic Java, ….

– Command promptJava (compiled)

Analysis (histograms, tuples, fitting) based on AIDA standard

Not technically backwards compatible with JAS2But migration is straightforward.

AIDA Overview

AIDA = Abstract Interfaces for Data AnalysisCovers key areas for data analysis

Histograms, Tuples, Fitting, Data Points, Plotting, Management

Developed collaboratively at series of workshops by groups at CERN, LAL, SLAC.Next workshop June 30-July 4 -- CERN

Interfaces developed for C++ and Java ( and maybe Python?)Several implementations/tools available

Anaphe/Lizard/LCG PI – CERNOpen Scientist – LALJAIDA/JAS/AIDAJNI – SLAC

JAS3 and AIDA

JAS3 has adopted AIDA for analysis AIDA allows us to leverage experience and skill of other

developersAIDA is functionally more complete than JAS2 analysis packageAIDA allows JAS to exchange data with other AIDA toolsAIDA provides bridge to C++ programs (e.g. Geant4)AIDA encourages creativity and innovation

JAS3 HEP Analysis tools based on JAIDAJAIDA = Java implementation of AIDA

JAIDA is part of FreeHEP libraryUsable as standalone library for any Java Application

AIDAJNI = Interface between C++ and Java AIDAAllows C++ programs to use JAIDA, JAS3

JAS3, AIDA and C++

C++ program

AID

A

C++ AIDA Implementation

AIDA- JNI

AID

AJAIDA

JAS3

.aida file(XML)

Java programA

IDA

JAS3 and AIDA

JAS3 supports all AIDA functionality, includingHistograms (includes arithmetic, projections, etc.)Clouds (unbinned histograms, scatterplots)PlotterTuplesFitting – AIDA interfaces allow for multiple fitters

Uncmin -- pure java minimizerMinuit -- Fortran called by Java Native Interface (JNI)

IOAIDA XML, PAW, Root

JAS3 supports user interaction with AIDA in three waysScripting (Pnuts, Python etc)Compiled (Java) codeGUI – Plotting, Fitting, Cuts etc.

JAS3 Scripting

JAS3 has multi-language OO scripting supportCommand line, Console, EditorMajor components (e.g. AIDA) have scripting interfaces

Currently have plugins to supportPnutsPnuts – syntax almost identical to Java, fast, well documented

and feature completePythonPython (using Jython)More scripting languages can be added

not restricted to Java implementations (e.g. could use C-Python, JPE)

JAS3 Lightning Tour

Tour designed to

give you an

overview of the

capabilities of JAS3,

you can try them

out for yourself this

afternoon.

Welcome Page, gives initial info and links to example

scripts and programs

Memory monitor

Opening Files

Use file menu

Drag from explorer

Graphical Interface to AIDA

Histograms, Clouds, Tuples all presented in

AIDA tree

.aida files, .hbook

files, .root files all presented

as AIDA objects

Drag items onto page, or use

(popup) menus

Printing

Can send individual

plots or full page

direct to printer

Or save as PS, EPS, PDF, SWF, SVG, PNG, GIF…

Or copy/paste into Word, PowerPoint

etc.

Java Editor, Compiler and Loader

Built-in Java compiler

Tree shows loaded

programsBuilt-in

editor for writing

analysis code

Unlike JAS2 which only supported “event analyzers” JAS3 allows any Java program

to be loaded. This example “main routine” is taken directly from the AIDA manual

Scripting

Can also write and run

scripts

Console allows direct interaction

with scripting language

Pnuts Language

Currently support Pnuts scripting languageComplete and well documented

http://javacenter.sun.co.jp/pnuts/doc/guide.html

Fast (although not as fast as compiled Java)Syntax very similar to JavaCan easily call compiled Java classes from scripts – best of

both worlds

Plan to support other languages in futureIn particular Python

Record Sources

Opening record (or event) based files

causes the run control toolbar

to appear

Works similarly to JAS2 Job control, but now also supports random access and “tagged” data sets

(mainly for event displays)

Histogram Profile

ScatterPlot

XY Data(More appropriate forsmaller data sets)

Tuple Explorer - Plots

Works with any tuple, read from file or

dynamically created

Tuple Explorer – Define Columns

Tuple Explorer - Cuts

Tuple Explorer - Tabulate

To be used with record loop

Tuple Explorer – Record Source

JAS3 Spreadsheet

Simple spreadsheet pluginfor

Displaying resultsCalculationsSimple Plots

Supports reading/writing.csv filesExcel files

Cut/Paste with Excel etcComing Soon…

Scripting interfaceGUI for building plotsUser defined functions

– Java, scripting

Miscellaneous Features

Save/Restore configuration

User Preferences

Plugin Manager

Status

Currently released JAS3 version 0.7.1AIDA functionality is quite solidCompiler, Loader, Record Loop all quite recently added,

Certainly still some rough edges

Documentation limited but availableBuilt-in example scripts and programsTutorial on web

If you are used to JAS2 you will find some functionality

not yet ported to JAS3Remote (client/server) access to data.3D Lego/Surface plots

JAS3 and the GRID

We plan to add client-server/distributed capabilities to

JAS3 similar to those in JAS2Will be based on (distributed) AIDA

Next AIDA workshop (at CERN next week) will discuss this

Want to use Grid standards where they existWork with others (PPDG-CS11,???) to define standards where they

do not exist

Want to be compatible with C++ serversTech-X have submitted phase II SBIR and if approved will work

closely if approved

JAS3 Links, More Info

JAS – Java Analysis Studio - http://jas.freehep.org

JAS3 – http://jas.freehep.org/jas3

JAIDA – http://java.freehep.org/jaida/

AIDA – http://aida.freehep.org

FreeHEP - http://www.freehep.org

FreeHEP Java Libraries - http://java.freehep.org

WIRED – http://wired.freehep.org