SITE

REP

ORT

University of JohannesburgSouth Africa

Stavros LambropoulosNetwork EngineerI.C.S Department

OVERVIEW• History of the UJ Research Cluster• User Groups• Hardware• South African Compute Grid (SA Grid)• Status• Applications• Issues• Future• Links• Contributions

HISTORY OF UJ RESEARCH CLUSTER

• UJRC started as an initiative of the High Energy Physics Group

• March 2009 - The UJ-OSG Compute Element passes validation and is registered on VORS (Resource Selector)

• March 2009 – 56 CPU Cores available• April 2009 – UJ Hosted Grid School

USER GROUPS• High Energy Physics (Physics) • Astrophysics (Physics) • Molecular Dynamics (Chemistry) • Quantum Chemistry (Chemistry) • Applied Mathematics • Numerical Studies (Engineering)

HARDWARE• 1 Head Node comprising :

Dell 2950, 2 x 4 Core Xeon Processors16 GB RAM900 GB – RAID5Scientific Linux 4/64 Bithosts : NFSv4, Accounts, Torque, Ganglia

HARDWARE (CONTD)• Separate Virtual Machines (VMWare

Server) for :OSG CE (1 GB RAM)OSG UI (submit node) – (2 GB RAM)gLite CE (1 GB RAM)gLite UI (submit node) – (1 GB RAM)

HARDWARE (CONTD)• 7 Worker Nodes comprising :

Dell 1425, 2 x 4 Core Opteron Processors16 GB RAMScientific Linux 4/64 BitgLite sw locally installedOSG sw from NFS

• Alcatel 6400 Gig Switch

HARDWARE (CONTD)

SA GRID• The South African National Grid is a

project to provide a national grid computing infrastructure to support scientific computing and collaboration. This project is managed by a consortium of universities, national laboratories and the Meraka Institute, under the cyber infrastructure programme, based on the gLite middleware .

STATUS• OSG is operational on SL4 base

STATUS (CONTD)• Started Discussion on the choice of

referent Tier1/Tier2 for ATLAS and ALICE• WN’s, CE’s and UI’s to be updated to SL5

as requested by LHC Computing Grid• Cobbler and Puppet to be used for the

new SL5 node installation and management

• Updating of Head Node from SL4/VMWare to SL5/Xen is planned

APPLICATIONS• 2 Commercial Applications Running

Locally :ANSYS FLUENT – Flow modeling SoftwareStar-CCM+ - Computational Fluid Dynamics

• Other Local Applications :Geant4 for NA63, MineralPETNA63 dedicated simulation codeDiamond Lattice Deformation

APPLICATIONS• On OSG :

Full ATLAS VO Support ENGAGE VO runs a few jobs local ATLAS users submit remote jobs from local UI Initial discussions have started to allow DOSAR VO

• On SAGrid : Will allow SAGrid VO’s ALICE VO ATLAS e-NMR VO WISDOM VO GILDA

The UJ Research Clusterand the OSG GRID

UJ – Physics High Energy Physics, ATLAS experiment at CERNKetevi Assamagan, Simon Connell, Sergio Ballestrero, Claire Lee, Neil Koch, Phineas Ntsoele ATHENA installed, using Pythia event generator to study variousHiggs scenarios.

UJ – Physics Diamond Ore Sorting (Mineral-PET)Sergio Ballestrero, Simon Connell, Norman Ives, Martin Cook, Winile SibandeGEANT4 MonteCarlo

Online diamond detection

Online diamond detection

Monte Carlo simulation

ISSUES• Limited International Bandwidth

Currently Using 11Mb/sTo be Upgraded early next year with the SEACOM cable

ISSUES (CONTD)• Research Funding – for HW and training• Additional complexity to manage both

OSG and gLite• Lack of caching by OSG installer, partially

solved with local Squid cache• No automated install & config system

yet, starting to work on Cobbler and Puppet

• NFSv4 problematic on SL4• Monitoring, need to add detailed job

monitoring/stats for Torque

ISSUES (CONTD)• Manpower – Grid services not primary

job/role for the 3 people – Addressing problem with single national Operations Team

• Low Usage – Marketing of services and availability has been done but researchers are slow to start

• No experience gathered on utilization of resource in terms of constraints on memory, disks, CPU and network

• Final VO acceptance policy required

FUTURE• Hardware Upgrade :

Additional 4 x WN’s being configured 1 x Dell MD1000 Storage shelf (6TB raw)

will be connected to the Head Node. – Ordered

16 x WN’s (Dell M605 blade chassis, with 2 x 6 Cores, 32GB Ram) – Ordered

224 Cores will be available• DOSAR Workshop in South Africa in

2010

LINKS• University of Johannesburg

http://www.uj.ac.za• UJ Physics

http://physics.uj.ac.za/cluster• South African Grid

http://www.sagrid.ac.za

CONTRIBUTIONS• Prof. S. Connell – UJ Physics Department• Sergio Ballestrero – UJ Physics & CERN

ATLAS TDAQ• Bruce Becker – SA Grid Co-ordinator• Francois Mynhardt – UJ I.C.S Department

Questions