CERN Batch System, Monitoring and Accounting - HEPiX Fall 2012 · 16 { Batch Accounting Overhaul...

PES

CERN IT DepartmentCH-1211 Geneve 23

Switzerlandwww.cern.ch/it

CERNITDepartment

CERN Batch System, Monitoring andAccounting

HEPiX Fall 2012

Jerome BellemanCERN – IT-PES

October 2012

http://www.cern.ch/it

http://www.cern.ch

PES



CERNITDepartment

2 – CERN BatchSystem, Monitoring

and Accounting

Context

Growing community

Busier batch system

Agile Infrastructure project


http://www.cern.ch

PES



CERNITDepartment


and Accounting

Outline

1 Batch System Challenges

2 Batch Monitoring Tools

3 Batch Accounting Overhaul


http://www.cern.ch

PES



CERNITDepartment

4 – Batch SystemChallenges

Section 1

Batch System Challenges


http://www.cern.ch

PES



CERNITDepartment


CERN Batch Setup

Platform LSF 7.0.6

All resources to one cluster

Different shares for different customers: public, grid andseveral for CERN experiments

LSF Master NodeNFS Server

LSF Master Failover

WNWN WN WN WN WN WN WN WN WN

Local Jobs Grid Jobs


http://www.cern.ch

PES



CERNITDepartment


A Large Batch System

> 4 000 physical nodes

> 60 000 cores, some SMT-enabled (25% overcommit)

> 55 000 job slots, > 400 000 jobs/day:


http://www.cern.ch

PES



CERNITDepartment


Future of the Batch Service

Agile Infrastructure Project:

Virtualise resources in CC: batch nodes to be fat VMs

Uniform IaaS layer

Configuration management with Puppet


http://www.cern.ch

PES



CERNITDepartment


Today’s Operational Issues

High submission and query load → Slow response

Ensuring fairshare scheduling

Complex LSF setup

Poor dynamism requiring daily reconfiguration

Scalability


http://www.cern.ch

PES



CERNITDepartment


Possible Alternatives to LSF

Goal for 5 years:

4 000→ 12 000 physical nodes

60 000→ 300 000 cores

Support frequent structural changes

Possible alternatives (unordered):

LSF 8

Condor

Grid Engine

Torque

SLURM ←−


http://www.cern.ch

PES



CERNITDepartment


Evaluating SLURM

From the SLURM Web site:

Free

65 000 physical nodes

120 000 jobs/hour

Active community

Extensible via plug-ins

Test bed:

Implement and test hierarchical fairshare model

Controllably submit queries and jobs

Reproducible load

Scale number of hosts, jobs, slots and queries


http://www.cern.ch

https://computing.llnl.gov/linux/slurm/

PES



CERNITDepartment

11 – BatchMonitoring Tools

Section 2

Batch Monitoring Tools


http://www.cern.ch

PES



CERNITDepartment


Technology Overview

Oracle, Python, Matplotlib & Django → Stats

Cassandra → Fairshare monitoring

OpenTSDB → Live monitoring

Splunk → Historical usage


http://www.cern.ch

PES



CERNITDepartment


Live Monitoring with OpenTSDB


http://www.cern.ch

PES



CERNITDepartment


Historical Usage with Splunk


http://www.cern.ch

PES



CERNITDepartment

15 – BatchAccounting Overhaul

Section 3

Batch Accounting Overhaul


http://www.cern.ch

PES



CERNITDepartment


New Batch Accounting: Goals

Make portable to other schedulers

Publish local job information

Publish correct normalisation factor per job

Use the new APEL software

Remove complexity, improve consistency


http://www.cern.ch

PES



CERNITDepartment


Old vs. New Batch Accounting

CEsBLAH

File

LRMSAcct.File

Acct.

ReportsAcct.Page

APELAcct.Portal

Daily

FilterLocalAPEL

SSMMessaging


http://www.cern.ch

PES



CERNITDepartment


Old vs. New Batch Accounting

CEsBLAH

File

LRMSAcct.File

Acct.

ReportsAcct.Page

APELAcct.Portal

Real-T

ime

FilterLocalAPEL

SSMMessaging


http://www.cern.ch

PES



CERNITDepartment


and Accounting

Conclusion

We need to scale

We’re moving to new infrastructure tools

CERN batch service being prepared for future challenges


http://www.cern.ch

PES



CERNITDepartment


and Accounting

Thanks!

Questions?


http://www.cern.ch

CERN Batch System, Monitoring and Accounting - HEPiX Fall 2012 · 16 { Batch Accounting Overhaul...

Documents

Transcript of CERN Batch System, Monitoring and Accounting - HEPiX Fall 2012 · 16 { Batch Accounting Overhaul...