JRA1 Activity Feedback Frédéric Hemmer EGEE Middleware Manager and the JRA1 team
CERN - European Laboratory for Particle Physics DESY November 2, 1998 PC Farms at CERN Frédéric...
-
Upload
eustacia-carson -
Category
Documents
-
view
220 -
download
3
Transcript of CERN - European Laboratory for Particle Physics DESY November 2, 1998 PC Farms at CERN Frédéric...
DESY November 2, 1998
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC Farms at CERN
Frédéric HemmerCERN-IT/PDP
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Disclaimer
This will cover farms which imply an involvement of CERN’s computer center.
There are other farms in strict online environments or “private” farms in building.
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Overview
Off line farms• Linux farms• NT farms• Issues
PC Technology & Performance Online Farms & quasi online farms Cost of ownership Conclusions
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Linux Farms - Nomad
Proof of concept in Summer 97 Straight NQS port SHIFT SW client port CERNLIB port NOMAD observed a quasi linearity with
clock frequency compared to Alpha’s !!!• I.e. Alpha@266 MHz = PII@266 MHz
Now 17 PC’s dual, 3 types of MB
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Cern Units
3Q97 4Q98 1Q98 2Q98 3Q98
NOMAD Installed Capacity
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Linux Farms - NA49
NA49 already deployed privately a PC farm in their premises
Request a new farm to be deployed in order to benefit from the computer center infrastructure (people and equipment …) in 1 H98
Trivial deployment, running with NQS Most PC’s are branded PC’s (HP) Now completely off RISC for CPU 18 DUALS @ 300->400 MHz
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
NA49 Analysis - data access
SONY DMSSONY DMS
UnixUnixServerServerUnixUnixServerServerUnixUnixServerServer
CORECORETapeTape
ServersServers
HiPPIHiPPI
From experimentFrom experiment10-12 TB / month10-12 TB / month
1 month/year1 month/yearManual FeedManual Feed
100 GB Cartridges100 GB Cartridges
HPHPK260K260
HPHPK260K260
HPHPK260K260
HPHPK260K260
HPHPK260K260
FDDIFDDI
600 GB600 GB1 Run1 Run
PCPCPCPCPCPCPCPCPCPCPCPC
100BT
100BT
SGISGIChallengeChallenge
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 7C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Linux Farms (NA48)
NA48 was using the QSW CS/2 (128 proc.)
CS/2 overload -> investigate PC’s in late 97
Installation of 12 Dual machines in 1Q98 and more ...
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 8C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
Linux Issues
EEPRO 100 B MP crashes AFS support (MP) NFS support (MP) Commercial software Manufacturer support for Linux Very few Linux experts
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 9C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
NT offline Farms
PCSF• Simulation facility but …
COMPASS• Evaluating & benchmarking technology
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF - Overview
Configuration Applications Data access Specific work & solutions Key issues Conclusions
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
1
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF - Goals
Make PC+NT a standard option for Physics Data Processing, starting with simulation
Establish a minimum management model for NT farm management
Address scalability issues Gain Windows NT experience
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF Milestones
Joined RD47 in Autumn 96 Price inquiry issued in 12/96 Hardware delivered 4/97 Ready to use 6/97 RD47 report 10/97 Expansion 5/98
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF Configuration (1) Server running NT 4.0 Server SP3
• 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB data disk (with mirroring). LSF central queues.
Server running NT Terminal Server Beta 2• 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data
disk. Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access
Servers running NT 4.0 Workstation SP3• 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB • 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB
All equipped with boot proms
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF Configuration (2)
Machines interconnected with 4 3com 3000 100BaseT switch
Display/Keyboard/Mouse connected to a Raritan multiplexor
PC Duo for remote admin access There were problems with other products All running LSF 3.0. LSF 3.2 does not work, support weak Completely integrated with NICE
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Applications on PCSF
ATLAS Dice simulation NA45 1996 reconstruction CMS reconstruction with Objectivity being
tested LHCB simulation code ready ATLAS reconstruction being ported ATLAS/Marseille event filter prototype
scalability tests
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Data access
NT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PC
Network
Network
Unix RFIOUnix RFIOServerServer
Unix RFIOUnix RFIOServerServer
Unix RFIOUnix RFIOServerServer
Unix RFIOUnix RFIOServerServer
Unix TapeUnix TapeServerServer
stagexxx commandsstagexxx commands
RFIORFIO
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
ATLAS Level 3 DAQ
Processor FarmProcessor Farm
Event BuilderEvent Builder
SFISFISFISFISFISFI
Storage (100 MB/s)Storage (100 MB/s)
Readout BuffersReadout Buffers
1 GB/s1 GB/s
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
ATLAS Event Filter
Testbed for evaluating algorithms & sizing
Architecture & simulation studies Monitoring, system management,
feedback, etc… Interface prototypes (SFI, SFO) Timescale : prototype -1 (I.e. end 98) Status : sizing of an initial farm
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1
9
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PCSF Usage
0
1000
2000
3000
4000
5000
6000
7000
8000
43 45 47 49 51 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Week #
NC
U h
ou
rs
Idle
Used
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
1
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Specific work so far
Installation (Remote Boot, Winstall, NICE replica’s, Install Server)
User codes, CERNLIB, SHIFT Job Starter PC MGR WNTS Web Interface
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Installation Disk cloning + change SID Fastest method, but not very
automated Remote boot
• Remote boot install procedures with virtual disk
• Use unattended setup, installs Winstall and other things
• Third party packages installed through Winstall
boot prom support on some hardware
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Porting
Usually porting code from Unix to NT is easy (NA45 code ported in 1 week)
Usually porting production environment from Unix to NT is difficult (shell scripts)
Porting build environment is difficult, better to use native tools (Dev Studio)
Mixing Unix and NT build environment, revision control, etc.
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Jobstarter
Initially inherited from Unix LSF CERN JobStarter
Rewritten in C++, using PcMgrSvc for drive mapping
Check execution preconditions Clean up normal and abnormal job end Kill popup dialog windows Excel & Winzip in batch
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PcMgrSvc/Ctl
Checks• Status of monitored processes/services• Amount of scratch space• Drive mapping(s)
Map/Unmap drives Sync. with time servers Generate alarms on request Gets all parameters from registry
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Web Interface
As a solution to• Remote access from outside CERN• Access from non NT hosts
Implemented as ASP’s with VB Requires IIS on the server
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s Web Interface -
authentication
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Web Interface - Overview
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2
9
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Web Interface - bjobs
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Web interface - bjobs result
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
1
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Windows NT Terminal Server
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Next Steps
Finish and understand remote boot issues Complete remote boot - remote install AFS Integration Build up resilience Investigate how to use the new WfM, DMI,
PXE, ACPI, etc. initiatives Investigate whether WSH is an alternative Investigate NT’s I/O capabilities
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Key Issues
AFS access LSF support Boot proms, equipment interoperability CODE reintegration (Physics & CERNLIB) Think Windows Scalability & Management (home grown
solution vs. commercial apps.) Remote & external access
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC with NT
PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing
Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed
Scalability has started to be addressed, but the relatively small number of nodes does not help here
Considerable NT experience has been gained
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Issues so far
Linux• EEPRO 100 B MP support• Commercial software• Manufacturer support• Very few local Linux experts
NT• AFS access• LSF support
• Think Windows• Remote and external access
PC• Interoperability (cards/MB combination• Remote Boot support
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC Technology evolution in 97 Pentium Pro Pentium II
• 50 % raw performance increase• but 50 % cache performance reduction
SEC new motherboards 440 FX 440 LX (SDRAM, AGP) Recent MB’s embedded SCSI, E’net, VGA 100 Mbit E’net switches standard, 1000
Mbit arriving
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s PC Technology evolution in
98 Pentium II @300 MHz Pentium Xeon @ 450 MHz
• MP support• 50 % cache performance increase
Slot 2 new motherboards 440 LX 440 BX, 440 NX (100 MHz, EDO) Recent MB’s No more available through Intel,
TYAN 1000 Mbit/s E’net switches standard, >> 1000
Mbit/s arriving
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 3
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Racking evolution
1997 1998
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Fast Ethernet Switches (Oct. 98)
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
At the back of Fast Ethernet Switches (Oct. 98)
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Gigabit Ethernet Switches
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Network performance: Results PC’s interconnected through 100 BaseT 3Com
3000 switch Repeated with other H/W Half duplex behavior Block size does not matter Linux uses less CPU than NT
Good unidirectional performance Disappointing CPU consumption on NT
Disappointing bi-directional performance
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC to PC Network performance
Linux Windows NT Max
MB/s CPU % MB/s CPU % MB/s
1->1 9->11 17 9->11 40 12.5
1<-> 1 4.6 10 5.3 44 12.57.5 21 5.2 44 12.5
12.1 10.5 251->3 11.7 55 12.5
3.9 20 4.23.9 20 4.23.9 20 4.2
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Network performance: issues
Unexplained 0.5 MB/s observed with some eepro100 versions on PCRD hardware, but OK on PCSF
Recent DEC E'net boards with chipset > 21140 give poor performance on Linux
Surprising results PC/Alpha
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC/Alpha Network performance
Linux Alpha DUX Max
MB/s MB/s MB/s
1->1 11.1 11.1 12.5
1<-> 1 6.7 11.1 12.511.1 6.7 12.517.8 17.8 25
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 4
9
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC High Performance Networking
HiPPI (5/98) PII, 300 MHz,
440LX, SDRAM, Roadrunner to SGI O2000, 4 CPU, IRIX 6.4
Transmit: 50 MB/s Receive: 50 MB/s
(53 MB/s with SMP)
Gigabit Ethernet (10/98)
PII, 400 MHz, 440 BX, 100 MHz SDRAM, PCI 32/33, Tigon I
1500 bytes/packet: 28 MB/s, 40% CPU
9000 bytes/packet, 90 MB/s, 90% CPU
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Disk performance PC’s connected to SEAGATE ST19171W using
two Adaptec 2940 UW NT needs a lot of tuning (default behavior is to
swap data out!) Block size, BIOS settings, EDO/FPM does not
matter Poor performance
Windows NT even worse Memory bandwidth is suspected
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
1
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Disk performance
# Streams Linux Windows/NT MaxMB/s CPU % MB/s CPU % MB/s
1 10.5 33 8.5 35 112 21 63 9.2 35 703 21 100 13.5 60 70
• Striping has no effect
•1 stream 2 stripes : 21 MB/s (22 max)
•1 stream 3 stripes : 21 MB/s (33 max)
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Disk performance: issues
Memory bandwidth suspected Need to test with LX/SDRAM, BX
SDRAM@100 Mhz RISC PCI does not support variety of
boards Combined disk/network performance
even worse : 5-6 MB/s on Linux
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Memory bandwidth (lmbench)
PCRD IBM DEC Prioris DEC PWS
Read MB/s 160 160 216 190Write MB/s 55 55 69 190
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Memory bandwidth (lmbench)
0
50
100
150
200
250
300
350
MB/s
Ta
ho
e2
DK
440
LX
Th
un
de
r2
Tig
er2
GA
686
DL
X
GA
686
(CP
U1
)
GA
686
(CP
U2
)
DE
C P
WS
43
3
SU
N U
ltra
5
Th
un
de
r10
0
N4
40
BX
Ka
ya
k X
A's
Co
mp
aq
Pro
lia
nt
16
00
Equipment
Mem read
Mem write
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Technology issues
Technology evolves too fast (processors, chipsets, memory, motherboards, networking,...)• Changing environment/interoperability issues• Hard to maintain (obsolescence)• New NIC’s, drivers• Measurements valid only a few months Difficult to establish stable environments
Wide variety of solutions Some combinations work, other not
Local suppliers cannot help to solve problems
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC Performance summary CPU performance fine Network performance
• Some configurations do not work• Some configurations can saturate Fast Ethernet• Recent tests show excellent performance
Memory performance• Now better than low-end RISC
Disk Performance disappointing Linux better than NT
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Online and quasi online farms
NA48 Data Recording NA45 Data Recording in Objectivity
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA48 Central Data Recording
Cisco 5505Cisco 5505
3Co
m 3900
3Co
m 3900
FDDIFDDI
Fast EthernetFast Ethernet
Fast EthernetFast Ethernet
XLNT GbitXLNT Gbit
FDDIFDDI
HiPPIHiPPI
GigaRouterGigaRouter
3Com 93003Com 9300Gigabit EthernetGigabit Ethernet
HiPPIHiPPI
CS/2CS/22.5 TB Disk space2.5 TB Disk space
SUN E450SUN E450500 GB Disk space500 GB Disk space
Event BuilderEvent BuilderOnline PC FarmOnline PC Farm
Sub detectorSub detectorVME cratesVME crates
7 KM7 KM
OfflineOfflinePC FarmPC Farm
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 5
9
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA 48 Data Recording in 98 May September 1998 Raw Data on Tape
• 68 TB (1450 tapes, mainly 50 GB tapes)• 12.5 TB Selected Reconstructed Data• Total with 97 data : 96 TB
Average Data Rate : 18 MB/s (peaks @ 23 MB/s)
CDR system can do 40-50 MB/s; limitation is CPU Time available
Data recorded as files (4 million)
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA48 On Line Farm
11 Subdetector PC’s (dual PII-266, 128 MB) 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB
SCSI) 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI) All running Linux Software event building in the interburst gap Optional Software Filter (tags data) Send data to computer center (local disk buffers :
144 GB , 2 hours) On CS/2 : L3 Filtering and tape writing
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
1
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA48 Plans for 1999
Fast EthernetFast Ethernet
Gigabit EthernetGigabit Ethernet
HiPPIHiPPI
4 * SUN E4504 * SUN E4504.5 TB Disk space4.5 TB Disk space
EventEventBuilderBuilder
Sub detectorSub detectorVME cratesVME crates
7 KM7 KM
3Co
m 3900
3Co
m 3900
HiPPIHiPPI3Com 93003Com 9300
Gigabit EthernetGigabit Ethernet
Fast EthernetFast Ethernet
Cisco 5505Cisco 5505
On/OfflineOn/OfflinePC FarmPC Farm
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
2
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA45 Data Recording
Fast EthernetFast Ethernet
Gigabit EthernetGigabit Ethernet
HiPPIHiPPI
2 * SUN E4502 * SUN E450500 GB Disk space500 GB Disk space
Event BuilderEvent BuilderOn Line PC FarmOn Line PC Farm
Sub detector VME cratesSub detector VME crates
7 KM7 KM
3Co
m 3900
3Co
m 3900
HiPPIHiPPI
Gigabit EthernetGigabit Ethernet
Fast EthernetFast Ethernet
SCISCI
3Com 39003Com 3900
3Com 93003Com 9300
NA48NA48
PCSFPCSF
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
3
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
NA45 Raw Data recording in Objectivity October 98 ; November 98 Estimated bandwidth : 15 MB/s Processes translate Raw Data format to Objectivity Database files (1.5 GB) are closed, then written on
tape Steering done using a set of perl scripts on the disk
servers On line filtering/reconstruction/calibration possible Farm is running Windows NT Reconstruction can use PCSF
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
4
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s Current & Future Data rates at
CERN
Year Experiments BandwidthMB/s
Raw DataTB/year
ProcessingSPECInt95
1990-2000
LEP 0.5 1 100
1997-2000
SPS 15-20 30-70 500
2000-2008
SPS 35 300 2000
2004- LHC 100-1000 3000 50000
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
5
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
Summary
On line PC farms are being used to record data at sensible rates (Linux)
Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)
Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
6
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
PC Total Cost of Ownership
42%
51%
4%1%2%
HW
MUX
Rack
Network
Sysadm
• Software not included
• Install labor not included
• Assumes 3 years lifetime
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
7
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
DEC 8400 (12-Way) Cost of Ownership
• Software & SW maintenance not included
• Assumes 5 years lifetime
72.8%
0.1%13.8%0.9%
12.4%
HW
MUX
HW Maint
Network
Sysadm
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
8
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
General Conclusions (1)
PC’s are now used for online, quasi online and offline environments
The “offline” is now part of the online The I/O is still done using RISC/Unix
but recent MP Xeon may change this …
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 6
9
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
General Conclusions (2)
PC technology is moving very fast• Good for performance• Not so for stability, interoperability• Not so for understanding issues
The general management of large farms is not solved but …• Number of initiatives/standards/tools may
help us here : WfM, DMI, PXE, ACPI, SMS, TNG, etc.
DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 7
0
CE
RN
- E
uro
pea
n L
abo
rato
ry f
or
Par
ticl
e P
hys
ics
C
ER
N -
Eu
rop
ean
Lab
ora
tory
fo
r P
arti
cle
Ph
ysic
s
General Conclusions (3)
Linux vs. NT … the battle is over• Choose the one suitable to your
application• NT can be used• Linux is usable (and offers more
performance). PC real costs are usually not well
understood