Evaluation of the LDC Computing Platform for Point 2 SuperMicro X7DBE-X
-
Upload
ursa-deleon -
Category
Documents
-
view
14 -
download
0
description
Transcript of Evaluation of the LDC Computing Platform for Point 2 SuperMicro X7DBE-X
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs1
Evaluation of the LDCComputing Platform for Point 2
SuperMicro X7DBE-X
Andrey ShevelCERN PH-AID
ALICE DAQ (Internal report)CERN
November 2006
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs2
Purpose
Background: Test the machine Elonex Supermicro based on
motherboard X7DBE-X (2xDual Core Xeon 3.0 GHz) as LDC with 6 D-RORCs
Evaluation program: Linux installation. DATE V5 installation. D-RORC throughput with DATE. Perform a range of runs with internal D-RORC
generator and receiving through DATE. Perform a range of long runs (two days each) in
configuration with the test machine as LDC and another machine as GDC.
Quick comparison with another machine based on motherboard X6DHE-XB (Dual Xeon 2.8 MHz).
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs3
Architecture of motherboard
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs4
Supermicro X7DBE-X
4U (tower)Chassis
2x USB front, 2x USB back
Serial, PS/2Periphery
ATI Technologies Inc ES1000Graphics
120 GB, SATADisk
2x GbE, Intel 631xESC/632xESBNetwork
3x 64-bit 133MHz PCI-X, 3x PCI-X (64-bit 133 MHz)3x PCI-X (64-bit 100 MHz)
I/O
2 GB (max 32GB), 533 MHz,DDR2Memory
1333 MHzFSB
Intel 5000P (BlackFord) Chipset
2 x Dual Core Intel Xeon, cache L2 – total 4 MB
CPU
pcald44.cern.c
h
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs5
Software Installation
Linux SLC4.4 with kernel 2.6.9-42.0.3.EL.cernsmp
no specific problems: USB boot successful, BIOS – everything is OK To use D-RORC modules the BIOS setting is needed to be
changed: all PCI slots were set to 100 MHz DATE 5.25 installation
MySql mysqltcl BWidget SMI DIM Drivers: rorc and physmem
have been installed as rpm with additional scripts: DATE_basic_Install-SLC4x.bash, DATE_User_Install-SLC4x.bash, DATE_DriversInstall.bash
No specific problems
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs6
Common info about measurement
Total main memory volume = 2 GB• 1 GB for Linux (in /etc/grub.conf)• 1 GB for physmem
6 cards D-RORC rev 4 (internal data generator). All measurement were done with DATE 5.25. No
received event data were recorded to HD. The data from the database were analyzed with scripts. The pictures were built up with gnuplot.
In total there were performed 390 measurement runs. Because the figures (rates, bandwidth) are fluctuating every run was continued at least 5 minutes.
Two points: random event size with max=10**4 and max=10**6 Bytes were measured during two days each (LDC on test machine, GDC on different machine).
All measurements with randomly distributed event size were performed with event checking in DATE.
D-RORC Page Size was 10**5 Bytes.
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs7
Event rate of one card
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs8
Bandwidth of readout of one card
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs9
Example of editDb page
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs10
Example of Memory Banks page
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs11
Event rate for several cards
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs12
Bandwidth for several cards
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs13
Event rate: ConsistencyCheckLevel
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs14
Bandwidth: ConsistencyCheckLevel
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs15
Event rate: NumberOfCards, ConsistencyCheckLevel
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs16
Bandwidth: NumberOfCards, ConsistencyCheckLevel
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs17
Event rate: D-RORC Page Size
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs18
Bandwidth: D-RORC Page Size
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs19
X6DHE-XB vs X7DBE-X
Parameter name
X6DBE-XB X7DBE-X
CPU type2x Xeon 2.8 GHz, 1 MB 2x Xeon 5160 dual core, 3.0 GHz, 4
MB L2 cache
Chipset E7520 Intel 5000P (BlackFord)
FSB 800 MHz 1333 MHz
Memory 2 GB, DDR 333 2 GB, DDR-2 533 MHz
PCI I/O2x PCI-X (64/133)4x PCI-X (64/100)1x PCI-e (x4)
3x PCI-X (64-bit 100 MHz)
3x PCI-X (64-bit 133 MHz)
Network 2x GbE, BCM5721 2x GbE, Intel 631xESB/632xESB
SYSBENCH Memory bandwidth
1.5 GB/s 3.4 GB/s
RootMarks 614 1500
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs20
X6DHE-XB vs X7DBE-X (continuation)
Parameter name
X6DBE-XB X7DBE-X
Event rate for 1 D-RORC with the size 1 KB
88.9 KHz 87.2 KHz
Event rate for 6 D-RORCs with size 1 KB
6 KHz 12.2 KHz
Bandwidth of readout for 6 D-RORCs with size 10**6 Bytes
1.58 GB/sec 1.98 GB/sec
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs21
Recommendation and conclusion
Recommendation: if you need to allocate D-RORC cards in the
machine and like to have maximum bandwidth for X7DBE-X:
• it is better to avoid to use PCI slot 4 & PCI slot 5 at the same time for high
bandwidth information sources.
Conclusion In the test it was shown bandwidth (1.96 GB/sec) when all 6 D-RORC cards are in
operation and event fragment size is more 10**5 bytes when ConsistencyCheckLevel=0.
X7DBE-X shows increased event rate and bandwidth with 6 D-RORCs, also it has more CPU power in comparison to X6DHE-XB.
The configuration LDC on test machine and GDC on another machine shown about the speed 118 MB/sec with no error during two days test.
Randomly distributed event fragment size with maximum 10**6 bytes gives total bandwidth 1.53 GB/sec during two days run.
The machine on the base X7DBE-X is excellent to be used as LDC with 6 D-RORC cards.
4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs22
Thank you!
Klaus Schossmaier, Sylvain Chapeland, Ulrich Fuchs and
PH/AID crew