DONVITO Giacinto (INFN) ZANGRANDO, Luigi (INFN) SGARAVATTO, Massimo (INFN) REBATTO, David (INFN)
INFN-GRID-WP2.4: Computing Fabric & Mass Storage
description
Transcript of INFN-GRID-WP2.4: Computing Fabric & Mass Storage
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 11
LNL
INFN-GRID-WP2.4:INFN-GRID-WP2.4:Computing Fabric Computing Fabric & Mass Storage& Mass Storage
LNLLNLL. BertiL. BertiM. BiasottoM. BiasottoM. GulminiM. GulminiG. MaronG. MaronN. TonioloN. Toniolo
PadovaPadovaS. BalsamoS. BalsamoM. BellatoM. BellatoF. CostaF. CostaR. FerrariR. FerrariM. MichelottoM. MichelottoI. SaccarolaI. SaccarolaS. VenturaS. Ventura
CNAFCNAFA. ChiericiA. ChiericiL. Dell’AgnelloL. Dell’AgnelloF. GiacominiF. GiacominiP. MatteuzziP. MatteuzziC. VistoliC. VistoliS. ZaniS. Zani
LecceLecceG. AloisioG. Aloisio M. CafaroM. Cafaro Z. ZzzzZ. Zzzz L. DepaolisL. Depaolis S. Campeggio S. Campeggio E. FasanelliE. Fasanelli
TorinoTorinoA. ForteA. Forte
GenovaGenovaG. ChiolaG. ChiolaG. CiaccioG. Ciaccio
Roma 1Roma 1D. AnzellottiD. AnzellottiC. BattistaC. BattistaM. De RossiM. De RossiF. MarzanoF. MarzanoS. FalcianoS. FalcianoA. SpanuA. SpanuE. ValenteE. Valente
BolognaBolognaG.P. SiroliG.P. SiroliP. MazzantiP. Mazzanti
CataniaCataniaC. RoccaC. RoccaE. CangianoE. CangianoTecnologoTecnologo
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 22
LNLTerminologyTerminology
PCPC
LANLAN
PC network PC network + midleware+ midleware
= PC Farm/Fabric= PC Farm/Fabric PC ClusterPC Cluster
LALANN
LALANN
LALANN
LALANN
LALANN
LALANN
PC Farm network PC Farm network + midleware+ midleware
= = GRIDGRID
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 33
LNLWhy this WP?Why this WP?
Commodity components like PCs and LANs are mature to form Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabricinexpensive and powerful Computing Fabric
Computing Fabrics located in different sites are integrating to Computing Fabrics located in different sites are integrating to form a Computational/Data GRIDform a Computational/Data GRID
ButBut
– How to design a fabric of 1000s nodes balancing computing How to design a fabric of 1000s nodes balancing computing power and efficient storage access?power and efficient storage access?
– How to control and monitor the basic system components?How to control and monitor the basic system components?
– How to “publish” the monitored values to the GRID monitor How to “publish” the monitored values to the GRID monitor system?system?
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 44
LNLComputing Fabric WPComputing Fabric WP
This WP wants to address the mentioned problems adding a technological tracking This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic constituents task to follow and to test, with real use case, the evolution of the basic constituents of a fabric.of a fabric.
WP Break DownWP Break Down– Fabric Design (INFNGrid Wp 2.4)Fabric Design (INFNGrid Wp 2.4)
Overall architecture and fabric setupOverall architecture and fabric setup LAN and SAN (System Area Network) technologiesLAN and SAN (System Area Network) technologies Communication protocols for high speed network fabricsCommunication protocols for high speed network fabrics Storage Systems Storage Systems Microprocessor TechnologyMicroprocessor Technology
– Fabric Management (DataGrid WP4)Fabric Management (DataGrid WP4) Configuration management and automatic software installationConfiguration management and automatic software installation System monitoringSystem monitoring Dynamic System PartitionDynamic System Partition Problem managementProblem management
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 55
LNLFabric Design - INFNGrid WP 2.4 - InstitutionsFabric Design - INFNGrid WP 2.4 - Institutions
Dipartimento di Scienze dell’Informazione - Universita’ di Dipartimento di Scienze dell’Informazione - Universita’ di VeneziaVenezia
Dipartimento di Scienze dell’informazione - Universita’ di Dipartimento di Scienze dell’informazione - Universita’ di GenovaGenova
Ingegneria Informatica – Universita’ di LecceIngegneria Informatica – Universita’ di Lecce
8 sezioni INFN, 1 Laboratorio Nazionale8 sezioni INFN, 1 Laboratorio Nazionale
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 66
LNLFabric Management - DataGrid WP4 - InstitutionsFabric Management - DataGrid WP4 - Institutions
CERNCERN
Konrad Zuse Zentrum (Berlin)Konrad Zuse Zentrum (Berlin)
Kirchhoff Institute (Heidelberg)Kirchhoff Institute (Heidelberg)
IN2P3 (Lyon)IN2P3 (Lyon)
INFNINFN
NikhefNikhef
RALRAL
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 77
LNLFabric Design Detailed Program (2000-2001)Fabric Design Detailed Program (2000-2001) Fabric ArchitectureFabric Architecture
– Network topologiesNetwork topologies– Data Server connections and network file systemsData Server connections and network file systems– System break downSystem break down
Interconnection NetworksInterconnection Networks– 100/1000 Ethernet100/1000 Ethernet– MyrinetMyrinet– InfinibandInfiniband
Communication protocols for high speed network fabricsCommunication protocols for high speed network fabrics
Storage SystemsStorage Systems– Ultra SCSI (160/320/…)Ultra SCSI (160/320/…)– Ultra and Serial ATAUltra and Serial ATA– Storage Area Network (SAN)Storage Area Network (SAN)
Fibre Channel Fibre Channel Microprocessor TechnologyMicroprocessor Technology
– Dual slim processorsDual slim processors– IA64IA64
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 88
LNLFabric Design Deliverables Fabric Design Deliverables
DDeelliivveerraabbllee DDeessccrriippttiioonn DDeelliivveerryy DDaattee DD22..44..11 RReeqquuiirreemmeennttss ggaatthheerriinngg ffrroomm
ffaabbrriicc uusseerrss 33 MMoonntthhss
DD22..44..22 SSuurrvveeyy ooff eexxiissttiinngg ffaabbrriicc//cclluusstteerr aarrcchhiitteeccttuurreess,, aarrcchhiitteeccttuurraall iissssuueess ttoo ddeessiiggnn aa ffaabbrriicc ooff ccoommmmooddiittyy ppccss
66 ++ 66 MMoonntthhss
DD22..44..33 IINNFFNN ttoooollkkiitt ttoo ddeessiiggnn aanndd rreeaalliizzee llooccaall ffaabbrriiccss
1122 MMoonntthhss
DD22..44..44 IInntteeggrraattiioonn ooff tthhee DDaattaaGGrriidd ffaabbrriicc mmaannaaggeemmeenntt ssooffttwwaarree ttoooollss
2244 MMoonntthhss
DD22..44..55 IInntteeggrraattiioonn ooff tthhee FFaabbrriicc MMaannaaggeemmeenntt iinnttoo tthhee GGRRIIDD eennvviirroonnmmeenntt
3366 MMoonntthhss
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 99
LNLFabric Design MilestoneFabric Design Milestone
MMiilleessttoonnee DDeessccrriippttiioonn SSIITTEE SSttaarrtt DDaattee
EEnndd DDaattee
MM22..44..11 NNFFSS ccoonnnneeccttiioonn ttooppoollooggyy:: ddiisskk sseerrvveerr bbaasseedd
LLNNLL 99//22000000 33//22000011
MM22..44..22 NNFFSS ccoonnnneeccttiioonn ttooppoollooggyy:: ddiissttrriibbuutteedd ddiisskkss
CCNNAAFF//BBOO 99//22000000 33//22000011
MM22..44..33 MMiiccrroopprroocceessssoorr TTeecchhnnoollooggyy:: dduuaall sslliimm pprroocceessssoorrss
CCNNAAFF//BBOO 99//22000000 33//22000011
MM22..44..44 SSttoorraaggee SSyysstteemmss:: FFiibbrree CChhaannnneell -- SSCCSSII UUllttrraa 116600//332200
PPAADDOOVVAA 99//22000000 33//22000011
MM22..44..55 SSttoorraaggee SSyysstteemmss:: SSeerriiaall AATTAA PPAADDOOVVAA 55//22000011 77//22000011 MM22..44..66 MMiiccrroopprroocceessssoorr TTeecchhnnoollooggyy:: IIAA6644 PPAADDOOVVAA 1111//22000000 33//22000011 MM22..44..77 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: MMyyrriinneett
++ IIAA6644 LLEECCCCEE 99//22000000 1122//22000011
MM22..44..88 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: GGiiggaabbiitt EEtthheerrnneett aanndd eeffffiicciieenntt ccoommmmuunniiccaattiioonn pprroottooccoollss
GGEENNOOVVAA//RROOMMAA11
99//22000000 55//22000011
MM22..44..99 SSeettuupp ooff aa cclluusstteerr ooff mmoorree tthhaann 110000 nnooddeess
LLNNLL 11//22000011 1122//22000011
MM22..44..1100 IInntteerrccoonnnneeccttiioonn NNeettwwoorrkkss:: IInnffiinniibbaanndd
LLNNLL 55//22000011 1122//22000011
MM22..44..1111 IInnssttaallllaattiioonn ooff tthhee IINNFFNN ffaabbrriiccss AAllll 22000022 22000033 MM22..44..1122 TTIIEERR 00 IInntteeggrraattiioonn AAllll 22000033 22000033
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1010
LNL DATA GRID Fabric Management WP4DATA GRID Fabric Management WP4
Automatic Software Installation and Management (OS and Automatic Software Installation and Management (OS and Applications)Applications)
Configuration ManagementConfiguration Management
System MonitoringSystem Monitoring
Problem ManagementProblem Management
Local Authorization Services and Grid IntegrationLocal Authorization Services and Grid Integration
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1111
LNLWP4 Fabric Managemet DeliverablesWP4 Fabric Managemet Deliverables
Requirements document and survey of existing tools and Requirements document and survey of existing tools and technologies (month 6)technologies (month 6)
A configuration and installation management demonstrated to A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12)work on a cluster of more than 100 nodes (month 12)
A fully deployed service level monitoring system for a computer A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions like policies and quality measures, to allow schedule decisions (month 24)(month 24)
A fully integrated system to accept remote resource requests in A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting information about progress og requests, and final accounting report back to the sender (month 36)report back to the sender (month 36)
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1212
LNLNFS Disk Servers Based Farm (Legnaro)NFS Disk Servers Based Farm (Legnaro)
Disk Server:Disk Server:-Dual PIII 800 MHzDual PIII 800 MHz- Mem 512 MBMem 512 MB- SCSI AdaptersSCSI Adapters
Ethernet Switch:Ethernet Switch:- 10 FE10 FE- 1 Uplink GE1 Uplink GE
Computational Nodes:Computational Nodes:- Dual PIII 800 MHz (40+40 SI95)Dual PIII 800 MHz (40+40 SI95)-Mem 512 MBMem 512 MB
N1N1SSWWIITTCCHH
N10N10
GEGE
FEFEN2N2
Farm ModuleFarm Module
LNL is testing this farm module (with PIII at 450 MHz)LNL is testing this farm module (with PIII at 450 MHz)
DISKDISKSERVERSERVER
This Farm has been fundedThis Farm has been fundedBy Comm Calcolo for LNLBy Comm Calcolo for LNLOff-line computation (gr. 2/3)Off-line computation (gr. 2/3)
Requests for 2000:Requests for 2000:- 2 Raid SCSI Controllers - 2 Raid SCSI Controllers 10 Ml10 Ml
ANISANISBootBoot
ServerServer
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1313
LNLLNL PC Farm (phase I – May 200)LNL PC Farm (phase I – May 200)
N1N1SSWWIITTCCHH
N10N10
GEGE
FEFEN2N2
Farm Module IFarm Module I
DISKDISKSERVERSERVER
FrontEndFrontEnd(ANIS)(ANIS)
ASUS P3BFASUS P3BF440 BX chipset440 BX chipset1 PIII 450 MHz1 PIII 450 MHzPCI 33/32PCI 33/32
SuperMicro PIIIDMESuperMicro PIIIDME840 chipset840 chipset1-2 PIII 600 MHz1-2 PIII 600 MHzPCI 33/32-66/64PCI 33/32-66/64
ExtremeNetworksExtremeNetworksSummit 48Summit 4848 Fe48 Fe2 Ge2 Ge
2 ULTRA 160 SCSI2 ULTRA 160 SCSIADAPTEC 39160ADAPTEC 39160
Problemi con Problemi con Memoria SDRAMMemoria SDRAM
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1414
LNL
DISKDISKSERVERSERVER
LNL PC Farm (phase II – Nov 2000)LNL PC Farm (phase II – Nov 2000)
N1N1SSWWIITTCCHH
N15N15
GEGE
FEFEN2N2
Farm Module IFarm Module I
DISKDISKSERVERSERVER
SuperMicro 370DLESuperMicro 370DLEServerWorks III LE chipsetServerWorks III LE chipset2 PIII 800 MHz2 PIII 800 MHz512 Mbyte 512 Mbyte PCI 33/32-66/64PCI 33/32-66/64Case minitowerCase minitower
ExtremeNetworksExtremeNetworksSummit 48Summit 4848 Fe48 Fe2 Ge2 Ge
Enclosure Compaq 4354REnclosure Compaq 4354R14 dischi 36GB SCSI U314 dischi 36GB SCSI U3500 Gbyte Storage Array500 Gbyte Storage Array
Raid Controller U3Raid Controller U3CompaqCompaqSmartArray 5300SmartArray 5300
Case Test: SuperMicro SC810Case Test: SuperMicro SC810
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1515
LNL Low Price 1U case example: SM SC810Low Price 1U case example: SM SC810
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1616
LNLCMS Fabric at LNL (2001)CMS Fabric at LNL (2001)
N1N1N10N10FEFE
N2N2
Farm ModuleFarm Module
N1N1N10N10
FEFE
N2N2
Farm ModuleFarm Module
N1N1N10N10
FEFE
N2N2
Farm ModuleFarm Module
N1N1N10N10
FEFE
N2N2
Farm ModuleFarm Module
DISKDISKSERVERSERVER
DISKDISKSERVERSERVER
DISKDISKSERVERSERVER
DISKDISKSERVERSERVER
GigaEthernet SwitchGigaEthernet Switch
SWITCHSWITCH SWITCHSWITCH SWITCHSWITCH SWITCHSWITCH
HP2524
HP8000 FastIron 4000
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1717
LNLLNL FarmLNL Farm
32 PCs, PIII 600 MHz, 3 Farm modules, 32 PCs, PIII 600 MHz, 3 Farm modules, 120 Gbps GE/Cu switch, 120 Gbps GE/Cu switch, 500 Gbyte disk500 Gbyte diskUsed for:Used for:- Sadirc/CMS Event Builder prototypesSadirc/CMS Event Builder prototypes- Data Analysis Production for LNL expsData Analysis Production for LNL exps- NFS and topological tests - NFS and topological tests
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1818
LNLNFS Distributed Disks based Farm (CNAF)NFS Distributed Disks based Farm (CNAF)
SSWWIITTCCHH
N10N10
FEFEN2N2
Farm ModuleFarm Module
ANISANISBootBoot
ServerServer
N1N1
- Topology testsTopology tests- dual slim processors (rack mounted) testsdual slim processors (rack mounted) tests- remote file systems (nfs, ams, etc. ) testsremote file systems (nfs, ams, etc. ) tests
Requests for 2000:Requests for 2000:- 10 dual slim processors10 dual slim processors 90 Ml 90 Ml- Fast ethernet switchFast ethernet switch 10 Ml 10 Ml- 10 SCSI disks 36/72 GB 10 SCSI disks 36/72 GB 20 Ml 20 Ml ------------
120 Ml120 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1919
LNLStorage Systems: FC and SCSI road mapStorage Systems: FC and SCSI road map
SCSISCSI
Ultra3/Ultra160Ultra3/Ultra160160 MB/sec160 MB/sec
Ultra2Ultra280 MB/sec80 MB/sec
Wide UltraWide Ultra40 MB/sec40 MB/sec
Ultra320Ultra320320 MB/sec320 MB/sec
Ultra640Ultra640640 MB/sec640 MB/sec
20042004
20032003
20022002
20012001
20002000
19991999
19981998
19971997
19961996
19951995
Fibre ChannelFibre Channel
1Gb1Gb100 MB/sec100 MB/sec
2Gb2Gb200 MB/sec200 MB/sec
400MB/sec (full duplex)400MB/sec (full duplex)
8/10Gb8/10Gb1000 MB/sec1000 MB/sec
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2020
LNLStorage Systems: Fiber Channel/SCSI evaluation (PD)Storage Systems: Fiber Channel/SCSI evaluation (PD)
N1N1FFCC
SSWWTT
N4N4
FCFCN2N2
N3N3
N1N1
Arbitrated loop (AL) FCArbitrated loop (AL) FC
Switched FCSwitched FC
Requests for 2000Requests for 2000- 4 Disk Servers4 Disk Servers 30 Ml30 Ml- 1 8 ports FC Switch 1 8 ports FC Switch 20 Ml20 Ml- 4 FC adapters 15 Ml4 FC adapters 15 Ml- 2 Raid Arrays 30 Ml2 Raid Arrays 30 Ml- 4 Ultra/1604 Ultra/160 SCSI disksSCSI disks 8 Ml 8 Ml ------------
103 Ml103 Ml
N1N1
SCSISCSI
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2121
LNLStorage Systems: Serial ATA (Padova)Storage Systems: Serial ATA (Padova)
N1N1
SerialSerialATAATA
Requests for 2001Requests for 2001- 4 ATA Cntr.4 ATA Cntr. 5 Ml5 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2222
LNLInterconnection Networks: Gigabit Ethernet (Genova)Interconnection Networks: Gigabit Ethernet (Genova)
Communication Protocols for high speed networks:Communication Protocols for high speed networks:
- Communication System GAMMACommunication System GAMMA
- Use of Programmable GigaEthernet NICsUse of Programmable GigaEthernet NICs
- 1000 Base T GigaEthernet (on copper) reliability evaluation1000 Base T GigaEthernet (on copper) reliability evaluation
- Implementation of efficient parallel/distributed RAID systemsImplementation of efficient parallel/distributed RAID systems
Requests for 2000Requests for 2000- 8 PC8 PC 24 Ml24 Ml- 1 12 ports Geth Switch 1 12 ports Geth Switch 30 Ml30 Ml- 8 GEth NICs8 GEth NICs 16 Ml16 Ml- ------------TOTALTOTAL 70 Ml70 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2323
LNLInterconnection Networks: MyrinetInterconnection Networks: Myrinet
Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and interface portsinterface ports
Flow control, error control, and “heartbeat” continuity Flow control, error control, and “heartbeat” continuity monitoring on every linkmonitoring on every link
Low latency, cut-through crossbar switchesLow latency, cut-through crossbar switches
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2424
LNLInterconnection Networks: Myrinet (Lecce)Interconnection Networks: Myrinet (Lecce)
N1N1MMYYRRIINNEETT
N10N10
N2N2DISKDISK
SERVERSERVER
MMYYRRIINNEETT
N10N10
N2N2
N1N1Requests for 2000Requests for 2000- 10 Biproc Comp. Nodes10 Biproc Comp. Nodes 50 Ml50 Ml- 1 16 ports Myrinet Switch 1 16 ports Myrinet Switch 8 Ml 8 Ml- 10 Myrinet NICs10 Myrinet NICs 35 Ml35 Ml- 10 disks 72 GB 10 disks 72 GB 35 Ml 35 Ml- 1 disk server1 disk server 10 Ml10 Ml- Rack + cablesRack + cables 5 Ml 5 Ml- ------------TOTALTOTAL 143 Ml143 Ml
Myrinet based farmMyrinet based farm- Disk Server based - Disk Server based - Distributed disks based- Distributed disks based
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2525
LNLInterconnection Networks: InfiniBand (I)Interconnection Networks: InfiniBand (I)
TODAYTODAY
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2626
LNLInfiniBand (II)InfiniBand (II)
Legacy host architectureLegacy host architecture
The Infiniband ModelThe Infiniband Model
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2727
LNLInfiniBand (III)InfiniBand (III)
What?What?– Initial single link signaling rate of 2.5Gbaud Initial single link signaling rate of 2.5Gbaud
Means unidirectional transfer rate of 250MB/sec with a Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sectheoretical full duplex rate of 500MB/sec
– Initial support for single, 4, and 12 wide link widthsInitial support for single, 4, and 12 wide link widths– Point to point switched fabricPoint to point switched fabric– Message based with multicasting supportMessage based with multicasting support
MultiMultiStageStageSwitchSwitch
LinksLinksTCATCA
I/O ControllerI/O Controller
MemoryMemoryControllerController
PCI-X PCI-X HostHost
BridgesBridges
CPUCPU CPUCPU CPUCPU CPUCPU
HCAHCALinkLink
Fibre ChannelFibre Channel
SCSISCSI
Gig. EthernetGig. Ethernet
MemoryMemory
HCAHCA- Host Channel Adapter- Host Channel AdapterTCATCA - Target Channel Adapter - Target Channel Adapter
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2828
LNLInterconnection Networks: InfiniBand (LNL)Interconnection Networks: InfiniBand (LNL)
Simple test system (4 servers + Storafe Area Network + Simple test system (4 servers + Storafe Area Network + Network) for Network) for 20012001 is possible is possible
Early access to the productsEarly access to the products
Test Beds Test Beds
Requests 2001 (valuations) Requests 2001 (valuations) Comm V (Sadirc2000)Comm V (Sadirc2000)• 4 servers4 servers 50 Ml50 Ml• 1 IBA Switch1 IBA Switch 20 Ml20 Ml• IBA AdaptersIBA Adapters 10 Ml10 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2929
LNLMicroprocessor Technology: Intel Itanium (IA64)Microprocessor Technology: Intel Itanium (IA64)
Requests for 2001Requests for 2001
– Padova: Investigation IA64Padova: Investigation IA64 4 PC IA644 PC IA64 40 Ml 40 Ml
– Lecce: IA64 in MyrinetLecce: IA64 in Myrinet 10 Dual IA6410 Dual IA64 100 Ml100 Ml 1 Disk Server +1 Disk Server +
disksdisks 16 Ml 16 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 3030
LNLResourcesResources
SitesSites PersonsPersons FTEFTEMaterialsMaterials
(Ml)(Ml)2000 20012000 2001
Ital. Trav.Ital. Trav.(Ml)(Ml)
2000 20012000 2001
Inter.TravInter.Trav(Ml)(Ml)
2000 20012000 2001
LegnaroLegnaro 55 2.62.6 1010 8080
PadovaPadova 77 2.12.1 103103 4545
CNAFCNAF 66 1.71.7 120120 --BolognaBologna 22 0.60.6 -- --
LecceLecce 66 1.41.4 143143 116116
TorinoTorino 11 0.20.2 -- --GenovaGenova 22 1.01.0 7070 --
CataniaCatania 33 0.70.7 -- --
Roma 1Roma 1 77 1.51.5 ** **
TOTALTOTAL 3939 11.811.8 446446 161161241241