Sun sparc enterprise t5140 and t5240 servers technical presentation
Sun sparc enterprise t5440 server technical presentation
-
Upload
xkinanx -
Category
Technology
-
view
1.121 -
download
8
description
Transcript of Sun sparc enterprise t5440 server technical presentation
1
Sun SPARC Enterprise T5440Rack ServersTrain-the-Trainer Course
Presenter
Sun Microsystems, Inc.Sun ConfidentialCDA Required
Sun Internal and Approved Partners Only 2
NDA NotificationThis presentation contains information related to projects that are currently in planning and/or development stages. This information represents the current intentions of Sun Microsystems, Inc. However, all aspects of these projects, including, but not limited to funding, availability, shipping dates, configurations, capacities, performance, and all other characteristics are subject to change and/or cancellation without notice.This material is confidential to Sun Microsystems and should be disclosed to non-employees only under the terms of an executed confidential-disclosure agreement.Distribution of this material to Sun Field Representatives should be done through the GSO CDA process. Under no circumstances should these roadmaps be given directly to a Sun Field Representative.The information contained herein is considered privileged, confidential and protected from disclosure. If you are not the intended recipient or their employee/agent responsible for its delivery, you are requested to refrain from any duplication or distribution of this message or its contents, in respect of the rights of the designated recipient and sender and to delete it from your computer.
Sun Internal and Approved Partners Only 3
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC T2 Plus• Sun SPARC Enterprise T5440 server
> Overview of Features> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 4
Industry's Most Highly Threaded Servers Maximum Threading = Higher Throughput, Greater Energy & Space Efficiency
T5120/T5220Up to 64 Threads
T5140/T5240Up to 128 Threads
SB6320US T2 blade
T5440Up to 256 Threads
Sun Internal and Approved Partners Only 5
UltraSPARC T2: TrueSystem On a Chip
• Up to 8 cores @1.2GHz or 1.4GHz• Up to 64 threads per CPU • Up to 16 FB-DIMMs, 4 memory
controllers> Up to 64GB memory (4GB DIMMs)> 2.5x memory BW of UltraSPARC T1
• 8 x fully pipelined Floating Point units / core, 1 per core
• Dual 10Gbit Ethernet and PCI-E integrated onto chip
• 4MB L2 (8 banks) 16 way associative• Enhanced MAU/Security co-processor
per core> DES, 3DES, AES, RC4, SHA1,
SHA256, MD5, RSA to 2048 key, ECC,CRC32
• Advanced Power saving features• 65nm process technology
x8 @2.5GHz4GBytes/s bi-directional
Full Cross Bar
C0 C1 C2 C3 C4 C5 C6 C7FPU FPU FPU FPU FPU FPU FPU FPU
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
FB DIMM FB DIMM FB DIMM FB DIMM
FB DIMM FB DIMM FB DIMM FB DIMM
PCI-ENIU(E-net+)
Sys I/FBuffer Switch Core
2x 10GE EthernetPower ~95W<1.5W/thread
MCU MCU MCU MCU
MAU MAUMAUMAUMAUMAUMAUMAU
DMA Sub-System
Sun Internal and Approved Partners Only 6
UltraSPARC T2 ProcessorL2 Data
Bank 0
SPARC
Core 0
SPARC
Core 1
SPARC
Core 5
SPARC
Core 4L2 Data
Bank 1
L2 Data
Bank 4
L2 Data
Bank 5
L2 Data
Bank 7
L2 Data
Bank 6
L2 Data
Bank 3
L2 Data
Bank 2
L2B0
L2B1
L2B2
L2B3
L2B5
L2B4
L2B6
L2B7
SPARC
Core 2
SPARC
Core 3
SPARC
Core 7
L2TAG2
L2TAG3
L2TAG7
L2TAG6
L2TAG0
L2TAG1
L2TAG5
L2TAG4
MCU0
MCU1
MCU2
MCU3
DMU
PEU
RTX
RDP TDS
CCXSII
SIO
CCU
NCU
EFU
SPARC
Core 6
MACFSR
FSR
FSR
PSR ESR
For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4
SPC SPARCv9 CMT CoreCCX Core-Cache CrossbarL2$ Level 2 Cache BankCLC Coherence & Link ControllerMCU Memory Control UnitSIU System Interface UnitNCU Non-cacheable UnitMAC Media Access ControllerDMU Data Management UnitPEU PCI Express Unit
Sun Internal and Approved Partners Only 7
UltraSPARC T2 Plus: Multi-Socket US T2
• Per Socket> Up to 8 cores @1.2GHz or
1.4GHz> Up to 64 Threads per socket> 4MB L2$ 8 banks x 16 Way SA> Up to 8 FPU's, 1 per core> Up to 8 Crypto cores, 1 per core
> DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC,CRC32
> 2 memory Controllers, 16 DIMMS> 25Read+13WriteGB/s Memory BW> One x8 PCIe interface
• Per system> 2 sockets glue-less> 4 sockets using UST2 Plus XBR> 16 or 32 PCIe lanes on 2 or 4
Socket systems• 65nm process technology
x8 @2.5GHz4GBytes/s
bi-directional
Full Cross Bar
Power ~105W<1.5W/thread
CO
FPUMAU
C1
FPUMAU
C2
FPUMAU
C3
FPUMAU
C4
FPUMAU
C5
FPUMAU
C6
FPUMAU
C7
FPUMAU
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$CU CU CU CU
MCU MCU
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
4 x Coherency
Links
System I/F PCI-Express
SSI Bus
Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total4 x Coherency links give Snoop bandwidth of 51.2GB/s
Sun Internal and Approved Partners Only 8
UltraSPARC T2 Plus ProcessorSPC SPARCv9 CMT CoreCCX Core-Cache CrossbarL2$ Level 2 Cache BankCLC Coherence & Link ControllerMCU Memory Control UnitSIU System Interface UnitNCU Non-cacheable UnitNCX Non-cacheable CrossbarDMU Data Management UnitPEU PCI Express Unit
Sun Internal and Approved Partners Only 9
CMT Processor ComparisonUltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Technology 90nm 65nm 65nm# of Cores 8 8 8# of Threads 32 64 64# of Sockets 1 1 1 – 4
1 – 1.4 1.2 – 1.4 1.2 – 1.4Execution pipelines per core 1 2 2Instructions per cycle per pipe 1 1 1FPUs per chip 1 8 8I$ size (per core) 16KB 16KB 16KBD$ size (per core) 8KB 8KB 8KBL2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-WayMemory Type DDR2 FBDIMM-1 FBDIMM-1Memory Bandwidth (GB/s) 21.3
- -IO Bus JBUS x8 PCIe x8 PCIeIO Bandwidth (GB/s) 2.5 2 + 2 2 + 2On-chip Networking - 2 x 10GE -Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5
Power (Watts, Watts/thread) 72, 2.25 95, 1.5
Core Frequency (Ghz)
42READ + 21WRITE 25READ + 13WRITE
Coherence Bandwdth (GB/s) 4 x (6.4READ+6.4WRITE)
105, 1.6 *
Sun Internal and Approved Partners Only 10
UltraSPARC T2 Plus Cores, Crossbar, L2$(8 cores, 64 threads, 4MB L2$)
CoherenceUnit
CoherenceUnit
CoherenceUnit
CoherenceUnit
Memory Controller Memory Controller
PCI-Express
UltraSPARC T2 Plus Cores, Crossbar, L2$(8 cores, 64 threads, 4MB L2$)
CoherenceUnit
CoherenceUnit
CoherenceUnit
CoherenceUnit
Memory Controller Memory Controller
PCI-Express
UltraSPARC T2 Plus 2-Socket System
System IO (Network, Disk, etc.)
NCX NCU, DMU NCX NCU, DMU
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Coherency Links
Sun Internal and Approved Partners Only 11
UltraSPARC T2 Plus 4-Socket System
Coherence Hubs (4)
4 ports/hub
in each dir
VF
8 Cores8 Threads/Core
4MB L2$
US T2 Plus
8 Cores64 Threads
4MB L2$
PCI-Express
PCI-Express
PCI-Express
PCI-Express
Dual
Chan
nel
FBDI
MM
Dual
Chan
nel
FBDI
MM
Dual
Chan
nel
FBDI
MM
Dual
Chan
nel
FBDI
MM
Dual Channel
FBDIMMDual Channel
FBDIMM
Dual Channel
FBDIMM
Dual Channel
FBDIMM
Coherence Hubs (4)
4 ports/hub
in each dir
Coherence Hubs (4)
4 ports/hub
in each dir
Coherence Hubs (4)
UltraSPARCT2 Plus
XBR
4 ports/hub6.4GB/s per port
in each dir
6.4GB/s 6.4GB/s
6.4GB/s6.4GB/s
6.4GB/s
6.4GB/s
6.4GB/s6.4GB/s
6.4GB/s6.4GB/s
6.4GB/s6.4GB/s
6.4GB/s
6.4GB/s
6.4GB/s6.4GB/s
US T2 Plus
8 Cores64 Threads
4MB L2$
US T2 Plus
8 Cores64 Threads
4MB L2$
US T2 Plus
8 Cores64 Threads
4MB L2$
Sun Internal and Approved Partners Only 12
US T2 Plus XBR Coherency Hub Chips...• 4 XBR chips in a system• Each operates in its own
Coherency Plane(CP)> A region of addressable space, unique
to that plane> Each XBR only tracks addresses in its
own space> Serializes accesses to coherent loads,
stores and writebacks (TSO model)> 40 bit Physical Address used> VF Snoop links to pass transactions
• Cacheable transactions– Most Memory transactions
• Non-Cacheable transactions– IO, Non-cachable memory,
Interrupts etc.
• XBR routes non-cacheable PIOs to the appropriate IO device> Forwarded to the Zambezi that owns the
address space for processing• DMA reads and writes are done
through cachable memory space just like processor initiated memory accesses
• CP links are based on an FB-DIMM physical interface, but a Sun protocol> 6.4GBps/Direction/Port> 51.2GB/s bandwidth total> CP links are tolerant to a single lane failure
(14 lanes per link)> CRC and replay features in the protocol
allow no-stop recovery> Memory structures are Parity protectedCachable/
Non-Cachable
39 078
Coherency Plane Designator
31 30 10 9
Coherency Plane Designator
Coherency Plane Designator
Coherency Plane Designator
Coherency Plane Designator
Mem Interleave by Coherency plane
(512B/1GB)
40 Bit Physical Address
Sun Internal and Approved Partners Only 13
UltraSPARC T2/Plus CPU Core Block Diagram
EXU1
IFU
LSUSPU
TLU
MMU/HWTW
FGU
Gasket
xbar/L2
EXU0
• IFU – Instruction Fetch Unit> 16 KB I$, 32B lines, 8-way SA> 64-entry fully-associative ITLB
• EXU0/1 – Integer Execution Units> 4 threads share each unit> Executes one integer
instruction/cycle• LSU – Load/Store Unit
> 8KB D$, 16B lines, 4-way SA> 128-entry fully-associative DTLB
• FGU – Floating/Graphics Unit• SPU – Stream Processing Unit
> Cryptographic acceleration• TLU – Trap Logic Unit
> Updates machine state, handles exceptions and interrupts
• MMU – Memory Management Unit> Hardware tablewalk (HWTW)
– TLB miss handled by hardware > 8KB, 64KB, 4MB, 256MB pages
Sun Internal and Approved Partners Only 14
●
●
Data
• Each Core has its own Floating Point Unit. This removes completely the 2% FP restriction of T1.
• Only a 6 cycle penalty to the FPU> T1 required upwards of 28 cycles
• Fully-pipelined (except divide/sqrt)> Divide/sqrt in parallel with add or multiply operations of other
threads• Full VIS 2.0 implementation in hardware, no emulation• FPU performs integer multiply, divide, population count• Multiplier incorporates enhancements for offloading modular
arithmetic operations to the Crypto Unit
Floating Point Unit
Sun Internal and Approved Partners Only 15
Data
881---
1616166464644
2 rounds/cycle2 rounds/cycle
3 overlapped cycles4 words into 44 words6 words into 52 words8 words into 60 words
2 column over two cycles2 column over two cycles2 column over two cycles
2 cycles per step2 cycles per step2 cycles per step32 bits per cycle
16 rounds48 rounds
1---
10 rounds12 rounds14 rounds80 steps64 steps64 steps
1
DES3DESRC4AES-128KAES-192KAES-256KAES-128AES-192AES-256SHA-1SHA-256MD5CRC32c
UltraSPARC T2 Plus Crypto Performance
Wire Speed for UltraSPARC T2 Plus is around 5 GB/S16,000 new SSL sessions/sec using 1024 Asymmetric Key Exchange
8/9+25/2437683
323741
30/3036/3642/42
160/160128/128128/128
1
12.04.0
12.0---
6.365.284.564.806.006.00
48.00
Function CommentIterations/
BlockBlock(bytes)
Cycles/Block
GB/S
New
Ciphers
with U
ltraSPAR
C T2 Plus
Absolute maximums, measured by engineering using micro-benchmarksApplication throughputs will be lower than these numbers
Export restrictions apply to some of the ciphers, requires installation of the Crypto Enabling patch to use. Purchase a CD which contains the patch.
Affects IPsec related ciphers, SSL type acceleration is unaffected and does not need enabling.
Sun Internal and Approved Partners Only 16
●
●
Data
FB-DIMM Memory Interface
Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 PlusRequire Pins 1000 440 208Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socketMemory Capacity up to 16x DIMM slots
Memory Controllers 4 4 2Memory Latency 96ns 126ns 126nsDIMM speeds 400MHz 667MHz 667MHzDIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM
Up to 64x DIMM slotsT5x20 uses 16 DIMM slots
Up to 32x DIMM slots/socketT5x40 uses 8/16 DIMM slots
Sun Internal and Approved Partners Only 17
●
●
Data
10/100/1000/10GbE Network Interfaces• Uses the Neptune Chip on the Motherboard which provides
> Two RJ-45 ports dedicated for 10/100/1000 > Two RJ-45 ports capable of 10/100/1000
– Fitting a XAUI adapter disables the one of these ports per XAUI • Same 10 GbE XAUI cards as T5x20 server• 16 Rx and 24 Tx DMA channels for virtualization
– Integrated NIU in T2 has 16 Tx channels– Raw UDP Tx performance on Neptune > T2 NIU
– T2 processor network typically outperforms Neptune on most network benchmarks
• Packet classification> Rx data can be HW classified at layer 2 ,3 and 4
– Put into the separate Rx DMA buffers by classification rules
• Same 'nxge' device driver as T2 NIU.• Performance estimates, testing/tuning ongoing
– 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle– 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports)
Sun Internal and Approved Partners Only 18
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
19Sun Internal and Approved Partners Only
SPARC Enterprise T5440
Quad Socket US T2 Plus
Compute Density• 4RU chassis• 4-sockets – UltraSPARC T2 Plus
> Up to 32 cores @ 1.2GHz or 1.4Ghz> Up to 256 threads
• Up to 64 FBDIMM slots> 512 GB of memory (using 8GB DIMMS)
Reliable• 4 Hot-Plug 2.5” SAS disks (Raid 0,1)• DVD-RW• 4 x Redundant(2+2), hot-swappable PSUs• 4 x Redundant, hot-swappable Fans• 2N PSU Redundancy, 1900 watts typical
consumption Expandable• 4 x Processor+Memory card pairs• 8 x8 PCIe expansion slots (2 x XAUI) • 4 Gigabit Ethernet (or up to 2 x 10GbE and
2 x1GbE)• 4 x USB 2.0
● Up to 256 threads / 512GB memory in a robust, redundant chassis
● Taking CMT to the Mid-Range● LDOMs consolidation dream box
Sun Internal and Approved Partners Only 20
T5440 Standard Configurations (PTO)Part Number Description 4RU
SEVPCJF1Z
SEVPGLF1Z
SEVPHSF1Z
2 x 1.2GHz, 8 Core CPU, 32GB (16 X 2GB), 2x 146GB SAS 2.5”disks, 4x 1120W PSU, DVD-RW, 2 x Memory Expansion Boards
4 x 1.2GHz, 8 Core CPU, 64GB (32 X 2GB), 2x 146GB SAS 2.5”disks, 4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards
4 x 1.4GHz, 8 Core CPU, 128GB (32 X 4GB), 2x 146GB SAS 2.5”disks, 4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards
● All configs feature...● 4 x 1GbE interface,● 1 x DVD-RW slimline drive● 8 x PCIe 8 lane slots (2 shared XAUI slots), 2 x16 mechanical ● 4 x USB 2.0 slots (2 front/2 back)● 1 x DB9 Serial port● 4 x Redundant, Hot-plug PSUs● 4 x Redundant, hot-plug fans● 2 x 146GB 2.5” SAS drives (can accommodate up to 4 drives)● RAID 0,1 from onboard LSI 1068E controller● 1 Year On-Site Warranty● Rack mount kit with cable management arm● 4 x Power Supply cables (country specific)
Sun Internal and Approved Partners Only 21
Sun Fire T5240/T5440 Differences
Feature SPARC Enterprise T5240 SPARC Enterprise T5440Form Factor 2U, 28” deep 4U, 24.5” deep
CPU
Memory
Network
Internal Storage
Removable Media 1x DVD-RW 1x DVD-RWSerialPCI Express slots
Power SupplyFans 5x Redundant/Hot-Swap 4 x Redundant Hot-Swap
2x UltraSPARC T2 Plus1.2/1.4 GHz 64-128 threads
Up to 4 x UltraSPARC T2 Plus1.2/1.4 GHz 128-256 threads
FB-DIMM 128GB MAX32x slots
FB-DIMM, 512GB MAX64x Slots
4x GbE or3 x GbE + 1 x 10GbE or 2 x GbE + 2 x 10GbE
4x GbE or3 x GbE + 1 x 10GbE or 2 x GbE + 2 x 10GbE
Up to 16 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap
1x RS-232, 4x USB 1x RS-232, 4x USB6x (low profile) 8x (low profile)
2x 1100W AC/2 x1200Watt DCRedundant/Hot-Swap
4 x 1120W ACRedundant/Hot-Swap
Sun Internal and Approved Partners Only 22
Service Required LED
Power/OK LED
Power Button
Front Fan Fault LED
PSU Fault LED
System Overheat Fault LED
Front PanelD
ISK 0
DISK 1
DISK 2
DISK 3
Locator LED/Button
Serial Number label
DVD
-RW
Dual USB 2.0 ports
System Label
Sun Internal and Approved Partners Only 23
Rear Panel
PSU 1PSU 1
PSU 0PSU 0
PSUStatusLED's
PCI 0
PCI 0
SPSerialPort
ChassisStatus LED's
and Locator Button
SPNetwork
Port
Quad GigabitEthernet Ports
DualUSB 2.0
Ports
POSIX DB-9Serial Port
PSU 2PSU 2
PSU 3PSU 3
PCI 1
PCI 1
PCI 2
PCI 2
PCI 3
PCI 3
PCI 4
PCI 4
PCI 5
PCI 5
PCI 6
PCI 6
PCI 7
PCI 7
PCI Card Securing Latches (internal)
PCI 0 = PCIe x8, mechanical x8
PCI 1 = PCIe x8, mechanical x16
PCI 2 = PCIe x8, mechanical x8
PCI 3 = PCIe x8, mechanical x8
PCI 4 = PCIe x8, mechanical x8, or XAUI 0 Card
PCI 5 = PCIe x8, mechanical x8, or XAUI 1 Card
PCI 6 = PCIe x8, mechanical x16
PCI 7 = PCIe x8, mechanical x8
NET 2
NET 0 NET 1
NET 3
XAUI 1 XAUI 0
Sun Internal and Approved Partners Only 24
SSI SSI SSI SSI
Sun Fire T5440 Block Diagram
US T2 XBR0
US T2 XBR1
Zambezi2
US T2 XBR3
4x FB Links 4x FB Links 4x FB Links 4x FB Links
CL0 CL0
CL0 CL0
CL1
CL1
CL2
CL2CL3
CL3
CL1
CL1
CL2
CL3
CL2
CL3
US T2 XBR2
PLX1PEX8548
PCIe x8PCIe x8 PCIe x8
PCIe x8
x8 PLX0PEX8548
Neptune
QUAD GbEPHY
BCM5466R
1 2 30
LSI1068ex8 SAS
x4SAS
PLX3PEX8548
PLX2PEX8548
PCIe-PCI
USBhost
USB Hub USB-IDE
0 1 2 3 4 USB 2.0 USB 2.0 DVD NETMGT
SERMGT DB-9 Quad 10/100/1000 5 76
BACK panel
FRONT panel
BACK panel
4x SSILPC PCI
ILOMCard
x1x8
x8x8
x8 x8
x8 x8x8
x8
x8 x8 x8 x8
MemModule
3
MemModule
1
MemModule
2
MemModule
0
CPUModule
0
CPUModule
2
CPUModule
1
CPUModule
3
T2 PlusVRM
VRM
VRM
VRM
VRM
VRM
VRM
VRM
XAUI 0XAUI 1
LPC
LPC
LPC
LPC
FPGA
x8
Mem Module =- 12 x FB-DIMM Slots
CPU Module =- 1 x VF CPU- 4 x FB-DIMM Slots
T2 Plus
T2 Plus
T2 Plus
Sun Internal and Approved Partners Only 25
Internal View with Cards installed(T5440)
DiskDiskChassisChassis
4 x SAS Disk Chassis
4 Fans are required System Configuration Card (SCC)EEPROM
Chassis Intrusion SwitchExtension to top cover from MBD
Service ProcessorDaughter Card
PowerPowerDistributionDistribution
BoardBoard
PCIe 0 x8(m)/x8(e)
Power BUS BarsRED = +12VoltsBLUE = Ground
MB
/FT0M
B/FT0
MEM 0MEM 0
Power Power SuppliesSupplies
MB
/FT2M
B/FT2
MB
/FT3M
B/FT3
MB
/FT1M
B/FT1 CPU 0CPU 0
CPU 2CPU 2
MEM 2MEM 2
MEM 1MEM 1
CPU 1CPU 1
CPU 3CPU 3
MEM 3MEM 3
PCIe 1 x16(m)/x8(e)
PCIe 2 x8(m)/x8(e)PCIe 3 x8(m)/x8(e)
XAUI/PCIe 4 x8(m)/x8(e)
XAUI/PCIe 5 x8(m)/x8(e)
PCIe 6 x16(m)/x8(e)
PCIe 7 x8(m)/x8(e)
Sun Internal and Approved Partners Only 26
Internal View, cards removed(T5440)
DiskDiskChassisChassis
System Configuration Card (SCC)EEPROM
Chassis SeperatorFence
US T2 Plus XBR Coherency Plane Chips
Service ProcessorDaughter Card
PowerPowerDistributionDistribution
BoardBoard
PCIe 0 x8(m)/x8(e)
MB
/FT0M
B/FT0
MEM 0MEM 0
Power Power SuppliesSupplies
MB
/FT2M
B/FT2
MB
/FT3M
B/FT3
MB
/FT1M
B/FT1 CPU 0CPU 0
CPU 2CPU 2
MEM 2MEM 2
MEM 1MEM 1
CPU 1CPU 1
CPU 3CPU 3
MEM 3MEM 3
PCIe 1 x16(m)/x8(e)
PCIe 2 x8(m)/x8(e)PCIe 3 x8(m)/x8(e)
XAUI 0/PCIe 4 x8(m)/x8(e)
XAUI 1/PCIe 5 x8(m)/x8(e)
PCIe 6 x16(m)/x8(e)
PCIe 7 x8(m)/x8(e)
Sun Internal and Approved Partners Only 27
Internal View, Motherboard only(T5440)Neptune Quad
GbE/10GbE NIC
LSI1068E Disk Controller
DC-DC Converter Modules
Service ProcessorCard Connector
MB
/FT0M
B/FT0
MEM 0MEM 0
MB
/FT2M
B/FT2
MB
/FT3M
B/FT3
MB
/FT1M
B/FT1 CPU 0CPU 0
CPU 2CPU 2
MEM 2MEM 2
MEM 1MEM 1
CPU 1CPU 1
CPU 3CPU 3
MEM 3MEM 3
Quad GbE RJ-45Ports
NET MGT portDual USB 2.0 Port
SER MGT port
DB-9 Serial Port
System Status LEDs
Fan connectors, with speed control logic
PLX PCIe Switch Chips
'Southbridge' Peripheral Chip
Sun Internal and Approved Partners Only 28
Motherboard and CPU/MEM Card layout
MEM 0MEM 0
CPU 0CPU 0
CPU 2CPU 2
MEM 2MEM 2
MEM 1MEM 1
CPU 1CPU 1
CPU 3CPU 3
MEM 3MEM 3
Memory BoardCPU/Mem Board
Motherboard
Sun Internal and Approved Partners Only 29
Memory Configuration Rules
BANK 0Install ALL DIMMs
BR1/CH0/DO
• 1 or 4 CPU cards in a system> Each CPU card has an associated
Memory Card> Unpopulated Cards must be filled
with spacers> Unpopulated DIMMs must be filled
with spacers• All DIMMS on a CPU & Memory
card pair must be the same size> 2GB, 4GB or 8GB only (no
1GB)> May support Different DIMM
sizes on different pairs...• 3 Configs...
> Fill Bank 0> Fill Banks 0,1> Fill Banks 0,1,2,3
• All standard configs provide exactly 8 free DIMM slots simultaneously
UltraSPARC T2 Plus
CPU
BANK 3Install ALL DIMMs
BANK 2Install ALL DIMMs
BANK 1Install ALL DIMMs
Card/ChassisTop
BR1/CH1/DOBR0/CH0/DOBR0/CH1/DO
BR1/CH1/D2BR1/CH1/D3BR1/CH0/D2BR1/CH0/D3
BR0/CH1/D2BR0/CH1/D3BR0/CH0/D2BR0/CH0/D3BR0/CH1/D1BR0/CH0/D1BR1/CH1/D1BR1/CH0/D1
VRM
VRM
VRM
VRM
VRM
Sun Internal and Approved Partners Only 30
Network Ports and Options
XAU
I 0XA
UI 0
NOTESAll Ethernet ports provided by a single Neptune Chip on the motherboard
NET 0 and NET 1 are disabled by XAUI 1 and XAUI0,if installed, to take advantage of 10GbE networking
* If a XAUI card is installed, the matching RJ-45 port is disabled (NET 0 and NET 1)
XAUI cards take one of three current Fibre 10GbE transceivers
Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM)
Copper versions of the XAUI card are expected Post-RR
XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive with PCI cards in the same location
Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1Driver nxge0 nxge1 nxge2 nxge3 PCI Device PCI@500/0/c/0,0 PCI@500/0/c/0,1 PCI@500/0/c/0,2 PCI@500/0/c/0,3 PCI@500/0/c/0,0 PCI@500/0/c/0,1
nxge0* nxge1*
XAU
I 1XA
UI 1
NET 2
NET 0 NET 1
NET 3
Sun Internal and Approved Partners Only 31
10GbE XAUI Option Cards• Interface to the Neptune chip to upgrade two of its ports to 1GbE• Cards each consume a PCI-E slot if installed• Attach to dedicated XAUI sockets on special risers• 10Gbe XAUI Card
– SESY7XA1Z> Requires Fibre Tranceivers
– 10GbE Short Reach(SR)– SESY7XT1Z
– 10GbE Long Reach (LR)– SESY7XT2Z
– 10GbE Long Reach multimode (LRM)– part No. TBD
• Copper connector XAUI versions to follow post-RR
Sun Internal and Approved Partners Only 32
PCIe SlotsBuilt In
• No Riser Cards are used, slots are directly on the motherboard• Low Profile PCI-Express cards only• Each Slot is capable of 25Watts power• Total power capacity limited to 140Watts
> Systems design point was to accommodate 8 x 17.5Watt Atlas(Quad GbE using Neptune) card
• Each Slot has x8 PCIe electrical capability and at least x8 mechanical connectors• Slots 1 and 6 have x16 mechanical connectors• XAUI cards in slots 4 and 5 replace PCIe cards in those slots
Sun Internal and Approved Partners Only 33
PCIe External IO Expansion Unitaka Expansion Chassis, I/O Box
• Up to 2 x PCI Expansion Chassis are supported > P/N : SENX8BE1Z> 4RU, 19” rack compatible , 28” deep> 1 Host card supported per US T2 Plus> Each chassis is managed by host SP
• Only support PCIe card IO boats ( PCI-X ?)> Each 'boat' requires an Link Card in the host
(ensign) and a link Card in the boat (captain) with special card between them
> IO 'boat' supports 6 x PCIe x8 full-length, hot-pluggable PCIe cards
• Chassis default configs contain> 2 x Hot-plug redundant 600Watt PSU/FAN
FRU's> 1 x PCIe 'boat', 1 link card pair(host-chassis)
and cable> 6 x PCIe card carriers for x8 PCIe Cards> Customers can add a further PCIe 'boat'
– P/N : SENX8TE1Z and more
Sun Internal and Approved Partners Only 34
Disk Drives• Disk drives are all 2.5” form factor• Original Galaxy NEMO bracket (not Marlin Bracket)• Support SAS only (Can mix SAS/SATA after qual)• SAS Drives
– 73GB @ 10K RPM– 146GB @ 10K RPM– 73GB @ 15K RPM– SSD (post-RR)
• Same disks as T2x00 systems
Ready to RemoveFaultStatus
Disk LED's
Sun Internal and Approved Partners Only 35
Disk Controller Options
• T5440 has embedded LSI 1068E on the motherboard– JBOD up to 4 disks– Controls internal disks only– Unused 4 ports are not used– RAID 0 (striping) and RAID 1(mirroring)– Hot-plug supported– Hot-swap in non-RAID configs is done with 'cfgadm'
• SGXPCIESAS-R-INT-Z NOT supported> Not available for T5440 due to chassis design and only 4 disk capacity
• SGXPCIESAS-R-EXT-Z Supported– For external disk arrays only (no internal ports)– 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available)– Adaptec Card and Firmware with Intel SunRiseLake chipset– Battery Backed Write Cache (BBWC) of 256MB, at least 72 hours capacity, much
longer when new
Sun Internal and Approved Partners Only 36
Fans and Cooling• Front to Back cooling• Two rows of fans provide redundancy
> PSU fans provide sufficient redundancyfor Disk cooling
• Requires 4 Fans– Main System Cooling
– MB/FT0– MB/FT1– MB/FT2– MB/FT3
– PSU integrated Fan Cooling flow– Cools disk drives and PSU's– PSUFAN0– PSUFAN1– PSUFAN2– PSUFAN3
PSU'sPSU's
MotherboardMotherboard
FrontBack
Disks
Disks
MB
/FT2M
B/FT2
MB/FT3
MB/FT3
MB/FT1
MB/FT1
MB
/FT0M
B/FT0
MEM0MEM0CPU0CPU0CPU2CPU2MEM2MEM2MEM1MEM1CPU1CPU1CPU3CPU3MEM3MEM3
Sun Internal and Approved Partners Only 37
Fans modules and boards
• 4 Fans Required> 60 Seconds grace if FAN removed for service
• Fans are speed controlled by the Service Processor• Speeds are independantly monitored by SP• Speeds tuned to minimise vibrations
> Built in vibration sensor• Speeds chosen to minimise harmonic resonance
with disk drives and other fans.• Each fan has an Amber FAULT LED
– if lit the fan module is faulty
Fan Fault Amber LED
Lit if fan faulty
Sun Internal and Approved Partners Only 38
Power Supplies• PSU's are high efficiency• Output 12V (and 3.3v standby)• Most DC-DC conversion done in MBD• Integrated Fans• 4 PSU's
> 2 + 2 Redundancy (Grid Redundancy)> Rated at 1120Watts each.
> BEWARE... – Same form factor as SunFire X4600 PSU's ... DO NOT MIX
PSU Fault LED
Sun Internal and Approved Partners Only 39
Rack Mounting Options
• Two Rack mounting options are available..• Standard Rack Mount (4 post)
> P/N : SEVY9RK1Z> Same as Sun Fire X4600 server> Standard configs contain this kit
• Tool-less Rack Kit (4 post)> P/N : SEVY9RK2Z> Optional kit, purchased seperately> Snaps into Sun and most 3rd party Racks> Requires no tools to fit or remove> Will replace tool-required version once available
Sun Internal and Approved Partners Only 40
Solaris for Sun SPARC Enterprise T5x40• T5x40 servers will require Solaris10 5/08 (update 5) with
patches> A Patch CD is supplied in the box
– Likely to be out of date, so customers should check SunSolve...• This incorporates several specific UltraSPARC T2/T2 Plus
optimizations and for CMT servers in general• Also includes some specific T5x40 optimisations
> UltraSPARC T2 Plus MPO optimisations> nxge driver tweaks for 10GbE devices> MMU DMA relaxed scheduling > LDOMs pre-install upgrade to version 1.0.3 (US T2 Plus support)
Sun Internal and Approved Partners Only 41
Solaris FMA• Now aware of CPU Core offlining• PCIe Fabric diagnostics improved• Crypto Unit offlining• Neptune awareness (though XAUI's not diagnosed)• POST memory CE tolerance improved• FRU identity info improved in fault reports (part/ser #)• Diagnostic Engines (DE)
– Existing DE have been extended for T5x40– New Eversholt DE used for HostBridge, PCIe diagnostics
• Limitations...– Can't isolate Uncorrectable Errors to a single DIMM– PCIe riser cards not in topology– XAUI card and XFP not in the topology– SouthBridge chip is not diagnosed at this time (DVD, USB,...)– IO Expansion box is not diagnosed via FMA
Sun Internal and Approved Partners Only 42
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> System Architecture and Tours> Extra Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 3.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 43
ILOM Features at a Glance• Hardware Control, Inventory, Monitoring• Host Configuration and Administration• SP Configuration and Administration• Logging, Auditing and Alerting• Authentication :
– LDAP, RADIUS, Active Directory, Local Users• Services :
– SSH, HTTP(S), NTP, Syslog, SMTP• Out-of-Band
– Via Serial port, or dedicated Ethernet port• In-Band
– Via Host OS, firmware upload, Explorer, Syslog
Sun Internal and Approved Partners Only 44
ILOM – Industry Standard Manageability• ILOM enables Sun systems to be easily integrated into third
party management infrastructures• Key functionality
> IPMI 2.0> SNMP V1, V2c, V3> SSH 2.0> LDAP, RADIUS & MS Active Directory authentication> Browser UI, SMASH DMTF CLP + ALOM style CLI
• NOTE : No RKVMS functionality like Galaxy ILOM> No Java plugin required
• Browsers tested include, these base versions an later releases
– Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox 1.0 and 1.5 on W2K server
Sun Internal and Approved Partners Only 45
ILOM – Industry Standard Manageability
• Identical to ILOM 2.0 for T5x20• Default user is 'root' with password 'changeme'
> Use for ILOM WEB GUI access and SSH access and IPMI• The ALOM 'admin' user is not defined by default, create this
user with an ALOM CLI• Lost 'root' password and ILOM config is reset using jumpers
> T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2)> T5440 - SP Daughter card Jumper J1127 (short pins 1-2)
• Note... 'sunservice' user shares the password of the 'root' user ONLY FOR SERVICE ACCESS*
• Passwords must be 8 to 16 chars (used to be max 8)• Max of 10 users allowed (used to be 16)
Sun Internal and Approved Partners Only 46
ILOM Common Tasks
• Setting up an ALOM 'admin' user from the ILOM CLI> SUNSP00144F4641BD login: root
Password: changemeWaiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.
Warning: password is set to factory default.
->create /SP/users/admin role=Administrator cli_mode=alomCreating user...Enter new password: ********Enter new password again: ********Created /SP/users/admin2
-> exit
SUNSP00144F4641BD login: adminPassword: ********Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.
sc>
Sun Internal and Approved Partners Only 47
ILOM Common Tasks
• Setting up ILOM IP configuration from the ILOM CLI> SUNSP00144F4641BD login: root
Password: changemeWaiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.
Warning: password is set to factory default.
-> set /SP/network state=enabledSet ’state’ to ’enabled’-> set /SP/network pendingipaddress=10.8.31.163Set ’pendingipaddress’ to ’10.8.31.163’-> set /SP/network pendingipdiscovery=staticSet ’pendingipdiscovery’ to ’static’-> set /SP/network pendingipnetmask=255.255.252.0Set ’pendingipnetmask’ to ’255.255.252.0’-> set /SP/network pendingipgateway=10.8.31.248Set ’pendingipgateway’ to ’10.8.31.248’-> set /SP/network commitpending=trueSet ’commitpending’ to ’true’->
-> set/SP/pendingipdiscovery=dhcpSet ’pendingipdiscovery’ to ’dhcp’-> set /SP/network commitpending=trueSet ’commitpending’ to ’true’->
Static IP configuration
Dynamic IP (DHCP)
configuration
Sun Internal and Approved Partners Only 48
ILOM IPMI functionality• IPMI is a widely used management protocol for x86 servers• ... But its new for SPARC boxes...• Download the required tools from
> The Solaris Supplemental CD (an older binary package)> Latest is available at http://ipmitool.sourceforge.net
– Binary packages (usually older versions) are available for Solaris x86 and SPARC, as well as Linux, Windows, Mac...etc
> There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC now too)
– http://www.sun.com/blueprints/0107/820-1011.pdf
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr/MB/V_+1V0_VDD | 1.01 Volts | ok/MB/V_+3V3_STBY | 3.36 Volts | ok/MB/V_+3V3_MAIN | 3.36 Volts | ok/MB/V_+5V0_VCC | 4.97 Volts | ok/MB/V_+12V0_MAIN | 12.16 Volts | ok....-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set /SYS/LOCATE on/SYS/LOCATE | ON-bash-ventus2$
Get the Sensor Data
Repository
Set the System Locator LED
flashing
Sun Internal and Approved Partners Only 49
ILOM Common Tasks
• Logging into the ILOM Web GUI• Point Web Browser at the IP address of
the ILOM> Will automatically redirect to a
secure HTTPS port> Best to disable any Web Proxies for
best performance> Make sure port 80, 443 are open on
any firewall
• Log in as any defined user, default Administration user is 'root'
Sun Internal and Approved Partners Only 50
ILOM/ALOM Common Tasks
• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address configurations for ILOM
• Configure additional ILOM/ALOM users
• Access Sensor Readings
• Control System configurations
• Carry out firmware updates.
• etc....
Sun Internal and Approved Partners Only 51
ALOM 2.0 vs. 1.4 CLI Differences• Default CLI appearance is ILOM
– User is 'root'– Must create an 'admin' Administrator user with the ALOM CLI
• Flashupdate in new ALOM expects a TFTP server not an FTP server • ALOM IP address changes now active only after running ...
– 'setsc netsc_commit true'– Previously changes were active only after SC reset– DHCP is enabled by default
• View firmware versions with – 'showhost version'
• Only Admin and Operator rolesare defined
sc> showhost version Sun System Firmware 7.0.0_build_13 2007/08/10 23:44
Host flash versions: Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54 OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25 Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:51
Sun Internal and Approved Partners Only 52
ALOM backward Compatibilty CLI
• Uses ALOM-CMT 1.2 for a base feature set• Includes ALOM-CMT 1.4 features...
> break -d> Host Watchdog Timer> Bootmode LDOMs awareness> Forward SP events to Hot/Solaris syslog via inband channel
Sun Internal and Approved Partners Only 53
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> Comparison to SunFire T1000/T2000/T5x20> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 54
Logical Domains 1.0.3 Features
• UltraSPARC T2 Plus Platform Support> 128 domains supported at RR
– 256 domains work on T5440, may be qualified later, requires Solaris fixes• Feature Set very similar to 1.0.2• I/O domain can reboot while the other domains keep running
> Dependent domains would wait for I/O transactions to complete• LDoms SNMP MIB
> Modeled after industry standard DMTF CIM (Common Information Model)> Running on control domain to enable remote SNMP monitoring
• Domain minimization support> Minimal/reduced Solaris installation> Base minimal packages plus optional ones for guest, control, I/O or service
domains
Sun Internal and Approved Partners Only 55
Logical Domains 1.0.3 Requirements
• Solaris 10 05/08 (update 5) and required patches• Systems firmware version
– T5x40 Firmware 7.1.5b = SP FW 2.0.4.20+– T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+– T2000/T1000/T6300/Pelton Firmware 6.5.x
• SUNWldm.v LDOMs Manager Package– Version 1.0.3
Sun Internal and Approved Partners Only 56
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> Comparison to SunFire T1000/T2000/T5x20> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 57
Option Parts List for T5440(ATO)
Part Number Option Description AvailableSEVY1CB1Z T2 Plus CPU module, 8 core, 1.2GHz, 8GB(4x2GB) with memory board RRSEVY1CC1Z T2 Plus CPU module, 8 core, 1.2GHz, 16GB(4x4GB) with memory board RRSEVY1CD1Z T2 Plus CPU module, 8 core, 1.2GHz, 32GB(4x8GB) with memory boardSEVY1DD1Z T2 Plus CPU module, 8 core, 1.4GHz, 32GB(4x8GB) with memory boardSEVY1DB1Z T2 Plus CPU module, 8 core, 1.4GHz, 8GB(4x2GB) with memory board Subject to 1.4GHz availabilitySEVY1DC1Z T2 Plus CPU module, 8 core, 1.4GHz, 16GB(4x4GB) with memory board Subject to 1.4GHz availabilitySESY2A1Z 2GB Memory Expansion kit(2x1GB)SESY2B2Z 4GB Memory Expansion kit(2x2GB) Non-Extended ECC version, no mixingSESY2D1Z 16GB Memory Expansion kit (2 X8 GB)SESY2C1Z 8GB Memory Expansion kit(2x4GB)SESVY3A11Z 73GB 10Krpm, 2.5” SAS DriveSESVY3C11Z 146GB 10Krpm, 2.5” SAS DriveSESVY3A21Z 73GB 15Krpm, 2.5” SAS DriveSEVY9PS51Z 1120W AC Power SupplySEVY9DV1Z DVD, 8x DVD-RW Slimline moduleSEVY9RK1Z Slide Rail Kit, Screw MountingSEVY9RK2Z Slide Rail Kit, Toolless Mounting
Sun Internal and Approved Partners Only 58
Option Parts List for T5440 (cont..)Part Number Option Description Max number
SG-XPCIE1FC-QF4 RR 8SG-XPCIE2FC-QF4 RR 8SG-XPCIE1FC-EM4 RR 8SG-XPCIE2FC-EM4 RR 8SG-XPCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 8SG-XPCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 8SGXCPCIESAS-R-EXT-Z SAS, 8 Port External RAID Controller RR 2X7281A-2 RR 8X7280A-2 RR 8X1236A-Z RR 8X4447A-Z RR 8X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 4X5558A RR -X5560A-Z RR -
X3000A-Z XVR-300, PCI-E x8 Graphics cards (up to 1) RR 4SESY7XA1Z 10GbE XAUI card RR 2TBD 10GbE XAUI Copper 10g Base-T TBD 2X6000A Sun Crypto Accelerator 6000 RR 2
SENY8BE1Z RR 2SENY8TE1Z PCI-Express “boat” for External IO Expansion Unit (6 Hot Plug carriers) RR 4SELY8LK1Z Copper link kit for IO Expansion unit “boat” RR 4
AvialableQlogic, 4Gb, PCI-E Single PortQlogic, 4Gb, PCI-E Dual Port
Emulex, 4Gb, PCI-E Single Port Emulex, 4Gb, PCI-E Dual Port
Northstar Dual port GigE MMF Low profile PCI-ENorthstar Dual port GigE UTP Low profile PCI-E
Infiniband HBA, Dual 4x ports, PCI-EAtlas Quad Port, GigE Copper, PCI-E (up to 3/6)
Tranceiver 10GE SR XFP (XAUI/Atlas)Tranceiver 10GE LR XFP (XAUI/Atlas)
External IO Expansion Unit – PCI-E base Config
Sun Internal and Approved Partners Only 59
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> Comparison to SunFire T1000/T2000/T5x20> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 60
Tuning Out of the Box• Using Solaris 10 05/08 (update 5) with Patches
> MPO optimisations used> DMA relaxed Ordering implemented> Default pages sizes for
– Initdata Segments is 64kB pages– Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages– ISM and DISM use 256MB pages– New variables introduced to control these sizes if changes are needed
– max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize, max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize
> L2 Cache hashing implemented– helps randomise accesses to L2 caches and ease thrashing
> Stack biasing implemented– D-Cache is only 4 Way associative... 8 threads– Running multiple copies of a binary can lead process stacks aligning in the cache
and to hot lines in the cache– Stack Bias slews the stack variable for processes.
Sun Internal and Approved Partners Only 61
Tuning Out of the Box (continued..)> Processor Groups introduced
– Helps associate CPUs with some shared resources (int/FP pipelines, Crypto units etc)
– Enables improved multi-level CMT load balancing and affinity policies> 'bcopy'/'uiomove' Optimizations
– Key kernel operations– new implementation about 2x faster than previous
• Optimisations will require patching of S10 05/08> These will be post RR patches as they become available> Most fixes are already committed in update 5> Check patch levels for customers with performance issues> Also check Firmware versions as this can improve performance
too...
Sun Internal and Approved Partners Only 62
Tuning Recommendations• Less involved than with T2000• S10 5/08 is more 'automatic'
– OOB optimised page size selection, thread scheduling, block copy, IP instances for zones, page coloring etc...
• If tuning is necessary remember...> Improved pipeline results in Single threaded performance around 20% better> Floating Point is no longer an issue> Start with T2000 tuning options> Use updated trapstat, cpustat,
• LDOMs may be an easier way to scale poorly threaded applications• Solaris Crypto framework is still being tuned
> Great for big packets, working on improving small packets• A Blueprint is being prepared (Denis Sheahan)
Sun Internal and Approved Partners Only 63
Tuning Recommendations (cont..)• Refer to UltraSPARC T1 applications tuning best practices...
> http://www.sun.com/servers/coolthreads/tnb/applications.jsp• CoolTools are being updated for UltraSPARC T2 and T2 Plus
> http://www.sun.com/servers/coolthreads/overview/cooltools.jsp• Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware
> Recompiling without code changes should benefit T1/T2/T2 Plus apps• LDOMs configs can benefit significantly from a transition to T5x20 platform
> Multiple IO/Service domains with higher bandwidth PCI busses> Make sure appropriate LDOMs tuning and sizing is carried out
• Wider Crypto capability, so ensure code is using PKCS#11 framework> Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers),
n2rng(random number generator) drivers are used appropriately
Sun Internal and Approved Partners Only 64
PCIe Performance
• Easily two to four times the performance of T5x20 (two or four PCIe Busses)
• All PCI card slots are x8 PCI electrically> Single level hierarchy> Cut through switching> PCI-E relaxed ordering for higher performance
• Performance per bus...> Marketing performance = 4000MB/sec> Theoretical Rate with overheads = 3790MB/sec> Max. DMA Achievable(bidirectional) = 3096MB/sec
– with 128byte packets– And 512 byte read size
Sun Internal and Approved Partners Only 65
Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers
> Overview of Features> System Architecture and Tours> Memory, Networking and IO Expansion> Disks, Fans and Power Supplies> Solaris for T5x40
• ILOM 2.0• LDOMs for T5x40 • CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ...
Sun Internal and Approved Partners Only 66
Tools and Information
• Cool Tools– http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• SUNWIN Tokens> Sun SPARC Enterprise T5440
– Architecture Whitepaper : 526122– JTF : 526118
> T5440 RAS Whitepaper : TBA• Video Tours
> Check the Sun.com product pages• Power Calculators
> Check the Sun.com product pages
Sun Internal and Approved Partners Only 67
Further Information• Presto
> T5440> http://sunwebcollab.east.sun.com/gm/document-1.26.680780/Batoka_PRESTO.html
• Installed Business Base (IBB) Promotions> http://sunwebcollab.east.sun.com/gm/document-1.26.680782/Batoka_IBB_Promo.html
• Start Up Essentials program> http://sunwebcollab.east.sun.com/gm/document-1.26.680794/Maramba_Batoka_SU_April1.html
Sun Internal and Approved Partners Only 68
Reference Information• T5440 Engineering TOI
– http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/• ILOM/ALOM User Guides
– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf
• Solaris FMA– http://fma.eng– http://lightside.sfbay/n2-fma/docs.php
•