Post on 09-Jul-2020
Evaluation of AMD EPYCChris Hollowell <hollowec@bnl.gov>
HEPiX Fall 2018, PIC Spain
2
What is EPYC?
EPYC is a new line of x86_64 server CPUs from AMD based on their Zen microarchitecture
Same microarchitecture used in their Ryzen desktop processorsReleased June 2017
First new high performance series of server CPUs offeredby AMD since 2012
Last were Piledriver-based OpteronsSteamroller Opteron products cancelled
AMD had focused on low power server CPUs instead
x86_64 Jaguar APUsARM-based Opteron A CPUs
Many vendors are now offering EPYC-based servers, including Dell, HP and Supermicro
3
How Does EPYC Differ From Skylake-SP?
Intel’s Skylake-SP Xeon x86_64 server CPU line also released in 2017
Both Skylake-SP and EPYC CPU dies manufactured using 14 nm process
Skylake-SP introduced AVX512 vector instruction support in XeonAVX512 not available in EPYCHS06 official GCC compilation options exclude autovectorizationStock SL6/7 GCC doesn’t support AVX512
Support added in GCC 4.9+ Not heavily used (yet) in HEP/NP offline computing
Both have models supporting 2666 MHz DDR4 memorySkylake-SP
6 memory channels per processor3 TB (2-socket system, extended memory models)
EPYC8 memory channels per processor4 TB (2-socket system)
4
How Does EPYC Differ From Skylake (Cont)?Some Skylake-SP processors include built in Omnipath networking,or FPGA coprocessors
Not available in EPYC
Both Skylake-SP and EPYC have SMT (HT) support2 logical cores per physical core (absent in some Xeon Bronze models)
Maximum core count (per socket)Skylake-SP – 28 physical / 56 logical (Xeon Platinum 8180M) EPYC – 32 physical / 64 logical (EPYC 7601)
Maximum socket countSkylake-SP – 8 (Xeon Platinum)EPYC – 2
Processor InteconnectSkylake-SP – UltraPath Interconnect (UPI)EYPC – Infinity Fabric (IF)
PCIe lanes (2-socket system)Skylake-SP – 96EPYC – 128 (some used by SoC functionality)
Same number available in single socket configuration
5
EPYC: MCM/SoC Design
EPYC utilizes an SoC designMany functions normally found in motherboardchipset on the CPU
SATA controllersUSB controllers etc.
Each EPYC processor consists of four CPU dies,interconnected via Infinity Fabric
Multi-Chip Module (MCM) architecture”CPU Complexes” (CCX)
Each CCX attached to its own memory2 memory channels per CCX
All Skylake-SP cores are on a single die
AMD claims MCM results in a cost reduction by improving yields
Believed to scale better than monolithic die approach as core counts continueto increaseDrawback: higher memory latency for non-NUMA-aware applications
6
EPYC: MCM/SoC Design (Cont.)# lscpuArchitecture: x86_64CPU op-mode(s): 32-bit, 64-bitByte Order: Little EndianCPU(s): 64On-line CPU(s) list: 0-63Thread(s) per core: 2Core(s) per socket: 16Socket(s): 2NUMA node(s): 8Vendor ID: AuthenticAMDCPU family: 23Model: 1Model name: AMD EPYC 7351 16-Core ProcessorStepping: 2CPU MHz: 2400.000CPU max MHz: 2400.0000CPU min MHz: 1200.0000BogoMIPS: 4799.41Virtualization: AMD-VL1d cache: 32KL1i cache: 64KL2 cache: 512KL3 cache: 8192KNUMA node0 CPU(s): 0-3,32-35NUMA node1 CPU(s): 4-7,36-39NUMA node2 CPU(s): 8-11,40-43NUMA node3 CPU(s): 12-15,44-47NUMA node4 CPU(s): 16-19,48-51NUMA node5 CPU(s): 20-23,52-55NUMA node6 CPU(s): 24-27,56-59NUMA node7 CPU(s): 28-31,60-63
# lscpuArchitecture: x86_64CPU op-mode(s): 32-bit, 64-bitByte Order: Little EndianCPU(s): 72On-line CPU(s) list: 0-71Thread(s) per core: 2Core(s) per socket: 18Socket(s): 2NUMA node(s): 2Vendor ID: GenuineIntelCPU family: 6Model: 85Model name: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHzStepping: 4CPU MHz: 2700.000BogoMIPS: 5404.41Virtualization: VT-xL1d cache: 32KL1i cache: 32KL2 cache: 1024KL3 cache: 25344KNUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71
EPYC vs Skylake-SP (SNC Disabled) NUMA Configuration
7
Socket LGA 3647 & SP3
Both CPUs/sockets are quite large
Visible quadrants in the SP3 socket for the four CPU complexes in the EPYC processor
Skylake SP – Socket LGA 3647 EPYC – Socket SP3
8
Skylake and EPYC Model Lineup ComparisonModel Base Frequency Cores SMT TDP Memory Retail
Xeon Bronze 3104 1.7 GHz (no turbo) 6 No 85W 2133 MHz DDR4 $213
Xeon Silver 4110 2.1 GHz 8 Yes 85W 2400 MHz DDR4 $501
Xeon Gold 5115 2.4 GHz 10 Yes 85W 2666 MHz DDR4 $1,221
Xeon Gold 6130 2.1 GHz 16 Yes 125W 2666 MHz DDR4 $1,900
Xeon Gold 6136 3.0 GHz 12 Yes 150W 2666 MHz DDR4 $2,460
Xeon Gold 6148 2.4 GHz 20 Yes 150W 2666 MHz DDR4 $3,072
Xeon Gold 6150 2.7 GHz 18 Yes 165W 2666 MHz DDR4 $3,358
Xeon Platinum 8170 2.1 GHz 28 Yes 165W 2666 MHz DDR4 $7,405
Xeon Platinum 8180M 2.5 GHz 28 Yes 205W 2666 MHz DDR4 $13,011
EPYC 7251 2.1 GHz 8 Yes 120W 2400 MHz DDR4 $475
EPYC 7351 2.4 GHz 16 Yes 170W 2666 MHz DDR4 $1,110Uniprocessor (P) - $750
EPYC 7401 2.0 GHz 24 Yes 170W 2666 MHz DDR4 $1,850Uniprocessor (P) - $1,075
EPYC 7451 2.3 GHz 24 Yes 180W 2666 MHz DDR4 $2,400
EPYC 7551 2.0 GHz 32 Yes 180W 2666 MHz DDR4 $3,400
EPYC 7601 2.2 GHz 32 Yes 180W 2666 MHz DDR4 $4,200
9
EPYC vs Skylake-SP: HEP/NP PerformanceBenchmarks
HEPSPEC06“all_cpp” subset of SPEC-CPU2006 run in parallel
CERN Cloud Benchmark SuiteVarious benchmarks, run in parallel
DB12WhetstoneATLAS KV
Unless noted, memory configured to utilize all 8 channels per CPU on EPYC, and 6 channels per CPU for Skylake-SP, with at least 2 GB RAM/logical core
~11% HS06 performance degradation seen for EPYC 7441 whenonly populating half of the memory channelsAll 2666 MHz DDR4
Noted dual rank (DR) DIMMs downclocked to 2400 MHz for EPYC
All run under SL/CentOS/RHEL 7
SMT/Hyperthreading enabled, unless otherwise indicated
Systems are dual CPU, unless noted
10
EPYC HEPSPEC06: SMT Off vs On
CPU0
200
400
600
800
1000
1200
1400
368
489541
780
883
1101
872
1148
1078
1296
EPYC 7401P@2.0 GHz [Uniprocessor - 24 threads]EPYC 7401P@2.0 GHz [Uniprocessor - 48 threads]
EPYC 7351@2.4 GHz [32 threads]EPYC 7351@2.4 GHz [64 threads]EPYC 7451@2.3 Ghz [48 threads] DDR-2400EPYC 7451@2.3 Ghz [96 threads] DDR-2400EPYC 7551@2.0 GHz [64 threads]EPYC 7551@2.0 GHz [128 threads]EPYC 7601@2.2 GHz [64 threads] DDR4-2400EPYC 7601@2.2 GHz [128 threads] DDR4-2400
EP
YC
7401P
SM
T
EP
YC
7351
EP
YC
7351 S
MT
EP
YC
7451 S
MT EP
YC
7551
EP
YC
7551 S
MT
EP
YC
7601
EP
YC
7601 S
MT
EP
YC
7451
HS06
EP
YC
7401P
25%+ HS06 performance improvement with SMT (“hyperthreading”) enabled
11
EPYC vs Skylake-SP: HEPSPEC06
CPU0
200
400
600
800
1000
1200
1400
394
729
790
10681035
1261
489
780
11011148
1296
Xeon Gold 5115@2.4 GHz [40 threads] +Xeon Gold 6130@2.1 GHz [64 threads]Xeon Gold 6136@3.0 GHz [48 threads]Xeon Gold 6148@2.4 GHz [80 threads]Xeon Gold 6150@2.7 GHz [72 threads]Xeon Platinum 8170@2.1 GHz [104 threads] *EPYC 7401P@2.0 GHz [Uniprocessor - 48 threads]EPYC 7351@2.4 GHz [64 threads]EPYC 7451@2.3 Ghz [96 threads] DDR-2400
EPYC 7551@2.0 GHz [128 threads]EPYC 7601@2.2 GHz [128 threads] DDR4-2400
Xeon
Gold 61 3
0
Xeon
Gold 61 3
6
Xeon
Gold 61 4
8
EP
YC
7401P
Xeon
Platinu
m 817
0
EP
YC
7351
EP
YC
7451
EP
YC
7551
EP
YC
7601
Xeon
Gold 61 5
0
HS06
+ = System using only 3 memory channels per CPU* = Value reported by CERN
Xe
on Gold 51
15
12
EPYC vs Skylake: HEPSPEC06 (Cont.)
Larger values are better
Similar maximum HS06 (~1,275) performance for the models testedData for highest level EPYC (7601), but not highest model Skylake-SP (8180M)Can assume Xeon Skylake 8180M would perform better than the 8170 value listed
Same number of cores/threads as 8170, but higher clock speed2.5 GHz vs 2.1 GHz
Mid-range model HS06 performance also similar~700 HS06 - ~1100 HS06
TDP somewhat higher for EPYC CPUs vs Xeon Gold, in general165 W max Xeon Gold, vs 180 W max EPYCCan likely expect EPYC to use a bit more power as a result
13
EPYC vs Skylake-SP: CERN Cloud Benchmarks
DB12 (aggregate) Whetstone (aggregate) ATLAS KV (aggregate)0
200
400
600
800
1000
1200
1400
220
114
15
998
262
65
733
210
67
1256
361
120
Xeon Gold 5115@2.4 GHz [40 threads] +Xeon Gold 6150@2.7 GHz [72 threads]EPYC 7351@2.4 GHz [64 threads]EPYC 7551@2.0 GHz [128 threads]
Xeon
Gold 61 5
0
+ = System using only 3 memory channels per CPU
Xe
on Gold
5115
EP
YC
7351
EP
YC
7551
Xeon G
old 51 15
Xeon
Gold 61 50
EP
YC
7351
EP
YC
7551
Xeon G
old 5115
Xeon G
old 61 50
EP
YC
7351
EP
YC
7551
Dirac HS06 Est.
Events/Sec
BWIPS
14
Results for a limited number of CPUs: only possible to run the full suite(including KV) on systems with CVMFS client installed/setup
By default the CERN cloud benchmarks run one instance of a benchmarkper logical core in parallel
However, only reports performance per logical coreInterested in aggregate system performance, not performance/logical core
For DB12, and Whetstone, simply multiplied result by number of logicalcoresFor KV, average seconds/event per logical core is reported
Took the inversion, and multiplied by the number of logical coresto obtain total events/sec
Larger graphed values are better
DB12 and Whestone results fairly in line with HS06
Expected somewhat better KV performance for the Xeon Gold 6150
EPYC/Skylake: CERN Cloud Benchmarks (Cont.)
15
EPYC vs Skylake-SP: CPU HS06/Dollar
CPU0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.32
0.19
0.160.17
0.15
0.09
0.45
0.35
0.23
0.17
0.15
Xeon Gold 5115@2.4 GHz [40 threads] +
Xeon Gold 6130@2.1 GHz [64 threads]Xeon Gold 6136@3.0 GHz [48 threads]Xeon Gold 6148@2.4 GHz [80 threads]Xeon Gold 6150@2.7 GHz [72 threads]Xeon Platinum 8170@2.1 GHz [104 threads]EPYC 7401P@2.0 GHz [Uniprocessor - 48 threads]EPYC 7351@2.4 GHz [64 threads]EPYC 7451@2.3 Ghz [96 threads] DDR-2400EPYC 7551@2.0 GHz [128 threads]EPYC 7601@2.2 GHz [128 threads] DDR4-2400
Xe
on Gold 61
30
Xe
on Gold 61
36
Xeon
Gold 61 4
8
EP
YC
7401P
Xe
on P
latinum
8170
EP
YC
7351
EP
YC
7451 EP
YC
7551
EP
YC
7601
Xeon G
old 61 50
HS06/$
Only retail CPU cost accounted for in calculationsDoes not represent reality given memory and base server pricing
+ = System using only 3 memory channels per CPU
Xeon
Gold 51 1
5
16
EPYC vs Skylake-SP: Estimated 25kHS06 Cost
CPU0
50
100
150
200
250
300
350
400
450
332
256 258
233
257
417
256
189
221
256 251
Xeon Gold 5115@2.4 GHz [6*16 GB DIMMs, 64 servers] +
Xeon Gold 6130@2.1 GHz [12*16 GB DIMMS, 34 servers]
Xeon Gold 6136@3.0 GHz [12*8 GB DIMMs, 32 servers]
Xeon Gold 6148@2.4 GHz [12*16 GB DIMMs, 23 servers]
Xeon Gold 6150@2.7 GHz [12*16 GB DIMMs, 24 servers]
Xeon Platinum 8170@2.1 GHz [12*32 GB DIMMS, 20 servers]
EPYC 7401P@2.0 GHz [8*16 GB DIMMs, 51 servers]
EPYC 7351@2.4 GHz [16*8 GB DIMMs, 32 servers]
EPYC 7451@2.3 GHz [16*16GB DIMMs, 23 servers]
EPYC 7551@2.0 GHz [16*16 GB, 22 servers]]
EPYC 7601@2.2 GHz [16*16 GB, 19 servers]
Xeon
Gold 61 3
0
Xe
on Gold 61
36
Xeon
Gold 61 4
8
EP
YC
7401P
Xe
on Platinu
m 81
70
EP
YC
7351
EP
YC
7451
EP
YC
7551
EP
YC
7601
Xe
on Gold 61
50
$1k
Estimated total cost of 25kHS06 +-500HS06Assuming $1,500 irreducible server cost, and retail CPU/memory pricing
Xe
on Gold 51
15
+ = System using only 3 memory channels per CPU
17
Server counts to achieve 25kHS06 +-500HS06 (+-2%)
Majority of compute node cost in CPU and memoryAssuming no excessive local storage space or IOPs requirements
Typically the case for HEP/NP
Retail CPU costs used in estimateLikely to receive volume or competitive discounts
Estimate assumes a server without CPUs and memory costs $1,500Includes power supply, disk, NIC, etc.Only accounts for cost of servers themselves. Associated costs suchas racks, network switches, integration, shipping, etc. not includedServer vendors typically increase base prices for servers which supporthigher performing CPU models with higher TDP (i.e. due to bigger PSUs, etc.)
$1,500 server base cost may be lower than reality for systems with higher end CPUs
Memory costs: retail Samsung server DIMM pricingDDR4 2666 MHz, ECC, registered
8 GB DIMM - $13716 GB DIMM - $20832 GB DIMM - $380
EPYC vs Skylake-SP: Est. 25kHS06 Cost (Cont.)
18
Enough memory populated per system to provide 2 GB/logical coreFairly standard for HEP/NP
Ensured all 6 (Skylake) or 8 (EPYC) memory channels per CPU utilized formaximum bandwidth, and NUMA performance
Often ended up with more RAM than required to satisfy 2 GB/logical core6 channels makes this particularly difficult for Skylake: installed memory not a power of two
Problem compounded by server manufacturers not offering smaller(i.e. 4 GB), less expensive DDR4 DIMMs
Estimated cost for 25kHS06 fairly similar between Skylake-SP Xeon Gold and EPYC servers: most close to $250k
Dual-CPU EPYC 7351 systems appear very cost effective, however: est. $189k~25% less than the Xeon Gold 6148 Required memory/logical core and DIMM channel parityCPU itself is inexpensive compared to other Skylake/EPYC counterparts
EPYC 7401P uniprocessor system HS06/$ for CPU cost initially looked promising
Large number of servers required: irreducible per server cost added upEstimated in the $250k range like many of the other CPUs
EPYC vs Skylake-SP: Est. 25kHS06 Cost (Cont.)
19
Side-Channel Attacks
Jan 2018 - New class of side-channel information disclosure vulnerabilities in CPU hardware made public
Meltdown, SpectreExploit speculative execution and caching optimizations in CPUs
List of similar side-channel attack vectors continues togrow
Speculative Store Bypass VulnerabilityForeshadow (L1TF)
Meltdown Spectre
Microcode updates for Skylake-SP released for all of the above
AMD claims EPYC is not vulnerable to Meltdown or ForeshadowDue to existing protections in their paging architectureReleased microcode updates for Spectre
20
The Future: EPYC 2 and Cascade LakeEPYC 2 - Rome
Expected in early 20197 nm processSupport for DDR4-3200 MHz DIMMs expected
Still 8 channelsMax core count per socket increased to 64 (128 threads)AVX512?
Cascade Lake XeonExpected end of 201814 nm process
10 nm process expected in Ice Lake in 2020Max memory speed: DDR4-2600 DIMMs
Still 6 channelsMax core count per socket remains at 28 (56 threads)Expected to support VNNI instructions
Utilizes AVX512 unitsSupport for Optane 3D Xpoint memoryAnnounced that the Frontera supercomputer at TACC will be Cascade Lake based – estimated to provide 35-40 PFLOPS without GPUs
Both CPUs will have Spectre mitigations built into hardware
21
Conclusions
The EPYC MCM architecture is considerably different from Skylake-SP’s single die configuration
Similar HEP/NP benchmark performance from mid/upper range Skylake-SP Xeon Gold and mid/upper range AMD EPYC CPUs
Pricing also similarHowever, dual-CPU EPYC 7351-based systems appear to be a sweet spot for applications requiring 2GB/logical core (somewhattypical for HEP/NP software)
Competition in the server CPU market will likely help reduce cost and spur innovation
EPYC2 (Rome) with its 7nm process and 64 physical cores appears poised to disrupt the existing balance of server CPU market share