HS06 on new CPU, KVM virtual machines and commercial cloud

Michele Michelotto

1

The HEP server for CPU farm

2

Two socket Rack mountable: 1U, 2U, dual twin, blade Multicore About 2GB per logical cpu x86-64 Intel or AMD

AMD

3

4

Interlagos

5

Intel roadmap

6

New Intel NamingAfter Several generation of Xeon 5n xx

51xx (Woodcrest /Core 2c 65nm)

53xx (Clovertown / Core 4c 65nm)

54xx (Harpertown / Penryn 4c 45nm)

55xx (Gainestown / Nehalem 4c/8t 45nm)

56xx (aka Gulftown / Nehalem 6c/12t 45)

Now Xeon E5 26xx “Sandy Bridge” EP 8c/16t ( @32 nm )

7

The dual proc Xeon E5 26xx

8

Configuration Software

9

Operating System: SL release 5.7 (Boron) Compiler: gcc version 4.1.2 20080704 (Red

Hat 4.1.2-51) HEP-SPEC06 based on SPEC CPU 1.2 (32bit) HEP-SPEC06 64 bit (default config + remove

“–m32”) 2GB per core unless explicitly stated

AMD 6272 2x16core 64GB at 2.1( up to 2.6) GHz

10

AMD 6272 2x16core 64GB at 2.1 GHz in 64bit mode

11

Piledriver core

12

Substitute the “Bulldozer” core Smarter prefetching A perceptron branch predictor that supplements

the primary BPU Larger L1 TLB Schedulers that free up tokens more quickly Faster FP and integer dividers and SYSCALL/RET

(kernel/System call instructions) Faster Store-to-Load forwarding

Abu Dhabi Family

13

AMD 6378 2x16core 64GB at 2.4GHz vs 6272 2.1 GHz – 32bit

14

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 360

50

100

150

200

250

300

350

HepSpec 32bit

threads

HS

06

AMD 6378 2x16core 64GB at 2.4GHz vs 6272 2.1 GHz – 64bit

15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 360

50

100

150

200

250

300

350

400

HepSpec 64bit

threads

HS

06

Better architecture

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 361

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

Area of improvment

threads

HS

06

64bit

clock +14%

32bit

Intel vs AMD

17

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 360

50

100

150

200

250

300

350

400

Intel Xeon E5 2660 @ 2200 MHz"

Amd Opteron 6378 @ 2400 MHz

AMD Opteron 6272 @ 2100 MHz

Intel Xeon E5

18

After E5

19

E5 version 2? Ivy Bridge core with 22nm tri-gate process

Reduce TDP Up to 10 cores with 25MB (or 30?) L3 cache 15% - 20% clock update

Mining in the SPEC site

20

Selected CPU INT RATE 2006 as a proxy of HS06

Kept only measurements on dual proc configuration

Only x86-64 architecture intel or AMD

Clock range is stable

21

28 May

2005

10 Octo

ber 2

006

22 Feb

ruary

2008

06 Jul

y 200

9

18 Nov

embe

r 201

0

01 Ap

ril 20

12

14 Au

gust 2

0130

500100015002000250030003500400045005000

Dual processor X86 - Clock vs time

Since Aug 2006

22

May-05 Oct-06 Feb-08 Jul-09 Nov-10 Apr-12 Aug-130

100

200

300

400

500

600

700

800

Intel X86AMD Opteron

Elastic Cloud

23

European Project e-Fiscal Sergio Andreozzi (EGI) Kashif Iqbal (ICHEC, NUI Galway, Ireland) HS06 is the main benchmark in EGI How we compare it with Amazon Elastic Cloud

(EC2) Create Virtual Machines environment on SL Map to EC2 type of computing node

24

Xeon E5 2600 vs EC2

25

Intel Xeon E5 2600 16 core/32 logical cpu SL6.2 64GB

core mem 1 HS06 32 HS06Bare Metal 21.94 Stdev 100.00% 323.27 100.00%1 VM UP / 1 VM RUN 1 1.5 GB 21.22 0.17 96.72%32 VM UP/1 VM RUN 1 1.5 GB 19.41 0.37 88.49%32 VM UP/32 VM RUN 1 1.5 GB 9.11 0.19 291.62 90.21%

2HS06 32 HS06Bare Metal 44.04 100.00% 323.271 VM UP/ 1 VM RUN 2 3 GB 41.50 94.23%16 VM UP/1 VM RUN 2 3 GB 40.67 0.23 92.35%16 VM UP / 16 VM RUN 2 3 GB 18.73 0.35 299.68 92.70%

4HS06 32 HS06Bare Metal 82.58 100.00% 323.271 VM UP/ 1 VM RUN 4 6 GB 75.86 91.86%8 VM UP/1 VM RUN 4 6 GB 74.75 0.32 90.52%8 VM UP / 8 VM RUN 4 6 GB 36.47 0.57 291.78 90.26%

8HS06 32 HS06Bare Metal 150.38 100.00% 323.271 VM UP/ 1 VM RUN 8 7 GB 137.59 1.15 91.49%8 VM UP/1 VM RUN 8 7 GB 136.87 0.73 91.01%8 VM UP / 8 VM RUN 8 7 GB 74.03 0.72 296.12 91.60%

EC2 unitEC2 Medium 2 1 VM 3.75 GB 11.42 EC2 Large 4 2 VM 7.5 GB 18.13 EC2 Extra Large 8 4 VM 15 GB 44.41

Opteron 6272 vs EC2

26

Amd 6272 32 cores SL6.2 64GBcore mem 1 HS06 32 HS06

Bare Metal 13.26 Stdev 100.00% 244.90 100.00%1 VM UP / 1 VM RUN 1 1.5 GB 12.76 0.21 96.23%32 VM UP/1 VM RUN 1 1.5 GB 11.72 0.12 88.39%32 VM UP/32 VM RUN 1 1.5 GB 6.91 0.15 220.97 90.23%

2HS06 32 HS06Bare Metal 26.21 100.00% 244.901 VM UP/ 1 VM RUN 2 3 GB 23.84 90.96%16 VM UP/1 VM RUN 2 3 GB 23.39 0.05 89.24%16 VM UP / 16 VM RUN 2 3 GB 13.40 0.25 214.42 87.55%

4HS06 32 HS06Bare Metal 51.97 100.00% 244.901 VM UP/ 1 VM RUN 4 6 GB 46.61 0.10 89.69%8 VM UP/1 VM RUN 4 6 GB 46.01 0.29 88.53%8 VM UP / 8 VM RUN 4 6 GB 13.40 0.25 210.48 85.95%

8HS06 32 HS06Bare Metal 91.45 100.00% 244.901 VM UP/ 1 VM RUN 8 7 GB 85.42 0.32 93.41%8 VM UP/1 VM RUN 8 7 GB 85.09 0.50 93.04%8 VM UP / 8 VM RUN 8 7 GB 51.99 0.66 207.99 84.93%

EC2 unitEC2 Medium 2 1 VM 3.75 GB 11.42 EC2 Large 4 2 VM 7.5 GB 18.13 EC2 Extra Large 8 4 VM 15 GB 44.41

HS06 for Medium (SBridge)

• SPEC score < with the > no. of VMs• Virtualisation + Multi-tenancy effect on performance ~ 3.28%

to 58.48%• More realistic figure ~ 11.53 to 58.48

HPC/HTC vs. Cloud Benchmarking – eFiscal Workshop @ EGI TF 2012, Prague 27

Bare Metal

1 VM

n VMs + M

inimal lo

ad

n VMs + Fu

lly lo

aded0

5

10

15

20

25

EGIEC2

HS06

Sco

re

HS06 for Large (SBridge)

• Virtualisation + MT effect on performance ~ 9.49% to 57.47%• Note the minimal effect of > no. of VMs

HPC/HTC vs. Cloud Benchmarking – eFiscal Workshop @ EGI TF 2012, Prague 28

Bare Metal

1 VM

n VMs + M

inimal lo

ad

n VMs + Fu

lly lo

aded05

101520253035404550

EGIEC2

HS06

Sco

re

HS06 for Xlarge (SBridge)

• Virtualisation + MT effect on performance ~ 8.14% to 55.84%• Note the minimal effect of > no. of VMs

HPC/HTC vs. Cloud Benchmarking – eFiscal Workshop @ EGI TF 2012, Prague 29

Bare Metal

1 VM

n VMs + M

inimal lo

ad

n VMs + Fu

lly lo

aded0

102030405060708090

EGIEC2

HS06

Sco

re

HS06 for Medium (Opteron)

• Virtualisation + MT effect on performance ~ 3.77% to 47.89%

HPC/HTC vs. Cloud Benchmarking – eFiscal Workshop @ EGI TF 2012, Prague 30

Bare M

etal

1 VM

n VMs + M

inimal

load

n VMs + Fu

lly load

ed0

2

4

6

8

10

12

14

EGIEC2

HS06

Sco

re

HS06 for Large (Opteron)

HPC/HTC vs. Cloud Benchmarking – eFiscal Workshop @ EGI TF 2012, Prague 31

• Virtualisation + MT effect on performance ~ 9.04% to 48.88%

Bare M

etal

1 VM

n VMs + M

inimal

load

n VMs + Fu

lly load

ed0

5

10

15

20

25

30

EGIEC2

HS06

Sco

re