High CapacitySpectrum Scale Building Block - 2019

Oliver Killok@pro-com.org+49 173 31 95 701

2019 Lenovo. All rights reserved.

pro-com DATENSYSTEME GmbHHPC Solution providerGermany

Startpoint

• Recap last GUG

• New Block Design

• Systems

• Performance Numbers

• Best Practices (so far)

2019 Lenovo. All rights reserved.

Last Time on GUG@ISC-2018

• Introduction Lenovo High CapacitySpectrum Scale Storage Block…

– 2 Server

– 4 DS Storage Units

– 1008 Hardrives

– 2 Racks

• Based on a Costumer RFP

• Solid Performance– 15 GB/s Write / 20 GB/s Read per Block

– 3,75 GB/s Write / 5 GB/s Read per Controller

2019 Lenovo. All rights reserved. 3

New Systems 2019

2019 Lenovo. All rights reserved. 4

Update 2019

• Lenovo and Netapp announced Global Strategic Alliance

• Update of our Configuration

• Targets– Increase Density

– Increase Performance

– Same Price per TB Range

2019 Lenovo. All rights reserved.

DE6000-H / DE600s

DE6000-H

• Dual Controller

• RAID 0,1,5,6 / Dynamic Disc Pool

• FSWA Support

– Full Stripe Write Acceleration

• 16-64 GB Cache

• 12GB SAS / 32 GB FC / 25 GbE iSCSI

• Up to 480 Drives

• 2U and 4U Versions

– 24x 2.5“ or 60x 3.5“

DE600s Expansion Chassis

• Dual ESM

• 60x 3.5“

2019 Lenovo. All rights reserved. 6

DE6000-H – FSWA

• Full Stripe Write Acceleration (FSWA)

• Only with classical RAID6

• Massive performance increase

• Depending on Workload

2019 Lenovo. All rights reserved. 7

Q

(Parity)

P

(Parity)

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8

Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB

1 MB Data Block per Array

256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB

FSWA

8

Q

(Parity)

P

(Parity)

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8

1 MB Data Block

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

256 KB

128 KB

Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB

FSWA

9

Q

(Parity)

P

(Parity)

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8

1 MB Data Block

2 MB Data Block

256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB

Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB

FSWA

10

Q

(Parity)

P

(Parity)

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8

1 MB Data Block

2 MB Data Block

4 MB Data Block

256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB

Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB

FSWA

11

New Block 2019

2019 Lenovo. All rights reserved. 12

Setup

• How to Connected?– SAS direct attached

• How to Scale– Up to 300 Drives per Disk System

– 1200 Drives per Block

• How to Balance– 16 SAS Ports 12GB / 4x 100 GbE / IB

Ports

– 32 PCI-E Lanes vs. 32 PCI-E Lanes

2019 Lenovo. All rights reserved. 13

1200 drive configuration in two racks

• 2x NSD – Server

– Lenovo ThinkSystem SR650

– 2x Intel Xeon 6242 – 16 Core 2.8 GHz

– 12x 32 GByte Memory

– 2x High Speed Adapter (IB / OPA / 100 GbE)

– 4x 12 GBIT Quad Port SAS Adapter

• 4x Back End Storage

– Lenovo ThinkSystem DE6000-4U

– Dual Controller – 8x 12 GB SAS Ports

– RAID 0,1, 5, 6, 10 – DDP Technology

– Each 300 Drives via 4x DE600s Expansion Unit

2019 Lenovo. All rights reserved.

1410HEB

A2

Co

mb

o P

DU

&1

p, 3

2A

/2

1 2

1 2

1 2

<< Storage

Expansion_DE 7

7Y69CTO1

WW

<< Storage

Expansion_DE 3

7Y69CTO1

WW

7Y80CTO1

WW

<< Storage

Expansion_DE 9

7Y69CTO1

WW

<< Storage

Expansion_DE 8

7Y69CTO1

WW

1

7Y69CTO1

WW

<< Storage_DE6000 4

7Y80CTO1

WW

Server_NSD-Server 2

7X06CTO1

WW

1

Co

mb

o P

DU

&1

p, 3

2A

/2

7Y69CTO1

WW

<< Storage

Expansion_DE 12

7Y69CTO1

WW

<< Storage

Expansion_DE 11

7Y69CTO1

WW

1

1

<< Storage

Expansion_DE 4

<< Storage

Expansion_DE 10

<< Storage_DE6000 3

12

Co

mb

o P

DU

&1

p, 3

2A

/2

12

1

A2

U

Rack_Storage-P3-R11-

12 No. MT/M or PN

Co

mb

o P

DU

&1

p, 3

2A

/2

12

7

6

5

4

3

2

13

12

11

10

9

8

19

18

17

16

15

14

25

24

23

22

21

20

31

30

29

28

27

26

37

36

35

34

33

32

1410HEB

A1

Co

mb

o P

DU

&1

p, 3

2A

/2

1 2

42

41

40

39

38

<< Storage

Expansion_DE 1

7Y69CTO1

WW

1

1

Co

mb

o P

DU

&1

p, 3

2A

/2

1 2

<< Storage

Expansion_DE 2

7Y69CTO1

WW

<< Storage

Expansion_DE 1

7Y69CTO1

WW

<< Storage_DE6000 1

7Y80CTO1

WW

<< Storage

Expansion_DE 3

7Y69CTO1

WW

1 2

<< Storage_DE6000 2

7Y80CTO1

WW

Server_NSD-Server 1

7X06CTO1

WW

6

7Y69CTO1

WW

<< Storage

Expansion_DE 5

7Y69CTO1

WW

<< Storage

Expansion_DE 4

7Y69CTO1

WW

Co

mb

o P

DU

&1

p, 3

2A

/2

12

1

1

12

12

A1

U

Rack_Storage-P3-R11-

12 No. MT/M or PN

Co

mb

o P

DU

&1

p, 3

2A

/2

<< Storage

Expansion_DE 2

7Y69CTO1

WW

<< Storage

Expansion_DE

6

5

4

3

2

1

12

11

10

9

8

7

18

17

16

15

14

13

24

23

22

21

20

19

30

29

28

27

26

25

36

35

34

33

32

31

42

41

40

39

38

37

14

Benchmarks New vs. Old

2019 Lenovo. All rights reserved. 15

Benchmarks

• Started begin of June 2019 ☺

• Focus– DE6000-H Scaling

- 60,120,180… Drives

– DE4000-H Scaling

– SSD Performance

– Classical Raid FSWA vs. DDP

– Directly mapped LUN to SAS link vs. DMM round-robin

– …

– …

– …

2019 Lenovo. All rights reserved. 16

First Numbers DE6000

• DE6000

– 120 Drives

– RAID 6 – (8+P+Q)

– 12 LUNS

• NSD

– 2x SR650

- 2x 6134

- EDR IB

• Clients

– 24x Clients EDR

• Spectrum Scale– 5.0.3.0

– 16M Blocksize

– 16M IOR Transfer Size

2019 Lenovo. All rights reserved. 17

1

2

3

4

5

6

7

8

9

10

11

12

13

DE6000 - 120 Drives DE6000 - 120 Drives

DE6000-H-120 Drives

Write Read

Predecessor – System DS6200

• DS6200

– 252 Drives

– ADAPT RAID

– 16 LUNS

• NSD

– 2x SR650

- 2x 6142

- 100 GbE

• Clients

– 24x Clients 100 GbE

• Spectrum Scale– 5.0.1.1

– 16M Blocksize

– 16M IOR Transfer Size

2019 Lenovo. All rights reserved. 18

1

2

3

4

5

6

7

8

9

10

11

12

13

DS6200 - 252 Drives DS6200 - 252 Drives

DS6200-252 Drives

Write Read

DS6200 vs. New DE6000-h

2019 Lenovo. All rights reserved. 19

1,75 x higher Write Speed / 2x higher Read Speed

Write Read

1

2

3

4

5

6

7

8

9

10

11

12

13

DE6000 - 120 Drives DE6000 - 120 Drives

DE6000-H-120 Drives

1

2

3

4

5

6

7

8

9

10

11

12

13

DS6200 - 252 Drives DS6200 - 252 Drives

DS6200-252 Drives

Some syntenic numbers

2019 Lenovo. All rights reserved. 20

DE6000

DE4000

Bigger PictureDE6000 vs. Nativ Raid

2019 Lenovo. All rights reserved. 21

Some Nativ Raid Numbers

2019 Lenovo. All rights reserved. 22

Hardware Controller vs. Nativ Raid

• Nativ Raid still King– Performance & Data Integrity

• DE6000– ~70 MB/s per Disk in Write

– ~101 MB/s per Disk in Read

• Nativ Raid– ~85 MB/s per Disk in Write

– ~108 MB/s per Disk in Read

2019 Lenovo. All rights reserved. 23

Write Read

16 MB Blocksize and 16 MB Transfersize

0 2 4 6 8 10 12 14 16 18 20

DS6200 - 252 Drives

DS6200 - 252 Drives

DE6000 - 120 Drives

DE6000 - 120 Drives

DSS-G 201 - 82 Drives

DSS-G 201 - 82 Drives

DSS-G 202 - 166 Drives

DSS-G 202 - 166 Drives

Solution Performance comparison

Best Practices – so far

• Planning and Implementation Guide

– Installation, Configure and Tuning

– Lenovo, Netapp and pro-com

– Available late 2019

– Lenovo Press

• Performance Level

– RAID 6 with FSWA vs. Dynamic Disk Pools

– Performance vs. Data Integrity

• Full Stripe Write Acceleration

– Optimize Segment Size vs. LUNS vs. GPFS Blocksize

• LUNs QTY

– multiple of 8 (to equally spread the LUNs across the Host Ports) - if not possible DMM round-robin

• Linux Disk I/O Settings:– Scheduler

- Noop

– reah_ahead_kb

- 128 (Linux Default)

– nr_request

- 128 (Linux Default)

– max_sectors_kb

- 2048

– …

– …

2019 Lenovo. All rights reserved. 24

2019 Lenovo. All rights reserved.

2019 Lenovo. All rights reserved.