High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and...
Transcript of High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and...
![Page 1: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/1.jpg)
High CapacitySpectrum Scale Building Block - 2019
Oliver [email protected]+49 173 31 95 701
2019 Lenovo. All rights reserved.
pro-com DATENSYSTEME GmbHHPC Solution providerGermany
![Page 2: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/2.jpg)
Startpoint
• Recap last GUG
• New Block Design
• Systems
• Performance Numbers
• Best Practices (so far)
2019 Lenovo. All rights reserved.
![Page 3: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/3.jpg)
Last Time on GUG@ISC-2018
• Introduction Lenovo High CapacitySpectrum Scale Storage Block…
– 2 Server
– 4 DS Storage Units
– 1008 Hardrives
– 2 Racks
• Based on a Costumer RFP
• Solid Performance– 15 GB/s Write / 20 GB/s Read per Block
– 3,75 GB/s Write / 5 GB/s Read per Controller
2019 Lenovo. All rights reserved. 3
![Page 4: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/4.jpg)
New Systems 2019
2019 Lenovo. All rights reserved. 4
![Page 5: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/5.jpg)
Update 2019
• Lenovo and Netapp announced Global Strategic Alliance
• Update of our Configuration
• Targets– Increase Density
– Increase Performance
– Same Price per TB Range
2019 Lenovo. All rights reserved.
![Page 6: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/6.jpg)
DE6000-H / DE600s
DE6000-H
• Dual Controller
• RAID 0,1,5,6 / Dynamic Disc Pool
• FSWA Support
– Full Stripe Write Acceleration
• 16-64 GB Cache
• 12GB SAS / 32 GB FC / 25 GbE iSCSI
• Up to 480 Drives
• 2U and 4U Versions
– 24x 2.5“ or 60x 3.5“
DE600s Expansion Chassis
• Dual ESM
• 60x 3.5“
2019 Lenovo. All rights reserved. 6
![Page 7: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/7.jpg)
DE6000-H – FSWA
• Full Stripe Write Acceleration (FSWA)
• Only with classical RAID6
• Massive performance increase
• Depending on Workload
2019 Lenovo. All rights reserved. 7
![Page 8: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/8.jpg)
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
1 MB Data Block per Array
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
FSWA
8
![Page 9: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/9.jpg)
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
9
![Page 10: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/10.jpg)
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
2 MB Data Block
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
10
![Page 11: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/11.jpg)
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
2 MB Data Block
4 MB Data Block
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
11
![Page 12: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/12.jpg)
New Block 2019
2019 Lenovo. All rights reserved. 12
![Page 13: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/13.jpg)
Setup
• How to Connected?– SAS direct attached
• How to Scale– Up to 300 Drives per Disk System
– 1200 Drives per Block
• How to Balance– 16 SAS Ports 12GB / 4x 100 GbE / IB
Ports
– 32 PCI-E Lanes vs. 32 PCI-E Lanes
2019 Lenovo. All rights reserved. 13
![Page 14: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/14.jpg)
1200 drive configuration in two racks
• 2x NSD – Server
– Lenovo ThinkSystem SR650
– 2x Intel Xeon 6242 – 16 Core 2.8 GHz
– 12x 32 GByte Memory
– 2x High Speed Adapter (IB / OPA / 100 GbE)
– 4x 12 GBIT Quad Port SAS Adapter
• 4x Back End Storage
– Lenovo ThinkSystem DE6000-4U
– Dual Controller – 8x 12 GB SAS Ports
– RAID 0,1, 5, 6, 10 – DDP Technology
– Each 300 Drives via 4x DE600s Expansion Unit
2019 Lenovo. All rights reserved.
1410HEB
A2
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
1 2
1 2
<< Storage
Expansion_DE 7
7Y69CTO1
WW
<< Storage
Expansion_DE 3
7Y69CTO1
WW
7Y80CTO1
WW
<< Storage
Expansion_DE 9
7Y69CTO1
WW
<< Storage
Expansion_DE 8
7Y69CTO1
WW
1
7Y69CTO1
WW
<< Storage_DE6000 4
7Y80CTO1
WW
Server_NSD-Server 2
7X06CTO1
WW
1
Co
mb
o P
DU
&1
p, 3
2A
/2
7Y69CTO1
WW
<< Storage
Expansion_DE 12
7Y69CTO1
WW
<< Storage
Expansion_DE 11
7Y69CTO1
WW
1
1
<< Storage
Expansion_DE 4
<< Storage
Expansion_DE 10
<< Storage_DE6000 3
12
Co
mb
o P
DU
&1
p, 3
2A
/2
12
1
A2
U
Rack_Storage-P3-R11-
12 No. MT/M or PN
Co
mb
o P
DU
&1
p, 3
2A
/2
12
7
6
5
4
3
2
13
12
11
10
9
8
19
18
17
16
15
14
25
24
23
22
21
20
31
30
29
28
27
26
37
36
35
34
33
32
1410HEB
A1
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
42
41
40
39
38
<< Storage
Expansion_DE 1
7Y69CTO1
WW
1
1
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
<< Storage
Expansion_DE 2
7Y69CTO1
WW
<< Storage
Expansion_DE 1
7Y69CTO1
WW
<< Storage_DE6000 1
7Y80CTO1
WW
<< Storage
Expansion_DE 3
7Y69CTO1
WW
1 2
<< Storage_DE6000 2
7Y80CTO1
WW
Server_NSD-Server 1
7X06CTO1
WW
6
7Y69CTO1
WW
<< Storage
Expansion_DE 5
7Y69CTO1
WW
<< Storage
Expansion_DE 4
7Y69CTO1
WW
Co
mb
o P
DU
&1
p, 3
2A
/2
12
1
1
12
12
A1
U
Rack_Storage-P3-R11-
12 No. MT/M or PN
Co
mb
o P
DU
&1
p, 3
2A
/2
<< Storage
Expansion_DE 2
7Y69CTO1
WW
<< Storage
Expansion_DE
6
5
4
3
2
1
12
11
10
9
8
7
18
17
16
15
14
13
24
23
22
21
20
19
30
29
28
27
26
25
36
35
34
33
32
31
42
41
40
39
38
37
14
![Page 15: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/15.jpg)
Benchmarks New vs. Old
2019 Lenovo. All rights reserved. 15
![Page 16: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/16.jpg)
Benchmarks
• Started begin of June 2019 ☺
• Focus– DE6000-H Scaling
- 60,120,180… Drives
– DE4000-H Scaling
– SSD Performance
– Classical Raid FSWA vs. DDP
– Directly mapped LUN to SAS link vs. DMM round-robin
– …
– …
– …
2019 Lenovo. All rights reserved. 16
![Page 17: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/17.jpg)
First Numbers DE6000
• DE6000
– 120 Drives
– RAID 6 – (8+P+Q)
– 12 LUNS
• NSD
– 2x SR650
- 2x 6134
- EDR IB
• Clients
– 24x Clients EDR
• Spectrum Scale– 5.0.3.0
– 16M Blocksize
– 16M IOR Transfer Size
2019 Lenovo. All rights reserved. 17
1
2
3
4
5
6
7
8
9
10
11
12
13
DE6000 - 120 Drives DE6000 - 120 Drives
DE6000-H-120 Drives
Write Read
![Page 18: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/18.jpg)
Predecessor – System DS6200
• DS6200
– 252 Drives
– ADAPT RAID
– 16 LUNS
• NSD
– 2x SR650
- 2x 6142
- 100 GbE
• Clients
– 24x Clients 100 GbE
• Spectrum Scale– 5.0.1.1
– 16M Blocksize
– 16M IOR Transfer Size
2019 Lenovo. All rights reserved. 18
1
2
3
4
5
6
7
8
9
10
11
12
13
DS6200 - 252 Drives DS6200 - 252 Drives
DS6200-252 Drives
Write Read
![Page 19: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/19.jpg)
DS6200 vs. New DE6000-h
2019 Lenovo. All rights reserved. 19
1,75 x higher Write Speed / 2x higher Read Speed
Write Read
1
2
3
4
5
6
7
8
9
10
11
12
13
DE6000 - 120 Drives DE6000 - 120 Drives
DE6000-H-120 Drives
1
2
3
4
5
6
7
8
9
10
11
12
13
DS6200 - 252 Drives DS6200 - 252 Drives
DS6200-252 Drives
![Page 20: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/20.jpg)
Some syntenic numbers
2019 Lenovo. All rights reserved. 20
DE6000
DE4000
![Page 21: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/21.jpg)
Bigger PictureDE6000 vs. Nativ Raid
2019 Lenovo. All rights reserved. 21
![Page 22: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/22.jpg)
Some Nativ Raid Numbers
2019 Lenovo. All rights reserved. 22
![Page 23: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/23.jpg)
Hardware Controller vs. Nativ Raid
• Nativ Raid still King– Performance & Data Integrity
• DE6000– ~70 MB/s per Disk in Write
– ~101 MB/s per Disk in Read
• Nativ Raid– ~85 MB/s per Disk in Write
– ~108 MB/s per Disk in Read
2019 Lenovo. All rights reserved. 23
Write Read
16 MB Blocksize and 16 MB Transfersize
0 2 4 6 8 10 12 14 16 18 20
DS6200 - 252 Drives
DS6200 - 252 Drives
DE6000 - 120 Drives
DE6000 - 120 Drives
DSS-G 201 - 82 Drives
DSS-G 201 - 82 Drives
DSS-G 202 - 166 Drives
DSS-G 202 - 166 Drives
Solution Performance comparison
![Page 24: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/24.jpg)
Best Practices – so far
• Planning and Implementation Guide
– Installation, Configure and Tuning
– Lenovo, Netapp and pro-com
– Available late 2019
– Lenovo Press
• Performance Level
– RAID 6 with FSWA vs. Dynamic Disk Pools
– Performance vs. Data Integrity
• Full Stripe Write Acceleration
– Optimize Segment Size vs. LUNS vs. GPFS Blocksize
• LUNs QTY
– multiple of 8 (to equally spread the LUNs across the Host Ports) - if not possible DMM round-robin
• Linux Disk I/O Settings:– Scheduler
- Noop
– reah_ahead_kb
- 128 (Linux Default)
– nr_request
- 128 (Linux Default)
– max_sectors_kb
- 2048
– …
– …
2019 Lenovo. All rights reserved. 24
![Page 25: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/25.jpg)
2019 Lenovo. All rights reserved.
![Page 26: High Capacity Spectrum Scale Building Block€¦ · Best Practices –so far •Planning and Implementation Guide –Installation, Configure and Tuning –Lenovo, Netapp and pro-com](https://reader033.fdocuments.us/reader033/viewer/2022060322/5f0d7fe87e708231d43aa9c4/html5/thumbnails/26.jpg)
2019 Lenovo. All rights reserved.