Ron Scott-Adams, VMware SABU ( @Tohuw)Joshua C. E. Fidel, VMware SABU ( @jcefidel)

STO1372BU

#VMworld #STO1372BU

Best Practices forvSAN Design:

A Real-World Perspective

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#STO1372BU CONFIDENTIAL 2

VMworld 2017 Content: Not fo

r publication or distri

bution

Quick Level Set: vSAN Design (STO1264BU)

#STO1372BU CONFIDENTIAL 3

VMware vSAN Delivers a Native vSphere Architecture

• Runs on any standard x86 server

• Pools local Storage Media into a shared datastore

• Simple, vSphere-integrated administrationvSphere vSAN

vSAN DatastoreVMworld 2017 Content: Not fo

r publication or distri

bution

Start with Use Cases (STO1885BU)

• Is this app business-critical?

• Is this a systems management or “out-of-band” tool?

• Does this app have a DMZ aspect?

• Does this app need a remote office local presence?

• Is this a Virtual Desktopor Portal app?

• Is this a cloud-native or container-based app?

4

Business Critical Virtual Desktops (VDI)

DR / DA

Cloud NativeDatabases

(SQL/Oracle)

ROBOManagement

Clusters

ContainersvSAN

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

What Does This Thing Do? (PBO1721BU)

• The app has its own complexity

– Siloed Infrastructure adds complexity

– Isolated Practices adds complexity

• What are the apps?

– Interview the business and end-users

– Know the business intent

• Native structures in the app

– App-level resiliency

– Tiered designs

– Interop with other apps

5#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

What Does This Thing Need? (SER1872BU)

• Compute Resources

– CPU

– RAM

• Storage Resources

– Performance/Throughput

– Capacity

• Design for a best possible hybrid:

– Vendor requirements

– User expectations

– Infrastructure capabilities

– Assessment data

6#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Assessing a Workload (STO1515BU)

• Assessments of an environment should be collected as data over time

– vip.vmware.com

– vRealize Operations

• This reveals trends and aspects of the workload not visible from incidental captures

7#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Assess CPU, RAM, & Storage

• Collect key metrics with meaningful measurements

– 95th Percentiles

• CPU in GHz & CPU Ready

– (Add ~10% for vSAN)

• vRAM allocated & pRAM consumed

– VMware KB 2113954

• Storage throughput/IOPS & latency

• Storage Capacity

– Assigned to VMs & consumed by VMs

– Design against the balance of both

8#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Consuming vSAN: Pre-Designed, Templates, & Build Your Own

• Pre-Designed Solutions: Take a Close Look

– VMware Cloud Foundation™ (PBO1486BU)VxRack® SDDC (PBO1064BU)VxRail® (STO1742BU)

• ReadyNodes™ are a template system (STO2095BU)

– Start all “build your own” configurations as ReadyNodes

• “Don’t Be Weird” – Supportable and Validated

• Understand the VCGs (HCLs)

– vSAN HCL: I/O Controllers and Storage Media

– vSphere HCL: Other node components

9#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Updating: VUM & Config Assist Are Your Friends (STO1968BU)

• You can only update to the version of vSAN your hardware is compatible with

• VUM is vSAN-aware in 6.6.1 to help ensure your upgrade will be certified

• You might need upgraded drivers & firmware at times

• vSAN Configuration Assist handles this

• Firmware support on:Dell, Lenovo, Supermicro, Fujitsu

10#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Storage Media Selection for vSAN (STO1540PU)

11

Caching

DataPersistence

All-Flash

Write CachingReads from Anywhere

Capacity DisksFlash Devices

Reads primarily from here

SAS SSD PCIe NVMe

vSAN

Hybrid

Read and Write Caching

Capacity DisksSAS / NL-SAS

PCIe NVMeSAS SSD

All-FlashProvides Space Efficiency• Deduplication• Compression• Erasure Coding

Enhances Performance

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Disk Group Design (STO1926BU)

12

• Each Disk Group is one cache disk with 1-7 capacity disks

• You can have five disk groups per node

• Do not place non-vSAN disks on the I/O Controller for vSAN

– KB 2129050

– Keep things simple

– Boot from SATADOM, M.2, SD Card, or disks on a separate controller

• Cache Sizing

– Hybrid is 10% of usable

– All-Flash depends on how much data is being written

– Check the vSAN Design and Sizing Guide on storagehub.vmware.com

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Capacity in vSAN: Failures to Tolerate & Fault Tolerance Method

FTT / FTM Raw Disk Required Node Count

FTT = 1, Mirroring 200% (2 copies + 1 witness) 3 (HY or AF)

FTT = 1, Erasure Coding 133% (3 stripes + 1 parity) 4 (AF only)

FTT = 2, Mirroring 300% (3 copies + 2 witnesses) 5 (HY or AF)

FTT = 2, Erasure Coding 150% (4 stripes + 2 parity) 6 (AF only)

FTT = 3, Mirroring 400% (4 copies + 3 witnesses) 7 (HY or AF)

13

With SPBM, you have

many options

vSAN Sizer can help you

size for mixed policies

Objects will never have

resilient parts on the

same node

Dedupe/Compression

gains are on top of this

Plan for 2:1 Dedupe &

Compression gains

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Capacity in vSAN: Overheads

14

• Beyond the FTT, FTM, and Dedupe/Compression…

– There are other overheads, like with any storage system

– Disks are base 10, computers count bytes in base 2: every disk is ~10% smaller

– On-Disk Format: ~1% of each disk

– Checksums: space for verification hashes

– Deduplication Table

• Also, you need slack space: ~30% of the datastore should remain free

– Keeps off the proactive rebalance point: 80% full starts shuffling data around

– A little mercy for resynchronizing data from failed disks

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

vSAN Sizing Made Easy (STO1500BU)

15

To make sizing simple, use

vsansizer.vmware.com

Free: login with your

myVMware account

Has special use case sizing

Check out STO1500BU for

review of the sizing and

assessment tools!

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Dense Storage Media and Node Considerations (STO1540PU)

• “Cheap & Deep” is sometimes neither

• 10 TB spinning drive = ~100 IOPs…

– Resync will take awhile

– Disk engagement might be too low

• Flash Storage Media isn’t slower with size…

– Want 100+ TB rebuilds from 1 node failure?

• vSphere Design: Consider fault domain size

– More nodes = more engagement and fault domains

– 64 Nodes is cool, but are more clusters better?

– 45º scale-out: add disks and add nodes

16#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

Disk Group Design Considerations

17

Node Density and Disk Population3.8

4 T

B C

apacity #

1

1.6

TB

Cache A

3.8

4 T

B C

apacity #

2

3.8

4 T

B C

apacity #

3

3.8

4 C

apacity #

4

3.8

4 T

B C

apacity #

5

3.8

4 T

B C

apacity #

6

800 G

B C

ache B

3.8

4 T

B C

apacity #

7

400 G

B C

ache C

3.8

4 T

B C

apacity #

8

3.8

4 T

B C

apacity #

9

3.8

4 T

B C

apacity #

10

400 G

B C

ache D

3.8

4 T

B C

apacity #

11

3.8

4 T

B C

apacity #

12

3.8

4 T

B C

apacity #

13

400 G

B C

ache E

3.8

4 T

B C

apacity #

14

3.8

4 T

B C

apacity #

15

800 G

B C

ache A

400 G

B C

ache B

400 G

B C

ache A

3.84 TB Total Raw Capacity1.6 TB Total Cache

23.04 TB Total Raw Capacity1.6 TB Total Cache

57.6 TB Total Raw Capacity2 TB Total Cache

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

Disk Group Design Considerations

18

Disk Population and Controller Configurations3.8

4 T

B C

apacity #

1

3.8

4 T

B C

apacity #

2

3.8

4 T

B C

apacity #

3

3.8

4 C

apacity #

4

3.8

4 T

B C

apacity #

5

3.8

4 T

B C

apacity #

6

3.8

4 T

B C

apacity #

7

400 G

B C

ache C

3.8

4 T

B C

apacity #

8

3.8

4 T

B C

apacity #

9

3.8

4 T

B C

ap

acity #

10

400 G

B C

ache D

3.8

4 T

B C

apacity #

11

3.8

4 T

B C

ap

acity #

12

3.8

4 T

B C

ap

acity #

13

3.8

4 T

B C

ap

acity #

14

3.8

4 T

B C

ap

acity #

15

76.8 TB Total Raw Capacity1.6 TB Total Cache

400 G

B C

ache B

400 G

B C

ache A

3.8

4 T

B C

ap

acity #

16

3.8

4 T

B C

ap

acity #

17

3.8

4 T

B C

ap

acity #

18

3.8

4 T

B C

ap

acity #

19

3.8

4 T

B C

ap

acity #

20

I/O ControllerController #2Controller #1

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

(Em

pty

)

Disk Group Design Considerations

19

Disk Population and Controller Configurations3.8

4 T

B C

apacity #

1

3.8

4 T

B C

apacity #

2

3.8

4 T

B C

apacity #

3

3.8

4 C

apacity #

4

3.8

4 T

B C

apacity #

5

3.8

4 T

B C

apacity #

6

3.8

4 T

B C

apacity #

7

600 G

B C

ache C

3.8

4 T

B C

apacity #

8

3.8

4 T

B C

apacity #

9

3.8

4 T

B C

ap

acity #

10

3.8

4 T

B C

ap

acity #

11

3.8

4 T

B C

ap

acity #

12

3.8

4 T

B C

ap

acity #

13

3.8

4 T

B C

ap

acity #

14

3.8

4 T

B C

ap

acity #

15

80.64 TB Total Raw Capacity1.8 TB Total Cache

600 G

B C

ache B

600 G

B C

ache A

3.8

4 T

B C

ap

acity #

16

3.8

4 T

B C

ap

acity #

17

3.8

4 T

B C

ap

acity #

18

3.8

4 T

B C

ap

acity #

19

3.8

4 T

B C

ap

acity #

20

Controller #1

3.8

4 T

B C

ap

acity #

21

Controller #2

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Physical Networking for vSAN (STO1193BU)

• Provide a 10 Gb Ethernet network

– 1 Gb supported for hybrid…

– But should you?

• Simple is good

– vSAN traffic is just between vSAN nodes

– East-West traffic is a challenge for FEX’s

– No multicast needed in 6.6+

– Jumbo Frames are usually inconsequential

• Provide a discrete VLAN

– More secure and reliable

– Separate vSAN clusters onto their own VLANs

20#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Virtual Networking for vSAN

• vSAN operates via vmkernel ports

• vmkernel ports need a connection to VNICs

• Use a Virtual Distributed Switch (VDS)

– Free with your box of vSAN!

– Allows for centralized configuration

– Network I/O Control (NIOC)

– The responsible virtual networking choice

21#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Active/Standby Uplink Distribution

• Arrows indicate Active/Standby relationships

• Example: Active for VM traffic is “A”, Standby is “B”

22

VM

vMotion

Management

vSAN

A B C D

Replication

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Storage Policy-Based Management (STO1179BU)

vSAN surfaces many unique storage capabilities to vCenter Server for resiliency, performance, and planning purposes

23

When the policy is selected, vSAN rearranges the objects in real time with no need to power off or restart workloads!

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Cluster Configuration (STO1960BU)

• Deduplication/Compression

– Activates both across the whole cluster, generally minimal overheads

– Performs a Disk Format Conversion (DFC)

– Each disk group is a deduplication domain

– Data and dedupe table are striped verticallyacross the disk group

24

• vSAN Native Encryption

– Encrypts the whole datastore at once

– Works with deduplication

– Requires a KMIP 1.1 Keyserver

– Can transition to/from vSphere VM Encrypt

#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

Special Topologies

• Fault Domains for ”Rack Awareness”

– STO1179BU

– Multiple nodes per fault domain

• Stretched Clusters for “Active/Active Sites”

– STO1118BU

– Size one vSAN cluster, double it to Site 2

• Site Recovery Manager for “DR Sites”

– STO2063BU

– 2 independent vSAN clusters, built differently

• 2-Node Direct Connect for “ROBO Sites”

– STO1994BU

– 2 nodes, each have a copy

25#STO1372BU CONFIDENTIAL

vSAN 2-Node

Cluster

RAID-0

vSphere vSAN

Cluster

RAID-6

WITNESS

RAID-6

RAID-1RAID-0

StretchedCluster

VMworld 2017 Content: Not fo

r publication or distri

bution

Backing up vSAN (STO1770BU)

• Backing up vSAN is Easy

– vSphere APIs for Data Protection means most solutions work out of the box

– Check out STO1770BU for what’s coming next

• Moving Data around within the Same System is Not a Backup!

• “3-2-1 Rule”

– 3 Copies of the backup

– 2 Storage Media

– 1 Off-Site

• Secondary Storage and Continuity Systems

26#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

The Steps to Victory (STO2652BU)

• Evaluation

– Proof of Concept

– User Acceptance Testing

– Certification

• Go Live

• Validation

• Each of the above depend on engagementand well-defined success criteria

• vSAN Proof of Concept Guide on storagehub.vmware.com

27#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

vSAN Resources for Reference

• StorageHub:storagehub.vmware.com

• Virtual Blocks: blogs.vmware.com/virtualblocks

• Virtually Speaking Podcast:vspeakingpodcast.com

• More VMworld 2017 vSAN Sessions:

• Technical Deep Dive [STO2986BU]

• Networking Design [STO3276GU]

• Troubleshooting Deep Dive [STO1315BU]

• vSAN Encryption Deep Dive [STO1451BU]

• Hardware Deep Dive Panel [STO1540PU]

• Technical Customer Panel [STO2615PU]

• Ron’s Blog: tohuw.net

• Josh’s Twitter: @jcefidel

• Cormac Hogan: cormachogan.com

• Duncan Epping: yellow-bricks.com

• John Nicholson: thenicholson.com

• Pete Koehler: vmpete.com

• Dave Morera: greatwhitetec.com

• Peter Keilty: livevirtually.net

• Paul Braren: tinkertry.com

• William Lam: virtuallyghetto.com

28#STO1372BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

VMworld 2017 Content: Not fo

r publication or distri

bution

Ron Scott-Adams, VMware SABU ( @Tohuw)

Josh Fidel, VMware SABU ( @jcefidel)VMworld 2017 Content: N

ot for publicatio

n or distribution