VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts

Capacity Jail Break: vSphere 5 Space Reclamation

Nuts and Bolts

Aboubacar Diare, Hewlett-Packard

Abid Saeed, VMware

STO4907

#STO4907

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2

Disclaimer

All information related to non release ESX versions discussed in this

presentation is a shared as a technology preview and is subject to

change without notice


Agenda

1. Problem statement

2. What is UNMAP

− VAAI UNMAP recap

− UNMAP description

3. UNMAP in ESXi5.x

4. Test configuration

5. Results and considerations

6. Summary

7. Call to action


Problem Statement

VMFS datastore freed capacity from VM deletions is not release automatically at the storage and when a thin provisioned LUN is used

If this capacity is never re-used by the VMFS file system. It amounts to stranded capacity allocated to a volume no longer using this capacity

For improved capacity utilization efficiency, it is ideal to jailbreak this capacity so it may be consumed where needed

VMware ESX VMFS

Storage TP

SPACE


VAAI Space Reclamation Support

Primitive Equivalent

Names

ESX & vCenter Name SCSI Command

ES

Xi5

Pri

mit

ives

UNMAP • Space

Reclamation • /VMFS3/EnableBlockDel

ete

UNMAP

(SCSI OP Code:42h)

OUT OF

SPACE

CONDITION

• TP STUN

• VM Pause

• Temporary

lack of LBA

• Permanent

lack of LBA

NA

Return codes: • Not Ready Allocation in progress

• Chk Condition Allocation failed write

protect

QUOTA

EXCEEDED

BEHAVIOR

• TP Soft

threshold

reached

NA Return codes: • Unit Attention – Thin provisioning

Soft threshold reached

TP LUN

REPORTING

• Report thin

provisioning

capability

NA Read Capacity TPE Bit

(SCSI OP Code: 16h)


• When a LUN is thin provisioned, the

storage target assigns physical blocks

for each LBA on a LUN on the first write to

that LBA

• When a VMFS datastore is created on a

LUN, the file system allocates blocks as per

user requests and tracks the free blocks

• As files are created, written to and deleted

on the datastore, a thin provisioned LUN

effectively becomes “thick provisioned” –

i.e. all the LBA on the LUN are written to

and thus have physical storage assigned

• UNMAP is a SCSI primitive that allows

VMFS to communicate to the storage

target that certain LBA are no longer

used on a LUN backing the datastore

With VAAI

VMware ESX VMFS

Storage TP

SPACE

What Is UNMAP


Used Capacity (vCenter & Array Views)

After UNMAP

Before UNMAP

After VM deletion


Array Level vs. ESX Level Thin Provisioning

20GB

Thick or Thin SAN

Volume

VM

20GB

Thin SAN Volume

VM

Thin VMDK

20GB

Thick VMDK

VMFS VMFS

VMware ESXi VMware ESXi

Thin VMDK


Does Your Device Support UNMAP

ESXi5.0 / ESXi5.1 / ESXi5.5

esxcli storage core device vaai status get –d

esxcli system settings advanced list –option

/VMFS3/EnableBlockDelete (no affect on system for ESXi5.0 patch2, ESXi5.0Ux, ESXi5.1Ux and ESXi5.5)


VMWare vCG/HCL VAAI UNMAP Support


UNMAP in ESXi5.x Introduction

• In vSphere 5.0 GA, as part of a file delete operation, VMFS would

issue UNMAP commands to notify the storage target of the LBA

that were being marked as “free” in the FS metadata

‒ File delete operations were bound to the time taken by storage

targets to complete the UNMAP commands. vSphere operations

that involved a file delete failed when the storage target delayed

completing UNMAP commands

• To handle the above problem, in vSphere 5.0U1 and vSphere 5.1,

VMFS stopped issuing UNMAP commands as part of the file

delete operation. These releases introduced a command line

triggered “asynchronous” way of issuing UNMAP commands on

ALL the unused blocks on a VMFS volume (vmkfstools –y)

• In ESXi5.5 the vmkfstools –y was replaced with an esxcli

command and the reclaim operation was improved


UNMAP in ESXi5.x


Space Reclamation in ESXi5.0

Synchronous ‒ IDEAL

HP 3PAR

Disk Array

1

2

4

3

User deletes a VM/VMDK

VMFS frees the blocks in VMFS metadata

Storage frees physical blocks

ESX sends UNMAP commands to the storage

VMFS free capacity Storage free capacity

VMFS

unmap

STEP 1

VM

VM

VM

VM


Space Reclamation in ESXi5.0patch2

HP 3PAR

Disk Array

1

2

4

3



Storage DO NOT free physical blocks

NO UNMAP commands to the storage


VMFS

STEP 1

VM

VM

VM

VM


Space Reclamation in ESXi5.0Ux, 5.1Ux and 5.5

Asynchronous

1

2

4

3



Storage frees physical blocks

ESX sends UNMAP commands to storage to

reclaim temp file allocated capacity.

HP 3PAR

Disk Array

VMFS

1 User runs vmkfstools / esxcli command

STEP 1

STEP 2

unmap


2 VMFS inflates temp file

?

VM

VM

VM

VM


What Is VMFS Free Capacity?

VMFS only view

VM [50GB]

VM [180GB]

VM [160GB]

VM

[80GB]

FREED [100GB]

FREED

[10GB]

NEVER USED

[160GB]

NEVER USED

[100GB]

FREED

[20GB]

NEVER

USED

[10GB]

FREE CAPACITY= Freed + Never used

FREE CAPACITY= 400GB

FREED

[200GB]

VM [200GB]

FREE CAPACITY= 600GB


Space Reclamation in ESXi5.0Ux and ESXi5.1Ux

Stack view

VM [100GB]

VMDK [200GB]

VM VMD

K VM VMD

K

Thin

VMFS

Volume

VMDK

Thin

Thick

Physical

Storage

0

0

5TB

5TB

VMFS Free

Capacity

550GB

Storage Free

Capacity

550GB

vmkfstools –y 50 RECLAIM: 50% of 850GB = ANY 425GB of 850GB

UNMAP READ WRITE WRITE

425GB

[200GB]

[50G

B]

[50G

B]

[50G

B]

[200GB] [300GB]

VMFS Free

Capacity

850GB

UNMAP


FREE CAPACITY =

Freed + Never used

This command will reclaim 50% of the free

capacity all at once. Default is 60% of the free

capacity.


Space Reclamation in ESXi5.0Ux and ESXi5.1Ux

• vmkfstools –y X (X percentage of free capacity)

• X defaults to 60%

• Must be run via CLI from the ESX host

• Must navigate to root of VMFS filesystem to UNMAP

• Utilizes a temp balloon file: .vmfsBalloonsXZYAB

• Max size of balloon file is 2TB

- Bug in ESXi5.1 will truncate larger than 2TB reclaims down to 2TB

- Fix in upcoming patch of ESXi5.1 will create multiple 2TB files

• Returned capacity is random


Space Reclamation in ESXi5.5 (Tech Preview)

Stack view

VM [100GB]

VMDK [200GB]

VM VMD

K VM VMD

K

Thin

VMFS

Volume

VMDK

Thin

Thick

Physical

Storage

0

0

5TB

5TB

FREE CAPACITY =

Freed + Never used

VMFS Free

Capacity

550GB

Storage Free

Capacity

550GB

esxcli storage vmfs unmap –u uuid –n reclaim_unit

RECLAIM: 100% of 850GB in 200MB chunks

[50G

B]

[50G

B]

[50G

B]

[200GB] [300GB]

VMFS Free

Capacity

850GB

UNMAP READ WRITE WRITE UNMAP

[200GB] [250GB]

Storage Free

Capacity

850GB

Temp VMDK

This command will reclaim 100% of the free

capacity in chunks. The chunk size is specified

with the reclaim_unit value. Default 200 blocks. VMFS free capacity Storage free capacity


Space Reclamation in ESXi5.5 (Tech Preview)

• esxcli storage vmfs unmap –u uuid –n reclaim_Unit

• vmkfstools –y <datastore_path> --reclaimBlockUnit X

(deprecated)

• Reclaim Block Unit defaults at 200 blocks - 200MB on VMFS5, VMFS3 depends of FS block size (ex: 1600MB for 8MB blk size VMFS) .

• Can be run through vMA, vCLI (esxcli)

• Utilizes a temp balloon file: .asyncUnmapFile

• Max size of balloon file is ~64TB

• Max reclaimBlockUnit is ~max file size

• 100% capacity returned


So What’s in it for Me?

In depth understanding of how the ESXi

UNMAP process works

Avoid downtime and reclaim

capacity online


Test Configuration


Test Configuration/Methodology

- Storage VMotion VMs from DS1 to DS2.

- Reclaim freed capacity on DS1 using

command line tools


Results and Considerations


Considerations

Free Capacity reservation

Reclaim efficiency

Performance impact


ESXi5.0Ux ‒ ESXi5.1Ux

Free Capacity Reservation

ESXi5.5

[600GB] [600GB]

[600GB]

[600GB]

VM VM

VM VM

VM

VM

[600GB] VM VM

[600GB] VM

VM

vmkfstools –y 30

vmkfstools –y 60

vmkfstools –y 99

esxcli storage unmap --reclaimBlockUnits 200

esxcli storage unmap --reclaimBlockUnits 300GB

VM esxcli storage unmap --reclaimBlockUnits 600GB

VM


Considerations


Reclaim efficiency

Performance impact


Reclaim Efficiency

vmkfstools / esxcli before and after capacity utilization – STORAGE CAPACITY VIEW

ESXi5.5 ESXi5.1

0

5

10

15

20

25

30

0

100000

200000

300000

400000

500000

600000

30% 60% 99% 200MB 400MB 800MB 1600MB 3200MB 6400MB 12800MB

(mins) (MB)

ESXi5.1 and ESXi5.5 Reclaim efficiency and performance for ~150GB reclaim

Used Capacity before reclaim Used Capacity after reclaim Used capacity goal Reclaim Duration


ESXi5.1 30% Iterative Reclaim Efficiency

0

0.5

1

1.5

2

2.5

3

3.5

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

Iteration 1 Iteration 2 Iteration 3 Iteration 4

(Du

rati

on

min

s)

(Us

ed

Ca

pa

city

MB

)

30% reclaim efficiency – USED STORAGE CAPACITY

Used Capacity before reclaim Used Capacity after Reclaim Used Capacity goal Reclaim Duration


Considerations


Reclaim efficiency

Performance impact


ESXi 5.1 Performance Impact

30% reclaim of ~150GB Heading Heading

Duration 1 min 3 sec

Capacity

reclaimed

3793MB

~1%

Peak latency

observed ~9 ms.





Capacity

reclaimed

10041 MB

~7%

Peak latency

observed 9 ms.





Capacity

reclaimed 99%

Peak latency

observed ~10 ms.


Heading Heading


Capacity

reclaimed 100%

Peak latency

observed ~11 ms.

ESXi 5.5 Performance Impact (Tech Preview)

12800MB per iteration reclaim ~150GB

ESXi5.1


Capacity

reclaimed 99%

Peak latency

observed 10 ms.


Performance Impact for Larger Capacity Reclaim


ESXi5.1 Performance Impact

99% reclaim of 720GB

Heading Heading


Capacity

reclaimed 99%

Peak latency

observed ~10 ms.


Heading Heading


Capacity

reclaimed 100%

Peak latency

observed ~11 ms.

ESXi5.5 Performance Impact (Tech Preview)

12800MB per iteration reclaim of ~720Gb


UNMAP in ESXi5.1 vs. ESXi5.5 (Tech Preview)

Comparison summary

ESX5.1/ESXi5.0U1 ESXi5.5

Capacity reservation High Low

Reclaim efficiency Random 100%

Maintenance window recommended to

reclaim 100% YES

NO (array & capacity

dependent)

Uses esxcli cmd NO YES

Uses vmkfstools cmd YES YES (deprecated)

Performs better at higher reclaim

increments NO

YES (Up to a threshold &

dependent on various

factors)

Support for multiple descriptors in

UNMAP command NO YES

Efficiently reclaim small capacity chunks

online as they are freed NO YES


In Sum


Sync UNMAP in vSphere 5.0 GA

Pros:

• Freed VMFS capacity is immediately reclaimed in storage capacity pool

• VMFS issues UNMAP only on those LBA that were actually written to (FREE CAPACITY RESERVATION)

• 100% reclaim efficiency. (RECLAIM EFFICIENCY)

Cons: • Possible failures of SVMotion, Snapshot consolidation and other

hypervisor operations

• Impact on VM workloads (PERFORMANCE IMPACT)

• Array ability to quickly turn around an UNMAPed block if immediately allocated

FREE CAPACITY

RESERVATION

RECLAIM EFFICIENCY

PERFORMANCE IMPACT(*)

HP 3PAR StoreServ UNMAP

(*) Performance impact is array

and workload dependent

Reclaim not supported


Async UNMAP in vSphere 5.0Ux and 5.1Ux

Pros: • The file delete operations are decoupled from VM operations

Cons: • Freed blocks aren’t tracked. UNMAP sent percentage of free blocks

• FS block allocation may fail when the UNMAP operation is running (FREE CAPACITY RESERVATION)

• Nondeterministic reclaim inefficiency (RECLAIM EFFICIENCY)

• Higher impact on VM workloads as reclaim % increases (PERFORMANCE IMPACT)

FREE CAPACITY

RESERVATION

RECLAIM EFFICIENCY





Offline reclaim suitable


Async UNMAP in vSphere 5.5 (Tech Preview)

Pros: • The file delete operations are decoupled from VM operations

• FS block allocation will not fail when the UNMAP operation is running (FREE CAPACITY RESERVATION)

• 100% reclaim efficiency (RECLAIM EFFICIENCY)

• Allows online capacity reclaim with less overall impact on workload (PERFORMANCE IMPACT)

• Able to reduce the number of UNMAP commands required with multiple descriptor per command support

• The esxcli implementation allows users to script the operation across hosts over the network

Cons: • Freed blocks aren’t tracked. UNMAP sent to all free blocks all the time.

FREE CAPACITY

RESERVATION

RECLAIM EFFICIENCY



Online & offline reclaim suitable




HP Arrays UNMAP Support

Array UNMAP Support

HP 3PAR StoreServ YES

HP P9500 YES

HP StoreVirtual Planned

HP P6000 Planned

HP P2000 NO


Q & A



Call to Action

Attend or download these sessions:

STO4907: Capacity Jail Break: vSphere 5 space reclamation nuts and bolts

Monday, 8/26: 2:30pm – 3:30pm

VSVC6656: OpenStack for the Enterprise

Tuesday, 8/27: 11:30am – 12:30pm

STO5545: Top 10 things you must know about storage for vSphere

Tuesday, 8/27: 5:00pm – 6:00pm

EUC5672: Implementing a Scalable and Highly Available Desktop and Application

Architecture with a VMware AlwaysOn Solution

Wednesday, 8/28: 11:30am – 12:30pm

STO5787: Storage - The Next Frontier of Virtualization –

How VMware technologies can enable and accelerate Software Defined Storage

Wednesday, 8/28: 2:30pm – 3:30pm

Talk to HP experts

Booth 1405

• More then 20 demos including Cloud, Virtualization, and VMware integration

• On-going theater sessions

Booth 2235

• Software Defined Zone


https://vmworld2013.activeevents.com/connect/sessionDetail.ww?SESSION_ID=5672






THANK YOU

Capacity Jail Break: vSphere 5 Space Reclamation

Nuts and Bolts

Aboubacar Diare, Hewlett-Packard

Abid Saeed, VMware

STO4907

#STO4907

VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts

Technology

Transcript of VMworld 2013: Capacity Jail Break: vSphere 5 Space Reclamation Nuts and Bolts