041011_DMP_SMB14_ver2

1

Veritas Dynamic Multi-PathingOptimized, Centrally Managed Multi-Pathing

Ashish YajnikPrincipal Product Manager

Storage & Availability Mgmt GroupDynamic Multi-Pathing

SYMANTEC VISION 2011

Datacenter Multi-Pathing Trends & Challenges

Multi-Vendor storage & server environments increasingly a norm

Single tools across physical & virtual env critical for efficiency

Configurations are continually scaling up & scaling out

Bridging System & Storage Team Communication is a challenge

2Dynamic Multi-Pathing


DMP - Optimized Multi-Vendor Multi-Pathing Solution

• One interface for all OS, storage & architecture

– Enables multi-vendor environment & standardization

– Industry’s broadest hardware compatibility list

– ~1000 arrays supported

• Enterprise grade performance

– Best in class throughput

– Proactive path management

– Automated & reliable recovery

• Bridges storage & server boundaries

– Enables informed decision making

– Reduces risk of configuration errors

• Managed by Veritas Operations Manager

– Single-pane-of-glass datacenter wide path management


http://www.emc.com/

http://www.ibm.com.au/

http://www.huawei.com/

http://xiotech.com/index.php


Agenda

Dynamic Multi-Pathing 4

Advanced Usability & ManagementBridge the Storage & Server Information GapCentrally manage multi-pathing with Veritas Operations Manager (VOM)Identify bottleneck with I/O Stat tool

Maximize Availability & PerformanceProactive Path Failover & Reliable FailbackEffective Load Balancing

HeterogeneityStandardize across physical & virtual environmentsLeverage ANY StorageStandardize across any OS & any Storage Management Stack

SYMANTEC VISION 2011Dynamic Multi-Pathing 5



HeterogeneityStandardize across physical & virtual environmentsLeverage ANY StorageStandardize across ANY OS & Storage Management Stacks


Clustering

Native MP

Volume Manger

File System

Veritas Cluster Server

Volume Manger

File System

DMP

Veritas Cluster Server

DMP

Veritas Storage Foundation

Exploiting OS Native Stacks:Introducing DMP Standalone Product


Enable DMP EVERYWHERE

Simpler for You to Standardize on DMP

• Leverage DMP’s extensive array support

• Standardize on single multi-pathing across ALL storage management stacks

• DMP reporting and management through VOM

• Seamless migration from native multi-pathing to DMP

Any OS Stack


Exploiting OS Native Stacks:DMP Standalone SCL and HCL Overview

• Enabled by DMP Native Support feature

• Oracle ASM & RAC supported with DMP Standalone

• Leverages DMP’s existing HCL – 1000+ arrays supported

7

DMP Single node Native Stack

DMP

Native FS

Native LVM

Native Multi-Pathing

Native FS

Native LVM

DMP Native Stack with Clustering

DMP

VCS

Native FS

Native LVM

Native FS

Native LVM

VCS

Native Multi-Pathing

OS Version Architecture Clustering Storage Management Stack

AIX 5.3 & 6.1 PowerPC VCS AIX LVM, ASM

RHEL4/5, SLES10, SLES11 Intel/AMD X64 (no 32 bit) VCS Linux LVM, ASM

Solaris 9, 10 SUN Sparc & X86_64 VCS SUN ZFS, ASM

HP-UX 11.31 IA & PA VCS HP-UX LVM

Windows 2003, 2008 X86, X64, IA64 VCS, MSCS Windows LDM

Dynamic Multi-Pathing

Any OS Stack


Locked-In to Storage Vendors

Exploiting ANY Storage:Eliminate Vendor Lock-In With Dynamic Multi-Pathing

Multi-Vendor Benefits

• Enabling Multi-Vendor environments saves on storage cost

• Point tools are complex to manage or don’t support all environments

• Standardize on DMP across heterogeneous OS & Storage solutions

DMX Clariion USP-V

HDLMEMC PowerPath


Storage vendor lock-in

results in 30-100% price

premiums on storage

Free to Choose the Most Cost Effective Storage

DMX Clariion IBM XIVHDS – AMS

Dynamic Multipathing (DMP)

Any Storage


Exploiting ANY Storage: Modular Architecture to Optimize to Any Storage

9

• Challenge for Optimization

– Different Array modes are hard to support

• Array Support Library (ASL)

– HW specific plug in to DDL

– Automate DMP driver configuration

– Discover array specific attributes

• Array Policy Module (APM)

– HW specific plug in to DMP

– Dynamically loadable kernel module

– Allows array-specific integration

• Allows DMP to support 50+ array families & 1000+ array models


DDL vxdmp

ANY VM

ANY FS

SCSI Drivers

HBA Drivers

DISK

APM2

APM1

ASL1 ASL2

Any Storage


Exploiting Virtualization:Enhanced VIOS Support with DMP


LPAR LPAR

Virtualized AIX Server

VIO Server

DMP

VIO Server

DMP

DMP DMP

• DMP Enhances VIOS

– Enable standardization of DMP on AIX

– DMP resides inside the VIOS

– Ability to run DMP inside LPAR as well

• Leverage extensive DMP HCL

• Deep Visibility & Reporting

– DMP in LPAR gets deep storage visibility

– Enables consistency of device names

– Informed decision making

– Centrally managed by VOM

Deep Storage Visibility

Virtual Support


Summary of DMP’s Heterogeneous Support


• All Major Unix/Linux Support

• Windows LDM Support

• Oracle ASM support with DMP Standalone

OS & Native Stack

Support

• Adding value in VIOS environments

• Multi-Pathing in LDOMs, LPARs & KVM

• Utilize DMP across Physical & Virtual environments

Virtualization Support

• All Tier1, Tier2 & major Tier3 arrays supported

• 1000~ arrays part of extensive HCL

Extensive Storage Support




HeterogeneityStandardize across physical & virtualLeverage ANY StorageStandardize across any OS


Bridging the Storage & System Information GapMapping OS Devices to SAN Devices

13

c0t5d65

c1t5d65

c4t5d65c3t5d65

c0t5d65 c3t5d65

Challenges

• Same LUN, different name across servers

• OS names do not map to storage terminology

c0t5d65

c1t5d65

c4t5d65c3t5d65

LSN: D600172E015B

EMC0_19 EMC0_19

LUN

c0t5d65

c1t5d65

c4t5d65c3t5d65

AVID: 0AF4

EMC0_0AF4 EMC0_0AF4

WITHOUT DMP

LUN LUN

LSN sorting algorithm

• Consistent names across servers

AVID naming

• Consistent names across servers

• Meaningful names for storage teams


Manageability


$>cat /etc/vx/device.custom

enclosure vendor=DGC product=DISK serial=CK200070400359 name=CLARiion

dmpnode serial=600601606D121B007C778BC48EDBDB11 name=MyApp-lun1

dmpnode serial=600601606D121B008FB6E0CA8EDBDB11 name=CLARiion-lun2

dmpnode serial=600601606D121B0091B6E0CA8EDBDB11 name=CLARiion-lun3

dmpnode serial=600601606D121B0093B6E0CA8EDBDB11 name=CLARiion-oralun1

dmpnode serial=600601606D121B00BFAB99C28FDBDB11 name=CLARiion-oralun2

Customized Device Naming

14

• Ability to change device names to user-specified names

– Admin can decide the names for the devices

– Framework for a file containing user-specified names to be uploaded to hosts

– Ability to clear the user-specified names with a CLI

– User-specified names will be made persistent

EMC Clariion

LUN

EMC Clariion

LUN

EMC Clariion

LUN

EMC Clariion

LUN

MyApp-lun1 CLARiion-lun2 CLARiion-oralun1 CLARiion-oralun2

#> vxgetdmpnames > /etc/vx/device.custom#> vxddladm assign names file=/etc/vx/device.custom

• Vxdisk list output

DEVICE TYPE DISK GROUP STATUS

MyApp-lun1 auto:cdsdisk - - online

CLARiion-lun2 auto:cdsdisk - - online

CLARiion-lun3 auto:cdsdisk - - online

CLARiion-oralun1 auto:cdsdisk - oradg online

CLARiion-oralun2 auto:cdsdisk - oradg online


Bridging Storage & Server Team Information Gap


Raid Levels – RAID5, RAID10Replication Attributes & SnapshotsThin Or Standard DevicesTier Information – SSD, SATA, FC, SAS

5

DMP Discovers Device Characteristics

My application is on tier 1 servers, what tier of storage is it using? SSD, SATA, SAS, FC?App

Team

Server Team

My mission critical file system is being protectedor not? RAID5, RAID10?

Is my critical data being replicated underneath or not?

#vxdisk –e list

DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR

USP0_af14 auto USP0_53 DFDG2 online c0t50060482CAE053ECd25s2 PVOL,SSD

USP0_af15 auto USP0_54 DFDG2 online c0t50060482CAE053ECd26s2 SATA,RAID_1

USP0_af16 auto USP0_55 DFDG2 online c0t50060482CAE053ECd27s2 FC,RAID_10

Manageability


Proactively address risks to prevent downtime

• Reduce errors in your environment & increase HA

– Manage risks in your environment with proactive reporting & alerts

– Customized health check reports for different violations


Manageability


Better environment control with DMP I/O Statistics

17

New options in vxdmpadm provides:

• Filter out ‘0’ entries

• vxdmpadm –z iostat show <selection-criteria>

• Displayed queued (-q) and errored (-e) I/Os

• vxdmpadm –q iostat show <selection-criteria>

• Per second IO statistics

• vxdmpadm –s iostat show <selection-criteria>

• Display output in units that make sense for normal admins

• vxdmpadm –u iostat show <selection-criteria>

Aggregate statistics for capacity management

• HBA port level statistics

• Groupby option to get enclosure, controller, and disk level statistics

• vxdmpadm iostat show groupby=ctlr|dmpnode|enclosure <selection criteria>

5.1 CLI Scalability & Accuracy Improvements

• Filtering & grouping of data in the kernel

• Reduces burden on CLI thus improving reporting accuracy


Manageability

SYMANTEC VISION 2011 18

$>vxdmpadm -u m –s iostat show groupby=ctlr interval=30

cpu usage = 878us per cpu memory = 32768b

OPERATIONS/SEC BLOCKS/SEC

CTLRNAME READS WRITES READS WRITES

c0 33200 11600 207.25m 248.13m

c1 33400 9040 233.24m 229.73m

c2 17600 5040 61.53m 64.62m

c3 18080 5680 98.45m 99.48m

$>vxdmpadm -u m -s iostat show groupby=enclosure ctlr=c1 interval=30

cpu usage = 1396us per cpu memory = 32768b

OPERATIONS/SEC BLOCKS/SEC

ENCLOSURENAME READS WRITES READS WRITES

emc_clariion0 17200 4000 118.97m 73.65m

ams_wms0 12000 4400 82.07m 80.01m

eva4k6k0 4000 4000 30.76m 73.65m

Better environment control with DMP I/O Statistics


Manageability




HeterogeneityStandardize across physical & virtualLeverage ANY StorageStandardize across any OS


Optimize Application I/O Throughput

20

• MinimumQueue (MinQ) I/O Policy

– Smallest HBA queue depth path chosen

– Typical work load & SAN configurations

• Adaptive I/O Policy

– Paths with lowest response time are used

– End-to-End bottleneck identification

• Adaptive-MinQ

– Combines best of MinQ & Adaptive

• Balanced

– LUN segmented in equal size partitions,

– Each partition associated with a path

• Round-Robin, Single-Active, Priority

– Standard I/O policies for native multi-pathing


HBA - A HBA - B

SP-A SP-B

Maximize Performance


Proactive Path Management:Smarter & Faster I/O Error Analysis

21

• Device Discovery Layer (DDL)

– Multi-threaded for faster discovery

• DMP Device Driver (VxDMP)

– Multi-threaded core processes

• Normal I/Os go through SCSI layer

– Chance for SCSI to recover transient errors

• DMP Error Analysis bypasses SCSI layer

• “SCSI Bypass” Benefits:

– Faster error analysis

– Asynchronous interface

– More detailed information of I/O outcome


DMP

VxVM/ ANY VM/ASM

VxFS/ANY FS

HBA Driver

OS SCSI Drivers

ErrorAnalysis

RegularIO

Maximize Availability


Proactive Path Management

22

Throttling

• If I/O is backed up on the device for 10 seconds, mark the path “Suspect”

HBA API Notifications

• When Port logout notification happens, mark the associated paths “Suspect”

Sub-Path Failover Group Protocol

• Collectively taken an action by inquiring about few paths

• Good for scale out configurations and results in speedier analysis



DMP Proactive Path Management Capabilities

– Avoids build-up of I/O on failing paths by proactive methods

– Results in quicker I/O error analysis and speedier failover


Total Throughput Impact of a Fabric Failure

4096 LUN Paths Fail at Once, 2048 of them are Primary Paths with Live I/O

DMP Minimum Queue

0.00

1000.00

2000.00

3000.00

4000.00

5000.00

6000.00

0:00

:00

0:00

:20

0:00

:40

0:01

:00

0:01

:20

0:01

:40

0:02

:00

0:02

:20

0:02

:40

3:00

:00

0:03

:20

0:03

:40

0:04

:00

0:04

:21

0:04

:40

0:05

:00

0:05

:20

0:05

:40

0:06

:00

0:06

:20

0:06

:40

0:07

:00

0:07

:20

0:07

:40

0:08

:00

0:08

:20

0:08

:40

0:09

:00

0:09

:20

0:09

:40

0:10

:00

Run time (hh:mm:ss)

KB

/sec

Proactive Path Management:Maximize Uptime in Scaled Up Config

• 4 ports per controller, each acting as primary port for 128 LUNs. 1024 LUNs total (128 * 8).

• Workload: multiple vxbench, single striped volume, ensuring that all 4096 primary paths carry traffic.

• Each Fabric contains 2 switches. Fabric A failed by disabling all Inter Switch Links

Tier1 - ALUA

1024 LUNs, 8 Paths per LUN

A B

Failure of Fabric A

DMP Fails over of 4096

LUN-paths in 40 seconds

23



Real-Time & Reliable Path Recovery

24

• DMP Path Recovery is in Kernel for reliability

• DMP regularly checks the state of failed and idle paths

– Successful SCSI inquiry will cause DMP to bring the path back online

– SCSI inquiry failure will cause DMP to disable the path

• Path checking is triggered:

– By fabric events: a port login event triggers path recovery

– Utilizes Path Probing mechanism: Every 5 minutes (tunable)




Nimble Recovery of PathsLIPP - Low Impact Path Probing

25

• Ensure real time accurate path state awareness

• Optimized utilization of system resources

• Utilizes Sub-path Failover Groups (SFG)

• LIPP groups paths into logical entities

• Health of few paths determines the health of the rest in the group

• Reduces CPU overhead & avoids flooding of SAN with probes

• Tuning LIPP

• Tunable (dmp_low_impact_path_probe) to turn “on/off” LIPP

• Ability to change number of probes sent to “idle” or “disabled” paths




DMP - Optimized Multi-Vendor Multi-Pathing Solution

• One interface for all OS, storage & architecture

– Enables multi-vendor environment & standardization

– Industry’s broadest hardware compatibility list

– Utilize DMP to enable more arrays

• Enterprise grade performance

– Best in class throughput

– Proactive path management

– Utilize faster failover & effective load balancing

• Bridges storage & server boundaries

– Enables informed decision making

– Utilize VOM & DMP for deep visibility

• Managed by Veritas Operations Manager

– Download VOM & utilize it for storage/path management


http://www.emc.com/

http://www.ibm.com.au/

http://www.huawei.com/

http://xiotech.com/index.php

Thank you!

Copyright © 2011 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

27

Ashish Yajnik - [email protected]

Chad Bersche – [email protected]


mailto:[email protected]

mailto:[email protected]

041011_DMP_SMB14_ver2

Documents

Transcript of 041011_DMP_SMB14_ver2