OPENVMS I/O AND STORAGE

7/30/2019 OPENVMS I/O AND STORAGE

1/44

2010 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

2011 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

OPENVMS I/O AND

STORAGETips and Best Practices for good performance

Rafiq Ahamed K

OpenVMS Engineering


2/44

Agenda

2

OpenVMS I/O Facts I/O Evolution on Integrity servers

What to expect from hardware

What do you know about multipathing

Notes on IOPERFORM

NUMA and I/O

Storage Tips and Tricks EVA Best Practices

OpenVMS Connectivity

Have a sneak into EVA PA

Q & A

9/19/2011


3/44

Operating System performance is largely dependent

on the underlying hardware

..So know your hardware capabilities

9/19/20113


4/44

Integrity server I/O

4

Architecture has evolved drastically for I/O Devices within Integrity,Performance & Scalability - doubling in each new hardware release

9/19/2011

PCIInterconnects

PCI (66MHz,0.5GB/sec)

PCI-X(133MHz,1GB/sec)PCI-X(266MHz, 2GB/sec)

PCIe Gen 1(2.5Gb/sec/Lane,250MB/sec)

PCIe Gen 2(5Gb/sec/Lane,500MB/sec)

Core I/O Ultra SCSI 160 Ultra SCSI 320 3G SAS (LSI Logic) 6G SAS (p410i)

SAN I/O 1G FC 2G FC 4G FC 8G FC

Disk Size 300GB 600GB 1.2TB 7.2TB

# I/O Device 3 6 8 12/16


5/44

Examples of latest speeds andfeeds of I/O on Integrity

platforms

5


6/44

Each BL890c/rx2800 supports eight SFF SAS HDD, up to 7.2TB capacity

Leadership in I/O and Storage on i2 architecture

High performance, reliable and scalable

9/19/20117

SAS provides apoint-to-point

connection to each

HDD @6G Speeds

Parallel SCSI with rx7640has a shared busUltra160 SCSI

Provides four p410RAID controllers

(one per blade) onBL890c i2,

One 410 RAID onrx2800

Configured as RAID 0/1or HBA mode [Future]

Stripe data withinand across multiple

p410 RAIDcontrollers

(OpenVMSShadowing)

Striping within provideshigh Performance.

Striping across controllersprovides no SPOF storage


7/44

Data shows the impact of p410i Caching and Striping

Core I/O on i2 servers

8

0

500

1000

1500

2000

2500

1 2 4 8 16 32 64 128 256

IOPS

Load

rx2800 i2 - SAS Logical Disk (Striping)

1 disk with Cache 2 disk with Cache 4 disk with Cache

0

100

200

300

400

500

600

700

1 2 4 8 16 32 64 128 256

IOPS

Load

rx2800 i2 - Core SAS Caching

W/O Cache Cache

9/19/2011

Use p410i Cache Batter Kit for faster response Stripe across multiple disks to maximize the utilization and

throughput


8/44

How is I/O Performance on Integrity Servers?

How they compare against my existing high end alpha servers?

After migrating to new platform what should I expect?

Why is i2 server I/O market differentiator?

Customer Concerns..

9/19/20119


9/44

Software capabilities;

multipathing

10


10/44

Multipathing 1(4)

11

Multipathing (MP) is a technique to manage multiple paths to storage

device through failover and failback mechanisms

It helps user to load balance across multiple paths to storage

By default multipath will be enabled on OpenVMS

OpenVMS MP supports ALUA (Asymmetric Logical Unit Access) [> V8.3]

OpenVMS MP supports FAILOVER and FAILBACK

OpenVMS MP load balances the storage devices across all the pathsavailable It spreads # of devices evenly across all the paths during boot time

At any point in time only single path can be active to a device

Users are recommended to use static load balancing techniques

9/19/2011


11/44

MP Connections Good and Bad

12

Alpha

HSVHSV

switch switch

IA64 Alpha

HSVHSV

switch switch

IA64

IA64

HSV

switch

IA64

HSV

switch switch

IA64

HSV

switch

Single Two Paths Four Paths

Shows single controllerconfigurations

9/19/2011


12/44

Multipathing 2(4)

13

Device discovery initiates the path discovery and forms MP set for each

device MC SYSMAN IO AUTO, SDA > SHOW DEVICE DGAxx shows MP Set

First path discovered is considered primary

The active path is called current

Automatic path selection algorithms optimized to support Active/Active arrays

Always active optimized (AO) paths are picked for I/O, if no alternative then active non-optimized (ANO) is picked [ how to fix this will discuss during EVA best practices]

With latest firmware's on storage, very rare you will get connected toANO

9/19/2011


13/44

VMS switches its path to a LUN when:

Multipathing 3(4)

14

An I/O error triggers mount verification (MV)

Device is not reachable on current path and another path works

MOUNT of a device with current path offline

Manual path switch via SET DEVICE/SWITCH/PATH=

Some local path becomes available when current path is MSCP

Path switch from MSCP to local triggered by the poller [not if manual sw]

Note: Any MV might trigger a path switchMV due to loss of cluster quorum

MV due to SCSI error flush

9/19/2011


14/44

Multipathing 4(4)

15

MPDEV_POLLER is light weight, will poll all paths for availability SET DEVICE device/POLL/NOPOLL

MV is not bad. It only indicates that OpenVMS validated your device Shadow devices can initiate and complete a MV ; Each shadow member operateindependently on a path switch

But, MV followed with path switch is indication of failover/failback The operator logs will indicate the details

SHOW DEV device/full will show details of path switch [time etc]

SDA> SHOW DEV device logs lot of diagnostic information in MPDEV structure

9/19/2011


15/44

SCSI Error Poller

9/19/201116

We have seen customers reporting huge traffic of Unit Attention(UA) in

SAN resulting in Cluster hangs, slow disk operations, high mountverifications etc!

These UA are initiated due to any changes in SAN like FirmwareUpgrades, Bus Reset etc

SCSI_ERROR_POLL is poller responsible for clearing the latchederrors (like SCSI UA) on all the fibre and SCSI devices, which canotherwise cause confusion in SAN

By default the poller will be enabled SYSGEN>SHOW

SCSI_ERROR_POLL


16/44

After upgrading my SAN components, I see large number of mountverifications, does that indicate problem?

Does multipathing do load balance? Or policies?

I see too many mount verification messages in operator log, will it

impact the volume performance (especially latency)

How do I know if my paths are well balanced or not?

How do I know my current path is active optimized or not?

Does multipathing support Active/Active Arrays, ALUA, third partystorage, SAS device, SSD device

OpenVMS Multipathing

Customer/Field Concerns..

9/19/201117


17/44

Did you know QIO is one of the most heavily usedinterface in OpenVMS. We want to put it on diet.What should we do?

1. Optimize QIO

2. Replace QIO3. Provide alternative

9/19/201118


18/44

IOPERFORM/FastIO

19

Fast I/O is a performance-enhanced alternative to performing QIOs

Substantially reduces the setup time for an I/O request

Fast I/O uses buffer objects to eliminate the I/O overhead ofmanipulating I/O buffers locked memory doubly mapped

Performed using the buffer objects and the following system services:

sys$io_setup ,sys$io_perform, sys$io_cleanup

sys$create_bufobj (jacket) /sys$create_bufobj_64

$ dir sys$examples:io_perform.c

System management considerations:

SYSGEN parameter MAXBOBMEM limits memory usage (defaults to 100)

VMS$BUFFER_OBJECT_USER identifier is required for process

Creating buffer objects once and using them for the time of application is faster

9/19/2011


19/44

Impact of IOPERFORM/FASTIO

20

Resource usage is reduced by20-70% depending on load andsystem Small size random workloads doublesthe throughput with increased loads

Larger size sequential workloadsoperate same

0

50100

150200

250300350400

450

1 2 3 4 5 6 7Threads

I/O Data Rate (MB/sec)

128K_READ

128K_READ_QIO

128K_WRITE

128K_WRITE_QIO

05000

100001500020000

2500030000

350004000045000

1 2 4 8 16 32 64

Threads

Throughput (IOPS)

8K_READ

8K_READ_QIO

9/19/2011


20/44

What you should know?

NUMA/RAD Impact

22 9/19/2011

DGA100

P1

BL890c i2 (architecturally 4 Blades conjoined)


21/44

NUMA/RAD Impact

23

In RAD based system, each RAD is made of CPU, Memory and I/O

devices

The accessibility of I/O devices from local to remote domains results inaccessing remote memory and remote interrupt latency

3000

3100

3200

3300

3400

3500

3600

3700

0 1 2 3 4 5 6 7 8

IO

Rate

RAD #

Impact of RAD on I/O Device

Opt/sec9/19/2011

10-15 % overheadOptimized Performance


22/44

RAD Guidelines for I/O

9/19/201124

Keep I/O Devices close to process which is heavily accessing it

Make use of FASTPATHING efficiently Make sure to FASTPATH the Fibre Devices close to the process which is initiating the I/O

The overhead involved in handling the remote I/O can impact the throughput [chart]

FASTPATH algorithms assign the CPU on round robin basis

Statically load balance the devices across multiple RADs

Make use to SET PROC/AFFINITY to bind processes with high I/O

Use SET DEVICE device/PREFERRED_CPUS


23/44

STORAGE BEST PRACTICES

25


24/44

EVA4400 EVA6400 EVA8400 P6300 P6500

Model HSV300 HSV400 HSV450 HSV340 HSV360

Memory /controller pair 4GBytes 8GBytes 14/22GBytes 4GBytes 8GBytes

Host Ports / controllerpair

4 FC

20 w switches

8 FC 8 FC 8 FC, 0 GbE

4 FC, 8 1GbE

4 FC, 4 10GbE

8 FC, 0 GbE

4 FC, 8 1GbE

4 FC, 4 10GbE

Host Port speed 4Gb/s FC4Gb/s FC 4Gb/s FC 8Gb/s FC

1Gb/s iSCSI10Gb/s iSCSI/FCoE

8Gb/s FC1Gb/s iSCSI

10Gb/siSCSI/FCoE

Device Ports, # 4 8 12 8 16

Device Ports, Speed 4Gb/s FC 4Gb/s FC 4Gb/s FC 6Gb/s 6Gb/s

# 3-1/2 drives 96 216 324 120 240

# 2-1/2 drives 0 0 0 250 500Max. Vdisk 1024 2048 2048 1024 2048

I/O Read Bandwidth 780 MB/s 1,250 MB/s 1,545 MB/s 1,700 MB/s 1,700 MB/s

I/O Write Bandwidth 590 MB/s 510 MB/s 515 MB/s 600 MB/s 780 MB/s

Random Read I/O 26,000 IOPs 54,000 IOPs 54,000 IOPs 45,000 IOPs 55,000 IOPs

Speeds and Feeds

EVA Differences

9/19/201127


25/44

After upgrading the OS or applying the patch, I/O response has

become slowerWe see 5-6 millisecond delay in completion of I/O compared to yesterday

After moving to new blade in same SAN environment, our CRTL FSYNCis running slow

After upgrading, we see additional CPUs ticks for copy, delete andrename

Our database is suddenly responding slow

Some nodes in cluster see high I/O latency after mid-night

Customer wants to know if this storage is enough for next 5 years

Customer is migrating from older version of EVA to newer version, canyou advise

General I/O issues reported

9/19/201128


26/44

Maximum number of storage performance issuesreported are due to mis-configuration of SANcomponents

9/19/201129


27/44

Best Practices..1(6)

30

Number of disks influences performance - YesFill the EVA with as many disk drives as possible.

Tests have shown linear growth in throughput (small random)

Number of disk groups influences performance No

In mixed load environments, it would be ok to have random vs.sequential application disk groups

Vraid level influences performance YesVraid1 best performance over the widest range of workloads; however,

Vraid5 is better for some sequential-write workloadsVraid0 provides the best random write workload performance , but noprotection Use for non-critical storage needs

9/19/2011


28/44


31

Fibre channel disk speed - Yes10K vs 15K Speed 15K rpm disks provide highest performance

Large-block sequential I/O, speed doesnt matter, but capacity matters

Small-block random I/O, 30-40% gains in request rates is seen

Best price-performance

For the equivalent cost of using 15k rpm disks, consider using more 10k rpmdisks.

Combine disks with different performance characteristics inthe same disk groupDo not create separate disk groups to enhance performance

9/19/2011


29/44


32

Mixing disk capacities Yes

The EVA stripes LUN capacity across all the disks in a disk group. The larger disk willhave more LUN capacity leading into imbalanced density

No control over the demand to the larger disks.

Use disks with equal capacity in a disk group.

Read cache management influences performance and always ENABLE

LUN count Yes, No Good to have few LUNs per controller

Depends on Host Requests and Queue Depths

Monitor OpenVMS Queue depth

Transfer size Yes Impacts SEQUENTIAL Workloads Tune the write transfer size to be a multiple of 8K and no greater than 128K.

OpenVMS Max Transfer Size is 128K for Disk and 64K for Tapes!DEVICE_MAX_IO_SIZE

9/19/2011


30/44


33

SSD performance - Yes, Yes, YesSSDs are about 12 times better

OpenVMS performed 10x times better than FC [next slide details]

Workloads like transaction processing, data mining, and databasessuch as Oracle are ideal

Spread SSDs evenly across all available back end loopsSSDs and HDDs may be mixed in the same drive enclosure

Monitor your application and EVA, accordingly can assign SSD orHDD to individual Controllers, or enable write through mode for SSDs

can help [Experiment!!]Customers use SSD drives to keep the critical path data, where theresponse time is un-compromised

9/19/2011


31/44

10xFaster

SSD Drive Through EVA

Mixed Load, 8 Disks SSD/FC DG on EVA4400 Smaller IOs(4K/8K) showed 9-10x times sustained increased in IOPS and MB/sec

with increase in load for SSD carved LUNs compared to FC

With 10 times faster response time, SSD carved LUN was able to deliver 10 timemore performances and bandwidth for smaller size IOs

OpenVMS 8.4 Performance Results

0

2000

4000

6000

8000

10000

12000

14000

1 2 4 8 16 32 64 128 256

IOPS

Threads

4K Mixed QIO - FC vs SSD

QIO_FC QIO_SSD

0

50

100

150

200

250

0 50 100 150 200 250 300

ResTime(msec)

Threads

4K Mixed QIO FC vs SSD

FC SSD

10x Faster

OpenVMS V8.4

9/19/201134


32/44


35

Controller balancing Yes, Yes, Yes

Maximize the utilizationActive/Active present LUNs simultaneously through both controllers, Ownership is only one

Manually load balance LUN ownership to both controllers (use EVAPerf) either through usingCV EVA or using OpenVMS SET DEVICE/SWITCH/PATH='PATH_NAME' 'DEV_NAME'command

Preferred Path During the initial boot of the EVA the preferred path parameter is read and

determines the managing controller [see below figure for options]Verify LUN ownership is reassigned after a failed controller has been repaired

Balance the workload as evenly as possible across all the Host Ports

HSVxxxHSVxxx

DGA99

DGA99 does I/O on these ports

DGA99 answers Inquiry on these ports

Command View EVA Preferred Path Settings9/19/2011


33/44

Controller Load Imbalance & No Equal Port Load Distribution

Customer Scenario

36 9/19/2011


34/44


37

Ensure there are no HW issues Specially Battery failure (Cache Battery

failure causes to change to Write-Through mode, hence Writeperformance becomes an issue), Device loop failure etc. Drive reportingtimeouts

Deploy the array only in supported configurations

Stay latest on EVA firmware!! BC and CA have different best practices and beyond the scope of this

discussion

9/19/2011


35/44

Some Points to Remember

38

Large latencies may be quite natural in some contexts, such as an array

processing large I/O requests

9/19/2011

Array processor utilizations tend to be high under intense, small-block,transaction oriented workloads but low under intense large-blockworkloads


36/44

8K workloadspushing close to

EVA MaxThroughputs!

P6500 36G RAID 5 Volume 4G FC Infrastructure

OpenVMS IO Data

40

0

51015202530354045

138 286 367 383 404 411 412 412

resptime(msec)

MB/sec

Sequential Read (MB/sec)

0

1

2

3

4

5

6

6562 12935 22287 31888 38136 40052 43452 45504 46202

resptime(msec)

IOPS

Random Read (IO/sec)

Higher throughputs can beobtained with smaller blocks.

Usually smaller blocks need alot of processing power

Higher bandwidth can be obtainedwith larger blocks.Larger blocks can drain theinterconnects faster due large datatransfers.

9/19/2011

128K IOs pushing4G FC line speeds!


37/44

STORAGE PERFORMANCEANALYSIS TOOLS &

REFERENCES

9/19/201141


38/44

Storage Performance Tools

9/19/201142

OpenVMS Host Utilities

T4, TLViz Disk, FCP, VEVAMON (older EVAs) SDA > FC [for fibre devices], PKR/PKC [for SAS devices]

SYS$ETC: SCSI_INFO.EXE, SCSI_MODE.EXE, FIBRE_SCAN.EXE

Many more..

EVAPerf Command Line EVA Performance Data Collector EVA Performance Advisor [ Year End Release]

XP Performance Advisor

Storage Essential Performance Pack


39/44

Slated to release soon Can participate in early adaptor program

Salient aspects of HP P6000 Performance Advisor

Integrated with Command View 10.0 in single pane of glass

User centric design compliance

Features:Dashboard

Threshold monitoring & notificationKey metric

Chart

Report

Quick setup Events

Database

9/19/201143


40/44

HP StorageWorks Sizing Tool

Sizing EVA

44 9/19/2011


41/44

References EVA Performance

45

HP StorageWorks Enterprise Virtual Array A tactical approach to

performance problem diagnosis. HP Document Library

9/19/2011
http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/erl.aspx?cc=us&lc=enhttp://h20195.www2.hp.com/v2/erl.aspx?cc=us&lc=enhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdfhttp://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0994ENW.pdf


42/44

Business Manager (Rohini Madhavan) [email protected]

Office of Customer Programs [email protected]

Questions/Comments

9/19/201146
mailto:[email protected]?subject=From%202011%20BC%20Storage%20Updatesmailto:[email protected]?subject=From%202011%20BC%20Storage%20Updatesmailto:[email protected]?subject=From%202011%20BC%20Storage%20Updatesmailto:[email protected]?subject=From%202011%20BC%20Storage%20Updates


43/44

THANK YOU


44/44

EVAModel Controller Model Firmware

EVA3000 HSV1001.XXX, 2.XXX or 3.XXX (Latest 3110)

EVA5000 HSV110

EVA3000 HSV1014.XXX (Latest 4100)

EVA5000 HSV111

EVA4000 HSV200 or HSV200-A

5.XXX or 6.XXX (Latest 6220)


EVA4100 HSV200-B

EVA6100 HSV200-B


EVA8100 HSV210-BEVA4400 HSV300 09XXXXXX or 10000000

EVA6400 HSV400095XXXXX, or 10000000

EVA8400 HSV450

P6300 HSV34010000090

P6500 HSV360

EVA Models - Reference

OPENVMS I/O AND STORAGE

Documents

Transcript of OPENVMS I/O AND STORAGE