Using the AIX Logical Volume Manager to perform SAN...

28
Using the AIX Logical Volume Manager to perform SAN storage migrations Migrating AIX systems to new SAN storage subsystems with the Logical Volume Manager Skill Level: Introductory Chris Gibson AIX Specialist Southern Cross Computer Systems 13 Jul 2010 This article provides examples of how to migrate AIX® systems from old to new SAN storage subsystems. It covers both dedicated and virtual I/O systems. We will examine IBM and non-IBM storage migrations, using the AIX Logical Volume Manager (LVM). Introduction Connecting with Chris Chris is a popular author and blogger on AIX. Browse Chris's other articles and check out his blog and My developerWorks profile If you work with AIX long enough, a time will come when you will need to migrate an existing AIX system from an old SAN storage device to a shiny, new SAN storage subsystem. The lease may be up on the old device, or it may simply need to be replaced by newer technology. This storage device could be an IBM product or that of another vendor. Either way, you will be faced with the decision of how best to migrate your AIX system to the newer device. Of course, you will also want to minimize the impact to your running systems as a result of the migration. In this article, I will share some examples of how I migrated AIX systems from old to Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks © Copyright IBM Corporation 2010. All rights reserved. Page 1 of 28

Transcript of Using the AIX Logical Volume Manager to perform SAN...

Page 1: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Using the AIX Logical Volume Manager to performSAN storage migrationsMigrating AIX systems to new SAN storage subsystems withthe Logical Volume Manager

Skill Level: Introductory

Chris GibsonAIX SpecialistSouthern Cross Computer Systems

13 Jul 2010

This article provides examples of how to migrate AIX® systems from old to new SANstorage subsystems. It covers both dedicated and virtual I/O systems. We willexamine IBM and non-IBM storage migrations, using the AIX Logical VolumeManager (LVM).

Introduction

Connecting with ChrisChris is a popular author and blogger on AIX. Browse Chris's otherarticles and check out his blog and My developerWorks profile

If you work with AIX long enough, a time will come when you will need to migrate anexisting AIX system from an old SAN storage device to a shiny, new SAN storagesubsystem. The lease may be up on the old device, or it may simply need to bereplaced by newer technology. This storage device could be an IBM product or thatof another vendor. Either way, you will be faced with the decision of how best tomigrate your AIX system to the newer device. Of course, you will also want tominimize the impact to your running systems as a result of the migration.

In this article, I will share some examples of how I migrated AIX systems from old to

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 1 of 28

Page 2: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

new SAN storage subsystems. I will cover both dedicated and virtual I/O (VIO)systems. To demonstrate the approach, the examples will include both IBM andnon-IBM storage.

Preparation and planning

Any migration of this type requires careful preparation and planning. I’m going toassume that you or your SAN storage administrator have already cabled, configuredand connected your new SAN storage device to your existing SAN. Also, I'll assumethat your AIX systems already have SAN connectivity to your existing SAN fabric.

Before migrating, verify support with your storage vendor. Review the vendor'sstorage support matrix. These matrices are usually listed on the storage vendor'swebsite (or are available from the vendor's representative). They will highlight whichsystems are supported with their storage device. For example, for IBM Enterpriseclass storage devices (such as the DS8300), they provide a support matrix whichcan be used as a reference during your planning. The DS8300 Interoperability Matrix(see the Resources section) identifies important support and compatibilityconsiderations with respect to various host systems and adapters. They cover awide spectrum of support checks such as supported operating systems, patchlevels, required Fibre Channel (FC) adapter firmware, supported SAN switchtypes/firmware, and much more.

One of the most important pieces of the puzzle is the required multi-path I/O devicedrivers and recommended FC adapter firmware (Microcode) levels. I have seen allsorts of problems when these components are not checked prior to integrating a newstorage device into an existing SAN and AIX environment. Most vendors providetools to help with the planning and verification, for example, IBM provides the FixLevel Recommendation Tools website, as well the IBM HBA support site (see theResources section). If a vendor does not provide online tools to assist in theplanning, then I recommend you ask them for help directly. After all, it is in theirinterests to help you make their product work in your environment!

Another important (and surprisingly sometimes overlooked) stage in the planningand preparation is the design phase. Take the time to design how your new SANstorage device is going to fit into your existing SAN. Ask questions that will help thedesign process, such as:

• Can/should this device connect to the existing SAN or is this a good timeto provision a new SAN Fabric?

• How will the AIX systems connect to the new storage device?

• How will the AIX operating system and data be migrated from the old tothe new disk?

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 2 of 28

Page 3: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

If it helps (and it usually does), draw pictures to help demonstrate answers to thesequestions. It will also help others to visualize and understand what you are trying toachieve. Start with a diagram that encapsulates your current state, then another thatdescribes how the new device will fit, followed by the proposed migration process orprocesses. Finally, at the end, state how the environment will look once the olddevice is no longer needed and all the data has been migrated from it.

Will you be migrating from a dedicated I/O environment to a virtual one? If you are,then I recommend you review the latest IBM Redbook on migrating from physical tovirtual storage (see Resources). This publication guides you through the migrationprocess and offers several methods for migrating.

If you are migrating a dedicated I/O environment to another dedicated I/Oconfiguration, then consider the software requirements on each LPAR. For example,for IBM DS8300 storage you need to ensure that you have the appropriate SDDPCMMPIO device drivers (e.g. devices.sddpcm.53.rte) and the DS8300 Host Attachmentkit (e.g. devices.fcp.disk.ibm.mpio.rte) prior to the migration (or immediately after),along with the required FC adapter Microcode (firmware).

If you already have a virtual I/O environment (running a virtual I/O server, or VIOS,for disk traffic), then consider the requirements for the VIOS. If you are migratingfrom IBM to IBM storage, then it is likely that you will simply need to update theMPIO code, FC adapter firmware and supporting device drivers. However, if you aremigrating from Vendor A to Vendor B, you may need a different approach. Thedesign process should shake out these considerations beforehand.

For example, if you are migrating (VIOS presented disk) from IBM DS to NetAppstorage, then it is likely that you will need to consider provisioning new VIO serversfor the NetApp disk. Rather than mix two vendors' MPIO code on the same VIOS, itmay be simpler to manage if each storage type has its own VIOS. I recommend thisapproach and my examples will cover how I chose to deploy this type ofconfiguration.

Unfortunately, I won't be discussing N-Port ID Virtualization (NPIV) in this article.NPIV adds another dimension to storage virtualization on the Power platform. NPIVwith a VIOS, utilizes virtual FC devices to present disk (and tape) natively to theclient LPARs. See the resources section for more information.

I'm also assuming that you do not have an IBM SAN Volume Controller (SVC) inyour environment. If you do have an SVC and all your AIX systems are alreadybehind it, then I suggest you take advantage of this product's amazing capabilities. Itcan migrate storage transparently, from one storage device to another, without thehost system ever being aware of the move.

If you don’t have an SVC, and you are considering implementing one, I say proceedwithout delay! With an SVC in your environment, storage migrations (for the AIX

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 3 of 28

Page 4: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

administrator) become a thing of the past. And that’s just one of the manyadvantages to using this wonderful device. See the Resources section for moreinformation on the IBM SVC.

Without an SVC, AIX storage migrations will most likely involve the use of the AIXLogical Volume Manager (LVM). This is what I will cover in the examples that follow.

IBM SAN Storage Migration with AIX and dedicated I/O

Several years ago I needed to migrate a large number of AIX hosts from an old IBMESS (F20) to a shiny, new IBM DS8300 storage subsystem. The AIX hosts were allusing dedicated FC adapters (HBAs). Each host had at least two FC adaptersconnected to our SAN. The following diagram shows the high-level view of the SANstorage and AIX LPAR connectivity to the existing SAN Fabric:

Figure 1. IBM ESS to DS8300 storage migration - Dedicated I/O - Current state

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 4 of 28

Page 5: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 5 of 28

Page 6: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

The MPIO code for every AIX system connected to this type of storage had to beupdated to the latest SDD device driver and FC adapter Microcode. For example,our design document stated the following:

Ensure that the following software is installed on all AIX systems, at these levels:

AIX

5200-05

IY62165 Abstract: Target device rejects writes - AIX Host to McData switch.

Fileset devices.pci.df1000f7.com:5.2.0.50 is applied on the system.

IY62116 Abstract: after EEH error, attached hdisks failed

Fileset devices.pci.df1000f7.com:5.2.0.50 is applied on the system.

ibm2105.rte

32.6.100.24 COMMITTED IBM 2105 Disk Device

devices.sdd.52.rte

1.6.0.2 COMMITTED IBM Subsystem Device Driver for AIX V52

devices.fcp.disk.ibm.rte

1.0.0.0 COMMITTED IBM FCP Disk Device

Ensure that the latest Microcode for Fibre Channel adapters has been applied.Use the lscfg command to determine the Microcode level of the FC adapter.

$ lscfg -vpl fcs0 | grep Z9

Updating all of our 100+ AIX systems, before migrating, was a sizeable task.However, once completed we were able to move to the new storage device withoutan issue.

The approach to migrating data from the old disk to the new disk was to employ theAIX Logical Volume Manager. There were two LVM utilities at the core of our datamigration strategy (mirrorvg and migratepv). Both commands can copy/move databetween disks while the system is running. Due to the very I/O intensive nature ofthese commands, they could impose a slight performance impact to I/O on thesystem. Therefore, it was determined that we would not perform a data migrationwhen the system was running peak (disk I/O) load. We would schedule these tasksduring relatively quiet periods.

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 6 of 28

Page 7: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

The arrow (from hdisk0 to hdisk11) in Figure 2 represents the LVM mirroring (andmigratepv) process for data migration.

The mirrorvg command would be used to migrate the operating system (rootvg) fromthe old ESS to the DS8300. However, for the application/data volume groups, wechose to use the migratepv command. This would give us some level of control overhow much additional I/O activity we could unleash on the running system. Some ofthe migrating systems were in production and we did not want to flood the I/Osubsystem and cause unnecessary performance issues.

Obviously before we could start, our storage team had to first attach and configurethe new DS8300 into our existing SAN. Once this was completed, we worked withthe storage team to determine what type and how many LUNs we would require foreach of the AIX systems that were migrating to this new device.

Our planning also captured each LPAR's hostname, the existing AIX hdisk names,the existing SDD (vpath) configuration, the World Wide Port Name (WWPN) for eachFC adapter, the current FC adapter name (e.g. fcs0 and fcs1), the current LUNconfiguration and the proposed new LUN configuration.

With the storage carved up, we were able to assign the new LUNs to the existingAIX hosts and perform the migration. Figure 2 below shows the DS8300 isconnected to our SAN and a LUN from it has been allocated to an existing AIXLPAR. The LUN appears to AIX as a hdisk device (hdisk11). This disk has beenallocated to an existing volume group (rootvg).

Figure 2. IBM ESS to DS8300 storage migration - Dedicated I/O - LVM Mirror -Migration state

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 7 of 28

Page 8: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 8 of 28

Page 9: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Prior to migrating, we performed a backup of the system (including a mksysb). Themigration could execute while the applications were running on the system.However, at some point after the migration, a reboot would be required. The systemboots from SAN disk. We will be migrating the system to a new SAN disk for theoperating system. To ensure that the boot disk has migrated successfully we mustmake sure that we can boot the system on the new disk. The downside is that if thisfails, we would need to restore the system from a mksysb backup.

In hindsight (5 years later!), I could have suggested we use alt_disk_install (on AIX5.2) instead of mirrorvg. The alt_disk_install command (now replaced by thealt_disk_copy command in AIX 5.3) can clone an existing rootvg onto another disk.Using this method would have provided a more efficient back out path. Fortunately,we never had to initiate a back out during any storage migration.

Before making any changes to the existing system, we documented the systemconfiguration. The current disk, LVM and SDD (vpath) configuration was alsocaptured. Historically, SDD configured vpath devices to provide MPIO to hdiskdevices. To keep my examples simple and more generic, I will refer to hdisk devicesrather than the vpath devices. You would typically work with vpath devices in a pureSDD environment. Storage vendors will present MPIO disk differently. Please keepthis is in mind when dealing with your storage devices and MPIO code.

# lsdev -Cc disk# lsvg –l rootvg# lsvg –l datavg# lsvg -p rootvg# lsvg -p datavg# lsvpcfg

When the new LUN had been assigned to the AIX host, the cfgmgr command wasexecuted to pickup the new DS8300 disk. I confirmed that the new disk wasdiscovered and the paths had been configured correctly.

; A disk type of 2107 confirms it is DS8300 storage.# lsdev –Cc disk | grep 2107# lspv# lsvpcfg

At this point, I could add the new DS8300 LUN (hdisk11) to rootvg with the extendvgcommand. Then, I used the mirrorvg command to create an exact copy of the dataon the new disk. Using AIX LVM commands, this process was straightforward.

# extendvg –f rootvg hdisk11# mirrorvg –S rootvg hdisk11

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 9 of 28

Page 10: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

The mirroring process can take some time. As a precautionary measure, I madesure that there was a new (secondary) dump logical volume (LV) on the new disk,that a new boot image was created, and the boot list contained both the old and thenew hdisks. If I needed to restart the system at this point in the migration, I could beassured that I could boot from either disk.

# mklv -y hd71 -t dump rootvg 8 hdisk11# lspv -l hdisk11# sysdumpdev -s /dev/hd71

# bosboot -a -d /dev/hdisk11# bosboot -a -d /dev/hdisk0

# bootlist –m normal -o# bootlist -m normal hdisk0 hdisk11# ipl_varyon –i

Once the mirrors had synced (i.e. lsvg –l rootvg did not show any stale partitions), Iremoved the old hdisk from rootvg. First I had to unmirror rootvg (unmirrorvg) fromthe older disk. I also made sure that any active dump device on this disk wasremoved. I temporarily changed the primary dump device to /dev/sysdumpnull. ThenI removed the disk from the volume group and the AIX Object Data Manager (ODM),with the rmdev command.

# ps –ef | grep sync# lsvg –l rootvg | grep –i stale# unmirrovg rootvg hdisk0# sysdumpdev -Pp /dev/sysdumpnull# rmlv hd7; no output from the lspv command, means no LV data on disk.# lspv –l hdisk0# reducevg rootvg hdisk0# chpv –c hdisk0# rmdev –dl hdisk0

After making these changes, it was important that I re-create the boot image, checkthe boot list contained only the new hdisk and that the primary dump device was setcorrectly.

# bosboot -a -d /dev/hdisk11# sysdumpdev -Pp /dev/hd71# bootlist –m normal -o# bootlist -m normal hdisk11# ipl_varyon -i

With the operating system now residing on the new disk, I focused on migrating thedata volume groups. The storage team assigned new data LUNs to the host andprovided me with a list of the LUN ids. First I needed to identify the DS8300 diskswhich would be used for the data migration. Both the lspv and lsdev commands can

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 10 of 28

Page 11: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

display information relating to hdisks on an AIX system.

; hdisk NOT ASSIGNED TO A VG# lspvhdisk12 none Nonehdisk13 none None; A disk type of 2107 is displayed for DS8300 disks on AIX.# lsdev –Cc disk | grep 2107

With the correct disks identified (hdisk12 and 13), I could add these disks (withextendvg) to the existing data volume group (datavg).

# extendvg datavg hdisk12 hdisk13

The data migration from the old disk to the new disk would be accomplished by theLVM command, migratepv. I had to select the source and destination disks for themigratepv operation. For example, the following commands would migrate data fromhdisk2 to hdisk12 and hdisk3 to hdisk13. At the end of the migration, both hdisk2and hdisk3 would be empty and all of their data would now reside on hdisk12 andhdisk13 respectively.

# migratepv hdisk2 hdisk12# migratepv hdisk3 hdisk13

To confirm that all the data (logical volumes) for each of the old hdisks had beenmigrated, I ran the lspv command to list the contents of each disk. The command didnot return any output, confirming that the disks were indeed empty. It was now safefor me to remove the old disks from the volume group and remove them from theODM.

# lspv –l hdisk2# lspv –l hdisk3# reducevg datavg hdisk2 hdisk3# rmdev –dl hdisk2# rmdev –dl hdisk3

With the migration complete, I could now ask the storage team to reclaim the oldLUNs on the F20.

To ensure that the boot disk and boot list had been configured correctly, I wouldreboot the system to verify. This would also ensure that the newly migrated disks,volume groups, logical volumes and filesystems would continue to function asexpected after the migration. The desired end state for our dedicated I/O system hadbeen achieved. The old ESS storage could now be decommissioned and removedfrom the data centre.

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 11 of 28

Page 12: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Figure 3. IBM ESS to DS8300 storage migration - Dedicated I/O - End state

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 12 of 28

Page 13: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 13 of 28

Page 14: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Mixed vendor SAN storage migration with AIX and virtual I/O

In the next scenario, I’ll discuss how I performed a similar storage migration. Thedifference with this example is that I had to migrate from IBM to NetApp storageusing the virtual I/O server (VIOS).

Let’s review the current state (the state of the environment prior to migrating to thenew storage). Figure 4 below shows that there is an AIX LPAR, connected to a pairof VIOS. Both are connected to IBM DS8300 storage. The VIOS pair (hvio1 andhvio2) are serving the DS8300 LUNs as virtual SCSI (VSCSI) disk to the clientLPAR. The LPAR has a single volume group, rootvg for the operating system, on asingle disk. Other disks and volume groups also exist for application data but are notdepicted for simplicity.

The NetApp storage device has been attached to our existing SAN fabric. However,none of the VIOS or AIX systems are accessing it at this time.

Figure 4. DS8300 to NetApp storage migration - DS8300 VIOS - Current state

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 14 of 28

Page 15: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 15 of 28

Page 16: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Our procedure document outlined how we would migrate an existing AIX LPAR,which currently uses hvio1 and hvio2 for all disk (DS8300) traffic. The LPAR and theVIOS reside on a Power6 570 (570-1). The new NetApp VIOS (hvio11 and 12) alsoreside on the same managed system and will be used for all virtual I/O traffic to/fromthe NetApp storage device. These details are captured during our planning process:

Client LPAR: lpar1

Managed system: 570-1

DS8300 VIOS pair (source): hvio1 and hvio2

NetApp VIOS pair (target): hvio11 and hvio12

The current VIOS and LPAR FC adapter, DS8300 disk, virtual adapter and virtualdisk configuration is also captured and used in the planning for the new diskassignments. For example, the following extract from our planning spreadsheetshows each LPAR, the current DS8300 disks assigned, the current VIOS, theexisting vhost/vscsi relationship, the new NetApp LUNs required, the new VIOS andthe new vhost to vscsi mapping for each LPAR. For examples, refer to Figures 5, 6and 7.

Figure 5.

Figure 6.

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 16 of 28

Page 17: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Figure 7.

Before we could migrate to the NetApp storage array, our Storage administrator firstconfigured the LUNs that we require and prepared to present them to the NetAppVIOS, hvio11 and hvio12. We have provided them with the following information (ata minimum) for the allocation to take place:

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 17 of 28

Page 18: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

• Client LPAR name : lpar1 (LPAR on 570-1). SAN boot.

• Source DS8300 VIOS pair names: hvio1 and hvio2.

• Target NetApp VIOS pair names: hvio11 and hvio12.

• WWPNs for all FC adapters on all VIOS involved in the migration.

• Quantity, size and purpose of the LUNs required, e.g. 1 x 50GB Boot LUN- rootvg, 1 x 100GB Application data - datavg.

We capture the current disk and volume group configuration with several LVMcommands.

# lsdev -Cc disk# lsvg# lsvg –o | lsvg –l# lsvg –o | lsvg -il# lspv# lsvg | lsvg –pi

Of course, we backup our LPAR before we make any changes to it, just in case.This includes performing a mksysb and savevg backup, followed by a file levelbackup with our corporate backup tool.

Prior to starting the migration, we deployed two new VIOS (hvio11 and 12),specifically for use with NetApp. During our design phase, we determined that itwould be a good idea to deploy the latest version of VIO server, version 2.1, for thebuild of the NetApp VIOS. The NetApp MPIO Host Attachment software is installedon both NetApp VIOS. Both VIOS will SAN boot from NetApp storage. You can referto Figure 8.

Figure 8. DS8300 to NetApp storage migration - Introduction of NetAppstorage and VIOS

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 18 of 28

Page 19: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

The storage team allocated a NetApp LUN (LUNy) to hvio11 and hvio12. This diskwould be used by the client LPAR for rootvg (boot disk).

We also dynamically add (with DLPAR) two new VSCSI adapters to the client LPAR.

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 19 of 28

Page 20: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

That is, vscsi2 and vscsi3, where vscsi2 is connected to hvio11 and vscsi3 isconnected hvio12. Again, using DLPAR, we assigned a new virtual SCSI serveradapter (vhost) to hvio11 and hvio12. I also ensure that we update the VIOS andLPAR partition profile (on the HMC) with these newly created virtual adapters.

We then mapped the LUN (as Virtual Target Device or VTD) to the client LPAR oneach VIOS and presented it to the LPAR. The disk (hdisk11) appears as a virtualSCSI disk on client. This new hdisk has been included in the existing root volumegroup (rootvg, also shown in Figure 8).

Once the LUN has been assigned to the NetApp VIOS, we perform our standardVIOS disk mapping to the client LPAR, that is, with mkvdev and lsmap to create andverify the disk has been assigned to the correct LPAR.

On the client LPAR, we ran cfgmgr to discover the new hdisk. We verify that the newdisk and paths are available. There should be two paths to the disk, one via vscsi2(hvio11) and one via vscsi3 (hvio12).

# lsdev –Cc disk# lspv# lspath –l hdisk11Enabled hdisk11 vscsi2Enabled hdisk11 vscsi3

To migrate the data from the old to the new disk, the method is similar to theprevious example. Except this time we will use mirrorvg (instead of migratepv) for alldata migrations (including the data volume groups). These systems are eitherdevelopment or test systems. While we do not want to impact performance, some ofthese systems are relatively idle or have very low user numbers, so we can addadditional I/O activity without too much of a performance concern. Using mirrorvgalso simplifies the migration as we do not have to run a migratepv command againsteach of the source/target disks.

First, we add the new disk (hdisk11) to rootvg. Then we mirror the volume group withthe new hdisk. This mirroring process takes place in the background. You can referto Figure 9.

# extendvg rootvg hdisk11# mirrorvg –S rootvg hdisk11

Figure 9. Migration state - LVM mirror

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 20 of 28

Page 21: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

As a pre-cautionary measure, we verify that rootvg is now mirrored, include the newdisk in the boot list and create a new boot image for the mirrored volume group. Justas before, if I need to reboot the LPAR at this point, I can rest assured that I can

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 21 of 28

Page 22: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

boot from either disk.

# bosboot -a -d /dev/hdisk11# bosboot –a –d /dev/hdisk0# bootlist -m normal hdisk0 hdisk11# bootlist –m normal –o# ipl_varyon -i

With rootvg mirrored successfully, we can now unmirror again, remove the old disk(hdisk0) from the volume group and remove it from the ODM. Again, we also need tocheck the boot list and recreate the boot image for this now non-mirrored volumegroup.

# ps –ef | grep sync# lsvg –l rootvg# unmirrorvg rootvg hdisk0# chpv –c hdisk0# lspv –l hdisk0# lsvg –l rootvg# reducevg rootvg hdisk0# lsvg –p rootvg# rmdev -dl hdisk0# bosboot -a -d /dev/hdisk11# bootlist –m normal hdisk11# bootlist –m normal -o

With our AIX OS now residing on the new hdisk (the NetApp LUN), we can nowmigrate the data volume groups. First, we identify the NetApp LUNs which will beused for the data volume group migration. The data LUNs had already beenassigned to the VIOS and mapped and configured on the client LPAR.

Again, we use extendvg to add the NetApp disks to the data volume group (datavg).

# extendvg datavg hdisk12# lsvg –p datavg# lspv hdisk12

We mirror the volume group from the DS8300 disk (hdisk2) to the new NetApp disk(hdisk12). The mirroring process is set to run in the background with the –S flag. Toensure the volume group and logical volumes are synced, before proceeding, wecheck that there are no stale physical partitions (PPs) and that there are no LVMsync processes still running in the background.

# mirrorvg –S datavg hdisk12# ps –ef | grep lresync# lsvg –l datavgdatavg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINTdatalv jfs2 96 192 2 open/syncd /dataloglv01 jfs2log 1 2 2 open/syncd N/A

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 22 of 28

Page 23: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

With the mirroring process complete, we unmirror the volume group from theDS8300 storage (hdisk2).

# unmirrovg datavg hdisk2

To verify that there is no longer any data on the DS8300 hdisks, the lspv commandis run and should not return any output.

# lspv –l hdisk2

Now we can remove the DS8300 hdisks from the data volume group using reducevgand remove the disk from the ODM.

# reducevg datavg hdisk2# rmdev –dl hdisk2

As this is a virtual I/O environment, there are some additional steps we must executebefore we can hand back the DS8300 LUNs to the storage team.

First, we must remove the device mappings for the DS8300 disk from the DS8300VIOS (hvio1 and 2). We should also take note of the DS8300 LUN id (from pcmpathquery device) so we can provide the storage admin with a list of LUN ids that can bereclaimed. In the following example, we check the disk mappings for vhost2 andverify the backing device hdisk number. We then enter the OEM VIOS environmentto run the pcmpath utility and obtain the LUN ids associated with each hdisk. Next,we run the rmvdev command to remove any virtual target device (VTD) mappingassociated with the hdisk (e.g. vtscsi2). And finally we remove the hdisk from theODM on the VIOS. The old DS8300 LUNs can now be reclaimed by the storageteam.

$ lsmap –vadapter vhost2 | grep hdisk$ oem_setup_env# pcmpath query device 20 | grep Serial# exit$ rmvdev -vtd vtscsi2$ rmdev –dev hdisk20

The virtual SCSI server adapters (i.e. vhost2) are removed from each VIOS ($rmdev –dev vhost2). And using DLPAR, we remove these virtual adapters from theLPAR definition. The VIOS partition profile is also updated to reflect the removal ofthese virtual devices.

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 23 of 28

Page 24: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

The original virtual SCSI adapters on the client LPAR, vscsi0 and vscsi1, are alsoremoved now using rmdev and DLPAR (the LPAR partition profile is also updated).

Again, at this point, you would reboot the LPAR once you are satisfied that all datamigrations have completed successfully. Verify correct system operation after thereboot (volume groups online, logical volumes open/syncd, filesystems mounted,etc.).

# lsvg# lsvg –o# df# mount# lsvg | lsvg –il | grep close

With the changes completed successfully we now document the systemconfiguration after the migration. At this point, we have reached our desired endstate. The AIX system is now using NetApp storage via our new VIOS. You can referto Figure 10.

The DS8300 (in this case) will remain in our environment. Likewise, the VIOS usedto serve DS8300 storage to client LPARs, will also remain. As it has beendetermined that the DS storage will be used for AIX systems requiring a high level ofperformance and availability. While all other non-critical AIX test systems will utilizethe NetApp device.

Figure 10. VIOS end state

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 24 of 28

Page 25: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Summary

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 25 of 28

Page 26: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

AIX storage migrations can be time consuming and tricky. Without the latest storagevirtualization technology, such as IBM's SVC, the task requires several phases,stages and tasks.

Fortunately, the AIX LVM is a very mature, stable and robust tool. It can greatlyempower us when faced with these challenges.

Using LVM, we can reduce the outage required for migrating the data. Having theability to move data between source and targets disks, while the system/applicationsare still running, is a huge benefit in an operational computing environment.

As always, I strongly recommend that you test your procedures in a non-productionenvironment before attempting a storage migration on a production system.

If you’ve found this article interesting, then there are other LVM commands whichare also worth researching, such as replacepv, redefinevg and recreatevg. Youmay also find the LVM hot spare policy of interest. Review the Resources sectionbelow to find out more.

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 26 of 28

Page 27: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

Resources

Learn

• IBM Fix Level recommendation tools

• IBM TotalStorage support: Search for host bus adapters, firmware

• IBM System Storage DS8300 series – Interoperability matrix

• SDDPCM support site

• PowerVM migration from physical to virtual storage

• AIX Logical Volume Manager from A to Z: Introduction and concepts

• AIX Logical Volume Manager from A to Z: Troubleshooting and commands

• Virtual fibre channel (NPIV)

• migratepv command

• mirrorvg command

• LVM Hot-spare disk policies

• Replacing a failed physical volume in a mirrored volume group

• alt_disk_copy command

• replacepv command

Discuss

• Follow developerWorks on Twitter.

• Get involved in the My developerWorks community.

• Participate in the AIX and UNIX forums:

• AIX Forum

• AIX Forum for developers

• Cluster Systems Management

• IBM Support Assistant Forum

• Performance Tools Forum

• Virtualization Forum

• More AIX and UNIX Forums

ibm.com/developerWorks developerWorks®

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 27 of 28

Page 28: Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago

About the author

Chris GibsonChris Gibson is an AIX systems specialist located in Melbourne,Australia. He is an IBM CATE, System p platform and AIX 5L, and aco-author of the IBM Redbooks publication, "NIM from A to Z in AIX5L."

developerWorks® ibm.com/developerWorks

Using the AIX Logical Volume Manager to perform SAN storage migrations Trademarks© Copyright IBM Corporation 2010. All rights reserved. Page 28 of 28