VIO Server IMT France

© IBM France ITD La Gaude Unix team 2010 Author : Alain Lentini

IBM GLOBAL SERVICES

IMT France / ITD La Gaude

Pseries Department

VIO server

Installation Configuration Standardization

Author : Alain Lentini

Document Management

Changes history

Version Date author Changes 1.0 May 2008 A.Lentini Initial version – DRAFT 2.0 February 2010 A.Lentini

- all types of SEA detailed with figures and build steps - command line mode for all the SEA configuration - rules and naming convention for SEA build - SAN disks config detailed with 3 typical configuration- command line mode for all the SAN disk configuration- rules and naming convention for SAN build - design and build recommendations for ITD/SSO lge

Reviewers list Name Date Ling Wai Hung February 2010 Stéphane Saleur February 2010 Guy Aluno February 2010 Sylvain Desnoes February 2010 Nicolas Doz February 2010

© IBM France ITD La Gaude Unix team 2010 - 2 -

1. INTRODUCTION..............................................................................................................7

2. STANDARD DESIGN EXTRACTS ..................................................................................7

3. INSTALL OS VIO.............................................................................................................9

3.1. DOWNLOAD FIXPACK AND VIOS ..................................................................................................... 9

3.2. UPGRADE VIOS FIXPACK ................................................................................................................. 9

3.3. VIOS BACKUP/RESTORE .................................................................................................................. 9

4. SHARED ETHERNET ADAPTER CONFIGURATION ....................................................9

4.1. BASIC SEA WITHOUT VLAN TAGGING............................................................................................ 9

4.1.1. BASIC SEA LOGICAL VIEW.......................................................................................................... 9 4.1.2. CREATE SEA ON VIO1 ................................................................................................................ 10

4.1.2.1. Create virtual Ethernet adapter on VIO1 .......................................................................... 10 4.1.2.1.1. command line mode........................................................................................................ 11

4.1.2.2. create control channel on VIO1 ........................................................................................ 11 4.1.2.2.1. command line mode........................................................................................................ 12 4.1.2.2.2. rules and naming convention .......................................................................................... 12

4.1.2.3. sea creation ........................................................................................................................ 13 4.1.3. CREATE SEA ON VIO2 ................................................................................................................ 13


4.1.3.2. create control channel on VIO2 ........................................................................................ 14 4.1.3.2.1. command line mode........................................................................................................ 15

4.1.3.3. sea creation ........................................................................................................................ 15 4.1.4. CLIENT PARTITION CONFIGURATION ...................................................................................... 15

4.1.4.1. Create virtual Ethernet adapter on client partition.......................................................... 16 4.1.4.2. IP configuration on client partition................................................................................... 16

4.2. ETHERCHANNEL.............................................................................................................................. 16

4.3. SEA WITH VLAN TAGGING ON THE SAME DEVICE ..................................................................... 16

4.3.1. IMPORTANT : AVOID THIS SOLUTION FOR OUR ARCHITECTURES ..................................... 16 4.3.2. SEA WITH VLAN TAGGING ON SAME INTERFACE LOGICAL VIEW ...................................... 16 4.3.3. CREATE SEA ON VIO1 ................................................................................................................ 17 4.3.4. CREATE ETHERCHANNEL ON VIO1.......................................................................................... 17


4.3.4.2. create control channel on VIO1 ........................................................................................ 18 4.3.4.2.1. rules and naming convention .......................................................................................... 19

4.3.4.3. sea creation on VIO1.......................................................................................................... 20 4.3.5. CREATE SEA ON VIO2 ................................................................................................................ 20


4.3.6. CREATE ETHERCHANNEL ON VIO2.......................................................................................... 20 4.3.6.1. Create virtual Ethernet adapter on VIO2 .......................................................................... 20

4.3.6.1.1. command line mode........................................................................................................ 21 4.3.6.2. create control channel on VIO2 ........................................................................................ 21 4.3.6.3. sea creation on VIO2.......................................................................................................... 22

4.3.7. CLIENT PARTITION CONFIGURATION ...................................................................................... 22 4.3.7.1. Create virtual Ethernet adapter on client partition.......................................................... 22 4.3.7.2. IP configuration on client partition................................................................................... 23

4.3.8. ADD A NEW VLAN ....................................................................................................................... 23

4.4. SEA WITH VLAN TAGGING – RECOMMENDED SOLUTION ......................................................... 24

4.4.1. SEA WITH VLAN TAGGING LOGICAL VIEW.............................................................................. 24 4.4.2. WHY THIS SOLUTION IS RECOMMENDED................................................................................ 25 4.4.3. CREATE SEA ON VIO1 ................................................................................................................ 25 4.4.4. CREATE ETHERCHANNEL ON VIO1.......................................................................................... 25

4.4.4.1. Create virtual Ethernet adapters on VIO1 ........................................................................ 25 4.4.4.1.1. command line mode........................................................................................................ 26

4.4.4.2. create control channel on VIO1 ........................................................................................ 27 4.4.4.2.1. command line mode........................................................................................................ 27 4.4.4.2.2. rules and naming convention .......................................................................................... 28

4.4.4.3. sea creation on VIO1.......................................................................................................... 28 4.4.5. CREATE SEA ON VIO2 ................................................................................................................ 28 4.4.6. CREATE ETHERCHANNEL ON VIO2.......................................................................................... 28


4.4.6.2. create control channel on VIO2 ........................................................................................ 30 4.4.6.2.1. command line mode........................................................................................................ 30

4.4.6.3. sea creation on VIO2.......................................................................................................... 31 4.4.7. CLIENT PARTITION CONFIGURATION ...................................................................................... 31

4.4.7.1. Create virtual Ethernet adapter on client partition.......................................................... 31 4.4.7.2. IP configuration on client partition................................................................................... 32

4.4.8. ADD A NEW VLAN ....................................................................................................................... 32 4.4.8.1. Create virtual Ethernet adapters on VIO1 ........................................................................ 32

4.4.8.1.1. command line mode........................................................................................................ 33 4.4.8.2. Update SEA on VIO1 .......................................................................................................... 34 4.4.8.3. Create virtual Ethernet adapters on VIO2 ........................................................................ 34

4.4.8.3.1. command line mode........................................................................................................ 35 4.4.8.4. Update SEA on VIO2 .......................................................................................................... 35 4.4.8.5. Client partition configuration ............................................................................................ 36

4.4.8.5.1. Create virtual Ethernet on client partition ........................................................................ 36 4.4.8.5.2. IP configuration on client partition................................................................................... 36

5. SAN DISKS CONFIGURATION.....................................................................................37

5.1. RULES AND NAMING CONVENTION .............................................................................................. 37


5.1.1. VTD NAMING CONVENTION ....................................................................................................... 37 5.1.2. SLOT NUMBERS AND VSCSI SERVER...................................................................................... 37

5.1.2.1. rules with LPM.................................................................................................................... 37 5.1.2.2. Rules without lpm .............................................................................................................. 38

5.2. REDBOOK REFERENCES FOR RULES AND NAMING CONVENTION ......................................... 38

5.3. VIRTUAL DEVICES ADAPTER VS QUEUE_DEPTH ....................................................................... 38

5.4. IMPLEMENTATION IN TYPICAL CONFIGURATIONS..................................................................... 39

5.4.1. CONFIGURATION 1 ..................................................................................................................... 39 5.4.2. CONFIGURATION 2 ..................................................................................................................... 40 5.4.3. CONFIGURATION 3 ..................................................................................................................... 40

5.5. FIBER CHANNEL AND HDISK DEVICES SETTINGS...................................................................... 41

5.6. CONFIGURATION 2 DETAILED IMPLEMENTATION ...................................................................... 41

5.6.1. CREATE VSCSI SERVER ADAPTER ON VIO1 FOR STORAGE B4 .......................................... 41 5.6.1.1. command line mode .......................................................................................................... 42

5.6.2. CREATE VSCSI SERVER ADAPTER ON VIO2 FOR STORAGE B4 .......................................... 42 5.6.2.1. command line mode .......................................................................................................... 43

5.6.3. CREATE CLIENT PARTITION DEVICE FOR VIO1 AND STORAGE B4..................................... 43 5.6.3.1. command line mode .......................................................................................................... 44

5.6.4. CREATE CLIENT PARTITION DEVICE FOR VIO2 AND STORAGE B4..................................... 44 5.6.4.1. command line mode .......................................................................................................... 44

5.6.5. CREATE VSCSI SERVER ADAPTER ON VIO1 FOR STORAGE G1.......................................... 44 5.6.5.1. command line mode .......................................................................................................... 45

5.6.6. CREATE VSCSI SERVER ADAPTER ON VIO2 FOR STORAGE G1.......................................... 45 5.6.6.1. command line mode .......................................................................................................... 46

5.6.7. CREATE CLIENT PARTITION DEVICE FOR VIO1 AND STORAGE G1..................................... 46 5.6.7.1. command line mode .......................................................................................................... 47

5.6.8. CREATE CLIENT PARTITION DEVICE FOR VIO2 AND STORAGE G1..................................... 47 5.6.8.1. command line mode .......................................................................................................... 47

5.6.9. CREATE VIRTUAL DEVICES ON VIO1 ....................................................................................... 47 5.6.10. CREATE VIRTUAL DEVICES ON VIO2................................................................................... 48 5.6.11. CLIENT PARTITION DEVICES SETTING................................................................................ 48

5.6.11.1. priority................................................................................................................................. 48 5.6.11.2. hcheck_interval .................................................................................................................. 49 5.6.11.3. queue depth........................................................................................................................ 49 5.6.11.4. vscsi_path_to ..................................................................................................................... 49

6. SCSI DISK DEVICES CONFIGURATION FOR ROOTVG CLIENT PARTITIONS .......49


6.1. CREATE THE LOGICAL VOLUMES AND THE VIRTUAL DEVICES............................................... 49

7. NTP CONFIGURATION ON VIO AND HMC .................................................................52

7.1. NTP CONFIGURATION ON VIO SERVERS...................................................................................... 52

7.2. NTP CONFIGURATION ON HMC...................................................................................................... 52


1. Introduction In this document, we assume that we must redundant VIO servers for each type of I/O. This means we can have configuration with 2 VIO servers for LAN and 2 others VIO servers for SAN, or we can have configuration with 2 VIO servers for LAN and SAN. You can use the document to adapt to other types of configurations, but you manage the risks. 2. Standard design extracts Here are architecture designs recommended in this document.

Basic SEA without vlan tagging


SEA with vlan tagging – recommended solution

Vscsi configuration 1 : 2 SAN VIO - 1 storage subsystem - 1 vscsi server per VIO server


Vscsi configuration 2 : 2 SAN VIO - 2 storage subsystems - 2 vscsi server per VIO 3. install OS Vio

3.1. download fixpack and vios http://www14.software.ibm.com/webapp/set2/sas/f/vios/home.html

3.2. upgrade vios fixpack updateios -dev /mnt/lpp_source/vios14_fixpack92_lpp -install –accept updateios -dev /mnt/lpp_source/vios15_fixpack101_lpp -install –accept reboot ioslevel

3.3. VIOs BACKUP/RESTORE backupios -file /mnt ( generate a tar file containing a mksysb and other informations ) The command creates a full backup tar file package including all of the resources that the installios command will need to install a Virtual I/O Server (mksysb, bosinst.data, network bootimage, and SPOT) from an HMC using the installios command. backupios –file /mnt/VIOS1_newP5_B2_220308.mksysb -mksysb ( generate only the mksysb file ) 4. Shared Ethernet Adapter configuration

4.1. Basic SEA without vlan tagging

4.1.1. Basic SEA logical view


4.1.2. create SEA on VIO1 Initialize all physical lan interfaces with the good settings : For example : 1000_Full_Duplex on the 2 VIOs

4.1.2.1. Create virtual Ethernet adapter on VIO1 Create the Virtual LAN adapter on the Hmc ( with dlpar and in the Vio partition profile ) : trunk priority = 1 on VIO1 A numerically lower priority receives a higher overall priority.


4.1.2.1.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 14 -a "ieee_virtual_eth=1,port_vlan_id=1101,is_trunk=1,trunk_priority=1" m=managed system id=partition id s=slot number VIO Profile update chsyscfg -r prof -m P01MMA03B49A04D07SN652F8BE -I "name=Normal,lpar_id=2,virtual_eth_adapters+=14/1/1101//1/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.1.2.2. create control channel on VIO1


4.1.2.2.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 15 -a "ieee_virtual_eth=0,port_vlan_id=3101" m=managed system id=partition id s=slot number VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+= 15/0/3101///1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.1.2.2.2. rules and naming convention slot number all slots > 10 slots < 50 : LAN slots slots > 50 : VSCSI slots same slots on both VIO PVID

1000 + Physical VLAN id


( Use virtual VLAN id different from the network VLAN on the switch ) 2000 + Physical VLAN id This rule is mandatory for configuration with several SEA on the same VLAN. For example, in case of VIO pro and pre-prod separated, you must use the same vlan on the SEA of prod and pre-prod. So you must use different VLANid between prod and pre-prod SEA. PVID for control channel

3000 + Physical VLAN id

4.1.2.3. sea creation

This will generate a device on the VIO server ( cfgdev or cfgmgr ) To find the device, use the slot number in the command lsdev –slots Exemple lsdev -slots U9117.570.65A1DA0-V2-C14 Virtual I/O Slot ent8 ( slot 2 ) U9117.570.65A1DA0-V2-C3 Virtual I/O Slot ent9 U9117.570.65A1DA0-V2-C15 Virtual I/O Slot ent13 lsdev -Cc adapter|grep ent ent0 Available 0C-20 4-Port 10/100/1000 Base-TX PCI-X Adapter (14101103) ent8 Available Virtual I/O Ethernet Adapter (l-lan) mkvdev -sea ent0 -vadapter ent8 -default ent8 -defaultid 1101 –attr ha_mode=auto ctl_chan=ent13 ent14 Important: If you do not define the ctl_chan attribute to the Shared Ethernet Adapters, they will not be able to negotiate which one is providing the bridging functionality. At this point, both SEAs will be bridging and a spanning tree loop can be formed. To avoid this, always specify the these parameters with the SEA creation.

4.1.3. create SEA on VIO2 Initialize all physical lan interfaces with the good settings : For example : 1000_Full_Duplex on the 2 VIOs

4.1.3.1. Create virtual Ethernet adapter on VIO2 trunk priority = 2 on VIO2 A numerically lower priority receives a higher overall priority.


4.1.3.1.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 14 -a "ieee_virtual_eth=1,port_vlan_id=1101,is_trunk=1,trunk_priority=2" m=managed system id=partition id s=slot number VIO Profile update chsyscfg -r prof -m P01MMA03B49A04D07SN652F8BE -I "name=Normal,lpar_id=3,virtual_eth_adapters+=14/1/1101//2/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required



4.1.3.2.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 15 -a "ieee_virtual_eth=0,port_vlan_id=3101" m=managed system id=partition id s=slot number VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+= 15/0/3101///1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.1.3.3. sea creation mkvdev -sea ent0 -vadapter ent8 -default ent8 -defaultid 1101 –attr ha_mode=auto ctl_chan=ent13 ent14 Important: If you do not define the ctl_chan attribute to the Shared Ethernet Adapters, they will not be able to negotiate which one is providing the bridging functionality. At this point, both SEAs will be bridging and a spanning tree loop can be formed. To avoid this, always specify the these parameters with the SEA creation.

4.1.4. Client partition configuration


4.1.4.1. Create virtual Ethernet adapter on client partition

4.1.4.2. IP configuration on client partition lscfg -vl ent*|grep ent ent0 U9117.MMA.652F8CE-V27-C2-T1 Virtual I/O Ethernet Adapter set IP address on the device ent0.

4.2. etherchannel mkvdev -lnagg ent8 ent12 ent16 -attr mode=8023ad

4.3. SEA with vlan tagging on the same device

4.3.1. important : avoid this solution for our architectures This solution works properly but don’t allow a dynamic update when you have to add a new vlan on your network. It needs to delete the SEA to add a new vlan.

4.3.2. SEA with vlan tagging on same interface logical view


4.3.3. create sea on VIO1

4.3.4. create etherchannel on vio1 Etherchannel VLAN 101/501 mkvdev -lnagg ent8 ent12 ent16 -attr mode=8023ad ent34

4.3.4.1. Create virtual Ethernet adapter on VIO1 Trunck priority 1 Not required


4.3.4.1.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 14 -a "ieee_virtual_eth=1,port_vlan_id=808,"addl_vlan_ids=101,501",is_trunk=1,trunk_priority=1" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=14/1/808/101,501/1/0" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required



VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 16 -a "ieee_virtual_eth=0,port_vlan_id=895" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=16/0/895///1"

4.3.4.2.1. rules and naming convention The devices on VIO server must be set to “not required”. slot number all slots > 10 slots < 50 : LAN slots slots > 50 : VSCSI slots same slots on both VIO PVID

1000 + Physical VLAN id ( Use virtual VLAN id different from the network VLAN on the switch )

2000 + Physical VLAN id This rule is mandatory for configuration with several SEA on the same VLAN.


For example, in case of VIO pro and pre-prod separated, you must use the same vlan on the SEA of prod and pre-prod. So you must use different VLANid between prod and pre-prod SEA. PVID for control channel decrement from 900

4.3.4.3. sea creation on VIO1 lsdev –slots U9117.MMA.652F8CE-V2-C14 Virtual I/O Slot ent36 U9117.MMA.652F8CE-V2-C16 Virtual I/O Slot ent38 lsdev –Cc adapter ent34 Available EtherChannel / IEEE 802.3ad Link Aggregation mkvdev -sea ent34 -vadapter ent36 -default ent36 -defaultid 808 -attr ha_mode=auto ctl_chan=ent38 ent39


4.3.6. create etherchannel on vio2 etherchannel VLAN 101/501 mkvdev -lnagg ent8 ent12 ent16 -attr mode=8023ad ent34

4.3.6.1. Create virtual Ethernet adapter on VIO2 Trunck priority 2 Not required


4.3.6.1.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 14 -a "ieee_virtual_eth=1,port_vlan_id=808,"addl_vlan_ids=101,501",is_trunk=1,trunk_priority=2" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+=14/1/808/101,501/2/0" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required



VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 16 -a "ieee_virtual_eth=0,port_vlan_id=895" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+=16/0/895///1"

4.3.6.3. sea creation on VIO2 lsdev –slots U9117.MMA.652F8CE-V2-C14 Virtual I/O Slot ent36 U9117.MMA.652F8CE-V2-C16 Virtual I/O Slot ent38 lsdev –Cc adapter ent34 Available EtherChannel / IEEE 802.3ad Link Aggregation mkvdev -sea ent34 -vadapter ent36 -default ent36 -defaultid 808 -attr ha_mode=auto ctl_chan=ent38 ent39




4.3.7.2. IP configuration on client partition lscfg -vl ent*|grep ent ent0 U9117.MMA.652F8CE-V27-C2-T1 Virtual I/O Ethernet Adapter set IP address on the device ent0.

4.3.8. add a new vlan As explained above, this action is not dynamic. That’s why we recommend not to use this solution in our environments. Modify the virtual device on VIO1 and VIO2. See example below :


Create the client virtual device and set the IP address on the client partition

4.4. SEA with vlan tagging – recommended solution

4.4.1. SEA with vlan tagging logical view


4.4.2. Why this solution is recommended This solution is recommended because adding new vlan on the SEA is dynamic. Even when a client network is stable, the client need can change at any time.


4.4.4. create etherchannel on vio1 Etherchannel VLAN 101/501 mkvdev -lnagg ent8 ent12 -attr mode=8023ad ent34

4.4.4.1. Create virtual Ethernet adapters on VIO1 Create one virtual Ethernet adapter per vlan on each VIO server. Trunck priority 1


4.4.4.1.1. command line mode VIO Dynamic Device creation ( dlpar )


chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 14 –a "ieee_virtual_eth=1,port_vlan_id=1101,"addl_vlan_ids=101",is_trunk=1,trunk_priority=1" chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 15 -a "ieee_virtual_eth=1,port_vlan_id=1501,"addl_vlan_ids=501",is_trunk=1,trunk_priority=1" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=14/1/1101/101/1/1" chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=15/1/1501/501/1/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required


4.4.4.2.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 16 -a "ieee_virtual_eth=0,port_vlan_id=895" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=16/0/895///1"


4.4.4.2.2. rules and naming convention slot number all slots > 10 slots < 50 : LAN slots slots > 50 : VSCSI slots same slots on both VIO PVID

1000 + Physical VLAN id ( Use virtual VLAN id different from the network VLAN on the switch )

2000 + Physical VLAN id This rule is mandatory for configuration with several SEA on the same VLAN. For example, in case of VIO pro and pre-prod separated, you must use the same vlan on the SEA of prod and pre-prod. So you must use different VLANid between prod and pre-prod SEA. PVID for control channel

decrement from 900

4.4.4.3. sea creation on VIO1 lsdev –slots U9117.MMA.652F8CE-V2-C14 Virtual I/O Slot ent36 U9117.MMA.652F8CE-V2-C15 Virtual I/O Slot ent37 U9117.MMA.652F8CE-V2-C16 Virtual I/O Slot ent38 lsdev –Cc adapter ent34 Available EtherChannel / IEEE 802.3ad Link Aggregation mkvdev -sea ent34 -vadapter ent36 ent37 -default ent36 -defaultid 808 -attr ha_mode=auto ctl_chan=ent38 ent39 ( new device genereted )


4.4.6. create etherchannel on vio2 etherchannel VLAN 101/501 mkvdev -lnagg ent8 ent12 -attr mode=8023ad ent34

4.4.6.1. Create virtual Ethernet adapter on VIO2 Trunck priority 2


4.4.6.1.1. command line mode


VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 14 –a "ieee_virtual_eth=1,port_vlan_id=1101,"addl_vlan_ids=101",is_trunk=1,trunk_priority=2" chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 15 -a "ieee_virtual_eth=1,port_vlan_id=1501,"addl_vlan_ids=501",is_trunk=1,trunk_priority=2" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+=14/1/1101/101/2/1" chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+=15/1/1501/501/2/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required


4.4.6.2.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 3 --rsubtype eth -s 16 -a "ieee_virtual_eth=0,port_vlan_id=895" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=3,virtual_eth_adapters+=16/0/895///1"


virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.4.6.3. sea creation on VIO2 mkvdev -sea ent34 -vadapter ent36 ent37 -default ent36 -defaultid 808 -attr ha_mode=auto ctl_chan=ent38 ent39 ( new device genereted )




4.4.7.2. IP configuration on client partition lsdev –Cc adapter ent0 U9117.MMA.652F8CE-V27-C2-T1 Virtual I/O Ethernet Adapter set IP address on the device ent0. ent1 U9117.MMA.652F8CE-V27-C3-T1 Virtual I/O Ethernet Adapter set IP address on the device ent1.

4.4.8. add a new vlan

4.4.8.1. Create virtual Ethernet adapters on VIO1


4.4.8.1.1. command line mode VIO Dynamic Device creation ( dlpar )


chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 17 -a "ieee_virtual_eth=1,port_vlan_id=1105,"addl_vlan_ids=105",is_trunk=1,trunk_priority=1" chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 18 -a "ieee_virtual_eth=1,port_vlan_id=1505,"addl_vlan_ids=505",is_trunk=1,trunk_priority=1" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=17/1/1105/105/1/1 chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=18/1/1505/505/1/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.4.8.2. Update SEA on VIO1 lsdev -slots U9117.MMA.652F8CE-V3-C17 Virtual I/O Slot ent56 U9117.MMA.652F8CE-V3-C18 Virtual I/O Slot ent57 chdev -dev ent39 -attr virt_adapters=ent36,ent37, ent56, ent57

4.4.8.3. Create virtual Ethernet adapters on VIO2


4.4.8.3.1. command line mode VIO Dynamic Device creation ( dlpar ) chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 17 -a "ieee_virtual_eth=1,port_vlan_id=1105,"addl_vlan_ids=105",is_trunk=1,trunk_priority=2" chhwres -r virtualio -m P01MMA02B2A69D07SN104700D -o a --id 2 --rsubtype eth -s 18 -a "ieee_virtual_eth=1,port_vlan_id=1505,"addl_vlan_ids=505",is_trunk=1,trunk_priority=2" VIO Profile update chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=17/1/1105/105/2/1 chsyscfg -r prof -m P01MMA02B2A69D07SN104700D -i "name=Normal,lpar_id=2,virtual_eth_adapters+=18/1/1505/505/2/1" virtual-slot-number/is-IEEE/ port-vlan-ID/additional-vlan-IDs/trunk-priority/is-required

4.4.8.4. Update SEA on VIO2 lsdev -slots U9117.MMA.652F8CE-V3-C17 Virtual I/O Slot ent56 U9117.MMA.652F8CE-V3-C18 Virtual I/O Slot ent57 chdev -dev ent39 -attr virt_adapters=ent36,ent37, ent56, ent57


4.4.8.5. Client partition configuration

4.4.8.5.1. Create virtual Ethernet on client partition

4.4.8.5.2. IP configuration on client partition


lscfg -vl ent*|grep ent ent1 U9117.MMA.652F8BE-V34-C5-T1 Virtual I/O Ethernet Adapter (l-lan) ent0 U9117.MMA.652F8BE-V34-C4-T1 Virtual I/O Ethernet Adapter (l-lan) set IP address on the device ent0 and ent1 5. san disks configuration

5.1. rules and naming convention

5.1.1. vtd naming convention Depending on your ios version and fixpack level, it’s possible that the vtd name can be renamed by LPM. In this case, it’s useless to define vtd naming convention. Depending on the customer specific context, it can be usefull that the vtd contains several information such as the geographical or functional information. Here is the recommended vtd syntax : [ additional information ] [ _ ] < client partition name > [ _ ] < lun_id > additional information : any geographical or functional information client partition name : this name can be reduced if necessary lun_id : lun id on the storage subsystem Examples : usr210_0543 us : counry r210 : a part of the client partition name 0543 : lunid on DS8000 saxsap12phd4 saxsap12p : a part of the client partition name hd4 : corresponding disk on the client partition

5.1.2. slot numbers and vscsi server

5.1.2.1. rules with LPM Vscsi server adapter slot number No specific rule for slot number on VIO server if LPM is used (LPM redefines slot numbers). Vscsi client adapter


Use Odd numbered slots starting at 21 for connection to the first VIO server, and use even numbered slots starting at 20 for connection to the second VIO server.

Allocate slots in pairs, with each client using two adjacent slots such as 21 and 20, or 31 and 30.

5.1.2.2. Rules without lpm

Vscsi server adapter

Use a specific VSCSI server for each storage subsystem. The first VIOS should use Odd numbered slots number starting at 51, and the second

should use even numbered slots starting at 50. In a two server scenario, allocate slots in pairs, with each client using two adjacent slots such as 51 and 50, or 61 and 60.

Vscsi client adapter

Use the same slot number between the vscsi server and client adapter. 5.2. redbook references for rules and naming convention

see explain in the following redbook : “IBM System p Advanced POWER Virtualization Best Practices” redp4194.pdf chapter 4.3.2 : “Virtual device slot numbers” “After establishing the naming conventions, also establish slot numbering conventions for the virtual I/O adapters. Slot numbers are shared between virtual storage and virtual network devices. In complex systems, there will tend to be far more storage devices than network devices because each virtual SCSI device can only communicate with one server or client. We recommend reserving the slot numbers through 20 for network devices on all LPARs in order to keep the network and storage devices grouped together. Management can be simplified by keeping slot numbers consistent between the virtual I/O client and server. However, when partitions are moved from one server to another, this might not be possible. For more information, see 4.6.3, “Storage planning” on page 108. In environments with only one virtual I/O server, add storage adapters incrementally starting with slot 21 and higher. When clients are attached to two Virtual I/O Servers, the adapter slot numbers should be alternated from one VIOS to the other. The first VIOS should use odd numbered slots starting at 21, and the second should use even numbered slots starting at 22. In a two server scenario, allocate slots in pairs, with each client using two adjacent slots such as 21 and 22, or 33 and 34.”

5.3. Virtual devices adapter vs queue_depth As explained below, you have to take care of the maximum disks per virtual adapter server on the VIO. Use the following process to evaluate this max value : ( 512 – 2 ) / ( 3 + queue depth ) See redbook “power VM Virtualization Managing and monitoring” sg247590.pdf chapter 2.5.2 Virtual device slot numbers:


“Because virtual SCSI connections operate at memory speed, there is generally no performance gain from adding multiple adapters between a Virtual I/O Server and client. For AIX virtual I/O client partitions, each adapter pair can handle up to 85 virtual devices with the default queue depth of three. For IBM i clients, up to 16 virtual disk and 16 optical devices are supported. In situations where virtual devices per partition are expected to exceed that number, or where the queue depth on some devices might be increased above the default, reserve additional adapter slots for the Virtual I/O Server and the virtual I/O client partition. When tuning queue depths, the VSCSI adapters have a fixed queue depth. There are 512 command elements of which 2 are used by the adapter, 3 are reserved for each VSCSI LUN for error recovery and the rest are used for I/O requests. Thus, with the default queue depth of 3 for VSCSI LUNs, that allows for up to 85 LUNs to use an adapter: (512 - 2) / (3 + 3) = 85 rounding down. So if you need higher queue depths for the devices, then the number of LUNs per adapter is reduced. For example, if you want to use a queue depth of 25, that allows 510/28= 18 LUNs per adapter for an AIX client partition.”

5.4. Implementation in Typical configurations

5.4.1. Configuration 1

2 SAN VIO 1 storage subsystem 1 vscsi server per VIO server LPM used

vscsi1 U9119.FHA.832EE32-V234-C21-T1 VIO2 Storage B4 vscsi0 U9119.FHA.832EE32-V234-C20-T1 VIO1 Storage B4 Client adapter slot numbers


5.4.2. configuration 2

2 SAN VIO 2 storage subsystems 2 vscsi server per VIO : 1 for each storage subsystem LPM used

Storage bay B4 ( slot nb range 20 to 29 ) Storage bay G1 ( slot nb range 30 to 39 ) vscsi3 U9119.FHA.832EE32-V234-C31-T1 VIO2 Storage G1 vscsi2 U9119.FHA.832EE32-V234-C30-T1 VIO1 Storage G1 vscsi1 U9119.FHA.832EE32-V234-C21-T1 VIO2 Storage B4 vscsi0 U9119.FHA.832EE32-V234-C20-T1 VIO1 Storage B4 Client adapter slot numbers

5.4.3. configuration 3

2 SAN VIO 2 storage subsystems 4 vscsi server per VIO : 2 for each storage subsystems ( for nb device limitation ) LPM used

Storage bay B4 ( slot nb range 20 to 29 ) Storage bay G1 ( slot nb range 30 to 39 ) vscsi7 U9119.FHA.834EE9F-V132-C33-T1 VIO2 Storage G1 vscsi6 U9119.FHA.834EE9F-V132-C32-T1 VIO1 Storage G1


vscsi5 U9119.FHA.834EE9F-V132-C23-T1 VIO2 Storage B4 vscsi4 U9119.FHA.834EE9F-V132-C22-T1 VIO1 Storage B4 vscsi3 U9119.FHA.834EE9F-V132-C31-T1 VIO2 Storage G1 vscsi2 U9119.FHA.834EE9F-V132-C30-T1 VIO1 Storage G1 vscsi1 U9119.FHA.834EE9F-V132-C21-T1 VIO2 Storage B4 vscsi0 U9119.FHA.834EE9F-V132-C20-T1 VIO1 Storage B4 Client adapter slot numbers

5.5. Fiber channel and hdisk devices settings For each disk device on the vio server : Activate the pvid on all the disks devices on both vioservers : chdev -l hdiskX -a pv=yes Set the reserve_policy parameter : chdev -dev hdiskX -attr reserve_policy=no_reserve

For each Fiber channel card on the VIO server Initialize all physical fiber channel interfaces with the good settings chdev -dev fscsi0 -attr fc_err_recov=fast_fail dyntrk=yes –perm

5.6. Configuration 2 detailed implementation Vio servers : VIE1P01MMA04B4 VIE2P01MMA04B4 Client partition : PODCLIENT_TEST6 building : B4 and G1 dual storage : 1 storage subsystem in each building LPM used Doesn’t need the same slot number between server and client adapter In this example, the values of vscsi server adapters slot number is not important as LPM is used.

5.6.1. create vscsi server adapter on VIO1 for storage B4 The adapter must be not required for LPM


5.6.1.1. command line mode vio1 device, server slot 50, client slot 20, storage B4 chhwres -r virtualio -m 9119-FHA-SN834EE9F-BA1A05D12 -o a --id 4 --rsubtype scsi -s 50 -a "adapter_type=server,remote_lpar_name=PODCLIENT_TEST6,remote_slot_num=20" chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=4,virtual_scsi_adapters+=50/server//PODCLIENT_TEST6/20/0"

5.6.2. create vscsi server adapter on VIO2 for storage B4 The adapter must be not required for LPM


5.6.2.1. command line mode vio2 device, server slot 51, client slot 20, storage B4 chhwres -r virtualio -m 9119-FHA-SN834EE9F-BA1A05D12 -o a --id 5 --rsubtype scsi -s 51 -a "adapter_type=server,remote_lpar_name=PODCLIENT_TEST6,remote_slot_num=21" chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=5,virtual_scsi_adapters+=51/server//PODCLIENT_TEST6/21/0"

5.6.3. create client partition device for VIO1 and storage B4


5.6.3.1. command line mode client partition device, vio1, server slot 50, client slot 20, storage B4 chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=128,virtual_scsi_adapters+=20/client//VIE1P01MMA04B4/50/1"

5.6.4. create client partition device for VIO2 and storage B4

5.6.4.1. command line mode client partition device,vio2, server slot 51, client slot 21, storage B4 chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=128,virtual_scsi_adapters+=21/client//VIE2P01MMA04B4/40/1"

5.6.5. create vscsi server adapter on VIO1 for storage G1 The adapter must be not required for LPM


5.6.5.1. command line mode vio1 device, server slot 60, client slot 30, storage G1 chhwres -r virtualio -m 9119-FHA-SN834EE9F-BA1A05D12 -o a --id 4 --rsubtype scsi -s 60 -a "adapter_type=server,remote_lpar_name=PODCLIENT_TEST6,remote_slot_num=30" chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=4,virtual_scsi_adapters+=60/server//PODCLIENT_TEST6/30/0"

5.6.6. create vscsi server adapter on VIO2 for storage G1


5.6.6.1. command line mode vio2 device, server slot 61, client slot 31, storage G1 chhwres -r virtualio -m 9119-FHA-SN834EE9F-BA1A05D12 -o a --id 5 --rsubtype scsi -s 61 -a "adapter_type=server,remote_lpar_name=PODCLIENT_TEST6,remote_slot_num=31" chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=5,virtual_scsi_adapters+=61/server//PODCLIENT_TEST6/31/0"

5.6.7. create client partition device for VIO1 and storage G1


5.6.7.1. command line mode client partition device, vio1, server slot 60, client slot 30, storage G1 chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=128,virtual_scsi_adapters+=30/client//VIE1P01MMA04B4/60/1"

5.6.8. create client partition device for VIO2 and storage G1

5.6.8.1. command line mode client partition device, vio2, server slot 61, client slot 31, storage G1 chsyscfg -r prof -m 9119-FHA-SN834EE9F-BA1A05D12 -i "name=default,lpar_id=128,virtual_scsi_adapters+=31/client//VIE2P01MMA04B4/61/1"

5.6.9. create virtual devices on VIO1 lsdev -slots U9119.FHA.834EE9F-V4-C50 Virtual I/O Slot vhost0 U9119.FHA.834EE9F-V4-C60 Virtual I/O Slot vhost5

vhost0 will be used for B4 storage subsystem. vhost5 will be used for G1 storage subsystem.

rootvg hdisk13 B4 hdisk52 G1 su - padmin -c ioscli mkvdev -vdev hdisk13 -vadapter vhost0 -dev p10_0230 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk52 -vadapter vhost5 -dev p10_0231 ;


ssh PODCLIENT_TEST6 cfgmgr hdisk2 to hdisk6 : storage subsystem B4 su - padmin -c ioscli mkvdev -vdev hdisk2 -vadapter vhost0 -dev usp010_3102 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk3 -vadapter vhost0 -dev usp010_3103 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk4 -vadapter vhost0 -dev usp010_3104 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk5 -vadapter vhost0 -dev usp010_3105 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk6 -vadapter vhost0 -dev usp010_3106 ; ssh PODCLIENT_TEST6 cfgmgr hdisk39 to hdisk43 : storage subsystem G1 su - padmin -c ioscli mkvdev -vdev hdisk39 -vadapter vhost5 -dev usp010_2050 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk40 -vadapter vhost5 -dev usp010_2051 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk41 -vadapter vhost5 -dev usp010_2052 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk42 -vadapter vhost5 -dev usp010_2053 ; ssh PODCLIENT_TEST6 cfgmgr su - padmin -c ioscli mkvdev -vdev hdisk43 -vadapter vhost5 -dev usp010_2054 ; ssh PODCLIENT_TEST6 cfgmgr

5.6.10. create virtual devices on VIO2 lsdev -slots U9119.FHA.834EE9F-V4-C40 Virtual I/O Slot vhost0 U9119.FHA.834EE9F-V4-C45 Virtual I/O Slot vhost5 The vhosts are the same as the vhosts of VIO1, the script to generate the virtual devices is the same.

5.6.11. Client partition devices setting

5.6.11.1. priority Change the priority on all the disks : You have to share the workload between both vioservers. Here are the different strategies :

Set all the disks of each partition on one vioserver with priority 1 Spread all disks of each partition between both VIO servers

chpath -l hdisk0 -p vscsi5 -a priority=2 chpath -l hdisk1 -p vscsi5 -a priority=2


Check : lspath -AHE -l hdiskx -p vscsiY

5.6.11.2. hcheck_interval Change the following setting on each disk device : hcheck_interval needs to be greater than the R/W timeout interval.

1) check the rw_timeout value on the physical device disks on the vio 2) put at least the same value on the hcheck_interval on the virtual device disks on the partition

chdev -l hdiskX -a hcheck_interval=60 –P Check lsattr -El hdiskX| grep hcheck_interval

5.6.11.3. queue depth For all the device disks of the client partitions, set the queue depth at the same value as the queue depth value of the physical disks of the VIO servers. See recommendation redbook "Virt Best Practices Redp4194" chapter 4.8 SCSI queue depth: “We recommend that you have the same queue depth on the virtual disk as the physical disk”

5.6.11.4. vscsi_path_to see redbook sg247940 “advanced power virtualization” chapter 4.2.1 MPIO in the client partition” On all the virtual scsi devices of the client partition, put the following value : chdev -l vscsiX -a vscsi_path_to=30 –P 6. scsi disk devices configuration for rootvg client partitions Create the vhost in WebSm menu. properties -> Virtual IO Adapter->onglet scsi->create server adapter Naming convention.You can use the following rules :

Start the slots numbers to 20 for the Vscsi devices. Use < 20 for the LAN devices. Use even slots numbers for VIO1 and odd numbers for VIO2 Use the first slot number for Aix disk and the other for the swap disk Set the maximum slot number to a value that prevents the VIO partition stop.

assign the new virtual device to its specific client partition. On the VIOs : cfgdev Check with lsdev -slots

6.1. Create the logical volumes and the virtual devices On vioserver1


Create the virtual server device in WebSm. assign the new virtual device to its specific client partition. See example below : In our naming convention, we advice to chose the same slot number for the client and for the server virtual scsi adapter ( see example below ) :

mkvg -f -vg vioserver1VG hdisk1 hdisk5 extendvg vioserver1VG hdisk2 hdisk6 partition mopesaxsap15p mklv -lv saxsap15phd0LV vioserver1VG 30G hdisk1 mklv -lv saxsap15phd2LV vioserver1VG 20G hdisk5 mkvdev -vdev saxsap15phd0LV -vadapter vhost0 -dev vtsaxsap15phd0 mkvdev -vdev saxsap15phd2LV -vadapter vhost1 -dev vtsaxsap15phd2 on vioserver2 Create the virtual server device in WebSm mkvg -f -vg vioserver2VG hdisk1 hdisk5 extendvg vioserver2VG hdisk2 hdisk6 partition mopesaxsap15p mklv -lv saxsap15phd1LV vioserver2VG 30G hdisk2 mklv -lv saxsap15phd3LV vioserver2VG 20G hdisk6 mkvdev -vdev saxsap15phd1LV -vadapter vhost0 -dev vtsaxsap15phd1 mkvdev -vdev saxsap15phd3LV -vadapter vhost1 -dev vtsaxsap15phd3 On the client partition mopesaxsap15p Create the vscsi client on the client partition profile ( same slot number as the vscsi server )


Here is an example of the result after the creation of 4 virtual scsi client adapter :


7. NTP configuration on VIO and HMC Enable NTP on all VIO servers and all HMC.

7.1. Ntp configuration on VIO servers See redbook sg247590 “power VM managing and monitoring” chapter 4.3 “Network Time Protocol configuration” vi /home/padmin/config/ntp.conf startnetsvc xntpd ( stopnetsvc xntpd ) check your ntp.log file lsnetsvc xntpd ntpdate –d <server>

7.2. NTP configuration on HMC See redbook sg247491 “HMC V7 handbook” chapter 2.5.14 “Change Date and Time”


VIO Server IMT France

Documents

Transcript of VIO Server IMT France