Huntington National Bank Maintenance Standards v10

Doc. Version: 0.7

Title: Huntington National Bank Maintenance Standards Guide

Doc. Date: 10/29/2013 Page: 1 of 51

INTERNAL USE ONLY This document is controlled electronically. All paper copies of this document are uncontrolled.

Huntington National Bank Maintenance Standards Guide

z/OS and Software Maintenance Reference Manual Draft Version

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 2 of 51


Approvals

Signatures on File

Name Role Date

Name Role Date

Name Role Date

Document Information:

Document Revision History

Version Date Revised By Description of Changes 0.0 10/9/13 Deb Martina Table of Contents and creation of shell document. 0.1 10/15/13 Deb Martina Added notes from Chris 0.2 10/18/13 Deb Martina Moved notes to Notes section 0.3 10/25/13 Deb Martina Added sections after talking to Chris today 0.4 10/25/13 Deb Martina Added Chris’ content into appropriate sections 0.5 10/28/13 Deb Martina Added SMP/e document is to do with installation and

maintenance per Chris’ email 10/25/13. Fixed formatting issues with template (headers).

0.6 10/29/13 Deb Martina Ran spelling and grammar check. Added header and footer. Fixed objective paragraph & doc heading. Added logo to header.

0.7 10/29/13 Deb Martina Met with Chris and Doris and made changes to formatting. Took out the HNB Notes and other sections that were deemed inappropriate.

0.8 11/8/13 Deb Martina Ran spelling and grammar check. Incorporated recommendations.

0.9 11/11/13 Deb Martina Chris made section changes and resent for formatting.

0.10 11/11/13 Deb Martina Incorporated changes per Chris and formatted headings to be left-aligned.

Filename: Huntington Bank Maintenance Standards.doc Location: TBD

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 3 of 51


Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 4 of 51


Table of Contents

Huntington National Bank Maintenance Standards Guide ......................................... 1 Date ........................................................................................................................................ 2

1 Introduction ............................................................................................................. 7 1.1 Objective ....................................................................................................................... 7 1.2 Scope ............................................................................................................................ 7

2 Management Summary ........................................................................................... 8 2.1 Analysis Summary ........................................................................................................ 8 2.2 Recommendations Summary ....................................................................................... 8

3 Naming Standards ................................................................................................ 10 3.1 Naming Standards Analysis ....................................................................................... 10

3.1.1 z/OS ...................................................................................................................... 10 3.1.2 ISV ......................................................................................................................... 10 3.1.3 DB2 ....................................................................................................................... 10 3.1.4 CICS ...................................................................................................................... 10 3.1.5 MQ ......................................................................................................................... 10 3.1.6 ChangeMan ........................................................................................................... 11 3.1.7 USS Filesystems .................................................................................................... 11

3.2 Naming Standards Recommendations ...................................................................... 11 3.2.1 DASD..................................................................................................................... 12

4 Catalog Usage ....................................................................................................... 13 4.1 Catalog Usage Analysis ............................................................................................. 13

4.1.1 General .................................................................................................................. 13 4.1.2 Shared Catalogs .................................................................................................... 13 4.1.3 Unshared Catalogs ................................................................................................ 14

4.2 Catalog Usage Recommendations ............................................................................. 14 5 System Symbolics ................................................................................................ 15

5.1 System Symbolics Analysis ....................................................................................... 15 5.2 System Symbolics Recommendations ...................................................................... 15

6 Installation and Maintenance ............................................................................... 16 6.1 Installation and Maintenance Analysis ...................................................................... 16

6.1.1 z/OS ...................................................................................................................... 16 6.1.2 ISV Products .......................................................................................................... 16 6.1.3 DB2 ....................................................................................................................... 17 6.1.4 CICS ...................................................................................................................... 18 6.1.5 MQ ......................................................................................................................... 18 6.1.6 ChangeMan ........................................................................................................... 18

6.2 Installation and Maintenance Recommendations ..................................................... 18 6.2.1 z/OS Installation ..................................................................................................... 18 6.2.2 z/OS Maintenance .................................................................................................. 19 6.2.3 USS ....................................................................................................................... 19 6.2.4 MQ ......................................................................................................................... 19 6.2.5 ISV Installation (Including ChangeMan) .................................................................. 20 6.2.6 ISV Maintenance (Including ChangeMan) ............................................................... 21 6.2.7 Level-Set Housekeeping ........................................................................................ 21 6.2.8 DB2 Installation ...................................................................................................... 21 6.2.9 DB2 Maintenance ................................................................................................... 22

6.2.9.1 New Release Installation ............................................................................ 22

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 5 of 51


6.2.10 CICS Installation................................................................................................. 23 6.2.11 CICS Maintenance ............................................................................................. 23

6.2.11.1 New Release Installation ............................................................................ 23 7 Deployment ........................................................................................................... 25

7.1 Deployment Analysis .................................................................................................. 25 7.1.1 z/OS ...................................................................................................................... 25 7.1.2 CICS ...................................................................................................................... 25 7.1.3 MQ ......................................................................................................................... 25 7.1.4 DB2 ....................................................................................................................... 26 7.1.5 ISV ......................................................................................................................... 26

Note: ..................................................................................................................................... 26 7.1.6 ChangeMan ........................................................................................................... 26

7.2 Deployment Recommendations ................................................................................. 27 7.2.1 Level Set Deployment ............................................................................................ 27 7.2.2 z/OS Level Set Build .............................................................................................. 28

7.2.2.1 z/OS Creating the SYRES Volumes ............................................................ 29 7.2.2.2 USS............................................................................................................ 29 7.2.2.3 MQ ............................................................................................................. 30

7.2.3 ISV Level Set Build (Including ChangeMan) ........................................................... 30 7.2.4 CICS ...................................................................................................................... 30

7.2.4.1 Prepositioning for Rolled Out Maintenance Deployment .............................. 30 7.2.4.2 CICS Maintenance Deployment .................................................................. 31

7.2.5 DB2 ....................................................................................................................... 32 7.2.5.1 Prepositioning for Rolled Out Maintenance Deployment .............................. 32 7.2.5.2 Non-Disruptive Deployment ........................................................................ 32

8 Software Refresh and Maintenance Cycles ........................................................ 34 8.1 Software Currency ...................................................................................................... 34 8.2 Recommendations for Software Currency ................................................................ 34 8.3 z/OS Maintenance ....................................................................................................... 35 8.4 RSM’s Recommendation for z/OS Maintenance Practices ....................................... 37 8.5 DB2 Maintenance ........................................................................................................ 37

8.5.1 Service Installation ................................................................................................. 37 8.6 CICS Maintenance....................................................................................................... 37

8.6.1 Service Installation ................................................................................................. 37 9 Testing ................................................................................................................... 38

9.1 Testing Analysis ......................................................................................................... 38 9.1.1 ISV Testing ............................................................................................................ 38 9.1.2 CICS Testing.......................................................................................................... 38 9.1.3 DB2 Testing ........................................................................................................... 38 9.1.4 MQ Testing ............................................................................................................ 38 9.1.5 ChangeMan Testing ............................................................................................... 38

9.2 Testing Issues ............................................................................................................. 39 9.3 Testing Recommendations ........................................................................................ 39

9.3.1 Test Cases ............................................................................................................. 40 9.3.2 z/OS RSU Maintenance Testing ............................................................................. 40 9.3.3 DB2 ....................................................................................................................... 41

10 PARMLIB & PROCLIB ........................................................................................... 42 10.1 PARMLIB & PROCLIB Analysis ............................................................................... 42

10.1.1 SYS1.IPLPARM ................................................................................................. 42 10.1.2 PARMLIB ........................................................................................................... 42

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 6 of 51


10.1.3 SYS1.IBM.PARMLIB .......................................................................................... 42 10.1.4 SYS1.PARMLIB ................................................................................................. 43 10.1.5 IEASYM06 ......................................................................................................... 43 10.1.6 IEASYS00 .......................................................................................................... 44 10.1.7 IEASYSxy .......................................................................................................... 45 10.1.8 PROCLIB ........................................................................................................... 45 10.1.9 SYS1.IBM.PROCLIB .......................................................................................... 46 10.1.10 SYS1.HNBSPLEX.PROCLIB .............................................................................. 46 10.1.11 SYS2.HNBSPLEX.PROCLIB .............................................................................. 46

10.2 PARMLIB & PROCLIB Recommendations .............................................................. 46 11 Recommended Reading ....................................................................................... 47 12 Appendix ............................................................................................................... 48

12.1 Glossary of Technical Terms................................................................................... 48 12.2 Skeleton Test Report ............................................................................................... 51

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 7 of 51


1 Introduction 1.1 Objective Conduct a Review of Current Practices for System Maintenance in order to create a path to Best Practices for System Maintenance in a SYSPLEX environment.

1.2 Scope • All Huntington National Bank SYSPLEXES at primary site are in scope:

HNBSPLEX HNBTPLEX HNBXPLEX

• With consideration for the following environments:

z/OS Maintenance Environment SYSPLEX Test Environment Subsystems (DB2, CICS, MQ) ISVs

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 8 of 51


2 Management Summary This section will be completed at the end of the evaluation.

2.1 Analysis Summary

2.2 Recommendations Summary The recommendations included in this document are being introduced to Huntington National Bank (HNB), following discussions with members of their technical teams, where the current processes were analyzed and discussed.

There are several ways to install and maintain z/OS systems and the final process depends on the requirements of the business and technical departments. The two key design points are: • Flexibility – Whether a flexible process is required. • Resiliency – Whether a resilient process is required.

From the discussions, it was understood that the key factors are:

• Easy to understand • Easy to maintain • Emphasis on better testing

The benefits of a flexible process were discussed with the technical teams, where system symbolics were used, in conjunction with Extended Alias Support (Symbolic Relate) and the IEASYMUP utility.

In general this process was not well received, because of its complexity and there is a reluctance to use IEASYMUP (a use at your own risk, utility provided by IBM) because it is not fully supported by IBM (Note that z/OS 2.1 introduces the “SETLOAD xx, IEASYM” command).

With all of the above considerations taken into account, the process described in this document is designed to provide a resilient (resiliency) environment for HNB. This resiliency should be seen as the first step on the journey for HNB towards the typical challenging target of "five nines" or 99.999%, which provides an average annual down time of five minutes.

One of the key items mentioned by HNB was the lack of suitable testing for software installation and maintenance, before software is deployed to the SPLEX.

This document, and its associated processes, will provide the required levels of testing with all software (z/OS, Program Products and ISV) being tested as a single unit to confirm interoperability.

The new process provides a valid testing environment for the System Programming teams while allowing the Development and Production systems to be left intact for the purpose for which they are designed.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 9 of 51


Many applications currently use Data Sharing and there is a plan for further applications to exploit this feature. For this reason, an IPL of a single LPAR in a sysplex should not cause a problem due to application portability and the ability to remain available. This new process will require a single LPAR to be IPL’d but this will be a quick shutdown and IPL without the need to perform tasks such as copying datasets during shutdown.

It should also be noted that the requirement for a sysplex-wide IPL should be a very rare occurrence meaning that applications that utilize DB2 data sharing and CICS CPSM functionality would remain constantly available.

Both z/OS and ISV software will be implemented using a pair of ‘SYSRES’ volumes (1x z/OS, 1x ISV) and will, by the nature of the process, require planning of ISV software deployments to minimize the number IPL’s required during a calendar year.

This process creates a stable environment where all products are tested together, removing issues that can be encountered, where a new ISV product is introduced in isolation, risking incompatibilities with existing software.

To summarize, the new process will provide:

• Easily understandable processes. • Reduced complexity, therefore reducing the possibility of errors. • Full product testing; where z/OS, Program Products and ISV Products have been

tested together throughout the deployment phase. • IPL’s of single LPARs only; removing the need for sysplex wide IPL’s. • IPL’s for implementation and back-out with zero intervention and minimum timescale. • Continuous application availability when DB2 Data Sharing and CICS CPSM

functionality is used.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 10 of 51


3 Naming Standards 3.1 Naming Standards Analysis

3.1.1 z/OS The following rules apply to z/OS datasets:

• All z/OS datasets are named SYS1.* • All HFS and zFS datasets relating to z/OS are named SYS1.OMVS.* • The ServerPac dialogs are used to change the dataset names to SYS1.*. • None of these datasets is DFSMS managed.

Note: USS datasets exist for other products with an HLI of SYS5.* and SYS6.*. Also, several ‘Huntington Application’ USS Datasets seem to be randomly named with the SYS1.* HLI).

3.1.2 ISV ISV products use two HLI’s:

• SYS3.* - Used for all LINKLIST and LPA datasets. SYS3 data sets are catalogued in the master catalogue but note that LPA/LINKLIST data sets can now be catalogued in user catalogues as long as they have their VOLSER coded.

• SYS5.* - Used for all other datasets.

3.1.3 DB2 There is a Huntington document where DB2 standards are well documented.

• The dataset naming convention is SYS6.DSN&ssid.* • Aliases with SymbolicRelate are utilized.

- A set of Aliases for each Data Sharing Group (DSG) and a set for the standalone subsystem have been defined.

- The naming convention is SYS6.dsg.*

Note: All DB2 SMP/e datasets have a HLI of SYS9.DB2.*

3.1.4 CICS The CICS datasets are named:

• SYS6.CICSTZx.&SYSPLEX.SDFH* where “x” – Initially was environment; but now cycles round the values as each set of maintenance is applied.

3.1.5 MQ MQ is received as part of z/OS so all datasets follow the z/OS standard.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 11 of 51


3.1.6 ChangeMan ChangeMan managed datasets:

• CGP.* - SPLEX • CGQ.* - TPLEX Exceptions:

• SYS2.CHGMAN.PROD.PROCLIB • SYS2.CHGMAN.PROD.INCLUDE • SYS2.CHGMAN.TEMP.PROCLIB • SYS2.CHGMAN.TEMP.INCLUDE

The above four datasets are in the JES2 concatenations and are on the SYxMCT volumes so are not affected by the Cloning process.

All libraries for the ChangeMan product itself have a HLI of SYS5.CHGMAN.*.

3.1.7 USS Filesystems All HFS and zFS datasets relating to z/OS are named SYS1.OMVS.*

USS datasets for other products have a HLI of SYS5.* and SYS6.*.

Several ‘Huntington Application’ USS Datasets seem to be randomly named with the SYS1.* HLI.

3.2 Naming Standards Recommendations The naming standard recommendations made throughout this document are made so that:

• Any person who does not have specific Huntington National Bank knowledge (temporary workers, software vendor support staff, new-hires) can easily maintain systems.

• ISV software can integrate with z/OS with minimal change (e.g., when specifying LE Libraries).

Note: At the request of Huntington National Bank, where possible, new standards have been suggested that differ to the current standards so that there is no confusion during the implementation phase as to which datasets follow the new and old standards.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 12 of 51


3.2.1 DASD VOLSER Use SRvpnn e.g. SRXC03 SYSRES where:

P – Sysplex indicator (X,T,S) V – z/OS version (C for 1.12, 1 for 2.01) nn – Level Set (3 for 3rd build)

ISVpnn e.g. ISVX03 ISV SYSRES P – Sysplex indicator (X,T,S) nn – Level Set (3 for 3rd build)

ZSnnnn z/OS Base install and maintenance volumes IVnnnn ISV Base install and maintenance volumes

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 13 of 51


4 Catalog Usage Each SYSPLEX has a master catalog that is shared across all LPARs in the SYSPLEX. The master catalogs:

• CAT.ICF.HNBSPLEX.MASTER • CAT.ICF.HNBTPLEX.MASTER • CAT.ICF.HNBXPLEX.MASTER

4.1 Catalog Usage Analysis

4.1.1 General There are various data sets cataloged in the master catalogs that can reside in user catalogs:

• Data sets with HLQs for which there is no alias defined i.e., HB00917. • Data sets in LPALSTxx and PROGxx (LNKLSTxx) that can now be defined in user

catalogs, provided a volSer is coded in their LPALSTxx and PROGxx (LNKLSTxx) entries i.e., SYS1.BOOLE.&SYSPLEX..BBLINK, SYS1.CICTZP.&SYSNAME..SDFHLPA etc.

• ISV product data sets i.e., AbendAid, CBT, Easy+ etc. • Other data sets i.e., IODFs, Unix System Services zFS, SMP/E data sets etc.

4.1.2 Shared Catalogs The following table lists the User Catalogs shared across all three SYSPLEXes:

Catalog Name

VOLSER

HNBSPLEX # aliases

HNBTPLEX # aliases

HNBXPLEX # aliases

CAT.ICF.MOVES S7A104 142 142 1 CAT.ICF.MOVET S7A102 1 1 1 CAT.ICF.MOVEX S7A37A 1 1 1

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 14 of 51


4.1.3 Unshared Catalogs The following table lists the User Catalogs unique to each SYSPLEX:

Catalog Name

VOLSER

HNBSPLEX # aliases

HNBTPLEX # aliases

HNBXPLEX # aliases

CAT.ICF.DB2 SYS003 15 CAT.ICF.DB2Q# SYS050 6 CAT.ICF.PROD SYSDRP 179 CAT.ICF.SMPE SYS009 2 CAT.ICF.SOFT SYSMCT 5 CAT.ICF.USER0 S7A505 1 CAT.ICF.USER1 SYS055 1500 CAT.ICF.USER2 SYS052 1773 CAT.ICF.Y2K SYS009 2 CAT.ICF.DB2D SYT008 1 CAT.ICF.DB2T SYT008 13 CAT.ICF.PRODT S9B123 177 CAT.ICF.SMPET SYT005 1 CAT.ICF.SOFTT SYTMCT 5 CAT.ICF.USER1T SYT053 1138 CAT.ICF.USER2T SYT009 1537 CAT.ICF.DB2X SYX009 5 CAT.ICF.HNBXPLE7.MASTER SYXMCT 0 CAT.ICF.PRODX SYXDRP 175 CAT.ICF.SMPEX SYXMCT 3 CAT.ICF.SOFTX SYXMCT 6 CAT.ICF.USER1X SYX053 1192 CAT.ICF.USER2X SYX009 1360 ZOS111.ICF.SMPE SYXRSW 0 ZOS111.MASTER.CAT SYXRSW 0

Note: Tape library catalogs, i.e., CAT.VOLCAT.Vxxxxxxx, are excluded from the above table, as they are ‘special’ cases that do not have aliases related to them.

4.2 Catalog Usage Recommendations

Each SYSPLEX should have its own Master Catalog. Only the datasets on the SYSRES should be cataloged in the Master Catalog. All other datasets should be cataloged in additional catalogs. This method will simplify future upgrades to z/OS.

All dataset entries in the Master Catalog should be specified with VOLUME(******).

ISV software should be cataloged in its own catalog. All dataset entries in the ISV Catalog should be specified with VOLUME(&ISVRES.)

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 15 of 51


5 System Symbolics 5.1 System Symbolics Analysis Symbolics are mainly used by DB2 and CICS as described in the Naming Standards section.

For DB2, a set of aliases with SymbolicRelate are utilized. A set of Aliases for each Data Sharing Group (DSG) and a set for the standalone subsystem have been defined.

The CICS procedures utilize JCL and System symbolics to specify the datasets to use. The following is an example:

DSN=&PREFIX..&SYSPLEX..SDFHLOAD and PREFIX='SYS6.CICTZN'. The &PREFIX variable is specified in the Procedure but could be overridden in the started task JCL.

5.2 System Symbolics Recommendations Using the recommended process, there will be only one extra System Symbolic:

Symbolic Usage &ISVRES VOLSER of current ISV SYSRES. This is automatically created from the

&SYSR1 symbolic i.e., SYMDEF(&ISVRES='ISV&SYSR1(4:3).') Note: All existing Symbolics (e.g., those used by DB2 for the DSG specifications) will continue bring used.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 16 of 51


6 Installation and Maintenance 6.1 Installation and Maintenance Analysis Maintenance for the major software components varies for each one (even for those installed by the same team). This also leads to rollout schedules that may vary for each new Maintenance level. There is a stated intention to work towards bringing some of these in line with other software products.

6.1.1 z/OS There is only one SMP/e environment for z/OS and IBM Products. Maintenance is applied to this environment and then a set of SYSRES volumes are created (‘RS’ or ‘AL’ set) – these also include the necessary HFS and zFS datasets.

RSU’s are applied quarterly. They are applied to the SMP/e environment that was created by the ServerPac order.

Currently FIXCATs are not used but are on the list of things to do.

6.1.2 ISV Products ISVs are managed on a one-to-one basis in that each one is installed and upgraded on its own merits, generally using the vendor prescribed methods.

All installations are now done on the XPLEX but until recently were done on the SPLEX. This means that some existing software is still installed on the SPLEX but all new installations are being moved to the XPLEX.

ISV Products are rolled out using a cloning process as described below. z/OS, CICS and DB2 do not use this process. ChangeMan utilizes the Cloning process to copy the data but does not implement during the process.

The Cloning process runs once a week and is a set of jobs that does the following:

• Closes all LPARS in the TPLEX. • Copies the entire SYS Storage group from SPLEX to TPLEX i.e. all SYS3.* datasets

that contain the runtimes. • IPLs all LPARS in the TPLEX using the new copies. • Closes all LPARS in the XPLEX. • Copies the entire SYS Storage group from SPLEX to XPLEX i.e. all SYS3.* datasets

that contain the runtimes. • IPLs all LPARS in the XPLEX using the new copies.

On occasion, it is required to run different levels of software in the “production” LPARs and “App Dev” LPARs on SPLEX. To do this, PARMLIB members are changed for the relevant LPARs to specify using the TPLEX version of the Dataset via the use of Symbolics.

For TPLEX and XPLEX, before the Cloning takes place, a Backup is taken of the #SYS and $SYS Storage Groups. This backup can be used for a complete Cloning Back-Out if

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 17 of 51


necessary. This method backs out all software implemented in a Cloning Cycle. The Back-Out process will be a repeat of Deployment, where the TPLEX or SPLEX is closed and the #SYS or $SYS Storage Group is refreshed before the LPARs are IPL’d. A preferable Back-Out is to maintain multiple versions of the software under HNBSPLEX, HNBTPLEX and HNBXPLEX dataset names. That way Back-Out can be accomplished without system outage by changing aliases/PROCS/PARMLIB to use the SYSPLEX named Datasets on the previous software version.

For SPLEX, the reverse of the Deployment Procedure is used so that all but one LPARs are closed, and then Manual Copies are performed once all ENQ’s are removed. The LPARs are then IPL’d.

6.1.3 DB2 It is planned to do RSU Maintenance is applied twice per year and if this could be synchronized with the z/OS RSU then that would be better. Approximately 6 weeks after the maintenance has been rolled out to all environments, it is accepted and a new SMP/e zone is created.

There will be one SMPE environment for all DB2s across HNB SMPE environment residing on volumes that are replicated to DR

In the DB2 V10 SMPE environment, there will be a target zone for each unique DB2 application maintenance level. We anticipate using three target zones in off release years.

• SMPE.DB2.DSN910.TRSU1304.SD* • SMPE.DB2.DSN910.TRSU1308.SD* • SMPE.DB2.DSN910.TRSU1312.SD*

The process of applying DB2 maintenance has been designed to have a standard set of Execution Datasets per DB2 environment that will continue to be reused over time.

Execution Datasets will be reused from Maintenance Cycle to Maintenance Cycle and DB2 Release to DB2 Release.

High-level DB2 maintenance process steps:

• Move DB2 SMPE target datasets to the LPAR in which DB2 maintenance will be applied.

• Remove traffic from DB2. • Development – cancel threads. • Production – Drain initiators, stop batch and quiesce CICS and DDF threads. • Once all threads are removed from a given DB2, a Stop DB2 command is issued. • Verify DB2 has been stopped. Delete/Compress execution datasets (pds clear, pds

clean). For example: - Production - YS6.DSNPDB2.HNB1.SDSNLOAD - Development - SYS6.DSNDB2.HNB2.SDSNLOAD

• Use IEBCOPY to copy contents of SMPE target zone to execution datasets • LLA Refresh ** Only on DB2s that have datasets in Linklist **. • Start DB2.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 18 of 51


• Execute a series of IVP jobs/binds to verify basic DB2 functionality is available and functioning.

6.1.4 CICS Maintenance is currently applied on an ad-hoc basis although that is because CICS v5.1 is scheduled for release, and maintenance would probably be applied once this was installed.

6.1.5 MQ Maintenance is applied to MQ at the same time as z/OS unless specific PTFs are required. The z/OS team does this. The MQ team does all the customization.

The team would like to start applying maintenance 3 times per year in line with the distributed MQ environments.

6.1.6 ChangeMan To maintain ChangeMan, there is another copy on SPLEX (HNB7 but could be any LPAR), CHGMANT. This is because the standard way to update ChangeMan is to use ChangeMan itself – what is known as ‘ChangeMan on ChangeMan’.

Note: This process is performed as defined by the Vendor.

6.2 Installation and Maintenance Recommendations

6.2.1 z/OS Installation z/OS should be ordered, installed and maintained using the following processes:

• z/OS should be ordered and installed via the ServerPac process. • All dataset names should be kept as the default specified by IBM. • Use the default device type when in the ServerPac dialog panel “Automatic Data Set

Assignment”. This is currently 3390-9 but could change in future. • Use the ServerPac dialogs to append a new HLQ to every dataset of ZOSvvv where

‘vvv’ is the version. For example, SYS1.LPALIB will be called ZOS201.SYS1.LPALIB for z/OS v2.01.

• Use the ServerPac dialogs to append ‘/Service’ to all USS Pathnames. • Once downloaded, the resulting volumes will become the new base install and

maintenance volumes where all SMP/e work is done and from where the new SYSRES volumes will be created. This is called the “z/OS Master Level Set”.

• All SMP/e DDDEFs should point to the cataloged ZOSvvv.* datasets and will not have VOLSERs coded.

• IBM supplied PARMLIBs and PROCLIBs i.e. ZOS201.SYS1.PARMLIB should not be changed.

• Several extra datasets will need to be manually created on the z/OS Master Level Set to support this process. All PARMLIB customizations are listed below.

ZOSvvv.SYS1.XPLEX.PARMLIB PARMLIB members specific to XPLEX ZOSvvv.SYS1.TPLEX.PARMLIB PARMLIB members specific to TPLEX ZOSvvv.SYS1.SPLEX.PARMLIB PARMLIB members specific to SPLEX

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 19 of 51


ZOSvvv.SYS1.XPLEX.PROCLIB PROCLIB members specific to XPLEX ZOSvvv.SYS1.TPLEX.PROCLIB PROCLIB members specific to TPLEX ZOSvvv.SYS1.SPLEX.PROCLIB PROCLIB members specific to SPLEX ZOSvvv.SYS1.XPLEX.LINKLIB LINKLIB members specific to XPLEX ZOSvvv.SYS1.TPLEX.LINKLIB LINKLIB members specific to TPLEX ZOSvvv.SYS1.SPLEX.LINKLIB LINKLIB members specific to SPLEX ZOSvvv.BUILD.JCL Library for z/OS Level Set build jobs ZOSvvv.MAINT.JCL Library for z/OS Level Set maintenance jobs

6.2.2 z/OS Maintenance All z/OS maintenance should be applied on the XPLEX and against the z/OS Master Level Set i.e., the datasets named ZOSvvv.*

Note: FIXCATs should be used where possible to identify all potential requirements when installing new hardware and software. See the section on Software Refresh and Maintenance Cycles for more information.

All SMP/e APPLY jobs should be kept and named with the corresponding Level Set number. Use the following dataset naming convention:

Dataset Usage ZOSvvv.MAINT.JCL(LSnnREC) SMP/e RECEIVE for Level Set nn ZOSvvv.MAINT.JCL(LSnnAPK) SMP/e APPLY CHECK for Level Set nn ZOSvvv.MAINT.JCL(LSnnAPP) SMP/e APPLY for Level Set nn ZOSvvv.MAINT.JCL(LSnnACK) SMP/e ACCEPT CHECK for Level Set nn ZOSvvv.MAINT.JCL(LSnnACC) SMP/e ACCEPT for Level Set nn If multiple RECEIVE and APPLY jobs are run for a single Level Set then use the following naming convention: Dataset Usage ZOSvvv.MAINT.JCL(LSnnxREC) SMP/e RECEIVE for Level Set nn and

subset x Note: This will ensure that all applicable maintenance is SMP/e ACCEPTed at the time of Level Set Housekeeping – refer to the section on Level Set Housekeeping.

6.2.3 USS SYSPLEX sharing should be used, and the BPXPRMxx VERSION() statement should be used with a SYSMBOLIC.

Note: HNB uses this today and has VERSION('&SYSR1.') specified.

The SYSPLEX Root should be mounted R/W so that new Version Roots are able to automatically create the required mountpoint.

6.2.4 MQ MQ should be installed as part of z/OS.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 20 of 51


6.2.5 ISV Installation (Including ChangeMan) ISV software will be installed and maintained using the same processes detailed for z/OS. There will be a base set of install libraries and volumes where all software is installed – the ISV Master Level Set. As with z/OS, creating a ‘SYSRES’ then performing an IPL will be the method for implementing the ISV software.

• All ISV software should be installed using the vendor recommended processes, where the vendor gives the option for an SMP/e install or non-SMP/e install, and then the SMP/e install should be used. Reasons for using the Vendor processes include: o Simplicity: The vendor process may seem unwieldy but trying to deviate from it is

often more difficult. o Less error prone: Deviating from the official process could result in problems. o Consistent and documented: Each install of the product should be similar and

changes will be documented. o Known: Anyone (temporary workers, software vendor support staff, new-hires)

can install using vendor documented processes if they have not been changed.

• The default dataset names, as supplied by ISV’s, should be used but with an HLQ of ‘ISVSOFT’ appended. Often, the ISV will only specify the LLQ (Low-level qualifier). Where only the LLQ is specified then use:

ISVSOFT.<product name>. <version>. <llq>

Where:

o <product name> is a unique name for each product. o <version> is the version, generally in the form VxRxMx or PUTyymm. o <llq> is the Low Level Qualifier as specified by the vendor.

For example:

o ISVSOFT.FDR5478.LOAD (Vendor distributes as FDR5478.LOAD) o ISVSOFT.CAI.V11R6M0.CAILIB (Vendor distributes as

CAI.V11R6M0.CAILIB) o ISVSOFT.MODEQ.V6R4.LOADLIB (Vendor distributed v6.4 as only

LOADLIB)

• The following rules must be followed for naming ISV datasets:

o ISVSOFT HLQ o Unique product name o Version (might be incorporated with product name) o LLQ as specified by vendor

• All ISV software should be installed onto DASD Volumes with the names ISVnnn.

• All SMP/e DDDEFs should point to the cataloged ISVSOFT.* datasets so do not have VOLSERs coded.

• Vendor supplied parameter datasets should not be changed.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 21 of 51


• Several extra datasets will need to be manually created on the ISV Master Level Set to support this process. These datasets will depend on the ISV Product Set and the amount of customization required. Examples are listed below:

ISVSOFT.FDR5478.XPLEX.PARMLIB PARMLIB members specific to XPLEX ISVSOFT.MODEQ.V6R4.HNB.PARM PARMLIB members for all HNB ISVSOFT.BUILD.JCL Library for ISV Level Set build jobs ISVSOFT.<product/version>.MAINT.JCL Library for ISV Level Set maintenance jobs (specific

for each product)

6.2.6 ISV Maintenance (Including ChangeMan) All ISV maintenance should be carried out on the XPLEX and against the ISV Master Level Set i.e. the datasets named ISVSOFT.*

Note: FIXCATs should be used where possible to identify all potential requirements when installing new hardware and software. Several ISV’s are now supporting FIXCAT.

All SMP/e APPLY jobs should be kept and named with the corresponding Level Set number. This will ensure that all applicable maintenance is SMP/e ACCEPTed at the time of Level Set Housekeeping – see the section below.

Each product should have its own maintenance library – suffix the ISVSOFT.<product> datasets with MAINT.JCL as outlined in the Installation and Maintenance section.

6.2.7 Level-Set Housekeeping The creation of a new Level Set creates a new pair of SYSRES volumes with an n+1 sequence number. Generally there will be 1 or 2 Level Sets being used and in exceptional cases 3 Level Sets used.

Once a Level Set is implemented in all LPARs in all SYSPLEXes, the n-2 Level Set is then eligible for deletion. Note that the n-1 Level Set should be kept in case back out is required. For example, if Level Set 06 is implemented in all LPARs, then Level Set 05 should be kept for back out and Level Set 04 should be deleted.

To delete a Level Set, perform the following tasks:

• SMP/e ACCEPT all maintenance that was part of the Level Set. • Initialize the SYSRES volumes (z/OS and ISV) so that they can be re-used. Previous maintenance is accepted to ensure that any APARS/PTF’s that cause issues, or are flagged as being in error (IBM terms “PE”), can be SMP/e RESTOREd (removed) with the minimum amount of effort and impact on other maintenance.

6.2.8 DB2 Installation DB2 should be ordered, installed and maintained using the following processes: • Using ShopZ, new levels of DB2 maintenance should be ordered and matched to the

z/OS ServicePac as part of an IBM CST release, synchronizing the code base of key subsystems to known compatible levels (i.e. CICS to DB2).

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 22 of 51


• Individual site-sensitive HIPERS and PEs should be reviewed, applied or removed as per site requirements.

• ServicePac installs should happen twice annually with monthly reviews of HIPERS to be installed on an ad-hoc basis should there be a business requirement.

• The ServicePac should be installed on HNBXPLEX to target zones with a meaningful level and RSU information and using IBM default dataset names.

• Use the following naming standards for all datasets DSNnnn.RSUrrrra.* where ‘nnn’ is the DB2 release (i.e. 910 or A10) and ‘rrrra’ is the current RSU of system i.e. RSU1206 plus adhoc fix level “a”, for example:

E.g. DSNA10.RSU1206A.SDSNLOAD • Increment the adhoc fix level “A” (B,C,D) when applying site specific fixes and fresh

HIPERS making it easy to identify the latest version.

6.2.9 DB2 Maintenance All DB2 maintenance should be applied using SMP/e on the XPLEX. See the section on Software Refresh and Maintenance Cycles for more information regarding DB2 Maintenance.

6.2.9.1 New Release Installation

Each new release of DB2 usually has significant changes to the installation and migration procedures from the previous release.

The following steps should be performed based on previous versions of DB2:

• Build new release distribution libraries from SMP/e, accept IBM defaults for system libraries names.

• Depending on whether IMS is used at the customer site, consideration must be given to the sharing of SMP/e environments or a separate generation of them as they both generate the IRLM subsystem.

• Tailor and run the DSNTIJXZ job to tailor input libraries setting the defaults to match the current DB2 installation.

• Run DB2 install ISPF panels (DSNINST1 CLIST) working through parameter panels, these should match the current setup thanks to the previous DSNTIJXZ job.

• Review new release documentation to ensure table, buffer pool, EDM pool etc. and that sizings take in to consideration version related changes.

• Particular attention must be paid to new or obsolete zPARMs, or those whose default value may have changed.

• Run DSNTZAZP job to import the current bufferpool setup. RRS is required. • Be aware of subsystem parameters that are set via job DSNTIJUZ – not the

installation CLIST. • Use the MIGRATE option on the new installation panels. • After working through the panels relating to runtime, z/OS, application defaults

etc., only a job called DSNTIJUZ will be left. • Review and amend the panels as required before running DSNTIJUZ which will

generate the runtime libraries for the new DB2 installation.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 23 of 51


Note: Please be advised, the above steps are a brief approximation of roughly 500 pages worth of DB2 installation material and are only a guide as to the sequence of event required to generate a new DB2 system.

6.2.10 CICS Installation CICS should be ordered and installed using the following procedures:

• Using ShopZ, new levels of CICS maintenance should be ordered and matched to z/OS ServicePac as part of an IBM CST release, synchronizing the code base of key subsystems to known compatible levels (i.e., CICS to DB2).

• Individual site-sensitive HIPERS and PEs should be reviewed, applied or removed as per site requirements.

• ServicePac installs should happen twice annually with monthly reviews of HIPERS to be installed on an ad-hoc basis should there be a business requirement.

• The ServicePac should be installed on HNBXPLEX to target zones with a meaningful level and RSU information and using IBM default dataset names.

• Use the following naming standards for all datasets DFHnnn.RSUrrrra.* where ‘nnn’ is the CICS release (i.e. 420 or 510) and ‘rrrra’ is the current RSU of system (RSU1206 plus adhoc fix level “a”). Example:

DFH510.RSU1206A.SDFHLINK • Increment the adhoc fix level “a” (i.e. B,C,D) when applying site specific fixes and

fresh HIPERS making it easy to identify the latest version.

6.2.11 CICS Maintenance All CICS maintenance should be applied on the XPLEX. See the section on Software Refresh and Maintenance Cycles for more information regarding CICS Maintenance.

6.2.11.1 New Release Installation

Each new release of CICS may have installation and migration procedures from the previous release.

Initial steps include:

• Generate the new runtime libraries using the standard ServicePac process on HNBXPLEX. The ServicePac installation dialogs should have already been installed.

• Enter various options relating to where the system libraries will be installed and accept IBM defaults where possible.

The following list includes some of the steps required, covering many hundreds of detailed steps outlined in the CICS TS installation manual:

• Keep a backup of the generated untouched generated libraries. • Tailor the new procedure libraries in line with site requirements. • CICS v5.1 brings a new feature, the ability to dynamically install the CICS TYPE 3

supervisor call (SVC) using the new DFHCSVCU utility, this avoids having to IPL z/OS for future releases of CICS TS.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 24 of 51


• Authorize required CICS TS libraries. • Add required new release libraries to Linklist. • Create new release CSD(s) (migrate a copy of the old one(s)). • Batch edit any release- dependent CSD group lists to required levels. • Define new release CICS local and global catalogs. • Update security product with any new profiles required. • Start and test a generic new region before attempting to deploy in an active system.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 25 of 51


7 Deployment 7.1 Deployment Analysis How each of the following products is deployed varies. The Cloning process is referenced in the Maintenance section.

7.1.1 z/OS Maintenance and new versions are rolled out to each LPAR via an IPL using a new set of SYSRES volumes. This approach is generally used even if Maintenance could be applied dynamically.

Each Sysplex has two sets of SYSRES volumes – an ‘RS’ set and an ‘AL’ set which are toggled. A Spreadsheet is used to record which SYSRES set is currently in use and at what level of z/OS it contains.

7.1.2 CICS When rolling out new maintenance, a couple of the Test CICS region’s PROCs are edited to refer to the new libraries. Once all regions are tested, the relevant Procedure is changed so that all regions pick up the new libraries. This is done outside of IPL’s.

Each CICS region is named entttCS where:

• e – Environment (‘P’ Prod, ‘U’ QA, ‘T’, Test, ‘X’ XPlex) • n – number • ttt – type of region

These Call Procedures CICSP (Prod), CICST (Test), or CICSU (QA) depend on the environment. Note: TPLEX is a clone of the SPLEX AD environment (HNB2, HNB5) so the procedure is also CICST.

The procedures utilize JCL and System symbolics to specify the datasets to use the following example:

DSN=&PREFIX..&SYSPLEX..SDFHLOAD and PREFIX='SYS6.CICTZN'.

Note: The &PREFIX variable is specified in the Procedure but could be overridden in the started task JCL.

7.1.3 MQ MQ was, until recently, deployed with z/OS but due to issues with v7.1, the z/OS upgrade it was ‘decoupled’, and the upgrade to MQ to v7.1 was implemented before the z/OS upgrade. The SMP/e environment stays shared with z/OS.

The number of queue managers for each environment should include:

• 2 in Production (SPLEX)- MQP1 & MQP2

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 26 of 51


• 2 in QA (SPLEX) - MQU1 & MQU2 • 1 in Training (in Production environment - SPLEX) – MQE1 • 2 in Sandbox (XPLEX) – MQA1 & MQA2 • 2 in Test (XPLEX) – MQT1 & MQT2.

Training and Production regions (both SPLEX) are generally updated at the same time. If these regions were updated separately, the Procedure would need to be edited.

7.1.4 DB2 Each DB2 has its own version of SDSNLOAD and SDSNEXIT. This means that each DB2 can be closed individually for Maintenance. With Datasharing implemented, this means that DB2 remains available during this time. To make the changes, once DB2 is closed, the libraries are replaced with new versions (delete members, compress, copy new members) before DB2 is restarted. The time to do the copy process (30 seconds) is considered acceptable since the other member of the Datasharing group would be available. Note: The ISV cloning process is not used for DB2. This is because the Cloning process has caused issues in the past where maintenance was thought to be on DB2 but had been ‘lost’.

7.1.5 ISV Some products are SPLEX only e.g. CTG and Dynatrace. These products will be installed on XPLEX, and then moved straight to SPLEX, where they are tested and deployed. The Cloning process is not used for these products, as they are not used in TPLEX.

BetterGener (Syncsort) is an exception because it is installed on the Sysres using the z/OS installation process.

Note: Deploying with ‘A’ and ‘B’ versions and using a symbolic to switch between them is being considered.

7.1.6 ChangeMan The cloning process is used to copy the new libraries from CHGMANT (SPLEX) to CHGMANA (TPLEX) but this is for convenience since the process is already there.

Panel and Skeleton changes happen immediately; but Exits need a bounce of ChangeMan. If changes are required before the Cloning process runs then manual copies will be done.

For new releases of ChangeMan, the vendor states that the ‘P’ sites have to be updated first or all sites at the same time.

ChangeMan Deployment Issues This section describes issues related to deployment of other software but have an impact on ChangeMan.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 27 of 51


ChangeMan needs to point to old versions of DB2 until all instances in every sysplex have been updated. Recently some Alias’ have been added for ChangeMan to use during DB2 upgrades to resolve this problem.

COBOL & z/OS upgrade caused some compatibility issues in SPLEX. This was found in SPLEX but on the test LPARs. However steps had to be put in place to stop new code being promoted to production SPLEX while the new APAR was installed. This might have been noticed earlier if there had been a test ChangeMan on XPLEX and promoting to TPLEX rather than the first promotions going straight to SPLEX.

Easytreive update stated that compiles on the old version would run on the new but there were no compatibility fixes to enable compiles on the new version to run on the old. Special jobs were created which STEPLIB’d to the new version but implementation needed to be done at the same time on both TPLEX and SPLEX.

7.2 Deployment Recommendations

7.2.1 Level Set Deployment Both z/OS and ISV software is deployed as a Level Set. This comprises of a pair of SYSRES volumes, one for z/OS and one for ISV (although this process could be expanded for multiple ISV SYSRES volumes if required).

The advantages of deploying z/OS and ISV in a level set include:

• All levels of all products are tested and deployed at the same time. • Ease of implementation and back out.

The process for creating the Level Set depends on whether maintenance has been applied to z/OS only, ISV only or both as illustrated below:

ISV Maintenance No ISV Maintenance z/OS Maintenance Create z/OS SYSRES from

Master Create ISV SYSRES from Master

Create z/OS SYSRES from Master Create ISV SYSRES from current

No z/OS Maintenance

Create z/OS SYSRES from current Create ISV SYSRES from Master

N/A

In cases where no maintenance has been applied, a new SYSRES is created by copying the current (most recently built) SYSRES and then relabeling it with the new Level Set. In this situation, do not create a new SYSRES from the Master Level Set because application of maintenance may be in progress.

If copying the current z/OS SYSRES, as well as relabeling the VOLSER with the new level set, the zFS root filesystem will need to be renamed with the new Level Set SYSRES VOLSER as the 2nd level qualifier. E.g. ZFS.SRCT04.ROOT

See the sections on z/OS Level Set Build and ISV Level Set Build on how to build the SYSRES from the Master.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 28 of 51


Implementing the new z/OS maintenance is now simply a matter of performing an IPL from the new z/OS Level Set SYSRES. Should problems be encountered and back out is required then IPL from the old SYSRES.

If Back-Out is performed, and maintenance is required to fix the problem, then a new Level Set is created. Before implementing the new Level Set, the LPARs where the faulty Level Set was applied should all be rolled back to the previous version. The following table illustrates sample implementation data:

LPARs Level Set Roll back New Level Set being deployed

New Level Set Complete

HNB6 16 15 17 17 HNB8 16 15 17 17 HNB3 16 (faulty) 15 15 17 HNB4 15 15 15 17 HNB2 15 15 15 17 HNB5 15 15 15 17 HNB1 15 15 15 17 HNB7 15 15 15 17

If the faulty Level Set is not rolled back on all LPARs, then the implementation from Level Set 15 to Level Set 17 (in the above example), is not being tested or failing to test, has been known to cause problems.

7.2.2 z/OS Level Set Build Each build of a z/OS Level Set SYSRES requires that a new build job be created in the dataset ZOSvvv.BUILD.JCL. Creating a new job rather than amending the old job used for the previous level set is required for two reasons:

1. An audit record of how each Level Set was built. 2. During the upgrade to a new version of z/OS, the old job may still be required to build

a back-level SYSRES for any fixes required before the new version is fully deployed.

The new SYSRES volume should be created on a 3390-27 type device. Using a single volume reduces the effort and chance of errors when:

• Maintaining SYSRES volumes. • Maintaining volume entries for datasets in PARMLIB. • Maintaining catalog entries for datasets on the SYSRES. • Maintaining the zFS Root which is now approaching the size of an entire 3390-3.

The z/OS Level Set SYSRES build process copies the required ZOSvvv.* datasets from the Master Level Set volumes onto a new SYSRES volume and renames them by removing the ZOSvvv HLQ. This results in standard dataset names e.g. SYS1.LPALIB. Only datasets needed for running the systems are copied to the SYSRES. This means that one SYSRES volume (3390-27) should suffice for all required z/OS and USS datasets. When creating the SYSRES, increase the VOLSER by one to indicate the new Level Set (e.g. SRX101, SRX102, SRX103, SRX104...). This also makes it much clearer which SYSRES is newer rather than toggling between just two volumes.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 29 of 51


The procedure for building a new z/OS Level Set SYSRES:

• Apply maintenance to the ZOSvvv.* datasets. • Compare current SYSRES PARMLIBs and PROCLIBs with the ZOSvvv.*

equivalents. This should not be necessary but just in case someone has performed a dynamic change to a live SYSRES.

• Build the z/OS and ISV Level Set SYSRES volumes. • IPL with the new z/OS Level Set SYSRES.

7.2.2.1 z/OS Creating the SYRES Volumes

Step 1 - Compare current SYSRES datasets with the ZOSvvv.* datasets An ISRSUPC job should be created to compare the following datasets on the live SYSRES:

• SYS1.PARMLIB with ZOSvvv.SYS1.PARMLIB • SYS1.PROCLIB with ZOSvvv.SYS1.PROCLIB • SYS1.<sysplex>.PARMLIB with ZOSvvv.SYS1.<sysplex>.PARMLIB • SYS1.<sysplex>.PROCLIB with ZOSvvv.SYS1.<sysplex>.PROCLIB • SYS1.<sysplex>.LINKLIB with ZOSvvv.SYS1.<sysplex>.LINKLIB

Make sure any changes to the live SYSRES datasets are reflected in the ZOSvvv.* datasets before running the build job; otherwise, they will be lost. Step 2 - Sysres build job A new job should be created to perform the following tasks:

• Checks that the proposed new SYSRES volume is not in use on any LPAR. • Initializes the proposed new SYSRES volume with the VOLSER corresponding to the

new Level Set. • Writes IPL text. • Copies all necessary ZOSvvv.* datasets and renames them by removing the HLQ.

Note: The copied datasets will be uncataloged. • Copies the zFS Root from ZOSvvv.ZFS.ROOT and rename it to

ZFS.<VOLSER>.ROOT where <VOLSER> is the VOLSER of the newly created SYSRES. See the following URL for hints on doing this using indirect volume serials: http://pic.dhe.ibm.com/infocenter/zos/v1r12/index.jsp?topic=%2Fcom.ibm.zos.r12.idac100%2Fdefzfs.htm

• For documentation purposes, this job creates a new dataset called SYS1.BUILD.INFO on the new SYSRES giving a date/time of when it was built and a build comment stating the reasons for build.

If this job is changed to incorporate extra datasets then the Master Catalogs will need to be updated with the catalog entries remembering that all entries should be specified VOLUME(******).

7.2.2.2 USS

Each Level Set will have a new version of the Version Root which will automatically mount at IPL.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 30 of 51


7.2.2.3 MQ

MQ should be deployed as part of the z/OS Level Set.

7.2.3 ISV Level Set Build (Including ChangeMan) The process for deploying ISV maintenance is almost identical to the z/OS process:

• All datasets created on the ISV ‘SYSRES’, ISVpnn volume, will be cataloged in the ISV catalog and with a symbolic VOLUME(&ISVRES.).

• The ISVSOFT HLQ should be removed. • Any versions in the dataset name should be removed. Example:

o ISVSOFT. FDR5478.LOAD should be copied as FDR.LOAD. o ISVSOFT.CAI.V11R6M0.CAILIB should be copied as CAI.CAILIB).

7.2.4 CICS

7.2.4.1 Prepositioning for Rolled Out Maintenance Deployment

Some tasks required before attempting to install a new version of CICS TS:

• Review CICS PSP (preventative Service Buckets) before attempting a CICS release upgrade. PSP: http://www14.software.ibm.com/webapp/set2/psearch/search?domain=psp

• It is recommended that CICPLEX SM (if used) is upgraded to the level of the new CICS subsystem if a mixed release environment is to be supported as part of the new release roll out and/or toleration maintenance is applied to the MASs The CMAS must be at the newer release. Failing to do so will cause unpredictable results, including possible system outage. With ServicePac installations, the CICSPlex SM is usually upgraded along with the new CICS release.

• Review and document and SIT (System Initialization Table) changes that may be required for the new release, some parameters may be obsolete, others may have new defaults that should be honoured to avoid problems when migrating to the new release. For example the default MXT and memlimits for V5.1 are increased Failure to observe these recommended changes may cause the CICS regions to perform badly or fail to start altogether.

• Check existing applications for use of deprecated CICS API (or SPI) commands, including COBOL levels no longer supported. Running a mixed release of CICS TS servers may cater for certain applications but this situation should be avoided if at all possible.

• The CICS Load Module scanner (DFHEISUP) may be run against existing application libraries to look for obsolete API commands after new release libraries have been built.

• Look for obsoleted JCICS calls if JAVA applications are currently run under CICS. • Check changes to resource definitions – some changes are required to align

definitions with new incoming bundle definitions, the IBM manuals need to be checked.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 31 of 51


• Review any automated commands issued to CICS with changed or obsoleted CEMT commands in mind, the automation logic may have to be made CICS-release sensitive if run in a mixed level environment.

• Ensure any new CICS system transactions have the required security profiles. • Review global and task related user exits for conformity with any documented

changes. • Review changes to EXCI (if this interface is used). • Contact vendors for required actions regarding ISV software.

Once the new release of CICS TS is running and tested on HNBXPLEX a strategy for test and production can be devised based on current application tolerance of the new release.

Individual groups of regions (DOR, and TORs are usually the easiest) may be upgraded ahead of the AOR, which requires application changes.

The ability to roll-in new CICS TS releases is completely application dependent and cannot be accurately predicted.

7.2.4.2 CICS Maintenance Deployment

Once the new release of CICS TS is installed on HNBXPLEX a strategy for deployment can be devised based on current application tolerance of the new release. Individual groups of regions (DOR, and TORs are usually the easiest) may be upgraded ahead of the AOR, which requires application changes. The maintenance install process should run as follows:

• Copy the SMP/e Target system libraries to suitable datasets on the correct LPAR. Use the following naming:

• DFH.<ibm-name>.RSUrrrra where ‘rrrr’ is the RSU and ‘a’ is an additional suffix to denote extra maintenance. The 2nd set of maintenance would be ‘B’, the 3rd ‘C’ etc. making it easy to identify the latest. E.g. DFH.SDFHLINK.RSU1206B

• For region specific datasets use DFH.<region>.<ibm-name>.RSUrrrra. E.g. DFH.DSNPD1G.SDFHLINK.RSU1206B

• APF Authorize the required libraries • To apply to a single CICS region then alter the &PREFIX and &PREFIX2 overrides in

the specific PROCs. • To apply to a group of CICS regions then alter the &PREFIX and &PREFIX2 in the

generic CICSx Procedure that is called by the region specific Proc. Note the following recommendations:

• It is advisable to roll in new RSU to CICSPLex SM regions first (if applicable). • The decision on a big bang (entire CICS footprint recycled) or rolled-in maintenance

will have been made based on experiences with test system rollout. • Rolling in maintenance is performed on a group of regions at a time within an LPAR

by changing the procedure called (to pick up new system symbolics) and the region or regions be recycled.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 32 of 51


7.2.5 DB2

7.2.5.1 Prepositioning for Rolled Out Maintenance Deployment

DB2 datasharing must be employed and tuned to ensure the production workload is viable with a datasharing member shutdown or a spare datasharing member should be defined which can be started up to takeover the workload when a permanent member is shutdown for maintenance.

For assured availability, a minimum of two LPARs (and member groups) per central processor complex (CPC) is required to provide basic failover and non-disruptive maintenance rollout. Due to the storage constraint relief offered by DBV10 onwards, this configuration is becoming the IBM recommended standard.

Keeping the data share number to the bare minimum to meet the above requirements will reduce Coupling facility traffic, reduce group-wide locks and improve performance.

Group DDF locations (via dynamic VIPA) should replace individual member names for off host applications and DB2 datasharing group names should be used instead of individual member subsystem names for traditional workloads - applications should not be aware of individual DB2 subsystems.

Each CPC should host a Coupling Facility LPAR (ICF) with the two system-managed, duplexed and capable of handling a failover load.

This standardized setup should be implemented across HNBXPLEX, HNBSPLEX and HNBTPLEX (The ICFs can of course be shared) although the number of DB2 share members does not need to match Production.

Active or DASD-backed archive logs should always be available to at least manage the last full image copies as relying on forward recovery from tape is no longer viable in the real world.

7.2.5.2 Non-Disruptive Deployment

DB2 maintenance will require a restart of individual member DB2 subsystems and/or the IRLM (Inter Region Locking Manager) address spaces.

There is a special kind of code refresh, which updates the code which starts DB2 itself. This is called an "Early code refresh". It is generally rarely required and is usually positional for later implementations of new releases of DB2.

The sequence following steps should be taken when updating an individual member of a DB2 share non-disruptively:

• Copy the SMP/e Target system libraries to suitable datasets on the correct LPAR. Use the following naming convention:

DSN.<ibm-name>.RSUrrrra where ‘rrrr’ is the RSU and ‘a’ is an additional suffix to denote extra maintenance. The 2nd set of maintenance would be ‘B’, the 3rd ‘C’ etc. making it easy to identify the latest (i.e. DSN.SDSNLOAD.RSU1206B).

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 33 of 51


• For subsystem specific datasets use DSN.<subsys>.<ibm-name>.RSUrrrra (e.g. DSN.DSNPD1G.SDSNEXIT.RSU1206B).

• APF Authorize the required libraries. • Shutdown the individual DB2 subsystem, specifying "CASTOUT NO" to avoid cross-

invalidating all group-buffered pages that the member has interest in which would cause a group-wide slowdown.

• Update relevant catalog Alias entries for the datasets. These entries can still incorporate the system symbolics for the DSG as currently employed by HNB but will now point to the correct RSU level. (e.g. DEF ALIAS(NAME(DSN.PDB2.SDSNLOAD) SYMBOLICRELATE(DSN.DSN&PDB2..SDSNLOAD.RSU1206B))

• Apply any incidental maintenance, such as a change to attachment code associated with an image.

• Restart the member subsystem. • Repeat for other members. • Should Back-Out be required, then redefine the Alias and repeat the above steps to

close and raise DB2. Note that the “product specific” versions that are required by ChangeMan, BMC etc. where they either need the latest or oldest code levels should also be managed via the Alias entry pointing to the correct RSU level of the required datasets.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 34 of 51


8 Software Refresh and Maintenance Cycles In a z/OS environment, software currency is one of the key elements in maintaining highly available and robust systems.

Typically, software currency is broken down into the following headings:

• Software Currency (version is supported by vendor) • Maintenance (Corrective or Preventative)

Both IBM and Independent Software Vendors (ISV’s) products should be managed in the same way. However, there will be terminology differences between IBM and the ISVs, but the underlying practices should be identical.

8.1 Software Currency Both IBM and the ISVs will maintain many versions of their software that rely on several factors:

• Hardware Installed • Version of z/OS installed • Version of required software (CICS, DB2, etc.) installed

It is generally accepted, and in some business sectors, the vendor supports a legal compliance or regulatory requirement that all installed software.

Some installations will maintain a strategy of installing all software at the very latest levels. However, there are risks with this strategy, for example installing code that has not been tested in large commercial environments. Remember, IBM will not be able to test with the volumes seen in the largest z/OS installations.

The majority of RSM’s clients do currently mandate a strategy of supporting n-1 for supported software. However, given IBM’s release cycle for z/OS, which has been changed to every two years, RSM noticed that its clients challenge that thinking. So in the case of z/OS, with the latest version currently 2.1, RSM would recommend that clients are running at least z/OS 1.13 in all of their Production LPARs.

8.2 Recommendations for Software Currency As far as IBM products are concerned, RSM recommends that HNB adopt the n-1 strategy for all products apart from z/OS.

z/OS should only be upgraded to the latest version when two quarterly RSU Service Packages have been made available, unless there is a distinct and justified reason for upgrading.

For z/OS, IBM maintains the following information showing the latest supported versions: http://www-03.ibm.com/systems/z/os/zos/support/zos_eos_dates.html

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 35 of 51


As far as ISV products are concerned, RSM recommends an n-1 strategy for software currency.

However, for certain products this will not be possible, especially when upgrading z/OS which will often determine that the latest version of an ISV product must be installed.

8.3 z/OS Maintenance Maintenance can be corrective or preventive. Corrective maintenance is necessary after a system has already encountered a problem. Regardless of the impact of the actual problem, the need to install the corrective maintenance may result in an unscheduled outage, which can have a larger impact. A preventive maintenance strategy will enable HNB to schedule the servicing of the z/OS system based upon their business needs.

RSM recommends having an unambiguous process to install preventive maintenance on a regular basis. It is beneficial to understand the concepts and different aspects of the z/OS service principles in order to create a process that is well defined.

Preventive maintenance of IBM products is an integral part of system stability and essential for achieving the highest availability for z/OS systems. Proactively servicing the z/OS system will safeguard against failures and unnecessary outages that are caused by known problems with available fixes. Having a preventive maintenance strategy has proven to be effective, and is generally recommended by IBM.

IBM is committed to identify defects, to accurately diagnose the root cause of problems, and to provide quality fixes in a timely manner. IBM’s support team ensures that APARs are updated with precise information relevant to the acknowledged defect, and flags the APARs appropriately to aid in service decisions. To complement these commitments, IBM has created the Recommended Service Upgrade (RSU) Maintenance Strategy. Following the RSU Maintenance Strategy will reduce HNB’s exposure to known problems and improve overall availability.

Having a robust preventive maintenance process is a best practice in managing z/OS. By avoiding known defects, which can have a major impact on the functioning of the system, preventive maintenance improves availability. Having a proactive preventive service strategy reduces the number of rediscovered defects and avoids unplanned outages.

One of z/OS’s strengths is its tight integration of the technologies across the hardware, firmware, operating system and middleware. To continue that strength throughout the products’ lifecycle, IBM developed an additional test effort known as Consolidated Service Test (CST) and a recommended preventative maintenance strategy, which is directly tied to this effort.

The Recommended Service Upgrade (RSU) was introduced in 2001, and continues to provide a consistent, installable, and tested preventive maintenance-level for the z/OS operating system, key subsystems such as CICS, DB2, IMS, MQSeries, WebSphere Application Server, and many of the other IBM tools and products that run on the z/OS platform. For many years, each IBM product had its own preventive maintenance strategy and had its own recommendations as to what maintenance to install. The inconsistencies of the recommendations led to confusion. With the RSU

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 36 of 51


recommendation, the benefit is that all IBM products have the same recommendation for preventive maintenance.

CST is a service environment that exists to test the maintenance for the operating system and key subsystems. The intent is to verify that the PTFs and products work well together, and to identify problems that might affect this process. The result of the testing is a package of maintenance across the product set that has been tested together. The testing done in CST is in addition to any testing that would be done prior to the release of the PTF as corrective service. The RSU provides integrated and tested service packages for the user to install. The process used to create the RSU provides for a reduced risk of encountering defective maintenance (PEs). Since its introduction, many of RSM’s customers have accepted the RSU as the maintenance strategy.

The general suggestion is for installations to stage the roll-out of the Recommended Service Upgrade (RSU) by product, on any single system. Additionally, we suggest that the client not change all the major products (such as z/OS, DB2, IMS, CICS, CTG, DB2 Connect, GDPS, Java, WebSphere MQ, WebSphere Application Server for z/OS, IBM DB2 and IMS Tools and z/OS Problem Determination Tools) all at once.

Changing many products in a single system simultaneously may complicate the tasks of problem diagnosis and PTF back out, if a severe problem occurs.

RSM recognizes that some installations have limited maintenance windows and need to upgrade several products at the same time. Therefore, IBM does test the “Big Bang” (maintenance for all products at the same time on a single image) method of rolling out maintenance to ensure that it works. To avoid the need to back out all of the maintenance, thorough customer application testing be done in this case.

View additional information about the RSU and CST on the CST website:

http://www-03.ibm.com/systems/z/os/zos/support/servicetest/

RSUs can be obtained through SMP/e Internet Service Retrieval as well as in service orders placed through Shopz and ServiceLink. The deliverables include ++ASSIGN statements that identify the RSU maintenance to be installed.

The SOURCEID identifies the RSU ID in the form of RSUyymm. The SOURCEID is the date the service completed the CST test cycle and was recommended - it is not the date that the service became available. RSUs are available monthly in the first few days of the month, and are available in two cycles. NOTE: RSUs are not cumulative; therefore all RSUs must be installed. Quarterly RSUs (yy03, yy06, yy09, yy12)

RSU service packages delivered each quarter include ALL maintenance that has been available as corrective service at the beginning of the prior quarter. The PTFs are added to the CST environment at the beginning of the quarter and tested for a full three months. For example, a severity 3 PTF is made available on July 16th. It is added to the CST environment on October 1st, tested for 3 months and made available on the RSU delivered in January (RSUyy12).

Monthly RSUs (yy01, yy02, yy03, yy04 ...)

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 37 of 51


For those customers who prefer more regular and frequent service upgrades, a monthly RSU is available. It contains HIPER, PE, Security/Integrity and Pervasive fixes. The fixes are added to the CST environment on the first of the month and tested for one full month. The fix is then included on the next RSU released. For example, the corrective service for a HIPER APAR is made available on January 12th. The PTF is added to the CST environment on February 1st. It is then included on the RSU delivered in March (RSUyy02).

RSUs comprise maintenance from multiple products available on the z/OS platform, including CICS, IMS, DB2, WebSphere Application Server and MQ. Detailed information about CST is available at: http://www.ibm.com/systems/z/os/zos/support/servicetest/

8.4 RSM’s Recommendation for z/OS Maintenance Practices RSM recommends that HNB take an aggressive, but pragmatic approach to RSU maintenance:

• RSU maintenance should be installed at least two times per year. However, given ever changing business requirements, a more frequent schedule of four times per year should be considered to apply the latest RSU Maintenance.

• HIPER and PE fixes should be reviewed weekly, and installed weekly or monthly if deemed applicable.

• Security/Integrity APARs and Red Alerts should also be monitored regularly, and installed weekly or monthly if deemed applicable.

8.5 DB2 Maintenance

8.5.1 Service Installation New level of DB2 maintenance should be deployed as a part of the z/OS ServicePac, individual site-sensitive HIPERS and PEs should be reviewed on a case-by-case basis. Synchronizing release levels with other key subsystems using IBMs CST releases is recommended.

A CST release is a collection of IBM subsystem maintenance that has been tested for co-existence.

A new SMP/e targetzone for the base release is considered best practice.

8.6 CICS Maintenance

8.6.1 Service Installation New levels of CICS maintenance should be ordered and matched to z/OS ServicePac as part of an IBM CST release, synchronising the code base of key subsystems to known compatible levels (ie CICS to DB2). Individual site-sensitive HIPERS and PEs should be reviewed, applied or removed as per site requirements. ServicePac installs should happen twice annually with monthly reviews of HIPERS to be installed on an ad-hoc basis should there be a business requirement.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 38 of 51


9 Testing 9.1 Testing Analysis

9.1.1 ISV Testing Basic testing is carried out by the Systems Programmers but more thorough testing is done by other departments e.g. CA-View and CA-Deliver. Not all testing is done in XPLEX or TPLEX – some testing of products is done in SPLEX. This is because test environments for all products are not available in the XPLEX or TPLEX. In addition, the SPLEX is shared between Production and Application Development so the Applications Developers need the products in SPLEX to enable testing.

9.1.2 CICS Testing Basic testing is done in XPLEX by starting the CMAS, WUI’s, a Test CICSPlex, and the Tech region. There is no test data on the TPLEX for testing although there are several regions started and the ChangeMan application resides there. Most testing is done in SPLEX in the Application Development LPARs HNB2 and HNB5.

After initial testing of a new release on HNBXPLEX, the service should be rolled out to HNBTPLEX and then HNBSPLEX.

An individual Production Acceptance Testing (PAT) CICSPlex environment matching Production should be retained on HNBTPLEX for proving of emergency production changes.

A separate symbolic (&PATPLEX) should be used for the PAT test system to bring the base runtime code in line with Production.

9.1.3 DB2 Testing Maintenance would be running in the QA and Test environments before the Production install.

9.1.4 MQ Testing Basic MQ testing is first done in XPLEX and this includes an environment with Queue sharing. The connections to CICS are tested but there are no applications to test on XPLEX.

Application testing begins in TPLEX.

As each application has been moved to use a queue-sharing group, the scenario where an LPAR (and Queue Manager) fails is tested to make sure that the queues remain available to the application. This is done on TPLEX.

9.1.5 ChangeMan Testing To test ChangeMan, there are two sets of tasks:

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 39 of 51


1. CHGMANZ (DP task) and CHGMANY (P task-slave) on the SPLEX to test version of DP-P on the same LPAR.

2. CHGMANR (DP task) on XPLEX with CHGMANQ on SPLEX (P task-slave) to match cross LPAR (matching current production).

Normally there would only be one set of test-started tasks, except when a major release or major change is being customized.

9.2 Testing Issues Some products are not thoroughly tested before SPLEX – either they are not available in the other environments or there is no test data.

CHGMANA promotes to ‘test’ on SPLEX because the developers insist on working there due to production data being there. They will copy production data and use for testing or, if read only; actually use the production files to test code.

9.3 Testing Recommendations This section focuses on System Software and not the applications that rely on it. However, during the analysis phase, it was noted that Application Development (AD) and testing is performed in the SPLEX, which is the Production LPAR and therefore has the highest SLA targets.

For the following reasons, all Application Development work and testing should be moved to the TPLEX in its own isolated environment:

• Production system availability/reliability • Data integrity • Security

HNB has 3 SYSPLEXes and they should be used for the following purposes:

• SPLEX: Production Only • TPLEX: Application Development, Application testing, Enhanced System Software

testing • XPLEX: System Software installation and functionality testing

Moving Application Development to TPLEX will also alleviate the problems faced by ChangeMan.

Every software product should have its own documented test case, which specifies the tests required and results expected.

The test cases should reflect how the software is used in Production (including relationships with other software) and should also test error recovery scenarios.

There should be ongoing, regular reviews of the test cases to make sure they are current (e.g., whenever software is updated or new functionality is exploited).

Examples of the types of tests that should be carried out on XPLEX:

• Basic Start and Stop • Message Analysis (when does the software initialize and become available) • IPL

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 40 of 51


• Performing basic functionality testing • Performing failure testing (e.g. Loss of LPAR in SYSPLEX)

Examples of the types of tests that should be carried out on TPLEX:

• Full function testing • Application testing

9.3.1 Test Cases Typical headings for a test case and descriptions are listed in the following table:

Test Suite ID The ID of the test suite to which this test case belongs. Test Case ID The ID of the test case. Test Case Summary The summary / objective of the test case. Related Requirement The ID of the requirement this test case relates/traces to. Prerequisites Any prerequisites or preconditions that must be fulfilled prior to executing

the test. Test Procedure Systematic procedure to execute the test. Test Data The test data, or links to the test data, that is to be used while conducting

the test. Expected Result The expected result of the test. Actual Result The actual result of the test; to be filled after executing the test. Status Pass or Fail. Other statuses can be ‘Not Executed’ if testing is not

performed and ‘Blocked’ if testing is blocked. Remarks Any comments on the test case or test execution. Created By The name of the author of the test case. Date of Creation The date of creation of the test case. Executed By The name of the person who executed the test. Date of Execution The date of execution of the test. Test Environment The environment (Hardware/Software/Network) in which the test was

executed.

A sample test case skeleton has been provided.

9.3.2 z/OS RSU Maintenance Testing When applying RSU maintenance IBM has already performed Consolidated Service Testing (CST) to improve the quality of the RSU service.

CST tests all maintenance for a collection of IBM products in a single package leading to a SOURCEID(collection of PTF’s) where IBM has provided a level of assurance that this maintenance will integrate correctly. IBM currently performs CST tests for the following products:

• CICS Transaction Gateway for z/OS • CICS Transaction Server for z/OS • DB2 for z/OS • DB2 Connect • Geographically Dispersed Parallel Sysplex (GDPS/PPRC) • IMS

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 41 of 51


• IRLM • JAVA • WebSphere Application Server for z/OS • WebSphere MQ for z/OS • z/OS • z/OS Problem Determination Tools • IBM DB2 and IMS Tools • IBM Tivoli (products listed in quarterly report)

Refer to the IBM website, where details can be reviewed for the quarterly report. http://www-03.ibm.com/systems/z/os/zos/support/servicetest/

9.3.3 DB2 After initial testing of a new release on HNBXPLEX is complete, the service should be rolled out to HNBTPLEX and then HNBSPLEX.

An individual Production Acceptance Testing (PAT) DB2 subsystem matching production should be retained on HNBTPLEX for proving emergency production changes and should be connected to the PAT test CICSPlex.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 42 of 51


10 PARMLIB & PROCLIB 10.1 PARMLIB & PROCLIB Analysis In this section, items in bold will vary depending on SYSPLEX name, volume serial number etc., as appropriate; in an effort to make the terms easier to understand versus consistently using HNBpPLEX, SYpMCT etc. throughout the document.

10.1.1 SYS1.IPLPARM Each SYSPLEX has its own SYS1.IPLPARM data set on volume SYSMCT. The three instances of the data set are identical and each one contains members for all LPARs, not just the LPARs in its own SYSPLEX.

The member naming convention is LOADxy where x represents the LPAR number, i.e., HNBx, and y represents the IODF number, i.e., SYS1.IODF0y. Each LPAR has LOADx0 thru LOADx3 defined and all are currently using LOADx3. Additional members called LOADxA, LOADxB, LOADxC, LOADxD and LOADxX, which tend to have different IODF and/or SYSPARM values, are defined but not for every LPAR.

All the LOADxy members are similar to the following LOAD13 member: ARCHLVL 2 IEASYM 06 IODF 03 SYS1 OPSYS01 00 Y NUCLEUS 1 NUCLST 01 PARMLIB SYS1.PARMLIB SYSMCT PARMLIB SYS1.IBM.PARMLIB ****** SYSCAT SYSMCT113CCAT.ICF.HNBSPLEX.MASTER CAT SYSPARM 10 SYSPLEX HNBSPLEX

10.1.2 PARMLIB Each LPAR has the following logical PARMLIB defined:

1) SYS1.PARMLIB, which resides on the SYSMCT volume. 2) SYS1.IBM.PARMLIB, which resides on the SYSRS1 or SYSAL1 volume, as

appropriate.

10.1.3 SYS1.IBM.PARMLIB SYS1.IBM.PARMLIB is identical across all three SYSPLEXes and is as shipped by IBM. This data set should only be updated during z/OS installation and maintenance. If modifications are required, the member(s) should be copied to SYS1.PARMLIB and updated there.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 43 of 51


10.1.4 SYS1.PARMLIB SYS1.PARMLIB differs across the three SYSPLEXes although there are many similarities. Each instance of the data set contains members for LPARs that are out with its own SYSPLEX. This data set contains any HNB-customized members and parameters. Many of the members in this data set are shared by all the LPARs in a SYSPLEX. These common members are usually suffixed with 00, whereas LPAR-specific members have their own suffices. There is some usage of system symbolics within the members.

10.1.5 IEASYM06 All LOADxy members, with the exception of DR members LOADD1 and LOADD7, specify IEASYM 06. All three SYSPLEXes use an identical IEASYM06 member that covers all LPARs. Note that the member has been edited for conciseness.

SYSDEF SYSNAME (HNB1) SYSDEF VMUSERID() SYSNAME(NOTDRP) SYSDEF SYMDEF(&PRIHSM.='Y') SYSDEF VMUSERID() SYMDEF(&PRIHSM.='N') SYSDEF LPARNAME(HNB1) SYSNAME(HNB1) .....etc..... SYSDEF LPARNAME(HNB9) SYSNAME(HNB9) SYSDEF VMUSERID(HNB1) SYSNAME(HNB1) .....etc..... SYSDEF VMUSERID(HNB9) SYSNAME(HNB9) SYSDEF SYMDEF(&LPN.='&SYSNAME(-1:1).') SYSDEF SYMDEF(&SYSR2.='&SYSR1(1:5).2') SYSDEF SYMDEF(&SYSR3.='&SYSR1(1:5).3') SYSDEF SYMDEF(&UXROOT.='&SYSR1(4:2).') SYSDEF HWNAME(PROC02) LPARNAME(HNB7) SYMDEF(&PRIHSM.='Y') SYSDEF HWNAME(PROC02) LPARNAME(HNB4) SYMDEF(&PRIHSM.='Y') SYSDEF SYMDEF(&PROCNAME.='PROC01') SYSDEF HWNAME(PROC02) SYMDEF(&PROCNAME.='PROC02') SYSDEF HWNAME(PROC03) SYMDEF(&PROCNAME.='PROC01') SYSDEF LPARNAME(HNB1) SYMDEF(&SUBPLEXN.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&SUBPLEXN.='HNBSPLEX') SYSDEF LPARNAME(HNB1) SYMDEF(&HNB2TPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB2TPLX.='HNBXPLEX') SYSDEF LPARNAME(HNB1) SYMDEF(&HNB5TPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB5TPLX.='HNBXPLEX') SYSDEF LPARNAME(HNB1) SYMDEF(&HNB3XPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB3XPLX.='HNBXPLEX') SYSDEF LPARNAME(HNB1) SYMDEF(&HNB4XPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB4XPLX.='HNBXPLEX')

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 44 of 51


SYSDEF LPARNAME(HNB1) SYMDEF(&HNB6TPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB6TPLX.='HNBXPLEX') SYSDEF LPARNAME(HNB1) SYMDEF(&HNB8TPLX.='HNBSPLEX') .....etc..... SYSDEF LPARNAME(HNB9) SYMDEF(&HNB8TPLX.='HNBXPLEX') SYSDEF SYMDEF(&RESVOL.='&SYSR1(4:2).')

10.1.6 IEASYS00 HNBSPLEX and HNBTPLEX have an identical IEASYS00 and there are only a few minor differences between them and HNBXPLEX.

ALLOC=00, CEE=00, CLOCK=00, CLPA, CMB=1400, COUPLE=00, CSA=(2200,500M), CSCBLOC=ABOVE, DIAG=00, DUMP=(DASD,&LPN.0-&LPN.2), FIX=00, GRS=STAR, GRSCNF=01, GRSRNL=02, HVCOMMON=120G, ICS=02, ILMLIB=SYS1.ILMLIB, ILMMODE=NONE, IOS=00, LICENSE=Z/OS, LOGCLS=E, LOGLMT=999999, LOGREC=SYS1.&SYSNAME..LOGREC, LPA=(00,L), MAXCAD=50, MAXUSER=530, MSTRJCL=00, OMVS=00, OPI=NO, OPT=00, PAGTOTL=(20), PROD=00, PROG=00, REAL=0, RSVNONR=50, RSVSTRT=20,

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 45 of 51


SCH=00, SMF=00, SMS=00, SQA=(10,750), SSN=00, SVC=00, VAL=00, VIODSN=SYS1.&SYSNAME..STGINDEX, VRRI.E.,N=0, ZZ=YES

10.1.7 IEASYSxy Each LPAR has its own parameter overrides to IEASYS00 and the member naming convention is IEASYSxy where x represents the LPAR number, i.e., HNBx, and y is usually zero. Additional members called IEASYSxA, IEASYSxB, IEASYSxX and IEASYSxZ, which contain various different parameter overrides, are defined but not for every LPAR.

All the IEASYSxy members are similar to the following IEASYS10 member: CMD=(1C,1V,1J), CON=(00,NOJES3,SHARED), COUPLE=01, DEVSUP=10, DIAG=(00,02), GRSRNL=23, IXGCNF=00, PAGE=(PAGE.&SYSNAME..PLPA,

PAGE.&SYSNAME..COMMON, PAGE.&SYSNAME..LOCAL1, PAGE.&SYSNAME..LOCAL2, PAGE.&SYSNAME..LOCAL3, PAGE.&SYSNAME..LOCAL4, PAGE.&SYSNAME..LOCAL5, PAGE.&SYSNAME..LOCAL6, PAGE.&SYSNAME..LOCAL7, PAGE.&SYSNAME..LOCAL8, PAGE.&SYSNAME..LOCAL9, PAGE.&SYSNAME..LOCAL10,L),

PLEXCFG=MULTISYSTEM, RSU=0

10.1.8 PROCLIB Each LPAR has the following PROCLIB concatenation defined:

1) SYS1.HNBSPLEX.PROCLIB, which resides on a system SMS volume. 2) SYS1.IBM.PROCLIB, which resides on the SYSRS1 or SYSAL1 volume, as

appropriate. 3) SYS2.HNBSPLEX.PROCLIB, which resides on a system SMS volume.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 46 of 51


Note: Various Application Developer PROCLIBS that differ according to SYSPLEX follow these Datasets; however, these are out of scope for this document.

10.1.9 SYS1.IBM.PROCLIB SYS1.IBM.PROCLIB is identical across all three SYSPLEXes and is as shipped by IBM. This data set should only be updated during z/OS installation and maintenance. If modifications are required, the member(s) should be copied to SYS1.PROCLIB and updated there.

10.1.10 SYS1.HNBSPLEX.PROCLIB SYS1.HNBSPLEX.PROCLIB is almost identical across all three SYSPLEXes and is effectively as shipped by IBM. This data set contains any HNB-customized members and parameters for z/OS related products i.e., High Level Assembler, C/C++, COBOL, PL/I, JES2, VTAM, TSO, TCP/IP and zFS etc. There is very little usage of system Symbolics within the members.

10.1.11 SYS2.HNBSPLEX.PROCLIB SYS2.HNBSPLEX.PROCLIB differs across the three SYSPLEXes although there are many similarities. This Dataset contains any HNB-customized members and parameters for the following systems:

• IBM non-z/OS related products, i.e., CICS, DB2, MQ etc. • ISV software products i.e., BMC, SAR etc. • Local procedures i.e., TSO logon procs, GTF procs etc.

Note: There is very little usage of system Symbolics within the members.

10.2 PARMLIB & PROCLIB Recommendations

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 47 of 51


11 Recommended Reading

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 48 of 51


12 Appendix 12.1 Glossary of Technical Terms APARs IBM documents defects of z/OS products in APARs (Authorized Program Analysis Reports). IBM in a release level PTF (Program Temporary Fix) provides corrections for the defects. APARs also offer a means to supply a new function for a product that is already generally available. IBM may flag an APAR under special circumstances that warrant extra attention.

HIPER APARs are marked HIPER for high impact problems, in which the problem addressed is critical and normal processing is adversely affected. Circumstances include, but are not limited to:

• The need to re-IPL, restart or recycle a subsystem in order to recover • Loss of a major function • Data loss • Recursive or unrecoverable failures • Severe performance degradation

HIPER APARs are considered severe and IBM suggests that the fix is installed or the circumvention is implemented as soon as possible.

SPECIAL ATTENTION APARs are marked SPECIAL ATTENTION if they are considered important for the customer to install. Situations include, but are not limited to:

• A new function to the product • Problems encountered during the installation of the product • Problems encountered during the installation of service updates • Problems related to a particular condition or environment PERVASIVE HIPER APARs and SPECIAL ATTENTION APARs may also be marked PERVASIVE. A pervasive APAR has a high probability of affecting a large number of systems.

PE APARs may be marked to identify an error in a PTF after the PTF is made generally available. Situations that characterize a PE (PTF in Error) APAR include, but are not limited to:

• If the PTF solves the original problem but creates a new problem that did not previously exist.

• If the PTF does not solve the original problem. • If the supplied SMP/e application control statements in the PTF contain errors or

prohibit it from applying correctly.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 49 of 51


• There is a documentation error that negatively affects the system. INTEGRITY/SECURITY APARs may be marked as INTEGRITY to identify problems that allow unauthorized access and compromise system controls. Security APARs address problems with existing security measures that lead to security exposures to the system as a whole or to an IBM product that runs on the system. The content of Security/Integrity APARs is classified as IBM Confidential, in order to protect all clients on the IBM System z platform from the exposure.

Access to Security/Integrity APAR information is available through Resource Link™ by sending an email to [email protected]. The request is validated by the IBM account team and requires IBM management approval. More information on z/OS Security and Integrity is available at http://www.ibm.com/systems/z/advantages/security/integrity_zos.html

Notification of HIPER and PE Critical Fixes Being made aware of critical fixes is the first step to a successful preventive maintenance plan. IBM provides various notification methods to aid in this part of the process including HOLDDATA, PSP Buckets, and ServiceLink applications.

Enhanced HOLDDATA IBM supplies a single source text file with information pertaining to critical fixes called Enhanced HOLDDATA. This one file encompasses all IBM z/OS platform products and is updated daily. When the file is received, SMP/e ignores FMIDs that are not installed, so that HOLDDATA can be used on any system. The Enhanced HOLDDATA file is a reliable source for identifying HIPER and PE fixes that are available and comparing them to what is installed on the system when used as input to SMP/e. Security/Integrity HOLDDATA is available to authorized customers through the System z Security Portal. ERROR IBM identifies HIPER and PE PTFs in an Enhanced HOLDDATA file with the ERROR construct.

The HOLDDATA provides information to identify the reason for the hold and the fixing PTF. For a PE APAR, the hold is against the PTF in error. The Enhanced HOLDDATA is received into the SMP/e global zone. The SMP/e REPORT ERRSYSMODS command can then be used on any target zone to identify missing critical service that applies to the system. The report lists the exception SYSMODs, the APAR numbers, the resolving SYSMODs that have not been installed yet, and the hold symptoms.

FIXCAT In the Enhanced HOLDDATA file, a construct called FIXCAT is used to identify a function or product category that the PTF may be associated with. FIXCAT categorizes preventive maintenance to allow selectively choosing which service is applicable to your system and can be ordered. More information about the FIXCAT construct can be found in the SMP/e Reference manual.

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 50 of 51


More information about the z/OS Enhanced HOLDDATA, including downloading the latest data file, is available on the website:

http://service.software.ibm.com/holdata/390holddata.html

PSP Bucket A PSP (Preventive Service Planning) Bucket is an informational file created to help with the migration to a new release of z/OS and upgrading products and components that run on the z/OS platform. There are hardware and software PSP buckets that are updated daily, and include a list of HIPER APARs with available fixes, installation tips, and general information.

PSP Buckets are categorized into upgrades and subsets. Upgrades represent the product release and subsets represent an individual component within the release. For example, UPGRADE ZOSV1R11, SUBSET DFSMS™ contains all information pertaining to DFSMS Version 1, Release 11. You can search for and view PSP buckets at: http://www14.software.ibm.com/webapp/set2/psp/srchBroker

ServiceLink using IBMLink IBM provides methods of automatic notification of problems and fixes relating to IBM products through ServiceLink. The ASAP (Automatic Software Alert Process) application and the Alert for IBM eServer™ zSeries® application allow you to subscribe to IBM products of interest and receive electronic notification pertaining to critical software problems as they are identified.

Red Alert Red Alert is another IBM method used to communicate extremely critical APARs that may affect the availability of a system. Red Alerts are published, in addition to an APAR being flagged HIPER, for a small number of exceptionally severe problems. There are no specific criteria for a Red Alert. Subscription services are available for Red Alerts through ServiceLink.

More information on Red Alerts can be found at http://www14.software.ibm.com/webapp/set2/sas/f/redAlerts/home.html IBM z/OS Preventive Service Deliverables IBM has various resources for obtaining necessary fixes, including ShopzSeries and the automated SMP/e Internet Service Retrieval function.

ShopzSeries ShopzSeries is a web application that you can use to obtain individual PTFs or service product packages. You can customize the ordering process by uploading reports from the z/OS host, which indicate the current service level and products installed on the host system. CBPDO (Custom Built Product Delivery Option) is available to order multiple IBM software products, up to a specified PUT level, all packaged on one tape or available for internet delivery. A PUT (Program Update Tape) level identifies when the PTF became available with SOURCEID PUTyymm.

ShopzSeries is available at http://www.ibm.com/software/shopzseries

Doc. Version: 0.7


Doc. Date: 10/29/2013 Page: 51 of 51


There are other preventive service fee offerings available through CustomPac®, including RefreshPac, which will deliver preventive maintenance through physical media. The maintenance included on the deliverable is customized based on the SMP/e CSI dataset that is uploaded electronically. ROP (Refresh on Profile) is similar, except that the CSI is not required to be uploaded for each order, and based on the size of the order delivery may be using the internet or physical media. More information on CustomPac offerings is available at https://www.ibm.com/services/ca/en/custompac/

SMP/e Internet Service Retrieval SMP/e Internet Service Retrieval (available in z/OS release 1.7) is a one-step automated method to create and upload a service inventory file from your SMP/e CSI dataset, submit a service order, wait for the package to be ready, download the package electronically and processes the service package. The service order can be corrective service, naming specific APARs and PTFs. It can also be preventive service, requesting critical (HIPER, PE), recommended (RSUyymm) or all PTFs. It may also be just for a two- year file of Enhanced HOLDDATA without PTFs. The RECEIVE ORDER command accomplishes this task, and the process can be manually initiated or scheduled to complement your preventive service strategy.

12.2 Skeleton Test Report

Attached is a skeleton test reportTest_Case_Template

.doc .

Huntington National Bank Maintenance Standards v10

Documents

Transcript of Huntington National Bank Maintenance Standards v10