IPCS for Networkers Part 2 of 2: Tales from the Crypt: How we … · 2014. 8. 6. · Session 15511...

InsertCustomSessionQR if Desired.

IPCS for Networkers Part 2 of 2:Tales from the Crypt: How we solve a VTAM/TCPIP Problem using IPCSGeorg SenflebenIBM Germany

Thursday, August 7, 201411:15 AM – 12:30 PMDLLCC, Room 318Session 15511

https://ibm.biz/BdFXzc

2

TELNET Problem EZZ6035I TN3270 Server not starting

214 EZZ6045I B0TN3270 PROFILE PROCESSING COMPLETE FOR FILE 214 SYS1.C.TCPPROF(VTAMSSL) 214 EZZ6046I B0TN3270 PROFILE PROCESSING RESUMING FOR FILE 214 SYS1.C.TCPPROF(VTAMPLEX) 214 EZZ6045I B0TN3270 PROFILE PROCESSING COMPLETE FOR FILE 214 SYS1.C.TCPPROF(VTAMPLEX) 08.51.27 STC09943 EZZ6091I B0TN3270 JOINED XCF GROUP EZZTLUNS 08.51.27 STC09943 *EZZ6092I B0TN3270 LUNR PROFILE PENDING 08.51.29 STC09943 EZZ6035I B0TN3270 DEBUG TASK EXCEPTION 291 291 TASK: LUNR MOD: EZBTXLUR 291 RCODE: A013-02 LUNS/LUNR send failed. 291 PARM1: 0000A013 PARM2: 00000000 PARM3: 00000000 08.51.30 STC09943 EZZ6035I B0TN3270 DEBUG TASK EXCEPTION 342 342 TASK: LUNR MOD: EZBTXLUR 342 RCODE: 102B-00 Socket initialization failed. No retry. 342 PARM1: 00000462 PARM2: 76690324 PARM3: BPX1SND 08.51.38 STC09943 EZZ6035I B0TN3270 DEBUG TASK EXCEPTION 826 826 TASK: XCFRECV MOD: EZBTXRCV 826 RCODE: A013-00 LUNS/LUNR send failed. 826 PARM1: 0000A013 PARM2: 00000000 PARM3: SYC3 B0TN3270

3

TELNET Problem EZZ6035I TN3270 messages


A013 LUNS/LUNR send failed.A send error occurred while trying to write on the connection between the LUNS and LUNR. PARM1 is the return code. PARM2 is the reason code.

A013 LUNS/LUNR send failed.A send error occurred while trying to write on the connection between the LUNS and LUNR. PARM1 is the return code. PARM2 is the reason code.

4

TELNET Problem EZZ6035I IPCS ERRNO CLIST


IP ERRNO E 0462Errno: 00000462(1122) : ENOBUFS Description: Insufficient buffer space availableIP ERRNO 76690324ReasonCode: 76690324 Module: EZBTCFWR ErrnoJr: 804 JRSMNOSTORAGE Description: There is no storage available to allocate

IP ERRNO E 0462Errno: 00000462(1122) : ENOBUFS Description: Insufficient buffer space availableIP ERRNO 76690324ReasonCode: 76690324 Module: EZBTCFWR ErrnoJr: 804 JRSMNOSTORAGE Description: There is no storage available to allocate

5

TELNET Problem EZZ6035I IP CTRACE Command

IP CTRACE COMP(SYSTCPIP) SUBNAME((B0TCPIP)) LOC **** 07/23/2014 SYSNAME MNEMONIC ENTRY ID TIME STAMP DESCRIPTION ------- -------- -------- --------------- ------------- SYC3 INIT 00010002 08:51:08.975814 CTRACE Initialization Complete SYC3 IOCTL 6003004D 08:51:12.949942 ÜSIOCSVIPA/SIOCSVIPA6 - Define / . . .SYC3 TCP 40037D1F 08:51:27.066549 ÜBad SYN return code SYC3 VTAMDATA 10020041 08:51:27.471467 ÜStorage Error on BGRP SYC3 VTAMDATA 10020041 08:51:27.479456 ÜStorage Error on BGRP SYC3 VTAMDATA 10020041 08:51:27.495474 ÜStorage Error on BGRP SYC3 VTAMDATA 10020041 08:51:27.495659 ÜStorage Error on BGRP SYC3 VTAMDATA 10020041 08:51:28.009145 ÜStorage Error on BGRP SYC3 VTAMDATA 10020041 08:51:28.009741 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:28.406085 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:28.429218 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:28.634058 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:28.634319 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:29.041704 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:29.141347 ÜStorage Error on BGRPSYC3 VTAMDATA 10020041 08:51:29.690516 ÜStorage Error on BGRP

6

TELNET Problem EZZ6035I IP TCPIPCS Command

IP TCPIPCSDataset: ONTOP.GS033.P38668.C724.D0723.SYC3.DMPO4 Title: ISTRACSW - DEFAULT CSDUMP WITH TCPIP - ID=7CD3 The address of the TSAB is: 1A50F000 Tseb SI Procedure Version Tsdb Tsdx Asid TraceOpts Status1A50F040 1 B0TCPIP V2R1 1A4E8000 1A4E80C8 0085 00000000 00000000 Active

CSM Status ECSA Storage: OK Data Space Storage: OK Fixed Storage: CONSTRAINED AND CRITICALAlet: 01FF0020 Dspname: CSM64001 Alet: 00000000 Dspname: ........

SYSLOG:*EZZ9679E B0TCPIP CSM HAS BEEN CRITICAL FOR AT LEAST 60 SECONDS

SYSLOG:*EZZ9679E B0TCPIP CSM HAS BEEN CRITICAL FOR AT LEAST 60 SECONDS

7

TELNET Problem EZZ6035I IP VERBX VTAMMAP 'CSMALL'

The limit for CSM ECSA is 7D00000 ==>125MB.Currently in use are 9394D0 ==> 9MB.The limit for CSM FIXED is 7D00000 ==> 125MB. Currently in use are 7236510 ==> 114MB.

SYS1.PARMLIB(IVTPRM00)FIXED MAX(125M)ECSA MAX(125M)

SYS1.PARMLIB(IVTPRM00)FIXED MAX(125M)ECSA MAX(125M)

114/125 * 100 = 91,2% => critical !

IP VERBX VTAMMAP 'CSMALL'VTAMMAP input data CSMALL CSMALL Analysis IVTSMCST: 1DD71000 +0000 C3E2D440 34301000 89620C28 89622348 ! CSM ....i...i... ! +0010 1DD74E60 007D0000 0009394D 007D0000 ! .P+-.'.....(.'.. ! +0020 00723651 896287B0 8962F410 89637E88 ! ....i.g.i.4.i.=h ! +0030 00800000 0963C5F0 0963A000 0963EFF0 ! ......E0.......0 ! +0040 0963E050 1DD74ED8 00000004 00000001 ! ..Ö&.P+Q........ !

http://www-01.ibm.com/support/docview.wss?uid=pub1ga32090500

8

TELNET Problem EZZ6035I IP VERBX VTAMMAP 'CSMOWNER' Part 1

IP VERBX VTAMMAP 'CSMOWNER CSMOWNID(85)' CSMOWNER CSMOWNID(85) CSMOWNER Analysis CSM 4K ECSA Pool 1BA55000 for Owner ID 85 EXT 1B27C980 HDR 1B27CB20 PHDR N/A BUF 1B26F000 DSPNAME N/AEXT 1A619300 HDR 1A6198C0 PHDR N/A BUF 1A617000 DSPNAME N/AEXT 1A619300 HDR 1A1B170C PHDR 1A6198C0 BUF 1A617000 DSPNAME N/A...

EXT 18810380 HDR 198D1BD4 PHDR 18810F40 BUF 1880E000 DSPNAME N/A EXT 18810380 HDR 18810FA0 PHDR N/A BUF 1880F000 DSPNAME N/A EXT 18810380 HDR 198D131C PHDR 18810FA0 BUF 1880F000 DSPNAME N/A EXT 18810380 HDR 198D1364 PHDR 18810FA0 BUF 1880F000 DSPNAME N/A EXT 18810380 HDR 198D13AC PHDR 18810FA0 BUF 1880F000 DSPNAME N/A EXT 18810380 HDR 198D13F4 PHDR 18810FA0 BUF 1880F000 DSPNAME N/A Number of primary buffers owned by 85 in CSM 4K ECSA pool: 810

810 * 4K totals to 3 MB of ECSA Storage

9

TELNET Problem EZZ6035I IP VERBX VTAMMAP 'CSMOWNER' Part 2

IP VERBX VTAMMAP 'CSMOWNER CSMOWNID(85)' CSMOWNER CSMOWNID(85) CSMOWNER Analysis CSM 4K DSPACE64 Pool 1BA0B000 for Owner ID 85 EXT 1BA23300 HDR 1BA238C0 PHDR N/A BUF 0000F000 DSPNAME CSM64001EXT 1BA09980 HDR 1BA09D00 PHDR N/A BUF 00039000 DSPNAME CSM64001EXT 1B412980 HDR 1B412A00 PHDR N/A BUF 000E1000 DSPNAME CSM64001...

EXT 187C8380 HDR 187C8DC0 PHDR N/A BUF 07475000 DSPNAME CSM64001 EXT 187C8380 HDR 187C8E20 PHDR N/A BUF 07476000 DSPNAME CSM64001 EXT 187C8380 HDR 187C8E80 PHDR N/A BUF 07477000 DSPNAME CSM64001 EXT 187C8380 HDR 187C8EE0 PHDR N/A BUF 07478000 DSPNAME CSM64001 EXT 187C8380 HDR 187C8F40 PHDR N/A BUF 07479000 DSPNAME CSM64001 EXT 187C8380 HDR 187C8FA0 PHDR N/A BUF 0747A000 DSPNAME CSM64001 Number of primary buffers owned by 85 in CSM 4K DSPACE64 pool: 26202

26202 * 4K totals to 104 MB of Dataspace Storage

10

TELNET Problem EZZ6035I IP VERBX VTAMMAP 'CSMPOOL' Part 1

ip verbx vtammap 'csmpool csmsize(4K) csmtype(dspace) detail' CSM 4K DSPACE64 POOL CSMPOOL: 1BA0B000 +0000 D7D6D6D3 10694020 00001000 00000003 00006FA0 00000254 00000008 00000020 ! POOL.. . +0020 1BA23300 18121380 00000003 1BA739E0 00000000 00000048 1636B3AC 000005DB ! .s...... +0040 16290000 1DD71310 1C0EC5D8 00000100 00000040 0B980001 188A0380 00000080 ! .....P.. +0060 00000000 00000000 ! ........ CSMEXT: 1BA23300 +0000 C5E7E340 06800000 1BA0A980 00000000 1BA0B000 01FF0020 8000D800 0000022F ! EXT .... +0020 00000001 00001000 00010FFF 00000010 00000000 00000000 00010000 00000010 ! ........ +0040 C3E2D4F6 F4F0F0F1 00000000 00000000 00000000 00000000 00000000 FFFF0000 ! CSM64001 +0060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ! ........ CSMHDR: 1BA233E0 +0000 C8C4D940 00602080 00750000 00000000 00000000 00000000 00000000 00002000 ! HDR .-.. +0020 1BA233E0 00010000 00000000 00000000 00000000 00000000 CD7E70DA 9E54AB50 ! .s.Ö.... +0040 00000021 1B43F800 00000001 00000001 00000000 00000000 01010100 00000000 ! ......8. | '----> USERDATA '---> MONITOR ID

The major CSM storage allocations are for Monitor Ids 21 and 23.USERDATA is either a storage address or a channel unit address.

11

D NET,CSMUSE,POOL=4KECSA ---------------------------------------------------------------------- IVT5508I DISPLAY ACCEPTED IVT5574I PROCESSING DISPLAY CSMUSE COMMAND - POOL SPECIFIED IVT5584I USAGE DETAILS - 4KECSA POOL - POOL TOTAL = 50784K IVT5532I ------------------------------------------------------ IVT5576I AMOUNT MONITOR ID OWNERID JOBNAME IVT5532I ------------------------------------------------------ IVT5577I 20108K B1 006B B0TCPIP IVT5579I BUFFER USE FOR B1 : USECNT USERDATA MONITOR HISTORY IVT5580I 8306 28548EC0 000000B1 IVT5580I 27 28548DB0 912093B1 IVT5580I 5 28548DB8 99B199B1 IVT5580I 4 28548EC8 000000B1 IVT5532I ------------------------------------------------------ IVT5577I 8948K A9 006B B0TCPIP IVT5579I BUFFER USE FOR A9 : USECNT USERDATA MONITOR HISTORY IVT5580I 3497 7E7C86F0 2093B1A9 IVT5580I 66 7E7C86F0 A5A4B1A9 IVT5580I 40 7E7C86F0 9EA4B1A9 IVT5580I 11 7E7C86F0 9899B1A9

TELNET Problem EZZ6035I CSM Monitor Ids in VTAM Display

Monitor IDs are documented in z/OS Communications Server: IP and SNA Codes Chapter 4

12

TELNET Problem EZZ6035I CSM Monitor IDs

Monitor Ids are documented inz/OS Communications Server:IP SNA and Codes

https://ibm.biz/BdFHBR

13

TELNET Problem EZZ6035I IP VERBX VTAMMAP 'CSMPOOL' Part 2

ip verbx vtammap 'csmpool csmsize(4K) csmtype(dspace) detail' CSMHDR: 187CEA00 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AB000 ! HDR .-.. +0020 187CEA00 00100000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§...... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01000300 00000000 ! ....0602 CSMHDR: 187CEA60 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AC000 ! HDR .-.. +0020 187CEA60 00110000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§.-.... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01010300 00000000 ! ....0602 CSMHDR: 187CEAC0 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AD000 ! HDR .-.. +0020 187CEAC0 00120000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§.ä.... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01020300 00000000 ! ....0602 CSMHDR: 187CEB20 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AE000 ! HDR .-.. +0020 187CEB20 00130000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§...... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01030300 00000000 ! ....0602 | '--> MODULE '--> TIMESTAMP

TOD clock of the last activity is stored at offset 38.The address of the module last touched the buffer is at offset 3C.USERDATA is the channel unit address 0602.

14

TELNET Problem EZZ6035I LISTTOD and 'VTFNDMOD storage address'

ip verbx vtammap 'csmpool csmsize(4K) csmtype(dspace) detail' CSMHDR: 187CEA00 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AB000 ! HDR .-.. +0020 187CEA00 00100000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§...... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01000300 00000000 ! ....0602 CSMHDR: 187CEA60 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AC000 ! HDR .-.. +0020 187CEA60 00110000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§.-.... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01010300 00000000 ! ....0602 CSMHDR: 187CEAC0 +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AD000 ! HDR .-.. +0020 187CEAC0 00120000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§.ä.... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01020300 00000000 ! ....0602 CSMHDR: 187CEB20 .---> Fixed indicator +0000 C8C4D940 00602080 00850000 00000000 00000000 00000000 00000000 073AE000 ! HDR .-.. +0020 187CEB20 00130000 00000000 00000000 00000000 00000000 CD7E70ED 9A1C4C66 ! .§...... +0040 00002321 F0F6F0F2 00000001 00000001 00000000 00000000 01030300 00000000 ! ....0602 | '--> MODULE '--> TIMESTAMP

TOD clock of the last activity is stored at offset 38.The address of the module last touched the buffer is at offset 3C.USERDATA is the channel unit address 0602.

IP VERBX VTAMMAP 'VTFNDMOD 1A1C4C66' VTAMMAP input data VTFNDMOD 1A1C4C66 VTFNDMOD AnalysisModule name: ISTLLCRB Compile date: 13.058 PTF Number: HVT6210Address entered: 1A1C4C66 Module entry point: 1A1C4108 -------- Displacement into module: B5E

IP VERBX VTAMMAP 'VTFNDMOD 1A1C4C66' VTAMMAP input data VTFNDMOD 1A1C4C66 VTFNDMOD AnalysisModule name: ISTLLCRB Compile date: 13.058 PTF Number: HVT6210Address entered: 1A1C4C66 Module entry point: 1A1C4108 -------- Displacement into module: B5E

IP LISTTOD CD7E70ED07/23/2014 06:51:46.457600 STCK X'CD7E70ED 00000000'07/23/2014 06:51:25.457600 UTC X'CD7E70D8 F90C0000'07/23/2014 08:51:25.457600 LOCAL X'CD7E8BAB 6D8C0000'

IP LISTTOD CD7E70ED07/23/2014 06:51:46.457600 STCK X'CD7E70ED 00000000'07/23/2014 06:51:25.457600 UTC X'CD7E70D8 F90C0000'07/23/2014 08:51:25.457600 LOCAL X'CD7E8BAB 6D8C0000'

15

TELNET Problem EZZ6035I QDIOSTG and READSTORAGE

Keyword READSTORAGE on LINK and INTERFACE overrides VTAM Start option QDIOSTG for a specific device.

• READSTORAGE Global • causes the QDIOSTG values to be used

• READSTORAGE MAX, AVG or MIN• Causes the QDIOSTG option MAX, AVG or MIN to be used


16

TELNET Problem EZZ6035I TCPIP Profile Interface statementIP TCPIPCS PROFILEINTERFace TENGB1E0V056 DEFINE IPAQENET CHPIDTYPE OSX CHPID 1E IPADDR

10.232.75.123/25 NONROUTER VLANID 56 VMAC ROUTEALL

READSTORAGE GLOBAL INBPERF DYNAMIC WORKLOADQ SECCLASS 255

NOMONSYSPLEX NODYNVLANREG NOISOLATE NOOLM

READSTORAGE GLOBAL => use QDIOSTG definition INBPERF DYNAMIC WORKLOADQ => IWQ is enabled When QDIO inbound workload queueing (IWQ) is not enabled, only one inbound queue is active for each active device

When IWQ is enabled, one or more ancillary input queues might also be active


17

TELNET Problem EZZ6035I DISPLAY TRLE of the QDIO device

D NET,TRL,TRLE=IUTXT01FIST097I DISPLAY ACCEPTED IST075I NAME = IUTXT01F, TYPE = TRLE 221 IST1954I TRL MAJOR NODE = ISTTRL IST486I STATUS= ACTIV, DESIRED STATE= ACTIVIST087I TYPE = LEASED, CONTROL = MPC , HPDT = YES IST1715I MPCLEVEL = QDIO MPCUSAGE = SHARE IST2263I PORTNAME = IUTXP01F PORTNUM = 0 OSA CODE LEVEL = 0AAA IST2337I CHPID TYPE = OSX CHPID = 1F PNETID = IEDN . . .IST1221I DATA DEV = 0602 STATUS = ACTIVE STATE = N/AIST1724I I/O TRACE = OFF TRACE LENGTH = *NA*IST1717I ULPID = B0TCPIP ULP INTERFACE = TENGB1F0V056 IST2309I ACCELERATED ROUTING ENABLED IST2331I QUEUE QUEUE READ QUEUE IST2332I ID TYPE STORAGE STATUSIST2205I ------ -------- --------------- -------------IST2333I RD/1 PRIMARY 4.0M(64 SBALS) ACTIVE IST2333I RD/2 BULKDATA 4.0M(64 SBALS) ACTIVE IST2333I RD/3 SYSDIST 4.0M(64 SBALS) ACTIVE IST2333I RD/4 EE 4.0M(64 SBALS) ACTIVE

18

TELNET Problem EZZ6035I Summary

TN3270 Server not starting CSM fixed storage constraint and critical Default for QDIOSTG=MAX / READSTORAGE GLOBAL IQDIO inbound workload queuing (IWQ) enabled Each active OSA-Express QDIO Datapath device consumes a large amount of fixed storage (4 x 4 = 16MB). Defining a large number of QDIO devices can cause z/OS Communications Server to consume a significant amount of fixed storage


19

TELNET Problem EZZ6035I Things to think about when enabling IWQ

When configuring a large number of QDIO devices, review the following parameters:

VTAM QDIOSTG start option READSTORAGE specifications in the TCP/IP profile FIXED MAX specification in the IVTPRM00 parmlib member

Additional CSM 4K DSPACE64 storage is used when the QDIO inbound workload queueing is implemented

Display the VTAM TRLE name that is associated with the interface to determine the amount of CSM storage that is being used


20

Thank You for attenting this session!

Your feedback is important:This was session 15511 at SHARE in Pittsburgh 2014IPCS for Networkers Part 2 of 2

IBM SmartCloud: Georg Senfleben [email protected]

Learn more about IPCS at the IPCS Network Debugging Workshop (ZONDE0DE)Info at IBM developerWorkshttps://ibm.biz/IPCS4Networkers

Learn more about IPCS at the IPCS Network Debugging Workshop (ZONDE0DE)Info at IBM developerWorkshttps://ibm.biz/IPCS4Networkers

http://www.ibm.com/cloud-computing/social/us/en/

Slide 1page15page16page17page18page19page20page21page22page23page25page24page26page27page28page29page30page31page32Slide 20

IPCS for Networkers Part 2 of 2: Tales from the Crypt: How we … · 2014. 8. 6. · Session 15511...

Documents

Transcript of IPCS for Networkers Part 2 of 2: Tales from the Crypt: How we … · 2014. 8. 6. · Session 15511...