A modern look inside the black box: Unlock IBM Z with IBM ... · Consumer -driven innovation...

A modern look inside the black box: Unlock IBM Z with IBM Z Operations analytics and SplunkDan Wiegand

IBM

November 2019

Session OP

https://twitter.com/gseukc

https://www.linkedin.com/groups/1741877

https://www.facebook.com/pages/Guide-Share-Europe-UK-Region/174907039241415

The market is entering a new chapter in cloud and digital

Unlock IBM Z with IBM Z Operations analytics and Splunk/ October, 2019 / © 2019 IBM Corporation

Chapter 1

Consumer-driven innovation

Digital/AI experimentation

“User applications”

Public cloud

Chapter 2

Enterprise-driven innovation

Digital/AI embedded in the business at scale

“Mission critical” workloads

Hybrid cloud

56% of customers

have no succession

plan for IBM Z skills

56% of customers

have no

succession plan

for IBM Z skills

75% of customers

say “developing a

more integrated

approach to

operational

resiliency” is the top

priority this year

75% of customers say

“developing a more

integrated approach to

operational resiliency”

is the top priority this

year

IBM Z powers 72% of customer facing apps, but organizational challenges limit ability to support

Forrester 2018 Forrester 2018

Forrester 2018

33% of outages

cost more than $1M

per hour


Your journey to cloud is

impacting all areas of the enterprise

Siloed Operations

4

That’s weird. I saw some

application problems but they

seem to have stopped. Hope it

doesn’t happen again!

IT Ops

I’m starting to see some storage

violations…. I need to fix what’s

happening in the CICS application

quickly…. I hope this isn’t affecting

any of the LOB apps!

CICS SME

Issues identified by

internal operations


Best practice for Enterprise Operations – Hybrid Cloud Visibility

5

Application Performance ManagementEnd to end transaction visibility in Application specific views

Operations AnalyticsOps metrics from across the enterprise in a single location

Deep dive toolingDomain specific tooling for the SMEs

1. Avoid war rooms with swift component isolation

2. Eliminate black boxes with full enterprise visibility

3. Appropriate data for SME hand over

5Unlock IBM Z with IBM Z Operations analytics and Splunk/ October, 2019 / © 2019 IBM Corporation

Proactive Operations

6

I can see a slowdown happening

and not as many CICS

transactions are being

processed. I better call our CICS

SME!

IT Ops

Thanks for the notification. I can see

some storage violations…. I need to

fix what’s happening in the CICS

application quickly before affecting

our customers using the LOB apps!

CICS SMEUnlock IBM Z with IBM Z Operations analytics and Splunk/ October, 2019 / © 2019 IBM Corporation

Issues identified by

internal operations

through enterprise

visibility

Hybrid Cloud Visibility - Breaking down silos

7

I can see a slowdown happening

and not as many CICS

transactions are being

processed. I better call our CICS

SME!

IT Ops

Thanks for the notification. I can see

some storage violations…. I need to

fix what’s happening in the CICS

application quickly before affecting

our customers using the LOB apps!

CICS SME

That’s weird. I saw some

application problems but they

seem to have stopped. Hope it

doesn’t happen again!

IT Ops

I’m starting to see some storage

violations…. I need to fix what’s

happening in the CICS application

quickly…. I hope this isn’t affecting

any of the LOB apps!

CICS SME


IBM Z Operations Analytics


Make Data Availablecollect operational data from

different sources in near real time

Hybrid Cloud Visibilityexamine operational data in a

broader context

Resilient OperationsProactively identify problems in

advance of their occurrence






broader context



IBM Z Common Data Provider and Splunk

10

▪ Stream the widest range of SMF records and Z log data in “near real-time”

▪ Gain insight and eliminate blind spots in hybrid IT Operations

▪ Visualize impacts across your infrastructure from continuously delivering

applications and application enhancements

▪ Maximize Splunk Investment with advanced filtering

▪ Optimized to reduce data and save on ingestion costs

▪ Built to perform and reduce CPU overhead - 100% of runtime components are zIIP

eligible

▪ Save money with the CDP’s fixed pricing model and two vendor solution


New 2.1 release

now available

SMF Data

Log Data

CDP Streaming

API

IBM Z Decision

Support

z/OS Connect

IMS Connect

Extension

Netview

CDP Is Strategic to Enabling IBM Z IT Operations Analytics


IBM CDP

Workload

Scheduler

Common Data Provider - Splunk ArchitectureOptimized for data ingestion


Architecture designed in partnership with Splunk. CSV ingestion focused on reducing total data volume ingested to Splunk.

• Reduce data ingestion costs by 3-8x using the data receiver.

• For a fixed license capacity, stream more data to gain additional insight.

Alternate mechanism is available to stream directly to Splunk using the HTTP Event Collector but does not have the same optimization.

Data Receiver vs HTTP Event Collector


Data Receiver – Best Practice

• Optimized to reduce data volumes 3-8

times lowering Splunk ingestion and

storages costs in addition to network

traffic

• CSV Data format reduces CPU

consumption on the Splunk infrastructure

for data ingestion.

• Requires Splunk Buffered Ingestion

application to be installed in Splunk and a

data receiver to be installed in a distributed

environment. The ingestion application

currently needs updating to support new data

types and add filtering.

HTTP Event Collector

• No data optimization, leverages JSON to stream

key value pairs.

• Key Value Pair format requires additional

processing on the Splunk infrastructure.

• Requires a one time authentication token to

be configured for each index reducing the

coordination between distributed and

mainframe teams. Indexers can be

configured by specific tokens. No need for

installation on distributed environment.

14

Common Data Provider leverages a simple configuration UI

The easy to use configuration tool runs in z/OSMF or the included Liberty and is secured by

RACF for total control of data and subscriber(s).

Just select the data you need… add your data filters, target(s) and go!


System Data Engine – Advanced SMF Data Management

15

• Add new or extend existing data sources through configuration

• Easily stream 3rd party or proprietary data sources through CDPz

• Code updates are not required

• Minimize data streamed to reduce ingestion costs

• Record level filtering – only send records that meet a certain set of conditions

• Field level filtering – only send the parts of the record needed in the target

• Curate Operational Data

• Develop data streams with calculations (like the CICS Summary data stream delivered in IZOA)


Add new records through configurationDEFINE RECORD

16

DEFINE RECORD SMF_IMS_C2C2

VERSION 'CDP.110'

IN LOG SMF

IDENTIFIED BY SMIMSRTY = &IMS_SMF_RECTYPE

AND SMIMSSTY = &IMS_SMF_RECSTYP

AND SMIMSSRC = &IMS_SMF_SRCID

AND SMIMSID1 = 7

--------------------------------------------------

-- Start of SMF header

---------------------------------------------------

FIELDS (

SMIMSLEN LENGTH 2 BINARY,

......

----------------------------------------------------

-- Start of CDP product section

-----------------------------------------------------

SECTION PRODUCT

......

----------------------------------------------------

-- Start of IMS user record

-----------------------------------------------------

SECTION SMF_IMS_C2C2

......

FIELDS (

RECLL LENGTH 2 BINARY, -- Length of log record

RECZZ LENGTH 2 HEX, -- QSAM reserved bits

RECTYPE LENGTH 1 BINARY, -- Record type (X’C2’)

RECSUB LENGTH 1 BINARY, -- Record subtype (x’C2’)

LENGTH 4 CHAR, -- Filler

RECSPL LENGTH 6 CHAR, -- C’SPLUNK’

......

SMF

standard

header

Pointer to

product

section

Pointer to

data

section

Product

Section

Data

Section

Data: ....BB SPLUNK.........."Y....p.$u....p....

Header section Self-defining fields

What’s the layout of user data?

This record definition shows an example

of an IMS user log record “C2C2”

definition

Use the DEFINE RECORD statement to define the SMF record type that describes the layout of the new record

IBM provides

most common

types with the product


Stream only required recordsDEFINE UPDATE

17

Use the DEFINE UPDATE statement to specify how to process a specific record type and how to store the result in the target of the update

Optionally, you can use the update definition to enable the record-level filtering by coding WHERE conditions

DEFINE UPDATE SMF_110_1_KPI_SB

VERSION 'CDP.110 '

FROM SMF_CICS_T

WHERE (SUBSTR(TRAN,1,2) = 'QF' AND INTERVAL(START, STOP) > 2.0)

OR (ABCODEO <> ' ')

OR (ABCODEC <> ' ')

TO &IBM_UPDATE_TARGET

&IBM_CORRELATION

AS &IBM_FILE_FORMAT SET(ALL);

TRAN ... START ... STOP ABCODEO ABCODEC STREAME

D?

LENDING 3 6 18 YES

QF34298 2 6 YES

QF12345 0 1

LENDING 5 8

QF20345 3 4 72 YES

SMF records read from

Exit, Log stream, or

In-memory resource

This update definition

performs record level filtering

on CICS 110 Transaction

Records for specific

transaction that are long

running or end in abend


Stream only required fields in the order you wantDEFINE TEMPLATE

18

Use the DEFINE TEMPLATE statement to change the layout of the record in the following manner:

▪ Create additional output with the same data, or a subset of the data.

▪ Change the order of the fields.

▪ Alter the formatting of the fields

DEFINE TEMPLATE SMF_070 FOR SMF_070

ORDER (

SMF70LEN,

SMF70TME,

SMF70SID,

SMF70LAC,

SMF70WLA

)

TO &IBM_UPDATE_TARGET

&IBM_CORRELATION

AS &IBM_FILE_FORMAT;

Correlator

SMF70TME

SMF70SEG

SMF70RTY

SMF70LEN

SMF70DTE

SMF70SID

SMF70SSI

SMF70STY

......

SMF70WLA

SMF70LAC

......

SMF70LEN

SMF70TME

SMF70SID

SMF70LAC

SMF70WLA

Default layout of update definition

Layout of template definition

This template definition

only sends 5 fields from

the original record to

the subscriber


IBM Z Common Data Provider zIIP Savings for SMF Data


Performance result based on IBM Internal tests. Mainframe configuration: z14, 4GB heap, z/OS 2.3, zIIPs online , 100GB memory.

Sending a mixture of SMF 101 and SMF 110 at a rate of 300GB/day. Reading data from an SMF In-Memory Resource. CDP version is ZCDP 2.1.

This data was collected by testing in our environment and is not necessarily indicative of specific customer environments. As always, actual CPU consumption

may vary.

19

CPU Consumed

General CPU zIIP

99.79% of workload is

offloaded to zIIPfor processing SMF data -

saving a significant amount of

processing on General CPUs

IMS Log and RMF III Record Support

20

IMS Log Records supported

▪ x’07’, x’0A07’, x’08’, x’0A08’ (accounting records)

▪ x'10’ (security violation records)

▪ x’56FA (transaction level statistics records)

▪ x’CA01’, x’CA20’ (IMS Performance Analyzer Transaction Index records)

▪ IMS User Log records

RMF III Record Support

▪ Central Processor Complex (CPC) Report







broader context




22

• Leverage IBM Z Common Data Provider to gain access to IBM Z operational data in near real time.

• Curated Data Definitions help you get rapid value and make sense of raw operational data

• Gain hybrid cloud visibility by integrating operational data with data from the rest of your enterprise.

• Expand domain expertise through domain-specific out-of-the-box analysis, insights, and expert advice including z/OS, WebSphere, DB2, CICS, IMS, MQ, Network, and Security

• Insights are available as a native Splunk application

• IBM Z Specific KPIs to quickly integrate with Splunk ITSI

• Reduce outages by proactively detecting operational anomalies and notifying operations when they occur


New 4.1 release

now available

The Value of IBM Z Operations Analytics Curated Data


“This will save me 3 months, 6 months, or even a year of

development effort to build this”

Lead Technical Architect

Data Curation begins to build insight into the raw SMF operational data by providing predefined data

calculations and field filtering to produce new key performance indicators.

For example:

• Summary records for Db2 and CICS – 1 record per minute vs thousands per second

• Leveraging most important fields for Db2 / CICS over sending hundreds of extraneous fields

• Calculation of common variables such as CPU utilization, paging rates, I/O rates

• Scenario based records, e.g. security to quickly see login, command authorization, USS file

access, data set access, etc

• Out of the box Splunk and ELK support for additional logs e.g. USS syslog, CICS MsgUser and

EYULOGS, NetView logs, WAS Sysprint and Sysout.

Dashboards and Searches – Insight into Operational DataAvailable as a native application on Splunk

24

Domain-specific ‘Quick Searches’available out-of-the-box

• Based on the combined experience of subject matter experts, support teams and customers.

• Immediate value out of the box.

Subsystem Dashboards

z/OS, WebSphere, DB2, CICS, IMS, MQ, Network,

and Security


IBM Z Operations AnalyticsIBM Z KPIs for Splunk IT Service Intelligence


IBM best practice

KPIs for CICS, DB2,

IMS, and MQ

delivered out of the

box with IBM Z

Operations Analytics

for direct use in

Splunk IT Service

Intelligence.

Mobile and online

applications are

experiencing serious

issues.

From Splunk ITSI (thanks

to the integration of IBM Z

data) you can easily see

that a problem on your

mainframe applications is

impacting your entire

business.

Art of the Possible – IBM Z and Splunk IT Service Intelligence Correlate Business Information with IT Infrastructure


Art of the Possible – IBM Z and Splunk IT Service Intelligence Correlate Key KPIs and Drill Into the Raw Data

Deep dive in the

log related to

that service to

quickly take

down to the root

cause of the

problem.


IBM Z Operations AnalyticsDeeper IBM Z Insight – DB2 Drilldown

Enterprise Dashboard

System CPU

Lock Suspend / Timeout

Job CPU

System Dashboard

Subsystem Transactions

Subsystem CPU

Subsystem Lock / Latch Wait

Subsystem IO Wait

Subsystem GETPAGE Req.

Subsystem Dashboard

Thread Transactions

Thread CPU

Thread Lock / Latch Wait

Thread IO Wait

Thread GETPAGE Req.


Our application is having some issues.

If I take a quick look, it seems that we

are getting some lock conflict

messages. Let me see if I can find

more details for our Db2 SME.

IBM Z Operations Analytics - Problem InsightsAutomatically surface important messages and anomalies

29

Provides easy to read problem summary and suggested actions for problem resolution.

Total number of Problem

Insights found per Sysplex

Count of this message

over the timeframe

Link to search for this

message in this time

period

Click to show the suggested

actions for this message29Unlock IBM Z with IBM Z Operations analytics and Splunk/ October, 2019 / © 2019 IBM Corporation

Suggested ActionsNext steps for a given problem insight

30

Link to the Knowledge Center

for this message ID

Suggested Actions to

investigate/resolve the issue

Steps can be configured to

align with your environment.



Solution Demo

IBM Z Decision Support for Capacity Planning


z/OS ConnectDevOps Visibility Through API Visualization


http://ibm.biz/zosconnect-analytics

http://ibm.biz/zosconnect-analytics

IMS Connect ExtensionsVisualization Available on Splunkbase

35

Transaction by IMS

Data Store

Original and Target IMS

Data Store35Unlock IBM Z with IBM Z Operations analytics and Splunk/ October, 2019 / © 2019 IBM Corporation

Client Success Story: Rabobank

Instant Payments Infrastructure leveraging IBM Z data and

Splunk IT Service Intelligence

GOAL: Fully process payments in 5 seconds 24/7/365

Visualize and predict disruptions to the business before

customers are impacted

Reduce mean time to repair saving time in war room

situations

Watch a replay of a Rabobank’s presentation at Splunk .conf

2018:

http://ibm.biz/IBMZVisibilityConf2018


37

More InformationIBM Z Common Data Provider for Splunk

IBM Common Data Provider product page ibm.biz/CDPzInfo Product Page

IBM Z Operations Analytics product page ibm.biz/IZOAInfo Product Page

Rabobank Splunk .conf 2018 Presentation ibm.biz/IBMZVisibilityConf2018 Rabobank Splunk .conf Presentation

Common Data Provider on Splunkbase ibm.biz/CDPzPartner Product Information on Splunkbase

Common Data Provider Splunk Dashboards ibm.biz/CDPzSamples Sample CDP Dashboards

Common Data Provider Splunk Integration ibm.biz/CDPzSplunkWhitePaper White paper

IBM Z ITSM Newsletter ibm.biz/zITSMNewsletterSubscribe IBM ITSM Newsletter


http://ibm.biz/CDPzInfo

http://ibm.biz/IZOAInfo

http://ibm.biz/IBMZVisibilityConf2018

http://ibm.biz/CDPzPartner

http://ibm.biz/CDPzSamples

http://ibm.biz/CDPzSplunkWhitePaper

http://ibm.biz/zITSMNewsletterSubscribe

Please submit your session feedback!

• Do it online at http://conferences.gse.org.uk/2019/feedback/op

• This session is OP

A modern look inside the black box: Unlock IBM Z with IBM ... · Consumer -driven innovation...

Documents

Transcript of A modern look inside the black box: Unlock IBM Z with IBM ... · Consumer -driven innovation...