Everything a DB2 for z/OS Application Developer needs to ...• Stored Procedure –SQLPL, ... Index...

© 2016 IBM Corporation

Everything a DB2 for z/OS Application Developer needs to

know about the DB2 engine in 3.6 million milliseconds…

Randy Ebersole

DB2 Technical Specialist

DB2 Lab Services, IBM Analytics

#ibmiod

Agenda

• The Basics

• The DB2 Engine

• DB2 SQL Processing

• Stored Procedures

• DB2 Logging

#ibmiod

The Basics

• We assume the application wants to access data in DB2 for z/OS

• Regardless of the “driver” of the request for data, an SQL statement will execute

o Drivers:

• CICS – Programs that contain SQL

• DDF – Dynamic SQL

• Batch – Wide range of SQL processing

• Stored Procedure – SQLPL, COBOL, JAVA, contains SQL for data access

• RESTful API

– New with DB2 12

#ibmiod

DB2 Objects

Nonpartitioned Tablespace

Table T1 Table T2

Index Space

Index X1

Index Space

Index X2

Partitioned

Tablespace

Table

Part 1

Index Space

Part 2

Part 3

Partitioning

Index Part 1

Part 2

Part 3

Volume

4 Volume

25

Volume

6

Volume 3

Volume

2Volume

1

Storage

Group

G2

Storage

Group G1

Database A

Database B

#ibmiod

DB2 Engine

DBM1 - Data

MSTR - Recovery

IRLM – Locking

CICS

Application

DDF

SELECT COL1, COL2, COL3, COL4

FROM ACCT_TABLE

WHERE ACCT_NBR = ?

SQL

Data Returned

#ibmiod

DB2 Engine Components

• Application

o SQL processing, Synch I/O, Buffer updates, Lock requests, Logical logging, Global lock requests, GBP reads

• DDF - DIST

o Distributed Transaction Manager, Connection and DBAT management

• SYSTEM SERVICES - MSTR

o Archiving, BSDS processing, Physical log write/log prefetch, Update commit, Checkpoints, Backouts, Thread deallocation, incl. page P-lock unlock

• DATABASE SERVICES – DBM1

o Dataset Open/Close, Preformat Extend, Deferred write, Prefetch read, Parallel child tasks, DBM1 Full/System Contraction, Castout, P-lock negotiation, AsyncGBP write, GBP checkpoints, IRLM and XES global contention, Notify Exit, Delete Name*

• IRLM

o Error checking Management, Deadlock detection, Local IRLM lock/latch contention, P-lock negotiation, Async XES request*

6

#ibmiod

Program Execution

• Program enablement

o Getting the DB2 “pieces” in place for execution

o Some differences depending on access method

• Dynamic SQL, Stored Procedures, package use

• Thread Processing

o Allocation, Deallocation, REUSE

• SQL Processing

o Access or “UPDATE” of the data

#ibmiod

DB2 Response Time Flow

CICS/IMSTSODDF

Elapsed Time

1st SQL call

End of Call

2nd SQL call

End of Call

Class 2 Time

Class 2Time

Class 1Time

ProgramStart

ProgramElapsedTime

ProgramEnd

End UserResponseTime

Enter

Receive

“Network

“network”

#ibmiod

Performance Evaluation Basics

• Elapsed Time

o Waiting for what ?

• Locks

• I/O

• Other

• CPU

o SQL Workload driven

• Access path related

#ibmiod

SQL Access Path Impact

• A chosen access path drives the cost of execution

o Pattern of access drives Getpage activity

• Index pages

• Data pages

o Prefetch usage/volume

o Understanding predicate filtering

• Knowledge of application data

#ibmiod

DB2 Catalog Tables

DB2 Optimizer

CPU Cost Traversing Rows and Pages

Applying Predicates(RDS vs. DM)

Cost of sorting

"Least Cost Access Path"

I/O Cost TS Scan vs. Index Scan

Join Sequence Test

Index Scan (MATCHCOLS,

CLUSTERRATIOF)

Work File Cost

=

Access Path Selection

Valid SQL DB2 Internal Cost

Formulas

#ibmiod

Page Access Review

Leaf Pages

Data Pages

Index

. . .rows

Root Page

Non-leaf Pages

#ibmiod

SQL Considerations


FROM ACCT_TABLE

WHERE ACCT_NBR = ?

SELECT COL1, COL2, COL3, COL4, COL5, COL6, COL7

FROM ACCT_TABLE

WHERE ACCT_PAY_DATE BETWEEN ? AND ?

AND STATUS IN (?,?,?)

ACCT_TABLE

IX1 ACCT_NBR

IX2 ACCT_PAY_DATE

ACCT_INFO

#ibmiod

Page Access for SQL


FROM ACCT_TABLE

WHERE ACCT_NBR = ?

• Index access via a unique key

• Data page access required

• 3 Index getpages

• One data page access

#ibmiod

Page Access for SQL


FROM ACCT_TABLE



• Index access – scan of some number of non-leaf page and leaf pages

• Data page access required

o To qualify STATUS +

o To retrieve qualified rows

• Getpage Estimate ?

#ibmiod

Consideration for DB2 SORT

• RDS SORT invoked for SQL Requirements

o ORDER BY

o GROUP BY

o DISTINCT

o UNION (WITHOUT ALL)

o Some SUBQUERIES


FROM ACCT_TABLE



ORDER BY STATUS

• Special Sort Object

o Like a Tablespace

• GETPAGE Requests

• I/O

o Bufferpool Use

#ibmiod

Common Issues related to predicates

• SQL Predicate Related

• ....AND ACCT_TYPE = ?

• ACCT_TYPE is not in index

• Data page/ row filtering required to process SQL

• Stage 2 Predicate Usage

• COL BETWEEN COL1 AND COL2

• T1.COL1 = T1.COL2

• Stage 1 not indexable predicates(evaluated at stage 1 processing, during index screening, or after data page access)

• COL <> value

• COL NOT BETWEEN valA AND valB

• COL NOT IN (list)

#ibmiod

Common Issues related to data

• Date range check

• WHERE DATECOL BETWEEN ? AND ?

• Search

• WHERE LASTNAME LIKE ?

• And there is no management of how the host variable is populated

• Example - 'S%'

#ibmiod

Row Processing

Row 1 Row 2 Row 3

Row 6 Row 9 Row 12

Row 26 Row 44 Row 60

Row 2

Row 26

Row 60

Page 1



Row 226 Row 474 Row 690SQL

Row 27

Row 32

Row 690

Page 2


Row 365 Row 591 Row 712

Row 726 Row 874 Row 990

Row 19

Row 29

Row 365

Page 3


FROM ACCT_TABLE

WHERE ACCT_NBR = ?Examined Processed

#ibmiod

SQL Call

Buffer Pool Processing

BP1

• GETPAGE FOR

o DATA PAGES

o INDEX PAGES

• NOT FOUND ?

o I/O WAIT

DASD

BP7 DASD

DB2 Sort work

#ibmiod

TABLE T1 TABLE T2

Composite

Composite

TABLE T3

OUTER TABLESor

COMPOSITE TABLES

INNER TABLESor

NEW TABLES

Join Processing

#ibmiod

Index Structure

Root

Page

Non-Leaf

PageNon-Leaf

Page

Non-Leaf

PageNon-Leaf

Page

Non-Leaf

Page

Leaf

Page

Leaf

PageLeaf

PageLeaf

Page

Leaf

PageLeaf

Page

Leaf

PageLeaf

Page

Offset Key1

Key3...rid

Key1...rid1

Key2...rid

Key Value RID

RID = RID Prefix (1 byte) + RID (4 bytes)Offset Key2Offset Key3

#ibmiod

Indexes

• Indexes are not always the answer

o Take additional space

o All indexes changed on INSERT and DELETE

• 3 - 5 + % DB2 CPU overhead

o UPDATE may change index

• Indexed columns should not be in SET statement

o MERGE can experience the additional cost

#ibmiod

Workload Considerations

• Complexities of transactions are a major consideration

o SQL processing on common objects

o SQL volume

• Impact to I/O patterns

• Locking

• other

• Outside influences

o CPU

o I/O workload

o Other data store access

#ibmiod

What do we know about the data ?

• About the Data

• Total Rows, Total Pages

• Data is for a year, a month, for last 20 years

• Reorg Frequency

• About the Access

• Common, very specific data retrieval

• Searches(wild card – LIKE )

• UPDATE/INSERT/DELETE

#ibmiod

SMF – Basics

• SMF (System management facilities) collects and records system and job-related information

• SMF collects data, writes to SMF datasets, with a periodic Dump to Dump datasets for ongoing use

• For DB2:

o Record Type 100 – Statistics - DB2 writes record type 100 to record data collected at event monitoring points

o Record Type 101 – Accounting - DB2 writes record type 101 to account for resources during a transaction

o Record Type 102 - DB2 writes record type 102 to record performance information (Audit also)

#ibmiod

• Accounting class detail has information that provides details related to application programs, usually counts and times:

o Start and stop times

o Commits and aborts

o SQL statements are issued

o Bufferpool requests

o Locking events

o Processor resources consumed

o Thread wait times for various events

o RID pool processing

o Distributed processing

o Resource limit facility statistics

Accounting Information

#ibmiod

DB2 Times – Page 1

AVERAGE APPL(CL.1) DB2 (CL.2)----------------------------------------------------------------ELAPSED TIME 5:22.25677 4:13.82966NONNESTED 5:22.25677 4:13.82966STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

CP CPU TIME 1:19.53273 1:17.35850AGENT 1:16.91287 1:14.73884NONNESTED 1:16.91287 1:14.73884STORED PRC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000PAR.TASKS 2.619862 2.619665

SECP CPU 0.000000 N/A

#ibmiod

DB2 Times – Page 2

AVERAGE APPL(CL.1) DB2 (CL.2)----------------------------------------------------------------SE CPU TIME 5.724770 5.724770NONNESTED 0.000000 0.000000STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 5.724770 5.724770

SUSPEND TIME 0.000000 2:45.71139AGENT N/A 2:12.25768PAR.TASKS N/A 33.453715STORED PROC 0.000000 N/AUDF 0.000000 N/A

NOT ACCOUNT. N/A 41.108370DB2 ENT/EXIT N/A 671802.22

#ibmiod

DB2 Stored Procedures

• Allows for defined access to DB2

• External Stored Procedures

o available languages on the z/OS server are: COBOL, PL/I, C, C++, Assembler, REXX, and JAVA

o Looks like other SQL applications. Including use of:

• static or dynamic SQL statements, IFI calls, and DB2 commands

o Except for REXX, pre-compile, compile, link, and bind the host source program to create the appropriate load modules and packages

o Definition and program logic are two separate components

o Runs through Workload Manager address spaces

• Native Stored Procedures

o Coded in SQL/PL

o Definition contains the SQL logic

o Execution path is more efficient

• NO WILM path, runs internal to the DB2 engine

#ibmiod

Locking

• A process is suspended when:

o A lock request is requested on an object that is already locked by another concurrent process and The two locks are not compatible

o The suspended process temporarily stops running. Incoming lock requests are queued in the order of arrival

• The suspended process resumes running when:

o All concurrent processes that hold incompatible locks release them

• The suspended time is the lock wait time

o The requesting process times out or deadlocks and the process resumes to deal with an error condition.

#ibmiod

• What’s in The DB2 Log

o Unit of recovery

o Checkpoint data

o Database page set control records

o Other miscellaneous stuff!

• Unit of Recovery Data

o Type of activity (Insert, Update, Delete)

o Before and after images of rows/columns

o Redo and undo records

• How DB2 handles rolling back a change

DB2 Logging

#ibmiod

• Change activity level and tracking

• Report activity affecting DB2 catalog objects

• Display INSERT, UPDATE, and DELETE activity

• Application RI reporting

• Auditing of DB2 update, insert, delete activity

• Recovery of DB2 data

• Replication of DB2 activity

• DDL tracking and reporting

o Auditing Catalog Changes

• Translate the activity to

o GRANT, REVOKE

o CREATE, ALTER, DROP

HOW TO LEVERAGE THE DB2 LOG

#ibmiod

Summary

• Think about the SQL and the processing and what it will need to do

o Consider the data characteristics

o It’s Not Magic

• Understand the performance possibilities

o Even at a high level

o What is Normal (or not !)

#ibmiod

RANDY EBERSOLE

[email protected]

Everything a DB2 for z/OS Application Developer needs to know about

the DB2 engine in 3.6 million milliseconds…

Everything a DB2 for z/OS Application Developer needs to ...• Stored Procedure –SQLPL, ... Index...

Documents

Transcript of Everything a DB2 for z/OS Application Developer needs to ...• Stored Procedure –SQLPL, ... Index...