10 Techniques To Know To Become An Amazing …Know Your Peak MIP Usage Hours 20 Top 25 MIPS Daily...

10 Techniques To Know To Become An Amazing Developer.

Lori Zaremba & Tina Sullivan Progressive Insurance

Session Code: E05 Wed, May 01, 2013 (09:15 AM - 10:15 AM) | Platform: DB2 Z/OS

Copyright 2013 Progressive Insurance. May not reproduce without permission

Click to edit Master title style

Objectives

You will be able to: Enhance your skillset and help you become a more value resource to your

company.

You will be able to show what DB2 uses behind the scenes and how it affects SQL.

Give upfront details to your DBA on issues you encounter creating an excelled relationship with your DBAs.

You will be able to understand that there is much more to DB2 than writing SQL.

View our favorite new features and functionalities that V9/V10 has to offer.

2



Agenda

Speaker background

Our (access) path to becoming amazing Top 10 list to become amazing

We did succeed at becoming amazing

Our favorite new features from V9/V10

Q & A.

3



Speaker Background

Lori Zaremba Lead Application Developer/Performance Tester,

Progressive Insurance 17 years experience working with DB2 4 years working with SQL Server

Tina Sullivan Application Programmer Developer for 25 years in Billing

4

Sorry, no Flo this year at IDUG.

But she promised to upgrade her ‘tricked out name tag’ in our honor, though…



About us: DB2 at Progressive

5


Billing

Two Data Centers

Cleveland Colorado Springs Agents

Quoting

Data

Warehouse Claims

Billing


About us: DB2 at Progressive Skip level release CM V10 Sept 22, 2012

NFM V10 March 10, 2013

6

546 Databases

10 Way Data Sharing Env.

31,000+ Tables

Several Tables over 1 BILLION Rows

60+ Terabytes of data



About us: Progressive Billing System

Probill DB2 Facts

7

2609

Total

Programs

425

Total

Tables

429

Total

Views

661

Total

Indexes

17

Stored

Procs

8

Sequence

Objects

15

Total

Triggers 11 terabytes of Production Data



Our (access) Path to Becoming Amazing

8

Pilot group

Need for DB2 education

Met weekly

Review current issues

Review prod SQL changes

Created DB2 standards

Shared Techniques



AMAZING BETWEEN (1 and 10)

9

This is you before


DB2MSTR Log

10


For Example

-911

Go to Detector

To Find the “Loser”

DB2MSTR Shows

Who Lost

Good Resource For Contention Detection

What environment your jobs run in.

Ours is DB2*MSTR.

Check with your DBA to find out yours.


DB2MSTR Log

• 10.31.51 STC64713 DSNT375I =DB2Q PLAN=PMTCX000 WITH 604

• 604 CORRELATION-ID=POOLMQ960001

• 604 CONNECTION-ID=CT62

• 604 LUW-ID=NETPRO.DB2DB2Q.CAC51A0C68AF=54110

• 604 THREAD-INFO=CT60T:*:*:POOLMQ96

• 604 IS DEADLOCKED WITH PLAN=BILCX000 WITH

• 604 CORRELATION-ID=POOLPBDW0013

• 604 CONNECTION-ID=CT5A


• 604 THREAD-INFO=CICSUSER:*:*:POOLPBDW

• 604 ON MEMBER DB2Q

• 10.31.51 STC64713 DSNT501I =DB2Q DSNILMCL RESOURCE UNAVAILABLE 605

• 605 CORRELATION-ID=POOLMQ960001

• 605 CONNECTION-ID=CT62


• 605 REASON 00C90088 (reason = deadlock )

• 605 TYPE 00000302 (type = tablespace page)

• 605 NAME BILDBBCM.BCMBBPT .X'00F0061E' (Database,tablespace,page)

11


Represent the threads in contention

Represent the threads in contention

Identifies the thread that was terminated

Information about

object in contention

1


Reorg with Discard

Combines archiving (save data to a new file)purging and reorganizing in a single step

Stability

Save time vs. writing a program

Little or no maintenance once it is step up

Execution – runtime is faster

Processing and programs run faster

12



Reorg with Discard Decrease in run-time for several programs.

Example where DB2 CPU time dropped about 75% for program ECD.

13



SQL - Study your Queries a Lot • Main task of the DB2 review team

• Our team reviews every New/Change of SQL statements that is put in our programs

Modifying the SQL and run Studio for Explains

14

Modifying the SQL and run through Data Studio for Explains

Stage 1/2/3

View when the

SQL will run

Multi Row Fetch

with Cursors Inlist vs listing

variables out

Look at the context

Rewrite the queries to run faster

Peak hours of the day

Most common one first

IN (‘a’, ‘b’, ‘c’) Less trips to DB2

$/$$/$$$..



SQL - Study your Queries a Lot SELECT MAX(POL_EFFECTIVE_DT)

INTO :A-POL-EFFECTIVE-DT

FROM POLICY_TABLE

WHERE POLICY_NO = :WS-POLICY-NO

AND STATUS_CD = 'C'

IF SQLCODE EQUAL ZERO

GO TO 0340-GET-MAX-BPT-EX

END-IF.

15

0340-GET-MAX-BPT.

SELECT MAX(POL_EFFECTIVE_DT)

INTO :B-POL-EFFECTIVE-DT

FROM POLICY_TABLE


AND STATUS_CD NOT IN ('W','Q')

SELECT

(SELECT IFNULL (MAX(POL_EFFECTIVE_DT), '9999-12-31')

FROM POLICY_TABLE

WHERE POLICY_NO = :WS-POLICY- NO

AND STATUS_CD = 'C')

,(SELECT IFNULL (MAX(POL_EFFECTIVE_DT), '9999-12-31')

FROM POLICY_TABLE


AND STATUS_CD NOT IN ( 'W', 'Q'))

INTO :WS-POL-EFF-DT1

,:WS-POL-EFF-DT2

FROM SYSIBM.SYSDUMMY1

Improves Performance As It Now Only Has 1 Trip To DB2


Production Data Change Process Advantages of Data Change Process

• Provides Confidence – process checks each sql statement for correct syntax

• Checks for ‘where’ statements

• Limits the number of rows affected to prevent outages (ie. Reaching MAXNMLKS)

• Sox Compliant - using developers initials, date and time, # of rows affected

• Auditable can quickly find who was changing what

• Recoverable provides backups of all updated and deleted data

16



Production Data Change Process

17


Production Data Change Process

17


SQL

SOX AUDIT FILE

BACKUPS

AUDIT RPT

SQL

SQL FOR BKUPS

SQL EDIT RPT Validate SQL

Target Table

Execute SQL

Manual

Verification 1 – Programmer codes SQL statements to change the target

table (update/insert/delete)

2 – Programmer requests run of the batch job to validate the

update/insert/delete SQL

3 – Programmer manually verifies the output of the

validation job

4- If the output from the validation job looks good, the programmer

requests that the Execute SQL job be run


Know Your Peak MIP Usage Hours

18

Problem: Spike in MIPs

driving need to purchase

engine upgrades

Peak hours 9 – 11 am 2 – 4 pm

Charged based on highest peak hours

Processes that

can run off-

peak hours

IDENTIFY… Peak MIP abusers

Programs for

performance

improvements


Ad hoc queries/job

that can run off

peak hours



19


.

Removed SQL no longer

needed

Revised SQL &

scheduling

Run during Off-Peak

hours



20

Top 25 MIPS Daily Consumers for Probill DB2 Packages

M-F, Peak Prime Hours (ranked by last day actuals)

0

500

1000

1500

2000

2500

12

/16

/20

10

12

/17

/20

10

12

/20

/20

10

12

/21

/20

10

12

/22

/20

10

12

/23

/20

10

12

/24

/20

10

12

/27

/20

10

12

/28

/20

10

12

/29

/20

10

12

/30

/20

10

12

/31

/20

10

01

/03

/20

11

01

/04

/20

11

01

/05

/20

11

01

/06

/20

11

01

/07

/20

11

01

/10

/20

11

01

/11

/20

11

01

/12

/20

11

01

/13

/20

11

01

/14

/20

11

01

/17

/20

11

01

/18

/20

11

01

/19

/20

11

01

/20

/20

11

01

/21

/20

11

01

/24

/20

11

01

/25

/20

11

01

/26

/20

11

01

/27

/20

11

01

/28

/20

11

Application groups based on AuthID, PackageID, Collection ID. Select "refresh" for current data.

25 - _OTHER

24 - GS2_BIL

23 - PFN_BIL

22 - QLG_BIL

21 - PPS_BIL

20 - PPA_BIL

19 - INS_BIL

18 - PNA_BIL

17 - BPT_BIL

16 - TAB_BIL

15 - BIV_BIL

14 - BED_BIL

13 - PUB_BIL

12 - PNC_BIL

11 - LRT_BIL

10 - DEL _BIL

9 - BBH_BIL

8 - AX _BIL

7 - AD _BIL

6 - PY _BIL

5 - BMS_BIL

4 - PSD_BIL

3 - AA _BIL

2 - BBP_BIL

1 - SEL_BIL

RESULTS:

Peak MIPS for the month was barely over 2000 compared with 2900 a

couple months prior


Savings of $63,000 a month


Claimers/Drainers How They Affect Maintenance

What is a claim?

notification to DB2 that an object is being accessed (reading/insert/update/delete)

What is a drain?

process of allowing existing claims on an object to finish while preventing new claims from being taken on that object

Maintenance tasks need to DRAIN the CLAIMers.

If they cannot DRAIN the CLAIMers , the task can fail

What types of maintenance tasks are affected by claims?

On-line utilities: REORG

DDL: alter table structures

Rebinds

To help ensure success:

Commit to release claims even read only programs hold claims

Investigate using DRAIN(ALL) vs. DRAIN(WRITERS) for reorgs – V10 recommendation

21



RTS – Real Time Statistics for developers!

Determine what tablespaces changed after test:

Unit testing - see what has changed

Stress testing - only reload those tablespaces that changed

Get information about :

• tablespaces/indexspaces – like #rows, extents, active pages, and when: load replace/copy/runstats/reorg were last run

• how many rows were impacted by a program or process - SYSIBM. SYSTABLESPACESTATS column TOTALROWS (count before/after)

• hash access = SYSIBM.SYSTABLESPACESTATS column REORGHASHACCESS:

will indicate number of times data is read using hash access (since initial create or the last reorg) – if it is not zero – then hash is being accessed

DB2 V9 - column that indicate when an index or program is last used

Index last used – SYSIBM.SYSINDEXSPACESTATS, column LASTUSED

Program last used – SYSIBM.SYSPACKAGE , column LASTUSED

* Programs have to be bound under V9

22



RTS – For Developers Example - Added 1 row to a table and then deleted it. Then ran: Sample query (stress testing)

==========================

SELECT A.DBNAME, A.NAME AS TABLESPACE ,INTEGER(SUM(TOTALROWS)) AS TOTROWS,

INTEGER(SUM(A.REORGINSERTS)) AS REORGI, INTEGER(SUM(A.REORGDELETES)) AS REORGD,

INTEGER(SUM(A.REORGUPDATES)) AS REORGU, INTEGER(SUM(A.STATSINSERTS)) AS STATSI,

INTEGER(SUM(A.STATSDELETES)) AS STATSD, INTEGER(SUM(A.STATSUPDATES)) AS STATSU

FROM SYSIBM.TABLESPACESTATS A

WHERE A.DBNAME IN ('BILDBBCM','BILDBPRG') AND NAME IN ('BCMBBAI','BCMBBAM')

AND A.UPDATESTATSTIME > '2013-01-02-15.12.00.000000'

GROUP BY A.DBNAME, A.NAME

ORDER BY A.DBNAME, A.NAME

Query output

23



RTS – for developers

How are real time statistics created/collected: V8

Need to Create:

Real time statistics database (DSNRTSDB)

SYSIBM.TABLESPACESTATS (including indexes)

SYSIBM.INDEXSPACESTATS (including indexes)

V9 and higher

Part of DB2 catalog

Big Question? Does RTS drive up CPU

NO DB2 updates RTS counts whether you use them or not

24



Performance testing Fixing a performance issue in development ($) is much less costly than fixing it in production ($$$).

• What we stress test: Changes to critical processes

Changes to fix a performance issue

Performance enhancements

• Measure and compare changes to baseline tests: MIPS

CPU

Run Time

Wait Time

• All measurements are reviewed and signed off prior to installing to production.

25



Performance testing

• Post-production performance review - Performance Instrumentation Database Metrics- Reactive

30 days of detailed information for drilling down to research/solve current issues

Roll-up - Proactive

13 months of roll-up information for trending purposes

• Data originates from: SARQ for batch jobs and job step information

DB2 DBA tables for DB2 program information

Billing infrastructure tables for transaction information

• SQL Server Database using SSRS for reports/charts

26



zIIP’s • What is a zIIP? System z9 Integrated Information Processor (zIIP)

Hardware next to the standard General Processor (GP)

One time cost for hardware

Progressive = 1 zIIP per GP

• Why use a zIIP? zIIP processor mips vs. GP mips lower cost hardware

lower monthly software charges usage on the zIIP processors are not included in our monthly usage based software charges.

27


zIIP’s IBM determines zIIP eligibility - a percentage (approximately 40%) of 'certain' workload allowable to run on the zIIPs.

Tasks that are considered zIIP eligible for zIIP offload Native sql stored procs

Reorgs ** we use reorgs with discard to purge data

Unload/load utilitites

BUT It Depends: If the zIIP’s are busy – the processing will default back to the GP

So: Write native stored procedures (distributed connection) and utilities when possible

28



zIIP’s Production DB2 Subsystem Address Space(s) MIPS Usage

(DB2 Version 8 -vs- DB2 Version 10 – compatibility mode)

CM = The reduction in GP mips is 600 out of 6100 mips from DB2, a savings of about 10%

NFM = prediction of 300-500 MIPS by moving to zIIP processing

29



Committing – Releases Locks Don’t be afraid of commitment

• Doing updates, inserts, and deletes to multiple pages, acquires more page locks.

• A page lock takes a cross-memory service call to acquire • about 250 bytes of memory to track.

• Example: random updates on 1,000 rows, lock as many as 1,000 pages, consuming approximately 250,000 (1,000 * 250) bytes of memory (including the tablespace and table/partition locks).

• Committing causes the cross-memory services call to release the page locks.

30



Committing Makes Unit of Work Changes Permanent

Not COMMITTing Holds (implied) committing until the end of huge unit of work

Holds locks, claims, and log records until unit of work completes

If a rollback is needed can take longer than the actual execution of the program (We’ve seen this )

COMMITTing: Shortens ROLLBACK durations

Casts the changes in concrete

31



Committing COMMITS FREE UP LOG BUFFER SPACE

As you perform inserts/updates/deletes – you create UNDO and REDO log records The UNDO records are created just in case your program issues an explicit or implicit ROLLBACK

The REDO records are created for the RECOVER utility

They can be applied to an IMAGE COPY to bring the data forward from the time of the copy to the point of recovery

At COMMIT: You're telling DB2 that you'll no longer need the log records for ROLLBACK

Log records are removed from memory and the log buffers are flushed to an active log dataset

32



Committing - IPROC/SPROC/UPROC We all Proc!

When DB2 detects repetitive update/insert/select activity within a unit of work it does the following:

Builds insert (IPROC), select (SPROC), and update (UPROC) procedures

These Procs cache information for sequential detection and index lookaside

Reduces the path length for performing the SQL stmt

The result:

reduces CPU overhead and improves performance

How Bind parameters affect the use of IPROC/SPROC/UPROC

RELEASE(DEALLOCATE)

Use for Performance (high batch volume/on-line transactions)

Retains info on SPROCs, UPROCs, and IPROCs after commit - >improving performance

33



Committing - IPROC/SPROC/UPROC We all Proc!

How bind parameters affect the use of IPROC/SPROC/UPROC – Cont

RELEASE(COMMIT)

Use for Availability

Committing too frequently can degrade performance:

• The IPROCs, SPROCs, and UPROCs are destroyed or never built

• Higher overhead of building the PROCS but never utilizing them

To take advantage of performance gains of procs:

• Must use at least 5 to 7 times before commit to pay for the build overhead

• After about the 8th time, the SQL is faster and cheaper

34



Summary

Use the tools we have shown you and you will be a more value resource to your company.

We have shown you what happens with DB2 that you may never see.

Claims/drains, commits, zIIPs,etc.

Try to rewrite any new or changing SQL.

When you call your DBA you can give an abundant amount of information.

DB2MSTR info.

Look at your tables and the data on them.

Reorg with discard, RTS, and MIP usage.

Form a developer DB2 team.

Incorporate a data change process and start performance testing.

35


Top 10 List To Become An Amazing Developer!!

36

This is you now!!



DB2 V9/V10 top 10+ list 1. Hash Access

ability to directly locate a row in a table without having to use an index (1 sync I/O, 0 if memory table) – reduce search CPU expense

2. Native Stored Procs

zIIP offloadable (distributed connections)

3. Index Include

way to delete indexes by combining them

4. Truncate table

quicker way to delete table data (take advantage when reloading billing stress data)

5. Temporal tables

history on the data changes on tables

6. Rid Pool failure

uses memory work files instead of doing a tablespace scan

(** make sure the work files are sized correctly)

7. Access Path Stability

falling back to a previous access path if a new access path is causing performance issues

37



DB2 V9/V10 top 10+ list - Continued

8. Row Numbering

simply assigns a unique number to each row of the result

9. Production Modeling

Improve access path testing by updating the catalog statistics on your test/stress system to be the same as your production system.

10. Index on expression

You can create an index using an expression

11. Order by/fetch first

Available in subselects

12. Merge statement

combining insert/update = less trips to DB2

*Previously may have been accomplished with a conditional INSERT or UPDATE

13. 'ROW CHANGE TIMESTAMP‘

column added to a table that will automatically track the last time a row was updated

* NO application code to capture!

38


Lori Zaremba Progressive Insurance [email protected]

Tina Sullivan Progressive Insurance [email protected]

E05 10 Techniques To Know To Become An Amazing Developer

Evaluate my session online: www.idug.org/na2013/eval


mailto:[email protected]

mailto:[email protected]

http://www.idug.org/na2013/eval

10 Techniques To Know To Become An Amazing …Know Your Peak MIP Usage Hours 20 Top 25 MIPS Daily...

Documents

Transcript of 10 Techniques To Know To Become An Amazing …Know Your Peak MIP Usage Hours 20 Top 25 MIPS Daily...