Tainted LOB

“Looking for LOB in all the wrong places”

or “Tainted LOB”

Experiences with Large Objects in DB2 for z/OS

Marcus Davage

GSE 2014

Session: CC

This presentation is based on my personal views and are not in any way endorsed by Lloyds Banking Group.

None of the information contained within presentation can be used for reference or for any other purpose by another company or organisation.

• Introduction – What LOBs are and how they work

• Experiences – Naming conventions

– Using utilities

– Space, the final frontier

– Performance

• DDL parameters

• Inline LOBs

• Conclusions – Prepare

– Plan

Syn

op

sis

• Large data objects that cannot be

contained in conventional data types

• VARCHAR 32K increasingly became

not big enough

• Appeared in DB2 v6

• Each new release brings new

improvements (and new headaches)

Wh

at a

re L

OB

s?

• One LOB table space per LOB column

per partition

• One auxiliary table and index per table

space

• SELECT LOBCOLUMN FROM T1

– Done under the covers via base row and

index

• Separate from base table space

– AUX YES in IBM utilities

Wh

at a

re L

OB

s?

• One LOB table space per LOB column

per partition

• One auxiliary table and index per table

space

• SELECT LOBCOLUMN FROM T1

– Done under the covers via base row and

index


– AUX YES in IBM utilities

• CHECK LOB utility very handy!

Wh

at a

re L

OB

s?

• Implicitly created ROWID column

appended to end of row

• Table is incomplete until additional

objects created:

– TABLESPACE for LOB

– AUXILIARY TABLE for LOB

– INDEX for auxiliary table

• CURRENT RULES special register

Wh

at a

re L

OB

s?

Ta

ble

s w

ith 1

LO

B

Base table space

Aux Index

LOB TS

Row 1

Row 2 LOB1

LOB2

Ta

ble

s w

ith >

1 L

OB

Col1

Table

Aux 1 Index

LOB1

TS

Col2

LOBCol1

LOBCol2

LOB1 LOB2

LOB2

TS

Aux 2 Index

Experiences

Your mileage may vary…

• Many major projects now using LOBs

– Huge demand for DASD

• BLOBs

– Generally already-compressed and/or

encrypted data

• CLOBs

– True character data

– XML storage (for those whom XML data

types are yet arcane)

Ub

i su

mu

s?

PROJECT TABLESPACE MB GB

TYPE

DB2PDB NORMAL 23376 22

DB2PDB PBR 137148 133

OTHER NORMAL 251338 245

OTHER PBG 665 0

OTHER LOB 3444 3

OTHER XML 25 0

OTHER PBR 137242 134

SYS1 NORMAL 69098 67

SYS1 PBG 71509 69

SYS1 LOB 1522069 1486

SYS2 NORMAL 1877 1

SYS2 PBG 285 0

SYS2 LOB 296548 289

SYS2 PBR 19802 19

SYS3 NORMAL 14 0

SYS3 PBG 0 0

SYS3 LOB 7126 6

SYS3 PBR 2138411 2088

SYS4 NORMAL 2753 2

SYS4 LOB 381968 373

Curren

t usag

e

SELECT TYPE, COUNT(*)

FROM SYSIBM.SYSTABLESPACE

WHERE YEAR(CREATEDTS) >='2006'

GROUP BY TYPE ;

---------+---------+---------+-

TYPE

---------+---------+---------+-

1652

G 323

O 1110

P 6

R 70

---------+---------+---------+-

Cu

rren

t usa

ge

WITH TS (CREATED) AS (

SELECT YEAR(CREATEDTS)

FROM SYSIBM.SYSTABLESPACE

WHERE TYPE = 'O'

AND YEAR(CREATEDTS) >='2006'

)

SELECT CREATED, COUNT(*)

FROM TS

GROUP BY CREATED

WITH UR ;

---------+---------+---------+-

CREATED

---------+---------+---------+-

2006 1

2010 15

2012 180

2013 201

2014 713

---------+---------+---------+-

Cu

rren

t usa

ge

0

100

200

300

400

500

600

700

800

2006 2010 2012 2013 2014

No. of LOBs

No. of LOBs

Naming Conventions

• DB2 Object Naming Standards

– Generally a Good Thing™

– Ease of Control / Administration

– Housekeeping

– Diagnosis

• CURRENT RULES special register

– STD – let DB2 do it for you

– DB2 – do it yourself

Na

min

g C

on

ve

ntio

ns

• The Good

– More control

– Complies with local standards

– Great for limited growth

• The Bad

– Must be created beforehand

– Constant monitoring

– Not for Partition By Growth tablespaces

• The Ugly

– Can cause outages if not tightly controlled

CU

RR

EN

T R

UL

ES

DB

2

• The Good

– No monitoring/maintenance/administration

– Complete automation (TABLESPACE ALL)

– Great for Partition by Growth tablespaces

• The Bad

– No control over object names

– Contravenes local standards

• The Ugly

– Needs careful consideration of

housekeeping routines (TABLESPACE ALL)

CU

RR

EN

T R

UL

ES

ST

D

• Which CURRENT RULES currently

rules?

– STD (My opinions, YMMV)

• Gotchas

– Work our your TS naming standards

• Easily identify LOBs from names

– Plan your housekeeping

• ISV tools

– AUX YES

– Watch out for EXCLUDEs/INCLUDEs

• IBM – use LISTDEFs with ALL keyword

CU

RR

EN

T R

UL

ES

??

?

CREATE TABLE USER.TEST1

(INT1 INTEGER NOT NULL

GENERATED ALWAYS AS IDENTITY (

START WITH 1 INCREMENT BY 1

MINVALUE 1 MAXVALUE 2147483647 NO

CYCLE CACHE 20 NO ORDER )

,FLAG1 CHAR(1) NOT NULL WITH DEFAULT

,VARCHAR1 VARCHAR(128) NOT NULL WITH DEFAULT USER

,SMALLINT1 SMALLINT NOT NULL WITH DEFAULT

,DECIMAL1 DECIMAL(15, 2) NOT NULL WITH DEFAULT

,TS1 TIMESTAMP NOT NULL WITH DEFAULT

,DATE1 DATE NOT NULL WITH DEFAULT

,TIME1 TIME NOT NULL WITH DEFAULT

,BLOB1 BLOB(1 M) INLINE LENGTH 1024

,CLOB1 BLOB(1 M) INLINE LENGTH 1024

,XML1 XML

,BLOB2 BLOB(1 M) INLINE LENGTH 1024

,CLOB2 BLOB(1 M) INLINE LENGTH 1024

,XML2 XML

,CONSTRAINT INT1 PRIMARY KEY ( INT1 ) )

IN DATABASE DUSER

CCSID UNICODE NOT VOLATILE APPEND NO

PARTITION BY SIZE EVERY 4 G;

Na

min

g e

xa

mp

les

Table Name Database Tblspace

v----1----v----2----v----3----v----4----v----5--

USER.TEST1 DUSER TEST1

USER.TEST1BLOB1K8JVZOFT DUSER LK8JVP4A

USER.TEST1BLOB2K8JWV1TJ DUSER LK8JV9XB

USER.TEST1CLOB1K8JVCUGV DUSER LK8JV7XN

USER.TEST1CLOB2K8JW3K8V DUSER LK8JWTMK

USER.XTEST1 DUSER XTES0000

USER.XTEST1000 DUSER XTES0001

Index Name Table Name

--v----1----v----2----v----3----v----4----v----5----v-

USER.I_DOCIDTEST1 USER.TEST1

USER.I_NODEIDXTEST1 USER.XTEST1

USER.I_NODEIDXTEST1000 USER.XTEST1000

USER.ITEST1BLOB1K8JV6BC USER.TEST1BLOB1K8JVZOFT

USER.ITEST1BLOB2K8JW190 USER.TEST1BLOB2K8JWV1TJ

USER.ITEST1CLOB1K8JVISA USER.TEST1CLOB1K8JVCUGV

USER.ITEST1CLOB2K8JW919 USER.TEST1CLOB2K8JW3K8V

USER.TEST1ABC_#_YJU USER.TEST1

Na

min

g e

xa

mp

les

Using Utilities


– CHECK, LOAD, REORG, COPY,

RECOVER, RUNSTATS

• Keep base + aux in sync

• Be aware of differences between

LOGGED and NOT LOGGED

• CHECK

– LOB

– INDEX

– DATA

Usin

g U

tilities

• RECOVER base and LOBs together

(simple, best option, need good copies)

• REPAIR (if LOB can be restored from

elsewhere)

• SQL (INSERT INTO…SELECT FROM)

• DSN1COPY (TSMYOYO)

• Re-LOAD

• Correcting LOB errors

LO

B R

eco

ve

ry

• KEYCARD keyword doesn’t work

• ISVs slow off the mark collecting LOB

stats

RU

NS

TA

TS

• Only REFERENCE (V9) / CHANGE

(V10) reclaims space

• SHRLEVEL CHANGE recommended

• Use AUX YES

– Rebalance PBR partitions

– Consolidate PBG partitions

• Issues with DISCARD

• Excluded from our automated

housekeeping due to size of LOBs

RE

OR

G

• Try and use LISTDEFs

• INCLUDE / EXCLUDE TABLESPACES

TABLESPACE database.space ALL

• ISVs have issues with INCLUDES and

EXCLUDES with AUX YES

CO

PY

• Job 1 COPY TABLESPACE DLBGP01.*

EXCLUDE DLBGP01.LJ6KYEZR, Aux TS for LOB part 2

DLBGP01.LJ7Z83KR, Aux TS for LOB part 3

DLBGP01.LJ762HEZ, Aux TS for LOB part 4

DLBGP01.LJ79T9ME, Aux TS for LOB part 5

DLBGP01.SPF0063, Base TS for Base table

DLBGP01.SPF0279 Aux TS for LOB part 1

AUX YES

• Job 2 COPY TABLESPACE DLBGP01.SPF0279

AUX YES

• Job 3 LISTDEF DPEGP01_TS01

INCLUDE TABLESPACES TABLESPACE DLBGP01.SPF0279 ALL

LISTDEF DPEGP01_TS02

INCLUDE TABLESPACES TABLESPACE DLBGP01.* ALL

EXCLUDE TABLESPACES TABLESPACE DLBGP01.SPF0279 ALL

CO

PY

• Cannot unload from image copies

– Would be a nice enhancement

• Unload with File Reference Variables

– Simpler to code

– Uses PDSE, one member per LOB

– Takes ages...and ages…and ages…

• Unload/LOAD with SPANNED

– Much faster

– A pain to code

– Generated LOAD cards in SYSPUNCH

would be handy for spanned records

UN

LO

AD

/LO

AD

Space – The Final Frontier

• WebSphere Message Broker

– Write-only application (mostly)

– c. 1 million inserts per day

– 4.5M tracks per day/30M tracks per week

– 1.5 TB per week

– Base Table had 1 BLOB and 1 CLOB

– LOBs defined to BP32K

– Max BLOB =3772, max CLOB=23368

– Wasted space!!!

• Broker logging reduced from 4M to 1.8M

tracks per day through app changes…

Sp

ace

– T

he

Fin

al F

ron

tier

Sp

ace

Exam

ple

Table Mon Tue Wed Thu Fri Sat Sun

Bufferpool BP32K2 BP21 BP21 BP21 BP32K2 BP32K2 BP32K2

Status 22/8 truncated and

reorged before use.

2/9 reorged by auto

reorg.

1/9 truncated and

reorged before use

1/9 truncated and

reorged before use,

in use now

1/9 truncated and

reorged before use

??/? truncated and

reorged before use.

2/9 reorged by auto

reorg.

22/8 truncated and

reorged before use.

2/9 reorged by auto

reorg.

22/8 truncated and

reorged before use.

2/9 reorged by auto

reorg.

AUDIT_LOG

Base 9765 8745 840 240 8355 4305 1965

LOB1 749430 88980 7665 690 829350 331155 148590

LOB2 748425 94245 7665 690 827655 331155 148590

1,507,620 191,970 16,170 1,620 1,665,360 666,615 299,145

rowcount 1,094,706 985,872 in use 0 983,277 485,007 215,940

Sp

ace

Exam

ple

0

200000

400000

600000

800000

1000000

1200000

1400000

1600000

1800000

1 2 3 4 5 6 7

CLOB

BLOB

Base

Sp

ace

Exam

ple

Class 1 Class 2 Class 3 Comments

0.075636 0.00762 0.005501 Base(BP16K) LOBS(BP32K)

0.039277 0.006351 0.004304 Base(16K) LOBS(BP8)

0.173326 0.042717 0.024962 base table in BP16K0 with inline BLOB and inline CLOB - LOBS in BP8

0.258535 0.071942 0.049692 base table in BP16K0 with max inline CLOB

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

1

2

3

4

Class 1

Class 2

Class 3

• Answer?

– Change bufferpool from 32K to 4K

• Tracks reduced from 1.8M to <200K per day

• Saving of 87% in DASD footprint!

• CPU went down 50%

– INLINE LOBs and compression

• Saved a further 10% in DASD

• CPU and elapsed time went up considerably

• Not really worth considering

• As I said, a mostly-write-only application, so

SELECTs not benefiting.

Sp

ace

– T

he

Fin

al F

ron

tier

Performance

• GBPCACHE

– CHANGED

– ALL

– SYSTEM

• SYSTEM is the default for LOBs

• Do not use CHANGED for a LOB table space

defined with LOG NO

– NONE

• Saves data transfer in GBP write and GBP

access for castout

• But updated pages must be written to DASD at

or prior to commit

• Useful when an updated page is rarely re-read

Pe

rform

an

ce

CREATE TABLE …. clobcol CLOB(x) INLINE LENGTH(y)

– Differentiates between SLOBs and FLOBs

– Allows LOBs to be stored in the base table

– Max 32680 bytes

– If a LOB cannot be fully inline, the LOB is

“split”. The first “y” bytes are stored inline

and the remaining bytes are stored in the

LOB table space.

Inlin

e L

OB

s

• Advantages

– Saves disk space

– Fewer synchronous I/Os

– Less CPU time to access LOB

– Prefetch can access multiple LOBs

– Improve FETCH CONTINUE

– Index on expression for LOB data

Inlin

e L

OB

s

• Disadvantages

– If most LOBs are split, performance may be

hurt

– If LOBs aren’t accessed, inline LOBs may

hurt performance

– If LOBs don’t compress well, CPU time

increases

– Buffer hit ratio for base table may decrease

– Larger base table image copies

– More logging, if LOBs are not logged

Inlin

e L

OB

s

• Considerations

– Requires Universal Table Spaces

– Required Re-ordered Row Format

– May require multiple reorg steps to

implement

– Know your LOB size distribution

• SQL histogram queries available

– May require new page size for base table

– May not compress well

• DSN1COMP doesn’t work for LOBs

• CLOBs better than BLOBs

Inlin

e L

OB

s

• Considerations

– May need to retune buffer pools

– Don’t bother Inlining if fewer than 50% of

LOBs can be fully inlined

– Increasing to next page size will reduce the

number of split LOBs

– If a LOB is not fully inlined, base and

auxiliary objects should be reorganized

in the same step. Use AUX YES or the

ALL keyword and SHRLEVEL

REFERENCE

Inlin

e L

OB

s

• Considerations

– Preparation is key

• “Before” performance stats reports

• “After” performance stats reports

– Iterative development

• Try several incremental changes

– INLINE length

– Page size

– Compression

– Storage

• Local and Group buffer pool sized correctly?

• Backed up by real storage?

Inlin

e L

OB

s

Conclusions

“If you go down to the LOBs today, you’d better not go alone”

• Prepare

– Read IBM redbooks

– IDUG/GSE/IBM presentations

– Read The Friendly Manuals

“It’s lovely down in the LOBs today, but it’s safer to stay at home”

• Plan

– DASD – make sure you have enough

– Real Storage – make sure you have enough

– Housekeeping

– Naming conventions – get them right first

Co

nclu

sio

n

http://db2forz.blogspot.co.uk/2013/07/is-length-of-your-inline-lobs-optimal.html

http://www.redbooks.ibm.com/redbooks/pdfs/sg247270.pdf

http://www-01.ibm.com/support/docview.wss?uid=swg1PM73034

http://www.gsebelux.com/system/files/Inline-LOB-BMC-GSEBEmar2013.pdf

http://robertsdb2blog.blogspot.co.uk/2012/01/got-lobs-get-db2-10-for-zos-part-1.html

http://robertsdb2blog.blogspot.co.uk/2013/07/db2-for-zos-clearing-up-some-

matters.html

http://db2forz.blogspot.co.uk/2013/07/is-length-of-your-inline-lobs-optimal.html

Re

fere

nce

s

Th

an

k y

ou

!

This presentation is based on my personal views and are not in any way endorsed by Lloyds Banking Group.

None of the information contained within presentation can be used for reference or for any other purpose by another company or organisation.

Marcus Davage @spufidoo

http://about.me/spufidoo

GSE 2014

Session: CC

Tainted LOB

Documents

Transcript of Tainted LOB