High Availability Improvements - NEDB2UG...* A new record is inserted into SYSPENDINGDDL table after...

© 2013 IBM Corporation

Information Management

High Availability Improvements



Agenda

Ø Recover and Online Schema

ØPending alter supports in Sequoia *Drop Column *Online Alter Limit Key

ØDefer Define enhancements

ØWork file enhancements



Recover and Online

Schema Changes

PendingChange PBGUTS PBRUTS XMLTS LOBTS

AlterSegmentSize X X X

AlterDataSetSize X X X X

AlterBPT/SPageSize X X X

AlterBPIndexPage X X

AlterMemberCluster X X

ConvertClassic1-Table Part.T/StoUTSPBR

X X

ConvertClassicSimple T/S

X

ConvertClassic SegmentedT/S

IBMConfidential

X ©2013IBMCorporation


DB2 10 Online Schema Enhancements



DB2 10 PIT RECOVERy with Pending Alter

5

time

Pending ALTER issued

0

1

Change deferred

2 REORG Materialize changes

3 RECOVER to PIT started

4 Earliest PIT RECOVER point (“Brick Wall”)

ü ü ü

© 2013 IBM Corporation IBM Confidential


PIT Recovery with Pending alter ØDB2 11 PITR with Pending ALTER * Supports in NFM only * A new record is inserted into SYSPENDINGDDL table after RECOVER

ØTable spaces supported: * LOB * XML * Partition by Range (PBR) Universal Table Space

ØAt end of RECOVER to PIT prior to the materializing REORG: * Table space is in restrictive status: REORG-pending * REORG to finalise the PIT recovery process is MANDATORY * No other PITR during this period prior to subsequence REORG

Ø Schema definition is not recoverable *Only data is recovered to PIT

© 2013 IBM Corporation 7

time

ALTER issued

0

1

Change deferred


Enhanced PIT RECOVERy with Pending Alter 2 REORG Materialize changes Pending

3 RECOVER to PIT start

4 Earliest PIT

RECOVERy point

ü ü


time

Pending ALTER issued

materialises changes

0

1

Change deferred


PIT RECOVERy with immediate ALTERs 2 REORG

3

PiT start

5 Earliest PiT

RECOVERy point

Immediate ALTERs

4 RECOVER to



PITR with non-materialized pending ALTER Ø Pending alter can be dropped any time before REORG Ø Pending change is not required to drop when RECOVER starts Ø PITR would be business as usual

time

REORG materializing changes

0

1 Pending

ALTER issued

Change deferred

3 RECOVER to PIT start

2



After PIT RECOVERy ØThe subsequent REORG is required * Must on be on ENTIRE table space * SHELEVEL NONE is not supported * SHRLEVEL CHANGE is overridden by SHRLEVEL REFERENCE Ø Before the subsequent REORG to materialize the RECOVER * No CREATE/ALTER/RENAME/DROP TABLE on the TS or AUX objects * All other utility jobs fail: DSNU933I (REORG required) Ø Only REORG/REPAIR DBD/REPORT RECOVERY or RECOVER to the same PIT are permitted ØWhere there were pending changes on LOB table space: * First REORG the LOB table space

* Then REORG the base table space

2 REORG materializing changes


PIT Recovery after materialized Pending alter ØWhen recover pending alter to PITR after materializing REORG : *Recovering an index is not supported *Entire table space must be recovered *There must be NO other Pending changes when RECOVER starts *VERIFYSET NO option will always be overridden to VERIFYSET YES

time © 2013 IBM Corporation

0

1 Pending

ALTER issued

Change deferred RECOVER to PIT start

3

2 REORG materializing changes


PITR and Pending ALTER restriction Ø TOCOPY/TOLASTCOPY/TOLASTFULLCOPY options can be used for entire table space on a single object * RECOVER cannot use an image copy with SHRLEVEL CHANGE taken between the change and it being materialised by REORG * Use TORBA or TOLOGPOINT to reach point after the image copy completed ØNo clone table may exist at the time RECOVER is issued Ø RECOVERy to a different PIT cannot commence until the first recovery AND its REORG complete ØRecovery to PITR prior to materialize of TS type conversion pending changes is NOT supported

time © 2013 IBM Corporation

0

1 Pending

ALTER issued

Change deferred RECOVER to PIT start

3

PendingChanges LOBTable Space

PartitionbyRange UTS

XMLTableSpace

AlterSegmentSize ü ü AlterDataSetSize ü ü ü AlterBPT/SPage

Size ü ü AlterBPIndexPage

AlterMember Cluster ü

AlterLimitkey © 2013 IBM Corporation


DB2 11 PIT Recovery Prior to Materializing REORG: Supported Changes

ü

sc1.table1inDB1.TS1

Col_1 Col_2 Col_3

Data_1 Data_2 Data_3




Drop Column



ALTER TABLE xxx COLUMN

ALTER TABLE

DB2 10 ADD COLUMN ALTER COLUMN

RENAME COLUMN

DB2 11 DROP COLUMN

sc1.table1inDB1.TS1

Col_1 Col_2 Col_3 Col_4

Data_1 Data_2 Data_3 Data_4



§ Columns become obsolete as applications change § Could the column be left? – An abandoned column costs

• • • •

Real space in every row stored in the table Real space in every Image Copy of the tablespace Potential space taken up in the log records written for the table Additional CPU and elapsed time in all aspects of accessing and maintaining the data

• DBA time “remembering” that the column is redundant/obsolete • Developer time analyzing usage of the column


Why do you need to DROP COLUMN?

sc1.table1inDB1.TS1




sc1.table1inDB1.TS1

Col_1 Col_2 Col_4


Data_1 Data_2 Data_4 ©2013IBM Corporation


How do you DROP a COLUMN? § Before DB2 11 – You have to DROP and recreate the Table to remove Col_3

Application Outage / Scheduled Maintenance

ApplicationTable




ApplicationTable

Col_1 Col_2 Col_4


Data_1 Data_2 Data_4 X 18 © 2013 IBM Corporation


Improvements in DB2 11 § DB2 11 ALTER TABLE sc1.table1 DROP COLUMN Col_3 RESTRICT REORG TABLESPACE DB1.TS1 SHRLEVEL REFERENCE Application Outage / Scheduled Maintenance IBM Confidential


SYSPENDINGDDL Table

Catalog Field

Value


DROP COLUMN Process

DBA issues Alter table

DROP COLUMN

Validation and authorization checks are done as the ALTER is issued.

Catalog

Table

Sch ly new em


DBA issues Alter


Drop column process

Catalog Table

SYSPENDINGDDL

App

a

Applications

schema

REORG Table Space With Defined schema

Continue to access data in the old

Table Space With New defined schema

ALTER TABLE DROP COLUMN …

§At ALTER time : § authorization checks § SYSPENDINGDDL entries §TablespacesetinAREOR

REORG

materialized the new definition

Unload

Data

Inline Image copy


Removing a Pending DROP COLUMN

ØIf a pending ALTER needs to be cancelled before the materializing REORG...

ALTER TABLE DROP COLUMN

ALTER TABLESPACE DROP PENDING CHANGES

Time

ØNOTE: This removes ALL pending changes associated with the table space or any objects within the table space



ØA column cannot be dropped if * The containing table space is NOT a Universal Table Space * The table is a created global temporary table, a system-period temporal table, a history table, MQT or table referenced by MQT plus others....

* There are row permissions or column masks dependent on the table * There are triggers defined on the table * The column is an XML column, is part of the table partitioning key, RI, plus

others... * Hash Key column * DOCID / ROWID column * Tables with EDITPROC or VALIDPROC

ØNo immediate alters (e.g. Can‘t change data type, add column, add/rotate parts) combines with pending alter


Things to think about...



New SQL Codes SQLCODE -195 LAST COLUMN OF table-name CANNOT BE DROPPED Explanation An attempt was made to drop a column using an ALTER TABLE statement. The column cannot be dropped from table table-name because at least one of the existing columns must be preserved when altering a table.

SQLCODE -196 COLUMN table-name.column-name CANNOT BE DROPPED. REASON = reason-code. Explanation The column cannot be dropped for one of the following reasons:

** See Information Centre or Manuals for the full list

© 2013 IBM


ØRECOVER * RECOVER to a PIT prior to a materializing REORG is not allowed

• DSNU556I and RC8 * An inline copy will be taken by the REORG

• This single image copy is the only base for recovery

ØUNLOAD * will not be able to unload from an Image Copy which contains data for

dropped columns • Dropped columns is no longer exist in the catalog after REORG

materialzied data • DSNU1227I and RC8

* Consider UNLOAD with external formatting • If data corruption occurs, can be used with LOAD utility if necessary Corporation 24

Impact on Other Utilities



Impact on Other Utilities - continued

ØRUNSTATs *Should be run after materializing REORG • Ensures statistics are current for BIND/REBIND or PREPARE

ØREORG *Only table space level REORG materializes the changes

Partition Logical_Part Limitkey

1 1 200

2 2 300

3 3 400

4 4 500



Alter Partition Limit Key

New key limit 350

ALTER TABLE … ALTER PARTITION integer ENDING AT …

Part- ition

Logical _Part

Limitkey _Internal

2 1 200

3 2 300 350

4 3 400

1 4 500


Partition 1 DSNUM 2

DSNUM 3 DSNUM 4 DSNUM 1

logial view physical view

Partition 2 Partition 3 Partition 4

Alter Table ... Alter Partition 3 Ending At 350 Inclusive

REORP REORP

DSNUM 2 DSNUM 3 DSNUM 4 DSNUM 1

50, 125, 150, 175, 200

225, 250, 275, 300 325, 350, 375, 400 425, 450, 475 500

50, 125, 150, 175, 200 225, 250, 275, 300, 325, 350 375, 400 425, 450, 475 500 © 2013 IBM Corporation

keys status DB2 Catalog

Partition 1 Partition 2 Partition 3 Partition 4

Any Reorg including physical part 3 and 4 must run to remove the REORP status

SYSIBM.SYSTABLEPART

ØPrior to V11 behavior * Affected partitions are set to REORP. * These partitions could not be accessed. * Any REORG could be run to redistribute the data and remove the status.



Alter Limit Key in DB2 11

ØAlter limit key is treated as a pending alter in NFM ØThe affected partitions are set to AREOR.

ØOnline REORG must be run to materialize the pending changes. *REORG SHRLEVEL NONE does not materialize these changes.

ØSupported table spaces types are: *UTS – partitioned by range (PBR) *Classic partitioned table spaces (table controlled partitioning)

• Objects, in which the partitioning is index controlled, must be first converted ! * The new limit keys are materialized in SYSTABLEPART in the SWITCH

phase (new message DSNU2916I).

Part- ition

Logical _Part

Limitkey _Internal

3 2 350

Part- ition

Logical _Part

Limitkey _Internal

3 2 300

Option_Value Part-ition

…350 3



DSNUM 2 DSNUM 3 DSNUM 4 DSNUM 1

logial view physical view

Partition 1 Partition 2 Partition 3 Partition 4

Alter Table ... Alter Partition 3 Ending At 350 Inclusive

AREOR AREOR

DSNUM 3 DSNUM 4

50, 125, 150, 175, 200 225, 250, 275, 300 325, 350, 375, 400 425, 450, 475 500

225, 250, 275, 300, 325, 350 375, 400

keys status DB2 Catalog

Partition 2 Partition 3

SYSIBM.SYSTABLEPART SYSIBM.SYSPENDINGDDL

Online Reorg including physical part 3 and 4 must run to materialize the Alter

SYSIBM.SYSTABLEPART

Materialization takes place in the SWITCH phase

SYSIBM.SYSPENDINGDDL is empty



ALTER TABLE … ALTER PARTITION integer ENDING AT …

ØRemoved restriction: * If the table space is in a materialized query table relation. * RECOVER PIT is allowed, but requires a subsequent REORG due to setting

of REORP after the recovery.

ØChanged behavior: * REORG SHRLEVEL NONE does not materialize the pending changes. * LOAD REPLACE does not materialize the pending changes.

ØIf special REORG keywords are not allowed, message DSNU2917I is issued (RC=8):

* REBALANCE * AUX NO (if the table space has LOB columns)



ALTER TABLESPACE ... DROP PENDING CHANGES

ØCurrent: AREOR setting is not removed.

ØNew: AREOR setting is removed. *Exception: AREOR is still kept if transition to hash organization is in progress.



Workfile

Enhancements



Workfile Enhancements

ØIssue a warning message when the total amount of in-use work file storage (including DGTT) has reached a storage shortage threshold (e.g.10% of total configured storage) *WFSTGUSE_AGENT_THRESHOLD : agent level space usage alert threshold (percentage range 0 – 100, default = 0) * WFSTGUSE_SYSTEM_THRESHOLD: system level space usage alert threshold (percentage range 0 -100, default = 90)

Ø Once the warning message has been issued, DB2 will wait for 30 system check points before issuing another same warning message

ØUpdate space map page under S-Latch to reduce space map page contention



Workfile Statistics trace records

ØAdd additional instrumentation data in DB2 statistics trace records (QIST) : * Total configured DASD storage for a work file database (QISTWSTG) * Current storage used and Maximum total storage ever used (include DGTTs): available in DB2 10 * Maximum total storage even used by non-DGTT (QISTWFMXU) * Current storage used by non-DGTTs (QISTWFCTO) * Total preferred configured storage for DGTTs (QISTDGTTSTG) * Maximum total storage ever used by DGTTs since DB2 is up (QISTDGTTMXU) * Current storage used by DGTTs (QISTDGTTCTO)

0

DB2 11 for z/OS Application Compatibility

Problem

• Changes to SQL and XML behavior are problematic to adopt

– It is difficult to understand the impact of change to thousands of applications

– Change needs to be controlled at the application level

• Service stream incompatibilities cause maintenance issues, especially when key PTFs pre-req the PTFs that cause the changed behavior

• Changes at a release boundary allow for some planning, but it is difficult

to sync application changes with release migration plans

Solution Details • BIND/REBIND PACKAGE Syntax BIND/REBIND >----+----------------------------------------- +----------------------------->

'-------- APPLCOMPAT---(--V10R1--)--------' (--V11R1--)

• REBIND TRIGGER Syntax REBINDER TRIGG >----+-----------------------------------------

6

DSNT298I csect-name ATTEMPT TO USE COMMAND OR OPTION command WHEN THE APPLICATION COMPATIBILITY IS SET FOR A PREVIOUS LEVEL Explanation A new command or command option was issued when application compatibility bind option is set to a prior DB2 release. Another possibility is when a command or command option introduced in release n was used but the application compatibility bind option is lower than n. command The command or command option that can be used only when the application compatibility bind option is set to the compatible release.

7

Solution Details • CREATE/ALTER PROCEDURE (SQL native): Syntax >>-CREATE/ALTER PROCEDURE --------------------------------------------------------------> option-list: >-----+--------------------- +-----------------------------------------------------------><

|---APPLCOMPAT V10R1--|

+---APPLCOMPAT V11R1--+

8

• CREATE/ALTER FUNCTION (SQL native):

Columns TableName SYSPACKAGE

ColumnName APPLCOMPAT

DataType VARCHAR(10)

Description TheDB2release levelthattheSQL statementsinthis packageis compatiblewith.

SYSPACKCOPY APPLCOMPAT VARCHAR(10) TheDB2release levelthattheSQL statementsinthis packageis compatiblewith.

Updated Catalog

9

– –

•

•

Solution Details (cont)

• ALL SQL DML and XML changes are fenced by these options. This includes: • Function Resolution

New rules SQL function • Result type changes Incompatible SQL changes such as: • SQLCODE changes BIF changes …

SQLCODE -4743

-4743

ATTEMPT TO USE A FUNCTION WHEN THE APPLICATION COMPATIBILITY IS SET FOR A PREVIOUS LEVEL

Explanation

Functions that this release of DB2® introduces cannot be used when application compatibility setting is for a prior DB2 release. An attempt was made to execute one of these functions. Another possibility is when a function introduced in release n was used but the application compatibility setting is lower than n.

To use the new functions that have been introduced in this release of DB2, the application compatibility bind option or special register CURRENT APPLICATION COMPATIBILITY must be set to the current release. An attempt to use functions introduced in release n requires the application compatibility bind option or special register CURRENT APPLICATION COMPATIBILITY value to be at least n.

1 1

•

– Ensure

code level

behavior

Once

Interaction with NFM is intended to fence application and system behavior in a migration/co-existance environmentthat the “n-m”* level of code can process objects created in the “n” level of the – Ensure that applications bound on level “n” can run (after autobind) “n- m”*

applicationis consistent across releases of DB2 – As such, APPLCOMPAT for the current release “n” may not bespecified until NFM – mode in NFM MODE, either APPLCOMPAT(n) or APPLCOMPAT(n-1) may be specified (see the following charts)

* n-m where m = 1 or 2. DB2 10 is the lowest level that may be

specified

DB2 11 Enabling New Function Mode (ENFM10 )

DB2 11 Catalog

DB2 11 New Function

Mode (NFM)

CATMAIN T

UPDATE (DSNTIJTC )

CATENFM COMPLET E (DSNTIJNF )

CATENFM START

(DSNTIJEN)

DB2 11 Librarie s

DB2 10 Catalog DB2 10 Librarie s

DB2 10 à DB2

DB2 11 Conversion

Mode (CM10)

10

Normal Migration 11 Mode

(NFM) With

DB2 SPE New Function

1 – 2 months Data Sharing Coexistence 1 week

2 hours

Bind with APPLCOMPAT(V10R1) option only Bind with APPLCOMPAT(V10R1) or

APPLCOMPAT(V11R1)

Interaction with NFM (cont)

• What about New DDL and Authorization syntax and semantics in an application bound with APPLCOMPAT(n-1)?

– DDL and Authorization are fenced by NFM and are not affected by

the APPLCOMPAT level

• For example – You can use SPUFI - bound with APPLCOMPAT(V10R1) to “CREATE” a GLOBAL VARIABLE (proposed DB2 11 function) and authorize a user to “SET” it, but

you cannot use that same version of SPUFI to reference the variable you just created.

2 9

Summary • With the APPLCOMPAT bind option, we are striving to separate release migration from application changes that may be necessary to migrate to a new release.


®

DB2 for z/OS DB2 11

Extended RBA / LRSN Reliability, Availability, Serviceability

70

DB2 for z/OS DB2 11 RBA Today § DB2’s Relative Byte Address (RBA) for logging is 6 bytes – Gives 256TB of log record addressing capacity per DB2 subsystem/member § With faster processors, consolidation of DB2 members, and heavy sustained logging rates, DB2 can exhaust the 6-byte RBA • Warning messages issued at x’FFFF00000000’ – Some customers wrap a log today in 2 months, others just approaching limit § Manual recovery actions are needed to reset RBA to zeros – Data Sharing: Shut down affected member and start a new one in its place • Operational issues, such as automation, routing, and others – Non Data Sharing: reset PGLOGRBA field in every page to zero • Special form of Image Copy – this is extended outage (>1 day)

71

DB2 for z/OS DB2 11 LRSN Today § The data sharing Log Record Sequence Number (LRSN) is the high order 6 bytes of the 8-byte time-of-day clock which hits end of range in 2042 § Some groups have a non-zero LRSN “delta” which gets added to the TOD clock – Then LRSN will hit end of range prior to 2042 – Print Log Map (DSNJU004) shows the LRSN delta value • A “delta” value could be set when data sharing is enabled or re-enabled • Example: data sharing enabled when the highest non shared RBA written is > LRSN § Some non data sharing customers have enabled data sharing to circumvent RBA nearing end-of-range – This causes a non-zero LRSN delta if High Written RBA > LRSN

§ No current circumvention for end of LRSN § 6-byte LRSN value has precision to only 16 microseconds – Can cause LRSN ‘spinning’ which burns extra CPU and aggravates log latch contention – Changes in DB2 9 and 10 address most LRSN spin situations – Some still exist, mainly with row update and delete, and occasionally multi row insert

DB2 for z/OS DB2 11 Solution § Expand the RBA and LRSN to 10 bytes

– RBA addressing capacity of 1 yottabyte (2**80) • X’00000000112233445566’ – add positions to high order RBA to left

– LRSN extended on left by 1 byte, on the right by 3 bytes

• • •

x'00LLLLLLLLLLLL000000’ >30,000 years and 16M times more precision Numeric progression after end of 6-byte LRSN:

– – – –

Next value 01 000000000000 000000. The next value is 01 000000000000 000001. Eventually, it becomes 01 000000000000 FFFFFF And next value is 01 000000000001 000000

§ Reason for 10 – 8 bytes is not enough to solve LRSN issues and insufficient for longer term

• •

One byte on lower end of LRSN means DB2 could spin on current processors 8-byte RBA -- could exhaust the range in < week with technology now in IBM labs

72

DB2 for z/OS DB2 11 Solution

§ § § §

NFM only (6 byte RBA/LRSN continues to be used in CM) In NFM, DB2 uses 6-byte values until you take action to convert Two conversion tasks (order not important):

1. Convert BSDSes to new format to enable logging with larger RBAs/LRSNs 2. Convert DB2 objects to new page format via REORG in piecemeal fashion over days/weeks/months These tasks are optional (but recommended)

– If you don’t care about larger RBAs/LRSNs then you don’t have to convert – But performance can be better if you convert BSDSes

• •

Avoid internal conversion overhead on log write (1%) Eliminate LRSN spin

– Even if you are “safe” w/I limits, DB2 11 is 10 byte system and 6-byte RBA will be converted before processing

73

74

DB2 for z/OS DB2 11 Major areas of change § Lots internal – affects the bones of DB2 § Utilities - Most have no external changes § Catalog § Trace records: numerous IFCIDs are changed for extended RBA/LRSN § New page format to handle 10-byte PGLOGRBA equivalent § Utilities can handle either page format, so can SQL, DDL § Definition: “Basic” means 6-byte RBA/LRSN. “Extended” is 10-bytes § Check with your vendor for specific tools you use: they must be able to handle/tolerate as appropriate: • Basic and Extended RBA/LRSN • Both page formats

75

DB2 for z/OS DB2 11 Considerations for Replication products § Replication products must be updated for DB2 11 § IFCID 306 products will not work in DB2 11 CM – They always see the 10-byte log format. • Q-Replication • CDC • Others

§ Same for products that use log capture exit (E-Net / RRDF) – Interface changed significantly § In DB2 11 CM any coexisting DB2 10 members see log records in old format – Workaround for IFCID 306 based products if customer does not yet have the updated version installed

76

DB2 for z/OS DB2 11 Modes of DB2 11 implementation § CM – 6-byte RBA/LRSN and pages in current or “basic” format – New catalog objects for DB2 11 are created in CM in basic page format mode, but with extended RBA/LRSN columns, where applicable – RECOVER utility: TORBA, TOLOGPOINT, RESTOREBEFORE keywords extended to accept either 6- or 10-byte RBA/LRSN input – Standalone log utilities accept larger RBA/LRSN values

– REPORT RECOVERY output accommodates extended RBA/LRSN values

– Work files are in extended format § ENFM – DB2 10 RBA/LRSN related catalog columns are extended to 10 bytes

– Catalog objects updated/reorged may be converted to extended page format based on DSNZPARM OBJECT_CREATE_FORMAT (Basic/Extended) – DSNTIJNF SCA converted to handle 10-byte RBA/LRSN

77

DB2 for z/OS DB2 11 Modes of DB2 11 implementation § NFM – Can convert BSDSes for extended RBA/LRSN

– Begin to convert objects any time you REORG

– Almost all partitioned objects can be converted independently (by part(s))

– Conversion MUST have occurred before All Fs for either RBA/LRSN • Else read only object since updates cannot be made in basic format – New job DSNTIJCV converts catalog/directory basic > extended page format • Optional, if you are not close to end of the log • Converts any not yet converted; over time all are in extended page format

§ Any mode: The Unit of Recovery ID (URID) extends to 10 bytes – DISPLAY THREAD output, restart status table messages

– Padded in CM with zeros

78

DB2 for z/OS DB2 11 Procedure to Convert to Larger LRSN § New warning messages for LRSN reaching end of range – DSNJ034I - WARNING: one year or less – DSNJ035E – Critical: 2 weeks of LRSN end (note “E” for “action”) § Convert the BSDSes of all D/S members to activate the extended log record format (DSNJCNVT offline utility while DB2 is down) – Roll through one data sharing member at a time for continuous availability § Convert table spaces and indexes to the new page format – When LRSN advances past the 6-byte limit (x’FFFFFFFFFFFF’) • objects in the old format become read only but still opened as RW § No order implied if members not approaching 6-byte limit, but makes sense to convert BSDSes first § Procedure for non sharing DB2 subsystems similar – in b/g material

79

DB2 for z/OS DB2 11 Converting the Page Format § An object can be converted to / from extended format by REORG or REBUILD utilities § A partitioned TS or index can be converted one part at a time – Except PBG hashed TS which must be converted at the TS level (all parts) § CLONE tables cannot be converted – DROP CLONE

– Convert base table, then ALTER ADD CLONE

– Re add all the cloned authorizations § SYSINDEXPART and SYSTABLEPART indicate the page format

80

DB2 for z/OS DB2 11 Current 4K page format (Basic)

§ Pghead = PGLOGRBA (6 bytes stored)

§ E is PGTAIL (2 bytes) § Last byte is PGEND (parity) – “E” - x’11000101’ or

– “N” - x’11010101’

– Underlined bit called PGBigTailApplies – Zeros mean off (does not apply)

§ So DB2 uses 6-byte RBA/LRSNs in pghead area

PGLOGRBA

E

pghead PGBigTail(20)

Unused(6) Reserved(2)

PGBigRBA(10) PGTAIL(2)

81

DB2 for z/OS DB2 11 4K page with Extended RBA/LRSN

§ PGBigTail (20 bytes) 1. Unused – 6 bytes 2. Reserved – 2 bytes 3. PGBigRBA – 10 bytes – Extended RBA/LRSN stored 4. PGTAIL – 2 bytes byte 1 reserved Byte 2 – PGEND either

x’01000010’ or

x’01010010’ Parity is bits 2-4 (100 or 101) and PGBigTailApplies is bit 7 -

If one, it is ON, (Extended format PGBigRBA) if zero, it doesn’t (Basic format)

All page sizes have same format Maximum row length unchanged

82

DB2 for z/OS DB2 11 New DSNZPARMs § OBJECT_CREATE_FORMAT – Parameter states page format new objects use, basic or extended – Default is BASIC (6-byte RBA/LRSN format) – EXTENDED only takes effect in NFM (10-byte format) • If you are going to convert, set it in CM and catalog objects touched in ENFM will be converted for you § UTILITY_OBJECT_CONVERSION (ignored during CM) – Should page conversion be done by default by REORG, LOAD, REBUILD? • BASIC: the output object will be basic format (default) • EXTENDED: the output object will be extended format (only NFM) • NONE: the output object remains the same format – RBALRSN_CONVERSION utility parm of same options override DSNZPARM • Specify the format of the object on completion of the utility – Can’t specify in CM – Specify for catalog objects with altered columns during ENFM

83

DB2 for z/OS DB2 11 Main Utilities

§ §

RECOVER REORG / REBUILD INDEX / LOAD REPLACE

Other utilities support both basic and extended page formats

§ § §

DSN1COPY – When source and target have differing page formats, copied data may not reflect target catalog

– REPAIR – new option to correct catalog/OBD for format of actual data sets after DSN1COPY LISTDEF – adds Filtering on Basic/Extended page format Utilities display/messages are in extended format regardless of actual object format

84

DB2 for z/OS DB2 11 RECOVER § RECOVER accepts longer values for TORBA, TOLOGPOINT, RESTOREBEFORE – Six-byte RBAs may be specified as ten-byte operands with the six-byte value right justified in ten-bytes, i.e.x'00000000RRRRRRRRRRRR‘ – Six-byte LRSNs may be specified as ten-byte operands in byte locations two through seven, i.e. x'00LLLLLLLLLLLL000000' – Corrects any mismatch between recovered data and catalog • i.e. current = Extended, but TORBA = Basic

85

DB2 for z/OS DB2 11 Utilities in General § Use DSNZPARM UTILITY_OBJECT_CONVERSION globally – Unless overriding Utility statement parms • RBALRSN_CONVERSION

§ Mapping table handles extended RBA/LRSN – Automatically created! § Any SHRLEVEL can convert § Index and underlying object unrelated – Index can be left in different page format from underlying objects • Example: 50 parts extended – 500 parts in basic – NPI can be either

Utility Partition Partitioned Index

NonPartitioned Index

ReorgTablespace Y Y Y

ReorgPART Y Y N

ReorgINDEX Y Y

ReorgINDEXPART Y N

RebuildINDEX Y Y

RebuildINDEXPART Y N

LoadReplaceTablespace Y Y Y

LoadReplacePART Y Y N

86

DB2 for z/OS DB2 11 Utilities and Page Format Conversion

§ Tablespace reorg required for only: – Hashed tables (hash algorithm to spread data) – XML Versioned T/S – to expand START_TS and END_TS from 8 > 10 bytes - Must occur before or with conversion to extended page format - then Part Reorg OK

§ Key: NPI must be converted specifically outside tablespace level REORG

FromDB211 Version

ToVersion BasicPage format

Extendedpage format

NFM DB211NFM Yes Yes NFM DB29/10 Yes No CM DB29/10 Yes No CM* DB29/10 Yes No

87

DB2 for z/OS DB2 11 DSN1COPY from DB2 11 to another subsystem

§ Reason for conversion back and forth is porting to another DB2 § Table shows the page format for a DB2 11 object that must be copied to a DB2 subsystem that is on another version of DB2

§ A REORG is necessary if an object is already in extended page format – DB2 9/10 understands only 6-byte basic page formats

88

DB2 for z/OS DB2 11 Recovery Considerations Example for table space ‘TS1’: T1 - image copy T2 - quiesce T3 - convert to extended page format T4 - quiesce T5 - image copy T6 - quiesce

Q1: Can I recover TS1 back to time T2? Ans: Yes. And the catalog is updated to indicate “basic” format Q2: Can I then roll forward to time T4? Ans: No. RECOVER does not allow log apply across a REORG

Messages RBA (D/Sornon-D/S)

LRSN D/Sonly

ReasonCode

DSNJ032I X’F00000000000’ n/a n/a

DSNJ033E X’FFFF00000000’ n/a 00D10251 DSNJ034I n/a 1yearor

less n/a

DSNJ035E n/a 2weeks 00D10251 Advisorylimit(‘soft’) X’FFF800000000’ n/a -90400C2026D

AdvisorylimitDS(‘soft’) 2months -90400C2026D

89

DB2 for z/OS DB2 11 Messages and Limits

§ Messages appear at DB2 restart and every log switch – not just a speed bump! § Soft limit: – Rollback and abort allowed

– SQL updates not allowed

– DDL ok if DROP/CREATE for extended format tablespace – utilities okay too

91

DB2 for z/OS DB2 11 End of RBA Timeline

F00000000000 DSNJ032I Every log switch and Restart

Begin Converting To Extended

DB2 in MAINT mode Convert BSDS

FFF80000000 Soft limit -904 00C206D No SQL updates for basic objects

FFFF00000000 DSNJ033E Critical DB2 stops 00D10251

FFFFFFFFFFF Hard limit Convert BSDS

and objects NOW!

92

DB2 for z/OS DB2 11 End of LRSN Timeline

1 Year left DSNJ034I Every log switch and Restart

Begin Converting To Extended

DB2 in MAINT mode Convert BSDS

2 months left Soft limit -904 00C206D No SQL updates for basic objects

2 weeks left DSNJ034I Critical DSNJ035E DB2 stops 00D10251

FFFFFFFFFFF Hard limit Convert BSDS

and objects NOW!

93

DB2 for z/OS DB2 11 Benefits

§ §

Grants permanent relief for operational issues/outages caused by hitting the end of RBA or LRSN

– Overhead of 6-byte logs is about 1% – Converting logs to 10-byte and tables to extended format may get 3% CPU improvement (your mileage may vary) Complete Flexibility

– Conversion of BSDS and DB2 objects can be done in any order – Conversion of DB2 objects performed non-disruptively with Reorg/Load

• •

No extra effort – can be done over time Can be done either way – Basic to Extended or Extended to Basic

– Conversion optional if you are not close to end of log 1. But DB2 code always converts Basic RBA/LRSN to Extended 2. Better update performance with Extended page formats

94

DB2 for z/OS DB2 11 The Message

§DB2 11 is a 10-byte RBA/LRSN system

§If you think you will be “conservative” or “safe” by remaining in 6-byte RBA/LRSN mode,

§Think again –Every 6-byte RBA/LRSN that is touched is converted to 10-bytes before processing, and may need to be converted back to 6-bytes.


®

DB2 for z/OS DB2 11

Background slides

96

DB2 for z/OS DB2 11 Active Log Data Set Background

§ § §

Support for active log data sets up to 4GB was introduced in DB2 V6 – The previous maximum was 2GB – Total active log space: 124GB (31 active logs * 4GB) – Up to 1000 archive logs, 4TB DB2 V8 NFM and later allows up to 93 active log data sets

– Maximum active log space: 372GB – Up to 10000 archive logs, 40TB In DB2 11, active log formatter (DSNJLOGF) can preformat 10 log data sets per execution

– One in DB2 10

97

DB2 for z/OS DB2 11 Non Data Sharing Procedure to Convert to Large RBA § Convert the BSDS to activate the extended log record format – New job DSNJCNVT § Convert table spaces and indexes to the new page format – Once the RBA value has advanced past the 6-byte limit (x’FFFFFFFFFFFF’) then any object still in the old format is forced to R/O access by DB2 – No way to update the RBA now

98

DB2 for z/OS DB2 11 BSDS Conversion § Run new DSNJCNVT utility to convert BSDS after creating new ones. – Need new output BSDS because CI size is increased

– Rename old BSDS with IDCAMS ALTER NEWNAME (SYSUT1/2)

– Create new BSDS with larger CI size (SYSUT 3/4)

– Typically takes much less than one minute to execute

99

DB2 for z/OS DB2 11 Log Limits and behavior § Advisory logging occurs at RBA “FFF80000000’x or LRSN about 2 months before LRSN exhausted. Updates for rollback work – No SQL updates allowed § Actual limit occurs when RBA/LRSN no longer fit in 6 bytes – LOAD REPLACE, REBUILD, REORG works for conversion into EXTENDED format only – ignores DSNZPARMs and utility settings – RECOVER of object in BASIC format sets REORP/RBLDP

– RESTORE SYSTEM allowed for both limits for non-D/S. For D/S allowed only if SLB created after BSDS conversion or recover from tape.. § MERGECOPY – Basic format not allowed

®


IBM Software Group

Archive Transparency

IBM Software Group | DB2 information management software

DB2 11 – Part III: Archive Transparency

Ø Importance and Business Need

Ø Technical Details


DB2 11 Part III – Importance and Business Need Archive Transparency:

Ø It provides basic archive and retrieval functions using SQL Ø The retrieval of data from base table or base table plus its associated archive table is controlled by the setting of a built-in system defined global variable without changing SQL in the applications

Ø The majority of applications retrieve data from base table. There should be no performance degradation by UNION ALL to its associated archive table.

Ø The functionality is primarily of interest to Ø Ø Ø Ø

DBAs application developers end users consultants


DB2 11 Controls of ArchiveTransparent Access and Archive Data

Built-inglobalvariablesSYSIBMADM.GET_ARCHIVE : CHAR(1), value can be either 'Y' or 'N’ SYSIBMADM.MOVE_TO_ARCHIVE: CHAR(1), value can be either 'Y' or 'N'

Ø Read/write of the variables: The read authority is granted to PUBLIC. The authority rules for grant are consistent with user-defined global variables. The administrative authorities that can grant global variable privileges are install SYSADM, SYSADM, SECADM, and ACCESSCTRL. If the SEPARATE_SECURITY zparm is set, then SYSADM cannot grant global variable privileges.

Ø Default: 'N'

Ø Transparent archive data via DELETE: set the built-in global variable SYSIBMADM.MOVE_TO_ARCHIVE to 'Y', the deleted records are propagated to archive table automatically by DB2 via single DELETE SQL statement

Ø Transparent access archive data: set the built-in global variable SYSIBMADM.GET_ARCHIVE to 'Y' for all subsequent SQL statements including those from invoked function, stored procedure, and trigger. This allows the application to see both active and archive data without modifying the SQL statements in multiple packages. DB2 rewrites the query with UNION ALL operator.

DB211–ControlsofArchiveTransparency

Thefollowingbind/rou@neop@[email protected]_ARCHIVEglobalvariable:

Ø  ARCHIVESENSITIVE(defaultYES)Ø  BINDPACKAGEØ  REBINDPACKAGEØ  REBINDTRIGGERPACKAGEØ  CREATETRIGGER(implicittriggerpackage)

Ø  ARCHIVESENSITIVE(defaultYES)Ø  CREATEFUNCTION(SQLscalar)Ø  ALTERFUNCTION(SQLscalar)Ø  CREATEPROCEDURE(SQLna@ve)Ø  ALTERPROCEDURE(SQLna@ve)Ø  ARCHIVESENSITIVE(DB2Ipanels)Ø  DB2IPanelDSNEBP10Ø  DB2IPanelDSNEBP11Ø  DB2IPanelDSNEBP19


Page 106

DB2 11 – DDL Example of Archive Transparency Ø Two table approach: archive-enabled table stores active data; archive table stores archive

data CREATE TABLE policy_info_aet (policy_id CHAR(10) NOT NULL,

coverage INT NOT NULL);

CREATE TABLE policy_info_arc (policy_id CHAR(10) NOT NULL, coverage INT NOT NULL);

Ø ALTER table to enable archive transparency: ALTER TABLE policy_info_aet ENABLE ARCHIVE USE policy_info_arc;

Ø ALTER table add column: single ALTER statement on archive-enabled table also adds the column(s) to archive table

ALTER TABLE policy_info_aet ADD COLUMN customer_id;

DB2 also implicitly adds customer_id column to archive table policy_info_arc

Ø Temporal tables (STT, ATT, bitemporal tables) and archive-enabled table (AET) are mutually exclusive; AET cannot be referenced in same SQL statement with STT or ATT.


DB2 11 – Example of Querying Archive Data Ø Assuming --

Ø Ø Ø Ø

There are three archive-enabled tables AET1, AET2, and AET3 In an application, there is an SQL statement:

SELECT UDF_1(AET1.C1) FROM AET1, AET2;

In UDF_1, there is an SQL statement: SELECT * FROM AET3;

The application wants to get the result from both base and archive table

Ø Transparent access to archive data with DB2 11 feature --

Ø Ø

NO change of the application and UDF_1, before calling the application SET SYSIBMADM.GET_ARCHIVE = ‘Y’ ;

If the application just wants to access the active data SET SYSIBMADM.GET_ARCHIVE = ‘N’ ; *** 'N' is the default ***

Ø Transparent retrieval of archive data: there is no need to change SQL statements in an existing application to get current data only or to get both current and archive data. The setting of the SYSIBMADM.GET_ARCHIVE built-in global variable offers transparent access to archive data for queries.

Page 107


DB2 11 – Performance Consideration To avoid runtime performance degradation of static DML with the built-in SYSIBMADM.GET_ARCHIVE global variable, there is a design of two sections per static DML with AET reference.

Given a static SQL SELECT * FROM AET1, and if the package is bound with option ARCHIVESENSITIVE YES, then DB2 binds this SQL twice --

Ø Original section: bind the original SELECT * FROM AET1 for current data. The majority of transparent archive applications request for current data only. There is no performance degradation caused by UNION ALL query transformation. Ø Extended section: bind again an extended section by expanding AET1 to UNION ALL of AET1 and its associated archive table. Ø For the application, there is no change.

Ø At execution time: Ø Ø

If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'N' value (the default), DB2 chooses the original section If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'Y' value, DB2 chooses the extended section to get both current and archive data. Page 108


DB2 11 – GET_ARCHIVE and Data Change Operation Ø INSERT/UPDATE/DELETE/MERGE: if AET is the target of the data-change operation (either directly or indirectly thru views), NO implicit union all query transformation on the target AET when SYSIBMADM.GET_ARCHIVE global variable is in effect.

Ø Example: given a static SQL DELETE FROM AET1 WHERE EXISTS (SELECT 1 FROM AET2 WHERE CUSTOMER_ID = 'JOHN'), and if the package is bound with option

ARCHIVESENSITIVE YES, then DB2 binds this SQL twice --

Ø Ø Ø Ø

Original section: bind the original delete for current data. The majority of transparent archive applications request for current data only. There is no performance degradation caused by UNION ALL query transformation. Extended section: bind again an extended section by expanding AET2 to UNION ALL with its associated archive table. The target table AET1 of DELETE is not expanded with UNION ALL. For the application, there is no change on the DML statements.

At execution time: -- If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'N' value (the default), DB2 chooses the original section -- If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'Y' value, DB2 chooses the extended section to get both current and archive data from AET2 for the evaluation of EXISTS predicate

Page 109


DB2 11 – MOVE_TO_ARCHIVE and Data Change Operation

Ø DELETE: single DELETE statement can trigger transparent archive when MOVE_TO_ARCHIVE global variable is in effect; no additional privilege is required on archive table, only the privileges on delete of archive-enabled table are needed

Ø INSERT/UPDATE/MERGE (blocked in archive mode):

Ø Ø

If SYSIBMADM.MOVE_TO_ARCHIVE = ‘Y’, the INSERT, UPDATE and MERGE will fail. This is based on the assumption that in archive mode, only DELETE is meaningful. If SYSIBMADM.MOVE_TO_ARCHIVE = ‘N’, no archive mode failure

Ø Example -- given a SQL DELETE FROM AET1, regardless whether it is dynamic or static SQL, and whether the package is bound with option ARCHIVESENSITIVE YES or NO:

ØFor the application, there is no change on the data-change SQL statements.

ØIf the built-in SYSIBMADM.MOVE_TO_ARCHIVE global variable contains 'Y' value, for each row deleted from archive-enabled table AET1, DB2 will insert it into the corresponding archive table. This is an SQL performance improvement to help archiving data with a single DELETE.

ØIf the built-in SYSIBMADM.MOVE_TO_ARCHIVE global variable contains 'N' value (the default), DB2 does no data propagation to archive table. It is a regular delete statement.

Page 110


DB2 11 – DDLs and DML Error Scenarios Ø NO Impacts of GET_ARCHIVE and MOVE_TO_ARCHIVE on DDL Statements (NO package

generation) Ø

Ø

Ø

Ø

Ø

Ø

View MQT Inline UDF Row Permission Column Mask … ...

Ø Error Scenarios of GET_ARCHIVE on DML Statements Ø Inline table UDF references: the SQL table UDF contains AET, and the SYSIBMADM.GET_ARCHIVE global variable is in effect

Ø Application of access controls: the access control contains AET, and the SYSIBMADM.GET_ARCHIVE

global variable is in effect Ø Error Scenarios of MOVE_TO_ARCHIVE on DML Statements Ø INSERT/UPDATE/MERGE: if SYSIBMADM.MOVE_TO_ARCHIVE = ‘Y’, the INSERT, UPDATE and MERGE on AET will fail. This is based on the assumption that in archive mode, only DELETE is meaningful.

Ø Error Scenarios NOT tied to GET_ARCHIVE and MOVE_TO_ARCHIVE on DML Statements Ø WHEN clause of trigger: CREATE TRIGGER and REBIND TRIGGER PACKAGE will fail if the trigger has AET reference in WHEN clause and the trigger package is generated with ARCHIVESENSITIVE YES Page 111

SYSTEM_TIMEVersioning ArchiveTransparency

Schema --twotableapproach

currenttable&historytable samecolumn#,columnname,column attributes(datatype,etc)

archive-enabledtable&archivetable samecolumn#,columnname,column attributes(datatype,etc)

ROWBEGIN/ROW END/TRANSIDcolumns mandatory none

Periodconcept yes–SYSTEM_TIMEperiod none,notcompatiblewithSTT

CompatiblewithATT yes no

Bindoption SYSTIMESENSITIVE ARCHIVESENSITIVE

Implicitunionallquery transformation

controlledbyCURRENTTEMPORAL SYSTEM_TIMEspecialregister:there areimplicitsystem_timepredicates

controlledbybuilt-inglobalvariable SYSIBMADM.GET_ARCHIVE:union

allonly,noimplicitpredicatesfor archivetransparency

Datapropagationto history/archivetable

UPDATEandDELETE DELETE, SYSIBMADM.MOVE_TO_ARCHIVE

StaticDMLs twosectiondesign twosectiondesign


Page 116

sc.: SYSTEM_TIME Versioning vs Archive Transparency

IndexEnhancements

•  Suppress-nullindexes•  Indexduplicateskipping•  IndexSkipping•  AutoCleanupofPseudo-deletedIndexEntries

Migration Planning Workshop


DB2 11 for z/OS

Suppress-null indexes § Support creation of indexes where the default NULL values are excluded from the index – Reduced index size, improved insert/update/delete performance, 3rd party DBMS compatibility § New CREATE INDEX keywords – INCLUDE NULL KEYS (default) • DB2 will create an index entry even if every key column contains the NULL value – EXCLUDE NULL KEYS • DB2 will not create an index entry when every key column contains the NULL value. If any key column is not null the index entry will be indexed § Conditions: – EXCLUDE NULL KEYS must not be specified if any of the columns identified by column-name are defined as NOT NULL – Not supported if the index is defined as a partitioning index

IDX



DB2 11 for z/OS

Auto Cleanup of Pseudo-deleted Index Entries …

§ Pre DB2 11 pseudo deleted keys can accumulate over time – Index size grows with increasing number of pseudo-deleted index entries • More getpages and lock requests required • Increased CPU cost and longer elapsed times for index search

– Applications may encounter deadlocks and timeouts during INSERT/UPDATE/DELETE • Pseudo-deleted index entries are locked to enforce uniqueness • RID reuse by INSERT following DELETE can lead to deadlock

§ DB2 11 provides automatic cleanup of pseudo deleted entries – Clean up both pseudo empty index pages and pseudo deleted index entries • Reduce impact of pseudo delete entries • Reduce the need to run the REORG INDEX utility

IDX



DB2 11 for z/OS

Auto Cleanup of Pseudo-deleted Index Entries

§ DB2 11 automatically cleans up pseudo deleted entries – zIIP eligible and runs in the background – Designed to have minimal or no disruption to applications – New ZParm INDEX_CLEANUP_THREADS (0-128) to control number of concurrent cleanup tasks

– – – –

Default is 10 0 disables index cleanup Value can vary between members of a data sharing group Online changeable

– New SYSIBM.SYSINDEXCLEANUP catalog table to control auto cleanup at index level • Day of week/month, start/end time. • By default cleanup is enabled for all indexes – IFCID 377 written when index is cleaned up

§ Benefits of automatic pseudo delete cleanup – Reduce size of some indexes, fewer getpages – Improve SQL performance, Lower CPU, less elapsed time – Reduce need to run REORG INDEX

IDX



DB2 11 for z/OS

Pseudo-deleted Index Cleanup § Cleanup done under system tasks, run as enclave SRBs and zIIP eligible • Parent thread (one per DB2 member) reads through RTS to find candidates – Runs every 5 minutes • Eligible indexes sorted based on number of pseudo deleted rids to delete highest first

NAME IX1 IX2 IX3 IX4

… n n n n

NPAGES 100 1000 500 2000

… X X X X

REORGPSEUDODELETES 5000 20000 100000 75000

• Child threads assigned based on ZParm setting SYSIBM.SYSINDEXSPACESTATS

Parent thread Index IX3

SELECT FROM… ORDER BY

Child cleanup thread IX3 Child cleanup thread IX4

– Child cleanup thread only started if Index already open for INSERT, UPDATE or DELETE • ‘X’-type P-lock already held

IDX

IX4 In Memory

IX2 IX1



DB2 11 for z/OS

SYSIBM.SYSINDEXCLEANUP Example § All index spaces in DB_1234 are enabled for cleanup on Sundays from 4:30 until noon, except – Index space IX_9876 is always disabled for cleanup

§ All index spaces in DB_XYZ disabled for cleanup on Saturdays, and – Index space IX_ABC is disabled for cleanup on the 30th of each month

from 1:30 to 7:30

DBNAME INDEX- SPACE

ENABLE_ DISABLE

MONTH _WEEK

MONTH DAY START _TIME

END _TIME

DB_1234 DB_1234 DB_XYZ DB_XYZ

NULL IX_9876 NULL IX_ABC

E D D D

W NULL W M

NULL NULL NULL NULL

7 NULL 6 30

043000 NULL NULL 001300

120000 NULL NULL 073000

SYSIBM.SYSINDEXCLEANUP

IDX

High Availability Improvements - NEDB2UG...* A new record is inserted into SYSPENDINGDDL table after...

Documents

Transcript of High Availability Improvements - NEDB2UG...* A new record is inserted into SYSPENDINGDDL table after...