POP/FED: Progressive Query Optimization for Federated Queries in DB2

Wook-Shin Han, Volker Markl, Stephan EwenVijayshankar Raman, Holger Kache

Goal: Add Robustness to Complex Federated QueriesEnvironment:Complex federated SQL queries, e.g. in decision support applications

Problem:Sub-optimal query performance due to erroneouscardinality estimates used during federated querycompilation/optimization

Results:Queries much more robust. Improvement in execution time by orders of magnitude.

Solution:Monitor federated plan during runtime and trigger reoptimization, reusing intermediate results.

SELECT YEAR AS YEAR,

SUM(CASE WHEN NATION = 'BRAZIL' THEN VOLUME ELSE 0 END) / SUM(VOLUME) AS MKT_SHAREFROM (SELECT YEAR(O_ORDERDATE) AS YEAR, L_EXTENDEDPRICE*(1-L_DISCOUNT)

AS VOLUME, N2.N_NAME AS NATION FROM

ORA.SUPPLIER, ORA.CUSTOMER,ORA.NATION N1, ORA.NATION N2, ORA.REGION, DB2.LINEITEM, DB2.ORDERS,

WHERE S_SUPPKEY = L_SUPPKEY AND L_ORDERKEY = O_ORDERKEY AND O_CUSTKEY = C_CUSTKEY AND C_NATIONKEY = N1.N_NATIONKEY AND N1.N_REGIONKEY = R_REGIONKEY AND R_NAME = 'AMERICA' AND S_NATIONKEY = N2.N_NATIONKEY AND O_ORDERDATE BETWEEN

DATE('1995-01-01') AND DATE('1996-12-31')

) AS ALL_NATIONS

GROUP BY YEARORDER BY YEAR;

Example Query Without POP

Total execution time:57 sec.

Plan with POP

CHECK:Operator 3RANGE [47287, 1000000]:CARDINALITY: 17916OUT OF BOUNDS!Reoptimizing!

Reoptimize 3x

Total execution time:107 sec.

Query

Checked & Rematchable Code

Parser

Semantics

Rewrite

Code Generator

TAOB5) Add Matching Information

4) Generate CHECK Code

Thread:28F: CHK (0010) jf=2A0:0 op=13 lb=a ub=b2A0: TA (001F) jf=2C0:0 jb=0:0 evlo=0:0 taob=158:02C0: JNF (0008) jf=2D8:0 jump=308:0

CHECKCHECK

CHECK

CHECK

CHECK

TAOBTAOB

Optimizer

3) Place CHECKs

2) Place dams for eager materialization

1) Compute Validity Ranges

Checked Plan[lb..ub]

[lb..ub][lb..ub]

Check[lb..ub]

Temp

Code withCHECK

MatchableTAOBs

Plans withjudicious

materialization

Plans withoptimalitycriterion

[lb2..ub2]

[lb1..ub1]

[lb3..ub3]

Cheaper

MoreExpensive

CHECK

Query Compiler Runtime

8) Save Intermediate Results

Section Closure

Query ResultRelational Data Services

Compiler to Runtime Transition

TAOB

CHECK

CHECK

TAOB

Section Execution

6) Drop Redundant Interm. results

CHECK

TAOBTAOB

TAOB

Subsumes

TAOB

TAOB

TAOB TAOB

Cardinality ???

7) Execute Checks

Intermediate Results / Additional knowledge

Interm.Result

Interm.Result

Interm.Result

View Matching

RD

S M

on

ito

r C

ou

nte

rs

CHECKErr

a++

b++

c++

d++

e++

f++

lb < actual card < ub ?

Reoptimize ! N

OK Y

9) Match Previus Results

VMVVMV

Interm.Res.VMV

Legend

Est: 182K VR: [47K, 10M]

Act: 17.9K

Est: 50 VR: [25, 1M]

Act: 5

SHIP

DB2.LineitemOrders

Sort

Scan

NLJN

SHIP

ORA.Supplier

MGJN

Filter

Sort

ORA.NationRegion

Sort

GrpBy

Ret.

HSJN

SHIP

ORA.Customer

SHIP

NLJN

ORA.Nation

SHIP

Temp

Scan

Temp

Scan Scan

Scan

Est: 120M VR: [10M, INF]

Act: 1.2M

SHIP

DB2.LineitemOrders

Sort

Scan

NLJN

SHIP

ORA.Supplier

MGJN

Filter

Sort

ORA.NationRegion

Sort

GrpBy

Ret.

HSJN

SHIP

ORA.Customer

SHIP

NLJN

ORA.Nation

SHIP

Temp

Scan

Temp

Scan Scan

Scan

Check

Temp

Check

Scan

Check

HSJN

Sort

GrpBy

Ret.

MGJN

Filter

VMV

NLJN

ORA.Supplier

SHIP

SHIP

ORA.Nation

VMV

Scan

TAOB Table Object

Section Executable Code and Objects

[lb..ub] Validity Range[lower .. upper bound]

Virtual Materialized View

Remote Statement Object

Oracle Nickname

Data / Knowledge

Control Flow

VMV

SHIP

ORA

POP/FED Monitor