Highly successful performance tuning of an informix database
Informix physical database design for data warehousing
-
Upload
keshav-murthy -
Category
Technology
-
view
2.148 -
download
0
Transcript of Informix physical database design for data warehousing
0
Physical Database design for Data Warehousing
Keshava Murthy,
Architect, IBM Informix Development
1Source: Forrester
Query Tools
Analytics
BPS Apps
BI Apps
LOB apps
Databases
Other transactional data sources
I/O & data loading
Query processing
DBMS & Storage mgmt
Enterprise Data Warehouse
2
3
4
Data Warehouse Schema and Queries.
• Characterized by:
– “Star” or “snowflake” schema:
– Complex, ad hoc queries that typically
• Look for trends, exceptions to make actionable business decisions
• Touch large subset of the database (unlike OLTP)
• Involve aggregation functions (e.g., COUNT, SUM, AVG,…)
City
Region
Store
SALES
Product
Period
Brand
Month
Quarter
Category
Dimensions
Fact Table
5
Store Sales ER-Diagram from TPC-DS300GB database
287,997,024
20
73,049
1,920,800
1000
204,000
1,000,000
402
86,400
7200
2,000,000
6
select s_store_name, s_store_id,
sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales,
sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales,
sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales,
sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales,
sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales,
sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales,
sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales
from store_sales, store, date_dim
where d_date_sk = ss_sold_date_sk and
s_store_sk = ss_store_sk and
s_gmt_offset = -5 and
d_year = 2002
group by s_store_name, s_store_id
order by s_store_name, s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
Aggregates
Fact table
Dimension tables.
Equijoins between primary(dimension) and
foreign keys(fact)
Predicates on the dimension tables
Grouping and ordering.
7
select first 100 i_item_id,
avg(ss_quantity) agg1,
avg(ss_list_price) agg2,
avg(ss_coupon_amt) agg3,
avg(ss_sales_price) agg4
from store_sales, customer_demographics, date_dim, item, promotion
where ss_sold_date_sk = d_date_sk and
ss_item_sk = i_item_sk and
ss_cdemo_sk = cd_demo_sk and
ss_promo_sk = p_promo_sk and
cd_gender = 'F' and
cd_marital_status = 'M' and
cd_education_status = 'College' and
(p_channel_email = 'N' or p_channel_event = 'N') and
d_year = 2001
group by i_item_id
order by i_item_id;
Aggregates
Fact tableDimension tables.
Equijoins between primary(dimension) and
foreign keys(fact)
Predicates on the dimension tables
Grouping and ordering.
8
1) dwa_ds.store_sales: INDEX PATH (SKIP SCAN)
(1) Index Name: martinfu.sold_date_store_salesIndex Keys (Detached): ss_sold_date_sk (Parallel, fragments: ALL)Lower Index Filter: dwa_ds.store_sales.ss_sold_date_sk = stream from dwa_ds.date_dim.d_date_sk
2) dwa_ds.customer_demographics: SEQUENTIAL SCAN
Filters:Table Scan Filters: ((dwa_ds.customer_demographics.cd_education_status = 'College' AND dwa_ds.customer_demographics.cd_marital_status = 'M' ) AND dwa_ds.customer_demographics.cd_gender = 'F' )
DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_cdemo_sk = dwa_ds.customer_demographics.cd_demo_sk
3) dwa_ds.promotion: SEQUENTIAL SCAN
Filters:Table Scan Filters: (dwa_ds.promotion.p_channel_event = 'N' OR dwa_ds.promotion.p_channel_email = 'N' )
DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_promo_sk = dwa_ds.promotion.p_promo_sk
4) dwa_ds.item: SEQUENTIAL SCAN
DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_item_sk = dwa_ds.item.i_item_sk
5) dwa_ds.date_dim: SEQUENTIAL SCAN
Filters:Table Scan Filters: dwa_ds.date_dim.d_year = 2001
DYNAMIC HASH JOIN (Index Push Down Key: dwa_ds.date_dim.d_date_sk to dwa_ds.store_sales)Dynamic Hash Filters: dwa_ds.store_sales.ss_sold_date_sk = dwa_ds.date_dim.d_date_sk
9
Store_sales
customer_demographics
item
date_dim
promotion
287,997,024
300GB database
1,920,800
204,000
73,049
1000
(p_channel_email = 'N' or p_channel_event = 'N')
d_year = 2001
cd_marital_status = 'M' and
cd_education_status = 'College'
10
Store_sales
customer_demographics (1.9M)
Item (200K)
Promotion (1000)
287,997,024
100GB database
date_dim (73K)
Skip Scan
Sequential Scan
Hash Join:
ss_cdemo_sk = cd_demo_sk
Hash Join: ss_promo_sk = p_promo_sk
Hash Join: ss_item_sk = i_item_sk
Hash Join ss_sold_date_sk = d_date_sk
(p_channel_event = 'N' OR p_channel_email = 'N' )
Index Push Down Key: (d_year = 2001)d_date_sk to dwa_ds.store_sales
((cd_education_status = 'College'
AND cd_marital_status = 'M' ) AND cd_gender = 'F' )
Sequential Scan
Sequential Scan
Sequential Scan
Index scan Sort rowids
Probe
Build
Build
Build
Build
Probe
Probe
Probe
11
Store_sales
customer_demographics (1.9M)
Item (200K)
Promotion (1000)
Start Fact table scan
300GB database
date_dim (73K)
Skip
Scan
Sequential Scan
Hash Join:
ss_cdemo_sk = cd_demo_sk
Hash Join: ss_promo_sk = p_promo_sk
Hash Join: ss_item_sk = i_item_sk
Hash Join ss_sold_date_sk = d_date_sk
(p_channel_event = 'N' OR p_channel_email = 'N' )
Index Push Down Key: (d_year = 2001)d_date_sk to dwa_ds.store_sales
((cd_education_status = 'College'
AND cd_marital_status = 'M' ) AND cd_gender = 'F' )
Sequential Scan
Sequential Scan
Sequential Scan
FactIndex scan
Sort rowids
Probe
Build
Build
Build
Build
Probe
Probe
Probe
Open
Open
Open
Open
Open
next
next
next
next
next
12
Intersection
of rowids
13
Tasks for database design
• Logical design
• Physical design
• Hardware configuration
• Informix configuration
• Feedbacks, changing needs.
14
• Reduce IO
• Improve IO throughput
• Improve CPU and network
• Improves administration efficiency
• Do more with less
Why is physical database design necessary?
Performance. Performance. Performance.
15
1. Data type selection
2. Indexes
3. Summary tables
4. Compression
5. Memory allocation
6. Table partitioning
Tasks of physical database design
16
• Numeric is faster than character
• BIGINT, BIGSERIAL is faster than int8, serial8
• Fixed char is faster than varchar, lvarchar
• All the character types exploit light scan
• Larger types means larger indices
• Date-time-columns use integer in fact table..
• (RDS – storing the DOMAINS for publishing)
Data type selection
17
18
create table "dwa_ds".web_sales(
ws_sold_date_sk integer,ws_sold_time_sk integer,ws_ship_date_sk integer,ws_bill_customer_sk integer,ws_bill_cdemo_sk integer,ws_bill_hdemo_sk integer,ws_bill_addr_sk integer,ws_ship_customer_sk integer,ws_ship_cdemo_sk integer,.....ws_quantity integer,ws_wholesale_cost decimal(7,2),ws_list_price decimal(7,2),ws_sales_price decimal(7,2),ws_net_paid decimal(7,2),ws_net_paid_inc_tax decimal(7,2),ws_net_paid_inc_ship decimal(7,2),ws_net_paid_inc_ship_tax decimal(7,2),ws_net_profit decimal(7,2),primary key (ws_item_sk,ws_order_number) disabled
);
19
Select d_month,i_category,sum(ws_net_paid) as total_sum,substr(i_category,2) as lochierarchy
fromweb_sales
,date_dim,item
whered_year = 2002
and d_date_sk = ws_sold_date_skand i_item_sk = ws_item_skgroup by i_category,i_classorder by d_month, i_category;
20
1. Very few indices
2. Leaner indices – Informix can combine when necessary
3. Keep load, insert performance in mind
1. When possible disable
2. Attach/detach performance with indexes
4. Scan – Scan – Scan
5. Provide better options for Optimizer
6. Improve scan for tables – more for dimension than facts
7. Exploit multi-index scans
8. Star join can work with indexes or without indexes.
Index Design
21
1. Handle the “canned” queries by pre-computing the answers
2. Saves system resources for handling complex ad-hoc queries
3. Can create multiple levels of summaries
-- sales by region, by part, by (day, week, month)
4. Need time to create the summary tables ahead of time
Summary tables
Fact table > billons of rows
Summary level 1 (day)
Summary level 2 (week)
Summary level 3 (week)
22
1. Very few indices
2. Leaner indices – Informix can combine when necessary
3. Keep load, insert performance in mind
1. When possible disable
2. Attach/detach performance with indexes
4. Scan – Scan – Scan
5. Provide better options for Optimizer
6. Improve scan for tables – more for dimension than facts
7. Exploit multi-index scans
8. Star join can work with indexes or without indexes.
Traditional Compression
23
Compression On Data Page With Multiple Rows
compress repack
Uncompressed Compressed Compressed
shrink
Multiple Compressed
Pages
Empty Data Pages
Animated
Slide
• To use repack and shrink, the data need not be compressed
• repack and shrink can be done independently
• Shrink does not do an implicit repack
24
1. IDS can configure page sizes that are multiples of the
2. default page size
3. Depending on OS can be 2k, 4k, 8k, or 16k
4. Each page size will necessitate its own buffer pool
5. Can be useful if OS supports larger page sizes for buffer transfers.
6. Ensure adequate memory exists
Pagesize Considerations
25
• Motivation
–Performance, Performance, Performance
–Performance of the queries
–Performance of Warehouse tasks
• Extract jobs
• Load performance
• Transformation
• Summary data creation
26
• Starting point
– Logical design
• Observations
– Presence or absence of indices and constraints
• What are your expected maintenance tasks
– Daily tasks
– Weekly tasks
– Monthly tasks
– Quarterly tasks
– Yearly tasks
27
Topics for the day
• Create table:fact an dimension
• Create Index: Deciding to which indices to create and their keys and attributes
• Attach/detach
• Drop/Truncate
• SELECT, UPDATE, DELETE
• Update Statistics
• Distribution
28
Features• External table
• Options for loading/unloading
• Fragment level stats
• Fragmentation – interval/etc
• PDQ
• Interplay with configuration
• Keeping things ONLINE
29
Fragmentation = Partitioning
• In Informix, they are synonymous
• Informix was the first to implement this feature. Industry is using partitioning for some time now
• Within Informix syntax, we’re making fragmentation and partitioning synonymous.
• Informix Fragmentation has nothing to do with disk fragmentation.
30
Fragmentation = Partitioning
• In Informix, they are synonymous
• Informix was the first to implement this feature. Industry is using partitioning for some time now
• Within Informix syntax, we’re making fragmentation and partitioning synonymous.
• Informix Fragmentation has nothing to do with disk fragmentation.
31
Situation
1. OLTP– Data volume
– performance
2. Data Warehousing– Data Volume
– Performance
– Data Management
3. Mixed workload– Do (1) and (2) together
32
Problem areas and Opportunities
• Capacity
• Storage IO throughput
• Performance
• Availability
• Storage management
• Reliability
• Time cyclic data management
• Security
33
Capacity• By default, a table gets one partition
• Limited number of pages in each partition
• Capacity is limited by the number of pages.
– 2KB pagesize on most platforms.
– 4KB pagesize on Windows and AIX
• Increase the capacity by creating dbspace with up to 16KB pagesize.
• Compress the table table to reclaim space
• Consider fragmenting the table.
34
Storage IO throughput• Warehouse and complex queries and DDL
operations depend on IO throughput significantly.
• By creating multiple fragments in multiple dbspaces, you give Informix opportunity to parallelize operations
35
Performance• Enable parallel operations for queries
– SELECT
– INSERT
• Exploit fragment elimination (aka partition pruning)
• Improve create index timings
• Improve update statistics operation
36
Availability• Fragments distributed in multiple disks or
volumes are more tolerant of failure
• If a disk failed, you can skip over those fragments via DATASKIP
37
Storage management• Manage the data distribution on different storage
disks or storage managers
• Match disk speeds to different quality of service
– Recent data on faster/expensive storage
38
Time cyclic data management• The table would maintain specific period of data.
– Last 25 hours
– Last 3 months or 13 months
– Last 7 years
• At every interval do the three operations
– Detach the data no longer needed
– Attach the new data or make room for it.
– Modify the expressions to represent the new window
39
4Q2008 1Q2009 2Q2009 3Q2009 4Q2009 1Q2010 2Q2010
working window
Time cyclic data management
40
Security• Grant or revoke access per fragment
– Grant and revoke is done on the tables
• Example
– Table is fragmented on states
– Table is fragmented on departments
– Table is fragmented on date or datetime
41
Fragmentation Overview
42
DBSPACE
CHUNK CHUNK CHUNK
Extent Extent
Extent
CHUNK
Pages
Pages
Pages
Partition
Extent
43
Customer_table Partition
idx_cust_id
Customer_table
Partition
Paritition
Storesales_table
Idx_store_id
Partition
Partition
Tables, Indices and Partitions
44
Fragment1 Fragment2 Fragment3 Fragment4
Fragmentation by Round Robin
Fragmentation by Expression
September October November December
45
Fragmentation by Round Robin
CREATE TABLE customer (id int, state char(2))
FRAGMENT BY ROUND ROBIN
in dbspace1, dbspace2, dbspace3;
CREATE TABLE customer (id int, state char(2))
FRAGMENT BY ROUND ROBIN
PARTITION part1 IN dbspace1,
PARTITION part2 IN dbspace1,
PARTITION part3 IN dbspace2;
part1 part2 part3
46
Fragmentation by Round Robin
• Predictable distribution of data and use of storage.
• Create indices with either round robin or expression strategy
• No fragment elimination possible
• Use expression strategy for indices to get benefit of elimination
• Index expression depends on the access pattern.
• Blobs avoid extra writes in RR strategy.
47
Fragmentation by expressionCREATE TABLE customer (id int, state char (2), zipcode decimal(5,0))
FRAGMENT BY EXPRESSION
(state = “CA“) in dbspace1,
(state = “KS") in dbspace2,
(state = “OR") in dbspace3,
(state = “NV") in dbspace4;
CREATE TABLE customer (id int, state char (2))
FRAGMENT BY EXPRESSION
PARTITION part1 (state = “CA") in dbspace1,
PARTITION part2 (state = “KS") in dbspace1,
PARTITION part3 (state = “OR") in dbspace1,
PARTITION part4 (state = “NV") in dbspace1;
CA KS OR NV
48
Fragmentation by expression
CREATE TABLE customer (id int, state char (2), zipcode
decimal(5,0))
FRAGMENT BY EXPRESSION
partition partca93 (state = “CA“ and zipcode < 93000) in dbspace1,
partition partcagt93 (state = “CA“ and zipcode >= 93000) in dbspace5,
PARTITION part2 (state = “KS") in dbspace2,
PARTITION part3 (state = “OR") in dbspace2,
PARTITION part4 (state = “NV") in dbspace3;
KS OR NVCA
< 93000
CA
>= 93000
49
Fragmentation by expression
CA KS OR NV
• Destination of the row depends on the row data
• Design of expressions requires understanding and estimation of data sets
• Expressions provide a flexible mechanism
• With flexibility comes complexity
• Can make fragment elimination complex
50
jan_partition
feb_partition
march_partition
april_partition
may_partition
june-partition
july_partition
august_partition
september_partition
october_partition
november_partition
1/1/2010 to 1-31-2010
2/1/2010 to 2-28-2010
3/1/2010 to 3-31-2010
4/1/2010 to 4-30-2010
5/1/2010 to 5-31-2010
9/1/2010 to 9-30-2010
8/1/2010 to 8-31-2010
7/1/2010 to 7-31-2010
6/1/2010 to 6-30-2010
10/1/2010 to 10-31-2010
11/1/2010 to 11-30-2010
Orders Table for 2010
51
Attached Index on a Fragmented Table
• Large table DSS or OLTP environment.
• Attractive index parallel scans.
• Attractive index fragment elimination and smaller btrees.
• Attractive scans on data pages in parallel.
• Balanced I/O for indexes and data pages.
52
Detached Fragmented Index on a Non-fragmented Table
• OLTP environment with high index hits vs. data page hits (key only reads).
• Attractive index scans in parallel
• Attractive index lookups with fragment elimination and smaller btrees.
• Unattractive scans on data pages in series.
53
Detached Index on a Fragmented Table
• DSS environment with some selective queries.
• Attractive scans on data pages in parallel.
• Unattractive index read in series.
54
Detached Fragmented Index on a Fragmented Table
• Mixed OLTP and DSS environments with data fragmented for DSS and index fragmented of OLTP or Selective queries and non-selective queries on different columns in a DSS environment.
• Attractive index parallel scans.
• Attractive index fragment elimination and smaller btrees.
• Attractive scans on data pages in parallel.
• Balanced I/O for indexes and data pages.
55
Three things important in determining
your strategy and expressions
Workload. Workload. Workload.
56
Create table history (
Docid char(14),
..
) fragment by expression
Partition p1 (docid >= “abcdef20112001” and docid<(babcde20111000) in dbs1,
Partition p2(docid >= “babcde20111000” and ..
Docid: abcdef20110001
57
Factors to consider
• Query Performance
– What kind of queries?
– Data Distribution
– Which applications use this table?
• Storage Management
– How do you handle data growth?
– What’s the table management strategy when dataset grows
58
Fragmentation Objectivesscan threads *
fragments
ParallelismFragments are accessed in
parallel, decreasing scan or insert
time.
Fragment EliminationUnneeded fragments are
eliminated, decreasing scan or
insert time, and reducing disk
contention.
scan threads *
fragments
scan threads *
fragments
Fragment Elimination &
ParallelismBoth goals are achieved.
XX XX
XX
59
OLTP characteristics:
• high volume of short transactions
• each transaction accesses a few rows
• index access method is used.
For this environment:
• Fragmentation by round
robin or expression strategy
• Reduce index access times using fine grain expressions
on the index fragment.
– Fragment recent data by day and older data by week or month.
60
Data Warehousing characteristics
• low volume of long running queries
• queries access most of the rows in each fragment
• very few indexes are generally required
• preferred access method is sequential scan
• preferred join method is the hash join
For this environment:
• fragment elimination
• parallel scan the needed fragments
61
• Multi-threaded Dynamic Scalable Architecture (DSA)– Scalability and Performance
– Optimal usage of hardware and OS resources
• DSS Parameters to optimize memory– DSS queries
– Efficient hash joins
• Parallel Data Query for parallel operations– Light scans, extensive
– calculations, sorts, multiple joins
– Ideal for DSS queries and batch operations
• Data Compression
• Time cyclic data mgmt– Fragment elimination, fragment
attach and detach
– Data/index distribution schemas
– Improve large data volume manageability
– Increase performance by maximizing I/O throughput
• Configurable Page Size
– On disk and in memory
– Additional performance gains
• Large Chunks support
– Allows IDS instances to handle large volumes
• Quick Sequential Scans
– Essential for table scans common to DSS environments 17
Top IDS features utilized for building warehouse
Source:
62
• Multi-threaded Dynamic Scalable Architecture (DSA)– Scalability and Performance
– Optimal usage of hardware and OS resources
• DSS Parameters to optimize memory– DSS queries
– Efficient hash joins
• Parallel Data Query for parallel operations– Light scans, extensive
– calculations, sorts, multiple joins
– Ideal for DSS queries and batch operations
• Data Compression
• Time cyclic data mgmt– Fragment elimination, fragment
attach and detach
– Data/index distribution schemas
– Improve large data volume manageability
– Increase performance by maximizing I/O throughput
• Configurable Page Size
– On disk and in memory
– Additional performance gains
• Large Chunks support
– Allows IDS instances to handle large volumes
• Quick Sequential Scans
– Essential for table scans common to DSS environments 17
Top IDS features utilized for building warehouse
Source:
Fragmentation Features
63
Fragmentation Objectives
scan threads *
fragments
ParallelismFragments are accessed in
parallel, decreasing scan or insert
time.
Fragment EliminationUnneeded fragments are
eliminated, decreasing scan or
insert time, and reducing disk
contention.
scan threads *
fragments
scan threads *
fragments
Fragment Elimination &
ParallelismBoth goals are achieved.
XX XX
XX
64
Typical Query with Non-PDQ vs. PDQ
Scan
Join
Sort
Send to client
Scan Scan
Sort
Scan
Join
Sort
Send to client
65
PDQ/Fragmentation
• Consider fragmenting any large table in a dbspace that is getting a lot of IO activity
• Consider fragmenting any large table if scans must be done against the table
• Do not put multiple fragments of the same table on the same physical device– Be aware of the I/O throughput of your storage
• Avoid using round robin fragmentation for indexes.
• Do not over-fragment.
– The cost of managing fragmentation can outweigh the benefits when there are excessive fragments.
66
PDQ Configuration
• MAX_PDQPRIORITY
– Set highest percentage of PDQ resources that a single client can use
• b
– Max number of DSS queries that can be run together
• DS_TOTAL_MEMORY
– Total memory reserved for PDQ
• DS_MAX_SCANS
– Max number of parallel scans allowed. Leave at default (1048567)
67
PDQ Configuration
• If the site is primary a DSS system, then it is recommended that most of the allocated memory be in the virtual buffers and that DS_TOTAL_MEMORY be very large
• PDQ can be used in smaller memory environments by setting PDQ_PRIORITY to 1 so that parallel scans can be done.
68
PDQ Configuration
• onmode can be used to dynamically change PDQ parameters
– onmode –M (DS_TOTAL_MEMORY)
– onmode –Q (DS_MAX_QUERIES)
– onmode –D (MAX_PDQPRIORITY)
– onmode –S (DS_MAX_SCANS)
69
Design for Time Cyclic data mgmtcreate table mytrans(
custid integer,
proc_date date, store_loc char(12)
….
) fragment by expression
......
(proc_date < DATE ('01/01/2009' ) ) in fe_auth_log20081231,
(MONTH(proc_date) = 1 ) in frag2009Jan ,
(MONTH(proc_date) = 2 ) in frag2009Feb,….
(MONTH(proc_date) = 10 and proc_date < DATE ('10/26/2009' ) ) in frag2009Oct ,
(proc_date = DATE ('10/26/2009' ) ) in frag20091026 ,
(proc_date = DATE ('10/27/2009' ) ) in frag20091027,
(proc_date = DATE ('10/28/2009' ) ) in frag20091027 ,
(proc_date = DATE ('10/29/2009' ) ) in frag20091027 ,
(proc_date = DATE ('10/30/2009' ) ) in frag20091027 ,
(proc_date = DATE ('10/31/2009' ) ) in frag20091027 ,
(proc_date = DATE ('11/01/2009' ) ) in frag20091027 ,
;
70
Can eliminateCan eliminateCan eliminateEquality expression
Cannot eliminateCannot eliminateCan eliminateRange expression
Nonoverlapping
Multiple column key
Overlapping on a
single column key
Nonoverlapping
Single fragment key
Type of filter (WHERE clause)
Fragment elimination
71
create table f1(a int, b varchar(32)) fragment by expression
partition p1 (a = 1) in rootdbs,
partition p2 (a = 2) in rootdbs,
partition p3 (a = 3) in rootdbs,
partition p4 (a = 4) in rootdbs,
partition p5 (a = 5) in rootdbs;
insert into f1 select 1, 'oneworld' from systables;
insert into f1 select 2, 'twoworld' from systables;
insert into f1 select 3, 'threeworld' from systables;
insert into f1 select 4, 'fourworld' from systables;
insert into f1 select 5, 'fiveworld' from systables;
insert into f1 select * from f1;
create index f1ab on f1(a,b);
create index f1b on f1(b);
update statistics high for table f1;
72
QUERY: (OPTIMIZATION TIMESTAMP: 9-29-2010 00:56:12)
------
select * from t1 where a=1
Estimated Cost: 5
Estimated # of Rows Returned: 136
1) keshav.t1: INDEX PATH
(1) Index Name: keshav.t1abIndex Keys: a b (Serial, fragments: 0)
Fragments Scanned: (0) p1 in rootdbsLower Index Filter: keshav.t1.a = 1
73
QUERY: (OPTIMIZATION TIMESTAMP: 9-29-2010 01:06:54)
------
select * from t1 where a <3
Estimated Cost: 11
Estimated # of Rows Returned: 272
1) keshav.t1: SEQUENTIAL SCAN (Serial, fragments:
0, 1)
Fragments Scanned: (0) p1 in rootdbs, (1) p2 in
rootdbs
Filters: keshav.t1.a < 3
74
select count(*) from t1 where a = 3 and b =
'threeworld'
Estimated Cost: 1
Estimated # of Rows Returned: 1
1) keshav.t1: INDEX PATH
(1) Index Name: keshav.t1ab
Index Keys: a b (Serial, fragments: 2)
Fragments Scanned: (2) p3 in rootdbs
Lower Index Filter: (keshav.t1.b =
'threeworld' AND keshav.t1.a = 3 )
75
select count(*) from t1
where a = a+1 and b = 'threeworld'
Estimated Cost: 11
Estimated # of Rows Returned: 1
1) keshav.t1: INDEX PATH
Filters: keshav.t1.a = keshav.t1.a + 1
(1) Index Name: keshav.t1b
Index Keys: b (Serial, fragments: ALL)
Lower Index Filter: keshav.t1.b = 'threeworld'
76
Managing Change• You have a round robin strategy, but running out of
space
• You have a round robin strategy, but need to modify it into expression expression strategy
• Time Cyclic changes aka Roll-on Roll-off
– Daily, weekly, monthly, yearly
– Hourly anyone?
– I’m running out of space in a fragment!
– I have uneven distribution of data
– I can’t have any down-time!
77
Managing Change
• You have a round robin strategy, but running out of space
– Analyze how frequently you’re adding new fragments
– Consider moving to higher pagesize (2K to 16K)
– All fragments of the table have to be in same pagesize.
– But, the indices can be in a different pagesize than table. But, all fragments in an index use same pgsize
– On 11.50.xC4 or above, you have another option!• You can compress the table data
– Consider moving to higher pagesize (2K to 16K) for further capacity
78
Managing Change• You have a round robin strategy, but
need to modify it into expression expression strategy
–You need exclusive access to the table
–You need to schedule the down time.
–ALTER FRAGMENT… INIT
79
Managing Change
• Time Cyclic changes aka Roll-on Roll-off
4Q2008 1Q2009 2Q2009 3Q2009 4Q2009 1Q2010
Current working set
Older rolled-off data
New data waiting to
roll on
2Q2010
80
Managing Change
• Time Cyclic changes aka Roll-on Roll-off
Q1 Q2 Q3 Q4
Q1 Q2
81
Managing Change• Time Cyclic changes aka Roll-on Roll-off
– Daily, weekly, monthly, yearly
– Consider the data distribution and the queries on it.
– Fine granularity can help speed up analysis
– Consider hybridcreate table t(a int, b date) fragment by expression
partition p1 (month(b) = 10) in dbs,
partition p2 (month(b) = 11) in dbs,
partition p3 (b >= date("12/01/2009") and b <= date("12/15/2009")) in
dbs,
partition p4 (b >= date("12/16/2009") and b <= date("12/31/2009")) in
dbs;
– Test your strategies to ensure fragment elimination.
82
Source: Forrester
Query
Tools
Analytics
BPS
Apps
BI
Apps
LOB
apps
Databases
Other
transactional
data sources
I/O & data loading
Query processing
DBMS & Storage mgmt
Interconnect & Networking
Enterprise Data Warehouse Platform
83
Source: Forrester
Query
Tools
Analytics
BPS
Apps
BI
Apps
LOB
apps
Databases
Other
transactional
data sources
I/O & data loading
Query processing
DBMS & Storage mgmt
Interconnect & Networking
Enterprise Data Warehouse Platform
Data Loading
External Table
Online
attach/detach
Data & storage
management
Deep Compession
Interval and List
Fragmentation
Online attach/detach
Fragment level stats
Storage provisioning
Table defragmenter
Query Processing
MERGE
Hierarchical Queries
Light Scans
Multi-Index Scan
Skip Scan
Bitmap Technology
Star and Snowflake
join optimization
Implicit PDQ
Tooling
Microstrategy
Cognos
Pentaho
Jaspersoft
SQW
84
• Multi-threaded Dynamic Scalable Architecture (DSA)
– Scalability and Performance
– Optimal usage of hardware and OS resources
• DSS Parameters to optimize memory
– DSS queries
– Efficient hash joins
• Parallel Data Query for parallel operations
– Light scans, extensive
– calculations, sorts, multiple joins
– Ideal for DSS queries and batch operations
• Data Compression
• Time cyclic data mgmt
– Fragment elimination, fragment attach and detach
– Data/index distribution schemas
– Improve large data volume manageability
– Increase performance by maximizing I/O throughput
• Configurable Page Size
– On disk and in memory
– Additional performance gains
• Large Chunks support
– Allows IDS instances to handle large volumes
• Quick Sequential Scans
– Essential for table scans common to DSS environments
Top IDS features utilized for building warehouse
85
New fragmentation Strategies in Informix v11.70
• List Fragmentation
–Similar to expression based fragmentation
–Syntax compatibility
• Interval Fragmentation
–Like expression, but policy based
– Improves availability of the system
86
Time Cyclic Data management
• Time-cyclic data management (roll-on, roll-off)
• Attach the new fragment
• Detach the fragment no longer needed
• Update the statistics (low, medium/high) to keep everything up to date.
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield field
fieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
JanJan FebFeb MarMar AprApr
May 09May 09
Dec 08Dec 08
enables storing data over time
87
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
fieldfieldfieldfieldfieldfieldfield
JanJan FebFeb MarMar AprApr
May 09May 09
Dec 08Dec 08
• ATTACH, DETACH and rest of ALTERs require exclusive access– Planned Downtime
• These can be scripted, but still need to lock out the users– Informix 11.50.xC6 has DDL_FORCE_EXEC to lock out the users
• Expression strategy gives you flexibility, but elimination can be tricky.
Time Cyclic Data management
88
Fragment by Expressioncreate table orders
(order_num int,order_date date,customer_num integer not null,ship_instruct char(40),backlog char(1),po_num char(10),ship_date date,ship_weight decimal(8,2),ship_charge money(6),paid_date date ) partition by expression
partition prv_partition (order_date < date(’01-01-2010’)) in mydbs,partition jan_partition (order_date >= date(’01-01-2010’) and
order_date < date(’02-01-2010’) in mydbs,partition feb_partition (order_date >= date(’02-01-2010’) and
order_date < date(’03-01-2010’) in mydbs,partition mar_partition (order_date >= date(’03-01-2010’) and
order_date < date(’04-01-2010’) in mydbs,partition apr_partition (order_date >= date(’04-01-2010’) and
order_date < date(’05-01-2010’) in mydbs,…
89
Fragment by Interval
Interval Value
Initial Partition
Partition Key
dbspaces
create table orders(order_num int,order_date date,customer_num integer not null,ship_instruct char(40),backlog char(1),po_num char(10),ship_date date,ship_weight decimal(8,2),ship_charge money(6),paid_date date )
partition by range(order_date) interval(1 units month)store in (dbs1, dbs2)partition prv_partition values < date(’01-01-2010’) in dbs3;
90
Interval Fragmentation
• Fragments data based on an interval value
– E.g. fragment for every month or every million customer records
• Tables have an initial set of fragments defined by a range expression
• When a row is inserted that does not fit in the initial range fragments, IDS will automatically create fragment to hold the row (no DBA intervention)
• No X-lock is required for fragment addition
• All the benefits of fragment by expression
91
ONLINE attach, detach
• ATTACH
– Load the data into a staging table, create the indices exactly as you have in your target table.
– Then simply attach the table as a fragment into another table.
• DETACH
– Identify the partition you want to detach
– Simply detach the partition with ONLINE keyword to avoid attemps to get exclusive access
92
Attach Example
ALTER FRAGMENT ONLINE ON TABLE “sales”.orders
ATTACH december_orders_table as PARTITION december_partition
values < 01-01-2011;
93
Attach Example
94
Attaching online
95
December_orders_table
Table to attach
orders
query1 query2
Issue ALTER ATTACH ONLINE
query1query2
Query1 and Query2 continue and won’t access the new partition
Attach
Modify the dictionary entry to indicate online attach
is in progress. Other sessions can read the list but
cannot modify.
query1query2
query3
New queries will work on the table and won’t
consider the table fragment for the queries.
ONLINE ATTACH operation is complete.
Table is fully available for queries
query3
query4
Get exclusive access to the partion list (in the
dictionary) .The dictionary entry gets modified
and new dictionary entries for new queries from
here on
New queries will work on the table and willconsider the new table fragment .Attaching online
96
ONLINE operations• ATTACH a fragment• DETACH a fragment• MODIFY transition value• Automatic ADDing of new fragments on insert
or update• tasks eliminated by interval fragmentation
– Scheduling downtime to get exclusive access for ADD, ATTACH, DETACH
– Defining proper expressions to ensure fragment elimination
– Running of update statistics manually after ALTER operations
– Time taken to collect statistics is reduced as well.
97
UPDATE STATISTICS during ATTACH, DETACH
• Automatically kick-off update statistics refresh in the background –need to enable fragment level statistics
• tasks eliminated by interval fragmentation–Running of update statistics manually
after ALTER operations–Time taken to collect statistics is
reduced as well.
98
List fragmentation
CREATE TABLE customer
(id SERIAL, fname CHAR(32), lname CHAR(32), state CHAR(2), phone CHAR(12))
FRAGMENT BY LIST (state)
PARTITION p0 VALUES ("KS", "IL", "IN") IN dbs0,
PARTITION p1 VALUES ("CA", "OR", "NV") IN dbs1,
PARTITION p2 VALUES ("NY", "MN") IN dbs2,
PARTITION p3 VALUES (NULL) IN dbs3,
PARTITION p4 REMAINDER IN dbs3;
99
Smarter Statistics Collection
100
Fragment Level Statistics (FLS)
• Generate and store column distribution at fragment level
• Fragment level stats are combined to form column distribution
• System monitors UDI (Update/Delete/Insert) activities on each fragment
• Stats are refreshed only for frequently updated fragments
• Fragment level distribution is used to re-calculate column distribution
• No need to re-generate stats across entire table
101
Generating Table Level Statistics
• Distribution created for entire column dataset from all fragments.
• Stored in sysdistrib with (tabid,colno) combination.
• Dbschema utility can decodes and display encoded distribution.
• Optimizer uses in-memory distribution representation for query optimization.
DataDistribution
CacheFeed
Sorted
Data
Feed
Column
Data
Store
Encoded
Distribution
Decode
Distribution
BinGenerator& Encoder
SORT
SysdistribCatalogtable
Frag1
Frag2
Fragn
102
Generating Fragment Level StatisticsData
Distribution Cache
Feed
Sorted
Data
Feed
Column
Data
Store
Encoded
MinibinsDecode
Distribution
Mini-BinGenerator& Encoder
SORT
SysfragdistCatalogTable
Frag1
Frag2
Fragn
SORT
SORT
Mini-BinGenerator& Encoder
Mini-BinGenerator& Encoder
Mini-Bin Merger & Bin
Encoder
SysdistribCatalogTable
SORT
Feed
decode
Minibins
Store
Encoded
Distribution
103
STATLEVEL property
STATLEVEL defines the granularity or level of statistics created for the table.
Can be set using CREATE or ALTER TABLE.
STATLEVEL [TABLE | FRAGMENT | AUTO] are the allowed values for STATLEVEL.
TABLE – entire table dataset is read and table level statistics are stored in sysdistrib catalog.
FRAGMENT – dataset of each fragment is read an fragment level statistics are stored in new sysfragdist catalog. This option is only allowed for fragmented tables.
AUTO – System determines when update statistics is run if TABLE or FRAGMENT level statistics should be created.
104
UPDATE STATISTICS extensions
• UPDATE STATISTICS [AUTO | FORCE];
• UPDATE STATISTICS HIGH FOR TABLE [AUTO | FORCE];
• UPDATE STATISTICS MEDIUM FOR TABLE tab1 SAMPLING SIZE 0.8 RESOLUTION 1.0 [AUTO | FORCE ];
• Mode specified in UPDATE STATISTICS statement overrides the AUTO_STAT_MODE session setting. Session setting overrides the ONCONFIG'sAUTO_STAT_MODE parameter.
105
UPDATE STATISTICS extensions
• New metadata columns - nupdates, ndeletes and ninserts –in sysdistrib and sysfragdist store the corresponding counter values from partition page at the time of statistics generation. These columns will be used by consecutive update statistics run for evaluating if statistics are stale or reusable.
• Statistics evaluation is done at fragment level for tables with fragment level statistics and at table level for the rest.
• Statistics created by MEDIUM or HIGH mode (column distributions) is evaluated.
• The LOW statistics is saved at the fragment level as well and is aggregated to collect global statistics
106
Alter Fragment Attach/Detach• Automatic background refreshing of column statistics after
executing ALTER FRAGMENT ATTACH/DETACH on a table with fragmented statistics.
• Refreshing of statistics begins after the ALTER has been committed.
• For ATTACH operation, fragmented statistics of the new fragment is built and table level statistics is rebuilt from allfragmented statistics. Any existing fragments with out of date column statistics will be rebuilt at this time too.
• For DETACH operation, table level statistics of the resulting tables are rebuilt from the fragmented statistics.
• The background task that refreshes statistics is “refreshstats”and will print errors in online.log if any are encountered.
107
Not yetYesYesYesCreate index ONLINE
YesYesYesYesFLS
YesNoNoNoStorage Provisioning
YesYesYesYesSmarter Stats
Yes -- MODIFY transition value
NoNoNoMODIFY ONLINE
YesNoNoNoDETACH ONLINE
YesNoNoNoATTACH ONLINE
YesYesYesNoEquality Expression
YesYesYesNoRange Expression
YesYesYesYesParallelism
IntervalExpression
ListRound Robin
1088/26/2011Session TPM3 108