Good Faq Teradata

14th May

This post is about row partitioning and will not discuss columnar.

To explain it correctly, let's get back to the basics...Each Teradata tables (except NoPI type) have "Primary Index", aka. PI, which is not physicalindex, but rather a logical construction: one or more columns of the table which give theinput for hashing method. The hash value determines two things:

Which AMP will store the record

Storing order of the records within the AMPs

If the PI is non-partitioned then the records are stored in order of hash value of PI.

If you use (row) partitioning, you define it at the Primary Index.In this case Teradata will associate a 2bytes or 2/8 bytes (at V14.10) "partition code" to therecord*, and the storing order is <partition code>,<hash_value> (aka. RowKey).That way partitions are not sub-tables or other physical objects, but only influence therecord storing order.

* This implies that no more than 64k(2bytes)/9Q(8bytes) partitions can exist. For detailsread the appropriate Teradata version's documentation.

NUSI (Non Unique Secondary Index) can serve as similar purposes, but is absolutelydifferent.NUSI is a separate subtable, with analogue PI to base table, but different (value) ordering.For details please read Teradata documentation.

Non partitioned table:create table tablePI( Trx_id Integer, Trx_dt Date)

How to use partitioned primary index (PPI) Part 1

How to use partitioned primary index(PPI)

What is partitioning?

What is the difference between PPI and NUSI?

How to define?

Dynamic Views template. Powered by Blogger.

TERADATA http://tdpank.blogspot.in/search?updated-min=2014-01-01T00:00:00-08...

1 of 10 8/16/2014 12:34 AM

create table tablePPI ( Trx_id Integer, Trx_dt Date)PRIMARY INDEX (Trx_id, Trx_dt**)PARTITION BY RANGE_N(Trx_dt BETWEEN DATE '2010-01-01' AND DATE'2013-12-31' EACH INTERVAL '1' DAY , NO RANGE, UNKNOWN)

Highlights

**Partitioning key (Trx_dt here) can be part of the PI or not. This is veryimportant, see below.

Partitioning can be single or multiple (MLPPI) levels***

RANGE_N or CASE_N functions can be used for determining partition code

RANGE_N function has constant interval endpoints and partition length.

NO RANGE and UNKNOWN partitions will store the out-of-intervals and null valuerecords respectively

***MLPPI is a technique when multiple or nested partitioning is defined on the table.Logically it looks like sub-partitions, but in practice it only influences the calculation ofpartition code values, which is still a linear 2/8 bytes value overall the table.

PPI is a very useful feature, but not a silver bullet to use it everywhere. Look the tradeoffs:

(+) Partition eliminationOnly the relevant partitions are scanned while accessing data

(+) Interval filtering is supported

(+) Accelerates INSERTsIf we load increment data into a populated table. Very likely less data blocks areaffected, since few partitions are involved (if date is the partitioning basis)

(-) 2 or 8 bytes extra space allocation per record

(-) Compression is not allowed on PartKey column

(-) PartKey inclusion problem (see below)

(-) Partition elimination works only with literalsSubselects cause full table scans

These functions are used to define partitioning. RANGE_N is for concentrate date (integer)intervals into partitions, while CASE_N is like a CASE-WHEN-THEN expression, where theoutcome is the partition.

Typically RANGE_N is used when we partition a transaction table by its date or timestamp,while CASE_N is popular in special cases like categorizing. You can use more columns in thelogical expression, but take care, all of them must be used in filter condition to enablepartition elimination.

Pros - Cons of using PPI

Design aspects

RANGE_N or CASE_N



2 of 10 8/16/2014 12:34 AM

If we load transactional data into our partitioned table, the date column we use aspartition key is populated later and later dates, while we have a finite partition rangedefinition.Partition ranges can be added to RANGE_N definition periodically (depends on version), orwe can define partitions in far advance. (365 partitions required for a year, 65k partitionscover ~ 180years, which is more than enough) Note that empty partitions do not allocatespace.

One of the methods above should be applied, otherwise the NO RANGE partition will growextensively, which will cause performance degradation due to less effective partitionelimination.

This is the funny point.Partitioning key is the column(s) that determines the partition, say used in theRANGE_N/CASE_N definition. We can include it in the Primary Index or not, we decide.

Let's take an example. We have a master-detail pair of tables, nicely "equi-PI"-ed foreffective join:

CREATE TABLE ORDER_HEAD( ORDER_NO INTEGER, ORDER_DT DATE) UNIQUE PRIMARY INDEX (ORDER_NO);

CREATE TABLE ORDER_ITEM( ORDER_NO INTEGER, ORDER_ITEM_NO, PROD_NO INTEGER) PRIMARY INDEX (ORDER_NO);

We modify ORDER_HEAD's PI:UNIQUE PRIMARY INDEX (ORDER_NO, ORDER_DT)PARTITION BY RANGE_N(ORDER_DT BETWEEN DATE '2010-01-01' AND DATE'2013-12-31' EACH INTERVAL '1' DAY , NO RANGE, UNKNOWN)

Should we include ORDER_DT or not? Which is better, what is the difference?

Not includeORDER_HEAD and ORDER_ITEM tables will have similar AMP distribution, butdifferent physical order within the AMPs.Each join operation requires sort of the selected ORDER_HEAD records in spool,or ORDER_ITEMS table will be merge joined against each selected non emptypartitions of ORDER_HEAD sequentially (called sliding-window merge join)

IncludeORDER_HEAD and ORDER_ITEM tables will have different AMP distribution, each

RANGE_N: interval extension or intervals in advance?

Partitioning Key: include in PI or not?



3 of 10 8/16/2014 12:34 AM

copy the ORDER_DT to the ORDER_ITEM table also, and use the same "Included" version ofPI. Requires some more space, logic in load time, but great gain while accessing data.

This select will eliminate all partitions except those three:select * from ORDER_HEAD where order_dt between '2013-12-12' (date)and '2013-12-14' (date);

This select will generate all rows scan:select * from ORDER_HEAD where cast( order_dt as char(7)) = '2013-12';

This select will generate all rows scan* either (sub-query):select * from ORDER_HEAD where order_dt in (select max(calendar_date)from sys_calendar.calendar where year_of_calendar=2013 andmonth_of_year=5);Why? Optimizer has to determine which partitions to be accessed in time of generatingexecution plan. That time it cannot know what is the result of the subquery. That is it.

* I got a proper comment on this option to double check. Yes, right, this information is aout-of-date. With actual versions of Teradata (V13.10..V14.10) I experienced 3 differentresults:

Full scanEg. sub-query contains a "group by"

Dynamic partition eliminationSub-query is simple, indicates "enhanced by dynamic partition elimination"section in the plan

Plan-time partititon eliminationLiteral condition or very simple sub query. Parsing time evaluation enables PO todetermine which partitions to be scanned. Plan: "...We do an all-AMPs ... stepfrom 3 partitions of...". Do not really know exactly what decides between fullscan, dynamic- or plan-time elimination... Explanations welcome.

We join two tables: T1 and T2. The table shows what happens if they are partitioned,not partitioned and the partitioning key is included or not in the PI:

T2

T1PI:(a) PI:(a) PART(b) PI:(a,b) PART(b)

PI:(a)Join: T1.a=T2.aRowHash match

PI:(a)PART(b)

Join: T1.a=T2.aT1 sorted by hash(a)orSliding-window MJ

Join: T1.a=T2.aT1&T2 sorted by hash(a)or Sliding-window MJ(NxM combinations)

Use cases

Filtering

Join



4 of 10 8/16/2014 12:34 AM

RowKey based MJ

PI:(a,b)PART(b)

Join: T1.a=T2.aT1 Redistributed &sortedby hash(a)

Join: T1.a=T2.aT1 Redistributed by hash(a)T2 sorted by hash(a) and MJJoin: T1.a=T2.a andT1.b=T2.bT2 Redistributed and sortedby RowKeyRowKey based MJ

Join: T1.a=T2.a andT1.b=T2.bRowKey based MJ

Let's take a transaction table like ORDERS. In practice we load it periodically (eg. daily)with the new increment which is typically focused to a short interval of transactiondate/time. If the ORDERS table is not partitioned, then the outstanding hashing algorithmwill spread them all over the data blocks of the table evenly, therefore Teradata has tomodify far more data blocks than the increment was reside in.

But if the ORDERS table is partitioned, then the physical order of the records is primarilydetermined by the partition key. This means that the increment will reside in very fewpartitions, close together, and the insert operation requires approx the same number ofblocks to be written than the increment was in.For more details on PPIs please refer the documentation of the appropriate Teradataversion.

To be continued... [http://teradatatuning.blogspot.hu/2014/04/using-partitioned-primary-index-ii.html]

Posted 14th May by pankaj agarwal

Insert

0 Add a comment

14th May

NPPI: Non partitioned primary indexThe good old regular PI. The rows are distributed byHASHAMP(HASHBUCKET(HASHROW(PI))), and ordered by HASHROW(PI), nothingspecial

How to choose partitioned primary index (PPI)

How to choose partitioned primary index(PPI)

What is the difference between NPPI and PPI?



5 of 10 8/16/2014 12:34 AM

below).

The only difference is the storing order of the records (and the 2/8 bytes overhead).

The partitioning feature - like in many other databases - usually solves some performanceissues, say enables to eliminate some needless work in specific situations.

SELECTEligible "where conditions" result serious partition-elimination, which means thatusually only a small fraction of the table should be scanned instead of the wholeone.

INSERTCheck the storing order of the NPPI tables: the records are in "hash" order, that isif I want to insert a series of records into a Teradata table, they will reside inspreadly distributed data blocks. If the table is big enough, my new eg. 1000records will get into ~1000 different data blocks, what means 1000 pieces ofexpensive "random writes". However if my 1000 records got to a PPI table andthey will have the same PartitionID, they will get into far less than 1000 datablocks with high probability. In real life situations we often will write tocontinuous data blocks with much cheaper "sequential write"

DELETESame as INSERT

BACKUPTeradata allows archiving only one or more partitions, saving lot of time andtape. Older data in transaction tables usually does not change therefore it isunnecessary to backup them every time

Like all good things in the world, partitioning has trade-offs also:

Extra 2/8 bytes per record allocated storage spaceDepending on maximal number of partitions. See "Number of partitions" chapter[https://www.blogger.com/blogger.g?blogID=1435643009978252687#numofpart]

Slower "SAMPLE" scansProper random sampling is more complex, since the physical storing order is incorrelation with partitioning value

Extra sort operations / Sliding window joinsIf joined to a table which has NPPI or PPI with not exactly same definition willresult a preparation "sort" step, or leads to a "sliding window merge join", whichis technically N x M merge joins between the partitions of TableA and TableB.

How many partitions should I have?How many partitions do I have?How is an empty partition looks like?They are all interesting questions, let's analyze the implementation of Teradata

What is PPI good for?

"Good-to-know"s

Costs of partitioning

[https://www.blogger.com/null] Number of partitions



6 of 10 8/16/2014 12:34 AM

determine the physical storing order of the record. A partition will not allocate space, an"empty partition" technically means that no record exists with that partition's partitionID,nothing else.How many partitions I have in the table? As many different PartitionID in the existingrecords occure, which depends on the occurring values of the partitioning column.How many partitions can I have in the table? It depends on the table definition. One mustuse the RANGE_N or the CASE_N function to define the PartitionID calculation. Its definitionunambiguously determines how many different PartitionID values may occur. In versions upto V13.10 65535 is allowed, from V14.00 we can have as many as 9.2 Quintillion (8 bytesPartitionID). The table definition cannot be altered to switch between 2 and 8 bytes layout.

What is the drawback of having many partition? The sliding-window merge join. Mindincluding partitioning column into the PI if possible (otherwise PI based filtering will causeas many accesses as many partitions exist).

We have the clauses NO RANGE and NO CASE in the PPI definition. They mean an ID valuefor that partition that is out of the defined range or case, those records got into thispartition. It can be a hidden trap, if you forget to maintain your date partition definition ona transaction table, and all records got to get into this partition from a moment. And thepartition keeps fattening, queries keep go slowing somehow...

This is a good trick. One can define partitioning "hierarchically", which is simply a"Cartesian product" of the partitions at each levels, the result is a single PartitionID. In caseof 2 bytes partitioning, the "Cartesian product" should fall below 65535.

What is sensational in the Teradata implementation of multi level PPI? You can filter onlylower level partitioning key(s) also, partition elimination will happen. How? It calculates allpossible combinations, and produces the PartitionID list to be scanned, excellent.

The next good question is: how fine should I define partitioning?It depends. Basically I'd branch to two main cases:

"Temporal" (date) partitioningThe best partition size is the day. Most of the filtering is on day level, and wehave ~365 days a year, not too much partitions for your lifetime. If we partitionon monthly units, then the partition elimination ranges are more rough, and wehave 12 partitions a year, which is also too much in case of a PI-NPPI join.

All othersIt really depends. Depends on the goal, and the value demographics. It's good tocorrelate with the filtering pattern (what is the frequent relevant 'where'condition parcel).

Hope it helped, please ask, if something is missing or confusing.


What happens with the out-of-range records?

Multi level partitioning

Partitioning granularity



7 of 10 8/16/2014 12:34 AM

14th May

Teradata is a diligent RDBMS that runs sometimes millions of SQLs a day. You will see themin the DBQL (DataBase Query Logging area) - if it is switched on - but it's a hard job to knowaround in that mess of queries. How can I find a specific query? What did that query run by?If I want to analyze or modify something I need to find the source of the execution asexactly as can be.Queryband is a labelling possibility to flag the queries to let their source job/report/etc. beeasily found.

Setting the queryband is usually the responsibility of the query runner:

ETL software or solution that executes it

OLAP tool that issues it

Person, who runs it ad-hoc

Technically it is a quite simple stuff: Teradata provides a command to set it:

SET QUERY_BAND = {'<variable1>=<value1>;<variable2>=<value2>;...' / NONE} [UPDATE] forSESSION/TRANSACTION;

, where:<variable1>=<value1>;Queryband can consist of arbitrary number of "variable"-"value" pairs. Both are stringvalues. Do not forget to put the semicolon after each variable-value pair!

NONE: clears the queryband

UPDATE: is specified, then those variables that has been previously defined are updated bythe new value, others are added with the given value. Empty value string is a valid contentand will not remove the variable. Please note that deleting a value is only possible byredefining the queryband without that specific variable.

SESSION/TRANSACTION: what it says...

How to use query banding in Teradata?

How to use query banding in Teradata?What is queryband?

Who defines the queryband?

How to set the queryband?



8 of 10 8/16/2014 12:34 AM

dbc.DBQLogtbl.QueryBand. The following example shows its content:

SET QUERY_BAND='PROJECT=TeraTuningBlog;TASK=QB_example;' for session;

(For the logged in session)SELECT queryband FROM dbc.sessioninfoX WHERE sessionNo=session;----------------------------------------------------PROJECT=TeraTuningBlog;TASK=QB_example;

(For the formerly ran queries)SELECT queryband FROM dbc.dbqlogtbl WHERE Queryid=...;----------------------------------------------------=S> PROJECT=TeraTuningBlog;TASK=QB_example;

(For a specific variable, eg. "PROJECT")SELECT QB_PROJECT FROM( SELECT CAST((case when index(queryband,'PROJECT=') >0 thensubstr(queryband,index(queryband,'PROJECT=') ) else '' end) AS VARCHAR(2050))tmp_PROJECT ,CAST( (substr(tmp_PROJECT,characters('PROJECT=')+1,nullifzero(index(tmp_PROJECT,';'))-characters('PROJECT=')-1)) AS VARCHAR(2050))QB_PROJECT FROM dbc.sessioninfoX WHERE sessionNo=session) x ;----------------------------------------------------TeraTuningBlog

(Which queries has been run by the "LoadCustomers" project?) SELECT a.*, CAST((case when index(queryband,'PROJECT=') >0 thensubstr(queryband,index(queryband,'PROJECT=') ) else '' end) AS VARCHAR(2050))tmp_PROJECT ,CAST( (substr(tmp_PROJECT,characters('PROJECT=')+1,nullifzero(index(tmp_PROJECT,';'))-characters('PROJECT=')-1)) ASVARCHAR(2050)) QB_PROJECT FROM dbc.dbqlogtbl a WHERE QB_PROJECT="LoadCustomers";

We know how to set the queryband, it's quite easy to build in / configure in the ETL tool,OLAP software and other query running applications. But what variables should we define,and how should we populate them? I give a best practice, but it is just a recommendation,can be modified due according to your taste.

First of all, some things to mind:

Designing querybanding



9 of 10 8/16/2014 12:34 AM

values for those items you later want to differentiate. Using a lookup/hierarchytable you can easily merge what you need, but never can drill down what isaggregated.

I recommend these variables to be defined:

SYS: Maximum of 3 characters ID of the system that ran the Query, like INF(Informatica), MST (Microstrategy), SLJ (SLJM), BO (Business Objects), AH (ad-hocquery tool)

ENV: P (Production) / Tx (Test x) / Dx (Development x), the identifier ofenvironment. x may be neglected, if it does not matter

JOB: Which job or report contains that specific query (the name of it)

STP: (Step) Which SQL script, or other sub-structure does the query belong to(name of it)

VER: Version of the JOB. This will determine the version of the script (ifavailable)


0 Add a comment



10 of 10 8/16/2014 12:34 AM

Good Faq Teradata

Documents

Transcript of Good Faq Teradata