Teradata SQL Tuning Ver 1

Teradata SQL Tuning

Teradata SQL Tuning

of 16

Teradata SQL Tuning

Value Measurement Framework

Business Technology InfrastructureAgility Adaptability

Efficiency % Time reduction in Design & Development

Increase in system performance

of 16

Teradata SQL Tuning

Contents

1. Introduction......................................................................................4Reduce the Workload.........................................................................................4Balance the Workload........................................................................................4Parallelize the Workload.....................................................................................4Upgrade..............................................................................................................4

2. Improve SQL statement tuning........................................................52.1 Reviewing the Execution Plan......................................................................52.2 Restructuring SQL statement.......................................................................5

2.2.1 AND or ‘=’ clause................................................................................................52.2.2 IN or BETWEEN clauses....................................................................................62.2.3 LIKE clause........................................................................................................62.2.4 IN and EXISTS clauses......................................................................................62.2.5 DISTINCT clause................................................................................................82.2.6 UNION or UNION ALL clauses..........................................................................82.2.7 CASE statement.................................................................................................92.2.8 SELECT DISTINCT clause................................................................................9

2.3 Eliminate the use of Temporary Tables......................................................102.4 Avoid Mixed Type Expressions...................................................................10

3. Indexes..........................................................................................123.1 Secondary Indexes.....................................................................................123.2 Join Index...................................................................................................12

3.2.1 Single table Join Index.....................................................................................133.2.2 Aggregate Join Index.......................................................................................133.2.3 Sparse Index....................................................................................................133.2.4 Global (Join) Index...........................................................................................143.2.5 Join Index performance....................................................................................14

3.3 Partitioned Primary Index...........................................................................153.4. Index Usage..............................................................................................16

of 16

Teradata SQL Tuning

1. Introduction

Performance of Teradata SQL based application depends on several factors, including database design, network latency, and query optimization, hardware specifications. Poorly tuned queries often cause performance problems. The objective of tuning a system is to either reduce the response time for end users of the system, or to reduce the resources used to process the same work. Both these objectives can be achieved in several ways:

Reduce the Workload

This is what commonly constitutes SQL tuning: finding more efficient ways to process the same workload. It is possible to change the execution plan of the statement without altering the functionality to reduce the resource consumption.

Two examples of how resource usage can be reduced are:

1. If a commonly executed query needs to access a small percentage of data in the table, then it can be executed more efficiently by using an index. Creation and utilization of such an index, reduces the amount of resources used.

2. If a user is looking at the first twenty rows of the 10,000 rows returned in a specific sort order, and if the query (and sort order) can be satisfied by an index, then the user does not need to access and sort the 10,000 rows to see the first 20 rows.

Balance the Workload

Systems often tend to have peak usage in the daytime when real users are connected to the system and low usage in the nighttime. If non-critical reports and batch jobs can be scheduled to run in the nighttime and their concurrency during daytime reduced, then it frees up resources for the more critical programs in the day.

Parallelize the Workload

Queries that access large amounts of data (typical data warehouse queries) often can be parallelized. This is extremely useful for reducing the response time in low concurrency data warehouse. However, for OLTP environments, which tend to be high concurrency, this can adversely impact other users by increasing the overall resource usage of the program.

Upgrade

Software/Hardware upgrade can be another means for attaining the desired performance levels and should be used only after trying everything from SQL tuning/data model restructuring and Database tuning perspective.

of 16

Teradata SQL Tuning

To ensure performance after a major / minor software or hardware upgrade, perform the following tasks on the production system.

Recollect statistics where possible. Save the EXPLAINs for the base queries. Run the test bed again after the upgrade and get the new EXPLAINs. Compare the two EXPLAINs. Check for faster or slower response on any of the test queries. If slower then check for the before and after performance EXPLAIN.

2. Improve SQL statement tuning

2.1 Reviewing the Execution PlanWhen tuning (or writing) a SQL statement in an OLTP environment, the goal is to drive from the table that has the most selective filter. This means that there are fewer rows passed to the next step. If the next step is a join, then this means that fewer rows are joined. Check to see whether the access paths are optimal.

When examining the optimizer execution plan, look for the following: The plan is such that the driving table has the best filter. The join order in each step means that the fewest number of rows are being returned to

the next step (that is, the join order should reflect, where possible, going to the best not-yet-used filters).

There are any unintentional Cartesian products (even with small tables). Each table is being accessed efficiently:

Consider the predicates in the SQL statement and the number of rows in the table. Look for suspicious activity, such as a full table scans on tables with large number of rows, which have predicates in the where clause. Determine why an index is not used for such a selective predicate.

A full table scan does not mean inefficiency. It might be more efficient to perform a full table scan on a small table, or to perform a full table scan to leverage a better join method (for example, hash_join) for the number of rows returned.

If any of these conditions are not optimal, then consider restructuring the SQL statement or the indexes available on the tables.

2.2 Restructuring SQL statementRewriting an inefficient SQL statement is often easier than repairing it. Since SQL is a flexible language, more than one SQL statement may meet the needs of your requirements. Although two SQL statements may produce the same result, one may be processed faster than the other. The results of the EXPLAIN statement can be used to compare the execution plans of the two statements and determine which is more efficient.

2.2.1 AND or ‘=’ clause

To improve SQL efficiency, it is advantageous to use equijoins whenever possible. Statements that perform equijoins on untransformed column values are easier to tune.

of 16

Teradata SQL Tuning

2.2.2 IN or BETWEEN clauses

In case of a choice of using the IN or the BETWEEN clauses in the query, it is advantageous to use the BETWEEN clause, as it is much more efficient. For example:

SELECT customer_number, customer_nameFROM customerWHERE customer_number in (1000, 1001, 1002, 1003, 1004);

is much less efficient than:

SELECT customer_number, customer_nameFROM customerWHERE customer_number BETWEEN 1000 and 1004

Assuming there is a useful index on customer_number, the Query Optimizer can locate a range of numbers much faster (using BETWEEN) than it can find a series of numbers using the IN clause.

2.2.3 LIKE clause

If LIKE is used in a WHERE clause, it is better to try to use one or more leading character in the clause, if at all possible. For example, use:

LIKE 'm%' not LIKE '%m'

If a leading character is used in the LIKE clause, then the Query Optimizer has the ability to potentially use an index to perform the query thereby, speeding performance.

But if the leading character in a LIKE clause is a wildcard, the Query Optimizer will not be able to use an index, and a table scan must be run, which reduces performance and takes more time.

2.2.4 IN and EXISTS clauses

In certain circumstances, it is better to use IN rather than EXISTS. In general, if the selective predicate is in the subquery, then use IN. If the selective predicate is in the parent query, then use EXISTS.

A subquery when used with an IN clause can take advantage of selectivity specified in the subquery. This is most beneficial when the most selective filter appears in the subquery and there are indexes on the join columns. Conversely, using EXISTS is beneficial when the most selective filter is in the parent query. This allows the selective predicates in the parent query to be applied before filtering the rows against the EXISTS criteria.

of 16

Teradata SQL Tuning

Below are two examples that demonstrate the benefits of IN and EXISTS. Both examples use the same schema with the following characteristics:

There is a unique index on the employees.employee_id field. There is an index on the orders.customer_id field. There is an index on the employees.department_id field. The employees table has 27,000 rows. The orders table has 10,000 rows.

Example 1:

This example demonstrates how rewriting a query to use IN can improve performance. This query identifies all employees who have placed orders on behalf of customer 144.

The following SQL statement uses EXISTS:

SELECT e.employee_id, e.first_name, e.last_name, e.salaryFROM employees eWHERE EXISTS (SELECT 1

FROM orders o WHERE e.employee_id = o.sales_rep_id AND o.customer_id = 144);

Rewriting the statement using IN, results in significantly fewer resources used. The SQL statement using IN:

SELECT e.employee_id, e.first_name, e.last_name, e.salaryFROM employees eWHERE e.employee_id IN (SELECT o.sales_rep_id FROM orders o WHERE o.customer_id = 144);

Explanation:

In the query using EXISTS, an extra unnecessary step is being performed by the parent query. From the sub-query, we obtain a table that has equijoined the tables employees and orders on the basis of employee_id and sales_rep_id. Further, the table is filtered for customer_id = 144. The obtained table is the resultant table needed but, an extra step of comparing of employee_id is done once again when the parent query is performed.

However, in the query using IN, the subquery returns only a filtered table on the basis of the specified customer_id and the join function is performed by the parent query. Thus, the work performed by the query using IN clause is much less than that of the query using the EXISTS clause.

Example 2:

This example demonstrates how rewriting a query to use EXISTS can improve performance. This query identifies all employees from department 80 who are sales reps who have placed orders. The following SQL statement uses IN:

of 16

Teradata SQL Tuning

SELECT e.employee_id, e.first_name, e.last_name, e.department_id, e.salaryFROM employees eWHERE e.department_id = 80 AND e.job_id = 'SA_REP' AND e.employee_id IN (SELECT o.sales_rep_id

FROM orders o);

The following SQL statement uses EXISTS:

SELECT e.employee_id, e.first_name, e.last_name, e.salaryFROM employees eWHERE e.department_id = 80 AND e.job_id = 'SA_REP' AND EXISTS (SELECT 1 FROM orders o WHERE e.employee_id = o.sales_rep_id);

Explanation:

In the query using IN, the subquery returns the entire orders table with the required column data. The rest of the filtering and the joining are done in the parent query.

However, in the query using EXISTS, the subquery returns a table equijoined between employees and orders table. The parent query then performs the filtration according to the specified conditions. Thus, the work performed by the query using EXISTS clause is much less than that of the query using the IN clause.

2.2.5 DISTINCT clause

At times this clause is added to every SELECT statement, even when it is not necessary. The DISTINCT clause should only be used in SELECT statements if it is known that duplicate returned rows are a possibility, and that having duplicate rows in the result set would cause problems with the requirements. The DISTINCT clause creates a lot of extra work, and reduces the physical resources that other SQL statements have at their disposal. Because of this, the DISTINCT clause is used only if it is necessary.

2.2.6 UNION or UNION ALL clauses

When using the UNION statement, keep in mind that, by default, it performs the equivalent of a SELECT DISTINCT on the final result set. In other words, UNION takes the results of two like recordsets, combines them, and then performs a SELECT DISTINCT in order to eliminate any duplicate rows. This process occurs even if there are no duplicate records in the final recordset. Selecting a distinct result requires building a temporary worktable, storing all rows in it and sorting before producing the output. If it is known that there are duplicate records, and this presents a problem for the application, then, the UNION statement should be used to eliminate the duplicate rows.

On the other hand, if it is known that there will never be any duplicate rows, or if there are, and this presents no problem to your application, then the UNION ALL statement should be used instead of the UNION statement. The advantage of the UNION ALL is that it does not perform the SELECT DISTINCT function, which saves a lot of unnecessary resources from being used. UNION ALL requires no worktable and no sorting. In most cases it’s much more efficient.

of 16

Teradata SQL Tuning

One more potential problem with UNION is the danger of flooding temporary database with a huge worktable. It may happen if a large result set is expected from a UNION query.

2.2.7 CASE statement

Often, it is necessary to calculate different aggregates on various sets of tables. Usually, this is done with multiple scans on the table, but it is easy to calculate all the aggregates with one single scan. Eliminating n-1 scans can greatly improve performance.

Combining multiple scans into one scan can be done by moving the WHERE condition of each scan into a CASE statement, which filters the data for the aggregation.

Example:

The following example asks for the count of all employees who earn less then 2000, between 2000 and 4000, and more than 4000 each month. This can be done with three separate queries:

SELECT COUNT (*)FROM employeesWHERE salary < 2000;

SELECT COUNT (*)FROM employeesWHERE salary BETWEEN 2000 AND 4000;

SELECT COUNT (*)FROM employeesWHERE salary > 4000;

However, it is more efficient to run the entire query in a single statement. Each number is calculated as one column. The count uses a filter with the CASE statement to count only the rows where the condition is valid. For example:

SELECT COUNT (CASE WHEN salary < 2000 THEN 1 ELSE null END) count1, COUNT (CASE WHEN salary BETWEEN 2001 AND 4000 THEN 1 ELSE null END) count2, COUNT (CASE WHEN salary > 4000 THEN 1 ELSE null END) count3 FROM employees;

2.2.8 SELECT DISTINCT clause

The DISTINCT option is used in a SELECT statement to filter out duplicate results from a query's output. In a simple SELECT from one table this is the easiest and quickest way of doing things. However, with a more complex query, the query can be recoded to gain a performance advantage. Consider the following example:

of 16

Teradata SQL Tuning

Example: The query returns authors that have a book already published.

SELECT DISTINCT au_fname, au_lnameFROM authors a JOIN titleAuthor t ON t.au_id = a.au_id;

The same result can be obtained by writing the query in the following manner,

SELECT au_fname, au_lnameFROM authors aWHERE EXISTS (SELECT *

FROM titleAuthor t WHERE t.au_id = a.au_id );

The second example gives a slightly better performance than the first one. The reason is that the EXISTS clause causes a name to be returned when the first book is found, and no further books for that author are considered (we already have the author's name, and only want to see it only once)

On the other hand, the DISTINCT query returns one copy of the author's name for each book the author has worked on, and the list of authors generated subsequently needs to be examined for duplicates to satisfy the DISTINCT clause.

The DISTINCT clause involves a worktable, which does not happen in the EXISTS clause.

AND has precedence over OR

2.3 Eliminate the use of Temporary Tables

As queries become more complex, temporary tables are used more and more. While temporary tables may sometimes be unavoidable, they can often be sidestepped by using derived tables instead.

A derived table is the result of using another SELECT statement in the FROM clause of a SELECT statement. By using derived tables instead of temporary tables, the application's performance can be boosted. Temporary tables slow the performance dramatically. The problem with temporary tables is the amount of overhead that goes along with using them. In order to get the fastest queries possible, the goal must be to make them do as little work as possible. The biggest benefit of using derived tables over using temporary tables is that they require fewer steps, and everything happens in memory instead of a combination of memory and disk. Fewer the steps involved, along with less I/O, faster is the performance.

2.4 Avoid Mixed Type Expressions

It is always better to avoid mixed-mode expressions, and beware of implicit type conversions. Consider the following example:

of 16

Teradata SQL Tuning

Example:

Assume a table with a column charcol defined as VARCHAR(2) and is also the primary index of the table. The following SQL query is run on this table,

of 16

Teradata SQL Tuning

SELECT * FROM <tablename>WHERE charcol = <numexpr>;

where, numexpr is an expression of numeric type. In SQL, the default implicit type conversion is from Character to Numeric data type. Thus, in the above query charcoal is converted to numeric data type.

Since the primary index has been defined on a column of VARCHAR data type and the implicit conversion makes it of numeric type, a full table scan is performed. This affects the performance of the query.This can be avoided by specifying an explicit conversion. The reformed query is,

SELECT * FROM <tablename>WHERE charcol = CAST (<numexpr> AS VARCHAR (2));

In this case, since the column on which the primary index has been defined, remains of the same type, a single AMP data retrieval takes place. This improvises the performance of the query.

3. Indexes

3.1 Secondary IndexesSecondary Indexes (SI) supply alternate access paths and the use of appropriate secondary indexes can increase retrieval performance. For best results, secondary indexes should be based on frequently used set selections and on equality search.A table can have up to 32 Secondary Indexes that can be created and dropped dynamically. However, it is not a good idea to create a number of SIs for each table just to speed up set selection because SIs consume the following extra resources:

SIs require additional storage to hold their subtables. In the case of a Fallback table, the SI subtables are Fallback also. Twice the additional storage space is required.

SIs require additional I/O to maintain these subtables.

When deciding whether or not to define a NUSI, there are other considerations. TheOptimizer may choose to do a Full Table Scan rather than utilize the NUSI in two cases:

When the NUSI is not selective enough. When no COLLECTed STATISTICS are available.

As a guideline, choose only those rows having frequent access as NUSI candidates. After the table has been loaded, create the NUSI indexes, COLLECT STATISTICS on the indexes, and then do an EXPLAIN referencing each NUSI. If the Parser chooses a Full Table Scan over using the NUSI, drop the index.

3.2 Join IndexJoin indexes are defined in a way that allows join queries to be resolved without accessing or joining their underlying base tables and also, reduce the number of rows processed in generating result sets. A join index is useful for queries where the index structure contains all the columns referenced by one or more joins, thereby allowing the index to cover all or part of the query. For a join index the join result is persistently stored and automatically maintained.

of 16

Teradata SQL Tuning

3.2.1 Single table Join Index

A single table join index is very useful for resolving joins on large tables without having to redistribute the joined rows across the AMPs. A single-table join index partitions all or a subset of a base table using a primary index based on the table’s foreign key (preferably the primary index of the table to which it is to be joined).

3.2.2 Aggregate Join Index

Aggregate join indexes offer an extremely efficient, cost-effective method of resolving queries that frequently specify the same aggregate operations on the same column or columns. When aggregate join indexes are available, the system does not have to repeat aggregate calculations for every query.

An aggregate join index can be defined on two or more tables, or on a single table. A single-table aggregate join index includes:

A columnar subset of a base table Additional columns for the aggregate summaries of the base table columns

An aggregate join index can be created using:

SUM function COUNT function GROUP BY clause

The use of an aggregate join index entails:

Initial time consumed to calculate and create the index. Whenever a value in a join-index column of the base table is updated, time to

recalculate the aggregate and update the index.

3.2.3 Sparse Index

Sparse indexes index a portion of the table using WHERE clause predicates to limit the rows indexed. Allowing constant expressions in the WHERE clause of the CREATE JOIN INDEX statement limits the rows that are included in the join index to a subset of the rows in the table based on an SQL query result. This capability in effect allows creation of sparse indexes. When base tables are large, this feature can be used to reduce the content of the join index to only the portion of the table that is frequently used if the typical query only references a portion of the rows.

A sparse index can focus on the portions of the tables that are most frequently used. This capability:

Reduces the storage requirements for a join index. Makes the costs for maintaining an index proportional to the percent of rows

actually referenced in the index.

of 16

Teradata SQL Tuning

3.2.4 Global (Join) Index

A Global Index is a term used to define a join index that contains the Row IDs of the base table rows. Queries may use the join index to qualify a few rows, then refer to the base tables to obtain requested columns that aren't stored in the join index. Such queries are said to be partially-covered by the index. This is referred to as a partially-covered global index.

Because the RDBMS supports multi-table, partially-covering join indexes, all types of join indexes, except the aggregate join index, can be joined to their base tables to retrieve columns that are referenced by a query but are not stored in the join index.

A partial-covering join index takes less space than a covering join index. Not all columns that are involved in a query selection condition have to be stored in a partial-covering join index. The benefits are:

Disk storage space for the join index decreases when fewer columns are stored in the join index.

Performance increases when the number of selection conditions that can be evaluated on the join index increases.

3.2.5 Join Index performance

3.2.5.1 Collecting Statistics

To provide the Optimizer with the information needed to generate the best plans, you need to have collected statistics on the primary index columns of each join index.Consider collecting statistics to improve performance during:

• Creation of a join index• Update maintenance of a join index

3.2.5.2 Cost considerations

Join indexes, like secondary indexes, incur both space and maintenance costs. For example, insert, update, and delete operations must be performed twice, once for the base table and once for the join index. However, if join indexes are suited to your applications, the improvements in query performance can far outweigh the costs.

3.2.5.3 Load utilities

MultiLoad and FastLoad utilities cannot be used to load or unload data into base tables that have an associated join index defined on them because join indexes are not maintained during the execution of these utilities. If an error occurs, the join index must be dropped and recreated after that table has been loaded. The TPump utility, which perform standard SQL row inserts and updates, can be used because join indexes are properly maintained during the execution of such utilities.

of 16

Teradata SQL Tuning

3.3 Partitioned Primary Index

The Partitioned Primary Index (PPI) feature allows a class of queries to access a portion of a large table, instead of the whole table. This is achieved by hashing rows to different virtual AMPs, as is done with a normal PI, but creating local partitions within each virtual AMP. Normal PI access remains unchanged, but in the case of a range query, for example, each virtual AMP is able immediately to focus its search on specific partitions within its workspace.PPI’s are defined on a table in order to increase query efficiency by avoiding full table scans without the overhead and maintenance costs of secondary indexes.

A Partitioned Primary Index (PPI) allows the data rows of a table to be:

Hash partitioned to the AMPs by the hash of the primary index columns Partitioned on some set of columns on each AMP Ordered by the hash of the primary index columns within that partition

Two functions, RANGE_N and CASE_N, can be used to simplify the specification of a partitioning expression.PPI improves performance as follows:

Automatic optimization occurs for queries that specify a restrictive condition on the partitioning column.

Uses partition elimination to improve the efficiency of range searches when, for example, the searches are range partitioned.

Only the rows of the qualified partitions in a query need to be accessed avoid full table scans.

Provides an access path to the rows in the base table while still providing efficient join strategies

If the same partition is consistently targeted, the part of the table updated may be able to fit largely in cache, significantly boosting performance

Benefits that are the result of using PPI vary based on:

The number of partitions defined The number of partitions that can be eliminated given the query workloads Whether or not the user follows an update strategy that takes advantage of partitioning.

However, the main disadvantages of using a PPI are:

The PI access disadvantage occurs only when the partitioning column is not part of the PI. In this situation, a query specifying a PI value, but no value for the partitioning column, must look in each partition for that value, instead of positioning directly to the first row for the PI value.

The direct join disadvantage occurs when another table with the same PI is joined with an equality condition on every PI column. For two non-PPI tables, the rows of the two tables will be ordered the same, and the join can be performed directly. If one of the tables is partitioned, the rows won't be ordered the same, and the task, in effect, becomes a set of sub-joins, one for each partition of the PPI table.

of 16

Teradata SQL Tuning

3.4. Index Usage

The EXPLAIN facility expresses the ‘confidence’ level for retrieval of data from a table. Some of the phrases used are:

…with high confidence Restricting conditions exist on index(es) or column(s) that have collected statistics.

…with low confidence Restricting conditions exist on index(es) having no statistics, but estimates can be based

upon a sampling of the index(es). Restricting conditions exist on index(es) or column(s) that have collected statistics but are

“AND-ed” together with conditions on non-indexed columns. Restricting conditions exist on index(es) or column(s) that have collected statistics but are

“OR-ed” together with other conditions.

…with no confidence Conditions outside the above.

The execution strategy provided by the EXPLAIN facility gives a direct feedback on what steps the optimizer chooses to do. Studying EXPLAIN outputs is an excellent way to know the inner workings of the Teradata DBS and how it handles SQL. Use of EXPLAIN facility regularly, saves a lot of time and computing resources by pointing out problems in SQL statements before they are actually run.

of 16

Teradata SQL Tuning Ver 1

Documents

Transcript of Teradata SQL Tuning Ver 1