1Q 2002 FAQs

Q1:

What is the difference between a Teradata macro and a stored procedure? Which should I use when?

A1:

Although both a macro and a stored procedure can, in Teradata, be used for some of the same things—e.g., they both support parameterized SQL—they are really very different.

A macro has very restricted control logic that is limited to performing or not performing ABORT statements, which undo changes. A stored procedure, on the other hand, supports extensive procedural logic and more sophisticated error handling. It can also perform looping logic or pass along statement values during a procedure. However, macros can return a multirow answer set, while stored procedures can only send back a single set of values through the parameter list.

Under some conditions, such as the straightforward insert of multiple rows, a macro will perform better than a stored procedure because it bundles all statements into one multistatement reques—a single parsing-and-recovery unit. Stored procedures do not support multistatement requests, and each insert within a procedure is sent individually to the parser and dispatched as a separate request, eliminating the potential for parallel single-row inserts.

For simple activities, particularly those involving multiple, repetitious SQL or requiring multirow answer sets, use macros; where the required logic is beyond what a macro can accommodate, turn to stored procedures.

Q2:

How is data redistributed when I add new hardware to an existing Teradata configuration?

A2:

A reconfiguration utility is provided with the DBMS to enable you to add new hardware with minimal intervention and system unavailability. Hash buckets are assigned evenly across all parallel units (AMPs) within the Teradata system and are used by the file system to target data row placement so a table's data is evenly distributed across all AMPs. When new nodes (and with them new AMPs) are added, a subset of these hash buckets is reassigned in order to include the new nodes in the even spread of data. Data rows associated with the reassigned hash buckets are migrated onto the new AMPs on the new nodes.

This utility only moves the proportion of data equivalent to the percentage of increase in number of AMPs, while leaving the majority of the rows untouched. All tables and indexes are automatically relocated over the internal high speed interconnect, in parallel on all nodes simultaneously, by issuing just a single command. When the reconfigure is complete, the entire system, including all tables and indexes, is immediately and fully available, without the need to rewrite load scripts or make any changes to jobs, queries or database parameters.

Q3:

Can Teradata support star schemas?

A3:

Teradata has handled "star" and "snowflake" joins for a long time. Often, the optimizer will build a plan that initiates a product join among several dimension tables after qualifying them and then redistributing the result so it can easily join the large fact table. Teradata

customers, especially those in retail, have used this technique very effectively since 1988.

Snowflake joins occur when star tables are joined to other tables in the query. Some of those joins can reduce the cardinality of the star tables. The optimizer recognizes joins that reduce cardinality and executes them prior to performing the product join, resulting in fewer rows and, thus, a shorter merge join phase. The algorithm is sophisticated enough to recognize multiple star joins in a single query and to recognize when one of the star joins is a snowflake of another.

Most Teradata optimizations for star joins rely on a large fact table with a composite primary index of columns that are the equi-join fields for the dimension tables. All of the dimension tables with a join field included in the primary index must be included in the star join. This allows the optimizer to join the dimension tables to produce the composite primary index and thus execute a high-performing merge join known as a "row hash match scan."

Teradata is very good at dealing with queries that do not fit a pattern, no matter what the underlying physical model. You won't limit the type of queries you can ask by using dimensional modeling in your Teradata system. The Teradata optimizer can build an optimal plan for any query involving the tables, regardless of whether it fits the star join conventions.

For this reason, and because the Teradata optimizer is good at delivering on complex requests, and additionally because many users have found it easier to add subject areas using a third normal form model, a normalized schema is often recommended. In addition, many users find it easier to add subject areas with a normalized model. This recommendation is often misinterpreted to mean that third normal form is required with Teradata, when in fact the Teradata technology is completely neutral when it comes to the data model.

Q4:

What happens when one query runs faster than another other during a synchronized scan?

A4:

Teradata can perform a sync scan (synchronized full table scan) if more than one query is scanning the same table. It does this by starting the second query scan at the same data block in the table where the first query is positioned. Then the queries share the reading of data blocks going forward. When the first one finishes, the second query continues the scan until it reaches its starting point.

While synchronization is encouraged, it's not forced. The optimizer selects sync scan only on tables too big to fit into the data cache. Small table blocks are likely to be cached, and therefore their I/Os are already potentially shared.

Once a sync scan begins, the DBMS makes a note of it and periodically checks the progress of all participants. It keeps an eye on how many tables are involved in the scan and how much memory is available to support the activity, and it monitors each query's progress. When synchronization is no longer optimal, the DBMS will spin off one or more of the participants, preventing fast queries from being hampered by slow queries.

Q5:

When does Teradata reuse spool files within the same request?

A5:

The Teradata optimizer can generate a query plan that will produce a spool file or temporary work file as output in one step for use as input in multiple subsequent steps. This can save significant processing time, and the optimizer will choose to do this whenever it senses that the same intermediate file will be built in different places in the plan. For instance, it will reuse files when a subquery accesses the same tables with the same selection criteria as the outer query, even when the subquery goes on to aggregate the resulting data and the outer query does not.

Correlated subqueries often benefit from reusable spools, as do multistatement requests containing multiple insert/select statements that all select from the same source. Teradata will select from the source table once into a re-usable spool file, which will then feed inserts to the several tables.

Under some conditions, table triggers will result in a plan where reusable spool files maintain both the base table and a trigger-specified table. When tables containing multiple join indexes are updated in a batch mode, e.g., by means of an insert/select operation, the prejoining happens once and the results are held in a reusable spool file, which is then applied to each of the participating join indexes. This eliminates the need to do the same prejoin activity for each join index structure requiring maintenance.

2Q 2002 FAQs

Q1:

I am the Medical Director for a large (2.4 million member) healthplan insurer, and we are taking a serious look at an integrated enterprise decision support system that uses Teradata as the back-end warehouse and platform. I should note that although I am a physician, I also oversee the medical informatics area, and I am computer (hardware, software, database and SQL) savvy enough to be dangerous.

We are currently a mainframe shop with primarily DB2 data marts, which are called warehouses, and a number of distributed and specialized data marts in Oracle residing on UNIX platforms. This is an old company, and there is a lot of vested interest amongst some areas of the company in the existing systems, which some people had direct input in developing (many years ago).

My purpose in writing and my question is this. One of the "objections" I hear being raised from these areas about potentially moving to the new structure is that they cannot write SQL directly against the warehouse in a Teradata world. And, since that is true (in their view of the world) then obviously this has no value for power users and will not meet the needs of the organization. My view of the world is that we have been brought up on traditional relational databases (one can argue about DB2 filling that role) and the standard use of SQL query tools.

We (including I) do not yet know enough about star schema architecture and how to understand and access the data model. But, I refuse to believe that it is not possible to write queries directly against the warehouse. Healthcare is certainly not unique in its need to access data in many different and powerful ways.

So, can one write SQL directly against a Teradata warehouse? If not, how does one go directly against the data, bypassing the GUI front ends that power users hate in order to do complex queries in this environment?

A1:

If this is the only issue then we better put your system on the truck and ship it to you!

Seriously though, this is exactly what we do. Our whole philosophy is to integrate all the data in the organization into a single enterprise view—a logical data model that represents the enterprise. We enable access for all users who have a question, regardless of scope or size. All of those questions are asked directly against the enterprise copy of the data, not against extracted versions or data marts. Over time, we would help you consolidate all the disparate, disjointed decisioning systems you currently have into a single enterprise decision solution—one copy of the data, one version of the truth.

Among our current customers, most use one or more query tools (GUI tools, as you call them) from the leading BI tool vendors (Cognos, Business Objects, MicroStrategy ...). However, we expect each of those tools to write SQL to send to Teradata, receive the answer set back and format the reports as specified by the end user. Some of these tools can ask very complex questions and submit very sophisticated SQL to Teradata. For a small set of power users, these tools may well not be flexible enough to formulate the required SQL. When that is the case we have both a batch SQL query tool and GUI-style SQL query tool, which allow users to code SQL queries/scripts to their hearts' content.

Even better, though, is that Teradata loves complex queries. The more complex, the better. The more joins, expressions and nesting, the better. I keep a QFH file where users send me queries they or some tool have written. The current leader is a 25-page SQL statement (in 10-point font) that accesses 56 different tables in the data model. Derived tables, subqueries, case expressions, statistical functions (stdev, skew, kurtosis and linear regression) are all included in our ANSI-compliant SQL implementation. And we have implemented a new SQL construct called Ordered Analytics that allows what used to be complex jobs or large offloads to become simple SQL statements. Rank, Quantile, Moving average, Moving Sum—all can be done inside the DBMS. Where other DBMSs force multi-step scripts and intermediate tables to handle many way joins, we want you to write the single SQL statement in all of its glory and let Teradata's optimizer figure out how to execute it.

Speaking of the optimizer, we have a No Hint mechanism; instead of requiring you to tell us which joins to execute in which order and what indexes to use when, our optimizer performs all of those jobs automatically and consistently so you do not have to have an IT expert between your users and their data. I personally guarantee that your power users and your DBAs will be the most religious converts to Teradata once you bring it in. The freedom the power users will gain and their ability to answer questions that they could never before even consider asking will make them ask how they got along without it for this long.

If your power users are asking data mining questions, we have a data mining user interface that allows them to do very high-powered SQL underneath. Statistical analysis, decision trees, logistic regression and many other functions are supported, all taking advantage of the parallel power of the Teradata platform by writing very complex SQL.

Regarding star schema, we recommend to our customers that they implement a physical database design that closely mirrors the logical data model for the organization. Thus, on Teradata systems the physical model tends to be a normalized model, with only occasional denormalizations or summaries for especially performance-sensitive, high-volume queries. Star schema is okay if you know the questions ahead of time, but a normalized model can answer the next question.

Stars are notoriously difficult to extend to a new business area, but a normalized model is easy to add to and extend as the enterprise's questions grow and extend. An enterprise model

allows the SQL query writers to have maximum flexibility across all of the areas of the business, e.g. matching patients with procedures, costs and outcomes all in one query to the database. The industry will tell you that star schema is required to avoid database performance problems and overly complex queries. Teradata's optimizer and execution engine make those issues moot so that you can afford to get the value of an enterprise model.

Highmark and Anthem both are enthusiastic users of Teradata in your industry. One of my favorite customer stories comes from Anthem where, among many uses, Teradata helps analyze patient outcomes from several different perspectives. The implementation there helped identify doctors, hospitals and procedures that had higher percentages of negative outcomes. By identifying those situations so Anthem could correct them, Teradata contributed directly to better patient care. We have a lot of good stories about helping companies grow, save money or improve profitability from the revenue they get, but it is even more fun when we can help save lives.

Q2:

A person designs a temporary table and loads that table with many records. These records have many columns with "like" information, except for the Primary Index. From a Load and Extract perspective, is it more efficient to define the primary index as the first column in that table or can it be defined anywhere within the record with the same efficiency and speed using SQL/Queryman?

A2:

The position of the primary index column(s) as defined in the CREATE TABLE statement has no performance impact on loading, accessing or extracting table rows with Teradata. This is true whether the structure is a permanent or a temporary table. In fact, the file system often rearranges columns within a row before it physically stores the row. For example, any variable-length columns are moved to the end of the row.

The row hash of an individual row is used to directly access the row, and also to reflect where the row should be physically stored at the time it is inserted. Row hash information is carried in the header of each row, no matter where in the row the columns it is based on reside.

Information held within the table header (of which there is a copy on each AMP) shows the offset of each column within the row. Teradata will use this offset value to find the primary index column directly in the actual row—when it is needed—without having to search the entire row.

If the column selected for a primary index is, in fact, a variable-length column, then a small amount of additional overhead will exist, in that we have to go not to one, but to two different locations in the header information to determine the offset for that column. This is very minor, but in general, fixed length data types are preferable for primary index columns.

Q3:

I've been perusing the archives of Teradata Magazine and recently found a Spring 2000 piece Todd Walter authored in which he nicely explains the wonderful uses of SQL Views. You close this section with the statement that Teradata views entail no performance hit because "Teradata handles views differently from other DBMS products in this regard." Todd then refers to a query rewrite process that results in references to the base tables on which the view is defined, and he comments that the

rewritten query is what gets passed to the optimizer.

I'm wondering to which DBMS systems you refer as those that handle views differently than what you describe briefly. If I understand what you say in this piece, this is what IBM's research prototype, System R, did and is what DB2 has always done. I would be surprised if NonStop SQL and some others I might name didn't do the same thing in this regard.

The rewrite involves taking a query that references a view (in a FROM clause) and logically "merging" it with the original query that defined the view. This merged query, with references to the base tables only, is what the optimizer works on ("roughly speaking"). Isn't this (again "roughly") the sort of thing you say Teradata does?

A3:

There are easy views and hard views. An easy view would be a simple projection of a single table. A hard view might include some or all of joins, aggregation, subqueries, derived tables, etc. Most DBMSs today handle the easy views. As the views become more difficult, they switch to materializing the contents of the view and then performing the query over the result. Informix is the worst, followed quickly by Oracle. You can find in their classes and documentation warnings to avoid the use of any but the simplest views. The Data Mart databases are really bad or don't even support views, such as RedBrick or Sybase IQ.

3Q 2002 FAQs

Q1:

How do you adjust the time on a Teradata system. How would I adjust the time if the 'get time' Ferret command shows the time as 03:06:11, whereas the exact time is 03:36:11?

A1:

Adjusting time across all the nodes in an MPP system is handled by a UNIX command. First, to view the time on each node, issue this UNIX command:

rallsh -sv dateTo change all the nodes to a given time, issue this command from any node, and it will be proliferated out to all nodes that are BYNET connected:rallsh -sv date mmddHHMM2002mm=2-digit monthdd=2-digit dayHH=2 digit hourMM=2d digit minute

Only months, days, hours and minutes are changeable, not seconds.

Q2:

When we set Priority Scheduler Facility (PSF) settings on a Teradata server for rationing CPU time, does it affect all the CPUs in all the nodes of Teradata MPP system?

A2:

Priority Scheduler setup commands are issued by means of the UNIX schmon utility. Commands may be entered on any node in the system and will be proliferated out to all other nodes. All nodes will have the same, exact settings. Each node, however, will have its own instance of priority scheduler, and will manage its node's priorities independently of other nodes in the system. For more information, please refer to the Teradata RDBMS

Utilities Guide.

Q3:

Please validate that the following script will backup a Teradata database.

ARCHIVE Script:logon demo1099/dbc,dbc;ARCHIVE DATA TABLES (CRMDEMO),RELEASE LOCK,INDEXES,FILE=textcrm;LOGOFF;RESTORE:logon demo1099/dbc,dbc;RESTORE DATA TABLES (CRMDEMO),RELEASE LOCK,FILE=textcrm;LOGOFF;

Would the archive/restore option automatically update the Data Dictionary entries?

A3:

Your script is valid. You have an INDEX keyword in the script that will archive all the USI & NUSI for every table on the database. That typically, will increase the time taken for the archive and need more tapes. Also, when restoring, you will need to BUILD your indexes. May we suggest a better way, which would be to recreate the SIs after restoring.

Q4:

I have to load NULL into a date field if the input has '?'. The code is:

FIELD SALES_TRANS_DT = CHAR (08) NULLIF SALES_TRANS_DT = '?';

Note that the input is a text field of length 8 bytes. If the input field has ? in the first byte, the TPump works. If the ? is in any other position other than the first byte, TPump writes the error log with the message "Invalid date supplied for TEST11.SALES_TRANS_DT."

The same is the case for other type of fields like SMALLINT & DECIMAL. How do you solve this riddle?

A4:

Since the field is character, comparing it to '?' is equivalent to comparing it to '? '.

You can't say "nullif position('?' in sales_trans_dt) > 0", so you will have to clean up the input before running TPump.

Q5:

How can an individual user view what queries he/she currently has running on the Teradata system? We recently experienced a spool space problem while running a passthrough SAS job to Teradata and thought the query was not getting terminated on the Teradata side after canceling the job in SAS. The spool was not releasing after the jobs completed for some reason.

A5:

One suggestion is to use PMON if you are running V2R4.1.1 of Teradata. It shows what users are active. If there are processes blocking a user, for example in version (5.0.0), you

can pull up the SQL currently being executed (if the process is still active) by a given user. There is a lot of other information that it provides...this is just a part.

Another suggestion is to use Teradata Manager to help you research the issue.

Or, if you just run a select from dbc.sessioninfox that will get all sessions that your userid has running. Don't know if in SAS passthrough you logoff and on again for each query as that would have a bearing on what you would see, but the view sessioninfo (without the x) shows all sessions running on the system.

1Q 2003 FAQs

Q1:

What is the best way to sum up three monthly fact tables (annualize) and load the results? In the example below the tables are relatively big, with about 65 million rows and 30 amt columns (decimal(14,2)) per row: Case (acct exist in month1, month2, month3)(amt_month1 + amt_month2 + amt_month3) * 4 -> amtCase (acct exist in month1, month2)(amt_month1 + amt_month2) * 6 -> amtCase (acct exist in month1)(amt_month1) * 12 -> amt

A1:

Think of this as an OLAP problem, and the solution will be much simpler and more generic:\

Step 1: Combine the three monthly fact tablesDELETE AnnualTable ALL;INSERT INTO AnnualTable ( acct, monthnumber, amt ) SELECT acct, 1,amt_month1 FROM month1;INSERT INTO AnnualTable ( acct, monthnumber, amt ) SELECT acct, 2,amt_month2 FROM month2;INSERT INTO AnnualTable ( acct, monthnumber, amt ) SELECT acct, 3,amt_month3 FROM month3

Step 2: Sum (annualize) and load tableSELECT acct,monthnumber,SUM( amt ) OVER (PARTITION BY acctORDER BY acct, monthnumberROWS UNBOUNDED PRECEDING ) AmtToDate,AmtToDate * 12 / monthnumber AS ProjectedAnnualAmtFROM AnnualTable/* Optional - this piece gives you the row with the highest monthnumber only.*/QUALIFY Rank(monthnumber) = 1

Of course, there are many variations to the OLAP approach. In particular, you can see where an AnnualTable containing rows for ALL possible accounts and ALL possible months with NULL amounts for non-existing account-months would allow you to also analyze missing data problems, perhaps projecting annual amounts using algorithms for partial or incomplete data.

Q2 What are the pros and cons of assigning surrogate keys in the data warehouse, as

: opposed to inheriting your source system natural keys? I realize that some of the pros for surrogates are that you might have some space savings, and you can hide some complexity of the source system keys, etc. But is it a good idea to assign different levels of surrogate keys in each and every instance of parent and child tables and take it multiple levels down? Should you always assign a surrogate when you have multiple sources (with different number of composite natural keys) for the same entity?

A2:

There are many tradeoffs to be considered. Let's assume the term NK stands for all of the columns of the natural key while SK stands for a single-column surrogate key. While there are other design considerations, let's assume that the NK is NOT included in any tables other than a single cross-reference table (which indicates the relationship of the SK to NK).

The pros:1. Often improves performance of a large set of queries due to less I/O (assuming the NK is excluded from all tables). More rows fit in each block.

2. Improves performance on a large set of queries due to reduced join complexity. The impact of both this issue and #1 above can be significant. The savings is proportional to the number of bytes associated with the NK relative to the size of the rows returned in the intermediate and final result sets. In my testing there has been I/O and CPU savings from 10% to 50%.

3. Improves ease of joins between tables since the join now involves only one column.

4. Reduces the probability that the user might construct a "killer query" (product join) because less predicates may be necessary to perform joins. This issue alone can overshadow all of the other issues.

5. Eases transition to historical continuity—that is to say that one can more easily associate different values of the NKs to SKs. The cross-reference table can correctly record the fact that an account number has changed simply by creating another row associating the new account number with same surrogate key as the old account number. This may be important when any part of the NK may change over time.

6. Reduces storage in many tables where only the SK is used. Another table of the cross-reference between NKs and SKs are maintained and therefore increase storage requirements. However, the overall storage requirements can often be reduced significantly.

7. Isolates the impact of change to the columns of the NK. Should any column of the NK change it can be easily performed against a single table rather than the multitudes of tables where the NK may have been used as a primary key or foreign key reference.

The cons:1. All SKs must be assigned by some process. While there are many ways to assign SKs, one can leverage the parallelism of Teradata and assign new SKs directly through SQL by using the RANK or similar function.

2. May add a layer of obscurity by now having to use the SK where the business might be accustomed to the NK.

3. May impede performance on a small set of queries where the columns of the NK may provide selectivity for a secondary index on the columns of the SK.

There are cases where it makes sense to carry the NK along with the SK in all/some of the tables.

I would not suggest that one use SKs as a substitute for each and every primary key in the model. SKs bring a lot of maintenance and therefore should generally be applied only when the benefit far outweighs the cost. I may not use a SK to replace a single or small two column NK. It really depends on what can be gained in consideration of all of the other requirements.

Contributed by Doug Drake, Winter Corporation

Q3:

I'm trying to figure out how to implement row-level security without using a ton of views. Our BI folks say it should be through Teradata. I say it should be through BI.

A3:

Row-level security within Teradata is implemented using views. Having said that, it needn't involve a "ton" of views:

For an example you could implement row-level security using one view per fact table. Say you are a retailer with multiple chains and you keep all of your data in the same tables, where chain is part of the key on all tables. Some users work specifically for one chain and are not allowed to see other chains' data. Corporate users may see data for all chains. Some users may see two of the chains' data, etc.

Create a small table mapping chain groups to chain ids. Create another table with user ids and their chain group. A view on top of both gives you the chains a user may see. The users have access via views only and all the fact table views include a join to your security view with a WHERE clause of WHERE user id=USER.

The beauty of having it implemented within Teradata means that all access—whether through a BI tool or through a "power user" tool—is subject to the same rules.

Q4:

Is there a method by which you can calculate the number of tactical (single amp) queries a Teradata system can support? I have been approached by one of our business partners to support the development of an application that would be accessed by 5,000 users twice a month. Assuming the worse case—that all 5,000 users tried to access their statement at the same time—how long would it take for all queries to be satisfied?

A4:

The NCR/Teradata performance people should be able to help you size it, but in general, it will depend on the query, its duration, and the volume of data to be returned. And, of course, it depends on what else is running at the time. Since you mentioned that it's single amp, I'm assuming that it will be primary index select. You should be easily able to handle 30 or more queries per amp per second.

Do a single-AMP RETRIEVE step frommwdw_prd_qmtbls.sxs_repeat_callers by way of the unique primaryindex "mwdw_prd_qmtbls.sxs_repeat_callers.MTN = '2194033732'" withno residual conditions. The estimated time for this step is 0.03seconds.-> The row is sent directly back to the user as the result of

statement 1. The total estimated time is 0.03 seconds.

And, if you use persistent connections, there's no logon time. So with a boatload of sessions and on your configuration, you should be able to handle all 5,000 queries within a second or two—worst case.

2Q 2003 FAQs

Q1:

V2R5 Query Log FunctionalityIf you have a lot of ad hoc users writing their own SQL, then query logging is a necessity. How can Query Log help?

A1:

The V2R5 Query Log should not be thought of as a single, binary, on/off kind of thing. It has a number of options for controlling the logging performed. Some of these options are going to be reasonable to have on all the time. Others are going to be more costly in resources and space and will probably only be used for investigating a part of the workload or a new application in detail.

For ad hoc, strategic queries, expect query-level logging to be on all the time. However, logging every step executed for every one of those queries will probably be more expensive than is desirable. But when there is a performance issue or when a new application is being deployed, the ability to investigate at the step level will be extremely valuable.

Likewise for objects—when object logging is on, each object touched will result in a record in the log. This will be a LOT of logging for a high volume of queries. And of course if you don't wish to pay the price of that logging for every query, it will be difficult to use it to identify unused objects.

Note that if the primary reason for logging objects is to understand what is used, then a daily job to aggregate that log to capture one record per object rather than one per object per query will radically reduce the long-term storage for the information. In some cases, it will probably not be appropriate to turn on logging at all for certain user IDs. Most obvious perhaps is TPump where query-level logging will result in a record in the log for every pack sent by TPump—probably much more logging than desired relative to the value of the information.

Q2:

Unnecessary Spool StepIn the following explain plan, spool 2 is created in step 1, which is then all-rows-scan read (last use) and put in spool 1 in step 3.1. So, what is the purpose of step 3.1? Spool 1 and Spool 2 are the same, and step 2 could have directly created Spool 1.

1) First, we lock xxx_DB.yyy_TBL for access.

2) Next, we do a SUM step to aggregate fromxxx_DB.yyy_TBL by way of an all-rows scan with acondition of ("(xxx_DB.yyy_TBL.date_col >= DATE'2002-12-02') AND(xxx_DB.yyy_TBL.date_col <= DATE

'2002-12-04')"), and the grouping identifier in field 1030.Aggregate Intermediate Results are computed globally, then placedin Spool 2. The input table will not be cached in memory, but itis eligible for synchronized scanning. The size of Spool 2 isestimated with low confidence to be 101 rows.

3) We execute the following steps in parallel.

1) We do an all-AMPs RETRIEVE step from Spool 2 (Last Use) byway of an all-rows scan into Spool 1, which is built locallyon the AMPs. The size of Spool 1 is estimated with lowconfidence to be 101 rows. The estimated time for this stepis 0.17 seconds.2) We do a SUM step to aggregate fromxxx_DB.yyy_TBL by way of an all-rows scanwith a condition of ("(xxx_DB.yyy_TBL.date_col >=DATE '2002-12-02') AND(xxx_DB.yyy_TBL.date_col <=DATE '2002-12-04')"), and the grouping identifier in field 1.Aggregate Intermediate Results are computed globally, thenplaced in Spool 4. The input table will not be cached inmemory, but it is eligible for synchronized scanning. Thesize of Spool 4 is estimated with high confidence to be793,963 rows.

4) We do an all-AMPs RETRIEVE step from Spool 4 (Last Use) by way ofan all-rows scan with a condition of ("Field_4 > 0") into Spool 1,which is built locally on the AMPs. Then we do a SORT to orderSpool 1 by the sort key in spool field1. The size of Spool 1 isestimated with high confidence to be 264,755 to 793,963 rows. Theestimated time for this step is 0.66 to 1.56 seconds.

5) Finally, we send out an END TRANSACTION step to all AMPs involvedin processing the request.-> The contents of Spool 1 are sent back to the user as the result of statement 1.

A2:

In most complex queries, you will see a last step, which appears to simply re-spool the result spool file. It is not really re-spooling it; rather, it is reformatting the data to prepare it for return to the requesting application. This is required because all of the internal processing steps (joins, scans, aggregates, etc.) work on data in the Teradata internal formats, while your application wants the data in the client data-type formats (appropriate character sets, formatting applied, right floating point representation, big/little endian, and so on).

For aggregate results, the grouping key also needs to be stripped out. In this particular plan it is a little more confusing since the result comes from a UNION operation. Step 3.1 formats the result of the first SELECT (Spool 2) into client format.

Step 4 does the same for the result of the second SELECT (spool 4) after applying the HAVING clause and the ORDER BY. Note that step 3.1 and step 4 both write their results into Spool 1, which is the final result spool all formatted and ready to be returned to the client application.

Q3:

DISTINCTPlease explain how the DISTINCT command works in Teradata SQL queries.

A3:

Including the DISTINCT clause in a SQL query results in elimination of duplicate rows. DISTINCT applies to the entire row and not to any single column.

Ex. SELECT DISTINCT a, b, c FROM x;

Here DISTINCT does not mean just "a" being unique, but the combination of a, b & c

You can produce distinct result sets in two ways:

SELECT DISTINCT x FROM TABLE;-or-SELECT x FROM TABLE GROUP BY x;

Teradata uses two different methods to produce the result sets, however, and the performance varies dramatically depending on the data.

SELECT DISTINCT is designed for data sets that are nearly unique to begin with and works by sorting the entire intermediate spool file and discarding duplicate values. SELECT/GROUP BY is designed for data sets that have relatively few unique values and works by performing an AMP local grouping operation and then merging the partial result sets for final processing.

Q4:

Teradata Priority SchedulerWe are looking to use the Teradata Priority Scheduler to help us out with resource management.

A4:

The Teradata Best Practices Document recommends moving all non-system session and work to a separate resource partition. This is phase 1 of our plan. I have a couple of related questions:

1) If I just move all non-system sessions to RP1 and go with all defaults (RP1 = none, default, M1, AG 41) are there recommended weightings? Our plan was 100 (RP0), 75 (RP1).

2) With such a high weighting on each (RP0 and RP1), can you migrate clients and batch sessions in phases based on the fact that only "active" sessions are considered and that each of the RPs has a high weighting? Or should it be a "big bang"-type of move?

3) Because we are planning to create the remaining 4 RPs and only use the first two (initially), I created my "user partition" (RP1) (development) performance groups, half expecting that the system would create its own. I created 8(L1), 10(M1), 12(H1) and 14(R1), and when I displayed it after the fact, I was expecting to see 11(low1), 13(med1), 15(High1) and 17(Rush1). I did not see this.- Is it safe to assume that I have to create the system performance groups myself or are there system performance groups required beyond the default RP0 partition?A4: Regarding the RP weightings, keep in mind that the entire Teradata Priority Scheduler algorithm is relatively based. That means the RP weights are

"high" only relative to each other. The relative difference between the weightings is what's important. Therefore, in the instance where you have two partitions, the effect would be the same whether you had 100 and 75 or 1000 and 750. There would no difference in the way the system behaves.

As far as moving things out of the default partition (RP0), that's often a good thing to do but it is not always necessary. Typically, you should keep RP0 for system work and, optionally, for very predictable, high-priority work such as an application that does OLTP processing or canned reporting. With RP0 containing only system work, you could comfortably make the RP0: RP1 weighting ratio 2:1 (i.e. 100 vs. 50) or even higher. Just remember that the more extreme the weighting differential, the more important it is to ensure that ONLY system work runs in default. Otherwise, a very poorly written, highly weighted query that inadvertently gets in RP0 could have a dramatic impact on system performance. Also, if you haven't switched the DBS Control setting to have rollback occur at the same priority as the executing transaction, this could also impact performance in a way you might not have intended.

Migrating in phases is certainly doable. Just appreciate the fact that, all things being equal, as you migrate work from RP0 to RP1 the performance characteristics of the work that gets to RP1 first will probably degrade. (That's somewhat of a guess, since I don't know specifically what your workloads look like.)

An alternative approach might be to set the weight of both partitions to be the same (i.e. RP0 100, RP1 100), migrate out the workloads to RP1, and then change the RP weights accordingly. As the work migrates, the system should behave as if everything was in the same partition because, essentially, everything will be prioritized the same. And then you could more heavily weight the system work if that's your desired end state.

In response to your last question, you can only run the Teradata Priority Scheduler, schmon, from a node that is running Teradata. Schmon requires services from the PDE layer, which is not available on the application node. Although you can run schmon from any Teradata node, you will require root privileges to issue commands.

You may have already seen this information, but there's a small repository on SupportLink that contains technical papers (usability tips and techniques) around things like priority scheduler. If not and you're interested, here's how to get to it.

If you are on page 1 of SupportLink, scroll down and there should be a link under the Updates column that takes you to the active data warehouse repository. The icon you are looking for says "Teradata ADW Documents." Click on that icon, and you will see a list of papers that you can download. There are several Teradata Priority Scheduler how-to papers with things like sample code, templates, discussion of trade-offs, dos and don'ts, etc. They might make your next steps easier.

3Q 2003 FAQs

Q1:

What is NCR's 64-bit MPP Server Update?

A1 As you know, Teradata's roadmap is dependent on Intel's roadmap for server-class nodes.

: We planned to announce 64-bit MPP servers as early as the end of this year. However, the current 64-bit server nodes from Intel haven't achieved performance parity with the 32-bit nodes. Based on market predictions from industry analysts, including Gartner and IDC, we expect the performance difference to continue through 2006. The Teradata Database software is ready and is currently available on non-NCR Intel 64-bit SMP systems. Teradata Database is positioned to deliver on 64-bit MPP servers when they reach comparable performance levels to the 32-bit Intel servers. When that happens, Teradata will provide 32-bit customers with investment protection options for their current systems.

Q2:

What is difference between perm space and spool space?

A2:

The difference between perm space and spool space is that perm space is allocated to the database and will not share with other databases, while spool space is a reserved space that is shared by different users and, within the users' queries, is used to format the result sets.

Q3:

I am using the Teradata JDBC type 4 driver. What port number does the driver use to talk to the database?

A3:

With the type 4 driver it is not necessary to specify a port in the connection string since a gateway is not used. The type 4 driver communicates directly with Teradata, which uses port 1025.

Example: "server" is the name of the system where Teradata is runningString url4 = "jdbc:teradata://server";con=DriverManager.getConnection (url4,"uid","password");

Q4:

We have a DDL for our database layout. Currently, we have a column definition as follows:

activity_dt DATE NOT NULL FORMAT 'mm/dd/yyyy' DEFAULT -18989899,The programmer ensured us that the default date would be 01/01/0001. But it is not; the UNIX timestamp above gives a date in 1969.

What is the syntax to define 01/01/0001 as the default date?

A4:

Your programmer is right. The date field in Teradata is not a UNIX timestamp, but a different scheme altogether. The default shown above is correct, but for the best solution, use an ANSI date literal, as shown below.

activity_dt DATE NOT NULL FORMAT 'mm/dd/yyyy' DEFAULT DATE '0001-01-01'

Q5:

We are just starting to work on Teradata and we have a question about building views. Do you know of a tool that could help us build dimensional views over third normal form models? We want these to be linked to the physical but still be independent, to inherit definitions but be modifiable, and to be documented (i.e. catalogued).

A5:

If I read this correctly, you're saying that you have a normalized model and want to generate dimensional views to sit on top of it. You can do that by following the guidelines on how to

develop views in the documentation.

Q6:

Currently, there is no way to cancel a rollback. The rollback will survive a TPARESET. However, the GSC can stop it with a utility (I believe a chargeable event). They will perform a TPARESET to cap the new entries to the TJ, then stop the journaling by using the utility. You will lose all data in the transient journal, not just your particular rollback. All tables involved in the transient journaling will be in an invalid state and will either need to be dropped/recreated or restored. Is there a fix for this?

A6:

There will be a new command in V2R5.1 Recovery Manager that will take care of this. It is: CANCEL ROLLBACK ON TABLE.

Q7:

Suppose that incoming data targeted for an enterprise data warehouse is in various "character sets," i.e. kanji, Latin, etc. We would like to store the data in a consistent character set, but when the info is presented back to the client, we want to place it in the appropriate character set for that client.Is there a "super type" character set that would allow for conversion to/from any character set? UNICODE seems close but still has exceptions. Is there a method to dynamically '.set session charset' if we were to present data from a method other than BTEQ?

A7:

UNICODE is the "super type" server character set that Teradata offers. If it doesn't meet your specific need, then you would need to submit a request for change. We do offer some capability to customize conversions from client character sets to internal character sets, and this may address the exceptions that you mention. However, without specifics on the exceptions, it is unclear if there are solutions to this issue using current capabilities.

KANJISJIS might be another possibility to use as the server character set, but I suspect that would be less suitable than UNICODE. You can define the character column to be UNICODE or LATIN (you can also choose GRAPHIC, KANJI1 and KANJISJIS, but we encourage users to choose either UNICODE or LATIN).

e.g., create table test (col1 char(5) character set latin, col2 char(5) character set unicode);

Incoming data will be translated internally to the form specified in the column definition. Outgoing data will be translated to the character-encoding scheme specified by the user (session character set). Users can change their character set without logging off, i.e. the session character set can be changed during a session.

I think most Teradata interfaces, including BTEQ and CLIv2, support a way to set the session's client character set. However, this must be done before logon (of course, one could log off, change setting and log back on when a change is necessary). I am not an ODBC/JDBC expert, but I assume they would have this capability also. The client character set cannot be changed during a session. The default server character set can be changed using the MODIFY USER statement; it can also be overridden for a column by explicitly specifying the CHARACTER SET after the character type.

1Q 2004 FAQs

Q1 How do you calculate the average of timestamp values in Teradata? For example,

: Create table avgts(i int, ts timestamp(0));ins avgts(1, '2003-09-01 12:00:00');ins avgts(1, '2003-09-03 12:00:00');select i, avg(ts) from avgts group by 1;gives me the error:5407: Invalid operation on ANSI Datetime or Interval value

A1:

You can do conversions like this:

sel runname,qryname,sum(timeinsecs)/count(=) as avgtime, avgtime/3600 as timehours,(avgtime-(3600=timehours))/60 as timeminutes,avgtime-((3600=timehours)+60=timeminutes) as timeseconds,trim(timehours)||':'||trim(timeminutes)||':'||trim(timeseconds) as averagetimefrom resultsgroup by 1,2order by 1,2

Or for a much simpler expression for what you are doing, check into ANSIINTERVAL!select avg((cast(timestampfield as time) - time '00:00:00.000000') hour to second)from table

Q2:

Will the optimizer materialize a multi-table view before any additional table joins or constraints are made? Here is an example:

I have an aggregated view that is a join of tables A, B, C and D. When users query this view, they sometimes want to join the view to table D to further constrain on column Z.

Will the optimizer join tables A, B, C and D first without the extra table D constraint on column Z? Or will the optimizer recognize the additional column Z constraint before materializing the view?

Would the plan be different on Teradata Database V2R5 compared to Teradata Database V2R4?

A2:

The case is a bit abstract here, but it makes me wonder if the intent of the query will actually be realized.

You state that the view specifies an aggregation over the result of a join between tables A, B, C and D, but you do not state whether Z is in the group list and made visible via the view. I am guessing that it isn't; if it were, then it would be easy to apply the condition to Z without an additional join to D.

If Z is not visible in the view, then it is highly likely that the additional join to D constrained on Z will not create the desired result. The aggregation will be performed first as specified, then the additional join to the constrained version of D will be performed—it has to be executed this way to get the correct SQL result for the query. This is likely to be a very different result than if the constraint was applied to Z on D prior to the aggregation because rows will be eliminated prior to consideration for grouping.

Q3 With the arrival of Teradata Database V2R5 comes the long-awaited multi-value

: compression facility. Can you discuss its features and its pros and cons?

A3:

The use of multi-value compression within Teradata Database V2R5 can provide some big savings, both in disk space and in the general performance of queries.

As an example, one customer had a table in Teradata Database V2R4.1, which used single-value compression, and they saved 30% in disk space. Since an upgrade to Teradata Database V2R5, the customer has managed to save an additional 25% in disk space and has noticed a general performance improvement of around 40%, which the customer thinks had a lot to do with the fragmentation of the table as well. The use of compression and the additional CPU overhead was negligible, estimated around 0.5%.

To date, customers who have implemented multi-value compression under Teradata Database V2R5 are seeing favorable results all around. However, be aware that there is a tradeoff: Some middle-sized tables actually grew in size with multi-value compression due to overheads of table headers and where users had applied to columns that had previously been declared as VARCHARs.

All in all, however, users have noticed a "general performance improvement."

Q4:

We would like to be able to store the results of any and all queries (the actual answer sets) for subsequent auditing. Our fear is that just having the query is not enough since rerunning the query might not produce the same answer set the user originally got.

It seems to me it would have to be part of the database query log, with an option provided to store, say, up to 10K of the final spool before it's sent to the user. But that option is currently not available.

Any thoughts as to how we could store the results of any and all queries (or some portion thereof) for some period of time?

A4:

Run every query as a CREATE TABLE AS WITH DATA with ROWNUMBER as the first column in the select. Then select from that table to return the result to the user. This process requires a driver program to grab the SQL and rewrite it in order to do the create with an appropriate table name that can be tracked later (date, user, etc.). The driver should probably log on with a driver ID so the user can't delete/drop the table.

It would not be practical to put in a database query log since every result has a different table layout. WARNING: The log will be vastly larger than the data, so be sure to have enough space available to support this query.

Q5:

In using the Teradata Priority Scheduler Facility, the milestone type parameter is set at the performance group, not at the allocation group. Therefore, assuming no manual intervention through the PM/API, the query will be assigned to the allocation group associated with the performance group at logon and then be handled according to the milestone type rules defined.

One possible exception to this case might be a situation where you implement a process that switches your PSF schemes, either based on some event or at regular time intervals. If, in this scenario, you were to "redefine" your performance group such that you change the milestone type from "time" to "query," I'm not sure what would

happen to any "in-flight" queries. I'm not recommending that, but I'm guessing that it's possible.

Does PSF continually reevaluate the definition of performance groups and/or allocation groups when computing the relative weighting?

A5:

You can change a performance group definition at any time, with active work running under its control. This will cause a change to the Teradata Priority Scheduler definition, which will cause the system.GDO record to be rewritten. Any time the system.GDO record is changed, all CPU accumulations (whether at the session or the query level) across all performance groups are zeroed out. The system.GDO record exists on each node and is the operating system's copy of the current priority settings.

This means that if you issue any schmon command, it will change a Teradata Priority Scheduler component. Say an allocation group weight in another resource partition, or changes a system level limit, that change will cause all queries under the control of any milestone limit in any performance group to start over in their accumulations of CPU.

Most definitely, relative weights will be recalculated when the system.GDO record changes, as is continuously happening under normal circumstances.

2Q 2004 FAQs

Q1:

I am loading data into Teradata via TPump. There seems to be an issue inserting a row into one table from two sources at the same time. When I look at the performance monitor, I can see that the data is being blocked. Logically, I find it difficult to believe that multiple sources could be loading the same table simultaneously without locking issues. I guess it could work, depending on blocks, cylinders, etc., but it sounds like at some point (statistically) you'd run into a blocking scenario. Are there any best practices around using TPump and multiple sources loading the same table?

A1:

There are two parameters to control locking in TPump: PACK and SESSIONS. With a large PACK value, TPump will run faster, but blocking will happen more frequently. SESSIONS just indicates the number of users concurrently inserting data via TPump.

Start by lowering PACK and SESSIONS on the TPump jobs you are running. TPump has a SERIALIZE function that avoids cross-session collisions by hashing identified key values to particular sessions; this also preserves order. Ultimately, you can run with PACK=1, SESSIONS=1, SERIALIZE ON. This way you may end up with no blocking, but TPump may run slow.

Q2:

I need help regarding Teradata's file writing utilities. In my application, I have a procedure that retrieves data from more than one table and stores data in variables after manipulation. I need to write the data to a file, which in turn can be loaded into a target table using MultiLoad. (This is because direct writing to the table takes a lot of time for large data

volumes.) Which Teradata file writing utility can be used in this case?

A2:

In Teradata we always try to get things done using set logic. You do not state what kind of data manipulation you need to do. If there is any way to do it in SQL statements, then the best solution for this type of application is to use INSERT SELECT without the data ever having to leave the database.

If it is absolutely necessary to retrieve and process a large amount of data outside Teradata, then it sounds like a job for the FastExport utility. FastExport accepts any SQL statement and retrieves the result using an interface optimized for high-volume export. It also has an interface for OUTMOD or Access Module that allows you to process the data on the way out of FastExport and lets you to write it anywhere you desire.

Q3:

Test (1) Environment: database: dev1Tables: tab1, tab2

Test (2) Environment:Database: dev2Tables: tab1, tab2

Both databases run on the same Teradata system and have identical table structures, names, access rights, etc.

I have a query that looks like this:

select col1, col2, col3...from dev1.tab1, tab2

where inner-join conditions ...

When I try to run this query, one of several things happens: 1. When the default database is 'dev2,' I get a Table/View/Macro

name 'tab2' ambiguous error.2. When the default database is 'dev1,' the query runs fine.3. When the default database is either 'dev2' or 'dev1' without the

'dev1' qualifier on 'tab1,' the query runs fine.4. When the default database is either 'dev1' or 'dev2' with both the

tables properly qualified as dev1.tab1 and dev2.tab2...5.

Is this a bug or am I missing a link?

A3:

Think of the default database as one entry in a search list of databases, not as a text substitution for a missing database specifier. All other databases named explicitly in the request are also added to the search list. Then each object is looked up in the databases in the search list. An object that appears in multiple databases in the search list is ambiguous.

Q4:

I've looked as some test results showing that single-AMP (PK access) queries are not impacted by VLC to any notable degree. If compression is defined but not used (because the particular rows' columns were not compressible), then there was less than 1% overhead detected (-0.14%

throughput change) when performing five sessions doing PK selects. This is equivalent to nothing, and it is a heavy load.

Another test with 100% compression (all the columns had to be uncompressed) showed single-AMP selects under the same conditions also having less than 1% overhead (-0.53% throughput change). That is also too small a factor to care about.

With short all-AMP tactical queries you may be able to benefit from VLC to some degree, or at least not experience degradation. In addition, because VLC has the potential of reducing overall I/O demand on the platform, it can provide a secondhand benefit to short tactical queries because it can reduce contention for I/O. Tactical queries always do better in environments that are not maxed out on a particular resource. VLC can be helpful in restoring a healthier balance between CPU and I/O demand, if the platform tends to be I/O-bound.

What do you think?

A4:

Actually, Teradata VLC compression has shown in testing as well as customer sites to offer a benefit in the load utilities. There are two reasons for this:

1.) When rows are compressed prior to loading, more of the rows can be placed into a single data block. Therefore, the load utility is doing less physical I/O to store more rows.2.) With PPI, if you have partitioned on day, for example, and are loading rows that all go into the same partition, often those data blocks are so few and so concentrated that they stay in the FSG cache. While this benefit will depend on many factors (your mileage will vary), and you might not see this at all, let me give you an illustration. During one test we loaded a PPI table via TPump where the rows targeted one partition, and we experienced almost zero I/O load. That significantly reduces time to load, as you can imagine. ARCMAIN will benefit as well, as there will be fewer blocks to back up.

In terms of the test results, I've seen 20-25% improvement using FastLoad. But what you will experience will depend very much on how many columns and rows can be compressed, and how large they were to start with. However, you should always see some improvement.

3Q 2004 FAQs

Q1:

I'm trying to subtract one year or 12 months from the leap year 2004-02-29 and I get the following error message: <2665 Invalid date>.

SELECT CAST('2004-02-29' AS DATE) - INTERVAL '12' MONTHSELECT CAST('2004-02-29' AS DATE) - INTERVAL '1' YEAR

A1:

ANSI standard says MONTH/YEAR interval arithmetic should not adjust day of month.

Teradata implemented the ADD_MONTHS() extension specifically to address this problem:

SELECT ADD_MONTHS(date '2004-02-29', -12)

Results: 2003-02-28

Q2:

I am planning on using the schmon -i or the schmon -b command in Priority Scheduler. Do you know if I need to adjust the CPU percentages on a regular basis? If so, how often should this be done?

A2:

Here are some explanations regarding just what these Priority Scheduler commands do:

The schmon -i (lowercase i) enables the prioritized queuing of FSG I/O requests. It should be turned on by default. It is not recommended that you turn it off, but it may not make much difference to do so if you are not experiencing I/O waits of any magnitude on your platform. What it does is enforce the prioritization of requests that are waiting for an I/O, which is usually a good thing. I am not aware of any negative trade-offs with having I/O Prioritization turned on.

The schmon -b command creates or modifies resource partitions. It is not uncommon to issue several schmon -b commands to alter the relative priorities of one or more already-defined resource partitions. Often a small batch of schmon -b commands is packaged into a cron job and scheduled regularly as a way to change priorities at different times of day. The most common change pattern I have seen is twice a day, once to change settings just prior to the batch cycle at night and then just before the day cycle in the morning. The resulting impact on CPU usage will depend on how much the assigned weights of the affected resource partitions are changed and the nature of the work running within them and the other active resource partitions.

If you are using PSA (Priority Scheduler Administrator), PSA can automate the changes in settings at specific times of day, so the DBA no longer needs to submit cron jobs or schmon commands to establish that level of control.

Q3:

When does an AMP give up the allocated worker tasks (AWTs)? Will it wait until the complete step has been finished on all AMPs before it releases all AWTs, or will it release the AWTs AMP by AMP? What if the step is highly skewed?

A3:

Teradata will release a given AWT at the time that particular AMP completes that query step. Each AMP will make a notification that it has completed that query step and then free up its AWTs for other work.

Only when the last AMP is done with that step does the system notify the dispatcher that all AMPs are done so the next step can be sent. If the step is highly skewed, some AMPs will be holding AWTs on behalf of that step, while other AMPs will have completed and released their AWTs to do other work.

Q4:

I am executing a query with a value in the where clause for a NUPI (Non-Unique Primary Index) on a table with a PPI (Partitioned Primary Index). The query is not using the NUPI.

My question is this: When you have a PPI and a NUPI (and the PPI is not part of the NUPI) and you supply values for the NUPI but not for the PPI, should the optimizer use the NUPI or go around it? Also, is it necessary or just recommended to incorporate the PPI column into the NUPI?

A4:

The answer depends on the size of the table and number of partitions.

When only the NUPI value is specified on a PPI table, every partition has to be accessed in order to find the NUPI value. If there are lots of partitions and it would be faster to scan the table, then that is what the optimizer will choose. This is the trade-off of partitioning a table that also needs to be accessed directly by a single PI value. If the PPI is valuable enough, then it is probably appropriate to put a NUSI on the direct selection columns to avoid all the probes to the partitions.

Should you put the partitioning column in the PI (Primary Index)? Usually not. In this case, if it is possible to specify the partitioning column value and the NUPI value in the where clause, then do that so that you have good PI-type access. If the partition value is not available, then having the partitioning column in the PI makes the PI useless even more of the time.

Q5:

We need to use big comments columns because we include management rules in the comments. However, the comment columns in Teradata are only 250 characters long. Is there anyway to capture the complete comment?

A5:

Teradata offers Metadata Services (MDS), which provides the facility for both capturing the business metadata and for disseminating the business and technical metadata to users. It has the flexibility to contain the large amount of text that is desirable for the business metadata descriptions. It also helps with relationships between objects.

1Q 2005 FAQs

Q1:

During spool redistributions to different AMPs (merge joins where primary indices are different), some records come to the same AMP after rehashing on the basis of join columns. Do they really go to the BYNET or message-passing layer for rehashing, or are these records directly spooled locally?

A1:

When rows are redistributed during join processing, there is a same-node optimization that is done by the message system. The redistribution message, when it is sent, uses VPROC and node addressing rather than hash addressing (it could be a different node, different AMP, same node, different AMP, or same node, same AMP). The optimization consists of recognizing when the destination is on the same node as the sender and routing that message to the correct AMP, without leaving the node. Only rows being redistributed off the node from which they were sent will be handed off to the BYNET. A row being sent out and returned to the same AMP leaves the AMP within a sender task and returns to the same AMP within a receiver task.

Q2:

Does Teradata support API/CLI for pre-defined, vendor supplied, third-party supplied and user defined complex data types on the database? To clarify, I am looking to understand if things such as composite field types, create sub-tables, arrays of like types or free form arrays are supported.

A2:

The upcoming Teradata Warehouse 8.1 (Teradata Database V2R6.1) will support user defined types (UDTs), including complex data/structured types. It does not provide direct array support and array operations yet. Teradata Database V2R5.1 included BLOBs and CLOBs for storing large unstructured data and user defined functions (UDFs) for processing that data. While the typing is not explicit, any array or structured type can be stored using these types and can be processed in the UDFs. The data structure must be represented in C programming language within the UDFs rather than the database knowing the type structure. This ability continues and is extended by UDT implementation in Teradata Database V2R6.1.

Q3:

When insert selecting, I assume the system is doing something to the data on the way, such as creating a spool file. Further, because maybe only one field is being changed, I

assume that the spool file is uncompressed then recompressed on the other end. I am insert selecting a 600GB table to a new data structure with new columns, a couple of transformations and both partitioned primary index (PPI) and multi-value compression. Am I right to assume that I will need 2 x 600GB plus the size of the uncompressed data in spool (40%-50% of 600GB)? If so, I will need 1.5TB of space and the spool file will need complete cylinders. The system currently is 67% full CurrentPerm/MaxPerm.

A3:

You are correct that the spool is not compressed. How much space it requires depends on the plan. Does the plan need to spool it only once or does it have to do joins, for example, to get the additional columns and transforms? All of this should be taken into account. You will need to target the necessary source space as well as the space required by the uncompressed copy in spool.

Q4:

When performing INSERT SELECT within target tables and concurrently generating reports with access locking, we have found that large amounts of data disappear from the target tables. We know that other factors may influence this observation, such as a faulty description. We are running Teradata Database V2R4.1 and moving to Teradata Database V2R5.1.1 and want to know if this has been resolved in the new release.

A4:

This should not be an issue with INSERT SELECT. Based on your description, there is only one case where this is an issue. That is when UPDATE WHERE is being used with a join. This can result in an access lock reader seeing a lot of missing rows. You can always review the explain plan to see if it shows a delete step followed by a merge insert. If the select sees the table during the time that the delete has operated, then those rows will be missing until the merge step completes. In Teradata Database V2R5, merge update, in most cases, has been changed to perform a direct merge into the table; again this can be seen in the explain plan. Prior to Teradata Database V2R5, any Update that uses the delete/merge plan should be executed with a LOCKING EXCLUSIVE clause if access lock selects might possibly be performed concurrently.

Here are some additional questions answered by Carrie Ballinger, a Teradata Certified Master and a senior technical consultant in Teradata's Active Data Warehouse Center of Expertise. If you would like to get some additional tips and insight from Carrie, be sure to check out her column titled "Learning from each other."

Q5:

Can partitioned primary indexes (PPIs) replace value-ordered indexes (VOIs)?

A5:

In many cases, yes. However, it depends on the application. A VOI is a special type of index whose sub-table rows are stored in order by index value. With VOIs you have flexibility over the range of access. The same VOI structure will provide efficient access for ranges of two days, two weeks or two months. With PPIs, your partition boundaries are fixed. If you partition by month, then partition elimination cannot be less than one month, even if you only require data from one day. If your access range size varies, consider finer partitions in your PPI table—a day or a week instead. Often, this is preferable to supporting a VOI on the same column as the partitioning column.

Q6:

Are AMP worker tasks (AWTs) held by queries when they are returning answer sets? During a quiet time on my system, I noticed 10 sessions, all in RESPONSE mode; however, when I issued a puma -c command, there were no AWTs in use. What is happening here?

A6:

AWTs working on behalf of a query are released at the end of the final query step before the BYNET merge process begins. The last AMP to complete its work in building a query's

final spool collects row counts from all AMPs, responds to the dispatcher that the step is complete and gives the final row count. The dispatcher then tells the BYNET to start a merge process.

The BYNET driver has special data buffers where rows that will feed into the merge are to be placed. The BYNET driver also has a special place in memory to track details about a query's response processing, such as its merge-ID; the spool-ID where the rows to be returned are located; and row-ID, which tells how far that spool has been read. AWTs are acquired as needed and only for the brief period of time it takes to move data from the final spool file into the BYNET buffers.

After all rows in the first BYNET buffer have been returned to the client, and if there is more data in the spool, a new AWT will get involved briefly to move data from the spool into the BYNET buffer. Once the merge is complete, the BYNET driver tells the dispatcher it is finished, completing the response cycle.

So, except for very brief periods of time when AMP activity is required to fill the BYBNET buffers, there are no AWTs held during response processing.