Teradata Connectors User Guide

Important Notice

(c) 2010-2013 Cloudera, Inc. All rights reserved.

Cloudera, the Cloudera logo, Cloudera Impala, and any other product or service names orslogans contained in this document are trademarks of Cloudera and its suppliers or licensors,and may not be copied, imitated or used, in whole or in part, without the prior writtenpermission of Cloudera or the applicable trademark holder.

Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation.All other trademarks, registered trademarks, product names and company names or logosmentioned in this document are the property of their respective owners. Reference to anyproducts, services, processes or other information, by trade name, trademark, manufacturer,supplier or otherwise does not constitute or imply endorsement, sponsorship orrecommendation thereof by us.

Complying with all applicable copyright laws is the responsibility of the user. Without limitingthe rights under copyright, no part of this document may be reproduced, stored in orintroduced into a retrieval system, or transmitted in any form or by any means (electronic,mechanical, photocopying, recording, or otherwise), or for any purpose, without the expresswritten permission of Cloudera.

Cloudera may have patents, patent applications, trademarks, copyrights, or other intellectualproperty rights covering subject matter in this document. Except as expressly provided inany written license agreement from Cloudera, the furnishing of this document does notgive you any license to these patents, trademarks copyrights, or other intellectual property.

The information in this document is subject to change without notice. Cloudera shall notbe liable for any damages resulting from technical errors or omissions which may be presentin this document, or from use of this document.

Cloudera, Inc.1001 Page Mill Road, Building 2Palo Alto, CA [email protected]: 1-888-789-1488Intl: 1-650-362-0488www.cloudera.com

Release Information

Version: 1.x

Date: August 29, 2013

Table of Contents

Introduction.......................................................................................................................................5Prerequisites.....................................................................................................................................7Choosing a Teradata Connector......................................................................................................9Installing a Teradata Connector....................................................................................................11Using Teradata Connectors...........................................................................................................13

Using the Cloudera Connector Powered by Teradata............................................................................13Using the Cloudera Connector for Teradata...........................................................................................15

Uninstalling Teradata Connectors................................................................................................19Limitations......................................................................................................................................21Getting Support...............................................................................................................................23

Introduction

Cloudera offers two connectors that provide high speed data transfer between Teradata and CDH.

• Cloudera Connector Powered by Teradata A new connector that uses the Teradata Hadoop SDK to achievethe best performance. The current version is 1.0.0.

• Cloudera Connector for Teradata An older connector that utilizes Teradata's FastExport and FastLoad JDBCcapabilities for data transfers. The current versions are 1.0.5 and 1.1.1.

After installation, these connectors allow various Sqoop tools such as sqoop-import and sqoop-export, tooperate in highly efficient direct modes, and exposes some options that are specific to Teradata.

This document describes how to install, configure, and use these connectors within a Sqoop installation andprovides reference information for the operation of the connector where necessary. This document is intendedfor:

• System and application programmers• System administrators• Database administrators• Data analysts• Data engineers

Teradata Connectors User Guide | 5

Introduction

Prerequisites

In order to use any of the Teradata connectors, you must have a functioning CDH installation, including Sqoopcomponents. Depending on the way Sqoop is installed, you may need administrative privileges in order to createor modify configuration files.

The Teradata connectors use catalog tables and views to look up metadata information. Therefore the user whois making the connection must have the SELECT privilege on the DBC database. Check with your Teradataadministrators or operators if you are not sure whether you have the required privileges. You must have SELECTprivileges on at least one of the following DBC database object types:

• DBC.columns• DBC.databases• DBC.tables• DBC.indices

The Cloudera Connector Powered by Teradata requires the following additional permissions to enable use ofthe *.fastload data methods.

• DBC.table_levelconstraints• DBC.triggers• DBC.tvm• DBC.dbase• DBC.referencingtbls

For more information about how to install, configure, and use Sqoop, see the Sqoop documentation in the CDH4 Installation Guide.

Cloudera Connector powered by Teradata 1.0.0

• Teradata 13.00 or later• CDH4.2 or later• Sqoop 1.4 or later but not compatible with Sqoop2

This version has been tested against Teradata Express 13.10 with Sqoop 1.4 installed on a supported Linux OSplatform.

Cloudera Connector for Teradata 1.1.1 Requirements

• Teradata 12.00 or later• CDH4 or later• Sqoop 1.4 or later but not compatible with Sqoop2


Cloudera Connector for Teradata 1.0.5 Requirements

• Teradata 12.00 or later• CDH3u1 or later• Sqoop 1.3 or later but not compatible with Sqoop2



Prerequisites

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

Choosing a Teradata Connector

Cloudera connectors allow you to exchange information between Teradata appliances and Hadoop ecosystemcomponents such as Hive, HBase, HDFS, or HCatalog. Cloudera provides three Sqoop based connectors that youcan use for exchanging data between CDH and Teradata:

• Cloudera Connector Powered by Teradata 1.0.0• Cloudera Connector for Teradata 1.1.1• Cloudera Connector for Teradata 1.0.5

Important: Only use one of these connectors. If you install multiple connectors, when jobs are run,a connector is chosen arbitrarily, producing unexpected results.

When considering which connector to use:

• Use the Cloudera Connector Powered by Teradata, if possible. Both connectors provide fast performance, butthe Teradata powered Cloudera Connector is typically faster.

• Use the Cloudera Connector for Teradata if you cannot use the Cloudera Connector Powered by Teradata.For example, use the Cloudera Connector for Teradata if you are using CDH 4.1 or a Teradata 12 database.

The features supported by these different connectors is as follows:

Table 1: Teradata Connector Feature Support

Cloudera ConnectorPowered by Teradata

Cloudera Connectorfor Teradata

Feature

X (version 1.0.5 only)CDH3u1

XCDH4.0-4.1

XXCDH4.2 and later

XTeradata 11.00 databases and data appliances and later

XXTeradata 13.00 databases and data appliances and later

XSecondary Indexes

XQueryBand


Choosing a Teradata Connector

Installing a Teradata Connector

1. Install the Sqoop connector by opening the distribution archive in a convenient location such as /usr/lib.Opening the distribution creates a directory that contains the jar file of the compiled version of the connector.Note the path to this jar file. The directory that is created when the file is expanded varies according to whichconnector you are using. Examples of typical resulting paths include:

• Cloudera Connector for Teradata 1.0.5:/usr/lib/sqoop-td-connector-1.0.5/sqoop-td-connector-1.0.5.jar

• Cloudera Connector for Teradata 1.1.1:/usr/lib/sqoop-td-connector-1.1.1/sqoop-td-connector-1.1.1.jar

• Cloudera Connector Powered by Teradata 1.0.0:/usr/lib/sqoop-connector-teradata-1.0.0/sqoop-connector-teradata-1.0.0.jar

2. Copy the Teradata JDBC drivers (terajdbc4.jar and tdgssconfig.jar) to the lib directory of Sqoopinstallation. You can obtain these drivers from the Teradata download website:http://downloads.teradata.com/download/connectivity/jdbc-driver. Without these drivers, the connectorwill not function correctly.

3. Confirm that the managers.d directory exixsts in the Sqoop configuration directory.

Note: Depending on how Sqoop is installed, its configuration directory may be in /etc/sqoop/conf,/usr/lib/sqoop/conf, or elsewhere if Sqoop was installed using the tar-ball distribution.

If the managers.d directory does not exist, create it.4. Create a text file in the managers.d directory. Give the text file some descriptive name such as

cldra_td_connector.5. The cldra_td_connector file must have the connector class name followed by the complete path to the

directory where the connector jar is located.

For example, for the Cloudera Connector for Teradata 1.0.5:

com.cloudera.sqoop.manager.TeradataManagerFactory= \/usr/lib/sqoop-td-connector-1.0.5/sqoop-td-connector-1.0.5.jar

For example, for the Cloudera Connector for Teradata 1.1.1:

com.cloudera.sqoop.manager.TeradataManagerFactory= \/usr/lib/sqoop-td-connector-1.1.1/sqoop-td-connector-1.1.1.jar

For example, for the Cloudera Connector powered by Teradata 1.0.0

com.cloudera.connector.teradata.TeradataManagerFactory= \/usr/lib/sqoop-connector-teradata-1.0.0/sqoop-connector-teradata-1.0.0.jar

Note: The preceding command is shown on two lines, but this must be entered in a single line.

The TeradataManagerFactory acts as a single point of delegation for invoking the connector bundled withthis distribution. An alternate way to specify TeradataManagerFactory is to add the following inside asqoop-site.xml file, which must be inside a classpath directory:

<configuration><property><name>sqoop.connection.factories</name><value>com.cloudera.sqoop.manager.TeradataManagerFactory</value>



http://downloads.teradata.com/download/connectivity/jdbc-driver

</property></configuration>

This is the way to configure a Sqoop action to use the Teradata connector inside Oozie.

Note: If you are using a Cloudera Connector for Teradata, you may need to change the value of themapred.task.timeout property to be greater than the default of 10 minutes. Themapred.task.timeout property determines how long to wait before killing an unresponsive task,and mapred tasks may not respond within the default 10 minute limit while a staging table is beingused.

12 | Teradata Connectors User Guide


Using Teradata Connectors

After you have installed the Teradata connectors and copied the necessary JDBC drivers for Teradata to the libdirectory of Sqoop installation, you can use the connector. The specifics of how to use each connector varies.Follow the process that is appropriate for your installation:

• Using the Cloudera Connector Powered by Teradata• Using the Cloudera Connector for Teradata

Using the Cloudera Connector Powered by TeradataAfter you have installed the connector and copied the necessary JDBC drivers for Teradata to the lib directoryof Sqoop installation, you can use this connector by invoking the Sqoop tools with the appropriate connectionstring.

The connection string must be of the form jdbc:teradata://<td-host>/DATABASE=<td-instance> where:

• <td-host> is the host name of the machine where the Teradata server runs.• <td-instance> is the Teradata database instance name.

For example, the following command invokes the Sqoop import tool with three mappers:

$ sqoop import --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --table MY_TABLE --num-mappers 3 \--target-dir /user/sqooptest/MY_TABLE

The following command invokes the Sqoop export tool with three mappers:

$ sqoop export --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --export-dir /user/sqooptest/MY_TABLE \--table MY_TABLE_TARGET --num-mappers 3

You can control behavior of the connector using extra arguments. Extra arguments must appear on the commandat the end after all other Sqoop parameters using -- as the delimiter.

Table 2: Teradata Connector Feature Support

DescriptionToolParameter

Override default staging table name. Please notethat this parameter applies only in case thatstaging tables are used during the data transfer.

import and export--staging-table

Override default staging database name. Pleasenote that this parameter applies only in case thatstaging tables are used during the data transfer.

import and export--staging-database

Force the connector to create the staging tableif input/output method support the stagingtables.

import and export--staging-force



DescriptionToolParameter

Specifies which input method should be used totransfer data from Teradata to Hadoop.

import--input-method

Specifies which output method whould be usedto transfer data from Hadoop to Teradata.

export--output-method

Specifies number of rows that will be procesedtogether in one batch.

import and export--batch-size

Example of import using input method partition.by.hash:

$ sqoop import --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --table MY_TABLE --num-mappers 3 \--target-dir /user/sqooptest/MY_TABLE -- --input-method partition.by.hash

Input and Output Methods

Teradata-Powered Cloudera Connector currently supports three input methods that can be used to import datafrom Teradata to Hadoop:

• split.by.hash• split.by.value• split.by.partition

And three output methods that can be used to export data from Hadoop to Teradata:

• batch.insert• multiple.fastload• internal.fastload

Importing data with split.by.value method

Input method split.by.value will create input splits as a ranges on the split by column (usually table’s primarykey). Each split is subsequently processed by single mapper to transfer the data using SELECT queries. All splitsare very likely to access all AMPs for retrieving the data, so you should set number mappers to 20-30 as thereis a limit for all-AMP concurrently running operations on Teradata side. You need to make sure that user usedfor transferring data have sufficient spool space available for the SELECT queries.

Importing data with split.by.hash method

This input method is very similar to previous method. Instead of directly operating on value ranges of one column,this method will operate on hash of the column. Importing by hash can be used for extracting data in situationswhere split.by.value and split.by.partition are not appropriate. Similarly as in previous case, each mapper willvery likely hit all AMPs available system and therefore you should set the number of mappers to 20-30 as thereis a limit for all-AMP concurrent jobs on the Teradata appliance.

Importing data with split.by.partition method

Split by partition method is preferred method for extracting a large amount of data from Teradata system.Behavior of this method depends whether source table is partitioned or not, so we will discuss both optionsseparately.

In case that your input table is not partitioned, connector will create partitioned staging table and execute anINSERT into SELECT query to move data from the source table into the staging table. Subsequently each mapperwill transfer data only from one single partition resulting in single AMP operation. Due to the single AMP natureyou can use a large number of mappers to gain the best performance. You will need at least the same amountof permanent space as your source table and sufficient amount of spool space to execute the SELECT queries.



If your table is already partitioned, then no extra staging table is created. You can however force the connectorto repartition your data using --staging-force parameter to achieve better performance. Without forcing repartitionof the data this method will usually end up opening all-AMP operation and thus it’s recommended to use 20-30mappers. The only difference is when your source table is PI table and your split by column is the table’s primarykey. In that case connector will create only single AMP operations and thus you can use high number of mappers.

Exporting data with batch.insert method

Batch insert method is utilizing JDBC batch execution to export data to the Teradata appliance. This methodshould be used only in situations where other methods are not a viable options. This method will create partitionedstating table prior the export job and then subsequently each mapper will transfer data from Hadoop to onesingle partition. After all mappers will end finish INSERT into SELECT statement is called to transfer the datafrom staging to table to final destination. You need to ensure sufficient amount of permanent space availableto have two copies of your data. You will also need sufficient spool space for executing the INSERT into SELECTquery. Maximal number of mappers that this method can use is only limited by maximal number of concurrentsessions allowed in the Teradata appliance.

Exporting data with multiple.fastload method

This method is utilizing FastLoad over JDBC technology. Similarly as in batch.insert case connector will firstlycreate a partitioned staging table. Each mapper is then loading data to a single partitioned followed by INSERTinto SELECT statement that will move all staged data to final table. You need to make sure that you havesufficient permanent and spool space to support the two copies of data and INSERT into SELECT statement.Each mapper will start it’s own FastLoad job that will consume one utility slot on your Teradata appliance. Asthe number of utility slots is typically between 15 and 30, it’s advised to set number of mappers accordingly.

Exporting data with internal.fastload method

Similarly as in multiple fastload method even the internal fastload requires partitioned staging table. Data arefirstly exported by each mapper into different partition and subsequently moved to the target table using INSERTinto SELECT statement. You need to make sure that you have sufficient permanent and spool space to storetwo copies of your data and to move them from staging table to target table. Unlike in multiple.fastload casein this method all mappers are participating in one FastLoad job by coordination using internal protocol. This isthe fastest available method for exporting data from Hadoop to Teradata appliance. Because all mappersparticipates in the one FastLoad job, only one utility slot on your Teradata is taken for the export job. Subsequentlyyou can use much higher number of mappers that is limited only by total number of AMP available in yoursystem.

Using the Cloudera Connector for TeradataAfter you have installed the Sqoop connectors and copied the necessary JDBC drivers for Teradata to the libdirectory of Sqoop installation, you can use this connector by invoking the Sqoop tools with the appropriateconnection string.

The connection string must be of the form jdbc:teradata://<td-host>/DATABASE=<tdinstance> where

• <td-host> is the host name of the machine where Teradata server runs.• <td-instance> is the Teradata database instance name.

The connector automatically uses FastExport/FastLoad for better performance.

For example, the following command invokes the Sqoop import tool with three mappers and uses the ClouderaConnector for Teradata:

$ sqoop import --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --table MY_TABLE --num-mappers 3 \--escaped-by '\\' --fields-terminated-by ',' --lines-terminated-by '\n'



The following command invokes the Sqoop export tool with three mappers and uses the Cloudera Connectorfor Teradata:

$ sqoop export --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --export-dir /user/sqooptest/MY_TABLE \--table MY_TABLE_TARGET --num-mappers 3 --input-escaped-by '\\' \--input-fields-terminated-by ','

The connector uses short-lived tables for the import and export use-cases. The user can control some aspectsof creation and deletion of these tables through the following command line arguments. These arguments mustdirectly follow the tool name and precede any tool-specific arguments:

• -D sqoop.teradata.import.use.temporary.table: Specifies whether the import temporary table shouldbe used or not. The default is true.

• -D sqoop.teradata.import.delete.temporary.table: Specifies whether the import temporary tableshould be deleted after job completion or not. The default is true.

• -D sqoop.teradata.import.table.suffix: Controls the naming of the import temporary table, if thesource table name is foo, then the temporary table will be named: foo<suffix>. The default suffix stringis _temp, so in this example, the name will be: foo_temp.

• -D sqoop.teradata.import.free.form.temporary.table.name: Specifies the name of the importtemporary table for free-form query. When used with free-form query, Teradata manager requires a temporarystaging table, so this option is required.

• -D sqoop.teradata.export.delete.temporary.tables: Specifies whether the export temporary tablesshould be deleted after job completion or not. The default is true.

• -D sqoop.teradata.export.delete.temporary.tables: Specifies whether the export temporary tablesshould be deleted after job completion or not. The default is true.

• -D sqoop.teradata.export.tables.suffix: Controls the naming of the export temporary tables, if thetarget table name is bar, then the temporary tables will be named: bar<suffix><task_number>, note thatthere will be as many temporary tables as there are map tasks (specified using the -m sqoop argument). Thedefault suffix string is _temp_, so in this example, the names will be: bar_temp_0, bar_temp_1, etc.

• -D sqoop.teradata.export.temporary.table.name: Specifies the name of the export temporary table.Cloudera Teradata Connector needs to create multiple temporary tables and thus will add numeric suffix tothis name based on number of used mappers. Teradata has limit of 30 characters per table name and thismust be taken into account when specifying custom name. This parameter will override configuration ofsqoop.teradata.export.tables.suffix.

• -D sqoop.teradata.session.query.band: Allows arbitrary query bands to be set for all queries that areexecuted by Sqoop. Sqoop is expecting a semicolon separated key=value pair list. Please note that TeradataUser GuideCloudera Connector for Teradata | 11 Using Cloudera Connector for Teradatarequires a finalsemicolon after the last key=value pair as well (for example, Data_Center=XO;Location=Europe;).

For example, to keep the temporary table during import and to specify a suffix name (_cloud):

sqoop import -D sqoop.teradata.import.delete.temporary.table=false \-D sqoop.teradata.import.table.suffix=_cloud \--connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --table MY_TABLE --num-mappers 3

Note that each argument described above must be prefixed by the "-D" option.

By default, the temporary table is created under the same database where the import/export table is located.This can be changed by providing a qualified table name under a database different from the one that is currentlyconnected.

For example:

sqoop import --connect jdbc:teradata://localhost/DATABASE=sqooptest \--username sqooptest --password xxxxx --table tabledb.MY_TABLE



In this case, the temporary table is created under sqooptest while the import table MY_TABLE is located undertabledb.

Note: While the temporary tables ensure all-or-nothing atomicity for import and export operations,they also take up extra space in the database. If there is not enough space to accommodate thetemporary tables in the database, this will cause an error.



Uninstalling Teradata Connectors

To remove the Cloudera Connector for Teradata:

• Delete the cldra_td_connector file from managers.d directory located under the Sqoop configurationdirectory.

• Remove the files from the connectors distribution. These files might be in a location under /usr/lib/.• Remove the JDBC drivers for Teradata that you copied to the lib directory of the Sqoop installation. Sqoop

will no longer need these drivers. The Teradata JDBC drivers are named terajdbc4.jar and tdgssconfig.jar.


Uninstalling Teradata Connectors

Limitations

All Teradata Connectors currently provided by Cloudera have several functional limitations. These connectors:

• Do not support import into HBase.• Do not support AVRO format.• Do not support import-all-tables.• Do not support upsert functionality (parameter --update-mode allowinsert).

Cloudera Connector for Teradata version 1.1.1 has several functional limitations in addition to those limitationsthat apply to all connectors. The Cloudera Connector for Teradata version 1.1.1:

• Does not support imports from views.• Does not support data types INTERVAL, PERIOD, and DATE/TIME/TIMESTAMP WITH TIME ZONE.• Optional query band is not set for queries executed by Teradata JDBC driver itself (namely BT and SET SESSION

CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL SR).

Cloudera Connector for Teradata version 1.0.5 has several functional limitations in addition to those limitationsthat apply to all connectors. The Cloudera Connector for Teradata version 1.0.5:

• Does not support free-form queries.• Do not support imports from views.• Do not support data types INTERVAL, PERIOD, and DATE/TIME/TIMESTAMP WITH TIME ZONE.• Do not support upsert functionality (parameter --update-mode allowinsert).


Limitations

Getting Support

Support for the Cloudera Connector for Teradata is available via Cloudera Enterprise Support. Refer tohttp://www.cloudera.com/support for more details.


Getting Support

http://www.cloudera.com/content/support/en/home.html

Teradata Connectors User Guide -...

Documents

Transcript of Teradata Connectors User Guide -...