www.KnowBigData.comHadoop

• Interact - Ask Questions

• Lifetime access of content

• Class Recording

• Cluster Access

• 24x7 support

WELCOME - KNOWBIGDATA

• Real Life Project

• Quizzes & Certification Test

• 10 x (3hr class)

• Socio-Pro Visibility

• Mock Interviews

www.KnowBigData.comHadoop

COURSE CONTENTI Understanding BigData, Hadoop ArchitectureII Environment Overview, MapReduce BasicsIII Adv MapReduce & TestingIV MapReduce in JavaV Pig & Pig LatinVI Analytics using HiveVII NoSQL, HBASEVIII Zookeeper,HBASEIX Oozie, Flume, Sqoop, MahoutX Intro Spark, Storm, Compare DBsXI YARN, Big Data Sets & Project Assignment

www.KnowBigData.comHadoop

WHAT IS SQOOP?

• Open source tool

• Extract data from a structured data store

• Into Hadoop for further processing

• Can also move data from DB to HBASE

• Can export data back to the data store

• Has extension framework - connectors

www.KnowBigData.comHadoop

A QUICK HELLO!ssh hadoop2.knowbigdata.com

>sqoop helpAvailable commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information

You can further get more help by:sqoop help import

www.KnowBigData.comHadoop

CONNECTORS

Sqoop ConnectorA modular component that uses extension framework To enable Sqoop imports and exports for a data store

Available Connectors:Include MySQL, PostgreSQL, Oracle, SQL Server, DB2.Generic JDBC Connector - any database that support jdbcThird Party too - Netezza, Teradata

www.KnowBigData.comHadoop

A SAMPLE IMPORTStart mysql:

cd /usr ; /usr/bin/mysqld_safe &

Create Sample Table in mysqlmysql -u root -pCREATE DATABASE sqoopex3;GRANT ALL PRIVILEGES ON sqoopex3.* TO '%'@'localhost';GRANT ALL PRIVILEGES ON sqoopex3.* TO ''@'localhost';

GRANT ALL PRIVILEGES ON sqoopex3.* TO '%'@'knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO ''@'knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO '%'@'%.knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO ''@'hadoop2.knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO '%'@'hadoop3.knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO ''@'hadoop3.knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO '%'@'hadoop4.knowbigdata.com';GRANT ALL PRIVILEGES ON sqoopex3.* TO ''@'hadoop4.knowbigdata.com';

www.KnowBigData.comHadoop

A SAMPLE IMPORT

Create Sample Table in mysqlCREATE TABLE widgets(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, widget_name VARCHAR(64) NOT NULL,price DECIMAL(10,2),design_date DATE,version INT,design_comment VARCHAR(100));

INSERT INTO widgets VALUES (NULL, 'sprocket', 0.25, '2010-02-10', 1,'Connects two gizmos');

INSERT INTO widgets VALUES (NULL, 'gizmo', 4.00, '2009-11-30', 4, NULL);INSERT INTO widgets VALUES (NULL, 'gadget', 99.99, '1983-08-13', 13, 'Our flagship

product');

www.KnowBigData.comHadoop

A SAMPLE IMPORT - HDFS

Import to a HDFS File:sqoop import --connect jdbc:mysql://hadoop1.knowbigdata.

com/sqoopex3 --table widgets -m 1 --username root --password ''

Check the content of the imported File:hadoop fs -cat widgets/part-m-00000

Also notice that widgets.java was created.

www.KnowBigData.comHadoop

A SAMPLE IMPORT - HIVE

Creates just the table:sqoop create-hive-table --connect jdbc:mysql://hadoop1.

knowbigdata.com/sqoopex3 --table widgets --fields-terminated-by ',' --username root --password ''Imports:hive> LOAD DATA INPATH "widgets" INTO TABLE widgets;

In single stepsqoop import --connect jdbc:mysql://hadoop1.knowbigdata.com/sqoopex3 --table widgets -m 1 --hive-import --username root --password ''

www.KnowBigData.comHadoop

A SAMPLE IMPORT - HBASE

Creates just the table:sqoop import --connect jdbc:mysql://hadoop1.knowbigdata.

com/sqoopex3 --table widgets --hbase-table 'customer' --column-family cf1 --username root --password '' --hbase-create-table --columns a1,b1 --hbase-row-key id

www.KnowBigData.comHadoop

EXPORT - HIVE TO MYSQLCreate Data File:cat > sales.log1,15,120 Any St.,Los Angeles,CA,90210,2010-08-013,4,120 Any St.,Los Angeles,CA,90210,2010-08-012,5,400 Some Pl.,Cupertino,CA,95014,2010-07-302,7,88 Mile Rd.,Manhattan,NY,10005,2010-07-18

Create Hive Table:CREATE TABLE sales(widget_id INT, qty INT,street STRING, city STRING, state STRING,zip INT, sale_date STRING)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','

Load:LOAD DATA LOCAL INPATH "sales.log" INTO TABLE sales;select * from sales;

www.KnowBigData.comHadoop

EXPORT - HIVE TO MYSQL

Create MYSQL Table:CREATE TABLE sales(widget_id INT, qty INT,street varchar(100), city varchar(100), state varchar(100),zip INT, sale_date varchar(100))

Export:sqoop export --connect jdbc:mysql://hadoop1.knowbigdata.com/sqoopex3 -m 1 --table sales --export-dir /apps/hive/warehouse/sales --input-fields-terminated-by ',' --username root --password ''