Cirro Admin Guide (PDF) - s3.amazonaws.com€¦ · This document describes the installation and...

Cirro Admin Guide v2.0

i

Copyright © 2016 Cirro. Confidential and Proprietary. All Rights Reserved.

Admin Guide For use with v2.0

Document Version 2.0

Document Release: January 28, 2016

Copyright © 2016 Cirro, Inc. Confidential and Proprietary. All Rights Reserved.


i


Table of Contents

1 Document Overview ............................................................................................................................... 1

1.1 Audience .......................................................................................................................................... 1

1.2 Related Documents .......................................................................................................................... 1

1.3 Usage Restrictions and Legal Statements ....................................................................................... 1

1.3.1 Confidential and Proprietary Information ............................................................................ 1

1.3.2 Trademarks ......................................................................................................................... 1

1.4 Cirro Corporate Offices .................................................................................................................... 2

1.5 Getting Help ..................................................................................................................................... 2

2 Cirro Installation ...................................................................................................................................... 3

2.1 Step 1: Determine Your Deployment Type ...................................................................................... 3

2.2 Step 2: Prepare Your Hardware ....................................................................................................... 3

2.2.1 Cirro Data Hub Hardware Requirements ............................................................................ 4

2.2.2 Cirro Data Hub Operating System Requirements ............................................................... 4

2.2.3 Network Connectivity Requirements ................................................................................... 4

2.3 Step 2: Copy your License File to the Cirro Data Hub ..................................................................... 4

2.4 Step 4: Prepare Your Software ........................................................................................................ 4

2.4.1 Third-Party Software Requirements .................................................................................... 4

2.4.1.1 Third-Party Software Requirements for All Configurations .......................................... 5

2.4.1.2 Data Source-Based Third-Party Software Requirements ............................................ 5

2.4.2 Additional Software Pre-Requisites .................................................................................... 8

2.5 Step 5: Install/Upgrade the Cirro Software ...................................................................................... 8

2.6 Step 6: Install Hadoop Name Node Services .................................................................................. 9

2.7 Step 7: Connect to Data Sources: Add Drivers, Resources, and Systems ..................................... 9

2.7.1 Hortonworks Hadoop Permissions Setup ........................................................................... 9

2.8 Step 8: Add the RMS Service ........................................................................................................ 10

2.9 Step 9: Create the Hive Connection .............................................................................................. 10

2.10 Installing to a Non-Default Location ........................................................................................... 10

2.11 Installing MySQL ........................................................................................................................ 10

2.12 The cirro_install_dem.sh Install Script Options ......................................................................... 11

2.13 When setting the password outside of an automated process, you should use the Reset Password script. See “4.3 Determining your Cirro Version .................................................................. 12

2.13.1 cirro_version Command Line Script .................................................................................. 12


ii


2.13.2 SQL Command ................................................................................................................. 12

2.14 Specifying a Custom Password for MySQL ............................................................................... 13

2.14.1 Changing the Password for New Installations .................................................................. 13

2.14.2 Changing the Password on Existing Installations ............................................................. 13

2.15 Specifying a CIRRO_USERNAME and CIRRO_GROUPNAME for Services .......................... 13

2.16 Integrating Cirro with Active Directory ....................................................................................... 14

2.16.1 Prerequisites for Active Directory Integration ................................................................... 14

2.16.2 Sample System Registration using LDAP Integration ...................................................... 16

2.16.3 Enabling SSL with an SSL-Secured LDAP Server ........................................................... 16

2.17 Troubleshooting Installation Problems ...................................................................................... 17

2.17.1 Permissions Errors for the /home/ Directory ..................................................................... 17

3 System Administration .......................................................................................................................... 18

3.1 Cirro Web Administration Console ................................................................................................. 18

3.1.1 Connecting to the Web Console ....................................................................................... 18

3.2 User Security Policies .................................................................................................................... 18

3.2.1 Username Policies ............................................................................................................ 18

3.2.2 Password Policies ............................................................................................................. 19

3.2.3 Setting User Security Policy Values .................................................................................. 21

3.3 Determining your Cirro Version ...................................................................................................... 21

3.3.1 cirro_version Command Line Script .................................................................................. 21

3.3.2 SQL Command ................................................................................................................. 21

3.4 Resetting the Cirro Data Hub Root User Password ....................................................................... 22

3.5 SSL Support for RMI Traffic ........................................................................................................... 22

3.6 Excluding and Including Schemas ................................................................................................. 22

3.7 Managing Multi-Part Object Name Mappings ................................................................................ 23

3.8 Creating Two-Part Object Name Mappings for tablenamepattern=true Driver Settings ................ 23

3.9 Audit Logs ...................................................................................................................................... 24

3.9.1 Accessing Audit Logs ........................................................................................................ 25

4 Component Reference ......................................................................................................................... 26

4.1 Cirro Data Hub (DH/DEM/FQO) ..................................................................................................... 26

4.2 HadoopNameNode ........................................................................................................................ 26

5 Document Change History ................................................................................................................... 28


1


1 Document Overview

This document describes the installation and day-to-day monitoring and management of the Cirro solution.

1.1 Audience

Users of this document should be system administrators of the Cirro solution. Users of this document should have expertise in server administration and understand the basic concepts of cloud computing and relational databases.

1.2 Related Documents

Cirro provides this documentation suite. Contact your Cirro representative to obtain additional documents.

Cirro Admin Guide: Provides instructions for system administrators who will install, configure, and maintain the Cirro solution.

Cirro SQL Specification: Defines the Cirro SQL language.

Cirro Analyst User Guide: Provides instructions on advanced and ad hoc querying, including using Cirro Analyst for Excel.

Cirro Business Intelligence User Guide: Provides instructions for using third-party business intelligence tools such as Business Objects and Tableau to work with Cirro, as well as installation instructions for the Cirro ODBC and JDBC drivers.

Cirro Functions Guide: Provides definitions for the Cirro pre-defined function library and information on creating custom functions.

1.3 Usage Restrictions and Legal Statements

1.3.1 Confidential and Proprietary Information

This document contains the confidential and proprietary information of Cirro, Inc. This document is submitted on a confidential basis and parties accessing this presentation are required to maintain the confidentiality of all confidential information contained herein. This document may not be distributed or reproduced in whole or in part or shown to any person without the prior consent of Cirro. The information contained herein is believed to be reliable, but no warranty is made as to the accuracy of any such information. Circumstances could change since the date this information was supplied.

1.3.2 Trademarks

The following terms are trademarks of Cirro, Inc.:

Cirro Data Hub, Cirro Analyst for Excel

The following terms are trademarks or registered trademarks of their respective owners and no claims to rights to such marks are made by Cirro, Inc.:


2


Microsoft, Windows, Windows NT, Excel, PowerPivot and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Apache, Apache Hadoop, Hadoop, HDFS, Avro, Cassandra, Chukwa, HBase, Hive, Mahout, Pig, Zookeeper are trademarks of the Apache Software Foundation.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Twitter is a trademark of Twitter Inc. in the United States, other countries, or both.

Additional marks may be included in this document that are the trademarks, trade names, logos, and service marks of their respective owners and no rights to such marks are made by Cirro, Inc.

1.4 Cirro Corporate Offices

Address all questions or comments regarding this documentation to Cirro at:

Cirro, Inc. 120 Vantis, Suite 500 Aliso Viejo, CA 92656 Phone: (949) 373-9600

1.5 Getting Help

For assistance using Cirro products, contact Cirro Support at [email protected].

mailto:[email protected]


3


2 Cirro Installation

This section describes the steps performed when installing Cirro.

The steps below are provided for reference only, and may differ from the steps required for your installation. Contact your Cirro representative for assistance planning and installing Cirro. Do not proceed to the steps below unless directed to do so by your Cirro representative.

2.1 Step 1: Determine Your Deployment Type

Cirro supports both cloud and data center deployments. Cirro can assist you and your company to understand the implications of each deployment type.

2.2 Step 2: Prepare Your Hardware

Whether you will deploy to a cloud or data center, prepare all of the hardware to Cirro specs. Obtain the specs from your Cirro representative. Make sure all hardware is set up and meets these requirements prior to starting installation.

You may want to create a reference chart, summarizing the components for your system and their metadata, as shown in the chart below:

Cirro Node Machine Name Security Group

Public IP

Cirro Data Hub / DEM <uniquename> <ip>

Hadoop Name Node <uniquename> <ip>

MySQL <uniquename> <ip>

Oracle <uniquename> <ip>

Greenplum <uniquename> <ip>

SQL Server <uniquename> <ip>

Teradata <uniquename> <ip>

Netezza <uniquename> <ip>

Redshift <uniquename> <ip>

Vertica <uniquename> <ip>

InfiniDB <uniquename> <ip>


4


SQLFire <uniquename> <ip>

Aurora <uniquename> <ip>

Hadoop Slaves <uniquename>

<uniquename>

<ip>

<ip>

Hadoop Name Node Backup <uniquename> <ip>

2.2.1 Cirro Data Hub Hardware Requirements

Each server must include these components:

CPU: Dual core processor

RAM: Minimum 8 GB.

Hard Drive: 100 GB.

2.2.2 Cirro Data Hub Operating System Requirements

The operating system must be one of the following. Generally speaking, all servers should have the same operating system and version.

Red Hat Enterprise Linux 6.x

CentOS 6.x

2.2.3 Network Connectivity Requirements

Cirro provides a Network Topology diagram with detailed information on network connectivity requirements. Please contact Cirro Support to obtain a copy of this diagram.

2.3 Step 2: Copy your License File to the Cirro Data Hub

The license file was provided to your organization with your Cirro product purchase.

If you do not have a copy of the license file, contact Support for assistance.

Copy the license file to the installation folder on the Cirro Data Hub.

2.4 Step 4: Prepare Your Software

2.4.1 Third-Party Software Requirements

Once the servers are set up, you will need to download third-party applications that are required by Cirro but not included in the Cirro installation package.

Not all of the software listed below is required for all installations. See the instructions below or check with your Cirro representative if you are unsure of which tools are needed for your system.

Go to the URL for each of the packages you need. Follow the instructions on each of the sites to accept the license agreement and download the products to the server.


5


Download the software packages to /usr/share/thirdparty on the Cirro Data Hub but do not install

the software at this time.

2.4.1.1 Third-Party Software Requirements for All Configurations

The following software is required regardless of the data sources you will connect to through Cirro.

Java SE Development Kit 7 Update 51

http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html#jdk-7u51-oth-JPR

Select jdk-7u51-linux-x64.tar.gz.

(If upgrading, please don’t remove the old version 1.6 JDK folder (/usr/local/bin/jdk1.6.0_25). This folder is still in use by other components. New installations which start with the JDK 7 do not need the 1.6.0 JDK.)

MySQL JDBC Driver Version 5.1.32

The MySQL connector/JDBC driver 5.1.32 is required for all installations.

http://downloads.mysql.com/archives.php?p=mysql-connector-java-5.1&o=other

Select Mysql-connector-java-5.1.32.tar.gz.

2.4.1.2 Data Source-Based Third-Party Software Requirements

The following drivers are required for connecting to the data sources listed and are only required if you intend to connect to each of these data types.

These files can be installed to /usr/share/thirdparty or a location of your choice.

Amazon Aurora database analytics engine driver

The following driver is required only if your Cirro Data Hub will connect to an Amazon Aurora Database Cluster.

http://dev.mysql.com/downloads/connector/j/r

Select the latest tar.gz file.

Amazon Redshift driver

The following driver is required only if your Cirro Data Hub will connect to Amazon Redshift.

Download the PostgreSQL JDBC 4 driver from http://jdbc.postgresql.org/download.html. Cirro validated against this driver version: http://jdbc.postgresql.org/download/postgresql-8.4-703.jdbc4.jar. This is the same driver as used for Pivotal Hawq. If your installation uses both database systems, only one copy of the PostgreSQL driver is needed.





http://jdbc.postgresql.org/download.html

http://jdbc.postgresql.org/download/postgresql-8.4-703.jdbc4.jar



6


Cloudera Impala

Impala installs on top of Cloudera Hive. It uses the Cloudera Hive JDBC driver and associated files, which are version-specific. These files are installed as part of the Cloudera installation, and details can be found in the Cloudera online documentation. Information on the latest version of Cloudera at the time of writing (5.5) can be found at http://www.cloudera.com/documentation/enterprise/latest/topics/impala_config.html

Greenplum JDBC driver for Greenplum Database version 4.2.1.0

The Greenplum JDBC driver is only required if your DH will need to connect to a Greenplum data source.

Download this file from the Greenplum location listed below: postgresql-8.4-701.jdbc4.jar

http://www.greenplum.com/community/downloads/database-ce/

InfiniDB

No software downloads are required for InfiniDB data sources.

Microsoft SQL Server JDBC Driver version 4.0

The Microsoft SQL Server JDBC driver is only required if your DH will need to connect to an existing Microsoft SQL Server installation at your company.

Download this file from the location below: Microsoft JDBC Driver 4.0 for SQL Server.

www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=11774

Also, on the SQL Server database, enable mixed authentication mode so that the Cirro Data Hub (DH) can connect using SQLServer authentication mode.

Microsoft SQL Server Connector for Apache Hadoop version 1.0

The Microsoft SQL Server Connector for Apache Hadoop is only required if your Cirro Data Hub will need to connect to an existing Microsoft SQL Server installation at your company.

Download this file from the location below: sqoop-sqlserver-1.0.tar.gz.

http://www.microsoft.com/download/en/details.aspx?id=27584

Netezza JDBC driver version 6 and Cloudera Connector for Netezza version 1.0.5

The Netezza JDBC driver and Cloudera Connector for Netezza are only required if your DH will connect to a Netezza database.

Download the following files:

Component Version File(s) Download Location

http://www.cloudera.com/documentation/enterprise/latest/topics/impala_config.html

http://www.greenplum.com/community/downloads/database-ce/

http://www.microsoft.com/download/en/details.aspx?id=27584


7


JDBC Driver

6.0.8 nzjdbc3.jar Not available for public downloading; contact Netezza technical support.

Sqoop Connector

1.0.5 sqoop-nz-connector-1.0.5.tar.gz

https://downloads.cloudera.com/connectors/sqoop-nz-connector-1.0.5.tar.gz

Oracle JDBC Driver version 11.2.0.3

This page lists the drivers that are available and the compatibility with different Oracle versions: http://www.oracle.com/technetwork/database/features/jdbc/

This page links to the download for the 11.2.0.3 driver, which we have certified for use with Cirro: http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html.

Pivotal Hawq

The Hawq driver is required only is your DH will connect to Pivotal Hawq.

Download the PostgreSQL JDBC 4 driver from http://jdbc.postgresql.org/download.html. Cirro validated against driver version 8.4-703, which can be found at http://jdbc.postgresql.org/download/postgresql-8.4-703.jdbc4.jar. This is the same driver as used for Amazon Redshift. If your installation uses both database systems, only one copy of the PostgreSQL driver is needed.

PostgresSQL JDBC driver

The PostgresSQL JDBC driver is only required if your DH will connect to a PostgresSQL database.

Go to https://jdbc.postgresql.org/ to obtain the driver. We support driver versions 8 and later.

SQLFire JDBC driver

The SQLFire JDBC driver is only required if your DH will connect to a SQLFire database.

Check http://www.vmware.com/products/application-platform/vfabric-sqlfire/overview.html to determine the driver version to download. You will need to download all of these files: sqlfiretools.jar, sqlfire.jar, and sqlfireclient.jar.

Sybase jConnect JDBC Driver version 7.0

For systems that will connect to Sybase IQ, use SAP jConnect for JDBC version 16 or above.

Download the latest release from http://www.sybase.com/products/allproductsa-

z/softwaredeveloperkit/jconnect.

Also, consult with your Sybase IQ DBA and make sure that your Sybase IQ database supports this version of JDBC driver.

Teradata JDBC driver version 1.3 and Cloudera Connector for Teradata version 1.0.5



http://www.oracle.com/technetwork/database/features/jdbc/

http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html

http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html

http://jdbc.postgresql.org/download.html


https://jdbc.postgresql.org/

http://www.vmware.com/products/application-platform/vfabric-sqlfire/overview.html

http://www.sybase.com/products/allproductsa-z/softwaredeveloperkit/jconnect

http://www.sybase.com/products/allproductsa-z/softwaredeveloperkit/jconnect


8


The Teradata JDBC driver and Sqoop connector are only required if your DH will connect to a Teradata database.

Download the following files:

Component Version File(s) Download Location

JDBC

Driver

1.3 tdgssconfig.jar

terajdbc4.jar

http://downloads.teradata.com/download/connectivity/teradata-

jdbc-driver-13-0-0-6

Sqoop

Connector

1.0.5 sqoop-td-

connector-

1.0.5.tar.gz

https://downloads.cloudera.com/connectors/sqoop-td-

connector-1.0.5.tar.gz

2.4.2 Additional Software Pre-Requisites

These additional configuration steps need to be performed before installation can begin:

1. Install the Procmail package on all servers.

2.5 Step 5: Install/Upgrade the Cirro Software

Note: The Cirro installation process installs a copy of MySQL. Do not install on a machine with an existing MySQL installation without consulting Cirro Support for assistance.

Note: You should have a live internet connection on your server during installation. If you do not, you will need to contact your System Administrator to install the compatible MySQL RPM version after installation.

To install or upgrade the Cirro Software, follow these steps:

1. Copy the installation file to the Cirro Data Hub (DH). This file can be obtained from Cirro Support.

2. Untar the file. 3. Change to the cirro_dem directory. 4. For new installations only, copy cirro_env.local to the folder /etc. 5. Modify the default passwords in this file, following the instructions in “2.14 Specifying a

Custom Password for MySQL” on page 13. 6. You may choose to modify others. See “2.15 Specifying a CIRRO_USERNAME and

CIRRO_GROUPNAME for Services” on page 13. 7. Do not modify any other values in the cirro_env.local file without instructions to do so from

Cirro Support. 8. The default installation location is usr/local/cirro. You may change this location by creating a

soft link to the desired installation location. See “2.10 Installing to a Non-Default Location” on page 10 for information.

9. Execute the file cirro_install_dem.sh with the correct options for your installation. See 2.12 "The cirro_install_dem.sh Install Script Options” on page 10 for instructions.

10. When prompted, enter and reenter the Cirro root user password.

If you encounter errors during the installation, contact Cirro Support for assistance.

http://downloads.teradata.com/download/connectivity/teradata-jdbc-driver-13-0-0-6

http://downloads.teradata.com/download/connectivity/teradata-jdbc-driver-13-0-0-6

https://downloads.cloudera.com/connectors/sqoop-td-connector-1.0.5.tar.gz

https://downloads.cloudera.com/connectors/sqoop-td-connector-1.0.5.tar.gz


9


2.6 Step 6: Install Hadoop Name Node Services

Please contact Cirro Support for Hadoop Name Node installation instructions.

2.7 Step 7: Connect to Data Sources: Add Drivers, Resources, and Systems

Follow these steps on the DH for each data source that will be accessible through Cirro.

First, follow these steps for any data sources of these types: greenplum, InfiniDB, mysql, netezza, oracle, redshift, sqlfire, sqlserver, sybaseiq, teradata, and vertica. These data sources can be added in any order. Contact Cirro Support for assistance adding InfiniDB or SQLFire.

See the Cirro SQL Reference Guide to see example commands for these data source types.

Finally, if you have data sources that are of type Cloudera, Hortonworks, or IBM InfoSphere BigInsights, add those in any order. Contact Cirro Support to obtain commands for these three data sources. Also, for Hortonworks Hadoop, see 2.7.1 “Hortonworks Hadoop Permissions Setup" on page 9 for important information on security settings for this data source.

1. Create the driver for the data source, using the CREATE DRIVER command. See the Cirro SQL Reference Guide for instructions on the CREATE DRIVER command.

For example,

CREATE DRIVER oracle2 OF TYPE oracle WITH CLASSNAME

'oracle.jdbc.OracleDriver';

2. Create the resource for this data source, using the ADD RESOURCE command. See the Cirro SQL Reference Guide for instructions on the ADD RESOURCE command.

For example,

ADD RESOURCE TO DRIVER oracle2 AT '/usr/share/thirdparty/ojdbc6.jar';

3. Create the system for this data source, using the CREATE SYSTEM command. See the Cirro SQL Reference Guide for information on the CREATE SYSTEM command. For example,

CREATE SYSTEM oracle2 OF TYPE oracle USING HOST "198.51.100.5" OPTIONS

( USER 'cirro', PASSWORD 'Password', TEMPDATABASE 'oracle',

DEFAULTDATABASE 'oracle', DRIVER 'oracle2', READONLY true) ;

2.7.1 Hortonworks Hadoop Permissions Setup

When using Hortonworks Hadoop, the default security settings must be modified to allow access to Cirro through the hdfs user (default).

To grant the needed permissions, modify the core-site.xml file (which can be accessed through Apache Ambari) to include these lines:

hadoop.proxyuser.hdfs.hosts

*

hadoop.proxyuser.hdfs.groups

*


10


2.8 Step 8: Add the RMS Service

Add the RMS service, using this command:

CREATE SYSTEM <systemName> OF TYPE rdbmsrms USING HOST <hostList>;

where

systemName is a user-specified name for the data source or sources.

hostList is one or more network-accessible IP address or server name for the data source. These data sources do not need to appear in a particular order.

For example,

CREATE SYSTEM rdbms2 OF TYPE rdbmsrms USING HOST "198.51.100.7";

or

CREATE SYSTEM rdbms2 OF TYPE rdbmsrms USING HOST "198.51.100.7",

"198.51.100.8", "198.51.100.9";

2.9 Step 9: Create the Hive Connection

This step is required only when connecting to a hive data source. Please contact Cirro Support for assistance.

2.10 Installing to a Non-Default Location

By default, the Cirro installation/upgrade process installs to usr/local/cirro on the installation machine.

To specify a different installation location, create a soft link in that directory by following these steps prior to installation:

1. Create one folder to hold the new cirro installation, in a location of your choice. (We will refer to this folder as the “Cirro install folder”. Create one folder in a location of your choice to hold the files belonging to an older Cirro install (the one currently present on the system). We will refer to this folder as “Cirro backup folder". The Cirro backup folder should be empty prior to install.

2. Create a soft link named /usr/local/cirro pointing to the Cirro install folder.

3. Create a soft link named /usr/local/cirro.old pointing at the Cirro backup folder.

4. During install, Cirro will move any pre-existing files from the Cirro install folder to the Cirro backup folder. Additional files such as startup scripts, and backup images of the mysql databases used by Cirro will be placed in the Cirro backup folder as will. This will help recover the old installation and configuration in case of a failure.

5. After backup has been completed Cirro will install new software to the Cirro install folder.

2.11 Installing MySQL

If you are manually installing MySQL prior to Cirro installation, follow the instructions below.


11


If you are installing Cirro using the Unix root account, the installation process will install MySQL automatically and you can ignore the instructions in this section.

1. Install MySQL using yum:

yum install mysql-server

If yum failed to install MySQL, attempt to install MySQL from rpm. Install version 5.0 or 5.1 only:

rpm -Uvh <mysql-package-name.rpm>

If you are unable to install with yum or rpm, please contact Cirro support about other installation options.

2. Start MySQL using:

service mysqld start

or

service mysql start

3. Configure MySQL to start automatically so Cirro can start the MySQL at boot:

chkconfig --level 2345 mysqld on

or

chkconfig --level 2345 mysql on

4. Configure the MySQL ${DEM_MYSQL_USER} user for password and privileges. As the root Linux user, login to MySQL using: "mysql -uroot", then issue the following commands, replacing xxxx with the appropriate password."

use mysql;

grant ALL privileges on *.* to '${DEM_MYSQL_USER}'@'%' identified by

'xxxx' with grant option;"

delete from mysql.user where password='';

flush privileges;

5. Ensure the variable DEM_MYSQL_PASS is set correctly in ${CIRRO_HOME}/var/cirro_env.local".

2.12 The cirro_install_dem.sh Install Script Options

Use the cirro_install_dem.sh install script to install the Cirro software. See “2.5 Step 5: Install/Upgrade the Cirro Software” on page 8 for information on when to use this script.

Usage

./cirro_install_dem.sh --install [ --force ] | --upgrade

[ --install-<service_name>-service | --uninstall-<service_name>-service ]

[ --password <password> ]

Install Options:


12


Use --install to perform a new install.

Use --force to force a new install even if an older install already exists. MySQL password should be manually changed if a new password is configured in cirro_env.local…. See “2.14 Specifying a Custom Password for MySQL” on page 13 for more information.

Upgrade Options:

Use --upgrade to perform an upgrade where an older installation already exists.

Password changes to systems are not applied during upgrade

Service Installation Options:

Use any combination of --install-<service_name>-service | --uninstall-<service_name>-service.

The available services for <service_name> are: o rdbms, Hadoop, Sqoop, and hive.

To install or uninstall all services, use –install-all-services.

Specifying both install and uninstall for the same service will uninstall it.

If a specific service is not mentioned in the install / uninstall list, the installer will upgrade it if it is already installed, but otherwise leave it unchanged.

Password Options:

Use --password <password> to set the Cirro Data Hub root password for scripting purposes

only. A prompt otherwise allows specification of a password/reenter password value.

2.13 When setting the password outside of an automated process, you should use the Reset Password script. See “3.3 Determining your Cirro Version

After installation, you can determine the Cirro version installed on your system using these methods.

2.13.1 cirro_version Command Line Script

Run the cirro_version script:

/usr/local/cirro/etc/cirro_version

This script shows the Cirro revision (build) number, the last changed revision, and the last changed date.

Revision: 3769

Last Changed Rev: 3761

Last Changed Date: 2013-11-27 15:35:49 -0800 (Wed, 27 Nov 2013)

2.13.2 SQL Command

Retrieve the Cirro build number by running the loadFullVersion SQL procedure.

Syntax

CALL loadFullVersion ;

Returns

Returns a string similar to 10.9.0.0 alpha - (Cirro 1.5.2.3769)


13


Resetting the Cirro Data Hub Root User Password” on page 21.

Note: Do not use the cirro_install_dem.sh script to install only the Cirro services on a system. Use the cirro_install_svcs.sh script instead.

2.14 Specifying a Custom Password for MySQL

Cirro installs MySQL as part of the product installation.

The password for the Cirro account on this system is specified in the cirro_env.local file. (See “2.5 Step 5: Install/Upgrade the Cirro Software” on page 8.)

To change this password to your own value, follow the steps below.

2.14.1 Changing the Password for New Installations

During Step 4 of the installation, and immediately after copying cirro_env.local to your installation server, uncomment this line in the file and set the password to a value of your choice:

# export DEM_MYSQL_PASS=password

Save the file and continue with the installation as described in Step 4.

2.14.2 Changing the Password on Existing Installations

If you need to change the password after you have completed the Cirro installation, follow these steps:

1. Log into the administrator interface for MySQL on the server, using the Cirro admin user and current password as shown in the cirro_env.local file. The user is “root”.

2. Change the Cirro user’s password in MySQL.

3. Open the cirro_env.local file and change the password in the file to match the new value you specified in MySQL. If you have not previously used a custom password for MySQL, you may also need to uncomment the _PASS variable for that line.

# export DEM_MYSQL_PASS=password

4. Restart the Cirro Data Hub for this change to take effect.

2.15 Specifying a CIRRO_USERNAME and CIRRO_GROUPNAME for Services

Cirro uses externally located services to perform operations such as starting Hadoop jobs, managing driver jars used by Apache Hive, performing Sqoop imports and exports, and transferring tables between different remote systems

When a Cirro service is started, Cirro looks at the environment variable CIRRO_USERNAME in order to determine the identity assumed by the Cirro service.


14


In order to ensure that the Cirro service runs properly, the Cirro install script modifies the ownership for folders and files used by the Cirro service such that they are owned by the user CIRRO_USERNAME and group CIRRO_GROUPNAME. (This only applies to files and folders installed by Cirro)

The default username / groupname used by the Cirro service is cirroclient. The system admin/installer can override this setting in cirro_env.local.

If the Cirro user name and group name are left unchanged, the Cirro install tool will create the user and group during install. Otherwise, the admin must manually create the user and group.

The cirro_env.local file includes default values for the Cirro username and Cirro group name on each of these systems, under the variables CIRRO_USERNAME and CIRRO_GROUPNAME. The installation process will create the Cirro accounts with these values.

# Tell Cirro Which user name to use for services.

# If you don't use cirroclient you have to create that user before

attempting to install.

# export CIRRO_USERNAME=cirroclient

# Set CIRRO_GROUPNAME if the group ${CIRRO_USERNAME} does not exist.

Otherwise this variable

# Defaults to ${CIRRO_USERNAME}

# export CIRRO_GROUPNAME="${CIRRO_USERNAME}"

If you want to use other username and group name values, create the user and group yourself (or use an existing user and group) on the services. Modify the cirro_env.local file to include these values, instead of the default values. You may need to uncomment the variables in the file. These values can only be set during initial installation.

2.16 Integrating Cirro with Active Directory

Your Cirro installation can be configured to integrate with an existing Active Directory/LDAP instance on your network. Once integrated, Active Directory users can automatically log in to Cirro without separate user creation and management. These users can also receive securable rights based on their group or role within Active Directory.

Integration with Active Directory is performed after Cirro installation.

2.16.1 Prerequisites for Active Directory Integration

Before starting the Active Directory integration, make sure this information is on hand.

The Active Directory schema for the system to which Cirro will connect, down to the role level where Cirro permissions will be assigned.

Write access to the derby.properties file on the Cirro Data Hub. This requires the same user account which was used to install Cirro.


15


If connecting to an SSL-enabled LDAP directory, follow the steps in "2.16.3 Enabling SSL with an SSL-Secured LDAP Server" on page 16 to set up the keys prior to continuing with the integration.

To add the integration, perform these steps:

1. In derby.properties:

a. Turn on LDAP:

derby.authentication.provider=org.apache.derby.impl.jdbc.authenti

cation.CirroLDAPAuthenticationServiceImpl

b. Specify the location of the Active Directory server:

derby.authentication.server=ldap://127.0.0.1:389/

c. Specify the LDAP search base:

derby.authentication.ldap.searchBase=CN=Users,DC=engineering,DC=m

yCompany,DC=com

d. Provide an Active Directory search filter:

derby.authentication.ldap.searchFilter=(&(objectClass=user)(sAMAc

countName=%USERNAME%))

e. Provide information on how to log in to search. This step is not necessary if anonymous lookup is allowed by Active Directory.

derby.authentication.ldap.searchAuthDN=CN=Vishwas,CN=Users,DC=eng

ineering,DC=myCompany,DC=com

derby.authentication.ldap.searchAuthPW=password)

2. Save the derby.properties file and restart Cirro.

3. Within Cirro, create roles which will contain the privileges for an Active Directory role. Use the WITH EXTERNAL clause to include the distinguished name. See the Cirro SQL Reference Guide for information on CREATE ROLE. The following are example values only.

CREATE ROLE engineering WITH EXTERNAL

'CN=Engineering,CN=Users,DC=engineering,DC=myCompanyDC=com';

CREATE ROLE developers WITH EXTERNAL

'CN=Developers,CN=Users,DC=engineering,DC=myCompany,DC=com';

CREATE ROLE services WITH EXTERNAL

'CN=Services,CN=Users,DC=engineering,DC=myCompany,DC=com';

4. Grant or deny privileges to Cirro objects to these roles. The following are example values only.


16


GRANT PUBLIC TO engineering;

GRANT PUBLIC TO developers;

GRANT PUBLIC TO services;

GRANT SELECT ON SQLSERVER1.inventory.dbo.TableEng TO engineering;

GRANT SELECT ON SQLSERVER1.inventory.dbo.TableTest TO engineering;

GRANT SELECT ON SQLSERVER1.inventory.dbo.TableDev TO developers;

DENY SELECT ON SQLSERVER1.inventory.dbo.TableTest TO developers;

GRANT SELECT ON SQLSERVER1.inventory.dbo.TableSer TO services;

Active Directory users associated to the roles created in CREATE ROLE (e.g. "engineering") should now be able to log in with access as specified in the GRANT and DENY statements.

The ADD AUTHORIZATION command can also be used to add authorization for a role to a system. See the Cirro SQL Reference Guide for information on the ADD AUTHORIZATION command.

2.16.2 Sample System Registration using LDAP Integration

To use the LDAP credentials of the Cirro user when connecting to a remote system, you must configure the remote system. Here is an example of how you enable that feature for SQL Server.

NOTE: You must use the jTDS for SQL Server driver to enable LDAP functionality. Contact Cirro Support to obtain this driver.

Add the jTDS driver:

CREATE DRIVER JTDS_DVR OF TYPE SQLSERVER WITH CLASSNAME

'net.sourceforge.jtds.jdbc.Driver';

ADD RESOURCE TO DRIVER JTDS_DVR AT '/usr/share/thirdparty/jtds-

1.3.1.jar';

Add the SQL Server as a Cirro system. Note the use of the SAME_AUTHORIZATION ‘true’ option, which causes connections made by the Data Hub to the remote SQL Server system to be made using the credentials of the LDAP user who is connected to the Data Hub. The CONNECTION_PREFIX value may not be required for all system types.

CREATE SYSTEM SQLSERVER1 OF TYPE SQLSERVER USING HOST "198.51.100.4"

OPTIONS ( USER 'cirroclient', PASSWORD 'password'), TEMPDATABASE

'temppoc', TEMPSCHEMA 'dbo', DRIVER 'JTDS_DVR',

CONNECTION_PREFIX 'jdbc:jtds:sqlserver://', SAME_AUTHORIZATION 'true',

CONNECTION_STRING_PROPERTIES ';domain=ENGINEERING');

2.16.3 Enabling SSL with an SSL-Secured LDAP Server

When using an SSL-secured LDAP server, additional steps are required to set up the system.

Perform these steps from a command prompt on the Cirro Data Hub. Replace "/opt/cirro" in all commands shown below with the path for your Cirro Data Hub.

1. Get the LDAP SSL certificate and store it in a file:


17


echo -n | openssl s_client -connect <LDAP hostname/IP>:636 | sed -ne '/-BEGIN

CERTIFICATE-/,/-END CERTIFICATE-/p' > ldapserver.pem

2. Generate a list of current certificates in the keystore, for comparison later:

keytool -list -keystore /opt/cirro/var/hubTrustStore.key -storepass

7V0i5U65fd4q91k

3. Import the Certificate into the keystore:

keytool -import -file ~/ldapserver.pem -keystore

/opt/cirro/var/hubTrustStore.key -storepass 7V0i5U65fd4q91k -alias ldaps

4. List the current certificates in the keystore to verify the certificate was imported correctly:

keytool -list -keystore /opt/cirro/var/hubTrustStore.key -storepass

7V0i5U65fd4q91k

5. Restart Cirro. This step can be skipped if immediately preceding Active Directory setup, which includes a Cirro restart.

cirro_fqod restart

2.17 Troubleshooting Installation Problems

This section lists errors you may encounter when installing the Cirro software.

If you do not find the solution to an installation problem in this section, please contact Cirro Support for assistance.

2.17.1 Permissions Errors for the /home/ Directory

The Cirro installation creates user accounts and installs software and configuration files inside the users’ home folder. Normally this means writing to the /home directory. On some systems, this directory is not accessible during installation. You may see permissions or access errors to this directory during installation.

When this happens, you can either change permissions so that this directory is accessible, or follow the steps below to install to another directory.

1. Create a new user account for the components, with a home directory in a location other than /home/.

2. Modify the cirro_env.local file to uncomment USERS_HOME and specify the new home location for component installation. Save cirro_env.local.

3. Retry the installation.


18


3 System Administration

This chapter describes the administration tools and typical administration actions associated with your Cirro installation..

3.1 Cirro Web Administration Console

The Cirro Web Administration Console provides:

Information on active and recent sessions and queries, with the ability to cancel sessions and queries.

User and role management.

System and driver management.

A web-based SQL client.

Log access.

Access to Cirro drivers and tools.

Access to the full Cirro documentation set.

3.1.1 Connecting to the Web Console

Access the Web Console on port 8443 (https only) of the Cirro Data Hub. Use either the network name or the IP address. For example, if the IP of the DH is 123.123.123.12, browse to https://123.123.123.12:8443. The Web Console is available any time the DH is up. If you are unable to access the Web Console at this page and the DH is up, check that the DH is available to you on the network.

A login account is required to access the Web Console. These are the same accounts and credentials used to connect to the Cirro Data Hub.

Enter your username and password in the login form shown. Contact Cirro Support if you do not know the Web Console username and password.

3.2 User Security Policies

3.2.1 Username Policies

Administrators can specify the user security policies for Cirro user usernames. The values listed here are configurable per installation:

Policy Name Description Potential Values

1

userid_numeric_length_minimum The minimum number of numeric characters required in a username.

0-100

1 Specifying "0" for any policy disables that policy.

https://123.123.123.12:8443/


19


If this policy is disabled, there is no minimum number of numeric characters

userid_character_length_minimum The minimum number of non-numeric characters required in a username.

If this policy is disabled, there is no minimum number of non-numeric characters.

0-100

3.2.2 Password Policies

Administrators can specify the user security policies for Cirro user passwords. The values listed here are configurable per installation:

Policy Name Description Potential Values

2

failed_attempts_limit The number of consecutive failed login attempts before a user account is placed into a locked state.

0-100

password_history_lockout

The number of recent passwords which the user cannot use when specifying a new password. For example, if this value is set to '2', then the user cannot re-use their previous 2 passwords.

0-100

inactivity_duration_limit The amount of time, in minutes, that a user account can be inactive before the user must log in again. A user is considered active and the inactivity counter resets to zero any time the user submits a request to the Cirro Data Hub for that session.

0-100

password_length_minimum The minimum number of characters for a user password. This value must be less than or equal to password_length_maximum, or disabled by specifying "0".

If this policy is disabled, the minimum password length is zero.

0-100

2 Specifying "0" for any policy disables that policy.


20


password_length_maximum The maximum number of characters for a user password. This value must be greater than or equal to password_length_minimum, or disabled by specifying "0".

0-100

password_lower_case_minimum The minimum number of lowercase letters required in a user password.

If this policy is disabled, no lowercase letters will be required.

0-100

password_upper_case_minimum The minimum number of uppercase letters required in a user password.

If this policy is disabled, no uppercase letters will be required.

0-100

password_punctuation_minimum The minimum number of special characters required in a user password.

If this policy is disabled, no special characters will be required. When enabled or required, the supported special characters are any which are supported by SQL.

0-100

password_numeral_minimum The minimum number of numerical digits required in a user password.

If this policy is disabled, no digits will be required.

0-100

password_character_type_minimum The minimum number of character types (lowercase, uppercase, punctuation, and numeral) which must be included in a user password.

For example, if this value is set to '2', then the password must include at least two of the four types.

If this policy is disabled, users will be required to provide the minimum 4 character types.

0-4

password_expiration_days The number of days after their current password was created at which the user is

0-1000


21


automatically required to create a new password.

3.2.3 Setting User Security Policy Values

User security policy values can be set using either the Web Console or SQL commands.

To update the settings using SQL commands, use the root user to submit an UPDATE statement, as shown here.

UPDATE security.password_requirements SET requirement = newValue WHERE

requirement_desc = 'policyName';

Arguments

newValue: the value to assign to policyName.

policyName: one of the policy values listed in the table above.

Example

UPDATE security.password_requirements SET requirement = 3 WHERE

requirement_desc = 'failed_attempts_limit';

3.3 Determining your Cirro Version

After installation, you can determine the Cirro version installed on your system using these methods.

3.3.1 cirro_version Command Line Script

Run the cirro_version script:

/usr/local/cirro/etc/cirro_version

This script shows the Cirro revision (build) number, the last changed revision, and the last changed date.

Revision: 3769

Last Changed Rev: 3761

Last Changed Date: 2013-11-27 15:35:49 -0800 (Wed, 27 Nov 2013)

3.3.2 SQL Command

Retrieve the Cirro build number by running the loadFullVersion SQL procedure.

Syntax

CALL loadFullVersion ;

Returns

Returns a string similar to 10.9.0.0 alpha - (Cirro 1.5.2.3769)


22


3.4 Resetting the Cirro Data Hub Root User Password

Use this script any time the Cirro Data Hub root user password needs to be reset. This script will only be needed after installation; you will be prompted to create the root password during installation.

1. Execute the script reset_password.sh, located in /user/local/cirro/bin.

2. When prompted, enter and reenter the new Cirro root user password.

3. Restart the cirro_fqod service for the new value to be effective.

3.5 SSL Support for RMI Traffic

RMI traffic within the Cirro system (e.g. between the Cirro Data Hub, Hadoop, and other components), occurs over peer-to-peer authenticated SSL.

SSL is turned on by default, and uses a Cirro-provided default key.

If you need to turn off SSL, or change the key, contact Cirro Support for assistance.

3.6 Excluding and Including Schemas

Schemas can be excluded (hidden) from remote data sources. This functionality allows an admin user to prevent tools from seeing some schemas on the data sources.

Cirro automatically excludes some schemas on Oracle and SQL Server which are administrative and do not include user data.

To use the exclude schema functionality:

5. Add a new remote data source using the CREATE REMOTE DATABASE command. (See the Cirro SQL Specification for information on CREATE REMOTE DATABASE.)

6. View the schema which are excluded for the remote data source from the SYS.SYSEXCLUDEDSCHEMAS system table. See the Cirro SQL Specification for information on this system table.

7. Exclude a schema using the EXCLUDE command: EXCLUDE schemaname ON remoteDBName;

For example, EXCLUDE inventory ON MySQL1;

and EXCLUDE inventory ON ‘MySQL 1’;

8. Include a previously-excluded schema using the INCLUDE command: INCLUDE schemaname ON remoteDBName;

For example INCLUDE inventory ON MySQL1;

and INCLUDE inventory ON ‘MySQL 1’;


23


3.7 Managing Multi-Part Object Name Mappings

Because Cirro interacts with multiple RDBMS’, we work with four-part object names, in the form: “db.catalog.schema.object”.

Many BI tools are not equipped to handle four-part names, and some require two-part names. Cirro provides the ability to perform two configurations in order to accommodate these needs.

To create mappings to four-part names, use the CREATE REMOTE DATABASE command. (See the Cirro SQL Specification for information on this command.)

3.8 Creating Two-Part Object Name Mappings for tablenamepattern=true Driver Settings

The Cirro ODBC and JDBC drivers support two-part table names if mapping rows for these tables have been created in the SYS.SYSTABLES system table.

These two-part table names are required for some BI tools. Cirro’s mapping functionality has the effect of “flattening” the database-schema-table hierarchy, making it easier for BI tools which expect a default database and default schema to discover and display the database, while hiding the fact that Cirro is a multi-system, multi-catalog, multi-schema environment. See the Cirro BI User Guide for information on when and how this functionality with the tablenamepattern value with each Cirro driver.

To create the mapping rows, use the CREATE REMOTE DATABASE SQL command. (See the Cirro SQL Reference Guide.)

You can check the database mappings by selecting from Cirro system tables (SYS.SYSTABLES, etc). When the tablenamepattern value in a driver is set to true, queries against table names return only information from Systems/Catalogs for which three-part mappings have been defined in the TABLENAME value. User queries can then reference the table using the delimited three-part name as the table name.

For example, a name mapping can be created to map “MYSQLCUST.schema.cust2” to “MYSQLCUST.inventory.schema.table”. The mapped table name is “MYSQLCUST-schema-cust2”.

When tablenamepattern is set to “false”, selecting from the SYS.SYSTABLES table will behave as shown.

SELECT DATABASENAME,SCHEMAID,TABLENAME FROM SYS.SYSTABLES WHERE TABLENAME

LIKE '%cust2%';

returns

DATABASENAME | SCHEMAID | TABLENAME

------------------------------------------------------------

MYSQLCUST | inventory | cust2

and

SELECT * FROM MYSQLCUST.inventory.cust2 LIMIT 2;

returns

cust_id | name | state

--------------------------------------------------------

1 | Big Jim | CA

2 | Medium Sam | CA


24


When a mapping has been created and tablenamepattern is set to “true”, selecting from the same values in SYS.SYSTABLES behaves as shown below.

SELECT DATABASENAME,SCHEMAID,TABLENAME FROM SYS.SYSTABLES WHERE TABLENAME

LIKE '%cust2%';

returns

DATABASENAME | SCHEMA | TABLENAME

-------------------------------------------------------------------------

MYSQLCUST | inventory | MYSQLCUST-inventory-cust2

and

SELECT * FROM "MYSQLCUST-inventory-cust2" LIMIT 2;

returns

cust_id | name | state

------------------------------------------------------------

1 | Big Jim | CA

2 | Medium Sam | CA

3.9 Audit Logs

Cirro provides an audit log which records security-related events. The audit log provides system administrators and security staff with the ability to review changes and attempted changes affecting change access.

This functionality is enabled automatically on all Cirro installations. No additional setup or configuration is required.

Audit records are stored in a separate MySQL database on the Cirro Data Hub. (It is possible to store the audit log on a separate server; contact Cirro Support for further information.)

The audit log is protected against accidental or deliberate deletes or updates. Users are limited to INSERT and SELECT actions only. Deletions or changes to existing records are detected.

The audit log tracks:

All actions performed by the root user.

The following actions performed by any user:

o System additions, modifications, and deletions.

o Driver additions, modifications, and deletions.

o User additions, modifications, and deletions, including permission and role membership changes.


25


o Role additions, modifications, and deletions, including role permission changes.

o Failed log in attempts.

Each record in the audit log includes:

A timestamp with the date and time of the event.

The username of the user performing the action.

The IP address of the user performing the action.

Detailed information about the action performed or change made.

Copies of the audit log can be archived to another location using the SELECT command.

The audit logs is never automatically trimmed or deleted by Cirro.

If it becomes necessary to trim or delete an audit log, contact Cirro Support for assistance.

3.9.1 Accessing Audit Logs

Access to the audit logs is gained via a user and password supplied during installation.

Contact Cirro Support for information on this user account.


26


4 Component Reference

This section describes the main Cirro components of a Cirro installation, and the key reference items, including port numbers, directories, and file locations.

This section provides a summary of the main components of the Cirro system. You will need to be familiar with each of these components in order to install or manage your Cirro system.

Not all of the components listed below are included in all Cirro installations, and some components might be installed on the same physical hardware.

Contact your Cirro representative for training on the functionality and management of each of these components.

4.1 Cirro Data Hub (DH/DEM/FQO)

This is the node where the external connections are presented.

Cirro Data Hub

o Port 1527

/etc/init.d/cirro_fqod [stop/start]

o Web Console: Port 8443 (https)

/etc/init.d/cirro_webd [stop/start]

o Binaries

/usr/local/cirro/lib

/usr/local/cirro/bin

/usr/local/cirro/etc

o Logs /tmp/cirro_fqe.log

- MySQL that controls DEM

- Fuse/Samba optionally installed

- Amazon tools packs

- Cassandra

o Administration Port:

o Binaries:

o Logs:

4.2 HadoopNameNode

- RMIRegistry

o Binary /etc/init.d/rmiregistryd [stop|start]

- RDBMStoRDBMS

o /etc/init.d/cirro_rdbmsd [stop|start]

o Binary



o Logs


27


/tmp/cirro_rdbmsd.log

- HadoopInterface

o /etc/init.d/cirro_haddopd [stop|start]

o Binary



o Logs

/var/log/messages

- SqoopInterface

o /etc/init.d/cirro_sqoopd [stop|start]

o Binary



o Logs

/var/log/messages

o

- HIVE

o /etc/init.d/cirro_hived [stop|start]

o Binary

/home/hadoop/hive*

/usr/lib/hive

o Logs

/var/log/messages -- System messages

/tmp/hadoop/hive.log -- User messages

- Hadoop/HBASE/sqoop/zookeeper

o Web Console http to 50030

o /etc/init.d/cirrod [stop|start] -- discovery and starts all nodes, works out problems of

synching name node with slaves

o Logs

/home/hadoop/hadoop*/logs

/tmp/

o Registry information go into the registry folder on DEM controls DEM and HNN

o Binaries

/home/hadoop/hadoop*

Linked to /usr/lib/hadoop

o Data (name node and slaves full data)

/home/hadoop/hdfs

- ORACLE Jar and SQL Server Jars are not legally distributable so the end user must download

them after that we can place them in /usr/local/cirro/lib on Hadoop Name Node.


28


5 Document Change History

This section lists recent changes to this document.

Document Version Release Date Description of Change(s)

2.0 January 28, 2016 Added more detailed instructions for enabling SSL with an SSL-enabled LDAP server.

Added driver information for connecting to Amazon Aurora database clusters.

Updated the driver link for Cloudera Impala.

Cirro Admin Guide (PDF) - s3.amazonaws.com€¦ · This document describes the installation and...

Documents

Transcript of Cirro Admin Guide (PDF) - s3.amazonaws.com€¦ · This document describes the installation and...