Co Vertica Copy PDF

7/25/2019 Co Vertica Copy PDF

1/12

Copyright IBM Corporation 2015 Trademarks

Managing Vertica clusters in a SaaS environment, Part 1:

Copying Vertica databases across the cluster

Page 1 of 12

Managing Vertica clusters in a SaaS environment, Part

1: Copying Verticadatabases across the cluster

Amol D. Barsagade([email protected])

Software Architect

IBM

03 June 2015

In the SaaS environment, data information and services are the key to customer success. HP

Vertica is a critical database technology that assists customers to better manage their data in

a cluster environment. This article illustrates the step-by-step approach for Vertica installation,

configuration, and setup acrossthe clusters. As a part of the setup, it is important to complete

the installation and configuration and then to copy the database across the cluster. .

View more content in this series

Introduction

NIOD is an IBM Unica product under Enterprise Marketing Management (EMM) which serves

customers in a SaaS environment that uses Hewlett-Packard Vertica clusters. The first step in

setting up the Vertica database is to install Vertica on the target cluster. Each source and targetcluster has three nodes. The required packages need to be verified, and the prerequisites for

copying the database across the cluster need to be followed. An empty database in the target

cluster is created. The database then is copied from the source to the target cluster.

Logging in to the source and target Vertica cluster nodes

Ensure that the passwordless SSH for the dbadminuser works between and across each of the

source and target cluster nodes. The dbadminuser must be able to log in to all the source and

target nodes without a password. Also, this user must be able to log in in the reverse order, that is,

from all target cluster nodes to all the source cluster nodes.

For simplicity, assume that the source and target Vertica clusters and nodes are as follows:

Table 1. Source and target Vertica cluster details

Cluster Type Cluster Node Name IP DB_Name User

Source source01.niod.unica.net 10.10.10.1 unicl01 dbadmin

source02.niod.unica.net 10.10.10.2 unicl01 dbadmin
http://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/developerworks/library/?search_by=Managing+Vertica+clusters+in+a+SaaS+environmenthttp://www.ibm.com/developerworks/library/?search_by=Managing+Vertica+clusters+in+a+SaaS+environmenthttp://www.ibm.com/developerworks/library/?search_by=Managing+Vertica+clusters+in+a+SaaS+environmentmailto:[email protected]://www.ibm.com/developerworks/ibm/trademarks/http://www.ibm.com/legal/copytrade.shtml


2/12

developerWorks ibm.com/developerWorks/

Managing Vertica clusters in a SaaS environment, Part 1:Copying Vertica databases across the cluster Page 2 of 12

source03.niod.unica.net 10.10.10.3 unicl01 dbadmin

Target target01.niod.unica.net 10.10.10.4 dbadmin

target02.niod.unica.net 10.10.10.5 dbadmin

target03.niod.unica.net 10.10.10.6 dbadmin

Copy the Vertica installer to the target cluster node

Assume that the Vertica binary file vertica-5.0.12-0.x86_64.RHEL5.rpm is available on the source

cluster and can be copied to the target cluster.

Copy the binary file from the source cluster to the first node of the target cluster,

target01.niod.unica.net (10.10.10.4).

For example,:

[ambarsag@source01 Vertica-binaries]$ scp vertica-5.0.12-0.x86_64.RHEL5.rpm [email protected]:/local/

Vertica-binaries

Note: Verify that the binary file was copied to the target cluster node and give full permissions so

that the dbadminuser can run it.

[ambarsag@target01 Vertica-binaries]$ chmod -R 777 vertica-5.0.12-0.x86_64.RHEL5.rpm

Verify packages on the target cluster

Verify that the required packages and files exist on each node of the target cluster.

Verification of sudo and Python

Verify that sudo and Python are available on all the target cluster hosts. Verify that you haveprivileges to use them on these hosts by running the following command at a shell prompt:

[root@target01 dbadmin]# which sudo

/usr/bin/sudo

[root@target01 /]# which python

/usr/bin/python

[root@target02 ~]# which sudo

/usr/bin/sudo

[root@target02 /]# which python

/usr/bin/python

[root@target03 ~]# which sudo

/usr/bin/sudo[root@target03 /]# which python

/usr/bin/python

Verification of rsync and version

Verify that rsync 3.0.5 or later is installed on all cluster hosts.

[root@target01 /]# which rsync

/usr/bin/rsync

[root@target01 /]# rsync --version

rsync version 3.0.6 protocol version 30


3/12

ibm.com/developerWorks/ developerWorks


Note: Verify the same on the other two nodes, target02and target03.

Verification of the su file

Verify that the sufile exists in /etc/pam.don all the cluster nodes.

[root@target01 pam.d]# pwd/etc/pam.d


Verification of NTP demon

Verify that the NTP Demon is running on all the target cluster nodes.

Note: If the NTP demon is installed, the output resembles:

The output indicates the run levels where the demon runs. Verify that the current run level of the system

(usually 3 or 5) has the NTP demon set to ON.

[root@target01 pam.d]# chkconfig --list ntpd

ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

[root@target01 /]# runlevel

N 3


Define the loopback address

Ensure that the /etc/hostsfile exists. Ensure that the file contains the loopback address 127.0.0.1

with only the name localhostin both fully qualified and unqualified forms.

For example:

127.0.0.1 localhost.localdomain localhost


Setting up the host fileSetup the /etc/hosts file on each node of the cluster

Make sure that the /etc/hostsfile includes all of the hosts that become part of the cluster.

[root@target01 /]# vi /etc/hosts

Input the following entry into the hosts file:

#Vertica HB

10.10.10.4 target01.niod.unica.net





4/12



Ensure port 5433 is not in use

[root@target01 /]# netstat -atup

[root@target01 /]# vi /etc/services

This command does not list the port number if it is identified by a service that is defined in /

etc/services. Some common services that might use port 5433 include pyrrho, nibbler, andpostgresql.


Ensure /dev/ptsis mounted

[root@target01 /]# cd /dev/pts



Installing Vertica on the target cluster

Choose one of the cluster hosts to be the Administration Host for the purposes of installing Vertica

and for using the Vertica Administration Tools. Here, the node target01.niod.unica.net is used

for the installation.

Run the RPM package installer

Note: Since sudois used to log in to the root, sudo su - , sudomust also be used to run the RPM

package installer.

[root@target01 Vertica-binaries]# sudo rpm -Uvh /local/Vertica-binaries/vertica-5.0.12-0.x86_64.RHEL5.rpm

Preparing... ########################################### [100%]

1:vertica ########################################### [100%]

Vertica Analytic Database V5.0.12-0 successfully installed on host target01.niod.unica.net

Note: If you are root, then use the following command:

# rpm -Uvh pathname

Note: This command creates a folder /opt/vertica.

[root@target01 vertica]# pwd

/opt/vertica

[root@target01 vertica]# ls

bin config examples include java lib lib64 log oss packages sbin scripts sdk share spread

Run the install script

To start the installation, run the install script from the /opt/vertica/sbinlocation.

Note: If you run the Vertica installation script without parameters, the script performs a single-node

installation on localhost.

If you are root:


5/12



# /opt/vertica/sbin/install_vertica -s host_list -r rpm_package -u dba_username

If you are using sudo:

$ sudo /opt/vertica/sbin/install_vertica -s host_list -r rpm_package -u dba_username

Note: The -rand -sparameters are not required for single-node installations.

Example:

[ambarsag@target01 sbin]$

sudo /opt/vertica/sbin/install_vertica -s 10.10.10.4,10.10.10.5,10.10.10.6 -r /local/Vertica-binaries/

vertica-5.0.12-0.x86_64.RHEL5.rpm -u dbadmin -p xxxx -g dbadmin

where:

-s: host_list

-r: rpm_package -u: dbadminuser

-p: password for dbadminuser

-g: group of dbadminuser

On successful completion of the installation you get the following message:

Verifying spread configuration on whole cluster

Creating node node0001 definition for host 10.10.10.4

... Done


... Done


... DoneError Monitor 0 errors 0 warnings

Installation complete.

Prerequisites for copying the database from the source to the target

cluster

To copy the source database unicl01to the target cluster nodes, the following guidelines must be

followed.

Have the same number of nodes as the source cluster

Make sure that the source and the target Vertica clusters have same number of nodes.

Be accessible from the source cluster

You might need to add the host name of each target cluster node into the hosts file of the source

cluster. Since this copy of the database is initiated from one of the source cluster nodes, all the

source node host files must contain the target host names.

On the first node of the source cluster, source01.niod.unica.net , log in as root.


6/12



[root@source01 /]# vi /etc/hosts

#Vertica SOURCE

10.10.10.1 source01.niod.unica.net



#Vertica TARGET




Note: Verify the same on the other two nodes, source02and source03.

Copy the license file from the source cluster to the target

The license file, license.key, must be copied to all the target cluster nodes. Assume that the

license file is on the source at /opt/vertica/config/share . In this example, the license file is

copied to each target cluster node at the /opt/vertica/config/share location.

Example:

source01:/opt/vertica/config/share $ ls -la

-rw-rw-rw- 1 dbadmin verticadba 547 Nov 16 2011 license.key

source01:/home/dbadmin $ scp license.key [email protected]:/opt/vertica/config/share

Note: Similarly, copy the license file to the other two nodes of the target cluster, 10.10.10.5 and

10.10.10.6.

Creating an empty database on the target cluster

Now, create an empty database on the target cluster with the same name, unicl01, as the source

cluster. Have a database with the same name as the database that is being copied. This database

can be empty.

Steps to create an empty database on the target cluster:

Log in to target01.niod.unica.net as the dbadminuser.

Run Vertica Admintools as follows:

[dbadmin@target01 bin]$ /opt/vertica/bin/admintools

This opens the "Main Menu" window.

Select "Configuration Menu".


7/12



Figure 1. Configuration Menu

Click OK.

On the "Configuration Menu" window, click "Create Database".

Figure 2. Create Database

Click OK.

Enter the name of the database and an optional comment.dbname: unicl01

comment: Copy db from source to target vertica cluster.

Click OK.

Enter a password for the new database.

Click OK.

Reenter the same password and click OK.

Select the list of target cluster nodes on which you want to create the same database.[X] 10.10.10.4

[X] 10.10.10.5

[X] 10.10.10.6


8/12



Catalog path name: /home/dbadmin

Data path name: /home/dbadmin

Click OK.

On the "Current Database Definition" window, review the details and click "Yes".

Note: The database unicl01is created on all three nodes of the target cluster. You can view thedatabase from the Vertica Admintools and can connect, start, and stop the database.

Copying the Vertica database from the source to the target cluster

Create the backup directories on all the three target cluster nodes. Then, create a configuration file

to be run along with the copyclustercommand.

Create the Backup directory on target nodes

Create the backup directory on all the three nodes of the target cluster as /home/dbadmin/backups .

Then, change the owner to dbadmin.

[root@target01 /]# mkdir /home/dbadmin/backups

[root@target01 /]# chown -R dbadmin:dbadmin /home/dbadmin/backups

Note: Create similar directories on the other two nodes of this cluster, target02and target03.

Identify the database node name.

Identify the database node names on the source cluster that are to be used in the configuration file

by running the copyclustercommand. For example:

source01:/opt/vertica/config $ /opt/vertica/bin/admintools -t node_map -d unicl01

DATABASE | NODENAME | HOSTNAME

--------------------------------------------------------

unicl01 | v_unicl01_node0001 | 10.10.10.1



Note: Here, the name that follows the -dflag is the database name.

Create a configuration file

You need to create a configuration file, CopyCluster.ini, on the source node,

source01.niod.unica.net . For example:

File name: CopyCluster.iniLocation: /opt/vertica/config/CopyCluster.ini

Sample Contents of CopyCluster.ini

[Misc]

snapshotName = CopySourceDB

tempDir = /tmp

verticaConfig = False

retryCount = 5

retryDelay = 1

[Database]


9/12



dbName = unicl01

dbUser = dbadmin

dbPassword = xxxx

dbPromptForPassword = False

[Transmission]

encrypt = False

checksum = False

port_rsync = 50000

bwlimit = 0hardLinkLocal = False

[Mapping0]

dbNode = v_unicl01_node0001

backupHost = 10.10.10.4

backupDir = /home/dbadmin/backups

[Mapping1]


backupHost = 10.10.10.5

backupDir = /home/dbadmin/backups

[Mapping2]


backupHost = 10.10.10.6backupDir = /home/dbadmin/backups

Note:

dbNode: Node_Nameof each source cluster.

backupHost: The target cluster node name.

backupDir: Back up directory on target cluster node.

dbName: Name of the source database, unicl01.

dbUser: Administrator user, the same on the source and the target cluster.

dbPassword: Password of source database.

Run the copyclustercommand

To copy the cluster, you need to run vbr.pyfrom a node in the source database. Use the database

administrator account, passing it the --task copyclustercommand. If you created a configuration

file specifically to copy the database, use the --config-fileparameter to tell the backup utility

where it is located.

Run the following command as dbadminfrom source01.niod.unica.net . Browse to /opt/vertica/

config/CopyCluster.ini .

source01:/home/dbadmin $ cd /opt/vertica/config

source01:/opt/vertica/config $ vbr.py --config-file CopyCluster.ini --task copycluster

Note: Here you might get the following error:

Node v_unicl01_node0001 of target cluster is not DOWN; cannot copycluster.

copycluster failed!

You need to stop the database unicl01on source01.niod.unica.net by using Vertica Admintools.

This action stops the database on the other two nodes as well. After you stop the database, run

the CopyClustercommand again.


10/12



source01:/opt/vertica/config $ vbr.py --config-file CopyCluster.ini --task copycluster

You might get the following message based on the size of the source database:

Copying...

151093501882 out of 151093501882, 100%

Upon successful completion of copying the database from the source to the target cluster, you

can start the database. Start the database from any one node of the target cluster nodes by using

Vertica Admintools.

Conclusion

This article describes the importance of various Vertica packages that are required for the

installation. It also helps to understand various configurations and the settings that are required

for the installation. The step-by-step approach to copying the Vertica databases of IBM NIOD

customers in a SaaS environment includes creating an empty database, setting the configurationfile, and running the copyclustercommand by using vbr.py.

Acknowledgment

I would like to extend appreciation to Marjana Ivkovic ([email protected]), Team Lead, EMM

SaaS Ops Applications Architects, for the continuous support and valuable technical guidance for

completing this article.


11/12



Resources

Learn

Copy the Database to Another Cluster

Vertica Forum Vertica Forum/Links

HP Vertica Analytics Platform Version 5.1.x Documentation

HP Haven Documentation

"Direct Integration with WebSphere Commerce" provides information on the direct integration

model of OMS and WebSphere Commerce.

"Sterling Definition Framework" for information on the Sterling SDF.

"Agent and integration servers" for information on the Sterling agent servers and integration

servers.

In the Commerce Zone, get information about other Commerce resources.

Stay current with developerWorks technical eventsfocused on a variety of IBM products andIT industry topics.

Follow developerWorks on Twitter.

Watch developerWorks on-demand demosranging from product installation and setup demos

for beginners, to advanced functionality for experienced developers.

Get products and technologies

Evaluate IBM productsin the way that suits you best: Download a product trial, try a product

online, use a product in a cloud environment.

Discuss

Participate in the discussion forum for this content.

Get involved in the My developerWorks community. Connect with other developerWorks

users while exploring the developer-driven blogs, forums, groups, and wikis.
http://www.ibm.com/developerworks/community/http://www.ibm.com/developerworks/community/https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000333http://www.ibm.com/developerworks/downloads/http://www.ibm.com/developerworks/offers/lp/demos/http://www.twitter.com/developerworks/http://www.ibm.com/developerworks/events/http://www.ibm.com/developerworks/commerce/http://www-01.ibm.com/support/knowledgecenter/SS6PEW_9.4.0/com.ibm.help.perf.manage.doc/c_FND_HA_AgentOrIntegrationServers.html?lang=en-ushttp://www-01.ibm.com/support/knowledgecenter/SS6PEW_9.4.0/com.ibm.help.om.prodover.doc/c_Overview_ServiceDefinitionFramework.html?lang=en-ushttp://www-01.ibm.com/support/knowledgecenter/SS6PEW_9.4.0/com.ibm.help.integ.94wcsc.doc/scwcintegrationoverview_sdf_94.html?lang=en-ushttp://www.vertica.com/hp-vertica-documentation/http://my.vertica.com/docs/5.1.6/HTML/index.htm#1241.htmhttp://vertica-forums.com/viewforum.php?f=52http://vertica-forums.com/http://my.vertica.com/docs/5.0/HTML/Master/16027.htm


12/12



About the author

Amol D. Barsagade

Amol D. Barsagade is a Database Architect with EMM SaaS Ops, IBM Experience

One, India Software Lab, Pune. Amol provides database solutions for database

architecture, performance tuning, capacity management, application development,

and migrations to various partners and customers in Asia-Pacific. Amol has a

bachelor's degree in computer science and several years of experience working

with relational database systems, including DB2, Oracle, and SQL server. Since

2006, he has made valuable contributions to IBM developerWorks by reviewing and

publishing articles.

Copyright IBM Corporation 2015

(www.ibm.com/legal/copytrade.shtml)

Trademarks(www.ibm.com/developerworks/ibm/trademarks/)
http://www.ibm.com/developerworks/ibm/trademarks/http://www.ibm.com/legal/copytrade.shtml

Co Vertica Copy PDF

Documents

Transcript of Co Vertica Copy PDF