(c) Infinity 1

HADOOP MIGRATION OVERVIEW

dwivedishashwat@gmail.com

http://helpmetocode.blogspot.com

(c) Infinity 2

APPROACH

Preparation

Phase

•Access Grants

•Assessment of Current Infrastructure

•Inventory of Current Setup

•Meta-Data Backup

Methodology

•New Cluster Setup with Hadoop 2.x . OR Inline Migration to Hadoop 2.x from Hadoop 1.x

Staging

Environment

•Replica of Current Setup in Staging Environment

•Upgrade in Staging Environment

•Validation of Migration in Staging Environment

Productio

n Rollo

ut

•Upgrade of Production Environment

•Production Environment validation.

(c) Infinity 3

UPGRADE TO HADOOP 2.X WITH NEW CLUSTER SETUP

Upgrade a cluster by configuring a new cluster with same capacity and newer Hadoop version and then migrate the files/data from old cluster to new one.

(c) Infinity 4

UPGRADE TO HADOOP 2.X WITH NEW CLUSTER SETUP

CopyToLocal /CopyFromLocal The process flows, the files are copied to local drive using Hadoop

command CopyToLocal and the files are then pushed to the new cluster using CopyFromLocal, and the older cluster can be decommissioned.

Using Hadoop CP command This is a kind of cluster to cluster copy, using Hadoop ‘cp’ command the

files are transferred from one HDFS to other HDFS. As the version are different we need a mechanism called copy from HFTP where the command is executed from the target cluster by defining source as old cluster with HFTP protocol and target as HDFS protocol.

Using Hadoop DISTCP command This is a kind of cluster to cluster copy, using Hadoop ‘distcp’ command

the files are transferred from one HDFS to other HDFS. As the version are different we need a mechanism called copy from HFTP where the command is executed from the target cluster by defining source as old cluster with HFTP protocol and target as HDFS protocol.

(c) Infinity 5

INLINE MIGRATION TO HADOOP 2.X FROM HADOOP 1.X

Upgrading existing cluster from V1 to V2 inline by Installing/Configuring new cluster and updating metadata.

HDFS Upgraded Metadata HDFS

Hadoop V1 Hadoop V2

(c) Infinity 6

METHODOLOGY FOR INLINE MIGRATION FROM HADOOP 1.X TO HADOOP 2.X

Shutdown of Cluster/Nodes

Software Stack Stage Upgrade Phase

Validation of Upgrade Activity

Production Rollout

(c) Infinity 7

FOLLOW THE FOLLOWING LINK FOR STEP BY STEP UPGRADE PROCESS

http://wiki.apache.org/hadoop/Hadoop_Upgrade

(c) Infinity 8

Drop me mail @ dwivedishashwat@gmail.com

Visit my blog http://helpmetocode.blogspot.com