Build your own 4 node virtualized hadoop cluster
-
Upload
peter-avakian -
Category
Technology
-
view
194 -
download
5
Transcript of Build your own 4 node virtualized hadoop cluster
![Page 1: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/1.jpg)
![Page 2: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/2.jpg)
![Page 3: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/3.jpg)
Overall Setup
![Page 4: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/4.jpg)
Network Configuration
![Page 5: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/5.jpg)
Network Configuration and Host Setup
![Page 6: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/6.jpg)
Inspect Network Configuration
![Page 7: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/7.jpg)
Inspect Network Configuration (Cont.)
![Page 8: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/8.jpg)
Storage Capacity
![Page 9: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/9.jpg)
Increasing Online Storage Capacity - ESXi
![Page 10: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/10.jpg)
Increasing Online Storage Capacity – ESXi (Cont.)
![Page 11: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/11.jpg)
Increasing Online Storage Capacity – LVM
![Page 12: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/12.jpg)
Increasing Online Storage Capacity – LVM (Cont.)
![Page 13: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/13.jpg)
Increasing Online Storage Capacity – LVM (Cont.)
![Page 14: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/14.jpg)
Disable some services and update Yum
![Page 15: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/15.jpg)
![Page 16: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/16.jpg)
Some Prep Work
![Page 17: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/17.jpg)
Check for previous or unsuccessful installed CDH
![Page 18: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/18.jpg)
Check for previous or unsuccessful installed CDH (Cont.)
![Page 19: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/19.jpg)
Some log files that you might want took at
![Page 20: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/20.jpg)
Some Ports to keep track of
![Page 21: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/21.jpg)
Starting CDH Installation
![Page 22: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/22.jpg)
![Page 23: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/23.jpg)
Point Your Browser to Your Server http://<host_server_address>:7180
![Page 24: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/24.jpg)
Specify List of Hosts of Your CDH Cluster
![Page 25: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/25.jpg)
Specify List of Hosts of Your CDH Cluster (Cont.)
![Page 26: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/26.jpg)
Select CDH 5.4 Repository Version
![Page 27: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/27.jpg)
JDK Installation Options
![Page 28: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/28.jpg)
Enable Single User Mode (in my case I did not)
![Page 29: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/29.jpg)
Provide SSH Login CredentialsUsing private key or root password
![Page 30: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/30.jpg)
Wait for the Installation progressDepending on the size of the cluster and setup - it might take around +20 minutes
![Page 31: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/31.jpg)
Once the Installation is Successful>|Click Continue
![Page 32: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/32.jpg)
Distribution of CDH Parcels Will Start Downloading and Installing on the all Nodes
![Page 33: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/33.jpg)
Hosts Inspection (Refer to Linux Prep Work section in case you have some failures)
![Page 34: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/34.jpg)
Hosts Inspection (Cont.)(Scroll down the page to make all are fine)
![Page 35: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/35.jpg)
Select Types of Services and Roles you want to include
![Page 36: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/36.jpg)
Start Your Role Assignments(Make sure you have at least 3 Zookeeper hosts assigned )
![Page 37: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/37.jpg)
![Page 38: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/38.jpg)
View Role Assignment by Host
![Page 39: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/39.jpg)
Hive, CDH Reports Manager and Oozie Database Setup
![Page 40: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/40.jpg)
Review the Changes Before You Continue
![Page 41: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/41.jpg)
The Progress Screen for the Service Deployment
![Page 42: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/42.jpg)
The Progress Screen for the Service Deployment (Cont.)
![Page 43: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/43.jpg)
Cluster Setup is Complete
![Page 44: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/44.jpg)
What did CDH install?
![Page 45: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/45.jpg)
Check Status
![Page 46: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/46.jpg)
Check to see if you can login to the manager server
![Page 47: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/47.jpg)
![Page 48: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/48.jpg)
Some Spark Related Directories and Files
![Page 49: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/49.jpg)
Running Sample Spark
![Page 50: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/50.jpg)
Spark URL Job Monitor
![Page 51: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/51.jpg)
Spark Job Detail
![Page 52: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/52.jpg)
Spark Job Detail
![Page 53: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/53.jpg)
Running Spark in--deploy-mode client -- master yarn
![Page 54: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/54.jpg)
![Page 55: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/55.jpg)
![Page 56: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/56.jpg)
Running Spark in--deploy-mode client -- master local
![Page 57: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/57.jpg)
![Page 58: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/58.jpg)
![Page 59: Build your own 4 node virtualized hadoop cluster](https://reader036.fdocuments.us/reader036/viewer/2022062308/55c6748abb61ebd6608b46d1/html5/thumbnails/59.jpg)