Hortonworks Technical Preview for Apache Slider View for...

Architecting the Future of Big Data

Hortonworks Technical Preview for Apache Slider View for Ambari Released: May 30, 2014

Revision: 1.6.1TP.003

Architecting the Future of Big Data

© 2014 Hortonworks Inc. All Rights Reserved.

Welcome to Hortonworks Slider View for Ambari Technical Preview. This Technical Preview provides early access to upcoming features, letting you test and review during the development process. These features are considered under development. These features are not intended for use in your production systems and are not supported by Hortonworks but your feedback is greatly appreciated. Have fun and please send feedback on the Hortonworks Community forums http://hortonworks.com/community/forums/forum/slider/

Hortonworks Inc.

I. Introduction .......................................................................................................................... 4

II. System Requirements ....................................................................................................... 4 Operating Systems ............................................................................................................................ 4

III. Download and Install Ambari ....................................................................................... 5

IV. Download and Install Slider .......................................................................................... 6 V. Download and Install Slider View ................................................................................. 7

VI. Using the Slider View ....................................................................................................... 8 Accessing the View ........................................................................................................................... 8 Configuring Ganglia Metrics for Slider Apps ........................................................................... 9 Creating a Slider App ..................................................................................................................... 10 Viewing the Slider App .................................................................................................................. 12

VII. Further Reading ............................................................................................................. 13

Hortonworks Inc.

I. Introduction The Apache Slider View for Apache Ambari Technical Preview provides a way to deploy and manage YARN applications that use the Apache Slider from Apache Ambari. For background: Apache Slider is a technology that simplifies the deployment and management of applications that run on YARN. Slider eliminates the need for developing custom YARN Application Masters, to deploy and manage your applications on YARN. Apache Ambari is a framework for installing, managing, and monitoring Apache Hadoop clusters. Ambari exposes a robust set of REST APIs and a rich Web interface for cluster management. Ambari Views is a new Ambari capability (currently being developed in the Ambari community) to enable the extension of the Ambari Web interface via “Views”. The Slider View for Ambari uses this capability to deliver an integrated experience for deploying and managing Slider apps from Ambari Web.

II. System Requirements The Slider View for Ambari has the following minimum system requirements:

• Apache Ambari 1.6.1 Technical Preview • Apache Slider 0.30 Technical Preview • Hortonworks Data Platform 2.1 • Required Services: HDFS, YARN, ZooKeeper, Ganglia • Oracle JDK 1.7 (64-bit)

Operating Systems

• 64-bit Red Hat Enterprise Linux (RHEL) 6 • 64-bit CentOS 6 • 64-bit Oracle Linux 6

Hortonworks Inc.

III. Download and Install Ambari 1. Download the Ambari 1.6.1 Technical Preview repositories. Ambari Repository Configuration File: http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.6.1.tech/ambari.repo Ambari Repository Tarball: http://public-repo-1.hortonworks.com/ambari/centos6/ambari-1.6.1.tech-centos6.tar.gz 2. Setup and install Ambari. You can follow the Ambari 1.6.0 instructions found below and use the Ambari 1.6.1 Technical Preview repositories from Step 1. http://docs.hortonworks.com/HDPDocuments/Ambari-1.6.0.0/index.html 3. Launch the Ambari Cluster Install Wizard and install an HDP 2.1 cluster. Be sure to include HDFS, YARN, ZooKeeper and Ganglia services at a minimum.

http://ambari.server:8080

Hortonworks Inc.

IV. Download and Install Slider 1. Download and install the Slider 0.30 Technical Preview per instructions.

http://public-repo-1.hortonworks.com/slider/0.30/HortonworksTechnicalPreview-Slider.pdf

2. On the Ambari Server, create an /apps directory. cd /var/lib/ambari-server/resources mkdir apps cd apps

3. Download the Sample Application Packages included in the Slider Tech Preview

and place on the Ambari Server in the apps folder to make available to Ambari and the Slider View. Apache HBase (0.96.0): wget http://public-repo-1.hortonworks.com/slider/0.30/apps/hbase_v096.zip Apache Storm (0.9.1): wget http://public-repo-1.hortonworks.com/slider/0.30/apps/storm_v091.zip Apache Accumulo (1.5.1): wget http://public-repo-1.hortonworks.com/slider/0.30/apps/accumulo_v151.zip

4. Load the Sample Application Packages into HDFS to make available to YARN. su hdfs hdfs dfs -copyFromLocal {sample.application.package} /slider

5. Restart Ambari Server.

Hortonworks Inc.

V. Download and Install Slider View 1. On the Ambari Server, create a /views directory.

cd /var/lib/ambari-server/resources mkdir views cd views

2. Download the Slider View package to the /views directory. wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.6.1.tech/slider-1.6.1.7.jar 3. Restart Ambari Server to install the View.

ambari-server restart

4. Tail the Ambari Server log to watch progress as the view is installed. tail -f /var/log/ambari-server/ambari-server.log

5. Confirm the Slider View is available by browsing the Ambari REST API.

GET /api/v1/views/SLIDER

{ "href" : "http://ambari.server:8080/api/v1/views/SLIDER", "ViewInfo" : { "view_name" : "SLIDER" }, "versions" : [ { "href" : "http://ambari.server:8080/api/v1/views/SLIDER/versions/0.1.0", "ViewVersionInfo" : { "version" : "0.1.0", "view_name" : "SLIDER" } } ] }

6. Confirm the Slider View instance is available by browsing the Ambari REST API.

GET /api/v1/views/SLIDER/versions/0.1.0

{ "href" : "http://ambari.server:8080/api/v1/views/SLIDER/versions/0.1.0/", "ViewVersionInfo" : { "archive" : "/var/lib/ambari-server/resources/views/work/SLIDER{0.1.0}", "label" : "Slider Apps View", "parameters" : [ ], "version" : "0.1.0", "view_name" : "SLIDER" },

"instances" : [ { "href":"http://ambari.server:8080/api/v1/views/SLIDER/versions/0.1.0/instances/Slider", "ViewInstanceInfo" : { "instance_name" : "Slider", "version" : "0.1.0",

Hortonworks Inc.

"view_name" : "SLIDER" } } ] }

VI. Using the Slider View Accessing the View

1. Browse to Ambari Server

http://ambari.server:8080 2. Log into Ambari Web.

3. Select the “Views” control icon in the upper right, select “Slider”.

4. The Slider View will be presented.

Hortonworks Inc.

Configuring Ganglia Metrics for Slider Apps IMPORTANT: Be sure to install the Ganglia Service in your cluster if you want to configure the Slider apps to produce metrics.

1. Ambari can be configured to run named Ganglia Monitors for Slider app metrics.

The named Monitors will be Application1, Application2 and Application3. Use the Ambari configs.sh script to help set the global configuration.

ID Port Host

Application1 8667 Runs on the Ganglia Server host



2. Get the global configuration by using Ambari configs.sh script.

cd /var/lib/ambari-server/resources/scripts/ ./configs.sh get localhost MyCluster global global.txt

where MyCluster is the name of your cluster

3. Edit global.txt and add the follow property to enable the named Ganglia

Monitors for Slider app metrics.

"enabled_app_servers" : "Application1,Application2,Application3" 4. Set the global configuration.

./configs.sh set localhost MyCluster global global.txt

5. From Ambari Web, browse to Services > Ganglia and restart the Ganglia Service

6. You can confirm the named Ganglia Monitors are running with the following:

Hortonworks Inc.

ps -ef | grep hdp/Application nobody 13137 1 1 01:06 ? 00:00:04 /usr/sbin/gmond --

conf=/etc/ganglia/hdp/Application2/gmond.core.conf --pid-file=/var/run/ganglia/hdp/Application2/gmond.pid nobody 13208 1 1 01:06 ? 00:00:04 /usr/sbin/gmond --

conf=/etc/ganglia/hdp/Application3/gmond.core.conf --pid-file=/var/run/ganglia/hdp/Application3/gmond.pid nobody 13253 1 1 01:06 ? 00:00:04 /usr/sbin/gmond --

conf=/etc/ganglia/hdp/Application1/gmond.core.conf --pid-file=/var/run/ganglia/hdp/Application1/gmond.pid

IMPORTANT: When you create your application, review the appConfig and be sure to configure the application to emit metrics to the Ganglia Monitors. appConfig must be edited to point to the correct Ganglia Server and named Ganglia Monitor ID. For example, for the HBase package: "site.global.ganglia_server_host": <host name of the Ganglia Server to emit metrics to> "site.global.ganglia_server_port": <port at which the Ganglia Server is listening> "site.global.ganglia_server_id": <name of the Ganglia Server ID> For example, for the Storm package: "site.global.ganglia_server_host": <host name of the Ganglia Server to emit metrics to> "site.global.ganglia_server_port": <port at which the Ganglia Server is listening> "site.global.ganglia_server_id": <name of the Ganglia Server ID> "site.global.ganglia_enabled":"true", In addition, "site.storm-site.nimbus.childopts", "site.storm-site.worker.childopts", and "site.storm-site.supervisor.childopts" should be configured with Ganglia host and port.

Creating a Slider App IMPORTANT: Be sure to have installed the Apache Slider Technical Preview (as described in the Download and Install Slider section above) prior to attempting to create your first app.

IMPORTANT: Be sure to review the HDFS path settings and be sure to set the Ganglia properties for metrics in Step 4 below.

1. Click the “Create App” button to launch the wizard.

2. Select from the available Slider Apps. This is the list of Sample Applications deployed in the previous step. Give your App a name.

Hortonworks Inc.

3. Configure number of component instances and allocate YARN resources.

4. Review the default Slider appConfig details.

IMPORTANT: PREPARE HDFS FOR YOUR APPLICATION AND BE SURE TO REVIEW THE HDFS PATH SETTINGS IN THE DEFAULT appConfig. Before creating a Slider app, be sure to prepare HDFS for your application. Some apps such as HBase and Accumulo will need HDFS setup. The appConfig for the apps defines the HDFS path that you must create. For example, "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data" in HBase configures the HDFS path for HBase instance. Ensure that “/apps/hbase” exists in HDFS and has appropriate permission for user “yarn”. If you create multiple app instances of HBase, then ensure that a unique HDFS path is provided for each app instance. For HBase and Accumulo apps:

su hdfs

Hortonworks Inc.

hdfs dfs -‐mkdir /apps hdfs dfs -‐mkdir /apps/hbase hdfs dfs -‐chown yarn:hdfs /apps/hbase hdfs dfs -‐mkdir /apps/accumulo hdfs dfs -‐chown yarn:hdfs /apps/accumulo

5. Review the configuration and deploy.

Viewing the Slider App 1. Select the Application Name of your App to browse to the App page.

2. View a summary of the app including status and location of various app

components. Manage the application lifecycle (e.g. freeze, thaw, destroy) from the Actions menu. Quick Link to YARN and application web pages, such as the YARN application page and the HBase Master (in the case of the HBase app).

3. If you have configured Ganglia for the app, you can view app metrics.

Hortonworks Inc.

VII. Further Reading

• Learn more about Apache Ambari here: http://ambari.apache.org/

• Learn more about Ambari Views here:

https://cwiki.apache.org/confluence/display/AMBARI/Views

• Learn more about the Slider View here: https://github.com/apache/ambari/tree/branch-1.6.0.slider/contrib/views/slider

• Learn more about Apache Slider here:

http://slider.incubator.apache.org/

About Hortonworks Hortonworks is a leading commercial vendor of Apache Hadoop, the preeminent open source platform for storing, managing and analyzing big data. Hortonworks Data Platform provides an open and stable foundation for enterprises and a growing ecosystem to build and deploy big data solutions. Hortonworks is the trusted source for information on Hadoop, and together with the Apache community, Hortonworks is making Hadoop easier to install, manage and use. Hortonworks provides technical support, training & certification programs for enterprises, systems integrators & technology vendors.

3460 W. Bayshore Rd. Palo Alto, CA 94303 USA

US: 1.855.846.7866 International: 1.408.916.4121 www.hortonworks.com

Hortonworks Technical Preview for Apache Slider View for...

Documents

Transcript of Hortonworks Technical Preview for Apache Slider View for...