Developing eXtensions for HUE

Maksym DoroshenkoLviv Hadoop User Group 19 February 2015

Agenda

● What is HUE?

● Why use it?

● HUE architecture

● HUE features

● Add extensions to HUE

● Aggregates the most common

Hadoop components into single

interface

● Simplify and integrate

● Free and open source

Goal of HUE

Apps

Hive & Pig

Metastore & HDFS

Additional applications

Backend

Python + Django

(2.6+ / 1.4.5)

Frontend

jQuery

Bootstrap

Knockout.js

Stack

View from 30K feet

You

and your

colleagues

Hue, as a "container" web application, sits in between your

Hadoop installation and the browser. It hosts all the Hue Apps,

including the built-in ones, and ones that you may write

yourself.

The HUE Server

Django running on WSGI

container/web server:

● manages the url dispatch

● executes app logic code

● puts together the views

from their templates

● use DB to manage

session data and Hue

apps can use it as well

for their “models”

Some Hue applications run

daemon processes "on the

side". For example, Beeswax

runs a daemon

("beeswax_server") that keeps

track of query states.

Install SDK app to HUE

For example we use CDH 5.3.

1. Go to HUE directory – cd /usr/lib/hue

2. Run "create_desktop_app" to Set up a New Source Tree –

“./build/env/bin/hue create_desktop_app calculator”

3. Install SDK application – “./build/env/bin/python

tools/app_reg/app_reg.py --install calculator --relative-

paths”

The "app_reg.py" tool manages the applications that are installed.

Result of installation

After installing we restart HUE – “sudo service hue restart”

And then visit http://127.0.0.1:8888/calculator/ you should see:

Issue to solve

Suppose we need track history of workflow

and for that we need some dashboard with

specific logic which we have not in HUE.

Create Django application

setup(

name="dashboard",

version="0.1",

description='Dashboard',

author='Maksym Doroshenko',

packages=find_packages('src'),

package_dir={'': 'src'},

install_requires=['setuptools', 'desktop'],

entry_points={'desktop.sdk.application': 'dashboard=dashboard'},

package_data={

'ctc_sla_dashboard': expand_package_data(

["src/dashboard/templates", "src/dashboard/static"],

"src/dashboard/")

}

)

Walk-through of a Django view

How add your application to HUE ?

As the previous one :)

1. Copy your application to HUE directory

(or use path to application in next step)

2. Install application –

“./build/env/bin/python

tools/app_reg/app_reg.py --

install dashboard --relative-

paths”

Our results

Dashboard example

Accessing HDFS

The “webhdfs.py” module give the file system object that exposes

operations that manipulate HDFS.

It is pre-configured to access HDFS as the user that's currently logged in.

Operations available on FS object are similar to the file operations typically

available in python.

The list of functions available is as follows:

● chmod, chown

● exists

● isdir, isfile

● listdir (and listdir_stats)

● mkdir

● open (which exposes a file-like object with read(), write(), seek() methods)

● remove, rmdir, rmtree

Web apps in any language

To create a new app:

● sudo build/env/bin/hue create_proxy_app Tableau

https://online.tableausoftware.com/

● sudo tools/app_reg/app_reg.py --install Tableau --relative-paths

If you want to update the url later, change it in the ini:

[Tableau]

url=http://new_url

Show time

Q&A