DAP Servers and Services

Section 2

APAC ‘07 OPeNDAP Workshop

12 Oct 2007

James GallagherThanks to Jennifer Adams, John Caron,

Roberto De Almeida, Nathan Potter and Patrick West

Outline

• Different DAP Servers

• Hyrax Architecture and customization

• Example Configuration of Hyrax

There are several different DAP servers

• Hyrax, developed by OPeNDAP• TDS, developed by Unidata• GDS, developed by COLA• PyDAP, developed by Roberto De Almeida• Others we won’t cover: Dapper; FDS;

CODAR; OPeNDAP Server3; et cetera• Terminology: We often say ‘server’ when

describing what the technical press calls a ‘web application.’

Hyrax

• Supports multiple protocols– Data: DAP using HTTP/GET and HTTP/SOAP;

Direct access (via HTTP); WCS/WFS funded, in development

– Catalog: THREDDS; HTML directories

• Data formats: In binary distribution: NetCDF; HDF4; HDF5; FreeForm; many more available as source code.

• Includes ASCII data dump, HTML data access form, Info metadata page

Hyrax Architecture

• Two (or more) cooperating processes:– Front-end provides DAP interface– Back-end(s) read(s) data

• Both parts can be customized– Front-end: different network protocols– Back-end: different data formats/systems

• N-Tier design is flexible, secure

OLFS BES Data

TDS

• Java Servlet network interface• Supports multiple protocols

– Data: DAP; WCS; NetCDF Subset; Direct access (via HTTP)

– Catalog: THREDDS

• Data formats: NetCDF; HDF5; GRIB-1,2; NEXRAD; DORADE; BUFR; DMSP; GINI; more in development

• Can also read from any other DAP server• Can serve aggregations

THREDDS Data Server HTTP Tomcat Server

Datasets

catalog.xml

motherlode.ucar.edu

THREDDS Server

NetCDF-Javalibrary

Application

IDD Data

•HTTPServer

•NetcdfSubset

•WCS

•OPeNDAP

configCatalog.xml

PyDAP

• Python native implementation from the DAP2 specification; WSGI application

• Server uses run-time loaded modules• Protocols:

– Data: DAP (HTTP/GET); WMS/KML; EditGrid/Google Spreadsheets; JSON

– Catalog: THREDDS using a script/batch system; full support planned

• Formats: CSV; netCDF; SQL Matlab; GRIB; HDF5

PyDAP Server Architecture:Plugins and Responses

GDS

• Java Servlet network interface and C/C++ back-end• Supports multiple protocols:

– Data: DAP (HTTP/GET)– Catalogs: THREDDS

• Formats: GRIB; HDF4; netCDF; BUFR; Binary; GrADS Station

• Can also read from many DAP servers• GDS can perform sophisticated server-side

operations and build interim data sets as a result.• Server-side operations use GrADS command syntax.

GRIB, HDFNetCDF

Binary

datasets in any format supported by GrADS

Result cache

holds temporary data (uploaded, generated by a previous operation, or transferred directly from another server) for use in remote analysis

GrADSbatch mode

interface code

DODS server libraries

Serverperforms analysis

operations

manages sessions, translates dataset

names

Java servlet

supports extended request types for analysis, upload

GrADS Data Server

DAP Servers Summary• Four servers, all developed by different

groups, were described (Hyrax, TDS, PyDAP and GDS)

• Each supports DAP and THREDDS• Each has unique features:

– PyDAP: WMS, KML and python scripting– GDS: Extensive server-side operations– TDS: Aggregation, WCS– Hyrax: Customizable, SOAP interface, WCS/WFS

support funded

• These are more properly called ‘web applications.’

Hyrax Architecture

• Front-end (OLFS) customization

• Authentication & Authorization

• Back-end (BES) customization

• Single/Multiple Machine Installations

• Security

OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

File system with data files,SQL Database, …

DAP2

THREDDS

HTML

Optional THREDDScatalogs

Hyrax Architecture

• The OLFS uses an ordered list of Dispatch Handlers.

• Each handler on the list is asked if it can handle the incoming request.

• The first handler on the list that claims the request gets to handle it.

• List order is important, as some handlers (for example THREDDS) may claim to handle requests that should be handled by a different handler.

• Handlers are free to do anything they need to handle a request: Contact a remote system/process, read files, spawn threads, et cetera.

OLFS Dispatch Handlers

Extending the OLFS

• Extension ‘modules’ written in Java• Added to a directory within Tomcat• The new modules have complete access to

the request information• Both HTTP GET and POST requests• There is some significant processing done

before the handler is called:– Conditional GET Requests– Authorization & Authentication

OLFS and Authorization/Authentication

• Authorization & Authentication (Az/Ac) actually provided by Apache or Tomcat

• Apache/Tomcat provides a range of Az/Ac features:– Realms: Role-based Az/Ac

• Usernames & passwords -> Authorization• Roles -> Authentication• Realm info in RDBs, XML docs, LDAP, …

– TLS/SSL:• Server and client Az/Ac• Secure data transmission

• Tomcat provides single sign-on capabilities – Clients must support cookies– Suitable for portals

BES

Network Protocol andProcess start/stopactivities

Data Store Interfaces

BES Framework

PPT*Initialization/Termination

DAP2Access

NetCDF3 HDF4 FreeForm…

DataCatalogs

Commands**BES Commands/ XML Documents

*PPT: point to point protocol**Some commands are built in, most load at run-time

BES Extensibility

In what ways can you extend the BES?

• New request handlers (data handlers like netcdf, hdf4)

• New response handlers (Cedar's Flat, Tab, Info responses)

• New commands (like our hello world example)

• Initialization/Termination

• Exception Handlers

• Reporters

• Container and Definition storage

• Aggregation engines

• Methods of returning your data (return as netcdf)

Possible Installations

• Both OLFS and BES run on one machine

• OLFS on one machine and BES on another

• One OLFS and several instances of BES on different machines

• OLFS communicating with one or more BESs and other backend processors

Installation Security

• A separate issue from Ac/Az• The BES must be protected:

– With a firewall or– TLS & Client certificates

• Running the OLFS and BES on separate machines limits the scope of a compromise of the OLFS

• Ensure that the BES, Tomcat and Apache all run with limited access to the server host

Hyrax Architecture Summary

• Hyrax can be installed on one machine or several

• Installation security merits serious consideration

• Authentication & Authorization are handled by the web servers (Tomcat and Apache)

Hands on: Hyrax Configuration

• Choices:– Single or Multiple machine– Single or Multiple back-end servers– Tomcat or Apache web server– Data formats– Catalog customization– Security

• Testing– Command line tools for system administrators– Web browser

APAC Workshop Configuration

• Run a single BES and the OLFS on one host (the virtual machine running SLAX Linux)

• Use Tomcat running on port 8080

• Data formats: NetCDF and FreeForm

• No custom THREDDS catalog

• No firewall

APAC Hardware

• SLAX Linux VMware Virtual Appliance

• All the software needed has been built, installed and configured

• Start the virtual machine now…

Starting the Virtual Machine

• Insert the CD-ROM• Drag the opendap_vm and Wintools folders to the Windows

desktop• Copy the ISO image from the CD ROM to the harddisk (in a

command window, mkdir C:\SLAX and then use copy or dd to copy the iso to the new directory).– The dd command is in the Wintools folder– The C:\SLAX directory is coded into the virtual machine; use Vmware

Workstation/Fusion to edit

• Open the opendap_vm folder and double click on the opendap_vm.vmx virtual machine configuration file or start VMware Player/Workstation/Fusion and open opendap_vm.vmx from there.

• The virtual host will boot and the SLAX (slackware Linux with KDE) desktop will appear.

Installing the Software

• Already present on the virtual machine

• But, if it wasn’t, you’d go to the OPeNDAP web site* and download: – Binaries or source for the BES and the

data handlers needed. – Get the OLFS web archive file (which is a

compiled java servlet).– You also need the Tomcat servlet engine

* http://opendap.org/download/hyrax.html

Location of Server Files on the Virtual Machine

• On your virtual machine:– The Tomcat servlet engine is in /usr/local/javadev/apache/tomcat-5.5.12 (this is the value of $CATALINA_HOME)

– The BES, data handlers and related source files are in /usr/local/src. The BES has been built and installed in /usr/local/ ($prefix)

– The OLFS web archive file is $CATALINA_HOME/webapps/opendap.war

– BES: bes.conf, found at $prefix/etc/bes/bes.conf– OLFS: olfs.xml and catalog.xml, found at $CATALINA_HOME/content/opendap

Background: Starting the Server

• Start the BES (back-end data processing component)

• Use bescmdln to verify it’s working• Start Tomcat: This automatically starts

all installed servlets– Servlets are installed by copying the .war

file to the servlet’s webapps directory

• Verify it’s working using a web browser

Start the BES

Verify the BES is running

Start Tomcat & the OLFS

• Typical steps:– Unpack the olfs jar-file– Copy the opendap.war file to Tomcat’s

webapps directory– Start Tomcat

• Since all but the last step has been done already, start Tomcat:

– /usr/local/javadev/apache-tomcat-5.5.12/bin/startup.sh

…terminal view

Verify Tomcat is running

http://localhost:8080

…and Hyrax

http://localhost:8080/opendap

Complete the Configuration

• Steps you would typically perform:– Security: Set up a firewall! Limit access to

port 10002 to this host only – Custom catalogs: Edit the catalog.xml

configuration file. By default automatic catalogs are generated

– Logging: Edit the log4j.xml file. By default all accesses are logged

Stopping Hyrax

First, stop Tomcat using ‘shutdown.sh’

…then stop the BES using ‘besctl’

Hyrax Configuration Summary

• For our chosen configuration, we used only Tomcat, plus the Hyrax web application which consists of the BES and the OLFS

• The tools bescmdln, getdap and a web browser were used to test the installation

• There are a lot of options, but the default settings produce a working server

• Security is a must for a web application; use a firewall to isolate the BES so only the local host can connect to it.

Summary

• There are a number of DAP-compliant servers (Hyrax, TDS, PyDAP and GDS) were described

• All of the servers (read ‘web applications’) actually support several other protocols

• While DAP does nothing to support cataloging data, THREDDS supports just that

• Hyrax is customizable and can be installed in several different configurations