Gfdnavi: its design and implementation with Ajax and Ruby-on-Rails
description
Transcript of Gfdnavi: its design and implementation with Ajax and Ruby-on-Rails
Gfdnavi:its design and implementation
with Ajax and Ruby-on-Rails
Seiya NISHIZAWA,Takeshi HORINOUCHI,Chiemi WATANABE,
T. KOSHIRO, A. TOMOBAYASHI, S. OTSUKA,Y. MORIKAWA, Y.-Y. HAYASHI, M. SHIOTANI, and
GFD-Dennou Davis project
Introduction
• What’s “Gfdnavi”– A tool to archive, share, distribute, analyze, and
visualize geophysical fluid data and knowledge– desktop use to data provide server– fundamental technologies
• Ruby on Rails• GPhys
An introduction was done by T. Horinouchi yesterday.
Ruby on Rails
• an open source web application framework• written in Ruby• Model-View-Controller (MVC) architecture• Convention over Configuration (CoC)• Don't repeat yourself (DRY)
• swift development– ActiveRecord– helper methods (HTML, JavaScript, ajax)
ActiveRecord (AR)• a part of Rails products• a ruby implementation of the object-relational
mapping pattern
• Do not need to use SQL– For better performance, SQL can be used on AR.
node = Node.find(:first, :conditions=>[”id=?”,2]) # SELECT * FROM nodes WHERE id=2 LIMIT 1;path = node.path #=> “/samples”kas = node.keyword_attributes # SELECT * FROM keyword_attributes WHERE node_id=2;
id path parent
1 / null
2 /samples 1
id node_id keyword value
1 2 description sample directory
2 2 notice just sumple
nodes table keyword_attribute table
Metadata DB
used for search
1. name-value attributes2. geospatial- and time-coordinate information3. owner, groups and access mode4. link among data5. time-stamp, size, etc
1. Name-Value attributes• attributes in data file (self-describing files)
– unified access to attributes in differently formatted files with GPhys
• attributes in text file– YAML format– any name-value attributes
gphys_nc = GPhys::IO.open(“fname.nc”,”T”) # NetCDFgphys_nc.att_names #=> [“long_name”, …]gphys_nc.get_att(“standard_name”) #=> “air_temperature”
gphys_grib = GPhys::IO.open(“fname.grib”, “TMP”) # GRIBgphys_grib .att_names #=> [“long_name”, …]gphys_grib.get_att(“standard_name”) #=> ”air_temperatrue”
description: NCEP/NCAR reanalysisgfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB
YAML
• a human-readable data serialization format– easier to read/write than XML
puts “Array (list)”ary = [0,1,2]puts ary.to_yaml
puts “\nHash (associative array)”hash = {“key0”=>”value0”, ”key1”=>”value1”}puts hash.to_yaml
Array (list)---- 0- 1- 2
Hash (associative array)---key1: value0key0: value1
Back
2. Geospatial- and time-coordinate information– spatial region
• rectangle in longitude-latitude section
– temporal region• start time and end time
global, regional, or point swath
3. Owner, Groups, and Access mode– permission system like i-node
• readable and writable for groups and others
– Multiple groups are allowed.
4. Link among data– e.g. This data was calculated from these variables
Directory tree structure• nodes in the tree structure
– node types: directories, variables, images, knowledges, etc
• Each node can have some metadata.– inherited to children nodes
local or opendap directoriesdata files
variables
image files
attributesdescription = “……..”owner = userAgroups = [groupA,groupB]
description = “……..”param1 = value1param2 = [value21,value22]
attributesgroups
virtual aggregated files
fork a child process
Analysis model
open data file(s)
calculate draw
NetCDF PNG
analysis
draw
storage
GPhysDB
cached get cache
output output
cache
yes
no
programmable
Analysis and Visualization
• Analysis model (Analysis class)– all the parameters for analysis or visualization
• the form in the analysis page• instance variables of the Analysis object
It is able to construct one from the other• enable to reconstruct the analysis page from
• drawn image• history list
• Draw method and analysis function are not hard-coded.– Their definitions are in YAML files (editable)
• one method in one file
:name: spectrum:description: |FFT|^2 along a specific dimension:nvars: 1:script: | [gphys0.fft(*arg0).abs ** 2]
:arguments:- :description: the dimensions for spectrum :value_type: array_string :default: []
spectrum.yml
simple coding due to GPhys
can create and modifyvia web-browser
• examples of original draw methods in a Gfdnavi server providing an ensemble forecast data
User Interface
• bottleneck of network application– network bandwidth– machine power and system load of the server
• better usability– ajax
• Rails has many helper methods to write HTML and JavaScript to use ajax.
– cache
• Animation
Web service• local programming• cross-site use
– other Gfdnavi servers– non-Gfdnavi servers
SOAPAPIs for all the analysis functions and draw methods
use the Analysis classWSDL
(REST)
Summary• Metadata and Directory tree structure
– attributes in self-describing data files and YAML files– inheritanceunified access to attributes with GPhyseasy and swift development with AcriveRecord (Rails)
• Analysis/Visualization– programmable (with text editor or web-browser)easy and extensible coding with GPhys
• User Interface– good usability with ajax and cacheeasy development with helper methods (Rails)
Thank you
MVC architecture• Model
– Relational Database• ActiveRecord: a ruby object that wraps a row in a
database table
• View– HTML
• ERuby: a templating system that embeds Ruby into a text document
– JavaScript• ajax
• Controller
Web server
Web server
G
Controller
Model
ActiveRecord, VizShot
View
ERubyhelper methods
User
Web Browser
RDB
create, update, deleteanalysis, draw
request
HTMLJavaScript
Gfdnavi server
GPhysDCL
ActiveRecord (AR)• a part of Rails products• a ruby implementation of the object-relational
mapping pattern– a row in a table ↔ an instance object– a column ↔ an instance method– a reference between tables ↔ an instance method
part = Part.newpart.name = “Sample part”part.price = 123.45part.save
INSERT INTO parts (name, price) \ values (‘Sample part’, 123.45);
name = “gearbox”part = Part.find(:first, :conditions=>[“name=?”,name])
SELECT * FROM parts \ WHERE name=‘gearbox’ LIMIT 1;
RDB tables
• Directory tree structure– nodes (basic unit of a tree)
• directories, variables, images, knowledges, etc.
• Meta data– keyword_attributes– spatial_and_temporal_attributes– node_relations
• User’s information– users– groups
• instance method is used for reference between two tables with foreign key
class Cart < AcriveRecord::Base has_many :itemsend
class Item < AcriveRecord::Base belongs_to :cartend
id cart_id name amount
1 1 orange 3
2 1 apple 2
id user
1 userA
cart = Cart.find_by_user(“userA”) #=> cart whose # user = “userA”items = cart.items #=> array of # items whose cart_id = 1items[0].name #=> “orenge”items[1].amount #=> 2
carts items
Directory tree structure• Tree structure is good way to organize data
/…/data_top_dir/ncep.yml /ncep/UV.jan.nc (contains U and V) /UV_500hPa_vect.png /UV.jan.nc.SIGEN /UV_500hPa_vect.png.yml
id path name parent node_type
1 / / null directory
2 /ncep ncep 1 directory
3 /ncep/UV.jan.nc UV.jan.nc 2 directory
4 /ncep/UV.jan.nc/U U 3 variable
5 /ncep/UV.jan.nc/V V 3 variable
6 /ncep/UV_500hPa_vect.png UV_500hPa_vect.png 3 image
• acts_as_tree method of AR– “parent” column is assumed to have parent’s “id”– “parent” method return the parent node– “children” method return the array of children nodes
class Node < AcriveRecord::Base acts_as_treeend
node = Node.find(3) #=> node whose id=3node.children #=> array of nodes # whose parent column is 3node.parent #=> node whose id is value of parent column
• two types of meta data1. meta data in data file (self-describing file)
• NetCDF: variable attributes
variables: float U(level,lat,lon) ; U:long_name = “zonal velocity“ ; U:units = “m/s” ; U:missing_value = -999.f ; U:standard_name = “eastward_wind” ; U:statistic = “Long Term Mean\n” ; float V(level,lat,lon) ; V:long_name = “meridional velocity” ; V:units = “m/s” ; V:missing_value = -999.f ; V:standard_name = “northward wind” ; V:statistic = “Long Term Mean\n” ; float lon(lon) ; : snip
2. meta data in YAML and SIGEN files• ncep.yml
• UV_500hPa_vect.png.yml
description: vector plot of horizontal velocity at 500hPagfdnavi: owner: user2 references: - /ncep/UV.jan.nc/U - /ncep/UV.jan.nc/V other_mode: 0 rgroups: - groupA
description: NCEP/NCAR reanalysisgfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB
• keyword_attributes tableid node_id keyword num_value value
1 2 description NCEP/NCAR reanalysis
2 3 description horizontal velocity in January
3 4 long_name zonal velocity
4 4 units m/s
5 4 missing_value -999.0
6 4 standard_name
eastward wind
7 4 statistic Long Term Mean\n
8 5 long_name meridional velocity
9 5 units m/s
10 5 missing_value -999.0
11 5 standard_name
northward wind
12 5 statistic Long Term Mean\n
13 6 description vector plot of horizontal velocity at 500hPa
• spatial_and_temporal_attributes table
• node_relations table– UV_500hPa_vect.png (node id = 6) was drawn
from U (node id=4) and V (node id = 5)
id reference referenced_by title
1 4 6 drawn
2 5 6 drawn
id node_id lon_lb lon_rt lat_lb lat_rt starttime endtime
1 4 0 360 -90 90 2008-01-01-00:00 2008-01-31-18:00
2 5 0 360 -90 90 2008-01-01-00:00 2008-01-31-18:00
• user and groups– root, user1(groupA,groupB), user2(groupA), user3
• access permission– owner
• root: “/”, “/ncep”• user1: “/ncep/UV.jan.nc”, “/ncep/UV.jan.nc/(U|V)”• user2: “/ncep/UV_500hPa_vect.png”
– permission for reading• everyone: “/”• groupA and groupB: “/ncep“, “/ncep/UV.jan.nc/(U|V)”• groupA: “/ncep/UV_500hPa_vect.png”
• users and groups tables
• nodes table
id name owner
1 groupA user1
2 groupB user2
id name groups super_user
1 root 00000000 true
2 user1 00000011 false
3 user2 00000001 false
4 user3 00000000 false
id path owner other_mode rgroups wgroups
1 / 1 4 null null
2 /ncep 1 0 00000011 null
3 /ncep/UV.jan.nc 2 4 null null
4 /ncep/UV.jan.nc/U 2 4 null null
5 /ncep/UV.jan.nc/V 2 4 null null
6 /ncep/UV_500hPa_vect.png 3 0 00000001 null
The first bit is the flag for the group of id=1 (groupA),the second one is for that of id=2 (groupB),and so on.
The first bit is the flag for the group of id=1 (groupA),the second one is for that of id=2 (groupB),and so on.
They seem to be permitted to be accessed by anyone,but it is not so, because the permission for “/ncep” is inherited.They seem to be permitted to be accessed by anyone,but it is not so, because the permission for “/ncep” is inherited.
2: writable, 4: readable, 6:both2: writable, 4: readable, 6:both
• Permission of a node depends on all its ancestors.– calculated in advance– saved at additional Boolean-type columns– In order to keep consistency, callbacks can be
added to events (e.g. after saving) with AR.
• When permission is changed, the columns of its children are recalculated.– Using ActiveRecord, adding such callbacks to events
(e.g. after saving) is quite easy.
class Node < ActiveRecord::Base
after_save calculate_children_permissions # add a callback method
def calculate_permissions # calculation of permission other_readable = (other_mode & 4) & parent.other_readable groups_readable = rgroups & parent.groups_readable save! # save to database table end
def calculate_children_permissions # callback method children.each{|child| child.calculate_permissions } endend
ER diagram of RDB tables
(only important tables and columns are shown)
Registering script
• Directory tree structure on a storage is registered to the database table as it is.– The top directory to be registered can be
specified.
• Metadata contained in data files are registered to the database table.
• Any other metadata can be registered by writing in Yaml and SIGEN files.
Analysis and Visualization• GPhys
– IO handling– data slice– mathematical calculation
• DCL (Dennou-Club Library)– plot
• (Cache)– drawn plots are cached to DB
• Directory tree view
click
ajax request to get children directories
The ajax request will not be called again.Opening and closing the directory is doneby changing “display” attribute of CSS.
RESTful
• Representational state transfer• Uniform interface
– All resource share a uniform interface like HTTP methods such as GET, POST, PUT, and DELETE.
• Resource Oriented Architecture– Every resources are uniquely addressable using URI.
• Statelessness– Each request is treated independently.
improvable– still some separate codes
• web application, web service
– SOAP ignore HTTP methods.• Some cache systems could not work well.
– Programming with SOAP tend to be complicated.
“RESTful web service + client(s)”– one unified code
• Multiple output formats are still supported (HTML, XML).
– acts as definition of HTTP methods• cache system friendly
– simpler local programming
RESTful webservice• URI syntax
– variable• http://host/…/ncep/UV.jan.nc/U.(html|xml)• http://host…/ncep/UV.jan.nc/children.(html|xml) # U and
V
– analysis• http://host/ .../{variables}/analysis/{function}.(html|xml)?{options}
– drawing• http://host/ .../{variables}/draw/{draw method}.(html|xml)?{options}
clients• flex
– ActionScript– (AIR)
• (HTML + JavaScript)• (desktop ruby client)• another gfdnavi server
UI with ajax
• asynchronous JavaScript and XML (originally)– not only JavaScript and XML, but also other pairs
• update a part of the existing page– better usability– reduce bandwidth usage
• Rails has many helper methods to write HTML and JavaScript to use ajax.
click
get node list of the directory vie ajax call
• Analysis page
check
get dimension’s information with ajax
click
send valuesin the formwith ajax
• Animation– the 1st loop
• ajax requests (one request for one image)• may take a little long time
– the later loops• cached image (cached as JavaScript object)