Use of OpenEarthTools to make netCDF
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
Extract Transform Load Provide
Subversion repository
2
Contents
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
2a
use tools >
Extract Transform Load Provide
2b
read data
2c
> save as netCDF
OpenEarthTools
• Download and install TortoiseSVN Subversion clienthttp://subversion.tigris.org/
• Make a checkout of OpenEarthTools (full or just Matlab subset)• Start Matlab and go to OpenEarthTools
>> cd F:\checkouts\OpenEarthTools\matlab\• Add OpenEarth paths to Matlab:
>> oetsettings• Inspect OpenEarthTools• When many people work on oen thing, we need a few basic rules• Document your functions and toolboxes
• Exploit the clickable links to files + directories in new Matlab version>> help functionname % in path>> help directoryname % in path• Displays:
> all H1 lines (first comment line in file)> or displays help in contents.m
• Add crosslinks: with 3 crosslinks, 6 clicks connect 729 files ≈ size OpenEarthTools• See also
OpenEarthTools documentation
other directories in path
other m function
www links
>> help applications
OpenEarth applications rijkswaterstaat - data: rijkswaterstaat data types (donar) delft3d - delftd file formats knmi - read knmi wind and meteo timeseries swan - swan toolbox
>> help rijkswaterstaat
donar_read - read ASCII text file from www.waterbase.nlgetwaterbase - get data from waterbase.nlgetwaterbasestation - get stations names for parametergetwaterbaseparameter - get parameter from waterbase.nlhmcz_read - read meteo file from hmcz@ rws
>> help donar_read
DONAR_READ read ASCII text file from www.waterbase.nl
DAT = donar_read(fname,<keyword,value>)
See also: getwaterbase, www.waterbase.nl
% DONAR_READ read ASCII text file from www.waterbase.nl%% DAT = donar_read(fname,<keyword,value>)%% See also: getwaterbase, www.waterbase.nl
OpenEarthTools documentation
H1 line
contents.m
donar_read.m
Contents
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
Extract Transform Load Provide
2a
use tools >
2b
read data
2c
> save as netCDF
Read a raw dataset
• Always read data into one struct
• Separate columns (Excel) into fieldnames• Include as much meta-info as possible in the struct• Use English names• Use SI units• Use documentation: cells are block starting with %% (~ chapters)• Align
Read a raw dataset
• Dedicated matlab tools>> xlsread>> textscan>> csvread
• Dedicated OpenEarthTools• www.waterbase.nl
>> donar_read• http://www.knmi.nl/klimatologie/onderzoeksgegevens/potentiele_wind/
>> KNMI knmi_potwind• http://www.knmi.nl/klimatologie/daggegevens/download.html
>> knmi_etmgeg• www.NODC.nl / SeaDataNet profile data (CTD):
>> odvread• Dedicate yourselves by peeling the file & add to OpenEarthTools:
>> oetnewfun, load_template>> fgetl, fscanf % ascii>> fread % binary
Read a raw data waterbase
Read a raw data KNMI wind
Read a raw data KNMI meteo
Read a raw data NODC ODV
Test with some raw data
Test for yourselve some raw data with dedicated scripts, choose one.Find the associated matlab function using the help documentation.
• Rijkswaterstaatwww.waterbase.nl
• KNMIhttp://www.knmi.nl/klimatologie/onderzoeksgegevens/
potentiele_wind/http://www.knmi.nl/klimatologie/daggegevens/download.html
• NODCwww.NODC.nl / SeaDataNet profile data (CTD):
Inspect a raw dataset
• Different types of raw data• Different structs for time series. Hmm.
> Waterbase > KNMI wind> KNMI meteo
• How to unify?• No guidelines for structs• International guidelines for netCDF!
• Idea: collaborate!• Each BwN participant transforms
raw data they use anyway into netCDF
• Everyone can readily access previously used data in a uniform manner
raw data struct
raw data file
netCDF
No standards available
International standards available
Contents
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
Extract Transform Load Provide
2a
use tools >
2b
read data
2c
> save as netCDF
netCDF
• What is netCDF• How to make netCDF
• Matlab netCDF interface
• Conventions (vocabularies):> Standard meta-info: CF convention +
GNU/CC/?> standard quantities: CF convention> Standard units: Unidata UD units> Standard coordinate codes: EPSG
• Save script in repository
• OpenEarthRawData: for dedicated script
• OpenEarthTools: for more general script• Inspect and plot netCDF
netCDF = linked dimensions, variables, attributes
t =f(t)z = f(z)
# = f(#)
chlorophyll = f(t,z,#)
disclaimer
institute
copyright
long_name
standard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
Self-describing file:• meta-info
never lost• no manual
needed• open file format
guarantees
future access
Let’s look at a basic netCDF file
nc_cf_example2nc.nc (in OpenEarthTools checkout)• >> nc_dump(‘nc_cf_example2nc.nc’)• ncBrowse • HDFview
Hierarchical structure as file system• Attributes• Dimensions• Variables
netCDF (1)
• Any amount of meta-data (Attributes) can be appended. • Handles large datasets (file > 4GB allowed with in 64-bit offset mode). • Scaleable for future applications (divide data over multiple files). • Open source specifications and interfaces.• Fast data storage and retrieval (direct access binary). • Small data storage (binary is small + internal zip option). • Simple yet robust.
“Following examination by 20 reviewers, NASA's Earth Science Data Systems Standards Process Group has concluded that netCDF classic should be adopted as a recommended standard.A major strength of netCDF classic, according to the reviewers, was that it has fostered data interoperability and exchange through its self-describing file format, platform independent architecture, and robust access methods. Additionally, its overall file format and metadata attributes were simple enough to be easily understood and applied yet robust enough to describe and store multidimensional data of different types in the same file” (source: Unidata news)
netCDF (2)
• 2 independent implementations of NetCDF interface (C & Java):
• reliable
• redundant
• Generally used, so there’s always someone who can offer help available.
• netCDF is de facto standard for coastal ocean circulation models (GOTM, ROMS, ECOM-SED and in the near future also Delft3D-FLOW).
• Lots of simple and free tools on the web that operate on NetCDF files:
• ncBrowse (NOAA)
• NCO
• Panoply (NASA)
• ncview (linux)
• Implemented in numerous commercial tools too
• Quickplot (Delft3D)
netCDF (3)
• Interfaces to sophisticated scientific data manipulation languages: • C-interface• Java-interface• Fortran• Python• IDL• Matlab
> Through java-interface (in OET)> mexcdf for C-interface (in OET)> Native matlab interface (since release 2008b)
• Merged with OPeNDAP-interface• Handle netCDF file on web as if it on your own PC.
• Additional, widely accepted conventions for NetCDF• implemented in main NetCDF viewers • CF meta data convention
Creating a netCDF file
• See : nc_cf_example2nc.m• Now step by step walking through• Use nc_cf_example2nc.nc as template when doing it yourselves
>> nc_create_empty % 1
disclaimer
institute
copyright
long_name
standard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
t =f(t)z = f(z)
# = f(#)
chlorophyll = f(t,z,#)
>> nc_create_empty % 0
• Web-sources (waterbase, knmi)
• Read directly from web
• Or download to local cache• Your own data:
• Save raw data in repository
OpenEarthRawData with
• INSPIRE XML meta-data• Then read data in Matlab• and create empty netCDF file
>> nc_attput % 1
disclaimer
institute
copyright
long_name
standard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
t =f(t)z = f(z)
# = f(#)
chlorophyll = f(t,z,#)
>> nc_attput % 1
• See CF meta-data conventions: http://cf-pcmdi.llnl.gov/• OpenEarth conventions:
• disclaimer + terms_for_use
• $Headurl$
>> nc_add_dimension % 2
disclaimer
institute
copyright
long_namestandard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
chlorophyll = f(t,z,#)
t =f(t)z = f(z)
# = f(#)
>> nc_add_dimension % 2
>> nc_addvar % 3
disclaimer
institute
copyright
long_name
standard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
chlorophyll = f(t,z,#)
t =f(t)z = f(z)
# = f(#)
>> nc_addvar % 3
CF convention
vocabulary
of
quantities
>> nc_varput % 4
>> nc_varput % 4
UNIDATA UD units package
vocabulary
of
units
>> nc_varput % 4
disclaimer
institute
copyright
long_name
standard_name
units
long_name
standard_name
units
long_name
units
long_name
standard_name
unitsconventions
title
website
t
#
z
chlorophyll = f(t,z,#)
t =f(t)z = f(z)
# = f(#)
>> nc_varput % 4
>> nc_dump % 5
disclaimer
institute
copyright
conventions
title
website
t
#
long_name
standard_name
units
z
long_namestandard_name
units
long_name
standard_name
units
long_name
units
chlorophyll = f(t,z,#)
t =f(t)z = f(z)
# = f(#)
>> nc_dump % 5
>> nc_dump % 5
Plot netCDF in free ncBrowse
Plot netCDF in free ncBrowse
Plot netCDF in Matlab
Plot netCDF in ArcGis
ArcGis also reads and writes netcdf files.
ArcGis also reads and writes netcdf files.
ArcGIS
Check netCDF in your own text editor (xml)
xml representation of a netcdf file
xml representation of a netcdf file
XML
Check netCDF in …
NCO#diffncdiff -v time file1.nc file2.nc#compression & packingncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% lossy compression)#selecting variables by regexncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc.
IDVVery usefulVery useful
Web hyperslabs, cool!Web hyperslabs, cool!
Not so stable.Not so stable.
JAVA netCDF toolsUsefulUseful
Summary current session
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
2a
use tools >
Extract Transform Load Provide
2b
read data
2c
> save as netCDF
Next session: using netCDF from others
1
3 D:\...
3 http://…
tools
models
add meta information
netCDF on web server
transform to netCDF
netCDF on OPeNDAP
server
data
data providers = data users
Extract Transform Load Provide
2
Top Related