Post on 31-Dec-2015
Tasks within WP4&
The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasks
Jussi Kaurola, Niilo Siljamo, Antti ArolaOzone and UV-Radiation Research Group
Finnish meteorological Institute
Contributions: Tim Martin (UG), Brian Gardiner&Peter Kirsch (BAS), Harry Slaper&Peter
den Outer (RIVM), Ola Engelsen (NILU), Martin Stanek (CHMI), Ann Webb&Richard Kift (UMIST),
Massimo Ferraguto (SSF), ...
Tasks within WP4 (deliverables):
4.1. Installation of new hardware, permitting... (FMI)
4.2. Installation of WWW data submission tools (FMI)
4.3. Addition of further Oracle structures for non-spectral and ancillary data (FMI, BAS)
4.4. Provision of web pages to provide search access to the Oracle database and guidance to users (FMI)
4.5. Update and maintain metadata pages for the database, including flag for WMO guidelines (UMIST)
4.6. Testing of and report on all database search and access tools, and publishing of user's guide and manual, with copy made available at web site (UG)
The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasks:
WP1 & WP2
recent changes in the database
screen shots
submission facilities
retrieval facilities
data integrity
hardware issues
plans for the future
FMI tasklist
summary
FMI task in WP1: "RAF for ozone and albedo"
Antti Arola
"The irradiance ratio of snow-covered to snow-free conditions tells how much snow enhances the irradiance on average.
The average albedo RAF in Sodankylä, calculated from 11 years of data, is about 1.19. In other words, snow enhancement is about 19%"
Report 1.4 (11pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/1.4/wp_1_4.pdf
FMI task in WP2: "2.1An evaluation of the impact of atmospheric
variability on the detection of trends in UV"Antti Arola
The developed methodology offers a way of separating the effect of a single factor, e.g., ozone, cloudiness, aerosols or albedo, on the short- and long-term variability of UV, using measurements of spectral irradiance
Report 2.1 (13 pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/2.1/wp2.1_finalreport.pdf
Arola A., K. Lakkala, A. Bais, J. Kaurola, C. Meleti, and P. Taalas, "Factors affecting short- and long-term changes of spectral UV irradiance at two European stations", JGR, doi:10.1029/2003JD003447, 2003.
Recent changes in the database
quality flags:old data flagged "manually"
incoming data flagged automatically
submission:auxiliary data
retrieval of data:flags included in the retrieval of spectra
broadband data
auxiliary data: daily values (hourly values not yet)
better information of recent submissions, enhanced user options for output (CHMI)
Data submission facilitiesspectral and broadband submission has been in operation since 2001
the submission of auxiliary data was delayed due to problems with (old) software
now the submission works also for aux data
"short and friendly guides" for spectral and auxiliary data
spectral guide should be OK (after several iterations)
auxiliary guide was revised two weeks ago
planned changes:
speeding up of submission process
make it work more as a background process
Data retrieval facilitieschanges made since the last meeting
new oracle table: measurement_type(spectral, broadband, ozone, cloud, dursun, snowcover)
broadband data can be retrieved
daily auxiliary can be retrieved (currently not hourly data)
flags included in headers and can be retrieved separately
some fixes of previously reported problems
planned changesfix problems with hourly auxiliary data
new user requirements for WPI design, give us feedback!
spectral retrieval somewhat slow and error-prone (queue problems). A completely new way to make SQL query and consequent data handling is planned
flagging of one spectrum
149263#056 GREEN GREEN Cloudy 3.639e+02 -1.000e+05 8.380e+01 -1.000e+05 -1.000e+05 GREEN -1.200e-02 14 GREEN 1.700e-02 67 GREEN 1.010e-05 6.600e-06 GREEN 1.850e+00 0 0 0 GREEN CLOUDS 2.860e-01 3.207e-01 SOME_VARIATION_SCAN 9.240e+00 9.766e-04 slf00567 /UVROOT/DB/DBroot/no_ntnu_trh/1997/ol752/11010000.gxa 56
Data integrity: duplicates
what is a duplicate? a measured spectrum which has several copies in
the database several different spectra have the same
measurement time
what is an overlapping spectrum? several different spectra have overlapping
measurement time
both types are included in the data retrievals both types need to be removed
Data integrity: duplicates
duplicates caused by errors in submitted dataautomatic identification: very complex task, not possible with current software
manual work is needed
current status (August run by the BAS tool):belsk bre064 128bru jytot 148855
spurious zeroes
during the flagging spectra with only zero values in irradiance column were recorded
41345 spectra found
only 93 cases with sza < 90 degreesat_boku_sonnblick 2at_boku_vienna 1be_iasb_bru 53fi_fmi_sod 2fr_ustl_vda 5gb_umist_reading 11it_jrc_ispra 27no_nilu_andoya 8
Hardware status
the current hardware is becoming old(purchased in 2000 with $40 000)
disks are almost full, Oracle disk at critical levelrecent submissions have been very large
total disk space 106 GB
Oracle data size 56 GB
Flexstor data size 23 GB
CPU power is adequate, flagging might be lagging a bit in the case of large submissions
disk failure on last Thursday, which was handled very well by the FMI IT department
Planned changes
some hardware upgrading is needed:poor mans solution to disk problem:new disks: 4 x 150 GB (4x600 euro)to current disk box (2 for data, and 2 for mirror)
better solution: new disk array (about 15 keuro)and some new disks. Much more tolerant for errors.
best solution: new server (20 keuro), new disk array (15 keuro) and new disks
a short break in service, a longer break in uploading of new data to Oracle
operating system upgrade in November-December causing 2-3 days break
upgrading of e.g. WPI related software (TOMCAT)
Planned changes...
currently everything is in ozone2.fmi.fi server which is outside FMI firewall ("grey area")
an alternative solution:server with Flexstor and WPI in "grey area"
Oracle inside firewall
a better integration of EUVDB Oracle database to FMI other Oracle databases. Benefits:
security
maintenance
even performance?
FMI task list
follow the link on "General Information Page" updated regularly check the "wish list" if you have ideas for new
improvements
Some remarks:-campaign data
manual submission by FMI retrieval from www pages needs planning and
work
Summary
personal opinion: now the functionalities of the database are rather good
some FMI milestones were delayed considerably the whole project time was more or less
development period, which certainly has caused headache to some users
remaining work: hourly auxliary data planned improvements:
smoother submission of large datasets faster retrieval of spectral data improved WPI design
our interest is to continue with the database maintenance. Funding to be discussed later...