Data Warehouse
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
The MeteoSwiss Data Warehouse
Real-time access to data and metadata – basis for the WIS-implementation in Switzerland
WMO RA VI Workshop on the implementation of WISSofia, 1-3 November 2011
Stephan Vogt, Meteorological Data Coordination, MeteoSwiss
2 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Agenda
• Situation in 1999• Towards the MeteoSwiss Data Warehouse• Metadata in the DWH• Ongoing improvements• Considerations on the WIS-implementation
3 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Situation in 1999 – and the visionYear 2000 Problem: maximum credible data accident to be expected?
4 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Objectives of the MeteoSwiss Data Warehouse
• Implementation of a central, integrated data platform for the collection, storage and processing of meteorological data and its metadata.
• Storage of metadata in one central metadata repository
• Subdivision in a processing and an application area
• Creating a basis for a customized access to data respecting official standards for internal and external clients
5 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
2000
2001
2003 Phenology & Pollen
2002 Surface data (rain gauge measurements, AWS data, observations)
2004 Aviation messages
2005 Lightning data
2006 International surface data (SYNOP)
2007 National sounding measurements
2008 Homogenized Data
2009 National & International Upper Air Data
2010 Gridded Datasets
Start of the MeteoSwiss Data Warehouse Project
Model Data2011
Metadata
6 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Conceptual Architecture
Dat
a so
urc
esS
tagi
ng
area
D
ata
stor
age
Database for point data analysis
Calc, Check, Data Transformation & Integration
Data delivery and products (on-/offline)
Weather Forecast
Data for the Public Internet
Durable data archive
Database for point data processing
(incl. raw data repository)
Light-ningdata
RadarClimaAnalysis Tools
Model Validation Research ClimatologyClimate Information Service
Dat
a an
alys
is
Autom.Observ.systems
UpperAir data
Aviationdata
Geodatabase(raster data, ...)
Data analysis & Interfaces (API, DB Tools, Web access, GIS, R …)
VERACMSAFNWP
TL
PartnerObserv.systems
QC
Met
a D
ata
Obser-vations
Met
adat
a M
anag
emen
t Too
ls
7 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Metadata in the MeteoSwiss DWH
…stored in more than 200 tables in the DWH
8 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Enhanced DataClean Data
customers
Raw Data
level 2data
level 1data
level 3data
level 5data
level 4data
quality control II calculation of
derived quantities
data collection quality control I
measurement
simulation
metadata
Raw data repositor
y
Data repositor
y for external
data
applications for data distribution
data
sou
rce
quality control III errorr correction
extreme value messages
imm
ed
iate
ly
days
years
exte
rnal
data
sou
rce
applications for data processing
immediately
durable data
archive
MeteoSwiss data archives
durable data archives outside MeteoSwiss
MeteoSwiss data sources
data sources outside MeteoSwiss
data transmission and distribution
Legend
Level 1 measurement data
Level 2 raw data
Level 3 cleaned data
Level 4 corrected data
Level 5 homogenized data
homogenization
Instrument Lifecycle
Spatial reference
Data import
AggregationcalculationContract
data
Station history
Homo-genization
Quality control
Parameters
9 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Development cycle & metadata management
Development
Deployment
Production
Failover Production
Master Metadaten Repository
10 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
How are the metadata accessed?
• DWH-Webtool with user guidance for all important use cases
• MS ACCESS for all other casesKontext-
datenbankDWHPROD
Kontext-datenbankDWHFOPR
Kontext-datenbank
PMETAREP
Intranet
WebLogic
KDB-Tool 2
SecPro
Client Internet Browser
Benutzerinteraktion
Zugriff Security Provider
Zugriff Kontextdaten
Active Directory(LDAP)
11 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Look and Feel of DWH-Webtools
12 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Advantages of the Data Warehouse
• measurement data• metadata • control data• quality tests
Allows an efficient and economic maintenance
company-wide data integration platform for the long term
independant from technical realisation
specified in a uniform way, centrally administrated and configurable
13 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Lessons learned
Do‘s Dont‘s
Listen to the data; conduct broad-based reviews
Listen too much to single data users
Follow a spiral development model, gain experience in operation, grow the infrastructure accordingly
Do everything in one step waterfall procedure
Follow the golden rules of database design
Fix technology too early
Understand the business process Don‘t let consultants talk you into a solution
Provide one attractive application for data analysis from the very beginning
Segregated architectures
14 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Do things the same way all over the organisation data governance
• Data architecture documents• Gremium with representatives from all MeteoSwiss
departements
15 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Not everything is solved...
• RENAISSANCE = Renewal and ENhAncement of the servIceS baSed oN the mEteoswiss data warehouse
• Subprojects:• Data supply of COSMO from DWH• Generic delivery mechanism for data marts• Station history• Consolidation of production tools
16 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Generic delivery mechanism for data marts• Existing delivery mechanisms have been generalized• Towards event-based triggering of processes• Generic delivery mechanism includes
• Measurement data• Metadatabased on XML standard
Primary purpose: Data delivery to GIN (Swiss joint Information Platform for Natural Hazards)
17 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Lots of tools for lots of products…
18 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
…and what about WIS?
• All remarks so far refer to our status as NC• MeteoSwiss has been asked to develop GAWSIS as a
DCPC for WIS • Today GAWSIS is not yet integrated into the DWH• Two options for the next steps:
• solution on the basis of the current GAWSIS Provide the data with an OAI-Service ready for collection by GISC
• Integration of GAWSIS into the DWH, make use of the planned Web Service-infrastructure
19 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Vision regarding WIS
20 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Vision regarding WIS
NC
MeteoSwissData Warehouse
Internet
GISC
XMLMetadata
21 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in SwitzerlandStephan Vogt
Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss
Questions?
Thanks for your attention!
Top Related