Post on 30-May-2020
3/14/2013
1
Time Series Data Management in
Air Quality Monitoring
AQE 2013, Telford/UK
Edgar Wetzel, 14 March 2013
Brief company presentation
� KISTERS AG
� software company with core competence: RDBMS-based time-series management
� +1.000 B2B and G2B customers world-wide
� +500 employees in offices all around the globe (4 continents)
� 50 years of experience since the company started back in 1963
© KISTERS AG 14/03/2013 - 2 -
3/14/2013
2
Measurement data: short term use
monitoring station
• permanent = continuous in time
• fixed = stable in space
monitoring station
• temporary = discontinuous in time
• mobile = moving in space
measurement data
• <site><date+time><value><flags>
data user
validation +
analysis
agency
regulatory
reporting archiving
multiple measurement data
� a time series!
© KISTERS AG 14/03/2013 - 3 -
multiple software packages
from distinct datalogger
and instrument manufacturers
Measurement data: quantity and variety
� data are measured all over the place
meteorology
ambient air quality
CEMS
concentration
quantity
deposition
NO2
NOX
O3
PM10
PM2,5
CO2
wind speed + direction
°C
© KISTERS AG 14/03/2013 - 4 -
3/14/2013
3
Measurement data: quantity
� monitoring networks consist of numerous stations
back-
ground
traffic industrystation type:
� each stations monitors several meteorological and air quality parameters
� large-scale trends can be determined using trend analysis on long-term data sets
© KISTERS AG 14/03/2013 - 5 -
What happens to the measured data?
� it is processed and ends up in regulatory reports
© KISTERS AG 14/03/2013 - 6 -
3/14/2013
4
What happens to the measured data?
� next it is archived and more often than not ...
... forgotten
© KISTERS AG 14/03/2013 - 7 -
� further use is limited due to ...
� limited availability
� difficult accessibility
� incompatible file formats
� inconsistencies
� unstructured archiving
� unrelated data sets
Each data points has an economic value! -1-
� measuring data costs money
� equipment
� human resources
� maintenance
� archiving data costs money
� albeit prices for data storage are steadily declining
� thus price is no limitation to long-term archiving of mass data!
© KISTERS AG 14/03/2013 - 8 -
3/14/2013
5
Each data points has an economic value! -2-
� there is a scientific and economic value in the information
potentially derived from the data
� information must be derived from the data
� data from different disciplines must be correlated
� communication and warning:
� defects – preventive maintenance – disaster
� early detection of critical situations – transparent information policy
� information quality increases by means of ...
� tried & tested validation procedures
� optimized analysis/evaluation tools
� data correlation algorithms
© KISTERS AG 14/03/2013 - 9 -
Long-term data analysis
� each new measurement point increases simultaneously both
� the economic value of the time series
� the statistical significance of the data pool
� BUT ...
� using many software makes evaluation difficult
� data piles don’t have any statistical or economic value if they are not
� properly organized/structured/managed/documented(meta data)
� accessible
� available
� quality assured
© KISTERS AG 14/03/2013 - 10 -
Data piles need to be managed by a single enterprise solution
software to maximize return on investment!
3/14/2013
6
How to improve?
� Time-Series Data Management Systems
� Enterprise Solution ... RDBMS-based advanced ...
� time-series data management
� time-series data analysis
� multi-tier architecture
� manage in one DB both
� current data (measured & calculated)
� legacy data (measured & calculated)
� assure all of the following functionality:
– centralization,
– validation,
– aggregation,
– processing,
– alerting,
– reporting,
– dissemination, and
– public information.
GUI applications
application server
RDBMS
data base
© KISTERS AG 14/03/2013 - 11 -
AquisNet – in brief
� AquisNet == Air quality information system – NETworks
� software family providing modules for the 2 main blocks of an AQM network:
� Data Center:
– centralization, validation, aggregation, processing, alerting, reporting &
dissemination
– RDBMS-based advanced time-series data management and analysis
� DAS: optional data acquisition system, i.e. the monitoring station software
– sample data from analogue, digital and smart sensors/analyzers
– run function tests and calibration cycles
– plausibility testing and local data storage
© KISTERS AG 14/03/2013 - 12 -
Field-proven software built on 15 years of experience
in managing huge amounts of current and legacy AQM
data!
3/14/2013
7
Air Quality Information System: core functions
Configuration &
Definitions
system widecatalogues/ key list
predefined and userdefinabledata groups
regulatory compliance rules
(EC, US EPA, UBA ...)
data access permissions
& data views
Operational Meta Data Management
sitestation
(measurement location)
parameter(channel)
time series definition
Time Series Data Management
data feeds(import/export)
data validation andediting
calculation of derived time series
data publishing
Time Series Data Analysis
statistics and
air quality indicators
manual or
automated reporting
graphics and
numeric tables
Performance Reporting (KPI)
© KISTERS AG 14/03/2013 - 13 -
What’s a time series for KISTERS?
� any data that is linked to a location and has a date/time stamp
� KISTERS will increase the value of the TS data by means of ...
� data centralization and subsequent validation
� appending quality and other information to the measured value
� creating an auditable track of data versions
� providing an organized data source for other uses
� providing an organized data sink for other data providers
© KISTERS AG 14/03/2013 - 14 -
transaction historycurrent state of
production time series
data quality state ,
changes Y/N +comments
3/14/2013
8
© KISTERS AG 14/03/2013 - 15 -
Typical time series data management workflow
reporting
produce report chosen report format report
processing
aggregate, analyze|station meta data| -> sequence
(<date+time><calculated value><quality flags>)derived data: temporary or permanent
validation
increase quality and set additional QA flags accordingly
|station meta data| -> sequence (<date+time><value><quality flags>)
2nd ... xth level QA checked production data
centralization and storage
receive from station and record in RDBMS|station meta data| -> sequence
(<date+time><value><quality flags>) original measurement data
data transmission
plausibility check <date+time><value><quality flags> 1st level QA checked data
measurement
measure <date+time><value> raw data
Data centralization and storage
� AquisNet SODA - KISTERS’ telemetry unit
� Data collection and communication on all channels
� Internet, phone, GSM & GRPS, radio communication, satellite, web services
� supports both push and pull modes!
� Central soft- and hardware for all data
� Consistent planning and monitoring
� Data security & user management
� Powerful and scalable
3/14/2013
9
Data validation
� AquisNet supports both ...
� fully automated algorithm-based validation
� visual inspection and validation
� All validation steps are ...
� documented in data quality flags
� in a versioning system allowing traceability & rollback
� commented
AquisNet DMO Graph/Table Editor
time series
explorer
interlinked
table + graph
views
multiple
time series +
scales/legends
+ graph types
© KISTERS AG 14/03/2013 - 18 -
3/14/2013
10
Data processing
� data security by means of controlled redundancy
� all incoming data are write-protected and will remain untouched
� a copy will be produced for validation and further processing
� automated data aggregation
� derived time series (mean, max, min, percentiles, ...) are calculated
automatically as new data comes in
� on the fly calculation of
� air quality indicators
� statistical indicators
� temporary time-series for ad-hoc analysis
AquisNet – time series analysis
programmable equations:
transform input time
series into derived time
series
regression calculation:
graph view + regression
equation
© KISTERS AG 14/03/2013 - 20 -
3/14/2013
11
Data reporting
� reports can be ...
� pre-defined by the user
� executed automatically following a pre-determined schedule
� executed manually on demand
� in the ownership of the system (all users), a single user, a user group
� numeric tables or graphs or a combination of both
� data formats for reports ...
� national or international standard formats (EC, US EPA, UBA, etc.)
� standard file formats (csv, pdf, jpg, ...)
� user-defined formats (optional)
If it’s in the DB (monitoring & station data, other data) it can be in a report!
All tables & graphs on the screen can be part of a report!
Reporting can be manual or automated!
AquisNet REP – time series graphs
time series line graphs:
multiple series, w/o
numeric table
3/14/2013
12
AquisNet REP – sample line/bar graphs
XY graphs with
labelsbox-whisker plotsline + step chart
tiled bar graphs
with labels
tiled line graphs
with sum bars
scatter plot +
linear regression
pollution rose
AquisNet – polar plots (w/o table)
wind rose
3/14/2013
13
AquisNet WEB–(option) web site content generation
Air Quality Index
AquisNet Web generates dynamic content that you can insert into
the existing framework of your own web site design!
© KISTERS AG 14/03/2013 - 25 -
air pollution network
synoptic map
AquisNet Dashboard –(option) desktop + mobile app
3/14/2013
14
... mobile apps ...
WISKIOther
�data hub: search and retrieve the data you need...
Why KISTERS is involved in interoperability ...
where do I find the
data I need for my
climate change
research?
KISTERS
DATA HUB
INTERNET
Web
Services
Other
WISKI
Other
WISKI
Other
WISKI
Web
Services
send request
receive data catalogue
Web
Services
Web
Services
send specific request
receive TS data
3/14/2013
15
AquisNet – hosting options
customer premisesAquisNet can be hosted at the
customer’s or remotely in the cloud!
AquisNet Data Center - future proof investment
� modern technology: flexible, scalable and customizable
� multi-tier software architecture:
� RDBMS – Business Layer – Rich Client
� RDBMS – Business Layer – Web Server – Browser Client
� single or multi-user with full-featured user-administration
� easy to integrate in existing IT environments:
� choice of RDBMS’s (Oracle, MS SQL) & operating systems (MS Windows, Linux)
© KISTERS AG 14/03/2013 - 30 -
� independent of any instrument or data logger manufacturer
3/14/2013
16
KISTERS + AquisNet – advantages in a nutshell
� KISTERS: 50 years of existence in the world-wide market
� solid business, markets and technological experience
� presence in all 4 corners of the world:
� close to the customers
� multi-lingual staff & intercultural experience
� dynamic + enthusiastic staff: solid mix of experienced and young employees
� AquisNet: vertical solution for the air quality community
� especially developed for AQM and related applications
� built on KISTERS time-series technology: solid, proven, future-proof
� +15 years of AquisNet
© KISTERS AG 14/03/2013 - 31 -
� +25 years of experience in time series data management
� close supervision, testing and implementation of new technology
� secured RDBMS-based technology with built-in redundancy (backup)
� transaction log and full rollback
� truly field-proven in various environments: +1000 B2B/B2G customers
KISTERS AG
Charlottenburger Allee 5 Tel. 02 41 96 71-0 info@kisters.de
D-52068 Aachen Fax 02 41 96 71-555 www.kisters.de
Vielen Dank für Ihre Aufmerksamkeit.
Thank you for your attention.
Nous vous remercions de votre attention.