Open Data and Open Source Software - Online Registrationmeetings.wmo.int/EC-69/Presentations/EC-69...

Open Data and Open Source Software

@Finnish Meteorological Institute

WMO EC SE, 11.5.2017

Harri Pietarila, Head of Expert Services, [email protected]

Roope Tervo, Development Manager, [email protected]

mailto:[email protected]

mailto:[email protected]

Motivation

• Ecosystem is changing

• NMHS will not control the whole value chain anymore

• Large companies like IBM and Google are becoming more and more

relevant players

11/05/2017

Open Source Software @ Finnish Meteorological

Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala

2

Charge for data

Charge for Service

Closed ecosystem

Open ecosystem

Motivation

• A role of NMHS is also changing

• Challenges:

• Ensure authoritative voice in warnings

• More efficiency in development and operations

• Ensure the impact of produced information

11/05/2017



3

Finnish Meteorological

Institute opened its data

11/05/2017 4

FMI Open Source Software

https://en.ilmatieteenlaitos.fi/open-data

FMI starts to open it’s

software

2013 2016






• Finnish Meteorological Institute

opened its data in 2013.

• Basically everything that FMI has

property rights was opened.

• Both (near) real-time and historical and climatological data.

• Data is provided in freely in machine

readable format.

11/05/2017 5

FMI Open Data







Data set Description Time

Interval

Estimated

publish date

Weather

Observations

Temperature, Wind, Humidity, Ground Temperature…

10 min Open,

older data to be

added

Weather forecast

model HIRLAM RCR

Point forecasts and grid data

Latest

model run

(4 times a

day) 0…54

h

Open

Aviation Observations METAR 30 min open

Weather Radars Precipitation Rate, Precipitation Amount…

5 min Open,

older data to be

added

Lightning Thunder Strikes in Finland

5 min Open

Some Example Data Sets

11/05/2017 6 Open Source Software @ Finnish Meteorological

Institute | Roope Tervo, Mikko Visa



Finnish Meteorological

Institute opened its data

11/05/2017 7

FMI Open Source Software


FMI starts to open it’s

software

2013 2016






Motivation

Boost Product Development of the Software

• NMHSs are well connected concerning NWP development

and GTS/WIS

• Why not extend collaboration to upstream as well?

11/05/2017



8

NWP Post-

processing Interpretation Visualization Applic

ations

Applications Observations

Collaboration and OSS Proprietary software

Motivation

Streamline collaboration with partners

• Open Source Software can boost the development

• A very powerful way to support developing countries

11/05/2017



9

NWP Post-

processing Interpretation Visualization Applic

ations

Applications Observations

Collaboration and OSS Proprietary software

Motivation

Increase appropriability of

weather and climate data

• NMHSs are successful when

weather and climate don’t

cause unanticipated

unwanted impact to society

• Data is valuable only when it

reach relevant audience

11/05/2017



10

Motivation

Increase appropriability of

weather and climate data

• Maximal coverage requires

several different channels

and services

• One organization can’t handle

them all

11/05/2017



11

Motivation

Support Research

• Open Source Software empowers research as well

• Easy and open methods to access and analyze the data

• Methods are repeatable when anyone can access the

tools

11/05/2017



13

Motivation

Public-private-partnership

• Open Source Software is a great tool to ease public-private

partnership

• OSS provides seamless access to weather and climate

information from multiple NMHS providers

11/05/2017



14

Motivation

• Boost and streamline collaboration between NMHSs

• Streamline collaboration with partners

• Support developing countries

• Boost product development of the software

• Support research activities

• Increase appropriability of weather and climate data

• Support companies

11/05/2017



15

Potential Users

• Other National Meteorological Services

• International agencies (such as ECMWF, EUMETSAT…)

• Other environmental agencies

• Research Institutes

• Environmental companies

• Private developers

11/05/2017



16

License

• MIT license is recommended for all FMI

OSS…

• …but every license is considered

separately.

• For example, other members of the project and/or

used third party projects may require some

specific license.

Commercial Use

Distribution

Modification

Private Use

License and Copyright Notice required

11/05/2017



17

What

• About 30 different software under

review

• Open source is the baseline…

• … but every software is considered

separately

• How easy the software is to isolate from

production system?

• Is the project vital? Is the software still

developed and maintained?

• Does the software potentially benefit

someone?

11/05/2017



18

What

Examples

• SmartMet Server (open) High-capacity and high-availability data and product server for

MetOcean data.

• SmartMet Workstation (2018) Tool for visualizing and editing meteorological data

• SmartMet Alert (2019) Tool for creating and disseminating (supports CAP) weather

warnings

• HIMAN (open) Post-processing suite for meteorological data

• PyTroll (open) Set of tools for the reading, interpretation, and writing of weather satellite data.

• SILAM (open) Airquality and dispersion model

11/05/2017



19

11/05/2017



20

FMI Open Data

Portal & FMI

INSPIRE Data

Services

Backend for

clients’ web

services

Integration to

clients’

systems Backend for

mobile

applications

Backend for

FMI Client

Services

Backend for

FMI public

pages

• Basis of FMI product generation

SmartMet Server

11/05/2017



21

• High-performing high-availability C++

data and product server for all kind

of MetOcean data

• FMI installation responses to over 30

million requests per day

• Over 99,95 % availability

• Also used at ECMWF Copernicus

Climate Data Store

SmartMet Server

11/05/2017



22

Web pages

with warning

maps and

texts

Integration to

clients’

systems

Automatic

text

messaging

Mobile phone

application

• SmartMet Alert provides

• A user interface for the forecasters to input warning information

• Automatic warning texts

• Several possible (automatic) dissemination channels

• CAP output

SmartMet Alert

CAP

TV

RSS Feeds Print

media

11/05/2017



23

• Used in operational warning production in Finland since February 2015

• Has been installed and trained in three Caribbean countries (Cuba,

Antigua and Barbuda, Jamaica)

• April-November 2016 has been installed and trained also in 6 Pacific

countries: Samoa, Vanuatu, Fiji,

Tonga, PNG and Solomon Islands

• Is going to be installed and trained

in Bahamas in 2017

SmartMet Alert

11/05/2017



24

• Tool for visualizing and editing meteorological data

• Support for all kind of meteorological data

• NWP, Satellites, Weather Radars, Observations, Lightning

Detection, Soundings…

SmartMet Workstation

11/05/2017



25

• Post-processing and grid manipulation suite

• Runs automatic post-processing scripts like statistical correlations or

derivative parameters

• Support GRIB1, GRIB2, CSV and QueryData formats

• Utilize GPU

HIMAN

11/05/2017



26

• Free and open source python

modules for the reading,

interpretation, and writing of

weather satellite data.

• Project started by SMHI (Sweden)

and DMI (Denmark)

• FMI is collaborating with many

others

• User base is global

PyTROLL

Adding Value for Satellite Users

11/05/2017



27

• Operational Emergency and Air Quality Modelling

System

• a global-to-beta-meso-scale (down to 1 km

resolution) dispersion model

• applied for wide range of atmospheric composition,

air quality, and emergency decision support

applications, as well as for inverse dispersion

problem solution

• Applications for nuclear accidents, wildland fires

and volcanic ash as well as for regulatory air

quality assessments and forecasting

SILAM

11/05/2017



28

• Data format conversion tools

• Product generation tools

• Observation network software and

quality control

• TAC IWXXM converters

• Radar composite and analyzing

software

• …

Many Others

Some Open --- Some Under Investigation

Policies and Guidelines

• Source code and documentation in

GitHub: https://github.com/fmidev

11/05/2017



29

https://github.com/fmidev



Examples

11/05/2017



30

Project Ongoing Project in Preparation

Completed cooperation or consulting project in the country

Scientific Collaboration

International co-operation & expert services

Pacific SIDS

Caribbean SIDS

Central Asia

Andean region

S-E Europe

11/05/2017 Open Source Software @ Finnish Meteorological


11/05/2017 Open Source Software @ Finnish Meteorological Institute | Roope

Tervo, Harri Pietarila, Mikko Rauhala 32

SmartMet installed

SmartMet – installations

• Used in operational warning production in Finland since February 2015

• Has been installed and trained in three Caribbean and six Pacific countries: • Cuba

• Antigua and Barbuda

• Jamaica

• Samoa

• Papua New Guinea

• Fiji

• Tonga

• Vanuatu

• Solomon Islands

SmartMet Alert installations

11/05/2017



33

11/05/2017



34

Andean region

• SmartMet installations:

Efficient data display and

editing tool for the

forecaster & automated

weather production for

web and mobile

- Peru SENAMHI

- Ecuador INAMHI

- Bolivia SENAMHI

- Columbia IDEAM

• ICI projects with Peru

Ecuador, Bolivia and

Columbia

• Regional project PRASDES

SmartMet products example

SEMANHI-Bolivia

http://www.senamhi.gob.bo/

New Android APP SENAMHI Bolivia

the alerts and forecast in Aymara language

new color coded Alert protocol of SENAMHI

11/05/2017



35



SmartMet in SIDS Caribbean

• FMI cooperation in SIDS Caribbean from 2000

• Establishment of Caribbean wide SmartMet (forecaster workstation and forecast production system) network (10 countries)

• Improvement in forecast quality

• Improvement in forecast production and end-user products

11/05/2017 36

SmartMet installed

Dominican Republic Cuba

Jamaica



SmartMet Alert in SIDS Caribbean

• Development of Early Warning services

• Installation of SmartAlert software in Jamaica, Cuba and Antigua & Barbuda

• Improvement in warning delivery and launching of modern end-user products such as maps etc…

• Systems in pilot phase (goal: operational after COPS project)

11/05/2017 37



Google map visualization works also as a mobile phone app - Example from Cuba.

SmartMet Alert

Warning map visualization – google map

11/05/2017



38

• FINPAC - Finnish-Pacific Project to Reduce Vulnerability of the Pacific Island Countries' Livelihoods to the Effects of Climate Change

• Project covering the entire value chain from improved observations to services and benefits to communities

• Funding of 3.8M€ to SPREP with Technical Assistance from FMI

• Partnership with SOPAC, Red Cross, USP and WMO

• Beneficiaries are the NMHSs from 14 Pacific Island Countries

• Kick-off on 1 July 2013 at the Pacific Meteorology Council meeting in Fiji

11/05/2017 39

SmartMet in SIDS Pacific

FINPAC (13-16)

MFA

FMI Project

Coordinator

Project Manager

Steering Committe

e

SPREP Manageme

nt Team

Financial reporting



1.4: Implement weather forecasting tools and production systems at selected NMSs

• SmartMet workstation for editing and viewing meteorological data was installed in six countries (Fiji, PNG, Samoa, Solomon Islands, Tonga and Vanuatu).

11/05/2017 40




• Examples:

11/05/2017 41


PNG met service web page

Tonga met service web page



SmartAlert in SIDS Pacific

forecaster interface to issue warnings

Forecaster

view - example

from Samoa

11/05/2017



42

• Examples (google map visualization of the warnings):

11/05/2017 43


PNG met service web page

Solomon Islands met service

web page

Samoa met

service

web page



Social media integration – Facebook

(Papua New Guinea Weather Service)

• Forecast data produced with SmartMet

• Facebook messenger inquiry to receive text format local

weather forecast

11/05/2017



44

Social media integration – Twitter

(Tonga Meteorological Service)

• Forecast data produced with SmartMet

• Automatized tweeting of forecast information in text

format

11/05/2017



45

Training to ensure sustainability

Tonga Vietnam

Papua New Guinea Cuba Peru

11/05/2017



46

www.fmi.fi

http://en.ilmatieteenlaitos.fi/open-source-code














Open Data and Open Source Software - Online Registrationmeetings.wmo.int/EC-69/Presentations/EC-69...

Documents

Transcript of Open Data and Open Source Software - Online Registrationmeetings.wmo.int/EC-69/Presentations/EC-69...