Well-founded decisions based on climate trend analysis

0 Copyright 2014 FUJITSU

Human CentricInnovation

Fujitsu Forum2014

19th – 20th November


Informed Decisions Based on Climate Trend Analysis

Gernot FelsGlobal Services & Solutions Marketing, Fujitsu

Manager for Cloud Infrastructures and Big Data Innovations, Fujitsu

Dr. Fritz Schinkel


The Promise of Big Data

Discover hidden secrets

Predict opportunities

Identify and minimize unknown risks

Take better and faster decisions

Accelerate business processes

Increase performance and productivity

Improve efficiency and effectiveness

Profitability and competitive advantage

Better utilize our planet‘s resources

A convincing value proposition.


Manufacturing

Energy

Maintenance

Agriculture

Big Data matters to every industry

Big Data

Healthcare Transportation

New opportunities, new values for enterprises and society.

Retail Finance

Public Sector …


Weather and climate trend predictions

Who needs to know credible long-term weather and climate trends?

Renewable power generation (solar, wind)

Power plant operation

Agricultural planning (flood, pest control)

Ski resort planning

Communities, counties, government

Transport, air-traffic control, shipping, sailing

Insurance

Manufacturing, retail, services

TV channels

…

Historical weather data required for trend analysis.


Historical weather data for trend analysis

European Centre for Medium-Range Weather Forecasts (ECMWF)

Analysis of weather development

Global weather data since 1979

Time series of weather maps

Usable for climate research and local trend analysis

Weather model ERA Interim

ECMWF Re-Analysis; Interim (highest resolution)

Model resolution

Time interval 6h

Measurement time: 0:00, 6:00, 12:00, 18:00 GMT

Grid of 0,25° (4 grid points per degree)

128 meteorological indicators

Time series of weather maps


Meteorological indicators

10 metre U wind component Large-scale snowfall Surface net thermal radiation Vertical integral of eastward cloud liquid water flux10 metre V wind component Logarithm of surface roughness length for heat Surface net thermal radiation, clear sky Vertical integral of eastward geopotential flux10 metre wind gust since previous post-processing Low cloud cover Surface pressure Vertical integral of eastward heat flux2 metre dewpoint temperature Maximum temperature at 2 metres since previous post-processing Surface roughness Vertical integral of eastward kinetic energy flux2 metre temperature Mean sea level pressure Surface sensible heat flux Vertical integral of eastward mass fluxAlbedo Mean wave direction Surface solar radiation downwards Vertical integral of eastward ozone fluxBoundary layer dissipation Mean wave period Surface thermal radiation downwards Vertical integral of eastward total energy fluxBoundary layer height Medium cloud cover Temperature of snow layer Vertical integral of eastward water vapour fluxCharnock Minimum temperature at 2 metres since previous post-processing TOA incident solar radiation Vertical integral of energy conversionClear sky surface photosynthetically active radiation Northward gravity wave surface stress Top net solar radiation Vertical integral of kinetic energyConvective available potential energy Northward turbulent surface stress Top net solar radiation, clear sky Vertical integral of mass of atmosphereConvective precipitation Photosynthetically active radiation at the surface Top net thermal radiation Vertical integral of mass tendencyConvective snowfall Runoff Top net thermal radiation, clear sky Vertical integral of northward cloud frozen water fluxDownward UV radiation at the surface Sea surface temperature Total cloud cover Vertical integral of northward cloud liquid water fluxEastward gravity wave surface stress Sea-ice cover Total column ice water Vertical integral of northward geopotential fluxEastward turbulent surface stress Significant height of combined wind waves and swell Total column liquid water Vertical integral of northward heat fluxEvaporation Skin reservoir content Total column ozone Vertical integral of northward kinetic energy fluxForecast albedo Skin temperature Total column water Vertical integral of northward mass fluxForecast logarithm of surface roughness for heat Snow albedo Total column water vapour Vertical integral of northward ozone fluxForecast surface roughness Snow density Total precipitation Vertical integral of northward total energy fluxGravity wave dissipation Snow depth Vertical integral of cloud frozen water Vertical integral of northward water vapour fluxHigh cloud cover Snow evaporation Vertical integral of cloud liquid water Vertical integral of ozoneIce temperature layer 1 Snowfall Vertical integral of divergence of cloud frozen water flux Vertical integral of potential+internal energyIce temperature layer 2 Snowmelt Vertical integral of divergence of cloud liquid water flux Vertical integral of potential+internal+latent energyIce temperature layer 3 Soil temperature level 1 Vertical integral of divergence of geopotential flux Vertical integral of temperatureIce temperature layer 4 Soil temperature level 2 Vertical integral of divergence of kinetic energy flux Vertical integral of thermal energyInstantaneous eastward turbulent surface stress Soil temperature level 3 Vertical integral of divergence of mass flux Vertical integral of total energyInstantaneous moisture flux Soil temperature level 4 Vertical integral of divergence of moisture flux Vertical integral of water vapourInstantaneous northward turbulent surface stress Sunshine duration Vertical integral of divergence of ozone flux Volumetric soil water layer 1Instantaneous surface sensible heat flux Surface latent heat flux Vertical integral of divergence of thermal energy flux Volumetric soil water layer 2Large-scale precipitation Surface net solar radiation Vertical integral of divergence of total energy flux Volumetric soil water layer 3Large-scale precipitation fraction Surface net solar radiation, clear sky Vertical integral of eastward cloud frozen water flux Volumetric soil water layer 4


GRIB for historical and forecast weather data

GRIdded Binary

Compressed binary format

Standard defined by WMO (World Meteorological Organization)

Used to store weather data

Based on rectangular grid

Geographic coordinates as grid points


Challenges and objectives

Challenges

Time series of global weather maps do not give immediate insight for certain location

Difficult and long-lasting evaluation(e.g. for wind probability estimation)

Late decisions

Objectives

Retrieve time series for certain location

Create puncture for relevant grid points over relevant period of time

Data transformation needed

How many files and which data volumes?


File quantities and data volumes

1 GRIB file per snapshot

4 snapshots per day

51,100 input files (time-related) over 35 years

360 degrees of longitude

180 degrees of latitude

4 grid points per degree of longitude and latitude

Add 1 grid point (north and south pole)

360 x 4 x (180 x 4 +1) =1,038,240 grid points =output files (location-related)

128 meteorological indicators, 4 bytes each

25 TB of historical weather data

Which solution concept will help?


Distributed Parallel Processing

DataNode

TaskTracker

DataNode

TaskTracker

DataNode

TaskTracker

NameNode

JobTracker

Clie

nt

Master

Slaves DFSConcept

Distribute data and I/O to server cluster nodes

Local server storage

Move computing to where data resides

Shared nothing architecture

Data replication to several nodes

Benefits

High performance, fast results

Unlimited scalability

Fault-tolerance

Cost-effective (standard servers with OSS)

De-facto standard


platform meets the challenges

ECMWF data as input

Load ECMWF data into HDFS

Use MR to invert data

From time series of world-wide weather maps

Into grid point based time series of weather data

Arrange as (sorted) KV pairs

Key = grid point and time combined

Value = Meteorological data

Java apps

Retrieve proximate time series, determine local weather development

Visualize results

Incremental update by short MR jobs

Import weather history

Invert time series

Retrieve proximate time series,determine local weather development

Visualizeresult


Example: Wind park planning

Time period: 14 years (2000-2013)

4 snapshots per day

20,456 input files (time-related) from ECMWF

21,221,625,600 records

1,038,240 grid points = output files (location-related)


Solution approach effects time to value

PERL

~100 min for processing data from 1 month

12 x 14 x 100 min ~ 12 days(over 14 years)


Not acceptable in practice



PERL





MapReduce

30 min for import to HDFS

141 min processing time

Read HDFS files (historical data)

Data transformation

Write results to HDFS files

~120 x faster than script approach

8 Slave Nodes (2-socket, 6C/12T)

Servers not fully utilized

Potential for improvement by removing other workload

Speed advantage by parallelization.



PERL





MapReduce

30 min for import to HDFS

141 min processing time

Read HDFS files (historical data)

Data transformation

Write results to HDFS files

~120 x faster than script approach

8 Slave Nodes (2-socket, 6C/12T)

Servers not fully utilized

Potential for improvement by removing other workload

Options for further acceleration?


In-memory platform as option

For single users Hadoop platform is sufficient

Retrieval and visualization within seconds

Increasing response times

Increasing number of users

Increasing number of queries

Complex queries, e.g. where in certain geographic area are most favorite locations for certain plans

Solution: IMDG

Accelerate retrieval and visualization

Pre-defined queries memory-resident

In-memory platforms help cope with any level of complexity.

Import weather history

Invert time seriesRetrieve proximate time series,

determine local weather development

Visualizeresult


Data transformation – Just 1x or more often?

Depending on use case no one-time act

New inversion of weather maps with new questions

Deduction from meteorological indicators in global weather maps

Example: Max. wind speed

Peak speeds of crossing weather front occur only shortly at one location

Often fail at 6 hrs grid

Determine front on weather map timely before and thereafter

What is the effort to realize new questions?


Dreamlike Big Data

Display weather data as table –Spreadsheet like Excel

Apply meteorological formulas directly to sample data

Check partial results at once in spreadsheet

Fast test run on significant (filtered) test data set

Simple expansion to total data set and visualization

Big Data for business users.

Process large data volumes, but avoid programming

MR jobs?


Dreamlike Big Data

Display weather data as table –Spreadsheet like Excel

Apply meteorological formulas directly to sample data

Check partial results at once in spreadsheet

Fast test run on significant (filtered) test data set

Simple expansion to total data set and visualization

Big Data for business users.


Is this a Big Data project?

1 data source

Structured data

Data is not generated at high speed

Analysis not always time-critical

25 TB x 2 is a considerable volume

Traditional technologies do not help

Big Data technologies solve customer problem

Affordable

Scalable with growth

Expected processing time can be controlled

Indeed no day-to-day Big Data project, but a very interesting one.

Volume Variety VelocityVersatility Value


Un- / Semi-/Poly-

structureddata

Big Data infrastructure

IMDB

Data Sources Analytics Platform Access

Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize

Consolidated data Distilled essence Applied knowledgeVarious data

IMDB

AppsServicesQueries

….

VisualizationReporting

Notification

DistributedParallel

Processing

IMDG

Dat

a at

res

tD

ata

in m

otio

n

FS

IMDG

DB / DW

DB / DWNoSQL NoSQL

IMDG

CEP


IMDG

Un- / Semi-/Poly-

structureddata


IMDB




IMDB

AppsServicesQueries

….


NotificationIMDG

Dat

a at

res

tD

ata

in m

otio

n

FS DB / DW

DB / DWNoSQL NoSQL

IMDG

CEP


Un- / Semi-/Poly-

structureddata


IMDB




IMDB

AppsServicesQueries

….


NotificationIMDG

Dat

a at

res

tD

ata

in m

otio

n

FS DB / DW

DB / DWNoSQL NoSQL

IMDG

CEP


How can help

Complete analytics platform

Infrastructure and services

Consulting, introduction, operation, maintenance

Apps for analysis and visualization

Integrated Systems for fast deployment

Location-based time series as cloud service

Weather prediction expertise

Everything from a single source: Simple, fast, without risk.


FUJITSU Showcase


Fujitsu showcase

What are the wind trends in a certain area?

Is weather better during weekends or on working days?

In which areas which differences?

Questions to be answered


Summary


Summary

Big Data – one of today’s megatrends

Promising value proposition

Exciting use cases across industries

Knowledge about future weather and climate is valuable for many target groups

Historical data available

Transformation needed to get desired insight and recognize trends

End-to-end solutions from Fujitsu

Integrated systems for fast-time to production

Supplementing services

You’d like to look into the future? Have a word with Fujitsu.

Thank you for listening


Appendix


Fujitsu Showcase: Wind Trends (1)

Select location in map or satellite view by click …

… or select previously saved location



Wait a second …



See the wind, temperature and

air pressure

Move over the charts and see individual

values



Select zoom windows to see details in

wind speed, temperature and

air pressure



Distribution of wind speed over time,

and distribution of wind frequency

and speed along wind direction

is displayed and …



… month to be taken into account can be

restricted and animated

Distribution of wind speed over time,

and distribution of wind frequency

and speed along wind direction

is displayed and …



Select year or span of years

for long term trends



MunichMecklenburg-Vorpommern

PaderbornBorkum

(off-shore)

Somalia

Cape Horn


Fujitsu Showcase: Weekdays and Weather (1)

Get complete page automatically published

Locations with most significant span between warmest and coldest weekday average

as map and as list

Number of grid points with maximum / minimum temperature on certain weekday

Locations with most significant span between warmest and coldest weekday average

and warmest day on a certain weekday



Visualization GUI to study the span of weekday mean

temperature at certain places and to look for possible reasons

Map colored for high span of weekday mean temperature



Sliders for span threshold,

contrast and opacity of coloring



And an adjustment for grid points with low temperature

span over the complete observation time



Using the color settings and the zooming into the map

we can find areas with significant differences of

weekday mean values in the observed timeframe



Click to a certain position shows the curve of

average temperature for the weekdays,

the coordinates and the total min/max temperature

of the point



Map and satellite can be used to find

possible reasons for mean temperature

related to weekdays



Zoom into the source of the color cloud

Industrial complex isshut down on Sunday?



US east cost is cooler on Sunday / Monday

Is traffic system heating

the atmosphere over the week?



South of Hudson Bay is an area with Wednesday

mean temperature approx. 1C higher than on Saturday

Does wood industry influence the temperature

in the rhythm of the week?

Well-founded decisions based on climate trend analysis

Technology

Transcript of Well-founded decisions based on climate trend analysis