OPEN DATA SCIENCE CONFERENCE

Real Time Training and Forecasting using LSTM/RNNs & AWS Lambda

Anoop Vasant KumarData Scientist

Hitachi Consulting, UK

Understanding the problem Nuclear Power Phase Out

Switch to low carbon,

environment friendly and

affordable energy.

Forecast of Energy Price

Trading Optimization and

Energy Price Forecast

Generation Optimization

Power Generation Optimization

Project objective #1: Cost effective method to provide real time market price forecasting service.

Project objective #2: A scalable and distributed solution, that forecasts prices for each of the 100+ energy products based on live market input feeds.

High Price 48 Euro

Generation Cost 30 Euro

Low Price 10 Euro

T-5 T-4 T-3 T-3 T-2 T-1 T

Time Series of a typical energy product

Time Series of Closing Prices

Time Series of Prices of Correlated Products

Time Series of Low and High Prices

Time Series of Volume Bought/Sold Forecasted Energy Price

Machine Learning Model - Per Product

Model

A model that could learn complex nonlinear relationships

Multiple features with each input being a time series on it own

Ability to remember the past trend and sequence

Ability to consider long term dependencies in data

Ability to improve forecasts and update model parameters real time based on new data inputs

Correlation of Products

Recurrent Neural Networks are networks with loops in them, allowing information to persist.

It makes use of sequential information unlike traditional neural networks.

RNN - A network that remembers

Long Short Term Memory - Recurrent Neural NetworksRemembering long term

dependencies is their default nature.

Think of LSTMs as exactly same as RNNs except the method is which the hidden state is calculated, is slightly different!.

Time Series of Closing Prices

Time Series of Prices of Correlated Products

Time Series of Low and High Prices

Time Series of Volume Bought/Sold Forecasted Energy Price

Machine Learning Model - Per Product

LSTM - Unrolled

X

y

Time Series of Closing Prices

Time Series of Prices of Correlated Products

Time Series of Low and High Prices

Time Series of Volume Bought/Sold Forecasted Energy Price

Machine Learning Model - Per Product

anoop.vasant.kumar@gmail.com

PerformanceProduct 2

Date upto which Neural Network is Trained : 2016-07-24

Compute as-a service

Provisioning and managing of the servers to run

code.

Event-driven compute service - runs code in response

to events, such as changes to data in an Amazon

S3 bucket.

Runs code on a high-availability compute

infrastructure and performs all of the

administration.

Compute Infrastructure - AWS Lambda

anoop.vasant.kumar@gmail.com

High Level Architecture /Neural Networks on the Cloud

High Speed Data Connection

Scheduled Daily Lambda Trigger

Pull

Dat

a Fe

eds

and

Dat

a Pr

epro

cess

Pandas DataFrame

Prd 1Prd 2

Prd 3

Object Upload

S3 Object Upload Event Notification

Prediction Lambda Triggered Per Object Upload

Prd 120

Prd 5Prd 4

Prd 3

Prd 1Prd 2

Prd 120

Prd 1

Prediction Date2016-08-03Key ValuePrd1 36Prd2 29…Prd 120 41

Amazon API Gateway

Time Series Product 2 in Market abc

Time Series - Product a

Time Series Product b

Preprocessed Data

Predicted Prices

X

y

anoop.vasant.kumar@gmail.com

Email: anoop.vasant.kumar@gmail.comTwitter: @vasant_anoop