June 7, 2017

Our journey towards a real time data platform

F.H.A. (Floran) Stuijt MSc

Alliander IT | Data & Insights

Real Time Analytics

Personal Introduction

2

Background

• Artificial Intelligence

• Computer Science

linkedin.com/in/floran-stuijt

floran.stuijt@alliander.com

Previous life

• Software & Systems Architecture/Engineering

(Recording, Robotics)

Interests

• Analytics,(Agile) systems architecture,

cognitive models

Floran Stuijt MSc

IT Architect @ Alliander

Datum

Titel van de presentatie

3

Alliander & Liander

3,0 million customers

5,7 million connections

7.200 employees

12 billion assets

85.842 km electricity grid

35.320 km gas grid

1,7 billion revenues

Datum

Titel van de presentatie

4

Using …

• smart technology

• data analysis

And by…

• closely monitoring the impact of

the energy transition

We know our grid’s status,

allowing us to…

• Prevent outages

• Fix outage faster

• Focused investments

Becoming a data driven business…

A Selection of Use Cases

5

• Fault Location Isolation and Supply

Restauration (FLISR)

• Real-time customer notification (status updates)

• Dynamic load balancing

• Monitoring and surveillance (e.g. meter alarms)

• Impact of Energy Transition

• Condition based modeling

• Data quality monitoring

opera

tional

str

ate

gic

streaming batch

First: a classification of data

6

Transactional Geographic

Real-time

Transactional Data

7Transactional

• Planned Work and Maintenance

• Asset (life cycle) Information

• Human Resource Information

Geographic Data

8Geographic

• Asset Geo Coding

- Asset coordinates

• Grid Topology

- Connectivity between assets

• Traveling distance

Real-time Data

9Real-time

• EMS/DMS Alerts & Metrics

• Digital Fault Indicator Alerts

• Social Media Events

• Power Quality (voltage threshold exceedance)

• Meteorological Events (e.g. lightning discharge)

• Smart meter power on/off notifications

Becoming a Data Driven Business requiresIntegration

10

Transactional Geographic

Real-time

BI&A

Business Intelligence & Analytics is at the heart of integration

How to deal with different dynamics of data?

Real-time

Our idea of real-time analytics

12

Filtering outliers

and corrupt data

Reduce, normalize, aggregate, etc.

Combining

Publishing

Enrichment

Filter

Pipe

Key

Eventreceiver

EventsourceCall to external service

External Service

Build on top of abstractions

13

Filteren

uitschieters en

corrupte data

Reduceren,

normaliseren,

aggregeren etc.

Verrijken

Externe Service B

Architecture

14

Event Stream

Processor

Basic

• Enrich

• Transform

• Aggregate

• Combine

Advanced

• Detect (spatio-temporal) patterns

• Parallelize (branch & merge)

• Consult external systems

Architecture

15

Event Stream

Processor

• Decouple source

and ESP

• Act as buffer in-between

Message

Queue

Architecture

16

Event Stream

Processor

• Store raw and

derived events for

later analysis

• Distill rules for ESP

Message

Queue

Measurements

& Event

Repository

Architecture

17

Event Stream

Processor

Message

Queue

Measurements

& Event

Repository

Smart Data Streaming

Real-time Data Platform (RTDP)

Real-time Data Platform

18

Real-time

Analytics +Analytics of

Real-time

Real-Time Data Platform

Some Challenges

19

• Event data modeling

• A relatively new paradigm

• Privacy & Security

• Estimating (initial) scale of the platform

• Which processing to do where (in ESP or outside)

• Integrating with Kafka

• Probably more to come…

20

Use Case:

Fault Location with IntellEvent

Traditional Grid Operator Tasks

21

Digital Fault

Detector AlarmsCustomer Calls Grid TopologyEMS Alerts

(domain) knowledge &

experience

IntellEvent Approach

22

Digital Fault

Detector Alarms

Customer Calls

Grid Topology

EMS Alerts

Asset Condition

Model

Faster and more

adequate response to

outageIntellEvent

IntellEvent Screenshot

23

Key takeaways

24

• Start experimenting, a lot can be done using open source

• Make sure business is involved early because of paradigm shift

• Be aware of the associated challenges

• Be prepared for an explosion of ideas by introducing this capability

25

Any Questions?