Building the next-gen digital meter platform for FluviusMaarten HerthogeTeam Lead Data Science & Engineering, delaware BeLux

64% 29% 17%got measurable results is successful reaches production

It’s easy to understand why so many projects fail

Deep Learning

ETL vs ELT

PaaS vs IaaS

Data Visualisation

Data Quality

Master Data

Big Data

Databricks

SMP vs MPP

Data Catalog

Spark

Storm

Data Mart

No-SQL

Cloud vs On-prem

Velocity, Variety and VolumeSemantic Layer

Predictive

Prescriptive

IoTStreaming

AI

Start with the problem,

not with the technology

The core data platform needs to support various use cases

Portal access to 3 years of history

Interfaces to allow scalable sharing of data

Digital meters collect data… a lot of data

7+ million IoT devices after full roll-out 7M+

LEGAL

12TB Sending up to 12 TB of data every day

GDPR and strict government regulations

The “golden hammer” does not exist,

use a portfolio approach

Think about the types of use cases, processing flexibility and enablement you require

Data consumers

LOB

CRM

ERP

#1 Increasing volume

Think about the types of use cases, processing flexibility and enablement you require

Data consumers

LOB

CRM

ERP

#2 Need to collect and combine any data

Think about the types of use cases, processing flexibility and enablement you require

Data consumers

LOB

CRM

ERP

#3 From data consumer to data explorer

Think about the types of use cases, processing flexibility and enablement you require

Data consumers

LOB

CRM

ERP

#4 Real-time becomes the standard

Think about the types of use cases, processing flexibility and enablement you require

Data consumers

LOB

CRM

ERP

#1 Increasing volume

#3 From data consumer to data explorer

#2 Need to collect and combine any data

#4 Real-time becomes the standard

Prioritize software-as-a-service to create a data portfolio

Managing all data

Social

LOB

Graph

IoT

Image

CRM

INGEST STORE PROCESS SERVE

Analyzing all data

Integrating all data

From GB’s

to TB’s

Seamless

scaling

1000+ API

requests/sec7M+ devices

Avoid the data lake pitfall

Data lakes often presented as the go-to solution in these cases

Massive scale

Optimized performance

Integration flexibility

Cost effectiveness

But data lake implementations fail in 60% of the cases

Massive scale

Optimized performance

Integration flexibility

Cost effectiveness

Becomes data swamp

Capabilities mismatch

Schema-on-read

Misused as “strategy”

Focus on collaboration

Azure Databricks

Aim for one unified analytics processing engine

Cloud storage

Data warehouses

Big data storage

IoT / streaming dataML models

BI tools

Data exports

Data warehouses

Collaborative Workspace

DATA

ENGINEER

DATA

SCIENTISTBUSINESS

ANALYST

Deploy and manage production jobs & workflows

ETL / ELT JOB SCHEDULER MONITORING DEVOPS

Single Runtime Foundation

DATA LAKE

MANAGEMENT

SERVERLESS

/ SaaSSCALE WITHOUT

LIMITS

PAY-AND-SCALE-

AS-YOU-GO

Narrow the scope and decouple storage from serve

INGEST STORE PROCESS SERVE

Why is this successful?

Unified, consistent data ingest path

Cost efficient for full detail

Why is this successful?

Allows for workload-specific platforms

tailored at specific cases independent

from the raw datastore

SQL DB Service Fabric

Data Lake Gen2& Delta

Databricks acts as glue between a data lake and various workload specific application platforms

INGEST STORE PROCESS SERVE

Databricks

Why is this successful?

Unified runtime and collaboration

One-click setup

Native integration with Azure services

Enterprise security & SLA

SQL DB Service Fabric

Data Lake Gen2& Delta

Fluvius’ data portfolio drives innovationon the Flemish energy market

Bringing consumption insights and awareness to millions of households

Lower barrier for data- and market-driven change

Portfolio projects TCO 20x lower compared to “golden hammer”

Focus on collaboration

Avoid the data lake pitfall

The “golden hammer” does not exist,

use a portfolio approach

Start with the problem,

not with the technology

Interested in more?Visit delaware.ai

Focus on collaboration

Avoid the data lake pitfall

The “golden hammer” does not exist,

use a portfolio approach

Start with the problem,

not with the technology

Thank you!Maarten HerthogeTL DS&E, delaware BeLux

Feedback

Your feedback is important to us.

Don’t forget to rateand review the sessions.