Toward A Model-Driven Design Tool for Big Data Architectures

Michele Guerriero

Saeed Tajfar

Damian Andrew Tamburri

Elisabetta Di Nitto

Introduction

Coordinator (Kafka)

Orchestrator (Hadoop Cluster)

Data Store

Batch LayerSpeed Layer

Serving LayerServing LayerServing Layer

Data Source

Data Source

Lambda architectureDistributedcomputation

Data streaming

HDFS

Distributed storage

Cloud infrastructure

1

How many Big Data technologies do I need to

know and combine?

Which resources, how many do I need and how

do we configure the deployed technologies?

What if I want to know properties andperformance of my application?

Simplify software design and reduce costs

Simplify Deployment

2

Questions Goals

Support Analysis

Analysis

Deployment blueprint

!

Model-Driven Engineering

3

Platform Independent Model (DPIM)

Technology Specific Model (DTSM)

Deployment Specific Model (DDSM)

is implemented by

is deployed onto

TOSCA  blueprint

Analysis

Analysis

Analysis & Optimization

M2M transformation

M2M transformation

M2T transformation

Model-Driven Big Data Design Architecture

4

DPIM Meta-Model

5

Core  DTSM  Package

Storm  Package

HadoopMRPackage

SparkPackage

<<Uses>>

<<Uses>>

Oryx  2Package

<<Uses>>

M2M Transformation

DPIM Model

Core  DTSM  Package

<<Uses>>

...Extensibility

DTSM Meta-Model

6

Application  Logic

Framework Logic

Framework  Configuration

The HadoopMR DTSM Package

7

● Build Deployment Topology

● Standards adoption (TOSCA)

● Deployment Technological Packages

DDSM Meta-Model

8

Transformations Set

9

● Preliminary steps toward the model-driven engineering of Big Data applications.

● Great potential behind MDE for data-intensive applications!

● Future steps:

○ increase models expressiveness (data quality, privacy concerns)

○ validation against industrial case studies

○ increase automation mechanisms

○ technological support

Conclusion and Future Works

10

Thanks!

11