Model driven engineering for big data management systems

www.modeliosoft.com

Model driven engineering for big data management systems

Marcos ALMEIDA [email protected]

Sarah DAHAB [email protected]

Andrey SADOVYKH [email protected]

1

Outlines

Introduction Model-driven development Big Data

Juniper Sample application Conclusions

www.modeliosoft.com 2

20 ME

2006

17,5 ME

2005

70 ME

2013

ParisRennesNantes

Sophia

SOFTEAM – a French IT services / Software vendor

• SOFTEAM, a growing company 25 years’ experience 900 experts Regular growth

• Specialist in OO technologies, new architectures, methodologies

• Banking, Defense, Telecom, …


23 ME

2008

Modelio for Software and System Engineering

• UML editor with 20 years’ historyo CloudMLo SysMLo MARTEo Code generationo Documentationo Teamwork


• Available under open source at Modelio.org

MODEL-DRIVEN DEVELOPMENT


It is all about models … Starting with UML


Requirements

UML Use Cases

Architecture

UML Componentsand Classes

Design

Refined Classesor Domain Specific Language

Implementation

Code generationJava, C++, Frameworks

Model = Code


Typical example: Control system for a frigate

• 800+ components• Developed by 100+ engineers• 1M+ LOC

• MDD fosters Productivity and Quality witho Code generationo Components reuseo Tracingo Automation


Curious DSL example: Ruby on Rails

Haml HTML%br{:clear => left’} <br clear=”left”/>%p.foo Hello <p class=”foo”>Hello</p>%p#foo Hello <p id=”foo”>Hello</p>.foo <div class=”foo”>...</div>#foo.bar <div id=”foo” class=”bar”>...</div>


Feature: User can manually add movie Scenario: Add a movie Given I am on the RottenPotatoes home page When I follow "Add new movie" Then I should be on the Create New Movie page When I fill in "Title" with "Men In Black" And I should see "Men In Black"

Cucumber and Capybara

HAML

What do we get from MDD?

Pros• Design once, deploy

everywhere!• Write your

transformation once, transform anything!

Cons• Transformations are

hard to write…• How to make sure they

are CORRECT? i.e.– Is there any

data/semantic loss?


BIG DATA


Volume, variety, velocity

1. @-mails sent every second : 2,9 million

2. Video uploaded to YouTube every minute: 25 hours

3. Data processed by Google every day: 24 petabytes

4. Tweets per day: 50 million

5. Products ordered on Amazon per second: 73 items


Only 0,5 % of data is analyzed

• In 2012, 2 837EB generated - just 0,5% actually analyzed.

That still amounts to 14EB (or 14.185 million terabytes)

Source: IDC & EMC


The main problem is Heterogeneity!

• Many different database management systemso Ex:

• MySQL (www.mysql.com/), • Big Table (http://research.google.com/archive/bigtable.html)• SimpleDB (http://aws.amazon.com/simpledb/)• Memcached (http://memcached.org/)• …

• Many underlying data representation paradigmso Ex:

• Relational Databases• Key-value Stores• Object-oriented Databases• Big Tables• …


http://www.mysql.com/

http://research.google.com/archive/bigtable.html

http://aws.amazon.com/simpledb/

http://memcached.org/

The basis of our solution is MDE… Why?

• Separating the problem from the solutiono In JUNIPER we model the solution

• Fostering automationo Analysiso Code generation


BusinessObjects Transformation

HDFS

MySQL

MongoDB

Abstract ModelsSpecific Models / code

Transformation

Transformation

Understanding the problem… Why is it so HARD? (1/2)• Target Technologies based on different paradigms• Example:


A

B

JPA@Entitypublic class A { @Basic public B getB(){ … }…}

SQLcreate table A (…)create table B (…)create table A_B (…)

Understanding the problem… Why is it so HARD? (2/2)• Target structure is variable• Example:


A

B

ER

NoSQL

A

BAB

Here A and B are

independent entities

Here, for performance reasons, B is

embedded in AA

B

Illustration: comparative features of MongoDB and PostgreSQL


Our solution: a component based approach to NoSQL heterogeneity

• Generic model transformation chaino Integrated to other Juniper tools

• Audit rules• Model to model transformations• Code generators

• Database specific instantiationsoApplication architecture modellingoData modelling oHardware architecture (deployment) modelling


The Juniper FP7 EU project

Website: http://www.juniper-project.org/Start Date: 2012-12-01Duration: 36 monthsTotal cost: 4 M€


JUNIPER integrates Big Data technologies over MPI


DOCs StreamsDBs

Data Processing

Stage 1 Stage N

BusinessIntelligence

Analytical DBs

Visualization

dbdb

DOCsDOCs

Data Processing in JUNIPER

S1

S3

S2

Analytical DBs

mpi

mpi

mpi

mpi

FPGA-enabled nodesHadoop

HPC

Modelling in Juniper


Models

High level Architecture

(Nodes,Programs, Streams…)

Real-timeconstraints

Java Code Code

Generation (+MPI initialization, communication, etc)

Reverse Engineering

SchedulabilityAnalysis

Tool

SchedulingAdvisor

Measurements & Advice

Deployment Scripts

ConfigurationModelExport

CodeGeneration

Mapping Programming Model, UML and MARTE


JUNIPERProgram

Channel

Cloud Node

ProgrammingModel

UML MARTE

Modelling the application and real-time constraints


Real-time constrains- response time- bandwidth

Big Data flowJUNIPER Programs

Modelling the hardware infrastructure at a high level


Cloud Node

CPU with 4 cores Hard drive

MPI code generation


Overview of the Juniper programming model concepts

Next step: integrating data modelling to the programming model


Business data modelling in Juniper

• Example


Uniquely identifying pieces of data

Partitioning dataIn different nodes

Business data modelling in Juniper

• Concepts


Approach taken for dealing with heterogeneity in JUNIPER

1. Define a generic template for Modelio modules to provide support for big data management systems

2. Instantiate the template for MongoDB and PostgreSQL


MongoDB modelling module


MongoDB Example (1/2)


MongoDB Example (2/2) + DEMO Video

Database schema configuration scripts

Deployment scripts

Configuration scripts


PostgreSQL modeller module


PostgreSQL Example

master installation scriptstandby installation script

configuration files


DATABASE MIGRATION SAMPLE APPLICATION


[VIDEO]


CONCLUSIONS


In short…

• Challenge: o Big data applications How should we handle heterogeneous data ??

• Juniper response:o Model driven solution for designing real-time big data systemso Component based solution to heterogeneity

• General business objects + big data concepts modelling • Database specific concepts

– Modelling– Model transformations– Code generation


… and Perspectives / Exploitation

• Source code and documentation available on our websiteo http://forge.modelio.org/projects/junipero http://forge.modelio.org/projects/mongodb-modelero http://forge.modelio.org/projects/postgresql-modeler

• Tutorial + Dissemination on our forum


http://forge.modelio.org/projects/juniper

http://forge.modelio.org/projects/mongodb-modeler

http://forge.modelio.org/projects/postgresql-modeler

Questions?Marcos AlmeidaSOFTEAM | ModelioSoft{name.surname}@softeam.fr

SOFTEAM R&D Web Site: http://rd.softeam.com

Modelio Web Site : http://www.modelio.orghttp://forge.modelio.org/projects/juniper

JUNIPER Web Site : http://www.juniper-project.org


*

*for your questions

Model driven engineering for big data management systems

Internet

Transcript of Model driven engineering for big data management systems