Automatic Effective Model Discovery

1

Automatic Effective Model Discovery

Sagar Sen

Thesis Director: Jean-Marc JézéquelThesis Supervisor: Benoit BaudryCo-supervisor : Hans Vangheluwe

DOMINO

2

There are many mansions in the mind of a scientist

Nikola Tesla

(Inventor)

3

Laws of Science

Experiments

Instruments

Model of the Electric Motor

design

4

Models : Are simplified abstract views of complex reality

Effective Models: Models that serve a purpose or satisfy an objective. It is created based on knowledge from heterogeneous sources.

The notion of effectiveness

5

DNA Molecule

Tesla Electric Motor

An effective model to predict and invent

An effective model discovered in nature

6

Effective Models in Software

Effective Model of Phone Behavior used by millions

7

An effective model is a product of mental tacit and manual work

Can we automate discovery of effective models?

In the context of Software Systems

8

Outline

Context and Problem The Solution Model Discovery Product Discovery Conclusion Future Work

9

Model-driven Engineering (MDE) of Software Systems

Modelling Domain Specification (A “Blueprint” : Concepts with Properties allowed in a set of

models)

Modelling Domain

Phone Behavior Model

“

Models are first-class artifacts in MDE

10

Set of all Finite State Machines

Example Modelling DomainMetamodel (a blueprint) of Finite State

Machines

11

Automatic Effective Model DiscoveryThe Problem

12

Modelling Domainspecification

13

Source 1 Source 2 Source N…

Heterogeneous sources of knowledge may guide discovery

14

Source 1 Source 2 Source N…

Effective Modelling Domain

Discovery Mechanism?

Effective Models

??Reliable Sources?

15

Outline


16

Conceptual Solution

Source 1 Source 2 Source N

…


Transform

Effective Modelling Domain specification

defines

Constraint Satisfaction Problem (CSP)

17

Source 1 Source 2 Source N

…

Modelling Domain

specification

Constraint Satisfaction Problem

Solve

Effective Modelling Domain

Conceptual Solution

18

Main Challenges1. Heterogeneous sources of knowledge are

expressed in different languages. How can we transform them to a single constraint satisfaction problem?

2. Knowledge is expressed on sets of objects in a model called meta-constraints. Eg. There must be at least one Final State in the set of all States in a Finite State Machine. How can we resolve such meta-constraints?

3. Models can be of infinite size! How do we control the generation of finite sized models?

Technical Solution

19

Model Mx

UMLOCL

…


Language Lxconforms to

CSP as Alloy Model

Transformation

Class Diagram Invariants

Why Alloy ?

20

1. Sets of concepts/classes with properties can be modeled as Alloy signatures with fields in an Alloy model A.

2. Meta-constraints on the concepts in A can be modeled as Alloy facts and predicates (based on first-order relational logic with quantifiers)

3. Alloy model A can be solved in a finite scope to give solution instances.

CSP as Alloy Model

21

Concrete Realization of Solution

Modelling Domain Specification

Metamodel Feature Diagram

Effective Model Discovery

in any Modelling Language

Effective Product Discovery

in the Variability Domain of a Software

Product

22

Outline


23

Model Discovery in the domain of a Modelling Language

Metamodel MM

Set of Valid Models in a Modelling Language

How to discover effective models

for a given objective?

24

Effective Model Discovery Objectivesin this Thesis

Objective 1: Test model discovery for detect bugs in model transformations

Objective 2: Partial Model Completion in Model Editors

25

Motivating Example

UML2RDBMS(For Object Persistence)

Model Transformation

(MT)

Adding a friend on FB (Change in Social Graph)

Examples

Input Models Output Models

Input metamodel MM_in

Output metamodelMM_out

26

Motivating Example


(MT)


There may be BUGS in the transformation MT!

27

Motivating Example


(MT)


Can we discover the Effective Modelling Domain

and Effective TEST Models

that detect these bugs?

28

What is the effective modelling domain for testing a

transformation?

29

Input Models

MM_in

Heterogeneous Sources of Knowledge define the Effective Modelling Domain

28

Input Models

MM_in


Required Typesand Properties

28

Input Models

MM_in

Invariants I on MM(OCL)



28

Input Models

MM_in

Invariants I on MM(OCL) Pre-condition

pre


Pre MT

Input Domain


28

Input Models

MM_in

Invariants I on MM(OCL) Pre-condition

pre

Test KnowledgeModelfragments F


Pre MT

Test Models

Input Domain


34

Automatic test model discovery : The Approach

35

Running Example :UML2RDBMS Transformation

Pre UML2RDBMS Post

UML Class Diagram Models

RDBMS Models

Bezivin, J., Rumpe, B., Schurr, A., Tratt, L.: Model transformations in practice workshop, october 3rd 2005, part of models 2005. In: Proceedings of MoDELS. (2005)

36

Input meta-model MM_in

Invariants I on MM_in(OCL)

Test Knowledgeas set of model fragments F

Test Model Discovery : The Approach

Pre MT

Pre-condition pre

Alloy Model

1.Metamodel Pruning

Effective metamodel eMM_in

2. MM2Alloy

4. MF2Alloy

3. OCL2Alloy(Manual/

Automatic)

Required Types and PropertiesT_req, P_req

37

Metamodel Pruning : The Process



1.Metamodel Pruning

MT(UML2RDBMS)

Required Types and PropertiesClass

Association

Attribute

PrimitiveType

Package

NamedElement:name

static analysis

All UML Models

Class Diagrams

38

Metamodel Pruning : The Results1. Required types and properties and their

obligatory dependencies are preserved.

2. All instances of effective MM are instances of input MM

3. Effective MM is a super type and a subset of of MM_in. This implies that all operations on eMM_in are valid for MM_insubsetOf

superTypeOf

UML Class Diagram Subset

39





Pre MT

Pre-condition pre

Alloy Model

1.Metamodel Pruning


2. MM2Alloy

4. MF2Alloy


Automatic)


40

Why MM2Alloy ? Structural Similarity

Metamodel classes specify a “Set of Objects

with Properties”

Alloy signatures specify “sets of atoms”

and Alloy fields, facts, and predicate specify allowed “relations between atoms”

Object and Properties Atoms and Relations

meta-levelsimilarity

instance-levelsimilarity

41

What aspects to transform by MM2Alloy?

Metamodel MM

Meta-metamodelling language

expressed in

Meta-metamodel (Eg. Ecore is an industrial standard)

specified by

42

MM2Alloy : Bridging the Semantic Gap

Bird’s Eye view of the meta-metamodel Ecore that specifies the set of all metamodels

43


We zoom into the concept of Classes and Properties (Attributes and References)

(abstract-)Class and Properties to Alloy Signatures and Fields

44


We zoom into the concept of multiple inheritance

Flattening Multiple Inheritance to Single Inheritance in Alloy

45


Zoom into characteristics of properties

Opposite, Multiplicity of , Composite Properties to Alloy Facts

46

Example Transformations to Alloy

47

Transformation to Alloy (1)

Lets take the Classifier concept from the UML Class Diagram meta-model

Transformed to an Alloy Signatureabstract sig Classifier { name : String

}

Lets take the Class concept concept that inherits from Classifier

Transformed to an Alloy Signature

sig Class extends Classifier {

is_persistent: one Bool,

parent : lone Class,attrs : some

Attribute}

UML Class Diagram Subset

48

Transformation to Alloy (2)Implicit Constraints to Facts

Containment is transformed to an Alloy factfact Association_containers{

(all o : Association |o in ClassModel.association)}

49


Example from Kermeta Type System MM to illustrate multiple containers

Multiple Containers to Alloy Facts

Containment is transformed to an Alloy factfact TypeVariableBinding_containers{

(all o1,o2,o3 : TypeVariableBinding| (o1 in ParameterizedType.typeParamBinding and o2 in ParameterizedType.virtualTypeBinding and o3 inVirtualType.typeParameterBinding ) anddisj[o1,o2,o3]}

50





Pre MT

Pre-condition pre

Alloy Model

1.Metamodel Pruning


2. MM2Alloy

4. MF2Alloy


Automatic)


51

Transformation of OCL to Alloy

No Cyclic Inheritance

OCL Version:

context Class:

inv :

not self.allParents()->includes(self)

Alloy Version:

fact noCyclicInheritance {

no c: Class | c in c.^parent

}

manually to an Alloy fact ?

52





Pre MT

Pre-condition pre

Alloy Model

1.Metamodel Pruning


2. MM2Alloy

4. MF2Alloy


Automatic)


53

Test Model Fragments1.We generate partitions of the effective

metamodel using “input domain partitioning” to cover various structural properties of the input domain.

“Franck Fleurey, Benoit Baudry, Pierre-Alain Muller, Yves Le Traon. Qualifying Input Test Data for Model Transformations. Software and Systems Modeling (SoSyM), Volume 8, Number 2, April 2009.”

2. We combine partitions using two strategies:

AllRanges and AllPartitions to give model fragments that cover wide range of characteristics the entire input metamodel

3. A model fragment is transformed to a predicate that must satisfied by a test model.

54

Test Model Fragment to Alloy Predicate

pred modelFragment1{some a1:Class ,a2:Class, a3:Class|#a1.name=0 and a2.name=0and #a3.name>0}

automatically transformed to an Alloy predicate

a1 : Class

name : {null}

a2 : Class

name : {« »}

a3 :Class

name : {.+}

55





Pre MT

Pre-condition pre

Alloy Model

1.Metamodel Pruning


2. MM2Alloy

4. MF2Alloy


Automatic)


56

Test Model Discovery: The Approach

Test Models

Alloy Model

5. Solve

57

Can these test models detect bugs?1. Mutation Analysis: Detecting artificially injected bugs

2. We perform an empirical study to compare test models generated using different sources of knowledge. We use mutation score as a measure of comparison

Sources:

(a) AllPartition (5 fragments) of 8 sets of 50 models/set = 400 models(b) Random in Input Domain of 8 sets of 50 models/set = 400 models(c) AllRanges (15 fragments) of 8 sets of 150 models/set = 1200 models(d) ) Random in Input Domain of 8 sets of 150 models/set = 1200 models

Total of 3200 models with multiple Alloy solutions (10) per fragment

Time to automatically discover: 3 hours on MacBook Pro with Intel Core Duo processor and 2GB of RAM

Time for mutation analysis: 7 days on a grid. Executing 200 mutant transformations for each 3200 models

58

Mutation Analysis

Y-axis Mutation Score

X-axis Test Strategy

59

Mutation AnalysisAllPartitions (93.3%) AllRanges (92.27%)

60

Summary of Model Discovery1. Automatic Discovery of Effective Models is

Possible!1.1 We present a software framework to transform knowledge from various sources to a CSP in Alloy1.2 We solve the CSP in Alloy model to discovery models

2. Empirical studies validate the effectiveness of models!

2.1 Automatically discovered test models help us detect 93 % bugs in the UML2RDBMS transformation!

3. The whole approach is implemented in the tool “Pramana” previously known as “Cartier” conforming to industry standard inputs

61

Outline


62

Product DiscoveryFeature Diagram (FD)

specifies a set of product configurations

How to automatically discover effective products for a given objective?

63

Motivating Example

E-Health Emergency Composite Service

Insurance Service

Treatment Service

Ambulance Service

Documentation Service

64

E-Health Service Workflow

Atomic Services in a Workflow

65

Variability in E-Health Service

E-Health Emergency Serviceis a

DYNAMIC SERVICE

In Traffic Jam!Choices in Insurance

66

Reconfiguration of Workflow

67

Variation Captured in a Feature Diagram

CHOICEMANDATORY

OPTIONAL

68

Variability in E-Health Composite Service

reconfigures

Removing unselected services

Orchestration Workflow

Selection of a set of features

69

All Possible Configurations

53,248 Configurations!

70

Variation in QoS such as Response Time

53,248 Possible “response time distributions” for E-health!

(Based on Sidney et. al. “Probabilistic QoS and soft contracts for transaction based Web services”, ICWS 2010)

Example response time distributions

71

Can this sample be used to better analyze E-health’s overall QoS ?

Is there small but effective set of configurations that widely samples the configuration space?

72

How do we sample?

What are the configurations that cover all

pairwise interactions between features (or services)?

J. Czerwonka, “Pairwise Testing in Real World”Proceedings of 24th Pacific Northwest Software Quality Conference, 2006

73

Pairwise InteractionsAmbulance_f Doctor_s

Pair1 N/A N/A

Pair2 N/A A

Pair3 A N/A

Pair4 A A

Eg: Pairs say what must be in a configuration that satisfy it. Total of 2380 pairs

74

We discover only 185 configurations covering all valid pairs in the 2380 feature interaction pairs

Configuration Discovery Methodology

Feature Diagram FD

T-wise StrategyT (Eg. T=2 implies pairwise)

1. FD2Alloy

2. Generation of Alloy Predicates for Tuples (pairs for T=2)

Alloy Model

3. Detection of Consistent Tuples (w.r.t FD)

4. Generate Configurationscovering valid pairs

(Divide and Compose)

Resulting Variation in QoS for 185 Configurations

75

Box Plot of 37 configurations out of 185 configurations

76

Y-axis Response Time

X-axis Configurations

Prob

abili

ty D

istrib

ution

Box Plot of 37 configurations out of 185 configurations

77

Extreme cases show up

Only mandatory services

Maximum number of services

Robust Determination of Service Level Agreement

78

Summary of Product Discovery

1. Automatic Discovery of Effective Product Configurations is Possible!1.1 This thesis presents a software framework to transform a feature diagram and T-wise interactions to a CSP in Alloy to discover products

2. Empirical studies validate the effectiveness of products!

2.1 We automatically discover configurations of a dynamic web service that helped determine better SLA (In collaboration with Ajay Kattepur, Team DISTRIBCOM)2.2 Two large case studies: Crisis Management, E-Health

3. The whole approach is implemented in the tool “Avishkar”

79

Outline


80

Conclusion1. Can we automatically discover effective models and

products ? Answer: Yes!

2. Were we able to qualify discovery using empirical studies? Answer: “Yes! Eg. Via mutation analysis in “Test Model Discovery”

3. Is there a software prototype for automatic model discovery? Answer: Yes! “Pramana” (previously Cartier)

4. Did we apply the general approach to other domains? Answer: Yes! The domain of Feature Diagrams implemented in tool “Avishkar”

5. Were we able to qualify product discovery using empirical studies? Answer: Yes! To “sample QoS and determine SLA” in a composite dynamic service

81

Conclusion8. Any key scientific contributions?

Answer:In our opinion,

a) First results in automatic discovery of models satisfying knowledge from various sources. Notably discovery of test models! Why not other forms of discovery?

b) We leverage the idea of model transformations to scale formal methods to address very specific and modular needs (Eg. using metamodel pruning) in MDE. Theory truly meets practice!

c) Automatic discovery facilitates large-scale empirical studies in MDE

82

Outline


83

Future Work1. Discovery of large and complex models using

discovery of small models followed by their composition.

2. Discovery of models using feedback knowledge from fitness computation

3. What John Koza did with “Genetic programming” we want to do with “Automatic effective model discovery”!

4. Industrial Impact: Automatic discovery and composition of products in a Very Large Feature Diagram (Video Surveillance System from Keeneo) for a given intention.

84

PublicationsJOURNAL ARTICLES (During Thesis)

(12) Sen S., Moha N. Baudry B., Mahe V., Barais Olivier, and J.-M.Jezequel, Reusable Model Transformations 2010 (Under Review) (11) Sen S., Mottu J.M. Baudry B., Automatic Model Generation for Transformation Testing 2010 (Under Review)(10). N. Moha, Sen S., Faucher C., Barais O., and J.-M.Jezequel, Evaluation of Kermeta on Graph Transformation Problems, Journal of

Software Tools and Technology Transfer, 2010(9). Sen S., B. Baudry, Vangheluwe H., Towards Domain-specific Model Editors with Automatic Model Completion, Simulation :

Transactions of the Modeling and Simulation Society, 2009

CONFERENCE ARTICLES (During Thesis)

(8) Kattepur Ajay, Sen S., Baudry B., Beneveniste A., Jard C. Variability Modeling and QoS Analysis of Web Services Orchestrations(7) G. Perrouin, Sen S., Klein J., Baudry B, Le Traon Yves, Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines, ICST 2010(6) Sen, S., N. Moha, B. Baudry, and J.-M.Jezequel, Meta-model Pruning. In Proceedings of MoDeLs. 2009. (5) Sen, S., B. Baudry, and J.-M. Mottu. Automatic Model Generation Strategies for Model Transformation Testing. In Proceedings of International Conference on Model Transformation. 2009(4) Sen S., Baudry B., Mottu Jean-Marie. On Combining Multi-formalism Knowledge to Select Models for Model Transformation Testing. ICST'08, 2008 (3) Sen S., Baudry B. Partial Model Completion in Model Driven Engineering. In Proc. of International Conference of Applications of Declarative Programming 2007(2) Sen S., Baudry B., Vangheluwe Hans, Domain Specific Model Editors with Model Completion, LNCS vol. on Models in Software Engineering,Workshops and Symposia at MoDELS 2007 (Best Paper)(1) Sen S., and Baudry B., Mutation-based Model Synthesis in Model Driven Engineering, Proc of IEEE ISSRE, Mutation Workshop. 2006

Model Discovery ArticlesEffective Modelling Domain ArticlesProduct Discovery Articles

85

THANK YOU

I AM PLEASED TO ADDRESS YOUR QUESTIONS

Automatic Effective Model Discovery

Documents

Transcript of Automatic Effective Model Discovery