Download - Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Transcript
Page 1: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

EXPLICATING AND REASONING WITH MODEL UNCERTAINTY

Marsha ChechikUniversity of Toronto

July, 2014

Acknowledgements: work done in collaboration with R. Salay, M. Famelis, A. DiSandro and many others

1

Page 2: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Why Do Engineers Build Models?• To understand

• …the interesting characteristics of an existing or desired (complex) system and its environment

• To predict• …the interesting characteristics of the system by analysing its

model(s)• To communicate

• …their understanding and design intent (to others and to oneself!)• To specify

• ...the implementation of the system (models as blueprints)

Source: Bran Selic’s Modeling Course

Page 3: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Software Engineering Models

Requirements

ArchitectureBehaviour Static Design

Use CasesDeployment

Concepts

StructureAbstraction Purposeful Reasoning

Page 4: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Model-Based Software Engineering (MBSE)• An approach to system and software development in which software

models play an indispensable role• Based on two time-proven ideas:

switch (state) { case‘1:action1; newState(‘2’); break; case‘2:action2; newState(‘3’); break; case’3:action3; newState(‘1’); break;}

(2) AUTOMATIONS1

S3

S2

e1/action1e2/action2

e3/action3

switch (state) { case‘1:action1; newState(‘2’); break; case‘2:action2; newState(‘3’); break; case’3:action3; newState(‘1’); break;}

(1) ABSTRACTIONS1

S3

S2

e1/action1e2/action2

e3/action3

Realm of

modelinglanguages

Realm of tools

Source: Bran Selic’s Modeling Course

Page 5: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

9

Model-Based Software Engineering (MBSE) Toolbox (partial list!)Support for:

Development,analysis,

refinement,transformation of models

+Communication,

documentation

Page 6: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

10

This is Natalie.

Natalie is a modeler.

Natalie faces uncertainty in her everyday work

Page 7: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

11

Alternative Designs

Hmm, I don’t know which

one, yet.

Page 8: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

12

Conflicting Stakeholder Opinions

What do I do until they decide?

Page 9: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

13

Incomplete Information

I don’t know everything about

this, yet.

Page 10: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

14

Uncertainty in Software Development

Uncertainty about the content of the model.

Many design alternatives Conflicting stakeholder opinions

Incomplete information

Page 11: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

15

Classifying Natalie’s Uncertainty[Aughenbaugh06] classification

Is inherently unknowable?

In the world(aleatory, “gambler”)

In the knowledge(epistemic, “science”)

Yes(irreducible uncertainty)

No(reducible

uncertainty)

Location of uncertainty

Uncertainty about the content of the model.

Page 12: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

MBSE Toolbox for Natalie?Existing tools do not applyOptions:

• Wait until uncertainty gets resolved (how long?)• Make a provisional decision and run with it (need backtracking!)

We propose:• Defer resolution of uncertainty but incorporate

uncertainty handling into the development processto allow progress

• Lift standard MDE tools to handle models with uncertainty• Add new tools:

• Help Natalie express her uncertainty• Help Natalie determine what she does not yet know• Help Natalie manage and reduce uncertainty

Mod

el

with

Unc

erta

inty

Page 13: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

17

• Motivation:• MBSE toolbox• Sources of uncertainty• Goal: MBSE toolbox for uncertainty-aware models

• Creating the toolbox• Modeling uncertainty• Transforming models containing uncertainty• Reasoning about models• Uncertainty-reducing refinement• Tool support

• Putting this into context

Page 14: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

18

EXPRESSING UNCERTAINTY

Page 15: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

19

Representing Uncertainty: ExampleSolver

SolverException

+ effect : String

Unsure if it should be an inner class.

Unsure if we need this field.

Notation means “inner class” in UML

Page 16: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

20

Some Decisions to Make

Options for assigning values to states of confidence

Quantitative

Examples: 0, 0.3, 0.75

Good for capturing likelihood of events

Hard to capture one’s own uncertainty

Qualitative

Examples: yes/no/don’t know strongly … strongly agree disagree

Good for capturing one’s own level of uncertainty

Less precise than quantitative

Page 17: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

21

How to Represent Uncertainty?Ask users

• Goals: model uncertainty explicitly, keep syntax familiar

[MiSE’13]

Page 18: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

22

Representing Uncertainty

Partial Models

• Points of uncertainty (“May elements”) explicated using syntactic annotations

Solver

SolverException

+ effect : String

Unsure if it should be an inner class.

Unsure if we need this field.

[ICSE12]

Page 19: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

23

Representing Uncertainty

Partial Models

• Points of uncertainty (“May elements”) explicated using syntactic annotations

Propositional variables: “the element exists”

Solver

SolverException

+ effect : String

X

Y

[ICSE12]

Page 20: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

24

Representing UncertaintySolver

SolverException

+ effect : String

X

Y

Solver

SolverException

Solver

SolverException

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : String

x=F, y=F x=T, y=F

x=F, y=T x=T, y=T

4 concretizations: 4 ways to resolve uncertainty.

Page 21: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

25

Representing Uncertainty

Partial Models

• Points of uncertainty (“May elements”) explicated using syntactic annotations

• Restrictions to the set of concretizations can be captured in the “May formula”

Solver

SolverException

+ effect : String

X

Y

X v Y

Page 22: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

26

Representing UncertaintySolver

SolverException

+ effect : String

X

Y

Solver

SolverException

Solver

SolverException

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : String

x=F, y=F x=T, y=F

x=F, y=T x=T, y=TX v Y

Page 23: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

-(S)securityRelated : (SV)Types1-numOfPassengers : Int

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

-numOfDoors : Int(V)C1

(M) (M)

(M)(M)

(MS)

(SV)Types1

Int

P1:ClassDiagram

?

?

? ?

?

?

?

More Complex Design Uncertainty

Not sure which security attributes - but know some will be needed.

Not sure which class will hold the numOfDoors attribute

Not sure if we will have Hovercraft or which class it specializes.

Not sure what other vehicle types there will be

Page 24: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

• May• Not sure if a model element should occur• annotations: M – maybe E - exists (default)

• Abs• Not sure how many elements exist that play a particular role• annotations: S - set, P - particular (default)

• Var• Not sure whether the element will be merged with another• annotations: V – variable, C - constant (default)

• OW• Not sure if the model is complete• annotations: M - maybe, E - exists (default)

Capturing Partial Information with MAVO

M – maybe E – exists (default)

S – set P – particular (default)

V – variable C – constant (default)

inc – incomplete comp – complete (default)

Page 25: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

A MAVO Class Diagram

Not sure which security attributes - but know some will be needed.

-(S)securityRelated : (SV)Types1-numOfPassengers : Int

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

-numOfDoors : Int(V)C1

(M) (M)

(M)(M)

(MS)

(SV)Types1

Int

P1:ClassDiagram

Not sure which class will hold the numOfDoors attribute

Not sure if we will have Hovercraft or which class it specializes.

Not sure what other vehicle types there will be

?

?

? ?

?

?

?

P1:MAVO(ClassDiagram)

Page 26: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

• Benefits• Partial model refinement = uncertainty reduction • Formal: allows analysis and automation• Semantics-independent – applies to models of different types• Use instead of concrete models …

… to allow deferral of decisions without delaying progress

Partial Models

A partial model represents the entire set of possible concrete models

Page 27: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

31

UNCERTAINTY TOOLBOX

Mod

el

with

Unc

erta

inty

Page 28: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

32

• Motivation:• MBSE toolbox• Sources of uncertainty• Goal: MBSE toolbox for uncertainty-aware models

• Creating the toolbox• Modeling uncertainty• Transforming models containing uncertainty• Reasoning about models• Uncertainty-reducing refinement• Tool support

• Putting this into context

Page 29: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

33

Transforming Models with Uncertainty

Natalie should be able to use model transformations

[MODELS’13]

Page 30: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

34

Transforming Models with Uncertainty

We need to lift Natalie’s transformations so that they can apply to models with uncertainty

Existing transformation techniques do not support this!

Natalie should be able to use model transformations

[MODELS’13]

Page 31: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

35

Reminder: Model Transformations With Graph Rewriting (“Classical”)

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

NegativeApplicationCondition

LeftHandSide

RightHandSide

EncapsulateVariable refactoring:Make fields private and add getter methodsunless they belong to some inner class

Example rule:

Page 32: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

36

Graph Rewriting: Example Input Model

SolverSolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Match

Page 33: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

37

Graph Rewriting: Example Input Model

SolverSolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

NAC also matches! ABORT !

Page 34: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

38

Graph Rewriting: Example Input Model 2

SolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Match

Page 35: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

39

Graph Rewriting: Example Input Model 2

SolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Delete

Page 36: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

40

Graph Rewriting: Example Input Model 2

SolverException

- effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Add

Page 37: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

41

Graph Rewriting: Example Input Model 2

SolverException - effect : String + getEffect() : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Add

Page 38: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

42

Graph Rewriting: Example Input Model 2

SolverException - effect : String + getEffect() : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

No more LHS matches.

Stop.

Page 39: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

43

And Now: Transforming Models With Uncertainty

Natalie wants to apply the rule to an input with uncertainty

Page 40: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

44

Why Transforming Models with Uncertainty is Hard?

Solver SolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Match???

X

YX v Y

Page 41: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

45

Why Transforming Models with Uncertainty is Hard?

Solver SolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Match???

X

YX v Y

Page 42: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

46

Why Transforming Models with Uncertainty is Hard?

Solver SolverException

+ effect : String

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

RHSLHSNAC

Should we delete?Should we add?

X

YX v Y

Page 43: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

47

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

?

Page 44: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

48

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : StringSolver

SolverException

?

Page 45: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

49

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : StringSolver

SolverException

Solver

SolverException

-effect : String+getEffect() : String

Solver

SolverException

+ effect : String

Solver

SolverException

Page 46: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

50

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : StringSolver

SolverException

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

+ - effect : String+getEffect() : String

X

Ya

b

Page 47: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

51

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

+ - effect : String+getEffect() : String

X

Ya

b

Page 48: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

52

Lifting Technique

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

(a) Find Match

Step 1: Determine applicability

Page 49: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

53

Lifting Technique

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

(a) Find Match

(b) Make sure the rule applies to at least one concretization

(requires solvinga SAT problem)

Step 1: Determine applicability

Page 50: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

54

Lifting Technique

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

Solver

X

Step 1: Determine applicabilityStep 2:Transform graph

SolverException

+ effect : StringY

(a) Copy over unchangedparts

Page 51: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

55

Lifting Technique

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

Solver

SolverException

+ - effect : String+getEffect() : String

X

Ya

b

Step 1: Determine applicabilityStep 2:Transform graph

(a) Copy over unchangedparts

(b) Perform additions and deletions

Added and deleted elements become Maybe

Page 52: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

56

Lifting Technique

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

X

Y

X v Y

Step 1: Determine applicabilityStep 2:Transform graphStep 3:Transform formula

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

+ - effect : String+getEffect() : String

X

Ya

b

Constrain Maybe elementsto ensure each thateach concrete modelis correctly transformed.

Page 53: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

57

Lifting Technique Summary

Solver

SolverException

+ effect : String

X

Y

X v Y

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

+ - effect : String+getEffect() : String

X

Ya

b

Step 1: Determine applicability (Henshin + Z3 SMT Solver)Step 2:Transform graph (Hanshin)Step 3:Transform formula (Java + Z3 input strings)

Page 54: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

58

Case Study

• Object–relational mapping (ORM)

• “Translate a class diagram to a relational database

schema.”

• Classic benchmark for model

transformation research

• Triple graph grammar with 5 layered graph rules [Varro06]

(Image from [Varro06])

Page 55: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

59

Case Study• Input model: the Ecore metamodel

• ORM for Ecore is important: cf. CDO and Teneo

• Manually flattened inheritance hierarchy and adapted to the

metamodel in [Varro06]

• Resulting model had 65 model elements:

• 17 classes, 17 associations, 6 generalization links, 25 attributes

• Manually injected points of uncertainty to create partial models with

increasing numbers of concretizations

Page 56: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

60

Setup And Results

• RQ: How does lifting scale with increasing uncertainty?

• Varied: number of concretizations of input

• Measured: time to complete the ORM transformation

• Ran on Intel Core i7-2600 3.40GHz×4core, 8GB RAM, Ubuntu-64 12.10.

• Runtime does not increase dramatically. Approach scales.

# concretizations 1 24 48 108 144 192 256Time (seconds) 32.6 32.8 32.7 32.9 32.6 33.0 48.4

Page 57: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

61

• Motivation:• MBSE toolbox• Sources of uncertainty• Goal: MBSE toolbox for uncertainty-aware models

• Creating the toolbox• Modeling uncertainty• Transforming models containing uncertainty• Reasoning about models• Uncertainty-reducing refinement• Tool support

• Putting this into context

Page 58: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

62

Verification

Example property:“Every inner class has at least one attribute”

Natalie’s favorite analysis

technique

How can you apply it to partial models?

Page 59: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

63

Verification: the Naïve ApproachExample property:

“Every inner class has at least one attribute”

Enumerate:

Page 60: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

64

Verification: LiftingExample property:

“Every inner class has at least one attribute”

Natalie’s favorite analysis technique,

LIFTED

• Applies directly to the partial model

• Does not enumerate concretizations

• Computes result using three-valued logic

…all concretizations

…some but not all

…none

Property holds for…

Page 61: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

65

Verification: Lifting

• Checking syntactic properties [ICSE’12]:• Language independent• SAT-based

• Checking semantic properties:• Depends on language semantics• Requires custom adaptation of verification technique• Examples:

• Goal satisfaction in i* models [RE’12, RE’14]• MTSA: Model checking and other analyses over MTSs [ASE’08]

Page 62: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

66

Scalability of Verification• Q: Is verification of models with uncertainty feasible?

• How does it compare with checking each model individually?• What about comparison with the classical verification?

• Q: How does the level of uncertainty affect this feasibility?

Experiments: with random inputs and real case studies

Random inputs:varying size of the model and level of uncertaintyspeedup = time for set of conventional models

time to check partial model

Page 63: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

67

Results

Page 64: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

68

Case Study: Checking Properties• Why? Triangulate experimental results (random inputs) with

observations from a real-word scenario

• Case study details:• Bug from the UMLet bugzilla• Realistic bug fixes• Uncertainty: which combination of actions that fix the bug

to choose from?• We checked two properties from the literature [SMSJ03]• Resulting partial model: 27,261 elements, 220

concretizations

[SMSJ03] R.V. Der Straeten, T. Mens, J. Simmonds, and V. Jonckers. “Using Description Logic to Maintain Consistency between UML Models”. In Proc. of UML’03, pp 326-340, 2003

Page 65: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

69

Results

Page 66: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

70

Results of Evaluation

• Is there a speedup?- Yes, it is consistently faster than reasoning with the set.

• How is speedup affected by changing model size and levels of uncertainty?- Speedup decreases with model size- Speedup increases with level of uncertainty- No slowdowns!

Reasoning with Partial modelsvs

Reasoning with a set of conventional models

Page 67: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

71

Property-Based RefinementSATSolver

SolverException

+ effect : String

X

Y

X v Y

Property Φ1: “Every inner class has at least one attribute”

Goal:Create a partial model of the concretizations for which the property holds.

(X v Y) Λ Φ1

We just need to update the May formula:

Page 68: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

72

Uncertainty-Reducing Refinement• Systematically incorporate

new information…• … Manually

• Edit the uncertainty annotations

• … Using automated transformations

Verify that a transformation is always uncertainty reducing.

Unsure if it should be an inner class.

X v Y

Unsure if we need this

field.

You know what? I want to get rid of

effect.[FASE’12, VOLT’12]

Page 69: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

-numOfDoors : IntCarTimeMachine Boat (MS)OtherVehicles

-lastLegalEntry : Timestamp-lastIDNumber : (V)IDType-previousIDNumber : (V)IDType-numOfPassengers : Int

Vehicle

Truck

(MS)

(V)IDType

Timestamp

P2:ClassDiagram

Int

A MAVO RefinementNot sure which security attributes - but know some will be needed.

-(S)securityRelated : (SV)Types1-numOfPassengers : Int

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

-numOfDoors : Int(V)C1

(M) (M)

(M)(M)

(MS)

(SV)Types1

Int

P1:ClassDiagram

Decision

Not sure which class will hold the numOfDoors attribute

Decision

Not sure if we will have Hovercraft or which class it specializes.

Decision: remove it

Not sure what other vehicle types there will be

Partial Decision

Refines

P1:MAVO(ClassDiagram)

Page 70: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Making the Toolbox... MMINT: Model Management INTeractive

• A platform for developing (typed) model management tools• Model management environment:

• Diff, refine, merge, etc.Available at https://github.com/adisandro/MMINT

CSPSAT/SMTHenshinAlloy

Page 71: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Making the Toolbox... Uncertainty-awareMMINT: Model Management INTeractive

• A platform for developing (typed) model management tools• Model management environment:

• Diff, refine, merge, etc.Available at https://github.com/adisandro/MMINTMU-MMINT – lifting of model management operators + new ones

CSPSAT/SMTHenshinAlloy

Mod

el w

ith

Unc

erta

inty

MU-MMINT(pronounced “Moomin”)

Page 72: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

76

• Motivation:• MBSE toolbox• Sources of uncertainty• Goal: MBSE toolbox for uncertainty-aware models

• Creating the toolbox• Modeling uncertainty• Transforming models containing uncertainty• Reasoning about models• Uncertainty-reducing refinement• Tool support

• Putting this into context

Page 73: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

77

PUTTING UNCERTAINTY INTO CONTEXT

Page 74: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

78

Back to Uncertainty Modeling• Our choice:

• Qualitative, possibilistic uncertainty modeling

• Keeping concrete syntax familiar• Encoding an exact set of potential

classical models• Refinement – reducing the set of

possibilitiesExplicit uncertainty modelingwith Partial Models + May formulas

Page 75: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

79

Representation of Sets

Uncertainty

Product Lines

Metamodels

Megamodels

ConcretizationsPartial model

ProductsProduct Line

InstancesMetamodel

MembersMegamodel

Page 76: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Inspired by: Modal Transition Systems (MTS)

LTS Extension:– Required transitions– Possible transitions– the rest is proscribed (prohibited)

– MTS toolbox – MTSA [ASE’08]

request?

request

reply

[Larsen/Thompson, 1988]

Page 77: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

reply? request?

request

reply

MTS = Set of LTS

request?reply?

request

reply

request

requestreply

reply

LTS

Page 78: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

MTS = Set of LTS

request?request

reply

reply? request?

request

reply

request?reply?

Refin

emen

t+

-

request

reply

request

requestreply

reply

LTS

Page 79: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Our Modeling of UncertaintyWe generalize MTSs in two ways:1. Make partiality modeling language-independent

• Benefit: consistent applicability across all phases of SE• Cost: reasoning is only about syntactic properties

2. Use different types of partiality for different types of uncertainty

• Benefit: methodological principles for expressing uncertainty

Page 80: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Aside: Complete w.r.t Scope and Span

• Scope – what is of interest (and what is not)• Span – level of abstraction

Valid call sequences of public methods of class Socket Political Map of Africa

sdkfj

sdkfj

sdkfj

sdkfjsdkfj

sdkfj sdkfj

sdkfj

sdkfj

Page 81: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

85

Some Similar Concepts• Non-determinism

• Only in behavioral models• Corresponds to omitted information

• Underspecification• Specification – saying what is known/required• Everything else about a given scope and span is

unspecified (uncertainty is implicit)

• With “our” uncertainty – can say what we do not know• Including statements about a given scope and span• Uncertainty is explicit

aa

Page 82: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

86

Some More Similar Concepts

• Incompleteness• Often same as underspecification• But can also include knowledge about the state of

completeness about the model w.r.t. a given scope and span

• “Our” uncertainty model can explicitly state which parts are incomplete

Page 83: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

87

Uncertainty is a Meta-Property! • Allows the to capture:

uncertainty of the specificationuncertainty in the variability modeluncertainty in a meta-modeluncertainty in parts of a non-deterministic model

Page 84: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

• Vagueness in modeling [Hermann]

• Incompleteness in databases• Representing null values in db [Fagin, etc.]

• Local Closed World Assumption [Cortes-Calabuig, etc.]

Other Similar Concepts

Page 85: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

89

Summary

Page 86: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

90

Some Current and Future Work• Front-end for modeling uncertainty [MiSE’13]• Extending the uncertainty toolbox to handle full MAVO models• Lifting other techniques that require semantics to handle

uncertainty• E.g., non-functional analysis

• Understanding origins and models of uncertainty• Mining developer conversations to infer uncertainty in collaborative

design [RSSE’14]• Building uncertainty dashboard to help the design process

• Uncertainty patterns• Bottom-up: useful for tool building• Top down: detecting uncertainty in workflows and conversations

Page 87: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

91

Some Current and Future Work• More complex modeling of uncertainty

• Combining “our” uncertainty with environment uncertainty• Handling richer qualitative models (fuzzy logic? probabilities?)

• Using explicit uncertainty modeling to bridge the gap with informal modeling/sketching• Bran Selic’s “formality slider”

• Uncertainty all the way down to code• Uncertainty as a meta-aspect, to be handled using language

workbenches• Lifting programming languages, e.g., Java• Symbolic execution of partial models

• Lifting analyses to other set-handling formalisms• E.g., transformations in product lines [ICSE’14]• Code analyses?

Page 88: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

92

Questions?

Page 89: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

93

Bibliography I[MODELS13] M. Famelis, R. Salay, A. Di Sandro, M. Chechik. “Transformation of Models Containing Uncertainty”. In Proc. of MODELS’13, 2013.

[ICSE12] M. Famelis, M. Chechik, and R. Salay. “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proc. of ICSE’12, 2012.

[ICSE14] R. Salay, M. Famelis, J. Rubin, A. Di Sandro, M. Chechik. “Lifting Model Transformations to Product Lines”. In Proc. of ICSE’14, 2014.

[FASE12] R. Salay, M. Famelis, and M. Chechik. “Language Independent Refinement using Partial Modeling”. In Proc. of FASE’12, 2012.

Page 90: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

94

Bibliography II[Varro06] D. Varro, S. Varro-Gyapay, H. Ehrig, U. Prange, and G. Taentzer. “Termination Analysis of Model Transformations by Petri Nets”. In Proc. of ICGT’06, pages 260–274, 2006.

[Classen09] A. Classen, P. Heymans, P. Schobbens, A. Legay, andJ. Raskin. “Model Checking Lots of Systems: Efficient Verification of Temporal Properties in Software Product Lines”. In Proc. of ICSE'10, pages 335-344, 2010.

[Svahnberg05] M. Svahnberg, J. Van Gurp, and J. Bosch. “A Taxonomy of Variability Realization Techniques”. Software: Practice and Experience, 35(8):705-754, 2005.

Page 91: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

95

I don’t know vs I don’t care• Knowledge vs relevance scale

I care I don’t careI know Specification

(classical)Out of scope

I do not know Uncertainty Out of scope

Page 92: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

96

MAVO • M – default is exists. Explicit only about uncertainty of

existance• A• V• O – by default, the world is closed unless considered

open

Page 93: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

97

REASONING OVER SETS

Page 94: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

98

“Plural” Models

“Singular” model m Plural

model p

“Deconstruct”D : p {m}

“Extract”E: p x r m

Lifted Transformation

T : p1 p2“Construct”C: {m} p

“Project” P : p1 x r p2

|p1| > |p2|

The intent behind each abstraction determines

pragmatics

Page 95: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

99

Sets of Models – A Useful Abstraction?

• Unified way of looking at set abstractions

• Reuse methods from one abstraction to another

• Chain abstractions?

• Finding connections between fields: principled but practical outcomes

Page 96: Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

100

Example: Lifting Transformations for Product Lines

1. Determine applicability2. Transform graph3. Transform formula

1. Determine applicability2. Transform graph3. Transform presence conditions

Transforming Partial Models Transforming Product Lines [ICSE’14]

May elements

May formula

Feature

Presence conditions