www.unamur.be

State Machine Flattening, a Mapping Study and Tools Assessment

Xavier Devroey <xavier.devroey@unamur.be> Maxime Cordy <maxime.cordy@unamur.be>Axel Legay <axel.legay@inria.fr>Pierre-Yves Schobbens <pierre-yves.schobbens@unamur.be>Patrick Heymans <patrick.heymans@unamur.be>

11th Workshop on Advances in Model Based Testing, A-MOST@ICST ’15Graz, Austria

THE CONTEXT Behavioural Model-Based

Testing (of Software

Product Lines)Framework

THE CONTEXT

THE CONTEXT

THE CONTEXT

Let’s check the literature!Let’s do a

Systematic Mapping

Study!

A SYSTEMATIC MAPPING STUDY ???

xavier.devroey@unamur.be 7

SMS: Small Introduction

• Largely used in medical research• … but quite new in Software Engineering• Goal: categorise published documents • “It requires less effort [than a systematic literature

review] while providing a more coarse-grained overview.” [Petersen, 2008]

A-MOST '15

xavier.devroey@unamur.be 8

Protocol

A-MOST '15

xavier.devroey@unamur.be 9

Protocol

A-MOST '15

xavier.devroey@unamur.be 10

Phase 1: Definition of RQs

• (RQ1) What are the input and output models used in the different flattening approaches?

• (RQ2) Do the different approaches support hierarchy (composite states) and parallelism (orthogonal states) in the input model?

• (RQ3) In which context are the different flattening approaches performed (e.g., code generation, test-case generation, semantic definition, etc.)?

A-MOST '15

xavier.devroey@unamur.be 11

Protocol

A-MOST '15

xavier.devroey@unamur.be 12

Phase 2: Search of Publications

• Exploratory search– Google Scholar (http://scholar.google.be/)– Science Direct (http://www.sciencedirect.com)– Computer Science Bibliographies (http://liinwww.ira.uka.

de/bibliography/)

• Inclusion criteria:– Books, articles, proceedings, technical reports and grey literature

• Exclusion criteria: – Slides

• Extension by related work– By word search in the documents (flat.* or hierarch.* or

orthogonal.*)A-MOST '15

xavier.devroey@unamur.be 13

Protocol

A-MOST '15

xavier.devroey@unamur.be 14

Set of Publications

A-MOST '15

• Initial set: 24 publications

• After step 3 : 12 new publications

• After step 3 (2nd): 1 new publication which did not match the inclusion and exclusion criteria

• Set of publications: 36 publications

xavier.devroey@unamur.be 15

Protocol

A-MOST '15

xavier.devroey@unamur.be 16

Phase 3: Keywording

• Step 1: tag papers using keywords– Adaptive reading depth– Guide the keywording process using research questions

• The purpose of the research (RQ3)• The input/outpout models of the transformation (RQ1)• Hierarchy/orthogonality support (RQ2)• Implementation (RQ3)

– In parallel by two reviewers + arbitrator

• Step 2: group keywords to build classification scheme

A-MOST '15

xavier.devroey@unamur.be 17

Classification Scheme

A-MOST '15

xavier.devroey@unamur.be 18

Classification Scheme

A-MOST '15

xavier.devroey@unamur.be 19

Classification Scheme

A-MOST '15

xavier.devroey@unamur.be 20

Classification Scheme

A-MOST '15

xavier.devroey@unamur.be 21

Protocol

A-MOST '15

xavier.devroey@unamur.be 22

Phase 3 bis: Quality Assessment

• Quality of the description of flattening in the publications• Based on Systematic Litterature Review protocol• Usability

– G 1.1 Is there a tool implementation?– G 1.2 Is there a small example?– G 1.3 Is there a more significant case study (even if not fully detailed)?– G 1.4 Are the input and output models described?– G 1.5 Does the publication present the limitations of the transformation?

• Degree of generality: – G 2.1 Are there guidelines for the transformation separated from the example

of the transformation (if any)?– G 2.2 Does it detail the transformation process for all the constructs of the

input model?– G 2.3 Does the flattening technique support hierarchy?– G 2.4 Does the flattening technique support orthogonality?

A-MOST '15

xavier.devroey@unamur.be 23

Phase 3 bis: Quality Assessment

• Quality of the description of flattening in the publications• Based on Systematic Litterature Review protocol• Usability

– G 1.1 Is there a tool implementation? 50% (Only 5 / 15 could be found!)– G 1.2 Is there a small example? 90%– G 1.3 Is there a more significant case study (even if not fully detailed)? 27%– G 1.4 Are the input and output models described? 90%– G 1.5 Does the publication present the limitations of the transformation? 67%

• Degree of generality: – G 2.1 Are there guidelines for the transformation separated from the example

of the transformation (if any)? 80%– G 2.2 Does it detail the transformation process for all the constructs of the

input model? 67%– G 2.3 Does the flattening technique support hierarchy? 97%– G 2.4 Does the flattening technique support orthogonality? 70%

A-MOST '15

xavier.devroey@unamur.be 25

Protocol

A-MOST '15

xavier.devroey@unamur.be 26

Phase 4: Mapping

A-MOST '15

xavier.devroey@unamur.be 27

Phase 4: Mapping

A-MOST '15

93% are solution papers

flattening is a transformation used to bridge high-level models with existing lower level analysis tools and frameworks

xavier.devroey@unamur.be 28

Phase 4: Mapping

A-MOST '15

2(!) evaluation and validation papers

xavier.devroey@unamur.be 29

Phase 4: Mapping

A-MOST '15

Model Checker input code, code dedicated to the synthesis of embedded systems, and Java code

xavier.devroey@unamur.be 30

Phase 4: Mapping

A-MOST '15

Consistency management, IP telephony, formal semantics to hierarchical state machines

xavier.devroey@unamur.be 31

Phase 4: Mapping

A-MOST '15

test case generation and selection

xavier.devroey@unamur.be 32

Phase 4: Mapping

A-MOST '15

UML with disparities in supported constructions

xavier.devroey@unamur.be 33

Phase 4: Mapping

A-MOST '15

Preferred for providing formal semantics

xavier.devroey@unamur.be 34

Conclusion about Flattening

• Mostly used for model-based validation (testing, verification) and code generation

• Step towards a more general objective

• Impacts the quality of the description of flattening algorithms

– Precise constructs supported by the flattening transformation are not always provided

– Validity of the flattening transformation is barely addressed

A-MOST '15

CONCLUSION ABOUT SMS

… and some lessons learned

xavier.devroey@unamur.be 36

Anonymous reviewer

““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no

content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the

categories, etc.But there is zero information on algorithms for state

machine flattening, thus the paper provides me no added value.”

A-MOST '15

xavier.devroey@unamur.be 37

Anonymous reviewer

““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no

content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the

categories, etc.But there is zero information on algorithms for state

machine flattening, thus the paper provides me no added value.”

A-MOST '15

xavier.devroey@unamur.be 38

Anonymous reviewer

““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature

review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no content at all. There are lots and lots of words used to describe the methodology, the keywords used to

search, the categories, etc.But there is zero information on algorithms for state

machine flattening, thus the paper provides me no added value.”

A-MOST '15

xavier.devroey@unamur.be 39

Anonymous reviewer

““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no

content at all. There are lots and lots of words used to describe the methodology, the keywords used to search,

the categories, etc.But there is zero information on algorithms for state

machine flattening, thus the paper provides me no added value.”

A-MOST '15

xavier.devroey@unamur.be 40

Anonymous reviewer

““I was unfamiliar with this term and had to look it up. It seems that this term refers to a systematic literature review. I found the paper completely uninteresting, despite having an interest in state machine flattening, because I found no

content at all. There are lots and lots of words used to describe the methodology, the keywords used to search, the

categories, etc.But there is zero information on algorithms for state

machine flattening, thus the paper provides me no added value.”

A-MOST '15

Categorise published documents

xavier.devroey@unamur.be 41

… lessons learned

• SMS is a good starting point for a thesis– Systematic => protocol

• Define• Follow

– Learn fast reading• Lots of papers

– Learn literature search engines manipulation

• Quality assessment (step 3bis)– Learn what to do/not to do when writing a paper

A-MOST '15

THANK YOU FOR YOUR ATTENTION !

Models and tools available at https://projects.info.unamur.be/vibes/