Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics
-
Upload
gabor-szarnyas -
Category
Engineering
-
view
150 -
download
1
Transcript of Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics
![Page 1: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/1.jpg)
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
MTA-BME Lendület Research Group on Cyber-Physical SystemsBudapest University of Technology and Economics
Fault Tolerant Systems Research Group
Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics
Gábor Szárnyas, Zsolt Kővári, Ágnes Salánki, Dániel Varró
![Page 2: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/2.jpg)
Motivation
Research Community
Problems of experimental evaluation of MDE papers
Difficult to find real industrial model
Tool Providers
Test generation for modeling tools
Scalability evaluationand stress testing of MDE tools
Smart CPS
Synthesis of prototypical test context/environment
Testing of autonomous robots(R3COP project)
![Page 3: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/3.jpg)
Motivation
Research Community
Problems of experimental evaluation of MDE papers
Difficult to find real industrial model
Tool Providers
Test generation for modeling tools
Scalability evaluationand stress testing of MDE tools
Smart CPS
Synthesis of prototypical test context/environment
Testing of autonomous robots(R3COP project)
How to automatically synthesize graph models…?
![Page 4: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/4.jpg)
Research Question and Objectives
• All well-formedness constraints satisfied
• Designated seed fragments included Consistent
• How to characterize realistic models?
• How to distinguish real and generated models?Realistic
• Guaranteed test coverage
• Required for tool qualificationDiverse
• Performance benchmarks
• Stress testing of tools and control algorithmsScalable
How to automatically synthesize graph models which are...
![Page 5: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/5.jpg)
Research Question and Objectives
• All well-formedness constraints satisfied
• Designated seed fragments included Consistent
• How to characterize realistic models?
• How to distinguish real and generated models?Realistic
• Guaranteed test coverage
• Required for tool qualificationDiverse
• Performance benchmarks
• Stress testing of tools and control algorithmsScalable
How to automatically synthesize graph models which are...
![Page 6: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/6.jpg)
Performance Experiments
![Page 7: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/7.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
![Page 8: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/8.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
![Page 9: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/9.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
![Page 10: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/10.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
![Page 11: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/11.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
![Page 12: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/12.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
o Again, what does that imply for real use cases?
![Page 13: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/13.jpg)
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
o Again, what does that imply for real use cases?
Qualitative description of models is required
![Page 14: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/14.jpg)
How to Obtain Models for Benchmarking?
• Difficult to obtain
• Obfuscated modelsIndustrial
• Quality of models?Student work
• Good quality models
• Small in sizeTutorial
• How realistic are these models?Generated
![Page 15: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/15.jpg)
What Makes a Model Realistic?
How to decide if a model is realistic
without domain-specific knowledge?
![Page 16: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/16.jpg)
Statecharts with Attributes
RedRed &
Orange
GreenOrange
RedRed &
Orange
GreenOrange
![Page 17: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/17.jpg)
Statecharts with Attributes
RedRed &
Orange
GreenOrange
RedRed &
Orange
GreenOrange
![Page 18: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/18.jpg)
Statecharts
S1 S2
S3S4
S1 S2
S3S4
![Page 19: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/19.jpg)
Statecharts
S1 S2
S3S4
S1 S2
S3S4
![Page 20: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/20.jpg)
Typed Graphs of the Models
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
![Page 21: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/21.jpg)
Typed Graphs of the Models
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
![Page 22: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/22.jpg)
Graph Metrics
Use graph metrics for characterizingthe graph of the model.
![Page 23: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/23.jpg)
Graph Metrics
![Page 24: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/24.jpg)
Graph Metrics
Number of vertices
![Page 25: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/25.jpg)
Graph Metrics
05
10152025
Number of vertices
![Page 26: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/26.jpg)
Graph Metrics
05
10152025
Number of vertices
Number of edges
![Page 27: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/27.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
![Page 28: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/28.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
![Page 29: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/29.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
![Page 30: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/30.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
![Page 31: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/31.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
![Page 32: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/32.jpg)
One-Dimensional Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
![Page 33: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/33.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
Centrality
![Page 34: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/34.jpg)
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
Centrality
00.20.40.60.8
1
![Page 35: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/35.jpg)
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
![Page 36: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/36.jpg)
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
![Page 37: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/37.jpg)
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
They are isomorphic.
Which is the graphof a real model?
![Page 38: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/38.jpg)
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
They are isomorphic.
Which is the graphof a real model?
Related finding: simplegraph metrics are unable topredict query performance
![Page 39: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/39.jpg)
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
![Page 40: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/40.jpg)
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
S1
S2
S3
S4
T1
T2
T3
T4
T5
E
![Page 41: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/41.jpg)
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
S1
S2
S3
S4
T1
T2
T3
T4
T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
![Page 42: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/42.jpg)
„Evaluation of Multidisciplinary Graph Metrics”
Typed graph (computer science)
Multi-layered networks (social network analysis)
Multidimensional networks (network theory)
Multiplex networks (physics)
Source: Wikipedia, Multidimensionalnetwork
![Page 43: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/43.jpg)
Multidimensional Metrics
Dimensional degree distributions
Node dimension connectivity
o ratio of nodes in the that belong to a dimension
Multiplex participation coefficient
o the connections of v are uniformly distributed among D
Node activity & pairwise multiplexity
o the ratio of nodes, which are active in both d1 and d2
![Page 44: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/44.jpg)
Methodology
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 45: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/45.jpg)
Methodology
1. Collect models
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 46: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/46.jpg)
Methodology
1. Collect models
2. Data Cleansing: removeRed
Red-Orange
Green Orange
T1 T2 T3 T4 T5
Entry
![Page 47: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/47.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 48: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/48.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 49: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/49.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 50: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/50.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 51: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/51.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 52: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/52.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 53: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/53.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 54: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/54.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
3. Calculate graph metrics
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 55: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/55.jpg)
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
3. Calculate graph metrics
4. Analyze results
o Statistical + exploratory
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
![Page 56: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/56.jpg)
Domains
AutoFOCUS
Building Information Model
Capella
JaMoPP
Train Benchmark
Yakindu
real
real
tutorial
synthetic
tutorial
tutorial
![Page 57: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/57.jpg)
Domain 1
Statistical Analysis
![Page 58: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/58.jpg)
Domain 1
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
![Page 59: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/59.jpg)
Domain 1
Domain 2
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
![Page 60: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/60.jpg)
Domain 1
Domain 2
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
![Page 61: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/61.jpg)
Statistical Analysis
0
1
0 2 4 6
![Page 62: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/62.jpg)
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
![Page 63: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/63.jpg)
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
![Page 64: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/64.jpg)
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Kolmogorov-Smirnov distance
![Page 65: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/65.jpg)
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Kolmogorov-Smirnov distance
![Page 66: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/66.jpg)
Homogeneity
Distinctiveness
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
0
1
0 2 4 6
Kolmogorov-Smirnov distance
![Page 67: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/67.jpg)
Homogeneity
Distinctiveness
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
0
1
0 2 4 6
Kolmogorov-Smirnov distance
![Page 68: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/68.jpg)
Dimensional Clustering Coefficients
![Page 69: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/69.jpg)
Dimensional Clustering Coefficients
KS distance
![Page 70: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/70.jpg)
Findings
1. Metamodel-level information is insufficient
![Page 71: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/71.jpg)
Findings
1. Metamodel-level information is insufficient
1. The ratio of containment edge types in the Capella metamodels: 75%
2. The ratio of containment edges in the Capella models: 42–50 %
![Page 72: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/72.jpg)
Findings
1. Metamodel-level information is insufficient
2. Containment edges dominate distributions
![Page 73: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/73.jpg)
Findings
1. Metamodel-level information is insufficient
2. Containment edges dominate distributions
3. Many edges follow the locality principle
![Page 74: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/74.jpg)
Future Directions
Use metrics for
o Instance model generators
o Query optimization
Improve performance of calculating metrics:incremental calculation
o https://github.com/ftsrg/model-analyzer
oWorks for both EMF and RDF models
All analysis results & code are available online:
o http://docs.inf.mit.bme.hu/model-metrics/
![Page 75: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/75.jpg)
The Train Benchmark
SOSYM paper – The Train Benchmark: Cross-Technology Performance Evaluation of Continuous Model Validation
o 6 queries, 12 transformations
o EMF, property graphs, RDF, SQL
o 12+ tools
o Automated visualization & reporting
http://github.com/ftsrg/trainbenchmark
![Page 76: Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics](https://reader031.fdocuments.us/reader031/viewer/2022030314/588698451a28abf6158b78b3/html5/thumbnails/76.jpg)
Ω