Towards a Macrobenchmark Framework for Performance Analysis of Java Applications
29
Budapest University of Technology and Economics Department of Measurement and Information Systems Budapest University of Technology and Economics Fault Tolerant Systems Research Group Towards a Macrobenchmark Framework for Performance Analysis of Java Applications Gábor Szárnyas, Dániel Varró
-
Upload
gabor-szarnyas -
Category
Engineering
-
view
191 -
download
5
Transcript of Towards a Macrobenchmark Framework for Performance Analysis of Java Applications
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Budapest University of Technology and EconomicsFault Tolerant Systems Research Group
Towards a Macrobenchmark Framework for Performance Analysis of Java Applications
Gábor Szárnyas, Dániel Varró
Performance issues
Model-Driven Development
Modeling
Codegeneration
Testing
Early validationsTransformations
Scalabilitychallenges
Model Sizes
Models = graphs with 100M–1B elements
o Car industry
o Avionics
o Software analysis
o Cyber-physical systems
Source: Markus Scheidgen, Automated and TransparentModel Fragmentation for Persisting Large Models, 2012
application model size
software models 108
sensor data 109
geo-spatial models 1012
Validation may take hours
Research Goal
Scalable query engine for evaluating complexqueries on large models.
Latest results are presented in tomorrow’s session:
o János Maginecz, Gábor Szárnyas: Sharded Joins for Scalable Incremental Graph Queries
Today: benchmarking
Motivation
Jennifer Widom: Tips for Writing Technical Papers
On performance experiments:
It’s easy to do “hokey” or meaningless experiments, and many papers do.
Many conferences expect experiments.
Benchmarks in Software Engineering
Performance experiments = benchmarks
Common goals
o Compare various tools
o Derive performance metrics
o Assess the scalability of the system
Microbenchmarks
oMethod-level
o Very difficult and not recommended
Macrobenchmarks
o Application-level
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed storage
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing
Distributed storage
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing
Distributed storage
Distributed query network
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Distributed indexer Model access adapter
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
INCQUERY-D Architecture
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
Server 0
Distributed query evaluation network
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
Separate processes
Operating System
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
• File system caching• Scheduled jobs• Swapping
Cloud Noise
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
AntijoinOther VMs runningon the same server
Parallel Execution
Server 1
Databaseshard 1
Server 2
Databaseshard 2
Server 3
Databaseshard 3
Transaction
Databaseshard 0
OS OS OS OS
Server 0
Indexer Indexer Indexer Indexer
Join
Join
Antijoin
Multiple processes in the same OS
Multithreaded processes
Threats to Validity
Operating system: caching, scheduled jobs
Parallel processes
Multithreaded execution
Environmental noise in the cloud
The hardest: Managed Runtime Environments
o Java Virtual Machine
o .NET CLR
Managed Runtime Environments
Just-in-timecompilation
Runtimeoptimization
The Effect of Warmup
Firstexecution time
Medianexecution timeof 5 executions
Two Java-based query engines
Performmultiple
executions
CASE STUDY:THE TRAIN BENCHMARK
Gábor Szárnyas, Benedek Izsó, István Ráth, Dániel Varró
2016
Database Benchmarks
Criteria for domain-specific benchmarks(Jim Gray, Benchmark Handbook, 1993):
Relevant
Scaleable
Portable
Simple
Two-phase commit, data cubes, ACID transactions
The Train Benchmark
Domain: railway network validation
Goal: measure the scalability of query evaluation
Supports various representations
Randomly generated instance models
o Increasing sizes
o Close to real-world instances
o Faults are inserted during generation
Railway Model
Synthetic model
Customizable model generator
Semaphore
Route
SegmentSwitch
Entry
ExitSwitch position
of the Route
Current
position
of the Switch
Validation Queries
Well-formedness constraints
Queries are looking for error patterns
o Lots of filtering, joins, etc.
Transformations
o Fault injections
o Quick fix-like repair operations
Benchmark Phases
1. Load 3. Transform 4. Revalidate2. Validate
Iteration: × 10Run:× 5
Change set size
Modelincreasing
size
Query Measure-
ments
Framework features
o Automated build and unit testing
o Automated visualization
Warmup
Warmup
Model Transformation
7 OOMs
Some Conclusions
Very hard to tell the actual memory consumption
oManual calls to the garbage collector – not enough
o Profiler – still not enough
o Setting a hard limit is the best approach
Benchmarking is difficult
o Lots of factors can add noise to the results
o Nothing works on the first try
Visualization helps a lot
oWorth investing time to learn R
o Not a nice language, but very productive
Related PublicationsBenchmark framework
Szárnyas, G., Izsó, B., Ráth, I., and Varró, D.,The Train Benchmark for Evaluating the Performance of Continuous Model Validation, SOSYM journal (work-in-progress)
Izsó, B., Szárnyas, G., Ráth, I., and Varró, D.,MONDO-SAM: A Framework to Systematically Assess MDE Scalability, 2nd Workshop on Scalable Model Driven Engineering, 2014
Benchmark applications
Szárnyas, G., Semeráth, O., Ráth, I., and Varró, D.,The TTC 2015 Train Benchmark Case for Incremental Model Validation, Transformation Tool Contest, 2015
Szárnyas, G., Izsó, B., Ráth, I., Harmath, D., Bergmann, G., and Varró, D.,IncQuery-D: A Distributed Incremental Model Query Framework in the Cloud, ACM/IEEE 17th International Conference on Model Driven Engineering Languages and Systems, 2014
Izsó, B., Szárnyas, G., Ráth, I., and Varró, D., IncQuery-D: Incremental Graph Search in the Cloud,Proceedings of the Workshop on Scalability in Model Driven Engineering, 2013
Ω
