Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation...
Transcript of Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation...
![Page 1: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/1.jpg)
July 20, PADTAD 2008, Seattle
Reproducible Testing of Distributed Software
with Middleware Virtualization and Simulation
Cyril Briquet
Department of EE & CSUniversity of Liège, Belgium
![Page 2: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/2.jpg)
2
Lighweight Bartering Grid
PADTAD 2008 - Reproducible Testing of Distibuted Software...
P2P Grid middleware developed
at University of Liege, BE:
Lightweight Bartering Grid
● study distributed scheduling policies
in unreliable environments
● provide software engineering tools
for testing and performance evaluation
![Page 3: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/3.jpg)
3
Goal: testing Grid middleware
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● reproducible testing of most of the middleware code
● accurate discrete-event simulation of a whole Grid
![Page 4: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/4.jpg)
4
How: abstraction of Grid nodes
PADTAD 2008 - Reproducible Testing of Distibuted Software...
Communications between Grid nodes,
as well as timers, multithreading...
(interactions with system APIs)
can be abstracted, then simulated
A whole P2P Grid can then run
within a discrete-event simulator
![Page 5: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/5.jpg)
5
Contents
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● Introduction● Context● Virtualization and simulation● Reproducible testing● Summary
![Page 6: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/6.jpg)
6
P2P Grid topology: 2-levels
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● worker node
(network-enabled
edge computer)
● Peer = controller of worker nodes,
server, consumer and supplier
of computational power
![Page 7: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/7.jpg)
7
2-step Task submission
PADTAD 2008 - Reproducible Testing of Distibuted Software...
informational opacity between Peers
![Page 8: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/8.jpg)
8
Features of a P2P Grid (1/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● P2P = bottom-up Grid formation
(Peers freely leave and join)
● nondedicated Grid nodes
● worker nodes can register/deregister dynamically
(1 Peer of a P2P Grid ~ Volunteer Grid, e.g. SETI@home)
● (most) interactions can be asynchronous
![Page 9: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/9.jpg)
9
Features of a P2P Grid (2/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● submitted Tasks can be cancelled – at any time -
by user agents, worker nodes and also Peers
(=> bursts of preemption)
● indeed, Peers can reclaim at any time for their own use
their computational power supplied to other Peers
![Page 10: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/10.jpg)
10
Contents
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● Introduction● Context● Virtualization and simulation● Reproducible testing● Summary
![Page 11: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/11.jpg)
11
Testing is difficult (1/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
P2P Grid middleware
● large software (for Peers, worker nodes, user agents)
● multiple sources of bugs
(scheduling algorithms, state consistency,
network, code execution, multithreading, data transfers)
![Page 12: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/12.jpg)
12
Testing is difficult (2/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● unit testing is nice,
but until P2P Grid middlewares can be fully verified,
integration tests for typical setups are necessary
● unit testing not sufficient, because of interactions
(nontrivial impact of scheduling policies on one another)
● big issue: difficult to reproduce a given P2P Grid state
(P2P Grid = complex distributed environment)
![Page 13: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/13.jpg)
13
Performance evaluation is difficult
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● scheduling algorithms: mostly heuristics
(analytical models too complex)
=>
performance evaluated experimentally
● big issue: difficult to reproduce a given P2P Grid state
(P2P Grid = complex distributed environment)
![Page 14: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/14.jpg)
14
Discrete-event system simulation
PADTAD 2008 - Reproducible Testing of Distibuted Software...
Purpose:
● observe a system in a controlled environment
Benefits:
● implementing a simulator
easier than setting up a large-scale testbed
● simulations totally controllable, repeatable
● fast (vs. real execution, or emulation)
Accuracy?
![Page 15: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/15.jpg)
15
Virtualization and simulation (1/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● our contribution:
use virtualization, simulation
as software engineering tools
● virtulalization = injection of an abstraction layer
between « layer n » and « layer n-1 »;
in our context: middleware and calls to system APIs
![Page 16: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/16.jpg)
16
Virtualization and simulation (2/2)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● virtualization of middleware (Virtual Machine, O.S.-level)
i.e. Grid nodes isolated from their environment,
i.e. scheduling code (+ ... ) uses an abstraction layer
● integration of scheduling code and simulator code:
Grid nodes are loaded in the simulator memory space,
each with its own data structures,
then interact as they would on real computers
![Page 17: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/17.jpg)
17
Consequences of virtualization, sim
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● Grid nodes not aware whether they run
on real computers or within the simulator
● massive code reuse between implementations
(« code once, deploy twice »)
● identical (not simplified) algorithms and architecture
between middleware and simulator
● new policies automatically deployed in middleware
● high accuracy of performance evaluation
![Page 18: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/18.jpg)
18
Example: simple scheduling event
PADTAD 2008 - Reproducible Testing of Distibuted Software...
![Page 19: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/19.jpg)
19
Simulator events
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● workload submission
● (completed, failed) Task execution
● timers
● multithreading (limited model)
● data transfers (doable, but requires fine-grained virtualization
of data transfer software, i.e. FTP and BitTorrent)
![Page 20: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/20.jpg)
20
Messaging in the middleware
PADTAD 2008 - Reproducible Testing of Distibuted Software...
![Page 21: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/21.jpg)
21
Messaging in the simulator
PADTAD 2008 - Reproducible Testing of Distibuted Software...
![Page 22: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/22.jpg)
22
Simulator I/O
PADTAD 2008 - Reproducible Testing of Distibuted Software...
Inputs:
● topology & computational power of Grid nodes
● configuration of Peer scheduling policies
● description of synthetic workload
Outputs:
● execution statistics (mean BoT response time, utilization)
![Page 23: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/23.jpg)
23
Contents
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● Introduction● Context● Virtualization and simulation● Reproducible testing● Summary
![Page 24: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/24.jpg)
24
Use #1: testing before deployment
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● unit testing a new scheduling policy:
● run a P2P Grid in the simulator
with a setup to activate some specific code chunk
● observe outcome,
i.e.crashes? assertions not met?
![Page 25: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/25.jpg)
25
Use #2: testing after deployment
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● when an issue arises with middleware (real deployment)
● P2P Grid with same configuration is run in the simulator
=>
determine whether a bug, crash, ... comes from the common code,
or may come from code interacting with environment
● enables co-development
● main branch: architecture + simulator
● secondary branch: Task execution, network
![Page 26: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/26.jpg)
26
Use #3: policy enumerator (1/3)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● example: introduction of an adaptive preemption policy
391 classes, 53.8kloc => 402 classes, 55.6kloc,
+ 73 classes refactored
● nothing broken by refactoring?
● unit testing not sufficient, because of interactions
(nontrivial impact of scheduling policies on one another)
![Page 27: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/27.jpg)
27
Policy Decision Points of one Peer
PADTAD 2008 - Reproducible Testing of Distibuted Software...
Peer =
server, consumer and supplier of computational power
![Page 28: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/28.jpg)
28
Use #3: policy enumerator (2/3)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● each Peer:
5 scheduling PDPs, 2 other PDPs, 30+ parameters
● let's enumerate
2892 combinations of policies = 2892 simulations
● outcome: 2859 OK, 40 bug #1, 3 bug #2
![Page 29: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/29.jpg)
29
Use #3: policy enumerator (3/3)
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● is it tractable? today's situation:
508 classes, 70.5kloc, 16850 combinations and growing
=> all simulations run... on a Grid
(a few hours to a few days, depending on Grid toplogy)
● future: insights from Skoll?
distributed continuous testing project,
using adaptive selection of test cases,
with additional tests performed
in the « neighborhood » of failed test cases
![Page 30: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/30.jpg)
30
Self-Bootstrapping
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● self-bootstrapping =
current, stable version of a given system
used to develop next version
● 1 middleware:
basic policies
● N simulators:
test and evaluate
advanced policies
![Page 31: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/31.jpg)
31
Limits of virtualization & sim
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● suitable for long-running Tasks, e.g. runtime > k seconds
● simulation of multithreading should be improved
● simulator is (currently) single-threaded;
could benefit from future many-core computers
● transfer of control messages simulated,
but transfer of data files (currently) not simulated
![Page 32: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/32.jpg)
32
Implementation remarks
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● event-driven architecture and asynchronous messages
particularly suitable to discrete-event simulation
● Java (J2SE 5.0) middleware, simulator & applications
● OO paradigm very useful:
interfaces with 2 implementations
(middleware, simulator)
● automated memory management very useful
(in the future, reuse of discarded data structures
between simulated Grid nodes would be helpful)
![Page 33: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/33.jpg)
33
Contents
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● Introduction● Context● Virtualization and simulation● Reproducible testing● Summary
![Page 34: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/34.jpg)
34
Summary
PADTAD 2008 - Reproducible Testing of Distibuted Software...
● « code once, deploy twice » based on virtualization
of Grid nodes at middleware level,
and simulation of a whole P2P Grid
● massive code reuse
=> co-development of simulator and middleware
=> immediate deployment of new policies
=> testing before and after deployment
=> systematic testing of policy combinations
=> accurate simulation results
● future work in multithreading, sim of data transfers
![Page 35: Reproducible Testing of Distributed Software · 2019. 11. 26. · 16 Virtualization and simulation (2/2) PADTAD 2008 - Reproducible Testing of Distibuted Software... virtualization](https://reader034.fdocuments.us/reader034/viewer/2022051907/5ffa9309001d8421d4589234/html5/thumbnails/35.jpg)
Thank You.