Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software...
Transcript of Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software...
![Page 1: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/1.jpg)
Testing StrategiesSoftware Engineering 2017
Alessio Gambi - Saarland University
![Page 2: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/2.jpg)
Generic Testing Strategies
• Automate as much as possible
• Manual not necessary bad, automatic not necessary good
• Automate the “right” part of testing input generation, execution, checking, …
![Page 3: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/3.jpg)
Automated Unit Test Generation• Generate the code of the test, the input data, and
possibly the behavior of the environment at once - Directed random generation
each test build incrementally, Randoop - Evolutionary method
whole test suite generation, Evosuite • Combine coverage and other metrics (e.g., branch
distance) to define a fitness function • False alarms and “impossible” behaviors
![Page 4: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/4.jpg)
Test Carving
• From the execution of system tests, generate a number of unit tests which execute only a subset of the observed method invocations
• Unit tests capture the same set of behaviors which are observed during the system tests
![Page 5: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/5.jpg)
Fuzz Testing or Fuzzing
• Generate a large number of random inputs
• Mostly focused on security testing Does system handle unforeseen inputs or crashes?
• Generative or metamorphic
• Structured inputs might be problematic
![Page 6: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/6.jpg)
Deep Fuzzing
• Generate a large number of random but syntactically valid inputs to reach deeper parts of the code
• Grammar based fuzzing: use a description of the input structure to guide the input generation Who provides the grammar? Can we learn it automatically?
![Page 7: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/7.jpg)
Capture and Replay
• Record the (manual) interactions between the system and its environment
• Replay interactions during testing
![Page 8: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/8.jpg)
Capture and Replay
• Record the (manual) interactions between the system and its environment
• Record the (manual) interactions between users and the GUI
• Replay interactions during regression testing
![Page 9: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/9.jpg)
Dealing with environmental and other dependencies
![Page 10: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/10.jpg)
Dealing with environmental and other dependencies
• The SUT interacts with external entities file system, database, remote services, …
• The code under test requires other code to be in place DB-bridge library, service API, …
• Avoid to directly interact with the actual entities brittle, dependent, and non-isolated tests
![Page 11: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/11.jpg)
Dealing with environmental and other dependencies
• Isolation
• Repeatability
• Reuse
• Separation of concerns setup/teardown vs test execution vs verification
How to achieve?
![Page 12: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/12.jpg)
Fixed states used as a baseline for running tests to ensure well known and fixed environment are
setup up before the test executions
Text Fixtures
![Page 13: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/13.jpg)
Fixed states used as a baseline for running tests to ensure well known and fixed environment are
setup up before the test executions
Text Fixtures
![Page 14: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/14.jpg)
Text FixturesFixed states used as a baseline for running tests to ensure well known and fixed environment are
setup up before the test executions
• Loading DB with known data
• Copying files to specific location
• Setup of test drivers, mocks, stubs, etc.
![Page 15: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/15.jpg)
Testing using Doubles
• Test Double. Generic term for any kind of pretend object used in place of a real object for testing purposes
To make a single unit work, you often need other units
![Page 16: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/16.jpg)
Testing using Doubles
• Dummy objects Object passed around but never used.
• Fake objects Working implementations with shortcuts.
• Stubs Provide canned answers to calls during tests.
• Mocks Objects pre-programmed with expectations on calls.
To make a single unit work, you often need other units
Gerard Meszaros
![Page 17: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/17.jpg)
State and Behavior Verification
• State verification examines the state of the SUT and its collaborators after the method was exercised.
• Behavior verification examines the interactions of the SUT with its collaborators during the method execution.
Martin Fowler
![Page 18: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/18.jpg)
Mocks vs Stubs
• Mocks always use behavior verification, Stubs can use also state verification.
• Setting up stubs includes defining canned responses.
• Setting up mocks also includes expressing expectations.
![Page 19: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/19.jpg)
Test with a DatabaseStub Mock
In-memory structure for storing records
Define data which will be written in the DB but not the logic to do it
SUT reads and writes records to the stub DB
SUT invokes mock, but no data are stored
Check state of DB and behavior of objects not related to the DB
Check what was written to DB and how values ended up there
![Page 20: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/20.jpg)
The Test Oracle Problem
Given an input for a system, the challenge of distinguishing the corresponding desired, correct
behavior from potentially incorrect behavior
![Page 21: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/21.jpg)
The Test Oracle ProblemGiven an input for a system, the challenge of
distinguishing the corresponding desired, correct behavior from potentially incorrect behavior
• Test preconditions specify the state before the test
• Test execution specifies what to do during the test
• Test postconditions (oracle) specify the expected results after the test
![Page 22: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/22.jpg)
The Test Oracle Problem
• Oracles are necessarily incomplete observable behavior, miss and false alarms
• Oracles are heuristics help testers to decide might point to wrong decision
Given an input for a system, the challenge of distinguishing the corresponding desired, correct
behavior from potentially incorrect behavior
![Page 23: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/23.jpg)
Software Test Oracles
• Not always possible to express the oracles non-testable programs
• Not always can be/need to be precise float/doubles, asynchronous interaction, …
• Not always possible to capture all the facets of “correct” behavior result is correct but it took hours to compute
![Page 24: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/24.jpg)
Types of Oracles• Constraint oracle
test values or relationships • Regression oracle
check results of current tests vs results of previous tests
• Self-verifying data as an oracle embed the correct answer in the test data
• Calculation oracles check the calculations of a program using other programs
• Inverse oracle Apply the inverse function, check if result is same (neg)
![Page 25: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/25.jpg)
Model as Oracles• Physical model
test a software simulation of a physical process
• Statistical model unlikely (yet formally correct) behaviors
• State model what the program does for each input in each state
• Interaction model how the SUT and the other programs behave in response to each other
![Page 26: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/26.jpg)
Model as Oracles• Physical model
test a software simulation of a physical process
• Statistical model unlikely (yet formally correct) behaviors
• State model what the program does for each input in each state
• Interaction model how the SUT and the other programs behave in response to each other
Do not forget that models may be wrong !
![Page 27: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/27.jpg)
The Test Oracle Problem
• Test with several oracles at once
• No matter what combination is used, some errors will escape anyway
![Page 28: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/28.jpg)
Anything else to test for ?What’s beyond functional correctness and coverage?
![Page 29: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/29.jpg)
Anything else to test for ?
• Load testing
• Performance testing
• Endurance testing
• Stress testing
• User Experience testing
What’s beyond functional correctness and coverage?
![Page 30: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/30.jpg)
Anything else to test for ?
• Acceptance testing
• Security testing
• Penetration testing
• A/B testing
• …
What’s beyond functional correctness and coverage?
• Load testing
• Performance testing
• Endurance testing
• Stress testing
• User Experience testing
![Page 31: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/31.jpg)
Specific Strategies. Why ?
• Testing different properties requires different approaches to testing adequacy criteria, test generation and execution, scaffolding, …
![Page 32: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/32.jpg)
Specific Strategies. Why ?
• Testing different properties requires different approaches to testing adequacy criteria, test generation and execution, scaffolding, …
• Different systems work in different contexts and have peculiar properties that must be taken into account during testing environment, distribution, limited resources, interactivity, …
![Page 33: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/33.jpg)
Testing Mobile Applications
![Page 34: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/34.jpg)
Challenges
• Limited memory/computational resources
• Hardware and software fragmentation
• Multiple network connectivity and carriers
• Unforgiving users
• Native, web-based, hybrid applications
![Page 35: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/35.jpg)
Main (Many) Strategies• Functional Testing
functionalities of application as per requirement specification
• Performance Testing client application, server, and network performance
• Memory Testing/Load Testing optimized (limited) memory usage by an application
• Security Testing secure connections, encrypted data, data leakage
![Page 36: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/36.jpg)
Main (Many) Strategies• Usability Testing, User eXperience Testing, and User
Acceptance Testing
• Interruption Testing incoming call, SMS, low memory/battery warning, …
• Installation Testing installation process including update and uninstall
• Carrier and Network Testing WiFI, 3G, 4G how switching network impacts on the application
![Page 37: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/37.jpg)
Functional Testing• Check the flow of actions
• Mostly at the level of user interface
• Tests can be local or instrumented developer runtime, emulator, real devices, cloud
• Combination of manual and automated tests capture&replay, monkey testing, …
![Page 38: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/38.jpg)
Memory/Load Testing
• Memory leakage testing performance slow down while using the app
• Find the breaking point of the application how many items in the cart make my app unresponsive?
Devices have limited memory and OS stop applications when they consume too much
memory
![Page 39: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/39.jpg)
Security Testing
• Data flow vulnerability - Check data are encrypted and channels are
secure - Do not save clear private data on client’s side
• Data leakage - Check log files
![Page 40: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/40.jpg)
Usability, User eXperience and User Acceptance Testing
• Entirely manual
• Alpha Testing: select a focused group of user
• Beta Testing: select a larger group of users with different devices/connections/location
• Crowd-sourcing: very large sets of unknown, and possibly untrustworthy, users
![Page 41: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/41.jpg)
Interruption Testing
• Go smoothly in the suspended state and restart afterwards
• Check that after the interruption the application does not end up in “frustrating” the users game not saved is lost, apps resume on home screen
Incoming notifications, text and calls, and memory/battery warning interrupt the operation of the app
![Page 42: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/42.jpg)
Carriers & Network Connectivity Testing
• Do not test only on WiFi connection
• Different carriers have different performances and features/standards
• Test for lost connectivity
• Test for connectivity downgrade 4G to 1G, 1G to Edge, …
![Page 43: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/43.jpg)
Tools• Monkey, DroidMATE: generate pseudo-random streams
of user events and system-level events
• UIAutomator: JavaScript to define tests which consists of complex user actions
• Robotium, Appium: support native and hybrid automated black-box tests
• MonkeyTalk: record and playback
• Screenfly: emulators to test different screen sizes and device profiles
there are many, many more
![Page 44: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/44.jpg)
Testing Web Applications
![Page 45: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/45.jpg)
Challenges• Client/Server architecture
- Part of the app runs in the browser (e.g., JS) - Part of the app runs in the server (CGI, DB, business
logic)
• Huge variability in browsers, versions, connection speeds, standard protocols, devices
• Open World (intranet vs internet audience)
![Page 46: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/46.jpg)
Challenges
• Fast rollout and updates
• Standard or specific requirements for appearance, graphics, or user interface
• Links between the pages, navigation
• Page content, consistent layout
![Page 47: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/47.jpg)
Main (Many) Strategies• Functionality testing
• Interface testing
• Database testing
• Performance testing
![Page 48: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/48.jpg)
Main (Many) Strategies• Functionality testing
• Interface testing
• Database testing
• Performance testing
• Security testing
• Usability testing
• Cookies testing
• Compatibility testing
• …
![Page 49: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/49.jpg)
Functionality Testing• Check all the links
- Outgoing and internal links must be valid (non broken)
- Orphan pages or dead-end navigation
- Back-button
• Test all the forms
- Fields validation, default values, wrong inputs
- Optional elements, and modification to form
![Page 50: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/50.jpg)
Interface Testing• Check the interactions between, client and Web
server, between Web server and application server, and between application server and database
• Handle errors gracefully and display proper messages to users
• Reset connections and interrupts running transactions
![Page 51: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/51.jpg)
Database Testing• Check data consistency, integrity and errors for
every DB related functionality edit, delete, modify
• Check all queries to database are sanitized and correctly executed
• Check that data is retrieved and updated correctly
![Page 52: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/52.jpg)
Performance/Load Testing• Subject the application to heavy load
- Define user model and runs as many concurrent users as possible
- Use small and large input data - Target specific pages (much like DoS)
• Stress testing: go beyond know system limits and break it; then, check if the system recovers.
![Page 53: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/53.jpg)
Cookies Testing
• Test if the application crashes if cookies are enabled/disabled
• Check if data are encrypted and cookies expire after sessions end
• “Fuzz” cookies (alter their content) or delete them
Cookies are small files stored on the user machine which enable server side user session
and might contains private data
![Page 54: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/54.jpg)
Compatibility Testing
• Cross-platform, cross-browser, and mobile
• CSS Style and AJAX/JS layout and interactions
• Security and validation
• Some operations (graphics) might not be available to OS
![Page 55: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/55.jpg)
Tools and Services• Many, diverse, and with tons of features
Load and Performance Test, Page Speed, Mobile Web/App, Link Checkers, HTML Validators, Web Accessibility, Web Services, Cross-Browser, Web Functional/Regression Test, Web Site Security, …
• Start with the most widely known: Selenium, JMeter
• Focus on the essential and what you need
• Outsource to the cloud (compatibility)
http://www.softwaretestinghelp.com/most-popular-web-application-testing-tools/
![Page 56: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/56.jpg)
Testing Distributed Applications
![Page 57: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/57.jpg)
Challenges
• Think of the problems and issues of testing normal applications and …
![Page 58: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/58.jpg)
Challenges
• Think of the problems and issues of testing normal applications and …
• … multiply them by multiple processes written in multiple languages running on multiple boxes that could potentially all be on different OS
![Page 59: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/59.jpg)
End-to-End Testing
• Order and timing of data arrival can cause bugs write after read, read after write, write over write
• Huge number of possible interleaving of operations difficult to predict and test
In a distributed application, data propagate to multiple parts. Hence, testing only a single system
at the time it not enough.
![Page 60: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/60.jpg)
• The number of components is small
• Run all the components locally bare metal or inside virtual machines
• Better control over component lifecycle
• No network related issues and delays
Simple Distributed Systems
![Page 61: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/61.jpg)
Complex Distributed Systems• Many components, impossible to test all the
configurations
• Test using a “good approximation” of production system vary the number of nodes
• Mimic any system that is not available during testing simulators, stubs, or doubles
• Generate positive and negative scenarios hard to enumerate and foreseen
![Page 62: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/62.jpg)
On Simulators/Stubs
• Valuable tools for development and debugging, but …
- are time consuming to develop
- do not behave like the actual systems
![Page 63: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/63.jpg)
Distributed Data Systems
• Asynchronous data delivery
• Nodes failure and recovery
Guarantee eventual consistency by means of data replication and distribution
![Page 64: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/64.jpg)
Asynchronous Data Delivery• Data is delivered to any the entry node which
replicates that data asynchronously to all other nodes.
• The entry node does not notify that the data has been successfully replicated.
• Testing needs to verify that data reaches the entry node and all the other nodes. Therefore, tests need to pull the data from each node where the data supposedly lives.
![Page 65: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/65.jpg)
Tests Asynchronous Data Delivery
• Change the SUT and introduce a notification when data is propagated.
- Guarantees on data delivery but changes the implementation
- Tests might not be representative
![Page 66: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/66.jpg)
Tests Asynchronous Data Delivery (2)
• Introduce artificial delays
- No guarantees on data delivery but highly changes it will complete
- Brittle tests: outcome of the tests is not predictable and depends on the environment the tests are executed (load, network speed, …)
![Page 67: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/67.jpg)
Tests Asynchronous Data Delivery (3)
• Introduce more and longer artificial delays
- Overall execution become exponentially longer
- It worsens as more tests are defined
![Page 68: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/68.jpg)
Tests Asynchronous Data Delivery (4)
• Introduce more and longer artificial delays but parallelize the tests execution
- Shorter time, but most likely more problems
- Problems appear in the test suite itself
![Page 69: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/69.jpg)
Asynchronous Data Delivery
• No ideal and “one-fits-all” solutions for testing
• Tailor testing to the specific requirements of the SUT
• Go for an hybrid solution that combines the different delays, small/large tests, and parallelization
![Page 70: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/70.jpg)
Node Failure
• A component that fail must not cause the entire system to fail, and ideally it should be invisible to the users.
• For example, data should be retrieved by a replica if the entry node is down. Data should propagate if a replica node is down and when it comes back synchronized.
![Page 71: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/71.jpg)
Test Node Failure• Setup and run the system
store data in the system
• Kill a node (take the node offline) which node? check that it is actually down
• Check data can be correctly retrieved is the data up-to-date or the “best version” (all the data) retrieving data from the broken node must fail (not hang)
• Restore the node (take the node online)
• Check data are synchronized (reconstruct the node) requires the ability to pull data from that particular node
![Page 72: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/72.jpg)
Model-Based Testing• Design the code to run in simulated environments and
deterministic execution order
• Add the ability to programmatically inject faults and perform sanity checks
• Encapsulate the correct behavior in a model and design tests on the model state searches, pseudo-random activity, and timing variations
![Page 73: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/73.jpg)
SUT
social network
![Page 74: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/74.jpg)
user userSUT
social network
![Page 75: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/75.jpg)
user userSUT
social network
![Page 76: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/76.jpg)
user userSUT
social network
![Page 77: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/77.jpg)
user userSUT
social network
![Page 78: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/78.jpg)
user userSUT
social network
![Page 79: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/79.jpg)
user userSUT
social network
![Page 80: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/80.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 81: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/81.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 82: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/82.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 83: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/83.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 84: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/84.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 85: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/85.jpg)
Init Role Reply Role
Existing
init.C
Declined Retracted Accepted
reply.Uinit.Dreply.D
observe.D observe.Dobserve.Dobserve.Dobserve.D
Existing
init.C
Accepted Declined Retracted
init.Dreply.Dreply.U
observe.D observe.Dobserve.Dobserve.Dobserve.D
Synchronized FSMs
![Page 86: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/86.jpg)
Additional Considerations• Ideal systems can be deterministically validated
not always possible!
• Design the system to run locally do not use global state
• Make sure you have dynamic logging timestamp, node ID, log levels, multiple sources
• Use APIs for messaging, time, timed events, and scheduling dependency injections during testing
to build testable distributed applications
![Page 87: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/87.jpg)
Summary• Testing is not only functional
• Automation is good but not always, manual testing is not always bad
• Focus the test effort on the requirements and the type of SUT
• Many tools… choose widely !
![Page 88: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/88.jpg)
References and Additional Readings
![Page 89: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/89.jpg)
• The Art of Software Testing, Glenn Myers
• Testing Computer Software, Cem Kaner
• Test-Driven Development by Example, Kent Beck
• How Google Tests Software, James Whittaker, Jason Arbon and Jeff Carollo
• Mock, Stubs, and Test Double:
• https://stackoverflow.com/questions/2665812/what-is-mocking
• http://cdn2.hubspot.net/hubfs/582328/GrammaTech-Img/mock-diagram.png
• https://martinfowler.com/articles/mocksArentStubs.html
• Test Oracle Problem
• http://www0.cs.ucl.ac.uk/staff/M.Harman/tse-oracle.pdf
• http://kaner.com/?p=190
• Test Mobile Applications:
• https://ymedialabs.com/17-strategies-for-end-to-end-mobile-testing-on-both-ios-and-android/
• http://www.softwaretestingclass.com/mobile-testing-tutorial-1-mobile-application-testing-strategy/
• http://cdn2.softwaretestinghelp.com/wp-content/qa/uploads/2015/08/Mobile-Testing-Tools1.jpg
• https://ymedialabs.com/17-strategies-for-end-to-end-mobile-testing-on-both-ios-and-android/
![Page 90: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/90.jpg)
• Test Mobile Applications:
• https://developer.android.com/training/testing/unit-testing/local-unit-tests.html
• https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html
• http://www.optimusinfo.com/top-10-mobile-testing-tools/
• https://www.ranorex.com/mobile-automation-testing/android-test-automation.html?gclid=CNrrjte2sNQCFekW0wodOroJTQ
• http://www.softwaretestinghelp.com/best-mobile-testing-tools/
• http://www.testingtools.com/mobile-testing/
• http://www.softwaretestinghelp.com/5-best-automation-tools-for-testing-android-applications/
• http://www.softwaretestingclass.com/overview-of-appium-mobile-automation-testing-tool/
• http://www.softwaretestingclass.com/overview-of-selendroid-mobile-automation-testing-tool/
• https://stackoverflow.com/questions/3337505/mocking-library-framework-that-works-best-in-android
![Page 91: Testing Strategies - Universität des Saarlandes · 2018-04-05 · Testing Strategies Software Engineering 2017 Alessio Gambi - Saarland University. ... • Manual not necessary bad,](https://reader036.fdocuments.us/reader036/viewer/2022070720/5ee117eead6a402d666c196e/html5/thumbnails/91.jpg)
• Test Web Applications:
• http://www.softwaretestinghelp.com/how-can-a-web-site-be-tested/
• Web Testing: A Complete guide about testing web applications. http://www.softwaretestinghelp.com/web-application-testing/
• http://www.softwaretestinghelp.com/website-cookie-testing-test-cases/
• http://www.softwaretestinghelp.com/best-cross-browser-testing-tools-to-ease-your-browser-compatibility-testing-efforts/
• http://www.softwareqatest.com/qatweb1.html
• https://dzone.com/articles/top-10-automated-software-testing-tools
• http://www.softwaretestinghelp.com/most-popular-web-application-testing-tools/
• Testing Distributed Applications:
• http://queue.acm.org/detail.cfm?id=2800697
• https://www.quora.com/How-do-I-test-a-distributed-system
• https://www.quora.com/How-do-you-design-a-distributed-system-so-that-it-can-be-deterministically-validated-by-tests