Post on 09-Oct-2020
TestingAn Introduction to Software Testing
Gholamhossein Tavasoli @ ZNU
Who Am I?
o Gholamhossein Tavasoli
o PhD Student in Artificial Intelligence
o System Developer for 8 yearso High Performance IDS / IPSo Industrial IDSo
o Security (Semi) Expert, Member of ZNU Cert for 3 years
Type of Bugso Security
o Regression
o Business Logic
o Accessibility
o Performance
o User Interface
o Integration
o Internationalization
o Scaling
Some Famous Software Failures
o Heartbleedo Heartbeat Extension of TLS is used to keep Datagram TLS sessions open.o Simple request and response scheme:o “send me back the following (padded) string which is n bytes long.”
o An attacker just had to request a long string, while telling it is short.o Other party responded with short string, then leaked potentially confidential data.
o Video (https://youtu.be/WgrBrPW_Zn4)
Some Famous Software Failures
o Ariane-5o Inertial navigation software taken from Ariane-4. Untested.o All other systems thoroughly tested component-by-component.o Ariane-5 had a different trajectory than Ariane 4.o Converting 64-bit floating-point data into 16-bit unsigned integer values. → Arithmetic overflow.o There was an exception handler for that. It had been disabled.o Not even 40 seconds after launch, Ariane-5 literally self-destructed. Successfully.
o Video (https://youtu.be/WgrBrPW_Zn4)
Why Testing?
o Software contains defects. The hide.
o Defects can cause software failures.
o Failures can cost money, and even be mortal!
o Testing assures the quality of the software.
o Testing accelerates software development.
o And much, much more.
Why Testing?
o Software contains defects. The hide.
o Defects can cause software failures.
o Failures can cost money, and even be mortal!
o Testing assures the quality of the software.
o Testing accelerates software development.
o And much, much more.
What is Testing?
The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they
satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
Do Software Testing. Right.
o Testing has to be planned.
o Testing costs easily up to 40% of a project's budget.
o Testing has to be performed in a reasonable way.
o Testing shall be independent and objective.
o Testing has to be managed.
Software testing is a fundamental part of professional software development!
Type of Tests
o Unit Testing
o Integration Testing
o E2E Testing
o Penetration Testing
o Performance Testing
o Regression Testing
o a11y Testing
o Stress Testing
o Fuzz Testing
o A/B Testing
o User Acceptance Testing
o Usability Testing
o i18n Testing
o Smoke Testing
Test Levels
o Unit Testingo Test individual units in isolation.o Find defects in software components (e.g., functions, classes).o Done by developers.o In general, white-box tests.
What is a Unit Test?
Test Levels
o Unit Testing
o Integration Testingo Test communication/interaction of units (i.e., their interfaces). o Maybe separate unit integration and system integration tests. o Done by developers. o Primarily white-box tests, but also black-box tests.
What is a Integration Test?
Test Levels
o Unit Testing
o Integration Testing
o System Testingo Test complete, integrated system. o Evaluate system compliance with specified requirements. o Stress, performance, usability etc. testing. o Done by (external) testers. o In general, black-box tests. Additional white-box tests possible.
Test Levels
o Unit Testing
o Integration Testing
o System Testing
o Acceptance Testing o Test complete, integrated system. o Evaluate system compliance with specified acceptance criteria. o May be performed at various times during development. o Done by customers/users. o Only black-box tests
App
o Technologieso Python
Python in Business
App
o Technologieso Pythono Flasko Flaskro NGINXo uWSGI
o Environmento Ubuntu Server (VM)o PyCharm
Setup
git clone https://github.com/gtavasoli/testing-workshop.git
cd testing-workshop
App Structure
o Unit Tests
o Simple Flask Apps
o Flaskr
o JMeter Profiles
Unit TestUnit test in python
Unit Testing
o Sample 1 – Calculatoro Test operatorso Floating point divisiono Division by zero
o Sample 2 – Employeeo Setup & Teardowno Setup & Teardown Class
Test Double
Test Double
o Dummy → I do nothing at all but to fill parameter list
o Stub → canned Answers
o Spy → stubs + interaction recording (for late interaction expectations verification)
o Mock → stubs + expectations on interaction
o Fake → I seem real but no
What Are Mocks?
o The term 'Mock Objects' has become a popular one to describe special case objects that mimic real objects for testing.
o Mocking makes unit testing easier!
o Verifies behavior (calls) to a method.
o A mock object simulates the behavior of real objects in controlled ways.
What Are Mocks?
Problems Mocks Solve
o Eliminates dependencies in the CUT (class under test)o Isolate Unite Tests
foo(x) bar(x)
Copyright: Based on “Mocking Strategies” – slides from 26 to end of mocking sample
Problems Mocks Solve
o Tests methods that have no return value
How do we know that bar(x) has been called?
Problems Mocks Solve
o Tests error handling
How do we generate this exception???
Problems Mocks Solve
o Eliminate dependency on database callso Speed up testing!
o Reduce test complexityo Don’t have to write complex logic to handle behavior of methods not under test
o Don’t have to wait to implement other methods
Problem: Friend Finder App
o Create a method to return a new, random friendship candidateo Must not show the same persono Must not show someone the user has already visit it.
Easy Method
get_next_person() get_random_person()
“Surely no one could
have seen EVERYONE
in the database!!!”
Write a Unit Test...
May be it Works!!!
May be don’t!
Easy Method
What if knew the result of get_random_person()???
Patching
But what if we call it multiple times???
Problem: Matching in Friend Finder App
o When a user send friend request…
o If the other user “likes” it:o Send them both a message with contact info
o If the other user “dislikes” it:o Let the user down gently…
o If the other user hasn’t evaluated yet:o Display the “give it time” message
Implementation
How do we test this??? No return values!!!
Behavior Verification
Mocking Exceptions
Simple JSON reader
How do we test something like this???
Testing JSON reader
o Test parsing a valid file
o Test an IOError (i.e. file missing)
o Test a ValueError (i.e invalid JSON)
How do I test open()
o Let’s just create a sample file!
Can you even mock a builtin?
What? Why Not???
o open() returns a File object
o open(filename).read()
o So we really need to mock File.read()o But it’s an instance!!! Oh no!
o Have you tried solving it with Mocks???
Mocks returning Mocks?
What about IOError?
What about ValueError?
Warning
o A warning about over-isolatingo Many people fall for relentless isolationo Magic tool for helping in isolating parts
o Mocking is about object communication and interface discoveryo Using it for isolation is a misuseo Specially from 3rd party code
o General rule of thumb: "do not mock what you don’t own"
Warning
o Avoid over-mockingo Duplicating implementation code in the testso Keep production and test code in sync
o We should refrain from this and try as much as possible to use real collaborators when possible.
Testing/Coding is so hard!!
Type of Software Testing
Non-functional Testing
o Performance Testingo To identify any bottlenecks or performance issues rather than finding bugs in a softwareo Speedo Capacityo Stabilityo Scalability
Non-functional Testing
o Performance Testing
o Load Testingo The process of testing the behavior of a software by applying maximum load in terms of software accessing and manipulating large input datao Toolso Load Runner, AppLoader, IBM Rational Performance Tester, Apache
JMeter, Silk Performer, Visual Studio Load Test, etc.
Non-functional Testing
o Performance Testing
o Load Testing
o Stress Testingo Includes testing the behavior of a software under abnormal conditionso Applying the load to the system and taking over the resources used by the software to identify the breaking point
Non-functional Testing
o Performance Testing
o Load Testing
o Stress Testingo Shutdown or restart of network ports randomlyo Turning the database on or offo Running different processes that consume resources such as CPU, memory, server, etc.
Non-functional Testing
o Performance Testing
o Security Testingo Confidentialityo Integrityo Authenticationo Availabilityo Authorizationo …
Performance Testingo Methodologieso Real User Monitoring - RUMo Synthetic Testing
o The Recipeo Step 1 – Collect Datao Step 2 – Analyzeo Step 3 – Experimento Step 4 – GoTo 1
Targets in Performance Testing
o Things we measureo Availability or uptimeo Concurrencyo Throughputo Response timeo Network utilizationo Server utilization
Tips
o Beware of failures
o Be aware of caching
o Automate bookkeeping
Tutorial 1 – Simple Test
o Step 1 – Start JMeter
o Step 2 – Create a TestPlan
o Step 3 – Create a Thread Group (Users)
o Step 4 – Add Sampler (HTTP)
o Step 5 – Add Listeners
o Step 6 – To Run the Test
Tutorial 2 – Using Assertions
o Response Assertion
o Duration Assertion
o Size Assertion
o HTML Assertion
Tutorial 3 – Flaskr
o Simple Microblog With Flask
o Parametric Request
Tutorial 4 – JMeter Plugins
o JMeter Plugin: Response Times Percentiles / Distribution
oTail Latency
The Tail at Scale – Dean and Barroso 2013
Tutorial 4 – JMeter Plugins
o What happens when there is not just a single server involved in servicing a request, but 100! (fan-out query, contribution of a large number of microservices)
Tutorial 4 – JMeter Plugins
Tutorial 4 – JMeter Plugins
o Tail Latency
o Root Causeso Shared resources (memory, CPU cores, CPU caches, shared file system)o Background tasks running on the servero Maintenance activities (GC, Log rotation)o Queues in general
https://blog.acolyer.org/2015/01/15/the-tail-at-scale/