Johan Seland Advanced Software Testing · • A black-box or system test • Will often require...
Transcript of Johan Seland Advanced Software Testing · • A black-box or system test • Will often require...
![Page 1: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/1.jpg)
SINTEF Group Head Office
Johan Seland
1
Geilo Winter School 2013
Advanced Software Testing
![Page 2: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/2.jpg)
SINTEF Group Head Office
• Solution is in the final branch on Github
• git clone git://github.com/johanseland/BowlingGameKataPy.git git checkout origin/final
2
Solution Example for the Bowling Game Kata
![Page 3: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/3.jpg)
SINTEF Group Head Office
• Testing Frameworks
• Writing good tests
• The mechanism of change • Refactoring
• Black-Box testing tools
• Continuous Integration
3
Overview of Lecture
![Page 4: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/4.jpg)
SINTEF Group Head Office
Regression Tests
System Tests
Integration Tests
Unit Tests
4
The Test Pyramid
![Page 5: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/5.jpg)
SINTEF Group Head Office
• Unit Tests: • Tests individual units of code • Function/Class level
• Integration Tests: • Tests how components communicate • Example: Is grid correctly initialized from init file?
• System Tests: • Are results valid? • Performance/Scaling/Resources/Stability
• Regression Tests: • Test for old bugs
5
The Test Pyramid Regression
Tests
System Tests
Integration Tests
Unit Tests
![Page 6: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/6.jpg)
SINTEF Group Head Office
• Testing software is made easier by test frameworks • At least for unit, integration and regression tests
• Typical facilities
• Automatic test detection • Minimize boilerplate-code • Assertion functions • Test execution
• Tests run in isolation • Command-line parsing • Standard output formats
• Common terminology for talking about tests
6
xUnit Test Frameworks
![Page 7: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/7.jpg)
SINTEF Group Head Office
• Assertions • The actual tests
• Test case • Function invoking assertions
• Suites • A collection of cases
• Fixtures • (Data) structures set up before tests
• Mocks objects • Objects that mimic behavior of real objects
7
xUnit Testing terminology
![Page 8: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/8.jpg)
SINTEF Group Head Office
Every language has a xUnit-based Test Framework
• C++ – GoogleTest, Qtest,
Boost.Test • Fortran:
– pfUnit, fUnit • .NET (C#, F#, etc.)
– Visual Studio Testing Framework
• Java – Junit
• Python – UnitTest
• R – Runit
• Matlab
– MUnit
• Open Source
8
![Page 9: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/9.jpg)
SINTEF Group Head Office
• Split test and source files! • Consider building application as
• Library + "App"-executable • Test-executables
• Separate project for different test layers
• Unit vs Integration vs System vs Regression
• make should build everything
• make check should run as reasonable amount of tests • Unambiguous answer if tests pass or fail
9
Test code organization
![Page 10: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/10.jpg)
SINTEF Group Head Office
• Test code is not a second-class citizen • Requires thought, design and care • It must be kept as clean as production code
• Agree on naming convention
• TestFoo() or FooTest()?
• Test one “concept” per test • Often multiple tests per class/function • Kent Beck school: One assert per test
10
Testing best practices
![Page 11: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/11.jpg)
SINTEF Group Head Office
• Be intelligent when writing unit tests! • Do not inflate their number
• Each test should have a meaning
• Collapse/Cleanup tests when cleaning up the production code • Typically test:
• Zero-case • Trivial Case • Corner Cases (division by zero?) • Error handling
11
What to unit test?
![Page 12: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/12.jpg)
SINTEF Group Head Office
• Fast • Tests should be fast. • You won’t run slow tests frequently
• Independent • Tests should not depend on each other
• Repeatable • They should be repeatable in any environment • Otherwise you have an excuse why they fail
• Self-Validating • Tests should have a boolean output • No manual evaluation should be needed
• Timely • Should be written just before the production code that make pass
12
Clean tests – F.I.R.S.T.
![Page 13: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/13.jpg)
SINTEF Group Head Office
• Short answer: You should only test the public interface of a class
• Reality: The crucial computation happens in private methods
• Possible fixes: • Split into impl-namespace • Mark as protected instead and let test-class be subclass • Let test be a friend class (C++) • Mark as package-private (Java)
13
Testing private methods
![Page 14: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/14.jpg)
SINTEF Group Head Office
• HERE BE DRAGONS!
• Floating point in general is not associative • a op b != b op a • Beware of parallel computations
• Floating point is sensitive to compiler settings!
• Fused operations • Compiler optimizations • Flush to memory
• Know your precision
14
Testing Floating Point Computations
![Page 15: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/15.jpg)
SINTEF Group Head Office
• For numerical algorithms, an estimate of the tolerance is needed • You can not simply test for equality • Absolute vs Relative error • Too low tolerance: Test might fail when implementation is correct • Too high tolerance: Test will not detect real errors
• Do you have (or can derive?) an error estimate/bound • Might be publishable itself! • Might just be asymptotic with unknown coefficient
15
Testing Floating Point cont'd
![Page 16: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/16.jpg)
SINTEF Group Head Office
• Unit tests is not the place to test for stability of iterative methods • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the implementation is correct
• Unit tests for iterative methods should test the implemenation • Constant input fields • Convergence criteria • Detecting invalid input
• Can you split it out so each substep has an analytical solution?
16
Unit Tests for Iterative Methods
![Page 17: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/17.jpg)
SINTEF Group Head Office
• Use analytic cases when available!
• Write tests to compare results between previous run of simulator • Typically require manual inspection • Often require dedicated post-processing tools
• Bitwise reproducibility is not attainable
17
Strategies for Testing Floating Point
![Page 18: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/18.jpg)
SINTEF Group Head Office
• Often you can not rely on real objects for tests • Databases • Sockets • Huge datasets • Displays • Amazon Instances
• FAKE objects have working implementation with shortcuts
• In-memory filesystem, constant grids
• MOCK objects are pre-programmed with expectations • Mock-libraries make it easier
18
Mocks and fakes
![Page 19: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/19.jpg)
SINTEF Group Head Office
• Use of InputFileReader is hardcoded in class (tight coupling)
• Problems: • I/O might take a long time • Input files must be distributed with tests
19
Using Fakes: Non-testable code
![Page 20: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/20.jpg)
SINTEF Group Head Office
• We instead use an abstract class defining our behaviour
• Parameter object is passed to simulator
20
Using Fakes: Introduce Parameter Object
![Page 21: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/21.jpg)
SINTEF Group Head Office
• In the test code, we add simplified version
• And pass this to simulator
21
Using Fakes: Introduce fake object
![Page 22: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/22.jpg)
SINTEF Group Head Office
Testable code will often have: • More classes
• Should not be a problem in modern IDEs • Use ECB in Emacs
• Smaller classes
• That follow the Single Responsibility Principle (SRP)
• Looser coupling • One of the benefits of object-oriented languages
22
Fake objects conclusion
![Page 23: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/23.jpg)
SINTEF Group Head Office
• Why do you want change?
1. To add a feature 2. To fix a bug 3. To improve the design (refactoring) 4. To optimize
23
Changing software
![Page 24: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/24.jpg)
SINTEF Group Head Office
Add a Feature Fix a bug Refactoring Optimizing Structure Changes Changes Changes Changes? New Functionality
Changes - - -
Functionality - Changes - - Resource Usage
- - - Change
24
The mechanisms of change cont'd
![Page 25: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/25.jpg)
SINTEF Group Head Office
• The main thing that distinguishes legacy code from non-legacy code is tests, or rather a lack of tests." Michael Feathers
• Alternative definition:
Code you are afraid to change, cause you can not see the consequences
BRING IT UNDER TEST!
25
What about legacy code?
![Page 26: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/26.jpg)
SINTEF Group Head Office
Refactoring
Code refactoring is the process of changing a computer program's source code without modifying its external functional behavior in order to improve some of
the nonfunctional attributes of the software
• Make it work • Make it right
– Maintainability – Extensibility
26
![Page 27: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/27.jpg)
SINTEF Group Head Office
Some refactoring techniques
• Rename field – Change a name into a new one that better reveals its purpose
• Extract method – Turn part of a larger method into a new method
• Move field – Move to a more appropriate class or source file
• Extract class – Move part of the code from an existing class into a new class
• Generalize Type – Create more general types to allow for more code sharing
• Many more at refactoring.com
27
![Page 28: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/28.jpg)
SINTEF Group Head Office
• IDEs have some support for automatic refactoring • Guarantees that behavior does not break • Java IDE have
• C++ is probably the most difficult language • Templates are specially difficult • But we are getting there as well (QTCreator in particular)
• Lean on the compiler 1. Alter declarations to cause compile errors 2. Navigate to errors and make changes 3. Rerun tests!
• Pair programming!
28
Refactoring in practice
![Page 29: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/29.jpg)
SINTEF Group Head Office
• A plethora of tools to analyse your program and tell you something about is status • At the source-code level or program level • Commercial or open-source
• White-box testing is biased
• You generally write them yourself
• Testing tools does not lie
29
Other testing tools
![Page 30: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/30.jpg)
SINTEF Group Head Office
• Programs that look at your source code • Identifies common errors
• Bounds checking for arrays • Memory leaks • Resource leaks • Stylistic errors • Code duplication
• Compute code metrics
• Subject to false positives and negatives
• Common tools: Cppcheck, Cpplint
30
Static Code Analysers
![Page 31: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/31.jpg)
SINTEF Group Head Office
• Tools that monitor the execution of a program • Allows you to understand their behaviour when running on real code
• Used to detect:
• Performance metrics • Memory leaks • Parallelization errors
• Program must be instrumented
• Compiler switch • Run in a virtual environment • Often program execution is slowed down dramatically
• Yet another benefit of many separate test programss
• Example Tools: Valgrind, gcov, Intel Parallel Studio
31
Profilers
![Page 32: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/32.jpg)
SINTEF Group Head Office
• Measure which code is executed at all
• Used to detect if you have code that is not covered your tests • Are you testing each direction of an if-statement? • Is it code you are not executing
• Example programs: gcov (GCC stack), Cobertura (Java), coverage (Python)
32
Code coverage
![Page 33: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/33.jpg)
SINTEF Group Head Office
CONTINOUS INTEGRATION
33
![Page 34: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/34.jpg)
SINTEF Group Head Office
• If daily builds are good • Continuous builds are better
• If daily testing is good • Continuous testing is better
• Detect issues as early as possible
• Jenkins is a web-service that happily builds code and executes test all-day • Jenkins will build on all your platforms • Execute long-running tests • Syndicate results across builds
34
Motivation
![Page 35: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/35.jpg)
SINTEF Group Head Office 35
![Page 36: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/36.jpg)
SINTEF Group Head Office
• Instant feedback • Everyone can see status
• Latest executable available • Build on all platforms all
the time
Downside: • Should really be run on dedicated server
36
Benefits
![Page 37: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/37.jpg)
SINTEF Group Head Office
• Email only when build breaks/tests start to fail
• Per project participant list • If people start filtering emails you have lost
• Everyone can look at build configurations
• Avoids mysterious cron jobs on private workstations • Jenkins is not a high-security system
• Do not build on your own workstation
• Highlights new dependencies
• Use clean builds
37
Jenkins best practices
![Page 38: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/38.jpg)
SINTEF Group Head Office
You need something not on Jenkins server • Matlab • Windows • GPUs • Fluent
• Hudson can start jobs on slaves • Extremely easy to set up
38
Jenkins slaves
![Page 39: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/39.jpg)
SINTEF Group Head Office
Jenkins tips
• Performance monitoring – Runtime for test is in XML-report – Small Python script to extract it
• Correctness monitoring – Compare output to prev. output
• Get cron jobs in there as well
• Back up Hudson!
39
![Page 40: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/40.jpg)
SINTEF Group Head Office
More possibilities
• Store profiling information
• Validate single-thread vs parallel implementations
• Validate against gold standard
• Analyze compiler warnings
• Static code analyzer
• Check for memory leaks (Valgrind)
• Let managers/supervisor know about available metrics?
40
![Page 41: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/41.jpg)
SINTEF Group Head Office
• Server was set up in summer 2010
• Specialized servers added later • GPU build server • Windows build server
• Informal tutorial session
• Quickly adopted for many projects
• People continue to use it!
41
Jenkins at SINTEF Applied Mathematics
![Page 42: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/42.jpg)
SINTEF Group Head Office
• Social Processes • Code-Reviews • Pair-Programming
• Bug/Issue tracking
• Acceptance Testing
• Fitnesse Framework
• Various code metrics
42
Thing not covered
![Page 43: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/43.jpg)
SINTEF Group Head Office
• You need a testing strategy
• Your testing strategy should consist of a battery of • White-box testing at the source level • Black-box testing at the program level
• Automate as much as possible
• Minimize the amount of human parsing necessary
• Execute tests as often as possible • Continuous Integration is an enabling technology for this
43
Concluding remarks
![Page 44: Johan Seland Advanced Software Testing · • A black-box or system test • Will often require manual inspection of results • Goal of automated tests should be to decide if the](https://reader034.fdocuments.us/reader034/viewer/2022050303/5f6c217e4177362a4d150299/html5/thumbnails/44.jpg)
SINTEF Group Head Office
Reading list
44