Why Johnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners
What, why and how? Johnny Bigert, 2012-10-25 [email protected].
-
Upload
zakary-leyman -
Category
Documents
-
view
213 -
download
1
Transcript of What, why and how? Johnny Bigert, 2012-10-25 [email protected].
![Page 2: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/2.jpg)
My View on Unit TestingPart of something bigger – testing on many
levels1. Fast and early feedback!
Tight feedback loop, results within seconds Early implies lower cost
2. Confidence! Less fear of refactoring
3. Better code! Testability requires decoupling and usable APIs
4. Reliability! Possible to remove all non-determinism
![Page 3: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/3.jpg)
Definitions and TerminologyUnit test, functional test, system test – what does it mean?
![Page 4: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/4.jpg)
Definitions and TerminologyThere are no definitive answersDifferent dimensions of automated tests:
Level – Unit, component, integration, systemAPI – Developer test, external testCharacteristic – Functional, non-functional
Acceptance testing, regression testing
![Page 5: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/5.jpg)
What Level Are We Testing On?Unit – The smallest unit of program code, normally
a single function, class with methods or data structure with functions
Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files.
Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server)
System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system
![Page 6: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/6.jpg)
What API Are We Using For Testing?Developer tests – Function calls to an internal
or external interface, normally test and production code running in the same process
External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.
![Page 7: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/7.jpg)
What Aspect Are We Testing?Functional – The behavior of the system,
normally expressed as a step-by-step instruction with some alternative paths
Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.
![Page 8: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/8.jpg)
FunctionalDeveloper External
Unit “unit test” N/A
Component “component test”
Integration “functional test”
System N/A “system test”
“Component test” – personal favorite. Provides stability just like a “functional test” (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a “unit test” (quick to write, easy to mock out problematic parts, results within milliseconds).
![Page 9: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/9.jpg)
Non-FunctionalDeveloper External
Unit N/A
Component e.g. test threading model of a component
Integration e.g. measure CPU consumption
System N/A e.g. “load test”
![Page 10: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/10.jpg)
Choose How to TestWhen choosing, consider
Cost (of writing, maintaining, execution resources etc.)
Feedback loop (time)Reliability
Trade-off example: Mocking or using system x?Mocking: cost of writing and maintaining, risk
of modeling wrong behavior.Using: cost/risk of non-determinism, cost of test
resources and longer execution times.
![Page 11: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/11.jpg)
Regression And Acceptance TestingDefinition: “regression” or “regression bug”
Something that was once considered “done” but does not fulfill requirements anymore
Acceptance testing: the use of a test to determine if a feature fulfills the requirements
Acceptance tests becomes part of the regression test suite once a feature is “done”
![Page 12: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/12.jpg)
How to Unit TestNecessities of effective unit and component testing
![Page 13: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/13.jpg)
DependenciesAssumption: we want to achieve fast
feedback and reliable testsNote, there are other kinds of tests that also
have meritRemove non-determinism: we need to be in
controlRandomness, time, file system, network,
databases, multi-threadingRemove dependencies to non-deterministic
components
![Page 14: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/14.jpg)
DependenciesWhat kind of problematic dependencies to
you have in your code base?
![Page 15: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/15.jpg)
Getting Rid of Dependencies, C++Example: Randomnessclass A {
public: void foo() { bool coinflip = rand() % 2 == 0; if(coinflip) … }};
Problem: dependency to non-deterministic component (randomness)
![Page 16: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/16.jpg)
Getting Rid of Dependencies, C++C++ solution: introduce interface and use
dependency injection
class IRandomness { virtual bool coinflip() = 0;};
![Page 17: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/17.jpg)
Getting Rid of Dependencies, C++class A {
public: A(IRandomness &r) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … }private: IRandomness &randomness;};
![Page 18: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/18.jpg)
What Is a Mock Object?Conforms to interfaceProgrammable behavior
For example, “if someone calls your send function with argument ‘hello’, return 7”
Can verify function invocations
Mock frameworks availableE.g. gmock (Google)Much more elegant in reflective languages
![Page 19: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/19.jpg)
Getting Rid of Dependencies, C++Test code:void testFoo() {
RandomnessMock random; // inherits IRandomness A a(random); a.foo(); …}
![Page 20: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/20.jpg)
Getting Rid of Dependencies, C++Production code:void main() {
Randomness random; // inherits IRandomness A a(random); …}
![Page 21: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/21.jpg)
Bi-Directional Dependencies
sendData
(inherits)onDataReceived
(inherits)Application
Network
INetworkReceiver INetworkSender
![Page 22: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/22.jpg)
Getting Rid of Dependencies, CFunction pointers = hard-to-read codeLink-time tricks
Link with other translation unit defining e.g. rand()
![Page 23: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/23.jpg)
Getting Rid of Dependencies, CRandMock m; // mixed C/C++ test code
RandMock &getRandMock() { return m; // used by test case to access mock}int rand() { m.rand(); // forward to mock object}
Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain
Or clean C solution
![Page 24: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/24.jpg)
Code CoverageDon’t focus on high levels of code coverage
per seNecessary, but not sufficient
Instead, focus on covering the most important user scenarios
After that, use code coverage to determine which code has not been covered
Analyze to understand what user scenarios are missing
![Page 25: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/25.jpg)
Unit Test FrameworksTest runners/frameworksGTestUnitTest++CppUnitC Unit Testing Framework
Mock frameworksGMockHippoMockppAmop
![Page 26: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/26.jpg)
VisionAgree on what to aim for
![Page 27: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/27.jpg)
Fast Feedback ExampleFrom a previous job:Proprietary hardware/software/OS platformVery expensive hardware = bottleneck in
testingEmulator to test software locally on Linux
machineRan unit tests in emulator, turn-around time
= 15 minsUnacceptable
![Page 28: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/28.jpg)
Fast Feedback ExampleMain problem: emulator start-up timeChallenge: proprietary types used (e.g. string
classes)Solution: fake proprietary types (string with
std::string, many others just empty implementations)
Ran unit tests on local machine,turn-around time < 15 seconds (from 15 mins)
![Page 29: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/29.jpg)
Fast Feedback ExampleInvest to speed up feedbackInvest to remove testability obstaclesUnit testing should be easy, fast and fun
![Page 30: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/30.jpg)
Development ExcellenceDefect prevention
Continuous improvement - feedback from bugs in order to improve software development process
Relentless testingRegression detection
Continuous integration running all regression tests often
Focus on both functional and non-functional requirements
Always releasableKeep CI builds greenKeep the product working
![Page 31: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/31.jpg)
Continuous IntegrationStaged
Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins
Stage 2 – extensive functional tests, smoke non-functional tests – 1 hour
Stage 3 – full functional and non-functional regression tests , static and dynamic analysis – 12 hours
Run stages perCommit (fine-grained tracability, expensive),
orRegularly (1. commit, 2. hourly, 3. nightly)
![Page 32: What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se.](https://reader035.fdocuments.us/reader035/viewer/2022070306/55173a3255034603568b60f1/html5/thumbnails/32.jpg)
Code Analysis ToolsStatic analysis (run on source code)
Lint – free, local analysis only, lots of false positivesKlocwork – commercial, global analysisCoverity – expensive, global analysis, good
precisionDynamic analysis (need representative data)
IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation.
Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.