White Box Testing• Agenda• The need• Benefits• Code review• Unit tests and automation• cppUnit Concepts• Memory leaks• Quantification• Coverage• APIs• Challenges• Best Practices• Don’ts• Metrics• Cyclomatic Complexity

Need for White Box Testing• Better to test anything from outside-in as well as inside-out• Most of the functional and performance issues arise due to bad coding• Ultimately the code matters – hence test the code left right and center• Black box tests are used to check cause and effect without getting into

internals• White box tests do the same by getting into the internals of every program• Every developer is by default a white box tester• Medical field requires general physicians and surgeons• Certain problems can be identified and solved only by surgeons• A person may look healthy but internally the person may have high bp and

sugar; they are not visible externally• So far, the importance of white box testing is not upto the mark• Product companies definitely need white box testing• If your application must scale to a very large extent, white box tests are

inevitable

Black Box Cannot Test These• It takes long time to check memory leaks• Any exceptions that happened but unnoticed or unhandled• Which portion of if condition is evaluated• What path the program took to achieve the end result• For every request what is the memory and cpu time taken• Is there any dead and unused code• Is there any extra code that is not needed• What are the potential breaking points in code• Is the code compliant to best practices• In large business apps, the data paths will be too large to test; it will

require senior business analysts; if done thru white box mode, normal programmers can bring out great bugs

Diagnostics tools• IBM Rational Purifyplus• JProbe• Parasoft tools• HP Mercury Diagnostics• Dynatrace• .Net profiler• AppPerfect• Netbeans• AQTime• All tools work on instrumentation and trace options• Some tools help in analyzing system memory space as well• Usually the system cpu executions are privileged, and hence we

may not get access to those

Business Benefits• Ensures that the core building blocks are intact, when

adding new features and modules• Go to market with confidence on product stability• Provides greater visibility in terms of technical and

internal compliance of the product• Vulnerability testing always needs a greater amount of

white box testing• Resource utilization reduction is key for marketing.

Products that use less memory and bandwidth are the most welcome ones

• Keeps business leaders informed on the underlying strengths and weaknesses of the application platform

Understand Design• Honestly speaking, how many of us use proper design before

coding?• Can the same happen to real estate industry without a proper floor

plan or for pharma industry without proper process and formula for drugs?

• Design has 4 parts – UI, database, Interface and Logic design• UI design is key aspect when it comes to usability and compliance• Database design ensures scalability and storage• Interface design is key for plug and play during integration• Logic is important for the business flow of the application• Logic design can be a flow chart or text algorithm or pseudo code• Most of the modelers do have pictorial representation of design• Ideally the code must reflect the translation of pseudo code• Traceability of design elements to code is one of the key areas we

need to look into• A change in design may impact several parts of code

Code Review• Most of the times, this is ignored or not done to its true

spirit• Code review must have design and code side by side• Code review can save upto 30% of testing cost at a later

point in time• Usual problems happen due to cut and paste of code

where it is not required• When there is a transition from one developer to

another, code goes thru challenges• Quick fixes on big projects cause enormous amount of

turbulence on production systems• Ideally senior people must review code• Spend at least 10% of coding time on code review

Code Review Checklist• Just try these 10 critical points in daily life1. No hard coding2. Ensure loop termination conditions3. No object creation inside loops4. Close every object that you open5. Give comments to every code block6. Use database connection only when you need7. Remove unused variables and code portions8. Follow a consistent naming convention9. Try to reduce overloading methods very often10. Check SQL or file status soon after every disk

operation

Unit Testing• The smallest piece of code that can be independently

run and tested is a unit• Unit can be a page or a function or a method within a

class or a whole program itself• It is the developer’s responsibility to unit test the code• Unit test must focus on data type of every parameter• Focus on data format of every parameter• Focus on boundary values of every parameter• For any given code, what deliverable do we provide as

part of unit testing?• Most of the times it is just the trust on the developer• Developer must spend at least 30% of coding time on

unit testing alone

How to draw boundaries

• What is the line that demarcates unit and other tests?– One expected output out of a function– No chaining

• When to stop unit tests?– When a set of parameter combinations are

tested, stop the same– Boundaries and equivalence partitions must

be tested before exit

Unit Testing Checklist• Are the naming conventions followed?• Are the comments done properly?• Is there any hard coding?• Test case for custom exceptions.• Test case for system exceptions.• Test case for “body” of an if condition.• Test case for “body” of an else condition.• Test case for every loop termination.• Test case for every recursion termination.• Test case for pointers release (for memory leaks).• Test case for every procedure entry and exit.• Test case for every parameter validation for procedures.• Test case for resource release (Closing DB connections,

releasing objects etc.)

Unit Test Matrix• A single test must make a call to a function with some

parameters combination and expect return value(s)• If a function has got multiple parameters that are of

different types, then combinations of all parameters are possible and it will become a Cartesian product table

• As the first step, a matrix needs to be prepared with all such combinations

• Then the matrix must be optimized to reflect the practicality of the inputs

• Possibilities of return paths must also be documented• The samples may go in 100s but not to worry on• Since automated unit tests run extremely fast without

human inputs, 1000s of tests can be executed in minutes

Automate Unit Tests• It is not practically possible to give all combinations of the inputs to

programs and run• To reduce time, we need to write a program, that calls the program

under test and pass parameters• Ideally the test program must be in the same language as the

program being tested• This will help in maintaining the homogeneity of the data type of

parameters passed• Since developers know the programming language, it is easy for

them to write the test program as well• These are called test suites or test scripts• The investment on the unit test automation is one time as long as

changes are minimal• The person who automates test for a program must know the

program well

The core of unit automation• Class A

– Method m1(p1, p2)– Method m2(p3, p4)

• TestClass TA– Testcase tm1

• ta = new Class A• rc = ta.m1(4, 5)• If (rc == exp_result)

– Pass

• Else– Fail

• // destroy ta

Level 1 Automation• Crude method – quick solution for short

term projects• Write wrapper callers • Hard code parameters• Cut and paste test code with no hesitation• Spend little time on test creation• Test code can be throw-away soon• Simple logs, more manual result analysis

required

Level 2 Automation• Parameters to tests must be separated

from test code• You can use csv or xml or any format for

test data• Useful to run same test on larger test data

sets• When a mixture of customers use the

product, the data set also will change largely – this will be helpful in that case

• Use your own test logs with more details

Level 3 Automation• Use test tools such as cppUnit or jUnit etc.• Build auto recovery options if tests fail in

between• Ensure every test has start-run-cleanup

structure• Use this on long term test engagements• When tests are to be reusable, this is the

best option• If you know cppUnit, the concepts are

exactly the same in nUnit, jUnit etc.

Stubs and Drivers

• When the program under test needs some pre-built data or state, we need to mimic the same

• This is a short-cut approach, but it serves the purpose

• Stub is the one that mimics a called function• Driver is the one that mimics a caller function• When we use stubs and drivers, it is a must that

we ensure that the data created by them are destroyed soon after the test

Testing APIs • APIs do exchange of data from one piece to another• APIs are usually consumed by many independent

programs• APIs in general are meant to provide a window of

information to 3rd parties• One change in API will affect all consumers at once• Most of the current day apps provide APIs• We can send tweets from our application using APIs• We can send emails from Amazon SES thru APIs• API calls must be assumed to be context neutral• Some APIs do require authentication mechanism• APIs fall under external interfaces

Wrapper Tools for APIs• A set of business critical APIs are now being

used in all EAI scenarios• FIX, SWIFT, News feeds, HL7, HIPAA, Part 11

standards etc• A variety of commercial tools are available in

each category• When messages are sent, these tools validate

the structures of XML or EDI against the templates

• Also, there are different versions of these APIs exist like html 5 etc.

cppUnit• Write test wrappers in cpp for cpp programs• If you are familiar with cpp programming, you may not

require cppUnit !• There are many versions of cppUnit itself as it is open

source – many people modified it for their convenience• You can write your own test library and framework• A set of pre-built classes and methods are already made

available to minimize test coding time• Testcase, Testsuite, Assert are the important aspects in

cppUnit• Testcase is the basic block – every single test• Testsuite contains multiple tests ( a batch)• Assert is the actual verification point where actual result

is compared with expected result

cppUnit• Fixture – known set of objects for test• setUp is the place where we create testcases• tearDown is the place where we remove all

created tests• Commonly used asserts

– Assert – Assert_message– Assert_equal– Assert_throw– Assert_equal_message

Instrumentation• White box tools and profiler tools use one single concept called

Instrumentation• Source code is compiled and built as binary• These tools understand the format of these binary files – where

function entry points are made, how they exit, where stack is maintained etc.

• Before profiling, they instrument the binary – inject their own log codes in the binary locations

• During the execution, the instrumented code is run and every detail is logged using these injected code (such as entered function x, executed loop y, exited function y, destroyed object o etc.)

• Using these logs, the tools provide internal details• This is very similar to sending a probe inside our body and doing a

scan of that probe

Overheads• When profilers run along with the instrumented

programs, it adds load to the machine• This is a necessary overhead we need to take• The performance overheads are negligible when

compared to actual object size• The statistics that are collected out of the

programs will vary from time to time; but if the variance is beyond 5%, then the tool has real issues

• Also, do not run 2 competing tools on the same machine. The effects are unpredictable

Memory Leaks• When a memory pointer is declared and

initialized and it is not released, it is a leak• Even in managed code sections, the leaks are

there for a time till the garbage collector acts• Leaked memory locations are useless and

vulnerable• One program leaking 10 bytes of memory per

call – can bring down the system in a day when 1000s of users use the same for 100s of transaction

• This is very important when we deal with device drivers and embedded systems

Memory Profiling

• This happens at runtime• Profiling is the basic block of performance

engineering• Concept of instrumentation and tracing happens• Call graph will tell what is the path the program

touched in terms of functions• Statistical profilers talk about memory and cpu• Memory consumed at that instance by variables

and functions• Memory released at any given point of time

Memory Quantification

• What is the consumption of memory and cpu is quantified

• Usual quantification happens in basic unit of measurement

• UOM can be KB or machine cycles• How many cpu cycles are spent in this

method when this is running• How many cpus cycles are spent to

initialize the class

Memory Usage

Method Kbytes

logon 138

do_registration 1607

view_bill 254

pay_bill 2897

dispute_bill 4876

view_summary 2346

get_call_details 1345

logon1%

do_registration12%

view_bill2%

pay_bill22%

dispute_bill36%

view_summary17%

get_call_details10%

Buffer Overflow• When arrays or memory blocks are declared and we try

to access beyond those boundaries it is called overflow• When program variable space is packed, the overflows

can cause damage to program counters or other stack areas

• When stack is compromised, the result is program abort• Dynamic arrays are also vulnerable to overflow when

people do not keep track of the max allocations• Buffer overflow can result on network buffers also when

the response buffer is underestimated• Protection of buffer overflow is usually made by

providing padding space to every memory space – but this needs to be done against memory optimization

Object Issues• Objects require space and the methods need

cpu cycles• When objects are declared and not destroyed,

they also cause leaks• Unused objects are vulnerable areas for attack• Closure and nullifying objects are essential for

optimized use of memory• Creating a series of objects in a multi user

environment can cause spiral problems

Function-wise Quantification• Every function has variables and statements• Parameters and variables take space from

symbol table as well as stack• Statements do take cpu cycles based on how

many times they get executed• When one function calls another, there is a huge

amount of context switch happening at heap and program registry levels

• How many times these context switches happen will also determine the complexity of the programs

Test case for coverage

• A path is defined as a route thru which a specific functionality is achieved by calling a series of steps

• If programs flow from top to bottom, it is easy• The more conditions, the more complex it is• Coverage must ultimately make up to 100% of the tested

function• Purecoverage will help in achieving this • Covering statements based on design is the best method• But when design is not in sync with program, we need to

apply simple logic to hit every statement that is present in a program

Coverage Output

Line Code Time

47 myproc (p1, p2) 7181

48 if (p1 < p2) 2

49 call proc1(); 388

50 if (p1 > p2) 0

51 call proc2(); 0

52 if (p1 == p2) 2

53 call proc3(); 6789

Coverage for conditions

• a < b && c > d– In the above c>d is evaluated only when a < b comes to true

• a < b || c > d– In the above c>d is evaluated only when a < b comes to false

• Though part of condition is not evaluated by compiler as part of optimization, they may be later used in the program

• Conditions need not be derived based on business case• Most of the time, the conditions are based on parameter

values of the functions to determine the path to be taken

Coverage for blocks

• This depends on how many conditions we have in the code

• Write one test for every relational operation of the condition

• Definitely test db, IO and memory exceptions• Write one condition for every catch block• In case the exception cannot be easily triggered, have

conditional compiled statements to raise such exceptions• Most of the time, people ignore exceptions thinking that it

will be taken care – remember, it will be caught, but how to treat the exception is up to us

Coverage goals• Coverage must have 100% at the end of all tests

for every function• There must be 1 test for every relational

operation in conditions• There must be 1 test for every loop termination• There must be 1 test for every exception• There must be 1 test for every test exit point (if

method has multiple return points)• There must be one test for successful sql

operation• There must be one test for unsuccessful sql

operation

Metrics Collection• Average number of unit test cases per page/component/any specific

unit in the project• Total number of unit test cases• Total number of unit test cases failed in test pass-1• Total number of unit test cases failed in test pass-2• Total number of unit test cases failed in test pass-n• Total number of memory leaks found• Percentage code coverage as per the tool statistics• Execution time for each test run• Total test execution time for all units• Ratio of # of defects found in unit testing to # of defects found in

code review• Defect distribution across units (asp related, db related, parameter

related etc.)

Metrics Analysis• Trend analysis helps in seeing whether the error

rate moves up or down• See pattern of failures and find out the corrective

action• Metrics vary a lot from developer to developer;

hence find out error rate pattern across developers

• Analyze failure pattern and time of failure • It is not easy to catch all errors in one round of

testing; hence do more rounds; more often

Configuration Management• Since we develop test code, we need to check in

these code sections to VSS or subversion or PVCS

• Maintain a uniform x.y version code to all test cases and suites

• Check in the test data along with test code as well

• Have one single person who can oversee all the CM operations

• Never allow people to have local copies of test code

Best Practices

• Keep adding more tests every day• More data preparation is the key• Test often - even twice a day• Never ignore warnings from IDEs• Run unit tests on a clean test bed• Review unit test code as well• Run regression unit tests daily• Run unit tests from other developer’s machine

Best Practices• Run unit tests by a different developer

• Make unit test code simple

• Never try to have if then else in a test code

• Do not have loops in a test code

• Run critical tests first

• Run all unit tests in a single session; never stagger

Don’ts

• Do not test entire functionality in one test• One test must satisfy just one condition• Never underestimate any parameter

combination• When skype or sony programs can come down

just by 1 bug, our program too can come down• If certain validations are performed in other

layers, do not do in unit tests• Never run tests on a test machine without clean

up

Don’ts• Never miss the null value test for any

parameter• Never miss any negative value tests for

database connection or query tests• Never miss trying to access an object or

record that does not exist, because these are the places where exceptions get triggered

• Do not miss to pass a wrong xml structure to web services

Challenges • White box testers must be developers• Hence companies use them to do core development

work and ignore writing white box tests• White box tests need tools and sometimes they are a

little expensive; hence companies avoid this route• Main challenge comes in not documenting design. Since

design is the basis, and if that is not present or incomplete, white box tests do not yield good results

• Project planning never considers the time for white box testing; especially in service companies, this is out of scope!

• When changes are made to the programs being tested, the changes are not documented well and hence unit tests go out of gear

Challenges• When same developer runs the unit tests, there

is a chance to ignore issues – so have developer A testing the code of developer B

• Initial unit test automation code takes some time; hence do not expect the ROI on day 1

• Results can be seen only after one or 2 quarters. Many management teams do not have patience to wait for such a long time

• People try to implement logic in different layers that they are designated for; instead of coding some logic in Java, they code in stored procedure and call that from java – this makes profiling and coverage difficult

Cyclomatic Complexity• This is to arrive at the testing complexity of code• Nodes, edges are the primary factors• Higher the complexity, higher the number of tests we need• This is based on graph theory• Many physical problems can be solved by graph theory• e.g. traveling salesman theory

• M = E - N + 2P• E - number of edges (lines)• N - number of nodes (rounds)• P is the connected components (start and destination points)

• The more if conditions and branches, the nodes and edges increase

• Call graph of all methods will show the independent connected components

Business Flow Tests

• When we do white box tests, we test individual components

• Hence it comes to unit tests• Divide flow into multiple components• Test them one by one• Aim at one transaction at a time• Club white box tools with functional test tool• Since business transactions need database

setup, ensure you have test stubs that pre-populate records in tables

How to speed-up• Let a senior create a sample test for every

function• Let a senior prepare unit test matrix• For a few tests, senior and junior

developers must do pair test automation• Then junior takes care of rest of the unit

tests for automation• Senior reviews the unit test code• Lead runs all tests every day

Get ready to take blame

• Unit tests are code; hence they may also have bugs. Developers may start blaming the tests themselves

• Tests may be redundant if not properly reviewed• Sequence of unit tests must be carefully done.

Always use bottom up approach on unit tests – test the smallest one first

• Run unit tests soon after build is released, and before BVT starts by the black box team

Never Give Up

• Teams give up white box tests after a quarter or so• This is because, they do not see visible returns• Remember – these things fail is a feel-good factor on

testing; at the same time, these things work is more important as well

• Most of the times, great systems are brought down due to missing white box tests and not black box functionality

• It may take years to find out, one great bug – but it is essential for big time products

• If you are in product company, be patient and white box will pay back