Designing Software For TestabilityA forgotten design pattern

Rohit Nayak

Talentica

Agenda Background and Motivation DFT in VLSI and hardware design Issues in Software Testing DFT for Software Development Automation and Testability Some Examples Cons, Review

The Anti-Pattern

If it works, its the developer, if not blame QA!

VLSI/PCB Testing Issues Mass produced Each piece needs to be validated Complexity

− Number of subsystems− Amount of logic

Access to internal logic Testing costs Cost of recall

Testability: “Effort required to

test a product to ensure that it

performs its intended function”

DFT Principles Controllability Observability

− Test Point Insertion Built-In Self Test (BIST) Fault Modelling Fault Simulation Test Pattern Generation

http://www.eviltester.com/wp-content/uploads/2008/03/bigbug.jpg

Software Testing“Testability is a design issue and needs to be

addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.”

- Bret Pettichord

Cost Of Bugs

10-25x10x1xCoding

25-100x15x10x1xArchitecture

10x-100x10x5-10x3x1xRequirements

ProductionSystem TestCodingArchitectureRequirements

Where Found

Where Introduced

Software Testability Suitability: clarity of specifications QA has Observability: we can only test what is visible Simplicity: easier design/UI makes testing easier Controllability: better the control, better the coverage

and automation Stability: how often it changes Performance: how fast it works Diagnosability: writing effective bug reports!

Software DFT Patterns BIST

− Automation suite w/ oracle

− Unit Tests− Assertions

Fault Simulation− Mock/Stub− Invalid params

Controllability− Decoupling− Bypassing− Mock/Stub− Set/Reset

Observability− Logging− Reporting− Test Interfaces

Good Automated Tests are Repeatable Easy to Write Easy to Understand Fast Way Easier with a Testable Software

When/How to Automate? Manager & Team are commited Testers/Dev either experienced or interested

in learning to script Product release cycles managed well Functionality / UI changes under control Early/Incremental approaches work best Build integration, reporting Each bug results in a automated test

Automation As Coding Automation scripts and frameworks are CODE! Use tools and scripting languages Evaluate tools on real problems Follow development processes

− Spiral/Agile Use Source Control same as rest of code These will have bugs as well!

Test “oracle” Expected Result

− Generated once from previous run or− Manually specified or− Legacy system

Time adjustment Results

− Database− File− Inline with code

Test Environment Multiple VMs Automated installs/images Automation tools Bug reporting tools

Unit Tests Each module has independent set Developer written/maintained setup, teardown xUnit, TestNG Runs on build Can run sub-set on install TDD

Diagnosability All environments: Test / Staging / Production Errors result in visual messages Errors are raised where they occur Errors can be localized Details are sufficient to fix issue Ability to send diag data to dev

Logging Module Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME) Define CRITICAL/ERROR levels well and use them! These should result in urgent notifications Lower levels in Production Time, App, Component, ThreadId, Message

8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334

Delimited columns for import Rotation based on Time, Size

Built-in Self Test Inserting test code/interfaces Set/Reset to bring state to known value Reporting to get current state Assertions about values/state

assert(order.billed==true)

“oracle” based regression

Some Examples Web Application

− Software As A Service− Browser Only UI− Ajax / Dynamic HTML

Banking (Client-Server)− Installed Application− Desktop Client− Server API

Web Application HTML Page Title Id values for important divs/controls Hidden values (non-textual, graphs, tables) Measurable Ajax responses Tools: Selenium, Sahi, Watir, WebTest, curl Logging incl. browser/ip/session cookie Ability to simulate time zone, language

Web Application - 2 Bypassing Captcha Mock TP APIs eg. Facebook, Google, OpenId SSL bypassing Transactions to be voided Multiple runs Always initialise controls

− Browser autofill

Desktop Unique identifiers to GUI controls Key Replay tools, SendKeys API test suites bypassing UI Client logs, ability to email Ability to run db queries in scripts

Desktop - 2 Automation Friendly Third Party Controls

− Ability to select cell− Copy-enable text fields− Key shortcuts to Forms UI for control focus,

clicking, navigation− Access by value (tree/list controls)

OCR for images/text consoles

Bypassing Credit card payment

− Test cards− Mock object− Dev auto-approval bypass code

Order placement− Dummy users, auto-fulfill− Dummy vendors (email order)− Admin screens move

Dates Never use system date directly Config override

<?xml version="1.0" encoding="utf-8" ?>

<configuration>

<appSettings>

<add key=“SystemDate" value=“$now"\>

</appSettings>

</configuration>

Nant: XMLPoke<xmlpoke file="App.config"

xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"

value="1-1-2009" />

Sending Emails Email subsystem should be a stub in dev Bypass SMTP: log to file … Email should be sent only to a whitelist Use Gmail Id & Atom API Use Unique Id/Timestamp to distinguish

Mocking / Stubbing Replace module during testing Different kinds:

− Sub-system (Email, Print)− Webservice (Credit Card, Flight Booking)− Hardware device (Biometric, CNC Machine)

Return default values, implement simple/fixed logic

Mocks - 2 Simulate Errors and Exceptions Provide more logging Jmock, Dependency Injection Con:

− Code level− Needs to change with API

Anti-Testability Viewpoints Security compromise by testability interfaces,

loggingRemove or lock down in prod

Extra coding timeMuch lower testing costs, better quality

Privacy issues in loggingShow only partial data

Performance Almost never an issue in practice!

Review BIST

− Automation suite w/ oracle

− Unit Tests− Assertions

Fault Simulation− Mock/Stub− Invalid params

Controllability− Decoupling− Bypassing− Mock/Stub− Set/Reset

Observability− Logging− Reporting