stp-2006-10

8/13/2019 stp-2006-10

1/40

A Publication

VOLUME 3 ISSUE 10 OCTOBER 2006 $8.95 www.stpmag.com

J2EE Application Performance Woes Brought to Light

Build a Quality-DrivenDevelopment Process

Thirteen Steps ToBulletproofing YourAutomation Scripts

Digging Into Root Causes

Digging Into

Root Causes

B E S T P R A C T I C E S :

T e a m T r a i n i n g

Findbugs Helps WithReport Generation
http:///reader/full//www.stpmag.comhttp:///reader/full//www.stpmag.com

8/13/2019 stp-2006-10

2/40

The Fast & Scalable Team Solution for...Defect & Issue Tracking Feature & Change Tracking Task & To-do List Tracking Helpdesk Ticket Tracking

OnTime is the market-leading project, defect and feature management tool for agile software development and test teams.OnTime facilitates tracking, analyzing and trending team-based software development efforts in an intuitive and powerful userinterface. A fully customizable UI, powerful workow, process enforcements, two-way email communications and custom reportscombine to help software development teams ship software on-time!

Available for Windows, Web & VS.NET 2003/2005

8006530024

www.axosoft.coms of t wa r e f o r s o ft wa r e d ev el o pm en t

Ship Software OnTime .

Only $495 for up to 5 Users Only $995 for up to 10 UsersFree Single-User Installations

OnTime 2006 Small Team Edition

For Teams up to 10 Members Free Single-User Installations $495 for 5-Team Members $995 for 10-Team Members

OnTime 2006 Professional Edition

For Teams of 10 to 1,000 Members From $149 Per User
http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/http://www.axosoft.com/

8/13/2019 stp-2006-10

3/40
http://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtesthttp://www.parasoft.com/jtest

8/13/2019 stp-2006-10

4/40
http://www.seapine.com/st&p

8/13/2019 stp-2006-10

5/40OCTOBER 2006 www.stpmag.com 5

ContentsA Publication

VOLUME 3 ISSUE 10 OCTOBER 2006

12 COVER STORY You Can Root Out J2EE PerformanceProblemsGet your hands dirty! Java 2 Platform, Enterprise Edition application issuesare often buried deep in the code. By Edie Lahav

Departments7 Editorial All things changefrom editors tosolar systems. By Lindsey Vereen

8 Out of the BoxNew products for developers andtesters. Compiled by Alex Handy

10 Peak PerformanceUse your numerical know-how tointerpret statistical data. By Scott Barber

35 Best PracticesIn pedagogy, the personal touch addsan essential element. By Geoff Koch

38 Future TestHow to address the challenge ofapplication manageability with healthmodels. By Mike Curreri

Are your automation scripts toofragile? Go step by step to help makethem bulletproof.

By Jos Ramn Martnez

24 Get Quality Into Your Head

29 AutomationScripts in 13 Steps

16 Shining Light On Java Code

This six-stage template can lead you toa successful quality-driven process.

By Nigel Cheshire

Static analysis tools aid in deliveringon-time code. By Alan Berg

8/13/2019 stp-2006-10

6/40

Copyright 2006 Instantiations, Inc. CodePro AnalytiX and CodePro are trademarks of InsAll other trademarks mentioned are the property of their respective owners.

Key Features of CodePro Analyti X

CodePro Analyti X for Eclipse, Rational and WebSphere

Drive out quality problems earlierin the development process.

Powerful code audits and metrics makeevery developer a Java expert.

Create tight, maintainable systems.Wizards generate unit and regression tests,and analyze testing code coverage.

Enforce quality measures across teams.Define, distribute, and enforce corporate codingstandards and quality measures, across individualsand teams no matter where theyre located.

Spend less time and money to develophigh-performance Java systems.Dramatically improve your applicationdevelopment process and code qualityleading to more reliable, maintainable systems.

Detect & correct code quality issues... automaticallyDefine, distribute & enforce qualitystandards across development teams800+ audit rules and metrics,350+ Quick FixesPowerful management reporting

Code metrics with drilldown & triggersAudit Java, JSP and XML files

JUnit test case generationCode coverage analysis of test cases

Dependency analysis & reportingIntegrated team collaborationJavadoc analysis & repairSeamless integration with Eclipse,Rational and WebSphere Studio

CodePro AnalytiX is developed by the experts who brought you the popular bookEclipse: Building Commercial Quality Plugins Eric Clayberg & Dan Rubel

www.instantiations.com 1-800-808-3737

the development tools experts

No one outside of IBM has more experiencecreating Eclipse-based Java development tools

8/13/2019 stp-2006-10

7/40

Once upon a time, way back in the last century, a year and a day werethought to be the samelength on the planet Mercury: 88 Earth days.Thanks to improved meas-urement techniques, wenow know that while the

planet does still revolvearound the sun every 88days, it rotates on its axisonce every 59 days.

Back when a day and a year onMercury were the same length, it wasone of nine planets in our solar system.But it seems Pluto has now beendrummed out of the planetary corps,thanks to its insufficient sphericity anda failure to clear its neighborhood, mak-ing Mercury a sibling in a family of only eight planets. Once Mercury was thesecond smallest planet next to Pluto,but with the disqualification of Pluto,it has gained the distinction of beingthe smallest.

The decision to blackball Pluto is not immutable. A number of scientists areup in arms about it, and perhaps thedecision will one day be rescinded.

Ipso Facto As anyone who has ever tried to test thecontinually moving target known as

enterprise software understands, anevent like the Plutonic expulsion sim-ply highlights the point that everythingthat is not true a priori is subject tochange. And astronomy is a field in which such changes are commonplace. Jupiter, for example, has way moremoons now than it did when I was a lad.

Speaking of changes, some are brew-ing right here at Software Test &Performance. After two years at thehelm of this magazine, I have decidedto move on. Next month youll see anew face at the top of this pagetheface belonging to Edward J. Correia.

You may know that name from SD

Times. My first encounter with Eddie took place sev-eral years ago at an indus-try dinner in San Francisco when I was working foranother publishing com-pany. I remember beingimpressed with his insight and industry knowledge.

Later, knowing his name, Iread his articles in SDTimes and grew furtherimpressed. When I joined

BZ Media, the company that publishesboth SD Times and Software Test &Performance, I was happy for theopportunity to become better acquaint-ed with him.

Curriculum VitaeEddie came to SD Times in 2000 andhelped with its launch in February of that year. He covered several beats,including integration and SOA,databases, XML and Web services,and embedded and wireless develop-ment.

Hed previously worked as a free-lancer with several publications, includ-ing Network Computing and UnixToday. Then he led the development team that built a publishing companysfirst centralized editorial repository andits wire-service storage and distribution

system.Later, Eddie took over as the first technical editor of CRNs Test Center, which was just being built. He spent four years with CRN building the Test Center team, developing performancetests for computer software and hardware, conducting the comparative testsand reporting the results.

Post Hoc And so I leave Software Test &Performance under Eddies extremely capable guidance, off to relearn every-thing I ever thought I knew about thesolar system.

OmniaMutanter

OCTOBER 2006 www.stpmag.com 7

A MESSAGE FROM THE EDITOR

Lindsey VereenEditor

VOLUME 3 ISSUE 10 OCTOBER 2006

BZ Media LLC7 High Street, Suite 407Huntington, NY 11743+1-631-421-4158fax [email protected]

PresidentTed Bahr

Executive Vice PresidentAlan Zeichick

Software Test & Performance (ISSN #1548-3460,USPS #78) is published 12 times a year by BZMedia LLC, 7 High Street, Suite 407, Huntington,NY 11743. Periodicals privileges pending atHuntington, NY and additional offices.

POSTMASTER: Send address changes to BZ Media,7 High Street, Suite 407, Huntington, NY 11743. Ridealong is included.

2006 BZ Media LLC. All rights reserved. SoftwareTest & Performance is a registered trademark ofBZ Media LLC.

Cover Photograph by Peter Chadwick

Editor Lindsey [email protected]

Senior Editor Alex [email protected]

Copy Editor Laurie OConnell

Contributing Editors Scott [email protected]

Geoff [email protected]

Editorial Director Alan [email protected]

Director of Circulation Agnes Vanek+1-631-421-4158 [email protected]

Customer Service/ Subscriptions [email protected]

EDITORIAL

READER SERVICE

Publisher Ted Bahr+1-631-421-4158 x101

[email protected]

Advertising Sales Manager David Karp+1-631-421-4158 [email protected]

Advertising Traffic Phyllis Oakes+1-631-421-4158 [email protected]

Marketing Manager Marilyn Daly+1-631-421-4158 [email protected]

List Services Nyla Moshlak+1-631-421-4158 x124

[email protected]

Reprints Lisa [email protected]

Accounting Viena Isaray+1-631-421-4158 [email protected]

SALES & MARKETING

Art Director LuAnn T. Palazzo

Art /Production Assistant Erin Broadhurst

ART & PRODUCTION
mailto:[email protected]:[email protected]:[email protected]

8/13/2019 stp-2006-10

8/40

Out of the BoxCompiled by Alex Handy

In late July, TestNG received a majorupdate that added support for organizingand categorizing thousands of unit tests within an application. While JUnit is themost popular unit testing tool for Java,

TestNG has been steadily receiving praisefor the past three years, thanks to its abil-ity to manage and group large numbersof tests within an application. Its thegrouping of tests that sets TestNG appart

from JUnit, and its also the featurethats most markedly improved in ver-sion 5.0 of the tool.

Cdric Beust, a software engineerat Google, began developing TestNGthree years ago when he decided that

JUnit was too limited in scope for prop-er enterprise use. The major feature

[in this release] is not brand new; itsabout more renaming and cleanup of annotation names to make them moreintuitive. Our reports are easier to readand better organized now. As we havemore and more users who have thou-sands of tests and dozens of groups, it becomes really important to makethose reports easy to read, said Beust, who is originally from France.

Most of the work on the project hasbeen done exclusively by Beust and Alexandru Popescu, who began con-

tributing code soon after Beust createdTestNG. Other developers have built plug-ins for the tool, but Beust and Popescuhave remained the primary contributors.

For the future, Beust hopes to seemore external contributions, but doesnt expect to do much work on the core of TestNG. For the past year, there were lessfeatures requested in the core and we were working on productivity around thecore, which is good because it means thecore is working and functional enough,said Beust of the 600-strong mailing list

for the tool. I think were going to seemore stronger integration with Webservers so we can drive TestNG fromremote machines. Theres also a work inprogress to have a distributed version so we can have distributed tests. The gen-eral message is going to be more about scaling. For people writing thousands of tests, we want to make it almost trans-parent for them to use as many machinesand as much power as they have.

TestNG is a free tool, and can be down-loaded at www.testng.com.

TestNG 5.0 Smoothes Out Annotations

TestNG offers charts and graphs beyond bugoccurences. This defect-tracking system can alsochart workflow and planning phases.

The newest version of GUIdancer adds Ant task support and a com-mand-line client foradded batch capabilities.This Swing and Java GUI

testing program is avail-able now from Bredex,GmbH at www.guidancer.com.

The past few monthshave seen considerabledevelopments for GUIdancer, said Hans-J.Brede, managing direc-tor of Bredex, GmbH.As well as improving the general usabil-ity of the tool, weve been working onoffering more options for testers batchtesting, a larger choice of actions, andtranslation of test data, for example. Thispreview is a taste of the full 1.1 version,

for which yet more features are in devel-opment.

GUIdancer can be downloaded andpurchased online. A single user licensecosts around $1,200, while a multi-userlicense costs around $5,000.

GUIdancer 1.1 Taps IntoTown

GUIdancer automatically tiptoes through your Swing apps tofind glitches and inconsistencies.

JAMon is a simple and free API that givesdevelopers a window into the performanceof their production applications. It can be

used during testing or deployment to trackstatistics and information related to anapplications behavior and speed. New tothis version is JDBC support and a handfulof bug fixes.

The JAMon API is quick and simpleto install. Simply place the JAMon.jar filein your class path, then insert the APIsstart and stop commands around thecode you wish to monitor. The tool canbe called multiple times in a single appli-cation, and requires no administrationprivileges to run, nor to view the datacollected. JAMon is available under aBSD license and can be downloaded at freshmeat.net/projects/jamonapi.

Java AppMonitor APIHits 2.2

8 Software Test & Performance OCTOBER 2006
http://www.guidancer.com/http://www.guidancer.com/http://www.guidancer.com/http://freshmeat.net/proects/jamonapihttp://freshmeat.net/proects/jamonapihttp://freshmeat.net/proects/jamonapihttp://www.guidancer.com/

8/13/2019 stp-2006-10

9/40

This fall, version 1.5 of the Web Application Testing in Ruby project willbe released. Watir, a Ruby library that works with Internet Explorer on Windows, is a free, open-source func-tional testing tool for automating brows-er-based tests of Web applications. Ruby

provides the ability connect to databas-es, read data files, export XML andstructure code into reusable libraries. Watir is designed to stomp all over Webapplications, chasing links and validat-ing forms.

On the Watir wiki, one user codifiedhis reasons for using Watir. SilkTest may be good for casual automation. However,Im talking about automating hundredsand hundreds of highly complex test sce-narios on a large Web application. Theautomated test cases will be maintainedby different people, be run for multiplereleases over many years. I just cannot imagine using SilkTest without incur-ring huge cost...

Using Watir + IE Developer Toolbar+ SpySmith, writing automation is very enjoyable and efficient. We are able to write very robust and concise test cases,such as automating drag-and-drop without knowing how such behavior isimplemented. We can strictly follow DRY principle and share our assets. We

can even embed powerful debuggingtools such as Rubys breakpoint with-out asking testers to write one extraline.

The user, who posted under thename Dannyy, used SilkTest 8.0 for hiscomparison.

Wati r is free, available at openqa.org/watir. As of this writing, version 1.5 was in ful l development and nearingrelease. By the time you read this, it should be available. To use Watir, Windows users must first install Ruby.

The open-source testingframework known as Selen-ium Core has reached version0.7.1 as of this writing.

Selenium Core is a testtool for Web applications.Selenium Core tests rundirectly in a browser: Internet Explorer, Mozilla and Firefoxon Windows, Linux orMacintosh. Written in pure J a v a S c r i p t / D H T M L ,Selenium Core tests are

copied directly into your appli-cation Web server, allowingthem to run in any supportedbrowser on the client side.Selenium Core is open sourcesoftware and can be down-loaded and used without charge. The tool is designedto make regression andacceptance testing of Web siteseasier. The project hassparked a number of off-shoots, including theSelenium IDE, a Firefox-basedtool that turns a browser intoan automated acceptance test machine.Selenium Remote Control and Seleniumon Rails have also made significant progressthis fall.

Blogger Scott McPhee wrote, of Selenium, Whats cool about it is that it understands the dreaded JavaScript. It actu-ally drives your local browser via a very cleverset of JavaScript functions... Its pretty raw at the moment, only young, but I think in

a short while it will show some definitepromise. All Selenium tests are written in a pro-

prietary scripting language known as

Selenese. But beyond that, the SeleniumIDE offers a far simpler way to write tests.Users of the Selenium IDE can record andplay back mouse- and keyboard-driven tests

within Firefox, then automate them for play-back. Tests can then be exported as Ruby scripts, and then automated and tied intoa nightly barrage.

While none of these projects have yet reached general release, theyre no less use-

ful at present. All of these Selenium proj-ects can be found at www.openqa.org/selenium. The tools are free and opensource.

Selenium IDE allows users to record GUI tasks for later play-back as tests.

Selenium Still Growing

Send product announcements [email protected]

Watir Flows

Over WebApps

Corrected table from Yuri Chernak's "Bringing Logic Into Play" (Aug. 2006).Erratum:


Disjunctive Syllogism Form Test ing Argument Form1. Either P or Q is true means 1. After all test cases have been executed, a feature sta-

tus can be either fail (P) or pass (Q) ( implication ).

2. P is not true means 2. We know that the feature did not fail the test for allof its test cases (evidence).

3.Then Q is true means 3.Then the feature passes the test (conclusion) .

TABLE 2: DERIVING A FEATURE PASS CONCLUSION
http://openqa.org/watirhttp://openqa.org/watirhttp://www.openqa.org/seleniumhttp://www.openqa.org/seleniumhttp://openqa.org/watirhttp://www.openqa.org/selenium

8/13/2019 stp-2006-10

10/40

If youre like me, you barely squeaked by in whatevermath class you took last. If

youre like two of the best pro-grammers Ive ever worked with, you failed your last mathclass miserably, dropped out of school and got a job writ-ing code. Or maybe you enjoy math and statistics, in whichcase Im happy for you andencourage you to put that enjoyment to practical use when designingand reporting on software tests.

Whatever your particular situation is,Im starting to feel like a math and sta-tistics teacher. As a whole, it seems to methat members of software development teams, developers, testers, administratorsand managers alike have an insufficient grasp on how to apply mathematics orinterpret statistical data on the job.

As an example, I just finished anotherseveral-hour discussion with someoneclaiming to understand statistical princi-ples who believed that a data set includingfive response-time measurements and astandard deviation roughly equal to the

mean was statistically significant. The dis-cussion reminded me that as performancetesters, we not only must know and be ableto apply certain mathematical and statisti-cal concepts, we must also be able to teachthem. Worse, we often have to teach theseconcepts to people who like math even lessthan we do. Over the years Ive stumbledupon some relatively effective explanationsfor the mathematical and statistical prin-ciples I most often use as a performancetester. Id like to share them with you.

Averages Also known as arithmetic mean , or mean forshort, the average is probably the most com-

monly used and most com-monly misunderstood statis-tic of them all. Just add up all

the numbers and divide by how many numbers you just addedwhat could be sim-pler? What most folks dont realize is that if the averageof 100 measurements is 4,that could imply one quarterof those measurements are 3,half are 4 and another quar-

ter are 5 (well call this data set A)or it could mean that 80 of those measurementsare 1 and the rest are 16 (data set B). If

were talking about response times, thosetwo sets of data have extremely different meanings. Given these two data sets anda response time goal of 5 seconds for allusers, looking at only the average, bothseem to meet the goal. Looking at the data,however, shows us that data set B not only doesnt meet the goal, it also probably demonstrates some kind of performanceanomaly. Use caution when using averagesto discuss response times, and, if at all pos-sible, avoid using averages as your only reported statistic.

PercentilesNot everyone is familiar with what percentiles represent. Its a straightforwardconcept easier to demonstrate thandefine, so Ill explain here using the 95thpercentile as an example. If you have 100measurements ordered from greatest toleast, and you count down the five largest measurements, the next largest meas-urement represents the 95th percentileof those measurements. For the purpos-es of response times, this statistic is readNinety-five percent of the simulatedusers experienced a response time of this

value or less under the same conditions

as the test execution.The 95th percentile of data set B above

is 16 seconds. Obviously this does not givethe impression of achieving our five-sec-ond response-time goal. Interestingly, thiscan be misleading as well: If we were to lookat the 80th percentile on the same data set,it would be one second. Despite this pos-sibility, percentiles remain the statistic that I find to be the most effective most often.That said, percentile statistics can standalone only when used to represent datathats uniformly or normally distributedand has an acceptable number of outliers.

Uniform Distributions

Uniform distribution is a term that repre-sents a collection of data roughly equiv-alent to a set of random numbers that are evenly distributed between the upperand lower bounds of the data set. Thekey is that every number in the data set is represented approximately the samenumber of times. Uniform distributionsare frequently used when modeling userdelays, but arent particularly commonresults in actual response-time data. Idgo so far as to say that uniformly distrib-uted results in response-time data are apretty good indicator that someoneshould probably double-check the test or take a hard look at the application.

Normal Distributions Also called a bell curve , a data set whosemember data are weighted toward thecenter (or median value) is a normal distribution . When graphed, the shape of thebell of normally distributed data can vary from tall and narrow to short andsquat, depending on the standard devia-

tion of the data set; the smaller the stan-dard deviation, the taller and more nar-row the bell. Quantifiable human activi-ties often result in normally distributeddata. Normally distributed data is alsocommon for response time data.

Standard DeviationsBy definition, one standard deviation isthe amount of variance within a set of

PerformanceTesting Plus:Do the Math !

Peak Performance

Scott Barber is the CTO at PerfTestPlus.His specialty is context-driven performance

testing and analysis for distributed multi-user systems. Contact him at [email protected].

Scott Barber

mailto:[email protected]:[email protected]:[email protected]

8/13/2019 stp-2006-10

11/40

8/13/2019 stp-2006-10

12/40

(J2EE) is that it decreases the need for complex programming while increasing the reusability of component-based modules. While its a boon for developers, this development model placesa new emphasis on the IT infrastructures middle tier. Working

with the different data types, hardware components and J2EEapplication servers to build an application is a new endeavor formany developers, with a variety of challenges for the test engi-neers in charge of software quality.

This distributed computing environment brings with it achanging group of variables. Unlike legacy architectures, when

a J2EE application goes into production, the distributed envi-ronment often behaves dynamically. A range of performance,application and configuration issues can occur that are uniqueto the dynamic J2EE environment. For the test engineer, dis-covering how the software components will behave during theapplications life cycle is key to successful QA. Performanceproblems are often discovered when testers are able to simu-late application trafficso load and stress testing is an impor-tant focus area for the test engineer. This article pinpointssome root causes of per formance issues in J2EE applications.

Testing applications is a challenge in the J2EE arena forseveral reasons. First, in many organizations, testing roles over-lap among QA, development, production and networking.Diversity and changeability in a J2EE systems many elements


By Edie Lahav

O ne of the reasons for the popularity of the Java 2 Platform, Enterprise Edition

J2EE Application Issues Are OftenBuried Deep In The Code

8/13/2019 stp-2006-10

13/40

makes diagnosing a problem very difficult. Varying loads andrequirements on J2EE applications and components can causeruntimes to fluctuate, and application performance problemsare often buried deep in the application code.

In addition, many applications havent been designed fortest. Even in most J2EE Web applications, performance and

memory concerns are the top issuesthe hooks for test-ing are typically not well defined in the application develop-ment stages. As developers and testers work more collabora-tively, this will change: More planning in the development stage will focus on the applications functional service objec-tives for per formance and scalability.

And last, but not least, when a J2EE application has a prob-lem, its generally found at the code level. With typically few clues on the surface, the test engineers spend hours re-cre-ating a situation and generally need a full team (development,networking and so on) standing by to watch test and QA try to pinpoint the errors root cause.

Compound CommunicationIn most organizations, the J2EE development team has a set of testing tools that typically include code profiling. This

ensures that the code works, but it doesnt test the applica-tion under load. So, after the code is complete, its turnedover to the QA team to determine if its ready for production.

In a more effective approach, test and QA engineers area part of the development team, and the testing plan is writ-ten when the specifications for the application are complet-

ed. In this model, the test and QA team know as much about the intended use of the application as the developers, and areable to build a solid plan as the application is being devel-oped.

However, if you cant swing a compound team like that, you can still stay in the loop by scheduling individual meet-ings with the development team and the applications spon-sor and asking them to describe the application for you. Learnabout the expected typical user behavior, including log-in,capacity expectations and any known technical details about the application. Obviously, the development team will sharemore information along the lines of a product specification,and the sponsor will tell you what to expect functionally, but with input from both, you should be able to piece together a

Edie Lahav is director of R&D at RadView.


P h o

t o g r a p h

b y

A c h

i m H

o l z e m

8/13/2019 stp-2006-10

14/4014 Software Test & Performance OCTOBER 2006

JAVA 2 APPLICATION TESTING

thorough understanding of the appli-cation.

Setting the SceneSo, youve written and executed a sol-id test plan. Youve tested the applica-tions capacity, throughput and log-infunctions, and youve profiled the hardware and software. Basically, youve exe-cuted the test plan and youve encoun-tered what look like performance issues.How do you do this in the most effec-tive way?

Youd like to tel l the development team more than the application brokehere or crashed there. You want toexplain exactly why the application

crashed and what caused it. As men-tioned earlier, to determine the root cause of problems in the distributed J2EE world, you often need to re-createthe problem for a team of colleagues.Its time-consuming to get the DBA, thenetwork administrator and a develop-ment engineer to watch different piecesof the system while QA runs the load test to re-create the problemand thats just the first step. After that, this team must sift through the data generated at allthese different system points to hunt down that broken moment and deter-

mine why the problem occurred.Logging Error Locations What you need is a log of your test exe-cutions that captures a complete recordof application performance across themultitier environment in one synchro-nized timeline. This will provide you withdata points at each location where theapplication may fail, with the details that the developers need to fix the problem.

After QA gathers this data, you must be able to analyze it. Looking throughthe data to locate the bottlenecks takessome J2EE background knowledge, solets get down to some basics.

Seeing Inside J2EESome of the very benefits that J2EEoffers developers lie in the features that make the language a challenge for thetest and QA engineer. For example, theability to use object-oriented program-ming and encapsulate programs cankeep the root cause of a problem buried

within a component.For many test engineers, J2EE com-

ponents limit testing to black-box tech-nology because they cant see inside.Test engineers need tools that provideinsight down to the component level topinpoint problems.

In addition, the environment posestesting concerns. In the evaluation of

any J2EE problem, your first step is todetermine if the error is caused by anapplication component or a systemissue. Test and QA engineers trained in

viewing load test metrics against serverfarms are challenged by the typical J2EEenvironment of firewalls, Web servers,EJB application servers and databaseservers.

When an application fails, as the test engineer, you must trace that failurethrough the entire system to the root cause. Along the way, you may encounter some of the following system

issues: Inadequate server resources (JDBC,threads, memory)

Thread pool sizes Inappropriate server parameters

(class path, JVM) Custom architectural wiring Low memory Configuration problemsFirst, you must understand the appli-

cations infrastructure and how the appli-cation components act under load. Once

youve determined that the problem isnot an infrastructure issue (all transac-tions between components and the infra-structure work), its time to drill down to

the components to look for the root cause. Youll need a tool that will enable you to drill down to the componentlevel.

Capturing Performance DataThere are several measurement inter-faces for capturing J2EE application per-formance data. You can use the inter-face provided by the application serv-er vendors for high-level monitoring, or you can use a sampling of a portion of events from an event stream. Sampling,unless implemented as a dynamic prac-tice, often misses specific or sequence-related problems, so make sure that yourtesting tool allows you to change yoursampling profile (what youre record-ingthe type and depth of information)on-the-fly. This allows you to switch backand forth between viewing high- and

low-level statistics during the test,because when you begin a sampling, youdont know when or where youll needto dive deeper.

A technique called total trace allows you to capture granular data for eachevent (including arguments) executed within the code. When included in test-ing tools, this technology often hasembedded intelligence that automatesresources to manage overhead andreturns to you a data log that you canthen analyze to pinpoint the problem.

The next step for the QA and test team is to replay and analyze the datato quickly pinpoint the root cause of theapplication problem. Look for tools that provide an interactive log that allows theteam to follow a transaction from theclient to the Web server to the distrib-uted environment and drill down intothe log to rapidly locate the issue.

The Usual J2EE OffendersIn testing, it helps to know what to look

for in a J2EE application. Lets exam-ine some of the more common J2EEperformance issues and their probablecauses.

Performance problems can becaused by any number of sources, fromincorrect business logic to incompati-ble components to an unexpected appli-cation error, such as an exception that isnt written to a log file. In your prob-lem-solving quest, begin by reviewingthe execution log and look for obvioussynchronization, deadlock or crash sit-uations. When you come to a failure inthe application, drill down to the com-ponent level to view what components

TABLE 1: APPLICATION SERVERS AND PACKAGES COMPARED

Application Server Properties

Application Server Version

JVM Version

JDK Type and Version

JTA Timeout

Maximum Heap Size

Servers Classpath

JDBC Driver

JDBC Pool Initial Capacity

OS Version

Application Package Properties

Deployment Date of WAR & EAR files

File Size

EJB Type

Classes Location

Implementation Class

Component Load Order

8/13/2019 stp-2006-10


JAVA 2 APPLICATION TESTING

are being accessed or are looking forinefficient database calls. Other issuesthat can cause performance problemsinclude:

Improper application settings. Thesecan include inadequate session-state provider or buffering disabledon a Web Forms page.

Improper application server settings .(For example, an insufficient num-ber of threads allocated to an appli-cation.) This can be diagnosed by recording code executed in syn-chronization with application andapplication server metrics, includ-ing EJBs. Then review the log at thesystem-level view to pinpoint theproblem.

Interoperability with legacy code. Infrastructure problems. For exam-

ple, poor network responses that

cause degradation in applicationperformance or insufficient hardware, limited processing power,lack of memory and so on.

Unexpected application errors. Thesecan be found by reviewing the his-tory of the code execution andobject states at each method call.The component-level view provides

visibility into the initial and futurefailure of the application.

Methods that perform poorly over many transactions. These can beidentified by reviewing metrics forEJBs, servlets, JSPs and so on,through a component-level view tofind the execution path that leadsto the cause of the poor perform-ance.

Memory thrashing and memory leaks . Consistently and/or intermittently slow

methods affecting specific user or data values.

Many J2EE applications use third-party Web services. These components

bring with them an entire range of issues, including inefficient operations,memory consumption, and insufficient application and framework settings. Todiagnose this problem, look for ineffi-cient operations such as the applicationopening a connection to the databaseupon every request, or poorly per-forming SQL statements or stored pro-cedures. Then, pinpoint the error by recording the complete test executionat the system level to provide informa-tion on a number of method calls andtheir duration in any window of time.

Memory consumption can also point out a performance problem. Identify a

possible error by recording perform-ance counters that can indicate that thecause is a memory issue, such as freememory and so on.

Having too many instantiations of asingle object running at the same timealso raises a red flag. Either the codeis creating too many copies of the object or the instantiations of the object arent being destroyed after use.

App lication and framework set -tingsfor example, the maximum num-ber of threads in the thread pool isinsufficientcan also cause perform-ance problems. Pinpoint them by recording all the necessary performancecounters and messages written to theevent log, as well as application andframework settings (J2EE configurationaccesses and so on). This data will thennote non-optimized settings that caused

the application to slow down.Long wait times or poor DB per-formance can be caused by non-opti-mized SQL statements or stored proce-dures. Pinpoint this problem by record-ing the JDBC performance and drilldown to the actual SQL statement that

was executed against the DB.

J2EE Configuration CriminalsTo be able to properly test J2EE applica-tions, the test and QA team needs back-ground on the J2EE configuration infra-structure. Ask for a detailed explanationof the applications infrastructure fromthe development team. This will provide

you with the information for identifyingissues such as insufficient permissions toaccess a resource, incorrect applicationsettings in a J2EE configuration file orconflicts with other applications. Due toits highly distributed nature, a J2EE appli-cation has many opportunities to get stuck in the infrastructure. Lets gothrough some of the more common con-

figuration issues and ways to diagnosethem. Insufficient permissions to access a

resource . For example, does the appli-cation have permission to write toan application directory? Solve by reviewing all accessed and attempt-ed accesses of the application by any resource on the computer.

Incompatible components. Locate thisissue by recording the interactionsof the application with any home-grown and third-party componentsto identify unsuccessful calls.

Conflict with other applications and or Web services. To catch this prob-

lem, record all the interactions of the users application with any exter-nal resources or services. The record-ing should show any unsuccessful

Web services calls or failures inaccessing a database.

New software that causes an application failure. Often, new deploy-ment of software can lead to anapplications failure. Review the his-torical log to determine if this isthe case.

Many times, poor performance is dueto configuration errors. Before testingoccurs on a J2EE application or when anew application server is rolled into pro-duction, its imperative that the configu-ration of the server be diagnosed.Reviewing the configuration before test-ing can resolve issues before a great dealof time and energy is spent trying to find

a problem that was ultimately caused by a server misconfiguration.Some testing tools allow you to com-

pare application servers to determineconfiguration errors. With this capabil-ity, youll want to compare the follow-ing application server and applicationfile properties, as in Table 1.

Functional FlawsFunctional application problems are typ-ically found earlier in the test cycle, but sometimes pop up later during load test-ing. Here are three common functionalproblems and some likely causes:

Incorrect business logic. Solve by recording the complete flow of managed and unmanaged compo-nents and public method calls.Captured exceptions may indicatea code problem.

Hang/Time-out. Solve by recordingthe execution of multiple threadsand processes at the system level.

Crash. Find crashes by recording

application execution at the systemlevel and automatically capturing thecrash events.

The J2EE Challenge J2EE offers a rich environment for thedeveloper, and companies investing inthese applications expect a promisedreturn on investment that only the test and QA team can ensure. J2EE alsooffers the tester a challengeto learnthe environment, avoid pitfalls and workmore closely with the development team. In that way, you can decrease thetime needed to deliver top-qualityapplications.

8/13/2019 stp-2006-10

16/40

QJPro and Findbugs make the automatic deliv-ery of on-time reports with indications of codequality viable. Here, Ill explore an example of report generation that uses nightly buildmethodology to deliver on-time agile codereviews. Further, Ill explain the nature of anautomatic code review, with emphasis on the value of the Findbugs tool (http://findbugs.sourceforge.net).

Bug HuntingFindbugs looks for antipatterns known as bug

pat ter ns in compiled Java code. The toolsearches for a large number of patterns that are common errors; for example, exceptionsthat are caught and then ignored or classesthat are serializable, but parts of which arereally not.

Findbugs works on compiled code usingthe BCEL library (http://jakarta.apache.org/bcel) and performs analysis on .class files.Obviously, before the tool analyzes, you needto compile the code. At an estimate, Findbugs

has a list of 100 antipatterns searched for; thislist expands with every new version, and if youre motivated, you can add your own pat-terns in what appears to be a relatively easily process. Please remember, as the tool is a com-munity effort, that its polite to pass back yourimprovements.

Why Use Findbugs?Static analysis comes of age for large codebases, with multiple teams striving to fulfillstringent targets with feature-rich function-

Shining


Static

Analysis

Tools Aid In

Delivering

On-TimeCode

Reviews

By Alan Berg

S tatic analysis is the process of checking code for patterns that indicate programming

faults. Tools such as PMD,
http://findbugs.sourceforge.net/http://findbugs.sourceforge.net/http://findbugs.sourceforge.net/

8/13/2019 stp-2006-10


P h o

t o g r a p

h b y

R a v i

T a

h i l r a m a n

i

al requirements. In this extreme andacidic type of environment, the

veloci ty of change and inconsistent code quality across teams makesit costly to QA by human effort alone.

Quality assurance focuses on themost important features of anygiven product, normally functionally testing a thin path within the wholecode base.

One issue that is not alwaysaddressed is the consistency of bad prac-tices by a given team, or the general lev-el of defects. Static analysis works fromthe bottom up and pays attention even-ly throughout the whole code base. Thetool can be considered a success if theanalysis doesnt generate too many falsepositives. Under these conditions, thetool acts as a neutral and objectiveobserver with abundant patience,defanging the potential for very realand human friction.

When one subproject is failing rel-ative to the quality generated by anoth-er, the statistics will point this out quick-ly and painlesslyFindbugs doesnt feelpain.

Further, static analysis captures alarge swath of real errors. Not all of

them are particularly interesting; some-times the reports sound more like nags.However, the number of real errors,trivial or not, is around the 50-75 per-cent correct rate.

This efficiency affords coding teamsthe time to breathe and then fire off reports and act on themwith a sig-nificant boost to quality. In the longrun, this will simplify debugging of themore challenging issues as backgroundnoise is diminished.

How to Boost Static ClingMotivating teams to use static analysiscan be a chore. Humans (and yes, evendevelopers) tend to be turned off by critical reports, and placing a new infra-

structure in an organization can betiring.Personally, I havent yet found an

optimized way of achieving this goal,but I suspect the best time to imple-ment is at the start of a project.Throwing in new technology at themost stressing parts of the QA processresults only in polite comments and pas-sive resistance. Instead, try to start at arestful time, perhaps as your team isgenerating functional requirementsand prototypes. Remember to consult

your target audience, asking questionslike What are the false positives?Which tests are the most valued? andWhere should we be looking first?

Inside FindbugsFindbugs has a number of interfaces,including the command line, Ant, Swingand even an Eclipse plug-in. BecauseEclipse is a visual environment, the plug-in is probably the handiest method togive you a feel for the tool.

You have two approaches to installa-tion: download a zip file and expand it into the Eclipse/plug-in directory, or go

via a live update. To install in Eclipse 3.2,follow the installation instructions men-tioned in the Findbugs installation at http://findbugs.cs.umd.edu/eclipse.

Once installed, to activate Findbugs,first create a Java project and then right-click on the project and select Properties> Findbugs. The configuration screenshown in Figure 1 pops up. Select Run

Findbugs automatically and set Minimum Priority to report to low. By doing this you can see more of the rulesas theyre automatically applied. I sug-gest that you browse the bug codes and

compare them with the descriptionsfound in the Findbugs bug descriptionat http://findbugs.sourceforge.net/bugDescriptions.html. Once the settingsare in place automatically, your code willbe tested and the errors will be report-ed in the Problems pane, which is conveniently located underneath yoursource code by default.

To gain insight into the error types,try writing deliberately bad code andthen test it to the Findbugs set of prac-tices (see Listing 1).

To generate the results shown inFigure 2, create the class file under yourown project and then right-click the proj-ect in the navigation window, choosingFind Bugs > Find bugs. Notice the seven warnings and no errors.

Working your way through the dis-covered bugs from top to bottom, by double-clicking on the issue listed inthe Problems dialog, moves the cursorto the line mentioned in the sourcecode. Table 1 describes each warning

in the relevant source code and a broad-er description of that warning. Again,for your own education, you shouldmake such a table yourself.

A Functional Plug-In At this point, you should have a func-

Alan Berg has been a lead developer at theCentral Computer Services at the Universityof Amste rdam for the past eight y ears.Contact h im at [email protected].

Light on Java Code
http://findbugs.sourceforge.net/bugDescriptions.htmlhttp://findbugs.sourceforge.net/bugDescriptions.htmlhttp://milto:[email protected]/http://milto:[email protected]/http://milto:[email protected]/http://findbugs.sourceforge.net/bugDescriptions.html

8/13/2019 stp-2006-10


CODE REVIEWS

tional plug-in. I strongly recommendgetting your hands dirty by writing somebuggy code and seeing what the toolmakes of it. Findbugs generates valuablefeedback for new developers, andEclipse interactions can serve as effec-tive training.

For large projects with many differ-ent players and teams involved, such anupward push is also helpful. The weak-est link is the one that the user tends toremember: If your service is working 99percent but the last 1 percent is trou-blesome, that troublesome code willprobably diminish your companys rep-utation. But if its enforced as a genericpart evenly applied across the whole of your software cycle, that last 1 percent may not be so problematic.

Next, lets look at the possibleapproach of integration into the devel-opment process.

Nightly BuildUsing the Eclipse plug-in and playing with

code can provide insight into the quality of the reporting and the value of the dif-ferent issues measured. However, thisapproach isnt scalable to most large pre-existing code bases. Findbugs, in combi-nation with a nightly build structure, isbetter suited to the task.

Via the command line, I generate adynamic HTML report. To achieve this,the source code must have been compiledpreviously, and so this approach sits snug-ly in the nightly build infrastructure.

The workflow could be similar to thefollowing:

Build the source code via Ant,Maven or with your own specialized

tools Deploy Perform static analysis of code Once satisfied, tag code and deploy

from development to acceptanceenvironment

Perform functional tests with toolssuch as Anteater (http://aft .sourceforge.net)

Perform static analysis of code viaextra tools such as PMD or QJPro

Perform functional tests Stress test if acceptance and pro-

duction are realistically similar Deploy to productionI test regularly in development with

one static analysis tool with its own ruleset and then try out a couple of others inthe acceptance phase, but many testers

choose to use all or several of the tools inparallel near the beginning of the cyclein development.

I choose not to, so as to avoid over- whelming the audience with false posi-tives and other information. However, as your audience becomes more sophisti-cated, you can use more sophisticatedtools, as well.

To run the default tests against yourcompiled package, youll need to get tothe Findbugs bin directory and executethe following command:

./findbugs -textui -maxHeap 1024 -html:fancy.xsl-outputFile test.html location_of_your_compiled_source code.

-maxHeap is the maximum amount of memory Findbugs may consume.-html:fancy.xsl tells Findbugs to generate

an HTML report via an XSL transformfrom the built-in fancy.xsl style sheet, andtest.html is the html name of the report.

Ive run this command against a440,000-line Java code base. The report generation cost two or three hours of computational effort on a dual-core desk-top computer.

If you apply this effort to the night-ly build process, the report will be view-able by the developers when they comein to work the next day. Just add thecommand to a cron job, making surethat the generated report lies under thedocument root of a conveniently con-figured Web server.

Figure 3 is a screen grab of a realisticHTMLized report. The top row offers theoption of viewing bugs by category or

FIG. 1:THE FINDBUGS PROPERTIES PANE

FIG. 2: RESULTS FOUND BY THE FINDBUGS ECLIPSE PLUG-IN
http://aft.sourceforge.net/http://aft.sourceforge.net/http://aft.sourceforge.net/

8/13/2019 stp-2006-10


CODE REVIEWS

package, offering a dynamic JavaScript/Dom tree that you may navi-gate and helps you locate problems.

Refactoring in ReviewsCode reviews are excellent tools for spot-ting smelly codecode thats rigid andeasy to break, or too complex to under-stand. From my own experience, Inotice that defects tend to clusterhid-den in the smoke there tends to be fire.Therefore, even if the reported issuesare trivial, the relative number of defectsper subproject can suggest the directionof your vectored refactoring efforts.

Assuming that you have time to refac-tor and dont get stuck in the franticproblem-solving stage, Findbugs can beextremely useful.

Here are some rules of thumb that can help with refactoring:

Caught Exceptions: If your code base ismore than a bit buggy, log previously ignored caught exceptions. You may finda large amount of information deliveredto the log files. Therefore, start by trawl-ing the HTML report per package.

One motivation for removingignored exceptions is that the end usermay believe that their transactions havesucceeded when the issue has merely

been ignored. If the code sits in this part of the application, add end-user feed-

back logic. Finally, the code may failsomewhere else, but the problem may originate elsewhere. Logging evenunsuspected code may bring this typeof error to the surface.

Code complexity : Once the class lengthin lines of code is greater than, say, thelength of a page in A4 format or if nest-ed if statements are too nested, devel-opers will have more problems under-

standing the code. This challenge trans-lates into higher defect levels and many

extra opportunities for failure. Worsestill, high code coverage via regressiontesting such as Junit tests is much moredifficult to write in these situations.Therefore, keeping the complexity out of the code increases code quality.Complex code probably means that youneed to reexamine your design patterns.

Other refactoring tools such as PMD(http://pmd.sourceforge.net) andQJPro (http://qjpro.sourceforge.net)are superior to Findbugs at finding com-plexity.

These tools also work on uncompiled Java code, thus avoiding the sometimesunnecessary step of compilation. By adding a second or third tool, youincrease the number of rules checkedbut you also boost the background noiseof more false positives.

Finding the balance can be difficult.Some testers try a varied approach: Apply one tool daily and then occasionally ver-ify with a second and third tool.

Junk DNA: As code bases grow, func-

tional requirements change. Throughthis erosion process, parts of your codebase may no longer get regularly exer-cised. Removing this junk DNA helpskeep the developer concentrated. PMDhas an Ant task that spots duplicated codeand outputs the duplicates to a text filefor further analysis. You should throw away or factor out into common utility classes.

Performance: Findbugs looks for about 1020 potential performance issues suchas string concatenation within loops or when to make an inner class static, thusdecreasing memory footprints. Per-formance problems created in the heat

FIG. 3: SCREEN GRAB OF A FINDBUGS HTMLIZED REPORT

LISTING 1: A CLASS WITH DUBIOUS PROGRAMMING PRACTICES

import java.io.IOException;

public class FailOne {String test;Object example;

public FailOne(){this.example=new Object();

}

public void setAction(String action){action="I have changed";

}

public void badException()throws IOException{

System.exit(0);System.gc();try{throw new IOException();}catch(Exception e){

// Het lets ignore bad news}

}

public Object getExample(){return this.example;

}

public synchronized void setExample(Object example){this.example=example;

}

}
http://pmd.sourceforge.net/http://pmd.sourceforge.net/http://qjpro.sourceforge.net/http://qjpro.sourceforge.net/http://pmd.sourceforge.net/

8/13/2019 stp-2006-10

20/40
http://www.stpcon.com/

8/13/2019 stp-2006-10

21/40
http://www.stpcon.com/

8/13/2019 stp-2006-10

22/40

of coding are hard to spot in a forest of

lines and powerfully provisioned systemresources.Common errors: One of the stated

focuses listed on the Findbugs home-page is to eliminate commonly founderrors and typos.

Excellentand FreeI found Findbugs to be an excellentandfreestatic analysis tool. It can be usedas an Eclipse plug-in as a training tool orfrom the command line as an enterprise- wide tool.

Every new version brings with it theability to discover new issues. All in all,the tool is an honest broker that helps

developers keep their code to a consis-

tent minimum standard. Whether its Findbugs, PMD, QJProor another static code analyzer, this typeof tool, when placed in a nightly buildstructure, allows for daily reports onquality and zooms in on particular issues.It enables you to detect defect numbersper package and helps you determine where to start your debugging and refac-toring efforts.

Whats the main weakness of staticanalysis? It produces high volumes of detailed information. However, oncedevelopers accept this approach, espe-cially if its vectored, theyll reap signif-icant improvements in quality.


CODE REVIEWS

}catch(Exception e){ // Het lets ignore bad news

}

System.gc();

System.exit(0);

public void setAction(String action){action="I have changed";}

public Object getExample(){return this.exam-

ple;}

public synchronized voidsetExample(Object exam-ple){this.example=example;}

String test;

Ignoring exceptions canstop your code from failingin the short term.However, errors indirectlycaused by this may meanthat youre looking in the

wrong place in the codewhenever an issue arises.

During garbage collection,everything else in the givenJVM stops.Therefore, thisaction has potentiallysevere performance issues.Use this command sparinglyand singularly.

Unless you really want tostop your application totallyand kill the JVM, you

should throw exceptionsrather than exiting.

The input parameter actionshould never be changeddue to risk of side effects.This is usually an accidentaland hopefully rare mistake.

If you synchronize the set,for the same reasons youshould probably synchronizethe get method.

The field test is never used.In general, this is due toone of two reasons: thecode has still to be writtenor its obsolete due to thespeed of change.

DE:FailOne.badException()might ignorejava.lang.Exception

Dm:FailOne.badException()forces garbage collection;extremely dubious except inbenchmarking code

Dm:FailOne.badException()invokes System.exit(...),

which shuts down the entirevirtual machine

IP:Parameter action toFailOne.setAction(String) isdead upon entry but over-written.

IS:Inconsistent synchronizationof FailOne.example; locked50% of time

UG:FailOne.getExample() isunsynchronized,FailOne.setExample(Object)is synchronized

UuF:Unused field: FailOne.test

TABLE 1: WARNINGS, SOURCE CODE AND EXPLANATION

Error Source Line Mentioned Description
http://www.itko.com/

8/13/2019 stp-2006-10

23/40
http://www.enerjy.com/visible

8/13/2019 stp-2006-10

24/40

Get Quality

8/13/2019 stp-2006-10

25/40

8/13/2019 stp-2006-10


code base need to be tested? Must high-priority bugs be completely eliminated?Perhaps these goals are too aggressive.The absolute values of the metrics dif-fer based on each organizations cor-porate standards. Once the process isin place, these goals can be achieved by implementing a source-code control sys-tem, a unit-testing framework and/or acode analysis tool at the start of any development project. Automating any task, such as a build process, will alsohelp to minimize human error and gainrepeatable results.

To facilitate trend analysis, the high-est level of this phased approach, eachstage must be built upon the last, as out-lined above. It isnt necessary, however,to halt all development work to imple-ment a quality processthe process itself

wil l be bui lt over time. For example,organizations with a large installed codebase may want to ensure that all new

work is unit-tested so that over time, theirtest coverage steadily increases. Similarly,

with a source-code control system, froma particular install date, all code changescan be tracked. Watching these quality metrics improve over time as their pos-itive influence spreads through theentire development team is just one of the benefits of trend analysis.

Now, lets examine each level in a bit more depth.

Standards & Goals Level 1To begin implementing quality process-es, standards and goals must be set.Every organization has standards andgoals, but determining which are suit-able and which are generally accepted

as industry best practices is a different matter. Existing development teams canderive some standards from the processthey already have in place. Setting highstandards or goals at the onset ensuresthat subsequent steps in the process aresimilarly focused.

When determining a set of standardsand goals, team involvement is impor-tant because it will make the remainingstages easier to implement and ensurea smoother flow throughout the process.

When key team members participate indefining standards, not only does it ensure their buy-in, it also helps gen-erates good ideas.

Some typical Java development stan-dards include:

General code compliance to a set of defined standards (for example,code documentation and unusedelements)

Unit testing coverage to an accept-able percentage

Tracking bugs and issues back to

the relevant files and authors Customer sign-off on working pro-totypes that conform to a subset of requirements

Enforcing design patterns todecrease cyclic complexity andtight coupling

Tools Level 2Once a set of standards and goals is inplace, the next stage is to implement thenecessary tools that will help develop-ment teams meet these standards. Tofacilitate teamwork and streamlineprocesses, all or a subset of basic codingtools may already be in place. These may

include IDEs, source-code control sys-tems, code analysis tools, testing frame- works, coverage tools and bug trackingtools, with the understanding that thesetools make the developer more efficient and the structure and control of theproject more manageable. Many organ-izations are currently at this level.

These tools dont necessarily have tobe standardized across development teams, because some developers may have experience and preference usingcertain products. Its essential, howev-er, that they are used throughout thedevelopment process to drive produc-tivity, increase the quality of output andease the day-to-day tasks that put devel-opers at risk for human error.

The output from these tools and themetrics they generate is a critical stepin bringing increased visibility and

improved quality practices into thedevelopment process; for an example,Figure 1 illustrates the trends of threeindividual developers and their adher-ence to standard compliance over aspecified period.

Automation/Repeatability Level 3Manual processes, especially repetitivetasks, are highly error-prone. With thecorrect setup and configuration, processautomation within the development lifecycle will eliminate the potential forhuman error and speed product deliv-ery through repeatability. Productivity wil l also rise when boring , repetit ivetasks have been automated, leaving thedevelopment team free to focus on com-petitive enhancements.

Process automation can range froma developer pushing a single button toentirely automatic scheduling. Usingsuch tools will allow computers to per-form time-consuming and processor-intensive tasks offline or after hours.

Other tools facilitate automated report-ing (such as alerting the system archi-tect of a failed build via an SMS messageor pager alert). Another example is anautomated build process that is essen-tial for repeatable, clean builds. Javadevelopment teams may use an Ant orMaven script for one-button processingfrom compilation to deployment.

Measurement Level 4 At this level, organizations begin to con-cern themselves with quality results. Theprevious levels provide a good founda-tion for quality practices with the result-ing productivity improvements, but

PROCESS IMPROVEMENT

FIG. 1:THE TRENDS OF THREE DEVELOPERS

8/13/2019 stp-2006-10


quality can be realized only when out-put from the lower-level processes (ora combination of them) is measured.

To introduce new development prac-tices, measurements must be used toidentify where resources should be con-centrated. For example, in unit testing, the pass rate is a key measurement. But what is anacceptable pass rate? Is it enoughthat all 5,000 unit tests havepassed? If 100 percent of thosetests pass, this may be a sufficient level of quality. However, if the5,000 unit tests cover only 20 per-cent of the code base, the appli-cation quality level is less certain.

Only through measurement and cross-referencing of resultscan organizations obtain a clearpicture of quality. In this case,

it might be useful to implement a standard in which developerscant check in their work untilthe testing percentage has met a predefined threshold.

Analytics Level 5 With the implementat ion of qua lity practices and the use of coding tools,organizations are now able to collect metrics on key data such as standardscompliance, test coverage, customer sat-isfaction and design flaws. For devel-opment teams, analyzing data allowsmanagers to identify behavioral trendsor insufficient processes, but in reality,few organizations are truly at this level.To determine if an organization is set up to gather and analyze key data, askquestions such as:

Who was responsible for the last bug reported by a customer?

Was the code tested? Are a significant number of issues

assigned to this person?

Who on your team has no bugs out-standing, no standards violationsand meets any of the thresholds of test coverage set?

Pulling this data together providesteams with instant visibility into developerbehaviors and trends, enabling develop-ment managers to focus on necessary train-ing and resource allocation. See Figure 2for a sample at-a-glance view of project trends in various quality areas such as unit test coverage and standard compliance.

Trend Analysis Level 6Organizations that use trend analysis by correlating data from different sources

to adjust and streamline processes areahead of most software development organizations in their industry and thushave a significant competitive advan-tage.

At thi s stage, an organizat ions

processes are optimizing the efficiency and quality of project output anddeployed applications. By trackingtrends over time, management can iden-tify what is happening throughout thecourse of the development project, andanalyze the results to identify and cor-rect problem areas well before the pro-

jects last few weeks.

Visibility Into the Life Cycle A quali ty-driven development processprovides visibility into the development life cycle by consolidating data from key processes, tools and standards, such asunit testing results, code coverage per-centages, best practices, compliance tocoding standards, overall activity and

bug detection. Most Java development tools create metrics from which limiteddecisions can be made, but these aretypically mere snapshots of data at a sin-gle point in time. Development toolsalso dont consolidate data from each

other; a code coverage tool may not show who is responsible foruntested code, an important fac-tor when analyzing developerbehaviors and trends. What isneeded is a system that consol-idates data from key develop-ment tools and systems, allow-ing development organizationsto track code-quality improve-ments over time.

Refining the quality processis definitely an ongoing task,especially with external factorssuch as Sarbanes-Oxley driving

the change process. However, if an organization is at the trendanalysis level in the process,these changes will be easier to

implement because a good manage-ment system is already in place. Thisallows the development organization totake positive action with less financialimpact.

By implementing a quality-drivendevelopment process and analyzing thesekey metrics, development managers canoptimize the performance of their teams,minimizing time wasted on avoidablerework, tracking down bugs, or lengthy or ineffective code reviews. Development teams can quantify and improve appli-cation quality at the beginning of thedevelopment process, when its easierand more cost-effective to address prob-lems, and predict future trends based ontheir own experience.

PROCESS IMPROVEMENT

FIG. 2: AN AT-A-GLANCE VIEW OF TRENDS

Only through measurementand cross-referencing of results

can organizations obtain aclear picture of quality.

8/13/2019 stp-2006-10

28/40
http://www.s-3con.com/

8/13/2019 stp-2006-10

29/40

dependentmany tools on the market can do this task, and the informationoffered here is appropriate no matter

which tool youre using. However, since Idont refer to any specific commercialtool, some of the described steps may not be necessary for some tools.

Here, Ill present 13 steps to con- vert the script that a toolany toolcreates into a program that is reusable,structured, nondependent, easily maintainable and bulletproofed. Theresult is the conversion of the initialscript into a structured program,avoiding the most common problems.But first, lets review some underlyingconcepts.

Automation Objectives When I refer to visual automation, Imean automation through a graphical

user interface. The objective of any visual automation project is to obtain atest script that reproduces the usersactions with the GUI. But for goodautomation, the objective is to obtainscripts that:

Are able to run without any human intervention, typically after the compilation during thenight.

Jos Ramn Martnez is a quality assurancemanager in the I+D department of Nextel inMadrid.


Are Your

Automation

Scripts Too

Fragile?

Go Step By

Step to HelpMake Them

Bulletproof

P h o

t o g r a p h

b y

J o e

S t o n e

13Steps to Automation ScriptSuccess

By Jos Ramn Martnez

T he purpose of this article is to show how to make effective automation scripts. The article isnt tool

8/13/2019 stp-2006-10


AUTOMATION SCRIPTS

Are able to guarantee theregression of the prod-ucts tested. Thus, faced with new characteristicsof the product, environ-mental changes (operat-ing system, database andso on) or after correc-tions, the scripts canguarantee that the func-tionality of the product isthe same as it was before.Here, you suppose that the former behavior wasthe good one and that it was verified.

Are able to work in any known or unknown con-dition until they finish allthe work, always makingall possible test cases.

Show error localizationclearly and unmistakably in their results.

Allow each test in the set to berun independently.

Permit most tests of the set to becombined in any possible order.Thus, the combined test can bearbitrarily long.

Guarantee that at the end of theexecution they show a clear andinformative report of the resultsthat is auto-registered in an exe-cution repository.

When to AutomateIn choosing automation, considerthree conditions:

You can automate when the cost isrecouped in a few runsthat wont be done by handand in ashort period of time. If theautomation takes more time thanmanual execution and youre exe-cuting the manual test only a cou-

ple of times, why bother automat-ing? You can automate when no impor-

tant bug is found during the process.If an important area is not tested ornot fully tested, or if bugs prevent the test from running, dont wastetime automating.

You can automate an interface when the visual presentation andthe behavior are stable and cor-rect. If not, youll have to redo

your test again and again.If any of these conditions arent

met, automation is a waste of timenomatter what your manager says!

Inside the Tools:The Object MapTo make a good script, you must understand how the tools workat least the most common tools. All toolsoffer some useful information (prop-erties) of each element that the usertouches. This information is used forrecognizing and differentiating theelements.

With this information, the programgenerates a map of the objects (thename varies according to the program), with all the information that the pro-gram considers useful for recognitionand use of the elements in the script.Therefore, you have two sets of ele-ments to manipulate: the script codeand the object map associated with it. Youll use both in the automationprocess.

First, the environment must be sta-ble in behavior, with no changes fore-

seen in the elements placement,although this last condition is not asimportant recentlynow, programsuse more than a property for identifi-cation of elements, employing thresh-old and lets look for it around tech-niques. However, its still important,since a program that has moving ele-ments is probably changing more thanthe elements positions.

Now youre ready to get automated,step by step.

1. Plan the ScriptBefore doing any automation, youmust plan what to include in it. You

need to: Decide which action or

feature to automate,such as printing a docu-ment.

Choose what screens andelements in each screento use.

Examine and reproducethe actions you want toautomate and ensurethat no bugs hamper theactions. The executionpath must be clear.

Its also very convenient to write (in the place andformat you choose) a lit-tle description that asso-ciates the name of thetest with the things it does, so it will be easy to

run and understand thetest in the future.Its challenging to translate a

manual test into an automatic one.Further, its impossible to create an auto-matic test like a manual oneand even if it were possible, its not usually desirable.The manual test has many details; it depends a lot on the tester observationsand doesnt need any initial stable state. All of these details, and more, are impos-sible to achieve in an automatic test with-out high cost.

You can start from a manual test,but usually youll have to simplify it considerably.

2. Save/Save the EnvironmentIn the recording process, you must first save the objects youll use in the script.Keep in mind that the program lets youautomate only the objects it knows.Thus, you must teach the program themain elements it has to learn. The pro-cedure changes depending on the pro-

gram, but the key point is the same: You can never suppose that the pro-gram knows automatically all thethings you can see onscreen (all themain windows or all of the tabs); it willlearn only the things you touch (andsometimes the local father or chil-dren). Thus, the best procedure is totouch with the mouse all the elementsand sub-elements (for example, cells ina grid). In this way the program willlearn them for sure.

In other programs, the only proce-dure for learning the objects is to savea test script that includes them.

After saving the objects, now you

8/13/2019 stp-2006-10


proceed to record the first script.Thus, you record all the actions youveplanned and then stop recording.

Voil! You have a complete script that includes all the actions you want toautomate. This script contains aunique function that creates every-thing. This function can be very long,in a daunting block of code.

But youre not done yet: if you runthis initial script again, most of thetime it wont work properlyyou needto shape it.

3. Eliminate Variable DependenciesIn the next step, youll edit the object map and change the properties of theobjects the program uses for recogni-tion that can spoil the script. The val-ues youre looking for are the onesthat can change in subsequent execu-

tions of the test (and this executioncan be in another computer). Forexample, if the name of a buttonchanges (the same button is used for

various purposes, for example), or you want the application in various lan-guages, the logical course is to elimi-nate the name of the button as a prop-erty or, if the program allows it, tochange it to a regularexpression such as *.

Also, in the code, youmust eliminate all thecoordinates dependen-cies. That way youll avoidthe problem of not findingan element due to achange in its screen size orposition inside the applica-tion. Sometimes its impos-sible to eliminate the coor-dinates dependenciesbecause the only way torecognize some elementsis by using them.

4. Modularize theScriptThe result of the first recording is a monolithicscript without separa-tions. Now you must modify the scripts codeto make it modular.

First, each script must deal with only one appli-cation or feature to allow for ease of reuse. Sometimes you must move (or even erase) all the refer-ences to other applications andinclude (or record again) them in

another script. Thus, each script willdeal with only one application or fea-ture.

The next logical step is to modify the scripts, dividing them in functionsthat implement the different actionsthat youve recorded: SaveFile,LoadFile and so on.

These functions will follow the cri-teria of maximum cohesion and mini-mum coupling: The name of the func-tion will define a unique action that

will be the only thing the functiondoes, and the function wont dependon any other function. Also, the func-tions must be short.

During this process, you abstract thefunctions and group the common ones.Thus, in a natural way, you obtain func-tions with arguments; for example,GiveValueToUser(cell x , cell y , name ),

that you can reuse with minimal effort.Now, in every script, you write (not record) a main function that calls allthe functions youve created. Thismain function wont include any recording logic; its only mission willbe to check that all the functions

youve made work properly.Next, you create a new blank script,

in which you design adirectory functionmain that will drive thecalls to all the rest of thescripts and functions.

This directory func-tion will call only thesefunctions, without any recording logic. With thismain function, its easy tochange the order of thecalls, to extend the test (for example, with aloop), to make new test combinations and so on.

Notice that this direc-

tory function wont callthe main functions of every script, but the func-tions they have imple-mented. This main func-tion will drive one test, which will be consideredto be a test unit.

Finally, in anotherblank script, you write(not record) a mainfunction that calls all the

test units youve created (in any order,repeating some of them).

This function will contain the whole test you want to run, and this

test main will call the main functionsof every test unit that lacks recordinglogic.

5. Write CommentsTo make the functions youve createdunderstandable, you must add the nec-essary comments, in at least two places:

In the header of the critical func-tions, they explain what the func-tion does, its parameters and itsresults (error exits included). Thisis very important in functions that are considered library functions.If you have classes, you must include header comments in them.

In situations that require difficult decisions or decisions that arent immediately understandable.Remember that code is read many times, but is written only once.

As a norm, document the scriptscode the same way youd like to read it from somebody elses scriptdont for-get the Golden Rule.

6. Make Test Units Short With test units, you have to keep it brief. That way, you can limit the dam-age of any error in the middle of theentire test and can combine different test units like puzzle pieces. You canalso change a whole unit without affecting the rest of the test.

Debugging the scripts code is alsoeasier with shorter tests, and the teststhemselves are more robust becausethey have fewer dependencies. And, asnoted before, short tests can generally call many different scripts functions.

7. Extend the Scripts CodeNow that the program has learned theobjects, you can make the test grow from within the script without furtherrecording. That way, you enlarge the

test without having to record anythingelse. For example, its usual to intro-duce a loop, changing some values inevery step. All this implies that it isnecessary to know the language of thescript and to have some basic pro-gramming skills.

At another level, by combining thetest units in different order, youll gen-erate new tests in an easy way.

In this process, if it becomes neces-sary to learn new objects, there aremechanisms in the programs to learnthem. In some programs this willoblige you to record more actions; insome others, it will be an object map

AUTOMATION SCRIPTS

Its impossibleto create an

automatic test like a manual

oneand even if

it were possible,its usually

not desirable.

8/13/2019 stp-2006-10


AUTOMATION SCRIPTS

addition. Of course, if youneed to add more functionsor scripts, you can recordthem and use them in thetest.

Also, for extending thecode, dont feel constrainedby the things that the pro-gram can record. Automationtools give you a computer lan-guage that usually allows youto program nonvisual actionsfrom directly under theinterface. For example, youcan usually make direct data-base access for checking aresult.

8. Ensure the EnvironmentFrom the StartThe scripts youre creating

must always execute underthe same conditions as they were recorded. Only in this way can you guarantee that the results are comparable tothose from the recording. So

you must create functionsthat guarantee that the envi-ronment is correct. These functions

will check the environment at the start of the test unit and force it to be ade-quate if necessary. Typical functions:

Determine if the process isalready executing and eliminateor create it (depending on thecase). Note that most automationtools have problems if there ismore than one instance of theprogram running.

Check/delete/insert nec-essary data in the data-base.

See if memory is ade-quate.

Check the screen size,

number of colors, pres-ence of a mouse andother factors.

Some of these functions, with some generalizations, willend up as library functionsthat youll reuse later.

Another good practice is toreset the environment at theend of the test unit. Usually this implies cleaning the data-base of the data introducedduring the test unit. In this case, theobjective is not to ensure a cleanexecutionyou ensure this at the start of the testbut to avoid the accumu-

lation of garbage that can spoil othertest units.

9. Add Trace Data AndVerification PointsNext, youll include trace informationin the scripts. Trace information willinform you where the execution is and what the test is doing at any time,including error messages with essen-

tial information about the problem. With trace information, you can

follow the evolution of the execution.The traces text must be clear and

comprehensive. From thistrace information, the execu-tion is explained in an intelli-gible way. You can even savetrace information and consid-er it a baseline for comparingdifferent executions. Of course, the saved trace must be in a fully correct execu-tion.

In some programs, theprogram can automatically check the values of some visu-al elements during the execu-tion, called verification points .On checking these verifica-tion points, you can decide if the action the script repro-duces has run well or not.The verification points haveto be carefully chosen for

their relevance, and must show unequivocally the cur-rent state of operations.

The values chosen must beunique in the current state; any deviation is an error.Traditionally, the values chosenare text literals, but this isnt

mandatory; they can be in the form of images, windows presence/absence andso on. You can also implement these veri-fication points without reading the inter-face; for example, instead of reading thetext box with the results, you can get thisdata from the database (thus workingunder the interface).

Without traces or verification points,the only thing you can know about an

execution is if its globally right or wrong, but this doesnt meanthat every action in the test isnecessarily right. With traces and verification points, you canensure that the entire executionis right or pinpoint where it goes

wrong. Without them, a test isnot valid.

10. Protect Code AndRestore StateNow you have the code divid-ed in a natural way into func-tions. You need to enforce it toprevent the script executionending prematurely. You must introduce try-catch-finally structures to deal with possible

exceptions.In addition, you must create recovery

functions that recover the system to itsstate at the time the problems arose.

With traces and verification points,

you can ensure that your entireexecution is rightor pinpoint

where it goes wrong.

8/13/2019 stp-2006-10

33/40

AUTOMATION SCRIPTS

These functions can even reinitiatenets or machines or execute databasefunctions (restore the database from abackup, for example), and are so use-ful they usually end up as library func-tions. No matter how complex they are, they dont have many mainte-nance problems because they workunder the interface.

11. Reuse the Code AndGenerate Libraries

When an application is automated,many of the tests share common ele-ments, and its logical to try to reusefunctions and their objects in different scripts.

These reused functions must be very robust and strongly parameter-ized. They must also have comments,as well as a clear and explicative

trace.If the automation functions are welldesigned, they can be easily reused.They can be extended without havingto be rerecorded; youll have only tocombine the functions created. Onthe other hand, the code will be read-able and mantainable like any quality software program.

Finally, one of the objectives of effi-cient design is the generation of agroup of library functions that can bereused in different automations. Inthis case, these functions wont bereused between various scripts of thesame project, but in entirely different automations.

This implies that these functionsmust:

Be very defensive in their code.They must check everything, eventhe most obvious details.

Be strongly documented. Have maximum cohesion and no

coupling at all.

Have a clear, comprehensive tracethat needs no further addition. Be very robust against any possible

error. Be parameterized in a flexible way. Include informative comments

about the functions, their use, com-plex decisions and so on. Headercomments are mandatory.

Be stored apart from the rest of thescripts in a different file under con-figuration control.

Take great care

stp-2006-10

Documents

Transcript of stp-2006-10