LLooaadd TTeessttiinngg:: The process of analyzing software applications and support-

ing infrastructures to determine acceptable performance, scalability, capac-

ity and transaction handling capabilities by modeling real-world application

usage conditions and executing them against the application and support-

ing infrastructure under test.

INTRODUCTION

The year 2000 was an exceptionally strong year of market growth for load testingtools and related services. The worldwide market (inclusive of remote/hosted loadtesting services) for distributed environments exceeded Newport Group forecastsand reached $359.3 million, up 68 percent from its 1999 level of $214 million, and

outpacing the previous years percentage growth rate of 55 percent. Newport Groupforecasts that the market for automated load testing solutions will continue to grow,but at a slower pace in 2001 given the challenging economic conditions. We believethe market will reach $445.6 million in 2001 and reach nearly $880 million by 2004(see figure 1, Worldwide Load Testing Market Growth, Distributed Environments,1998 - 2004).

N e w p o r t G r o u p , I n c .IT Trends Research and Reporting

Source: Newport Group, Inc. 2001

Numbers InMillions

Annual Load Test Market Summary and Analysis

Analysts: Billie Shea and Kevin Gallagher

As it relates to all products and related services comprisingthe $823.7 million automated software quality (ASQ) mar-ket for distributed environments, the submarket of tools andrelated services tackling load testing continues to be themost significant driver of ASQ market performance, repre-senting a full 43.6 percent of the total market. This is anincrease of 9 percent over last year when the load testingsubmarket was representative of 35 percent of the overallASQ market for distributed environments. For purposes ofthis report, distributed environments include traditionalclient/server application environments, packaged ERP(Enterprise Resource Planning) and CRM (CustomerRelationship Management) application environments andthe full range of Internet, Intranet and extranet applicationenvironments.

The primary reason behind the strong growth of the loadtesting market is the continued increase and widespreadreliance on Web-based computing environments as businessrevenue generators and customer service tools. RecentNewport Group research reveals that on average todaysGlobal 2000 business attributes 15 percent of bottom linerevenue to their Web-enabled business channel and thatnumber is expected to grow to 22 percent by the end of2001. Couple the business importance of Web applicationswith the complex and diverse nature of these computingenvironments and it is clear why performance, scalabilityand overall reliability must be a top priority in support ofbusiness operations and growth. Therefore, as the businessvalue of load testing for Web-enabled environments hasbeen proven and recognized, load testing efforts are nowmore readily supported by business executives as an integralpart of the application development, deployment and man-agement process. The result is that load testing solutionsaimed at Web-based environments realized overall growthof 114 percent, growing from their 1999 level of $123.5 mil-lion to $263.8 million in 2000.

And, although adoption of load testing solutions outside ofthe United States accounts for only 29.5 percent of the mar-ket revenue in 2000, growth rates for load testing solutionsin the International markets outpaced (+79%) growth ratesin the United States (+64%). Europe and Asia/Pacific real-ized the strongest growth internationally. General on-siteconsulting services for load testing remained very strongthroughout the industry, resulting in more widespread mar-ket education about successful load testing strategies andtechniques. Lastly, no company shined more than market-leader Mercury Interactive, increasing their overall marketshare for load testing tools and services in distributed envi-ronments by an impressive 14 percent, and widening the gapbetween themselves and everyone else in the load testingtools and services space.

This annual market summary report will discuss the state ofthe market for load testing tools and related services in dis-tributed environments, inclusive of remote/hosted load test-ing services for the year 2000. This report will provide fore-casts and market trends as well as highlight market share indistributed environments and its largest subsegment, Web orIP-based environments. Further, load testing offerings fromthe leading vendors shaping this space will be profiled andsome new vendors will be introduced. Vendors in this reportrange widely in size and vary from vendors providing awide array of products and services to those that offerfocused point solutions in the product or service arena only.

COMMENTARY ON MARKET TRENDS

The year 2000 was the strongest growth year for load test-ing solutions to date. Once being a market with only a hand-ful of players, Newport Group recognizes the introductionof many new players over the past 24 months. Along withnew players, new approaches to load testing have emerged,most with the intention of driving price for load testing toolsdown. For example, a hardware-based load-generationappliance approach was recently introduced in addition tonetworks of peer-to-peer (P2P) architectures for generatingtraffic over the Web to test IP-based applications.

As a result of converging market conditions, the averageexpenditure for an automated load testing tool in 2000 was$33,000, 33 percent less than the average price of $49,500in 1999. Newport Group believes that a number of marketconditions attributed to the decrease. First, there was anincrease of vendor competition, with newer vendors lever-aging price in competitive situations to win deals. In addi-tion, some vendors chose to blend the use of in-house loadtesting with outsourced, hosted/remote load testing services.Therefore, end-users tended to purchase tools with lowerVirtual User (VU) licenses for testing throughout develop-ment and then chose to supplement their in-house testingwith hosted/remote services when high volumes of simulat-ed users were required. In 1999, end-users purchased toolswith an average license for 600 VUs. In 2000 that numberdropped to an average of less than 500. The number of userssimulated via hosted/remote load testing services rangedfrom 1000 to 5000, on average.

Another reason for the drop in average price is that a broad-er range of companies many smaller in size than the mar-ket has previously experienced are adopting load testingsolutions as they move their business critical applications tothe Web. Thus, based on a broader market appeal for loadtesting solutions and based on the current performance/scal-ability requirements for a greater number of small to mid-size companies, the average VU license has decreased. The

2

last factor that may have influenced the decrease, but to alesser extent, is the challenging economic climate, whichcaused IT spending to be scrutinized a bit more carefullytoward year-end.

The core strength of the load testing market still resides withthe handful of traditional players, and, of those players,companies offering a wide array of testing products andservices that extend beyond load testing, such as test man-agement, functional testing, component testing and per-formance monitoring and management solutions, realizedthe strongest growth in 2000. We believe they will also bethe strongest performers in 2001. Mercury Interactiveremains the dominant market leaderand the one vendor, which all othersaim to compete against.

Given the steady upward trend ingrowth, and the clear business recog-nition for the need and importance ofmeasuring and managing applicationperformance and scalability, webelieve the market will sustain somenew vendors, but only those with thestrongest and most aggressive com-petitive strategies. These new ven-dors will thus broaden and furtherenhance the overall growth potentialof the entire market. However, it isimportant to note that new players inthis space have entered a challengingmarket when going up against strongand highly aggressive traditionalplayers such as market-leaderMercury Interactive. As it standscurrently, the top five vendors in theload testing tools and related ser-vices space including MercuryInteractive, Rational Software,Compuware Corporation, SegueSoftware and Empirix account for a full 87 percent of thetotal revenue in 2000. And, consider that toward the end ofthe year 2000 and into 2001, some shakeout and consolida-tion among vendors has occurred. For example, service-onlyperformance testing and management vendor iSharp closedits doors and Keynote Systems purchased technology assetsfrom Envive Corporation.

HOSTED/REMOTE LOAD TESTING SERVICES

In 2000, multiple vendors both new and traditional, intro-duced remote or hosted load testing services, and wewatched as these types of services began to take root. By the

time the dust settled at the end of the year, hosted/remoteload testing services had accounted for slightly less than 3percent of the overall market. Hosted load testing servicesleverage the Internet to conduct load tests over the wire.With this model, load tests can be driven from one to manygeographic locations around the world to stress a Web appli-cation and infrastructure from outside of its firewall.Vendors competing in this space report that on average theircustomers choose to have user loads driven from three dif-ferent geographic locations and that it takes an average offour test runs to complete most service engagements. Themain benefit to this approach is its ability to test the firewalland ensure that appropriate ISP bandwidth is allocated and

that routers, switches and peering relationships will supportanticipated user loads. A hosted load testing service pro-vides a user-centric perspective of what will typically beexperienced in the live environment.

Of the newer, hosted load testing services-only vendors,Keynote Systems was the strongest performer with itsKeyReadiness service. Of the traditional testing tool com-panies now enhancing their current offerings withremote/hosted load testing services, Mercury Interactive andCompuware Corporation were the strongest performers,with Mercury Interactive being responsible for better thanhalf of the revenue generated from these new services. True

Worldwide Load Testing Market ShareDistributed Enviroments, 2000

Total $359.3 Million

(product & related services, including hosted/remote services)

figure 2

Mercury

Interactive 54%

Cyrano 3%

Compuware 9%Other 7%Segue 7%

Empirix 6%

Rational 11%

RadView 3%

3

Source: Newport Group, Inc. 2001

to their market leadership position, Mercury Interactive wasfirst-to-market to deliver hosted / remote load testing serv-ices.

Better than 70 percent of customers for hosted/remote loadtesting services were new to the discipline of load testingand on average these types of services were able to deliver3X performance gains and an average of 10 load testing-focused consulting hours to each customer. Newport Groupexpects that the percentage of companies adopting theseservices, who are new to load testing, will decrease in 2001as more IT teams that have load testing products in-housechoose to supplement their internal efforts with these typesof service offerings. Traditional testing tool companies suchas Mercury Interactive, Compuware Corporation and SegueSoftware should really benefit by leveraging their large baseof existing customers. Also noteworthy is that for compa-nies like Mercury Interactive, Compuware Corporation andEmpirix that have broad product and service offerings, host-ed/remote load testing services work to enhance productsales. On average, 42 percent of customers signing on forhosted/remote services went on to purchase product for in-house use in 2000.

HEALTHY INCREASES CONTINUED IN 2000

Sustaining the momentum gained in 1999, healthy increasesin load testing revenue spilled into the year 2000. The larg-er than forecasted growth validates the importance load

testing plays in todays technology-driven economy. The worldwide mar-ket for product and related services(inclusive of remote/hosted load test-ing services) for distributed environ-ments reached $359.3 million, up 68percent from its 1999 level of $214million. Newport Group defines relat-ed services as any consulting servicesthat are sold to an end-user client,which are over and above any servicesthat may be sold as part of the tool pur-chase. We factor in consulting servicesprovided by the tool vendors and theirdirect channel partners.

Once again, Mercury Interactiveretains the leadership position in thismarket space. In fact, the company sig-nificantly strengthened their leadershipposition in 2000, capturing a full 54percent of the worldwide market, animpressive 14 percent gain in marketshare over 1999. Rational Softwareonce again captured the second place

position with 11 percent market share and Compuware with9 percent, Segue with 7 percent and Empirix with 6 percentrounded out the top five vendors in this category (see figure2, Worldwide Load Testing Market Share, DistributedEnvironments, product and related services, 2000).

If we look at market share for distributed environmentsbased on load testing product only, which reached $288.7million in 2000, Mercury Interactive is again the marketleader with 58 percent market share. Rational Softwareremains in second with 11 percent market share, Empirix,Segue Software and Compuware Corporation share the thirdplace spot, each holding 7 percent market share (see figure3, Worldwide Load Testing Market Share, DistributedEnvironments, product only, 2000).

Web load testing revenue for products and related servicesin 2000 totaled $263.8 million, a 114 percent increase over1999. Second to Mercury Interactives 57 percent marketshare in this catagory, Segue Software posted the most rev-enue in the Web load testing space and holds 9 percent mar-ket share. Not far behind are Compuware and Empirix shar-ing third place, each with 8 percent market share, RationalSoftware with 6 percent and RadView Software with 4 per-cent (see figure 4, Worldwide Load Testing Market Share,Web Environments, product and related services, 2000).

If we look at market share for Web load testing based onproduct only, which increased 121 percent to reach $216

Worldwide Load Testing Tool Market ShareDistributed Enviroments, 2000

Total $288.7 Million (Product Only)

figure 3

Mercury

Interactive 58%

Cyrano 2%

Compuware 7%Other 5%Segue 7%

Empirix 7%

Rational 11%

RadView 3%

Source: Newport Group, Inc. 2001

4

million in 2000, Mercury Interactiveremains the market leader with a remark-able 63 percent market share. Empirix andSegue share second place in this category,each with 9 percent market share (see fig-ure 5, Worldwide Load Testing MarketShare, Web Environments, product only,2000).

For year over year percentage growth inWeb environments, Mercury Interactiveand Empirix both outpaced the 114 percentgrowth of the entire market. MercuryInteractive grew revenue in the Web loadtesting tools and related services space animpressive 169 percent. Empirix was notfar behind, growing their overall revenuein the Web space by 164 percent (see fig-ure 6, Market Share Growth for LoadTesting, Leading Vendors in DistributedEnvironments, Web). RadView Softwarewas also nearly on par with overall marketgrowth, increasing their revenues in theWeb load testing space by 112 percent.

THE INTERNATIONAL EFFECT

In 2000, 70.5 percent of the total load testing market residedin the United States. From 1999 to 2000, the United Statesexperienced growth of 64 percent ($154.7M to $253.3M)and the International markets experienced growth of 79 per-cent ($59.3M to $106M). For individual vendors, Paris,France-based CYRANO recorded the greatest percentage oftheir total revenue, 47 percent, as being derived fromInternational markets. Compuware followed with 37 per-cent and Mercury Interactive attributed 30 percent of theirtotal load testing revenues to International markets (see fig-ure 7, Worldwide Load Testing Revenue, DistributedEnvironments, 2000: U.S. vs. International Markets).

VENDOR PROFILES

The following sections provide brief summaries on the loadtesting offerings from leading vendors. Each section beginswith one or several company highlights from 2000.

Compuware Corporation (Farmington Hills, MI)

Year 2000 Company Highlight: In 2000, Compuwaremerged the development labs for their EcoSYSTEMS prod-uct line (production monitoring/management) with thedevelopment labs for their QACenter product line (applica-tion testing). As a result, development initiatives have

accelerated and tighter integration among the two productsuites has occurred at a more rapid pace. This shouldstrengthen their competitive position in 2001.

For load testing, Compuware offers QACenter PerformanceEdition. Performance Edition provides a script developmentworkbench consisting of Easyscript modules, which are cus-tomized to record application traffic and convert it intoeditable load testing scripts. Tool support for multiple Weband application server protocols includes HTTP/HTTPS,SSL and digital certificates, FTP, SMTP, POP3, TCP/IP,UDP and LDAP. Tool support for multiple CORBA-basedmiddlewares, specifically Inprise, Visibroker, IonaOrbixWeb and others supported through IIOP as well assupport for Microsoft technologies including DCOM, SQLServer and MS Exchange are available. For distributed envi-ronment development, Compuware supports Oracle (OracleSQL*Net and Net8), Sybase (DBLIB and CTLIB) andTUXEDO (6.3 or higher) as well as support for testingenterprise Java and wireless applications. For ERP testing,SAP R/3, Peoplesoft and Oracle implementations can betested using Compuware solutions. Additionally,Compuware provides support for legacy character-basedapplications with Telnet VT100-520 support and support forcustom middleware implementations through WindowsSockets.

The load testing technology in the Performance Edition (for-mally sold individually as QALoad) relies on multi-thread-

Worldwide Load Testing Market ShareDistributed Enviroments, 2000

Total $263.8 Million

(product & related services, including hosted/remote services)

figure 4

Mercury Interactive 57%

Cyrano 2%

Compuware 8%Other 6%

Segue 9%

Empirix 8%

Rational 6%

RadView 4%

Source: Newport Group, Inc. 2001

5

ed virtual users and the use of compiled C scripts to improvethroughput, reduce overheads and make effective use ofmachine resources, thus reducing the amount of hardwareneeded to run large tests. Performance Edition includes fea-tures such as automatic discovery of remote virtual users,automatic configuration and remote downloading of files(data, runtime). Any information or file downloading isdone at the beginning of the test to reduce network over-heads and avoid impacting the load test itself. PerformanceEdition can take activity into account, such as throughputrates, connection speeds and user think times.

In the QACenter Performance Edition, Compuware has bun-dled its load testing technology with server analysis agentsand file and data management (File-AID). The server analy-sis agents monitor each tier of the application infrastructureduring the load test. The technology, derived fromCompuwares production monitoring solution EcoSystems,provides detailed performance data that is automaticallycorrelated to load testing results. The combination of loadtesting, resource utilization, data management and userexperience monitoring tools work to identify the origin ofperformance problems and then provide the drill-downdetail needed to locate and resolve those problems.

In September of 2000, Compuware launched its remote loadtesting service, PointForward. It should be noted that inaddition to scalability testing, PointForward includes relia-bility testing, integrity testing and performance monitoring.

Via the PointForward service, Compuware engineers lever-age the QACenter load testing technology to drive loads ofhundreds or thousands over the Internet to a customer siteand measure performance and scalability. WithPointForward, customers can opt to have user loads drivenfrom multiple geographic locations worldwide.

Compuware entered the market for distributed functionaland load testing in 1996 via the acquisition of UK-basedDirect Technologies. They strengthened their position in theASQ space in 1997 with the acquisition of NuMegaTechnologies. NuMega offers a tool set aimed at defectmanagement, error detection and code coverage analysis forWindows. A positive byproduct of Compuwares perform-ance solutions is that they can be used in and across multi-ple environments. To aid customers who may require out-sourced services, Compuware maintains a substantial pro-fessional services team of 11,000 plus, about 7 percent ofwhich focus on Compuwares performance testing andmanagement solutions for distributed environments.Compuware services for load testing range from traditionalon-site consulting to recently added hosted/remote services.With an effective launch of their hosted services in the sec-ond half of 2000 and quick uptick in attracting customers,Compuware should continue to grow its remote load testingservice, PointForward in 2001.

Compuware is publicly traded on the NASDAQ under thesymbol CPWR. A sample of Compuware clients include

Home Depot, IBM, PepsiCo, PlanetRx,Kmart, Kohls, Lycos, Lante,CustomTracks, Sandbox.com, Dell UK,Little Tykes, Xactware, the DetroitNewspaper Agency, Campbell Soup, BellAtlantic, EMC Insurance, Fidelity, GlaxoWelcome, NCR, Mass Mutual, and others.Key Compuware business partnersinclude Microsoft, IBM, Oracle, Dell,SAP, Peoplsoft, Baan, Seibel, Citrix,Compaq, Sun, and Hewlett Packardamong others. In business since 1973,more information can be viewed atwww.compuware.com.

CYRANO (Paris, France)

Year 2000 Company Highlight: Theproduct is free, the model is open sourceand service contracts are reasonablypriced. In 2000, CYRANO is the one ven-dor that underwent the most dramaticchange to their business model. CYRANOpositioned itself as the first testing vendor

Worldwide Load Testing Tool Market ShareWeb Enviroments, 2000

Total $216.0 Million (Product Only)

figure 5

Mercury

Interactive 63%

Cyrano 1%

Compuware 6%Other 3%Segue 9%

Empirix 9%

Rational 5%

RadView 4%

Source: Newport Group, Inc. 2001

6

to provide their technology via theopen source business model.CYRANOs open source solutionoffers a core architecture of loadtesting and production monitoringtechnology to organizations at nocost.

In 2000, CYRANO positioned itselfas the first testing vendor to providetheir technology via the open sourcebusiness model. CYRANOs opensource solution, OpenSTA, offers acore architecture of load testing andproduction monitoring technologyto organizations at no cost. By pro-viding software for Web perform-ance testing (OpenSTA), databaseperformance testing and analysis(Impact and Workbench), databasemonitoring (Production), networkand firewall vulnerability testing(NV e-secure), legacy understandingand testing (Wincap and Test) andstrategic consulting and educationofferings, CYRANO is an end-to-end quality assuranceprovider to its customers. CYRANO offers integrated solu-tions, service and support to companies that want to mini-mize risk, benchmark Service Level Agreements and enableCapacity Planning for their IT infrastructures.

The United States office of CYRANO, based inNewburyport, Massachusetts, continues to feel the effects ofre-focusing their sales efforts in the States. As a result, keyexecutive positions have been shifted from the Paris, Franceheadquarters to the United States. This effort coincides withCYRANOs goal to focus on their pervious testing strength,a strategy they had moved away from several years back.CYRANO is a public company listed on the EuroNM of theParis Bourse (Reuters: CYRA.LN, Sicovam 3922). Formedin 1989 and publicly traded since 1998, CYRANO is head-quartered in Paris, France, with regional headquarters in theUK and USA. CYRANOs Web site address is www.cyra-no.com.

Empirix (Waltham, MA)

Year 2000 Company Highlight: Empirix became an inde-pendent company by merging RSW Software with HammerTechnologies (testing in the communications space), twoformer independent business units of Teradyne. The com-bined technologies from each respective company uniquelypositions Empirix to serve the testing needs of individualand/or integrated Web and voice applications. Empirix is

among the top five vendors in the market space for load test-ing solutions. In 2000, the company worked aggressively toexpand their testing offerings and added hosted load testingservices, component testing tools and performance monitor-ing services through a partnership with Internet qualitymeasurement firm, Gomez, Inc. Based on their performanceand forward-looking company strategies, Empirix shouldcontinue to strengthen their competitive position in 2001.

Empirix (formerly RSW Software) offers e-Load as part ofits integrated e-TEST suite. e-TEST suite handles function-al testing with e-Tester, load/performance testing with e-Load, production monitoring with e-Monitor, advanced testdata collection with ServerStats, analysis and reporting withe-Reporter and WebReporter, test scheduling/managementwith e-Manager and defect tracking with d-Tracker. Whenused together, they provide end-to-end use of a single set oftest assets, which ensures consistency in testing throughoutdevelopment and deployment of Web applications. Themultiple components that comprise e-TEST suite are tightlyintegrated and work well together; they also can be licensedand used on a stand-alone basis.

Requiring no programming, e-Load utilizes automatically-generated Visual Scripts and supports data-driven user loadsfor hundreds or thousands of simultaneous virtual users.The e-Load Scenario Manager lets you define custom load,stress, and performance scenarios that can be changed on thefly using different sets of scripts to represent a range of busi-

ness functions. e-Load leverages TrueLoad technology,

Market Share Growth for Load TestingLeading Vendors in Distributed Environments Web

figure 6

Mercury

InteractiveCyrano, IncCompuware

Corporation

Segue

Software

EmpirixRational

Software

RadView

Software

0

100

120

140

160

+103%

-8%

+169%

+29% +164%+54%

+112%

1998 1999 2000

80

60

40

20

7

Source: Newport Group, Inc. 2001

ness functions. e-Load leverages TrueLoad technology, anobject-level recording method that insulates the user fromminor changes to the HTTP, to emulate real-world browsertraffic. This tool is sophisticated in its ability to handle fre-quently changing session IDs typical in Web environments,therefore not producing a host of false test failures that canhappen as a result of continuously changing site informationfor dynamic web applications. e-Load also features auto-matic content validation so the user is assured that the appli-cation under test provides the correct responses under load,without the need for additional programming.

e-Load allows QA engineers to review snapshots of virtualuser interaction with the application in real time to aid inidentifying and correcting any scalability problems. e-Loadsupports real-time graphing and reporting. Test reportingcapabilities (common for e-Load and e-Monitor) allow thetool user to layer and view multiple test runs on a singlegraph. Reports that display server-side metrics in correla-tion with client-side metrics are available, along with theability to export test results into Excel style graphs andcharts using a reporting wizard. Summary reports are auto-matically generated in HTML format.

e-Load ServerStats collects metrics that are tied to specificcomponents of a complete Web infrastructure. TheServerStat component of e-TEST suite supports collectionof metrics from multiple Web, application and databaseserver environments running multiple operating systems. Itsupports the collection of Web server metrics from ISAPI-

based, NSAPI-based, APACHE, NTSE,as well as all UNIX (including Linux)and Microsoft Windows platforms. Inaddition, ServerStats supports the col-lection of application server metricsfrom BEA WebLogic, IBM WebSphere,COM+, Sun/AOL/Netscape iPlanet,Oracle, BroadVision, Allaire ColdFusion, ATG Dynamo, SunsNetDynamics, Apples WebObjects, andMicrosoft Active Server Pages (ASP).Database support includes MicrosoftSQL Server and Oracle.

Powered by e-Load, e-LoadExpert is ahosted testing service designed to verifythe performance of Web applicationsover the Internet. Customers of e-LoadExpert can watch test engineersfrom Empirix design and execute theirload tests online and then work withthem to interpret and analyze results.Empirix launched the eLoadExpert serv-

ice in October of 2000.

For load testing at the component level, Empirix offersBean-test, a browser-based tool for testing the scalabilityand integrity of middle-tier applications built withEnterprise JavaBeans (EJB). Optimized for leading appli-cation servers from BEA (WebLogic), IBM (WebSphere)and Bluestone, Bean-test generates multi-threaded Java testclients to measure performance, scalability and functionali-ty of EJB component-based applications as workloadsincrease. Most recently, Empirix announced FirstACT, anapplication component testing solution that performs loadand functional testing of Web Services and COM compo-nents.

Also from Empirix, the Hammer IT Contact Center TestSystem is an automated solution for acceptance testing andload testing of contact center applications and systems. TheHammer IT Contact Center Test System works by generat-ing thousands of test calls for CTI, IVR and ACD installa-tions. Hammer IT duplicates realistic customer calling con-ditions, dialing into the systems under test, entering accountnumbers and PIN numbers, listening to ensure that the rightprompt responses are being played, and then measuring sys-tem and network response times throughout each test call. Ifresponse times exceed pre-established thresholds or if anyother problem is found, the Hammer IT logs the error and itslocation in the call flow to a database and displays testresults in real-time. Hammer IT provides standard alertingcapabilities.

Other

Segue

Empirix

Rational

RadView

Mercury

Cyrano

Compuware

0 50 100 150

In Millions

Worldwide Load Testing Revenue

Distributed Environments, 2000United States ($253.3M) vs. International Markets ($106M)

figure 7

From 1999 to 2000, the U.S experienced 64% growth for load testing solutions ($154.7M to $253.3M)

and the International markets experienced growth of 79% ($59.3M to $106M)

International

U.S.

8

Source: Newport Group, Inc. 2001

A sample of Empirix customers includes American Airlines,British Telecom, Countrywide, Sprint PCS, MCI Worldcom,Computer Sciences Corporation, Peoples Bank, ManulifeFinancial, H&R Block, and Royal Bank of Canada. Partnersinclude BEA, ATG, BroadVision, Compaq, Microsoft,PWC, AnswerThink and Sapient, among others. Empirix isthe combination of RSW Software and HammerTechnologies, both formerly units of Teradyne, Inc., whichbecame an independent company in December, 2000. Moreinformation can be viewed at www.empirix.com.

Mercury Interactive (Sunnyvale, CA)

Year 2000 Company Highlight: Mercury Interactive sig-nificantly increased their market share for load testing,gaining 14 percent in distributed environment testing. Thecompany was added to the NASDAQ-100 index and cement-ed a strategic partnership with Akamai, a leading providerof distributed application and content delivery services.Mercury Interactive strengthened their performance moni-toring/management offerings, part of which included theacquisition of Freshwater Software in 2001. Based on theirimpressive performance, Mercury Interactive continues toshape and dominate this market space.

Primarily addressing the testing needs of distributed envi-ronments, Mercury Interactives LoadRunner is an enter-prise level load testing tool that measures and predicts sys-tem behavior and performance. The tool exercises an entireenterprise infrastructure by emulating thousands of users toisolate problems, optimize performance and acceleratedeployment. LoadRunners Controller acts as a central pointof control in order to organize, execute and manage load testscenarios. The Controller executes load tests by running andsynchronizing combinations of business processes withmultiple types of virtual users concurrently. As the virtualusers run and emulate the application clients, the Controlleruses its integrated real-time monitors to provide perform-ance information on all parts of the infrastructure under loadin order to help isolate the bottlenecks. Performance moni-tors are available for the most popular web servers, applica-tion servers, databases, network components and firewalls.For instance, an enterprise application can be tested withhundreds or thousands of desktop, terminal, Web and Javaclients to determine how Java users affect a system as theyaccess an ERP application over the Web.

Using LoadRunner, load tests can be run against any server,regardless of platform or geographic location. LoadRunnercan be used to load test applications running over many dif-ferent architectures including Web, Wireless, Streaming,ERP and CRM applications. The tools open API allowsusers and/or third-party vendors to integrate LoadRunner

into their environments. Additionally, LoadRunner is inte-grated with Mercury Interactives application performancemonitoring and management technologies, allowingLoadRunner scripts to be re-used to monitor application per-formance once an application has been deployed into pro-duction. In addition to LoadRunner, Mercury Interactivealso offers Astra LoadTest as part of the Astra suite of test-ing tools. Based on Visual Basic, Astra LoadTest is a lighter and subsequently less expensive load testing tool, requir-ing less coding for script development and focused on Webapplications only. For either tool, load scenarios and resultscan be saved to Mercury Interactives TestDirector reposito-ry for ongoing scenario reuse, test management and defecttracking.

Mercury Interactives overall market strength has movedcompanies such as Compaq, Dell, Data General, IBM SPC,IBM AS/400 Benchmark Center, Informix, Intel, HP,Microsoft, Oracle, Peoplesoft, Sequent, Sybase, and Unisysto rely on LoadRunner in their benchmark and competencycenters. Further, e-business vendors such as BroadVision,I2, Vignette, CacheFlow, Allaire Corporation, Ariba, Inc.,Bluestone Software, Calico Commerce, Inc., GemstoneSystems, Inc., nCipher, Microsoft, SilverStream, Symantecand Vision Software have standardized on LoadRunner intheir benchmark and performance centers.

LoadRunner offers support for HTTP, XML, SOAP, WAP, i-Mode, CORBA, DCOM, plus many more protocols andtechnologies. LoadRunners virtual users can run on sever-al different platforms including Windows NT and 2000,Solaris, AIX, HP-UX and Linux. In addition, MercuryInteractives LoadRunner includes online performance mon-itors, allowing identification and isolation of performanceproblems during the load test. Supported performance mon-itors are available for Apache, IIS, iPlanet, WebLogic,WebSphere, SilverStream, Ariba, ATG Dynamo, Oracle,SQLServer, DB2 and more.

Mercury Interactive also offers ActiveTest a hosted-loadtesting service. ActiveTest is driven by MercuryInteractives LoadRunner, and is designed to verify the per-formance of Web applications over the Internet. Leveragingthe current hosted infrastructure, the ActiveTest Service candrive transaction load simulations of over 2 million concur-rent users or an approximate Web server load of three billionhits per day. ActiveTest results reveal which areas of theapplication are in need of tuning to achieve better perform-ance. Based on customer observations and summary per-formance test reports generated by the ActiveTest Service,customers can assess their application weaknesses and makeimprovements to their systems.

9

Publicly traded on the NASDAQ under the symbol MERQ,Mercury Interactive is listed on the Standard and Poor 500index and the NASDAQ 100. A sample of MercuryInteractives customers include Bank of America, E*Trade,Gap Inc., Citibank, Barnsandnoble.com, PHH VehicleManagement Services, Sabre Group, Ford Motor Company,COVISINT, Lufthansa, JCrew.com, and Prudential amongthousands of others. A sample of Mercury Interactives part-ners include Andersen Consulting, Ernst & Young,Microsoft, BEA, iPlanet, marchFIRST, BroadVision,Allaire, IBM Global Services and many others. In businesssince 1989, more information can be viewed at www.mer-curyinteractive.com.

RadView Software (Burlington, MA)

Year 2000 Company Highlight: RadViewSoftware experienced a respectable growthrate of 112 percent and put aggressiveplans in place to expand their product linebeyond load and integrity testing. In 2001,they have begun to deliver new testingsolutions, specifically a functional testingtool. RadView also opened offices in theUK and Germany and forged partnershipsin an effort to extend their internationalpresence to Asia-Pacific.

Formally in business since 1997, RadView continues togrow their business due to their focus on testing Web appli-cations. The Companys cornerstone product, WebLoad,generates virtual user loads of hundreds or thousands to sim-ulate real-world traffic and measure and monitor applicationperformance, scalability and integrity. Test scripts are gen-erated in JavaScript and are customizable, which providesflexibility for a range of simple to complex testing scripts.Likewise, WebLoad requires its user to have experience inJavaScript programming.

RadViews differentiator is their focus on combining loadtesting with application integrity testing coupled with theirDOM-centric approach. To accomplish integrity testing,RadView opted to focus their testing approach on the DOM(Distributed Object Model). So in essence, WebLoad focus-es on the integrity of data and is not focused on the presen-tation of the GUI interface. As a reminder, the DOM is aW3C specification that evolved to bridge competing objectmodels from Netscape and Microsoft. The DOM workstoward ensuring that Web pages are presented in the samemanner, independent of the browser that displays them. Inaddition to centering their approach to Web testing on theDOM, WebLoad incorporates support for HTTP/HTTPS,Java, ActiveX, WAP, XML, as well as a feature named

Cruise Control, which is a predictive tool feature for auto-matically measuring performance thresholds and determin-ing concurrent usage limits. WebLoad runs on anyMicrosoft Windows workstation and/or server and on theLinux and Sun Solaris platforms, Solaris version 2.6 orhigher.

In early 2000, RadView introduced WebLoad ResourceManager. WebLoad Resource Manager is essentially theproductization of RadViews vision of bridging the gap inresponsibilities between different IT departments, such asdevelopment and quality assurance, within an organization.Acting as a central point of administration, WebLoad

Resource Manager is able to define andadminister testing activities, such as sta-tus and scheduling across a single depart-ment or an entire enterprise. The highlevel benefit lies in the fact that the manystages and departments involved in thedevelopment life cycle standardize on asingle solution to promote an easyexchange of information across depart-mental and/or geographic lines.

The Company raised $37 million throughan initial public offering completed inAugust 2000. RadView has more than1,000 customers worldwide including:

Fidelity, American Express, eBay, Anheuser-Busch,Vanguard, Sun Microsystems, Autodesk, and NortelNetworks. More information can be viewed at www.rad-view.com.

Rational Software (Cupertino, CA)

Year 2000 Company Highlight: Rational Software isamong the top five vendors in the load testing space onceagain for 2000. Rational integrated their performance test-ing software, formerly Rational Suite PerformanceStudio,into the Rational Suite TestStudio in order to offer customersa wide range of testing solutions within a single productsuite.

Through tools, best practices and services, Rational helps itscustomers overcome a dilemma they call the software para-dox - the need to deliver higher quality software in less time.Rational Suite TestStudio helps companies address the chal-lenges of the e-software paradox by providing an integratedautomated software quality (ASQ) solution that tests alldimensions of quality, including functionality, reliabilityand performance.

Rational Suite TestStudio includes a complete set of auto-

10

The overall market forload testing tools andrelated services in distrib-uted environmentsreached $359.3 million in2000 and Newport Groupestimates that numberwill grow to $879.2 millionin 2004.

mated testing and defect tracking tools, as well as theRational Team Unifying Platform, to help teams work moreproductively by providing common access to projectrequirements, change requests, test assets, and more.Rational Suite TestStudio is supported by Rational e-devel-opment services to help accelerate application developmentand implementation.

Rational Suite TestStudio provides the three levels of diag-nostic information needed to isolate the cause of perform-ance bottlenecks by analyzing the business transactions,low-level client calls, and system resources that are causingpoor performance. For example, performance tests withRational Suite TestStudio helps users decide when to tunethe back-end server by increasing system memory or CPUspeed. It also identifies specific areas of client, middle-tier,or server-side code that are causing performance problems.

Tools within Rational Suite TestStudio include the follow-ing: Rational Robot for functional testingof the entire application; RationalPureCoverage to identify untested codeand provide code-coverage analysis;application profiling; Rational Purify tolocate hard-to-find run-time errors;Rational Quantify to pinpoint perform-ance bottlenecks; and RationalTestFactory to detect run-time errors with-out user assistance and generate optimalscripts for regression testing.

With Rational Suite TestStudio, testers are unified with therest of the team by providing them use of the same tools,methods, data and metrics as used by developers. By beingintegrated with the Rational Suite Team Unifying Platform,Rational Suite TestStudio helps teams work more produc-tively by providing common access to project requirements,change requests and test assets.

Tools within the Rational Team Unifying Platform includeRational TestManager, for managing all test activities andartifacts; Rational Unified Process, providing browser-accessible knowledge prompting the team to use develop-ment best practices; Rational RequisitePro to enable team-based requirements management; Rational ClearQuest todeliver integrated defect tracking and change request man-agement; Rational ClearCase LT to control artifacts thatevolve over the course of development; and Rational SoDAto automatically generate project documentation.

All Rational products are supported by a worldwide serviceand support organization, which offers professional educa-tion and training, customized services and projects, mentor-

ing, consulting and technical support services. RationalSoftware is publicly traded on the NASDAQ under the sym-bol RATL. More information can be viewed at www.ration-al.com.

Segue Software (Lexington, MA)

Year 2000 Company Highlight: Segue Software realizedrespectable growth for load testing tools and related servic-es. The Company released an updated version ofSilkPerformer in December of 2000, enabling it to supportMicrosofts DCOM architecture and enhancing the prod-ucts security and reporting features.

Segue Software offers SilkPerformer, an integrated Web,application and database server testing tool. By simulatinghigh-volumes of Internet and database traffic, SilkPerformercan create realistic models to test Web, client/server, andERP applications. With the tools e-business focus,

SilkPerformer is designed to provide loadtesting capabilities across a number of e-business-related technologies including,HTML, XML, Java, JavaScript, ActiveXand browsers. Additionally, SilkPerformeroffers support for the many Web and appli-cation server protocols such as HTTP,HTTPS, LDAP, SMTP, POP3, FTP, IIOPand WAP , as well as tool support to teststreaming media.

SilkPerformer leverages its multi-machinecontroller technology to balance its simulated user loadsacross any number of load-driving machines, resulting in theability to conduct very robust load tests. In addition,SilkPerformer combines multiple transaction types and var-ied user connection speeds to run against the web applica-tion, while automatically randomizing user think times.With support for SSL, SilkPerformer is able to conductsecure recording to gauge the impact that securing an e-business system has on overall performance. SilkPerformerutilizes a Results Repository that can store load test infor-mation for analysis of performance factors such as scalabil-ity, SQL structure execution, Web server throughput, andconcurrency. By storing information in the ResultsRepository, performance factors can be modeled and ana-lyzed to assist in performance and business decisions.

Segue offers a range of consulting services fromeConfidence programs and performance audits to full tech-nical implementations of automated test services, includingnewly added hosted load testing services. Drawing on overa hundred e-business consulting engagements, SegueseConfidence Programs deliver the testing expertise that

11

Load testing solutionsaccount for 43.6% of theoverall market for auto-mated software quality(ASQ) solutions in distrib-uted environments.

helps companies to define and achieve their specific e-busi-ness reliability requirements. Segues eConfidence pro-grams include: eConfidence Functionality, eConfidencePerformance and eConfidence Production. Consulting cus-tomers include Bank of America, British Airways, ChaseManhattan Bank, Citibank, Compaq, Dell, DiscoverBrokerage, eCircles.com, Fidelity Investments, Hotmail,Infoseek, Morningstar, Netscape, Quote.com, Rueters andTickets.com among others. Most recently, Segue announcedeConfidenceScale (eCScale). Powered by SilkPerformer,eCScale is an expanded set of hosted load testing servicesdesigned to provide performance and scalability assess-ments of e-business applications. Hosted by Navisite,Segues eCScale service can be delivered via a hosted loadtesting service or through traditional on-site consultations.

Segue is publiclytraded on theNASDAQ underthe symbol SEGU.Key Segue clientsfor SilkPerformerinclude Fideli tyInvestments, eBay,Sun Microsystems,Charles Schwab,AOL/Ne t scape ,

Dell, State Farm, Merrill Lynch, Sears.com and Telcordia.Key partners include marchFIRST, PriceWaterhouseCoopers, Razorfish, Systest Labs, StarQuality, BMCSoftware and IBM Global Services, among others. In busi-ness since 1988, more information can be viewed atwww.segue.com.

NEW MARKET ENTRANTS

As the market for load testing expands, Newport Group con-tinues to see emerging companies introduce solutions.Some are existing companies that have added load testing totheir current offerings. Others are newly formed companiesaimed at catering to smaller businesses or touting newapproaches to the mass market. What they all have in com-mon is that they are companies trying to capitalize on agrowing market. Market share for the following companiesis relatively small in comparison to other vendors profiled inthis report and therefore falls into the other category of themarket share graphs contained in this report.

Atesto Technologies, Inc. (Fremont, CA)

During 2000, hosted testing services gained increasedacceptance and Atesto Technologies formally entered theautomated testing market with a range of Web-focused test-

ing services. Organized as a service suite, Atesto offershosted testing services in the following areas:

Atesto Function Checker: Tests and validates complex,static and dynamic Web applications.Atesto Load Modeler: Facilitates capacity planning andmanagement by measuring the response of Web sites underload conditions from a globally distributed test infrastruc-ture.Atesto Response Watch: Monitors the response of a liveWeb site over time from the perspective of the end customer.Automated alerts notify site administrators when customiz-able service level thresholds are compromised.

As it relates to the focus of this report, the Atesto LoadTesting service utilizes a Web-based interface to manage theperformance testing process. The Atesto ScriptRecorderallows automatic generation of scripts that can be directlyexecuted from one-to-any of the fifteen geographic loca-tions where Atesto servers are located. The recorder is aWindows based thin client that captures the sequence ofbrowser clicks constituting a transaction or business processand translates it into a test script. Scripts can be further cus-tomized to add functions like checkpoints, user think timesand variable specifications for data driven testing. TheAtesto Load Test sys-tem can recreate com-plex Web applicationusage scenariosinvolving multipletypes business transac-tions executing atvarying connectionspeeds, incorporatingdynamic session vari-ables, embedded frames, pop-up windows and more. Theload test process can also be customized for ramp-up and -cool down patterns.

Once the script is written and uploaded to the Atesto servers,load tests can be scheduled and executed (following routinesecurity checks) with hundreds or thousands of virtual usersvia the Atesto Web site. Real time graphical and tabularreport formats provide a detailed view of server and appli-cation performance under load conditions. An executivesummary report provides a condensed view of the mostimportant test results. Detailed reports are available forresponse times (for each script, location and access typecombination), transaction summaries (passes and failuresfor each transaction), server metrics (page views, through-put and number of hits), and failure analysis (failure typeclassification and raw failure data for every script). Detaileddata on location and the Internet backbone is also available.

In 2000, the market for loadtesting tools and related serv-ices in distributed environ-ments grew 68%. The sub-segment of the marketfocused on the Web realizedan increase of 114%.

In 2000, the market forload testing tools andrelated services grew by64% in the United Statesand 79% Internationally.

12

Understanding that Atesto is new to the automated testingspace, company historical performance information is notavailable. Nonetheless, Atesto recently (April 2001) closedon $15 million of second round financing. As a result of thisfunding, Atesto plans to explore partnership and growthopportunities. Atesto is privately held, with venture fundingfrom the Sprout Group, Labrador Ventures, DraperRichards, Argus Capital, Wilson SonsiniLaw Group and others. Additional infor-mation can be found on their Web site atwww.atesto.com.

Caw Networks, Inc.(Mountain View, CA)

Founded in 1999, Caw Networks billsitself an Internet infrastructure company.Caw Networks builds appliancesdesigned to help users strengthen andscale the Internet. The companysWebAvalanche product produces large-scale stress tests and executes themagainst a complete Web application infrastructure. CawNetworks differentiates itself by taking an appliance-basedapproach to load testing, offering a sophisticated hardwareappliance able to generate very high volumes of Web usersfrom a single machine. The WebAvalanche appliance stress-es both the application and network layers of a Web appli-cation infrastructure simultaneously.

Built with best-of-breed hardware, WebAvalanche can beused to build and execute large-scale load tests to definetraffic handling capabilities and aid in capacity planning.WebAvalanche is a rack-mounted 4U high hardware devicebuild primarily on Intel-based technologies. TheWebAvalanche appliance sits just outside of the web infra-structure, which may consist of routers, firewalls, load bal-ancers, web, application and database servers, etc.

The Web Avalanche load generator consists of a high per-formance micro-kernel operating system, proprietaryTCP/IP stack and an optimized network driver. A singleWebAvalanche appliance is able to maintain up to 1 millionsimultaneous connections, simulating in excess of 10,000HTTP GETs per second. WebAvalanche can serve as a com-plement to script-based regression testing tools. Regressiontools could be leveraged to run limited instances of scriptsto test specific features and functions of an application whileWebAvalanche stressed that same application under veryhigh loads. This combination would ensure that an applica-tions functional integrity was not compromised underextreme load conditions.

Prior to test execution, WebAvalanche provides the func-tionality for users to define profiles of the types of simulat-ed users that are to be generated. This solution providesflexibility for profiling simulated users at the network level,ensuring that tests do not operate from the assumption ofclean LAN traffic only. For example, connection profilescan be specified to vary network latency, packet loss rates

and network noise for each simulateduser. The fashion in which simulatedusers are to be routed through a webapplication infrastructure can be spec-ified. With WebAvalanche, simulatedweb traffic is configured and generat-ed at an aggregate level in contrast todetailed scripting at a per simulateduser level, resulting in less overhead intest set-up and script maintenance.

Simulated users are drawn from alarge number of source IP addressesand user behaviors such as think timesbetween page requests and HTTP

aborts can be randomly varied and weighted.WebAvalanche has a graduated load feature that allows thatload to be added incrementally to determine the effects onsystem performance. In addition, WebAvalanche automati-cally invokes ramp-down intervals upon reaching the testduration limit or if a user manually stops the test. This auto-matic ramp-down feature is important for monitoringwhether connections close appropriately, ensuring the mostefficient turnover for computing resources and thus a high-er level of system performance.

Upon test execution, WebAvalanche cycles through a list ofsite URLs on a per simulated user basis. As simulated usersare generated, the load generator automatically handles bothsession and persistent-based cookies. WebAvalanche alsosupports SSL, HTML forms and HTTP POSTS, withupcoming support for distributed mode operation and otherprotocols. Finally, following test execution detailed report-ing per URL and detailed TCP/HTTP analysis is delivered.More information on Caw Networks can be found on theirwebsite at www.caw.com.

Dieselpoint, Inc. (Chicago, IL)

As another new entrant, Dieselpoint, Inc., is the creator ofDieseltest, a low cost windows application that simulateshundreds or thousands of users hitting a Web site. Designedto run on client machines, Dieseltest is priced at $99 perCPU. More information, and a demo version designed tosupport 25 users, is available from their Web site at www.dieseltest.com

Of the $359.3 million in over-all load test tool and relatedservices revenue for distrib-uted environments, 73.4% or$263.8 million was gained as aresult of the need to test Webapplication environments, a15% increase over 1999.

13

Keynote Systems(San Mateo, CA)

Last year, Keynote made its foray intothe load testing market with the acqui-sition of Velogic (May, 2000). As acompliment to Keynotes performancemanagement services, Velogics load testing technology wasabsorbed into the Keynote offering under the nameKeyReadiness. KeyReadiness is a hosted load testing andperformance management service. More specifically,KeyReadiness was organized into 3 separate performancelevels. KeyReadiness SilverTest is a low cost introductoryversion of the load testing service. It includes a standardtesting plan, a load test of up to 6000 sessions per hour,basic performance reports and returned results within twodays. The KeyReadiness GoldTest is more robust, includinga more comprehensive test plan, a load test of up to 30,000sessions per hour, and in-depth reports. Results for thislevel of service are returned within three to seven days.Lastly, the Key-Readiness Platinum Test provides expandedload testing, geographic profiling of users, up to 60,000 ses-sions per hour, advanced reporting that includes bottleneckanalysis, session information and lost revenue analysis.

What is unique about the KeyReadiness service is that it canprovide revenue projections vs. load projections to deter-mine lost revenue as a result of performance degradation. Itis understood that certain assumptions are made when thesetests are designed, however, the projections still offer aunique performance perspective.

Recent Keynote headlines (July, 2001) announced the acqui-sition of Envive Corporation. Envive, based in MountainView California, was essentially a competitor to Keynotethat offered geographically dispersed self-serve load testingservices via the Management Service Provider businessmodel. With the acquisition, Keynote changed the name ofthe Envive service to Test Perspective. Based in San Mateo,California, more information can be found atwww.keynote.com.

Parasoft Corporation (Monrovia, CA)

Founded in 1987, Parasoft released the load testing func-tionality of its WebKing 3.0 product in November of 2000to compliment its error prevention and error detection suiteof tools. WebKing is a comprehensive development, test-ing, and management tool designed specially for dynamicWeb site developers and testers. WebKing automaticallyexposes load, construction, functionality, presentation, con-tent, and design problems and requires no script develop-ment. WebKing also provides an infrastructure that allows

users to automatically compile,deploy, and test back-end programsand related output pages; promptingusers to take advantage of testing earlyin order to spot critical problemsbefore they become widespread.Integrating WebKing into the Web

application development process works to prevent siteerrors, increase site reliability and improve the efficiency ofthe Web development process. The Parasoft Web site can befound at www.parasoft.com.

softlight technologies, inc. (Hamilton, NJ)

Founded in 1999, softlight delivers SiteTools 3.0, a suite ofperformance testing tools. In addition to a Web site moni-toring tool, the SiteTools Suite includes the SiteToolsLoader, a performance-testing tool designed to generateload against a Web server by recording and running scriptswith any number of simulated users with differing dataparameters. Scripts are customizable with the use of anincluded editor.

The SiteTools Loader is comprised of a MainConsole,UserManager, DynamicUser, Recorder and an Editor. TheMainConsole allows for the loading and running of tests.All scripts are validated during the load process in order toensure that any manual editing or script custimization hasnot affected the validity of the scripts. If any script fails val-idation, a detailed message is displayed showing the reason,the script file, and the line number of the failure. A test canbe stopped at any point and restarted if needed. Once a testis completed, a comprehensive set of reports is created inHTML format. Users can also choose to create a databasethat collates all user response information for reporting pur-poses. The UserManager handles the management of allsimulated users on a machine. It is responsible for the cre-ation of all users, as either a separate thread or a separateprocess. Besides user creation, it routes all commands andmessages to the appropriate destination. The DynamicUserhandles the actual user simulation. It runs all requests, val-idates the response, and logs all relevant information aboutthe request.

The average virtual user license for a loadtesting tool is 469, down 22% from 1999.

The average expenditure for anautomated load testing tool was$33,000, down 33% from 1999.

14

The Recorder allows query scripts to be recorded from useractions performed on a Web site. The recorder scripts canthen be edited and saved. Scripts can be recorded in twoways. The first uses the browser to record scripts by way ofthe actual high-level calls made to the requested URL, linkscalled by redirection, or links called to fill a frame within aframeset. Internal links to images, style sheets or other typesof internal links are not recorded and will be parsed out ofthe returning HTML and called during playback.

The second way to record a scriptinvolves setting up the recorder toact as a proxy. This will record allHTTP traffic being sent to and fromthe Web browser and the Web serv-er. The SiteTools Recorder will record HTTP traffic sent byany ActiveX control or JAVA applet residing in the current-ly displayed page. At this time, HTTP is the only protocolthat can be recorded through the proxy recorder. Queriesthat use HTTPS must be recorded through the Recorderbrowser.

SiteTools 3.0 is packaged and sold in three different ways.Customers can opt for the Developer Edition, whichincludes a license for one console and 50 virtual users; theEnterprise Edition, which includes a license for one consoleand unlimited virtual users; or the SiteTools Suite, whichincludes one license for SiteTools Enterprise Edition and tenlicenses of SiteTools Developer Edition.

A downloadable trial version of SiteTools, and more infor-mation, can be found at www.softlight.com.

Web Performance, Inc. (Research Triangle Park, NC)

Recognized as a smaller player in the automated testingmarket, Web Performance Inc. develops and markets WebPerformance Trainer, a cross-platform load testing tool.

Designed for medium and small development and testingteams, Web Performance Trainer delivers ease of use, whilesimulating up to 10,000 or more virtual users. It also pro-vides features such as modem bandwidth simulation, loadbalancing of virtual user engines, form data replacement,SSL, etc. Web Performance Trainer has been shipping sinceMay of 2000, and is available on a trial basis as a downloadfrom their site at www.webperformanceinc.com.

WebmasterSolutions (San Diego, CA)

WebmasterSolutions provides outsourced monitoring andload testing services. For load testing, WebmasterSolutionsoffers SiteStress on a subscription-basis. With the SiteStressservice, customers have the option to load test up to threetimes within a 15-day window of engagement. Once testsare complete, WebmasterSolutions engineers will analyzethe results to determine what might be causing failures orbottlenecks. Comparing results to similar systems,

WebmasterSolutions engineers will thenmake appropriate recommendations onimproving performance or fixing prob-lems. Current clients range from onlineretailers to online bank. Imformation

about WebmasterSolutions and SiteStress can be found atwww.webmastersolutions.com

THE BOTTOM LINE

As witnessed by market growth projections and the amountof new players entering the market space, the demand forsolutions that can define the limits of distributed applicationinfrastructures and work to maintain optimal performance ison the rise. Businesses clearly recognize the value in reduc-ing the risk of failures and poor performance and areincreasingly leveraging solutions that deliver them controlover the performance, efficiency and growth of their distrib-uted application infrastructures. Clearly the Web andstrengthening business focus on electronic commerce isdriving market growth, amplifying the requirement toemploy solutions that define and continually manage boththe integrity and the performance handling capabilities ofWeb-based application infrastructures.

Businesses are steadily increasing their investments intoquality improvement. Newport Groups annual researchinto Web application integrity, conducted inNovember/December of 2000, reveals that on averageGlobal 2000 businesses are investing $250,000 into testingtools, $237,500 into the hardware necessary to support anadequate test environment, $36,000 into outsourced testingservices and slightly more than $790,000 per year for staffand training. Newport Group expects that these investmentswill continue and that, specifically, the market for automat-ed load testing solutions will continue to grow, but at a slow-er pace given continued market expansion.

All company names and products mentioned herein are registered trademarks of their respective companies.

Load testing solutions accountfor 35.2% of the overall market for automated testing solutions.

15

About Newport Group

Newport Group is an independent information technology research firm. Founded in 1997, NewportGroup was created with the intent and interest to provide detailed research services to major corporations and software vendors that share a vital interest in information technology. Drawing onyears of IT research experience, Newport Group concentrates on selective IT research topics andtrends. The benefits of this exclusive research approach are passed on to the client base with demonstrated expertise and insight.

The information contained in any and all final written materials by Newport Group will be based on accurate reliable information.However, because Newport Group information is gathered through numerous sources, including third parties, we cannot warrantthat the final publication is complete and error-free. Market statistics reported in this document are based on Newport Group estimates and analyst opinions. Newport Group disclaims all implied warranties, including, without limitation, warranties of merchantability of fitness for a particular purpose. Newport Group or its affiliates shall have no liability for any direct, incidental,special or consequential damages or lost profits.

Copyright 2001 Newport Group, Inc. Reproduction is forbidden unless authorizedAdditional copies may be obtained by contacting 1-508-362-9799

All company names and products mentioned herein are registered trademarks of their respective companies.

3230 Main Street, P.O. Box 909, Barnstable, Massachusetts 02630Telephone: 508-362-9799 www.newport-group-inc.com Fax: 508-362-5229