Geronimo 1.1.1 Performance Report -...

42
GERONIMO 1.1.1 PERFORMANCE Understanding the current performance prole Matt Hogstrom [email protected] November 2006 Version 1.0 Geronimo 1.1.1 Performance Update

Transcript of Geronimo 1.1.1 Performance Report -...

Page 1: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

GERONIMO 1.1 .1 PERFORMANCEUnderstanding the current performance profile

Matt [email protected]

November 2006Version 1.0

Geronimo 1.1.1 Performance Update

Page 2: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Legal Stuff

Java 2 Enterprise Edition (J2EE), Java 2 Standard Edition (J2SE) and Enterprise Java Beans are trademarks or registered trademarks of Sun Microsystems, Inc.

This report characterizes performance using a benchmark sample. Further evaluation is required for suitability of any Application Server in its unique deployment scenario. Don’t go buying $30 million worth of hardware and blame me because you didn’t do any further

testing.

This report is not sponsored by, or endorsed by, the Apache Software Foundation. It is simply the work of a committer on the project.

This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 2.5 License. To view a copy of this license, visit http://creativecommons.org/licenses/

by-nc-sa/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.

Derivative works should refer to this document and author.

Geronimo 1.1.1 Performance Update

Page 3: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Acknowledgments 1

Summary 2

Introduction 3Disclaimers 3

Testing Environment 5Hardware 5Software 7

Workload 8

Results 9Web Tier 9EJB Primitives 12Trade Scenario 14

Geronimo Server Information 16

Appendix B - Runtime Parameters and Results 17PingServlet 17PingServlet2Writer 18PingServlet2Include 19PingServlet2Servlet 20PingJSP 21PingJSPEL 22PingServlet2JSP 23PingHTTPSession1 24PingHHTPSession2 25PingHTTPSession3 26PingJDBCRead 27PingJDBCWrite 28PingServlet2JNDI 29

Table of Contents

Geronimo 1.1.1 Performance Report i

Page 4: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

PingServlet2Session 30PingServlet2EntityLocal 31PingServlet2EntityEJBRemote 32PingServlet2Session2Entity 33PingServlet2EntityCo&ection 34PingServlet2Session2CMROne2One 35PingServlet2CMROne2Many 36Scenario - JDBC 37

Appendix

Geronimo 1.1.1 Performance ii

Page 5: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Acknowledgments

I would like to acknowledge the generous contribution of hardware used for this report. Intel Corporation has been immensely helpful in providing 4 systems for testing.. Three of the systems have been used in performance and regression testing of Geronimo and are the foundation of this report. The other system is used for TCK testing and other miscellaneous functions.

On a side note I’d like to thank Marc The’Berge from Intel who has been my primary contact for coordinating the use of these systems. Marc has been a a great partner over the years and a good friend. He has been very supportive of the projects goals and been very attentive to supporting us. Thanks bubba ;-)

Belinda, my wife, get lots of credit for being patient with me and my odd habits. I married so well, not sure why she got stuck with me.

Geronimo 1.1.1 Performance Report 1

Page 6: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

SummaryFor those that prefer the bottom line, here it is. The performance of Apache Geronimo has been characterized using the Apache DayTrader Benchmark Sample. The comparison target of Apache Geronimo is a derived metric based on testing and observation of several application servers (both Open Source and Commercial). The Performance Target (PT) is not necessarily indicative of any specific server’s performance, but represents the best possible performance of the servers tested. In some cases the Performance Target was inflated based on my expectations.

The benchmark application used for testing is flexible enough to compare performance of the Web, EJB and an application that represents a stock trading simulation. The Web tests show that Apache Geronimo is very competitive against alternatives in the marketplace. In almost all cases Apache Geronimo either was neck in neck with the other containers or exceeded their best number. Also, scalability of the Web tier to almost 100% while servicing 100 simultaneous clients shows there are no inherent bottlenecks that artificially limit its scalability. Note that the 100 simultaneous clients were zero think-time and were to generate a load. A separate set of tests will be conducted on multi-user scalability.

The EJB tier shows that where pass-by-value semantics are required Apache Geronimo has area for improvement. Apache Geronimo lacks two significant features (a Stateless Session bean cache and the ability to specify SELECT FOR UPDATE on EJB queries) that do not allow for good performance under a heavy workload. In essence, where multiple concurrent updates to tables represented by CMP Entity beans -911 deadlocks may be inevitable. The Apache DayTrader Application highlights this problem. These issues are being addressed in the upcoming Apache Geronimo 1.2 release.

Overall Apache Geronimo is very competitive for Web based workloads. Also, workloads that do not require pass-by-value semantics with EJB workloads also perform acceptably. Workloads using Container Managed Persistence may encounter deadlocks under high load.

Geronimo 1.1.1 Performance Update 2

Page 7: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

IntroductionThe goal of this report is to help users understand the relative performance of the Apache Geronimo Server 1.1.1. Performance is always a touchy subject and I wanted to share information about Geronimo without causing a significant amount of fuss. To accomplish this I decided to NOT compare Apache Geronimo directly to any other Application Server (Open Source or Commercial). Doing so would most likely cause a lot of hand waving that would detract from the primary purpose of this report. However, a performance report without something to compare to yields little in the form of a positive comparison as its difficult to understand your relative benefit. Toward this end I compare Apache Geronimo to a derived metric I call the Performance Target (PT).

The Performance Target (PT) is a number that represents, in general, the best performance number of those Application Servers tested. There is no hidden AppServer A or AppServer B in the report. Based on my testing I found that in many instances, Open Source Application Servers outperformed Commercial Application Servers while in other areas (specifically the EJB workloads) the Commercial Application Servers outperformed their Open Source counterparts. The Performance Target (PT) is a worst case scenario for the Apache Geronimo Server in that it represents the best number and a bit more in some cases. Bottom line is, these numbers represent a respectable throughput for any application server. That said, performance isn’t everything.

An Application Server could be the best performing one on the market and still not meet many of the Non-Functional Requirements (NFRs) that people look at when making a selection. Price, Performance, Usability, Footprint, among others. are all factors in making that selection. This report comments primarily on the performance component.

D I S C L A I M E R SIn the interest of openness and full disclosure the reader should be aware of the following facts:

First, the author, Matt Hogstrom, works for IBM. I am also a committer on the Apache Geronimo project. I’ve been a performance analyst for several years and was the Performance Architect for the WebSphere Application Server. I participated in ECperf 1.0 (JSR-004) as well as ECperf 1.1 (JSR-131). In addition, I represented IBM to the Standard Performance Evaluation Corporation (SPEC) and participated in the development of SPECjAppServer 2001, 2002 and 2004 in the OSG-Java subcommittee. I also like to SCUBA dive and have three very cool children. I don’t like cats, but we have one anyway.

Geronimo 1.1.1 Performance Update 3

Page 8: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Second, there are lots of choices out there for Operating System, database, etc. I had to make a decision and I chose SuSE Linux and IBM’s DB2. I used SuSE Version 10 with the following Kernel version “Linux version 2.6.16.21-0.8-bigsmp.” I chose SuSE simply because it is the distribution I’m most familiar with and I like it. I’ve used it on Intel as well as zSeries for several years and its my preferred distribution, and a personal preference. I don’t think the performance of another Linux distribution would be substantially different in terms of a performance result.

As far as databases are concerned I decided to use DB2 for a couple of reasons. First, I use it all the time so I’m familiar with it. I wanted to complete this report quickly and decided to avoid a learning curve of another database. I’d like to produce other results with MySQL but given a limited timeframe I chose to use what I know and works well. Oracle would have required me to get permission from Oracle to release benchmark results and I didn’t want to go down that path of having someone else review the results.. This is an Application Server test and not a commercial benchmark result like SPECjAppServer 2004. (http://www.spec.org/jAppServer2004).

As with any testing there are an infinite number of possible combinations of tuning options. I chose to use a set of generally accepted defaults and test with those. One could certainly turn out other results that would be higher or lower when tweaking those options. I’m very interested in options that I may not have tried so please feel free to provide feedback. Please send me e-mail directly at [email protected] and I’ll do my best to incorporate your ideas and feedback into future results.

Next, the other significant feature of this report is the choice of a Java Virtual Machine. I decided to use IBM’s 1.5 Version of their JRE for Linux (32-bit SR3). I had originally started testing with the Java 1.5 Virtual Machine from Sun but because it wasn’t clear to me that benchmarking with that VM was clear of legal issues. That said, also, I could not for the life of me to get the Sun VM past 91% CPU utilization. I have to admit that played a factor in my decision as well.

Geronimo is certified on a 1.4.2 JVM. However, since most people are interested in Web based applications and less in CORBA applications I felt that 1.5 was more relevant to a larger number of people. Apache Geronimo runs with either version but lacks CORBA functionality when running on Java 1.5.

Finally, I used a commercial load driver to run the workload. There was no fancy scripting used for the tests and Open Source load drivers were not fast enough for the high volume, low-level, primitives. I’d very much like to use something well known like Grinder or JMeter from the Open Source world but they couldn’t keep up (or maybe I was too boneheaded to figure it out. If someone has some suggestions or donations I’m interested.

Geronimo 1.1.1 Performance Update 4

Page 9: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Testing Environment

H A R D W A R E

Image

First, these systems were so excellent I have to add some personal comments here. The AppServer (Gordy) was a two chip / dual core system and ran as a 4-way. I hadn’t realized how fast these machines were. With the 4MB of L2 cache they are almost unstoppable. They ran really well and the upgrade from dual-core to quad-core was so simple. Update the BIOS, swap the chips and bada bing...I had 8-ways. I used the 8-ways for the driver system (Wilbur) and for the database system (Porky). I hate shameless plugs but these machines really impressed me.

I had to run the AppServer system as a 4-way in these tests as there wasn’t enough driver capacity for the low-level primitives (like PingServlet). Rather than configure / re-configure the environment I chose to leave the setup in a constant configuration. Here is the block level-view of the setup.

Geronimo 1.1.1 Performance Update 5

Page 10: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Porky and Wilbur are 8-way 2.67 Ghz Intel Systems (2 Quad core CPUs) and Gordy (the Application Server test machine was left as a 2.67 Ghz Dual Core system (4-way). The Controller was actually a Mac Book Pro running the load controller which simply started the workload and gathered the run statistics.

The Application Servers tested were all run with a 1 gigabyte heap size.

Geronimo 1.1.1 Performance Update 6

Page 11: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

S O F T W A R EOperating System:

SuSE 10 Linux (used on all 3 servers)Linux e-gordy 2.6.16.21-0.8-bigsmp #1 SMP [snip] i686 i686 i386 GNU/Linux

Application Server:

Apache Geronimo 1.1.1 - Tomcat J2EE Distribution

Java Virtual Machine:

“J2RE 1.5.0 IBM Linux build pxi32dev-20061002a (SR3) “

Database Software:

DB2 V9.1 (used on Porky)DB21085I Instance “db2inst1” uses “32” bits and DB2 code release “SQL09010” with level identifier “02010107”.Informational tokens are “DB2 v9.1.0.0”, “s060629”, “LINUXIA32”, and Fix Pack “0”.

Geronimo 1.1.1 Performance Update 7

Page 12: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

WorkloadThe workload used to generate these test results is a J2EE application called DayTrader. It is an application built around the paradigm of online stock trading. Users log in, look at their portflio, look up quotes and then buy and sell shares of stock. In addition to the full workload runtime of Stock Trading the application also contains a set of primitives used for performance as well as functional testing.

The application is very flexible and can execute in a variety of runtime modes. Each of these runtime modes exercise various elements of the J2EE architecture. It includes runtime modes for standard Web Tier applications that are based on JSPs and Servlets, JDBC and EJB as well as a WebServices mode for conducting remote operations. For this report, since Apache Geronimo was not able to execute in EJB mode under load I only included the JDBC runtime mode.

The application was originally built by IBM as a way to characterize performance. It was used internally and donated to Apache Geronimo in 2005. Even though the application is maintained at the Apache Software Foundation it does not depend on Geronimo or in any way favor that Application Server. Work is currently being conducted on DayTrader Version 2.0 to enhance the runtime modes and prepare it as a benchmark example for Java Enterprise Edition 5.0.

More information about the application can be found at http://geronimo.apache.org/daytrader.

Note: The website is in flux (at the time of publish) If the content is not currently visible check back in a few days.

Geronimo 1.1.1 Performance Update 8

Page 13: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

ResultsThe results of the testing are broken into three categories. The categories are Web Tier, EJB Primitives and Trading Scenario.

The Web Tier contains simple primitives that start with the simplest of primitives, PingServlet, and build to increasingly complex tests up to PingServlet2JNDI. The goal of these tests are to characterize the runtime performance of the various components of the Web Container; these results were based on the Geronimo distribution that includes Tomcat.

The name of the primitives covered in this section all begin with the simple primitive PingServlet. All other primitives are built on this foundation. Since the other primitives are built on PingServlet, in most cases, the PingServlet prefix is not mentioned in the charts. So, for example, where you see 2Include you can assume that the actual primitive name is PingServlet2Include.

The tests were gathered by warming up the workload for a few minutes and then stopping it. The server was allowed to settle down and then the workload was restarted and run for approximately 2 minutes. During this time the server reached a steady state and the throughput was captured and is measured in Requests Per Second. The goal for all tests was to achieve 100% CPU utilization and a steady throughput.

W E B T I E RThe Web Tier is by far the best performing of the Apache Geronimo Server. For all operations good CPU utilization was obtained (in excess of 98% on the 4-way Application Server). Unfortunately, for PingJDBCWrite the database system was not able to keep up with the amount of disk write activity and only a CPU utilization of approximately 73% could be obtained. I’m working to improve performance by using RAM disks for the logs as I’m more concerned with the Application Server performance than the database. For this reason I have excluded the PingJDBCWrite from the tests as this was a problem for all tested Application Servers.

All of the following results are measured in Requests Per Second which is the number of round trip requests sent from the driver to the tested Application Server.

Geronimo 1.1.1 Performance Update 9

Page 14: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Servlet, HTTPSession1 and PingJSP all show excellent competitive results compared against the Performance Target (PT). In the initial set of tests show that Apache Geronimo is on a footing with the other competitors in the arena.

Just like the previous primitives the PingServletWriter and Servlet2Servlet show very competitive results.

Geronimo 1.1.1 PT

0

21,250

42,500

63,750

85,000

Servlet HTTPSession1PingJSP

Geronimo 1.1.1 PT

0

21,250

42,500

63,750

85,000

Writer2Servlet

2JSPEL

Geronimo 1.1.1 Performance Update 10

Page 15: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Finally, the JDBC primitives also show excellent performance. The significantly higher JNDI is due in part to the fact that Geronimo uses a simple HashMap for JNDI name resolution where the other implementations use an mutable Global JNDI implementation.

Overall for applications that are dependent on Web Tier and JDBC applications Apache Geronimo is very competitive in the variety of choices available to users.

Geronimo 1.1.1 PT

0

12,500

25,000

37,500

50,000

JDBCReadJDBCWrite

2JNDI

Geronimo 1.1.1 Performance Update 11

Page 16: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

E J B P R I M I T I V E SThe EJB Primitives exercise specific paths through the EJB container. Just like the primitives in the web container, these tests are driven from a servlet which in turn exercises the part of the container the performance engineer wants to focus on. These primitives start with simple Servlet2Session and then work through a variety of increasingly complex entity bean tests that include local and remote interfaces along with Container Managed Persistence including Container Managed Relationships (CMRs).

Apache Geronimo does not perform as well as the competition when running in a pass-by-value mode of operation (J2EE default mode of operation). In many instances this is not as much of an issue due to the fact that most containers run in a mode of operation that bypasses parameter copying but for those that require the isolation required through value copying Apache Geronimo is at a disadvantage. You’ll note that for operations involving Local Interfaces (2EntityLocal) the performance is again on par with the competition. EntityRemote simply highlights the performance degredation noted about parameter copying. Apache Geronimo allows for the disabling of parameter copying (like most Application Servers) but I have not re-run these experiments in this mode of operation.

The next set of primitives show an area that needs improvement which is that of complicated SQL operations using CMRs and Collections.

Geronimo 1.1.1 PT

0

7,500

15,000

22,500

30,000

2SessionEJB 2EntityLocal 2EntityRemote

Geronimo 1.1.1 Performance Update 12

Page 17: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

In each of the above primitives the sequence of events is PingServlet2Session2... One of the areas that needs to be investigated further is the comparison of these same primitives without parameter copying as indicated previously.

Overall, the performance of EJBs is competitive when using local interfaces. Where an application developer requires the isolation provided via pass-by-value, Apache Geronimo lags the other alternatives. However, as previously noted, many installations run in a pass-by-reference mode and for these deployments Apache Geronimo will continue to provide competitive performance for those applications.

Geronimo 1.1.1 PT

0

2,000

4,000

6,000

8,000

2EntityCollection2CMROne2One

2CMR2One2Many

Geronimo 1.1.1 Performance Update 13

Page 18: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

T R A D E S C E N A R I OThese numbers represent the Stock Trading Simulation running in JDBC Mode. In this mode of operation Servlets, JSPs and JDBC are used to simulate the trading scenarios. Apache DayTrader provides a built in Servlet to simplify testing in this mode of operation. This Servlet is called TradeScenarioServlet and controls the various interactions of the trades. Logins, Quotes, reviewing Profiles and selling holdings are executed in a random fashion based on the cardinality of the database tables. For these tests I ran with a user population of 500 and a stock quote population of 1000.

The overall CPU utilization for the Geronimo server being tested was 93.7 percent. it appears that the database system in use was somewhat limited in its disk subsystem. Other servers ran slightly hotter or about the same as the Apache Geronimo server.

Apache Geronimo is competitive in terms of delivering a respectable set of performance results when running Apache DayTrader in Direct Mode which depends on JSPs and JDBC for its operation.

Although DayTrader can run using EJB’s Apache Geronimo lacks two features required to sustain heavy load in this mode. These are a Stateless Session Bean Cache as well as the ability to specify whether SQL operations should be conducted with a SELECT FOR UPDATE or not. These features are being added to Apache Geronimo 1.2 which will be

Geronimo 1.1.1 PT

0

1,000

2,000

3,000

4,000

JDBC Direct

Geronimo 1.1.1 Performance Update 14

Page 19: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

available in a month or so but the 1.1.1 version could not successfully execute this runtime mode without encountering SQL -911 deadlocks.

Geronimo 1.1.1 Performance Update 15

Page 20: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Geronimo Server Information Version: 1.1.1 Download Location: http://geronimo.apache.org/downloads Distribution: J2EE - Tomcat

Download Size: 35MB Unzipped Disk Size: 41.6M Disk space after initial startup: 52.5MB

Initial Startup time: 15 seconds After DayTrader 1.1.1 installed: 19 seconds

Java Command Line: java -Xmx1g -Xms1g -jar server.jar

Appendix A - Geronimo Server Information

Geronimo 1.1.1 Performance

Page 21: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

Appendix B - Runtime Parameters and Results

P I N G S E R V L E T

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:21 Clients completed 0/100 Page elements 11772678 Page element throughput 83034.828 /s Transactions 11772660 Transaction throughput 83034.703 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 11772658 Page throughput 83034.688 /s HTTP data read 4412.206 MB HTTP data written 4591.368 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 17

Page 22: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 W R I T E R

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServletWriter.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:17 Clients completed 0/100 Page elements 11056283 Page element throughput 80652.758 /s Transactions 11056267 Transaction throughput 80652.641 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 11056265 Page throughput 80652.625 /s HTTP data read 4516.067 MB HTTP data written 4378.315 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 18

Page 23: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 I N C L U D E

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Include.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:21 Clients completed 0/100 Page elements 6785818 Page element throughput 47847.766 /s Transactions 6785800 Transaction throughput 47847.641 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 6785800 Page throughput 47847.641 /s HTTP data read 3367.056 MB HTTP data written 2700.786 MB HTTP avg. page element response time 0.002

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 19

Page 24: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 S E R V L E T

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Servlet.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:03:04 Clients completed 0/100 Page elements 10333710 Page element throughput 55950.438 /s Transactions 10333693 Transaction throughput 55950.344 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 10333691 Page throughput 55950.336 /s HTTP data read 4428.819 MB HTTP data written 4112.847 MB HTTP avg. page element response time 0.002

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 20

Page 25: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G J S PConfiguration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingJsp.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:29 Clients completed 0/100 Page elements 10174203 Page element throughput 68244.312 /s Transactions 10174186 Transaction throughput 68244.195 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 10174184 Page throughput 68244.180 /s HTTP data read 5920.621 MB HTTP data written 4425.805 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 21

Page 26: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G J S P E LConfiguration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingJspEL.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:35 Clients completed 0/100 Page elements 3004921 Page element throughput 19289.393 /s Transactions 3004903 Transaction throughput 19289.277 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 3004903 Page throughput 19289.277 /s HTTP data read 5680.636 MB HTTP data written 1153.917 MB HTTP avg. page element response time 0.005

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 22

Page 27: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 J S PConfiguration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Jsp.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:45 Clients completed 0/100 Page elements 8117802 Page element throughput 48908.020 /s Transactions 8117787 Transaction throughput 48907.930 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 8117786 Page throughput 48907.922 /s HTTP data read 5003.296 MB HTTP data written 3628.686 MB HTTP avg. page element response time 0.002

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 23

Page 28: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G H T T P S E S S I O N 1Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingHTTPSession1.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:53 Clients completed 0/100 Page elements 12417009 Page element throughput 71531.086 /s Transactions 12416998 Transaction throughput 71531.016 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 12416992 Page throughput 71530.984 /s HTTP data read 6266.697 MB HTTP data written 5513.171 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 24

Page 29: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G H H T P S E S S I O N 2Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingHTTPSession2.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:44 Clients completed 0/100 Page elements 10860713 Page element throughput 66191.164 /s Transactions 10860701 Transaction throughput 66191.094 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 10860699 Page throughput 66191.086 /s HTTP data read 5479.051 MB HTTP data written 4822.179 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 25

Page 30: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G H T T P S E S S I O N 3Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingHTTPSession3.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:48 Clients completed 0/100 Page elements 11287667 Page element throughput 66990.312 /s Transactions 11287652 Transaction throughput 66990.227 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 11287647 Page throughput 66990.195 /s HTTP data read 5968.018 MB HTTP data written 5011.745 MB HTTP avg. page element response time 0.001

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 26

Page 31: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G J D B C R E A D

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingJDBCRead.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:45 Clients completed 0/100 Page elements 3848492 Page element throughput 23309.361 /s Transactions 3848472 Transaction throughput 23309.240 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 3848472 Page throughput 23309.240 /s HTTP data read 2608.098 MB HTTP data written 1504.792 MB HTTP avg. page element response time 0.004

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 27

Page 32: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G J D B C W R I T E

Data omitted until database disk subsystem can be upgraded.

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 28

Page 33: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 J N D IConfiguration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2JNDI.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:59 Clients completed 0/100 Page elements 8538928 Page element throughput 47469.609 /s Transactions 8538914 ransaction throughput 47469.531 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 8538913 Page throughput 47469.523 /s HTTP data read 3694.145 MB HTTP data written 3372.908 MB HTTP avg. page element response time 0.002

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 29

Page 34: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 S E S S I O N

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Session.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:53 Clients completed 0/100 Page elements 2018866 Page element throughput 11602.878 /s Transactions 2018847 Transaction throughput 11602.769 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 2018847 Page throughput 11602.769 /s HTTP data read 1207.167 MB HTTP data written 803.541 MB HTTP avg. page element response time 0.009

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 30

Page 35: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 E N T I T Y L O C A L

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2EntityLocal.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:52 Clients completed 0/100 Page elements 1599402 Page element throughput 9272.917 /s Transactions 1599383 Transaction throughput 9272.807 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 1599383 Page throughput 9272.807 /s HTTP data read 1340.891 MB HTTP data written 642.992 MBHTTP avg. page element response time 0.011

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 31

Page 36: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 E N T I T Y E J B R E M O T E

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2EntityRemote.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:53 Clients completed 0/100 Page elements 250961 Page element throughput 1449.904 /s Transactions 250942 Transaction throughput 1449.794 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 250942 Page throughput 1449.794 /s HTTP data read 212.689 MB HTTP data written 101.170 MB HTTP avg. page element response time 0.069

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 32

Page 37: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 S E S S I O N 2 E N T I T Y

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Session2Entity.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:46 Clients completed 0/100 Page elements 229875 Page element throughput 1384.047 /s Transactions 229856 Transaction throughput 1383.933 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 229856 Page throughput 1383.933 /s HTTP data read 177.594 MB HTTP data written 93.132 MB HTTP avg. page element response time 0.072

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 33

Page 38: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 E N T I T Y C O L L E C T I O N

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Session2EntityCollection.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:59 Clients completed 0/100 Page elements 223824 Page element throughput 1244.926 /s Transactions 223804 Transaction throughput 1244.815 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 223804 Page throughput 1244.815 /s HTTP data read 335.064 MB HTTP data written 92.920 MB HTTP avg. page element response time 0.080

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 34

Page 39: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 S E S S I O N 2 C M R O N E 2 O N E

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Session2CMROne2One.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:53 Clients completed 0/100 Page elements 239119 Page element throughput 1374.856 /s Transactions 239102 Transaction throughput 1374.758 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 239102 Page throughput 1374.758 /s HTTP data read 197.422 MB HTTP data written 97.833 MB HTTP avg. page element response time 0.073

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 35

Page 40: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

P I N G S E R V L E T 2 C M R O N E 2 M A N Y

Configuration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name PingServlet2Session2CMROne2Many.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:47 Clients completed 0/100 Page elements 122210 Page element throughput 729.403 /s Transactions 122210 Transaction throughput 729.403 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 122210 Page throughput 729.403 /s HTTP data read 272.991 MB HTTP data written 50.147 MB HTTP avg. page element response time 0.137

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 36

Page 41: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

S C E N A R I O - J D B CConfiguration

Engine name wilbur Initial number of clients 100 Max. number of clients 200 Number of worker threads 20 Script file name Scenario.jxs Percent element delay 0% Startup delay 0 ms Max. connect timeout 30 s Max. read/write timeout 60 s Max. errors before exiting 500 Console print on Validate length checking on Dynamic DNS off Dynamic cookies on XML stats interval 5 s Script repeat 900000 Config file 50clients.conf

Results

Run time 00:02:53 Clients completed 0/100 Page elements 641013 Page element throughput 3694.919 /s Transactions 641013 Transaction throughput 3694.919 /s Network I/O errors 0 Web server errors 0 Num of pages retrieved 641013 Page throughput 3694.919 /s HTTP data read 5427.968 MB HTTP data written 276.956 MB HTTP avg. page element response time 0.027

Appendix - B Runtime Parameters and Results

Geronimo 1.1.1 Performance 37

Page 42: Geronimo 1.1.1 Performance Report - home.apache.orgpeople.apache.org/~hogstrom/Geronimo-1.1.1-PerformanceReport.pdf · This report characterizes performance using a benchmark sample.

The Java Runtime Environment (JRE) was obtained from http://www-128.ibm.com/developerworks/java/jdk/. The JRE used in this testing was J2SE 5.0 SR3 for Intel Linux 32-bit.

Bibliography

Geronimo 1.1.1 Performance 38