7/30/2019 Diagnosing J2EE Performance Problems
1/12
J2EE
DIAGNOSING J2EE PERFORMANCE PROBLEMS
THROUGHOUT THE APPLICATION LIFECYCLE
7/30/2019 Diagnosing J2EE Performance Problems
2/12
ABSTRACT
Many large-scale, complex enterprise applications are now built and
deployed using the J2EE architecture. However, many of these
applications suffer from poor performance and scalability because the
focus of the development process is on functionality, while performance
and scalability are dealt with as an afterthought.
This paper will present techniques for delivering high-performance
applications to production, managing and measuring the performance of
applications, and diagnosing the toughest J2EE problems throughout the
application lifecycle. It will examine the various types of performance
issues at each stage of the lifecycle and which diagnostic tools and
techniques can best resolve them.
Abstract 2
Challenges for J2EE Applications and Performance 3
Performance Evaluation and Diagnostic Needs
Throughout the Application Lifecycle 3
Typical J2EE Application Performance Problems 6
Diagnosing Performance Problems with Mercury Diagnostics for J2EE 7
J2EE Diagnostics for J2EE for Mercury LoadRunner and
Mercury Performance Center 8
Mercury Diagnostics for J2EE for Mercury Business Availability 8
J2EE Deep Diagnostics 9
Mercury Diagnostics Profiler for J2EE10
Summary 11
TABLE OF CONTENTS
7/30/2019 Diagnosing J2EE Performance Problems
3/12
Challenges for J2EE Applications and Performance
Todays enterprises are choosing to build business applications that leverage the power, portability, and
rapid development of Java technology. J2EE offers many advantages to developers, but introduces new
challenges for the development, performance diagnosis, tuning, deployment, and management of
applications. Successful delivery and management of typically complex J2EE-based applications require
evaluating performance throughout the entire application lifecycle.
An application may perform well in the development and QA environment, but fail to scale or exhibit
transient performance problems in production. It is important to understand the impact of the
infrastructure in which the application runs and the behavior of the many application components as
they interact under load. From the diagnostic perspective, it is important to be able to isolate the
problem by tier of the application architecture and by application component, and to have progressive
drill-down visibility into J2EE performance problems, the J2EE environment, and into the actual code,with sufficient detail to determine the root cause of the problems.
There are additional factors that can increase the difficulties of application delivery and management.
The deployment lifecycle for many J2EE applications is compressed, due to increased pressures for
quick time-to-market. Boundaries between development, QA, deployment, and production stages and
IT groups are blurred. Centralized IT organizations may be managing hundreds of applications, without
in-depth J2EE experience.
Many applications have not been sufficiently architected for performance and scalability, with thorough
consideration of design and usage patterns, and adequate attention to planning and testing
performance against well-defined service objectives. J2EE scalability capabilities, while extensive, do
not substitute for such efforts. Later in the lifecycle, applications may be pushed into production to
meet deadlines, with insufficient validation of performance or scalability and inadequate tools to see
into the applications internal behavior. This makes it difficult and expensive to fix problems. Getting to
the root of performance problems in the complex, distributed, dynamic J2EE environment is truly a
challenge.
Performance Evaluation and Diagnostic Needs Throughout the Application Lifecycle
During the application lifecycle, there are many IT stakeholders in application performance. They include
application architects, developers, load testers and engineers in QA, application support staff, and site
operations in production. They share common requirements for end-user application performance, but
also have specific roles and needs unique to their environments.
In the design and specification phases, a topic outside the scope of this diagnostic paper,
performance and scalability issues need to be considered and incorporated, and specific business
objectives need to be set.
J2EE
WWW.MERCURY.COM 3
7/30/2019 Diagnosing J2EE Performance Problems
4/12
J2EE
In the development and unit-testing phase, application profiling can be useful to validate
performance and functionality. Developers should test the performance of not just application
components, but also performance of the end-to-end integrated application (as it would be tested by
QA and/or used in production) against identified performance criteria. This step allows development
teams to find problems earlier in the lifecycle by preparing the application to be load tested.
During QA cycles, load testing typically follows integrated functional and regression testing. A
complete application, including all interfaces with external systems, should be fully load tested prior
to software release. Objectives include scalability and capacity estimation under load that realistically
represent expected live usage, along with visibility into the internal performance behavior of the
application and actionable data on bottlenecks. This should include transaction breakdown of
latencies for each J2EE tier and method, along with additional specific root-cause diagnostic
information.
Most try using profiling tools for root-cause diagnostics and fail. Traditional profiling tools used at the
developer desktop cannot be used in QA during load testing for performance diagnosis, due to the
high overhead they impose. What is needed is a profiling technology that can work in a development
environment to find and fix common problems in applications earlier in the lifecycle. Such a tool
should allow development teams to release a solid application into QA that is ready to be load tested
and where most obvious performance bugs have been found and addressed. J2EE diagnostic tools
used in QA need to be designed for load and should be integrated with load testing tools to boost
productivity, test effectively, and facilitate collaboration between application owners and QA teams to
triage, diagnose, and solve problems under load conditions.
During staging or pre-production, deployment teams should determine the performance andscalability of the application in the specific IT environment and configuration that will serve live users,
verify projected peak load capability, and tune the application and infrastructure to meet its designed
transaction throughput and response-time service levels, keeping the business objectives in mind.
Load testing and performance diagnostic tools that can optimize both application and infrastructure
and drill down to solve problems are essential during this stage.
In live production, operations and application managers need to continuously monitor the health of
the entire system, measure the performance of the application, and, if something starts to go wrong,
quickly alert, triage, and isolate the problem area. Once the problem area is identified, the
appropriate teams should be able to collaborate to pinpoint the root cause of the problem. It is
important to not only proactively monitor end-user business processes, but also to diagnose issues
that happen with real users using the systems.
WWW.MERCURY.COM 4
7/30/2019 Diagnosing J2EE Performance Problems
5/12
J2EE
Lets see an example of how a typical collaborative effort on diagnostics might look using Mercury
solutions for a new application being developed. The above diagram shows this process. The four key
stakeholders are shown as the four circles each representing the business owner, the application
owners, QA, and production operations. The application lifecycle is shown by dotted arrows on the
outside, and the interactions of the various roles in the lifecycle are shown as solid arrows for each
stage. The Mercury lifecycle solution facilitates collaboration between team members by providing a
single technology across the lifecycle and role-based web dashboards that can drastically reduce the
mean time to problem resolution.
Throughout the entire application lifecycle, J2EE-specific visibility and diagnostic capabilities should be
integrated with and complementary to multi-platform, multi-protocol tools for functional testing, load
testing, and application performance management. Common tools and measurements help facilitate
communication, while team members need specific features and capabilities suited to their roles.
WWW.MERCURY.COM 5
Figure 1: Mercury Diagnostics is a single solution that enables developers to find problems earlier in the lifecycle, QA teams to be
empowered with the right data set to share with development to fix problems under load, and production operations teams to manage
the application 24x7, and triage and diagnose issues as they occur.
MARKET
REQUIREMENTS/GOALS
BUSINESS
OWNER
DEVELOP
APPLICATION
OWNER
APPLICATION
READY
MERCURY DIAGNOSTICS FOR J2EE
FUNCTIONAL
APPLICATION
(1 USER)
LOAD-READY
APPLICATION
(5-20 USERS)
BUSINESS-
READY
APPLICATION
(50,000 USERS)
QA
DEPLOY
PRODUCTION
OPERATIONS
PRODUCTION-
READYAPPLICATION
MERCURY DIAGNOSTICS PROFILER
MERCURY BUSINESS AVAILABILITY CENTER DIAGNOSTICS
QA/TESTING
MERCURY LOADRUNNER/MERCURY
PERFORMANCE CENTER DIAGNOSTICS
7/30/2019 Diagnosing J2EE Performance Problems
6/12
J2EE
Typical J2EE Application Performance Problems
There are many problems that can surface during the application lifecycle. For J2EE applications, the
users experience of performance is affected by many external network infrastructure factors that are
independent of application behavior. A broad set of monitors that can cover the complex environment
is needed to triage problems quickly. From the J2EE application diagnostic perspective, it is essential to
be able to capture and correlate specific external parameters, such as method arguments, that can
drive performance problems in a J2EE method or sequence of transactions. Specific latencies and
parameter captures are also frequently required to identify problems at J2EE interfaces to external
systems, such as backend databases, legacy systems, and packaged software.
Within the J2EE environment, some of the most common problems include:
1. Code problems:
a. Slow methods
Consistently slow methods
Intermittently slow methods, related to specific user/data values driving problematic application
behavior
b. Memory problems, including memory thrashing and memory leaks
c. Coding practices, such as using exceptions as a means to transfer control in the application,
processing large result sets, etc.
d. Synchronization problems, including both under-synchronization and over-synchronization for
locks and threads
2. Application server configuration problems:
a. JDBC connection pool size
b. JVM heap size
c. Thread pool sizes
3. Architecture and design problems, with a wide range of issues, such as:
a. Data-marshalling problems resulting from filtering at the wrong tier
b. Serial execution resulting from over synchronization in design of custom code
An effective diagnostic solution must provide capabilities and techniques to triage and identify the root
cause of each of these common problems, regardless of when they emerge, from development through
production. Capturing application performance data requires the ability to tune up the data capture as
you go from identification and triage to root-cause diagnostics. This is best achieved with lifecycle
technology without having to install new tools to collect greater details for root-cause diagnostics.
WWW.MERCURY.COM 6
7/30/2019 Diagnosing J2EE Performance Problems
7/12
J2EE
Diagnosing Performance Problems with Mercury Diagnostics for J2EE
Mercury Diagnostics for J2EE provides the industrys first top-down, end-to-end lifecycle approach to
seamlessly manage, monitor, diagnose, and resolve critical problems in J2EE applications in both pre-
and post-production environments. It can be implemented alone or integrated with Mercury
LoadRunner, Mercury Performance Center, and Mercury Business Availability Center offerings.
Mercury Diagnostics for J2EE provides a common, consistent foundation of shared assets, metrics,
scripts, and monitors, as well as specific solutions for the differing requirements of application delivery
and application management.
Mercury Diagnostics for J2EE uses a combination of JMX, aggregation, and unique Total Trace
technology, which can capture data to diagnose all of the problems previously described. It is the only
solution that can capture, under load, necessary and sufficient information to get to the root cause for
the full range of J2EE performance problems. Unlike other techniques, Total Trace allows capturing of
every event (including arguments) at low overheads. Other transaction tracing techniques may use a
threshold for collecting a fixed number of sampled events regardless of whether they are related to thebusiness process in question or not. This could lead to capturing of sampled events that might be
totally unrelated to the problem at hand.
Another common technique used for tracing a transaction is to tag a server request or HTTP URL
request with a specific header. This header then can be followed through the application to
provide traceability across methods. Most common implementations of such techniques ignore the
business process name that a specific method belongs to and are limited in their ability to trace
across JVM boundaries.
WWW.MERCURY.COM 7
Figure 2: Mercury Diagnostics for J2EE combines J2EE-specific diagnostics with application delivery and application management
solutions.
BEA
IBM
SAP
ORACLE...
MERCURY IT GOVERNANCE CENTER
MONITOR TRIAGE DIAGNOSE
APPLICATION MANAGEMENT
MERCURY BUSINESS
AVAILABILITY CENTER
END USER, APPLICATION,
SYSTEM LEVEL AND
SERVICE LEVEL MANAGEMENT
APPLICATION DELIVERYAPPLICATION DEVELOPMENT
MERCURY DIAGNOSTICS PROFILER
MERCURY QUALITY
CENTER
MERCURY
PERFORMANCE CENTER
TEST MANAGEMENT
FUNCTIONAL TESTINGLOAD TESTING
APPLICATION LIFECYCLE
APPLICATION PLATFORMS - BEA, IBM, MICROSOFT, ORACLE, SAP, SIEBEL
MERCURY DIAGNOSTICS
7/30/2019 Diagnosing J2EE Performance Problems
8/12
J2EE
For example, if a business
transaction were to interact with
several JVMs hosted on remote
machines, it would be critical to
trace the business transaction
across the different JVM
boundaries and provide
diagnostics data without losing
the business context of the
transaction. The screen shot to
the right shows how the remote
calls crossing JVM boundaries are
distinguished with a Remote
Calls layer and critical data such
as SQL queries and application
exception information.
Mercury Diagnostics for J2EE for Mercury LoadRunner and Mercury Performance Center
Mercury Diagnostics for J2EE is fully integrated with Mercury LoadRunner and Mercury Performance
Center to expose and diagnose the most common J2EE performance problems under load testing.
Mercury Diagnostics for J2EE first correlates end-user problems to the web transaction tier for fast
isolation and resolution, then pinpoints the infrastructure, application, or database problems. It analyzes
the TCP/IP stack to break down a transactions end-to-end response time from an end-user perspective
to determine whether poor transaction time is caused by external network issues or by a specific server
tier. It then guides users through an intuitive top-down analysis process across multiple application
layers, tracing business performance problems from the end user all the way to the problematic
component.
In addition, Mercury Diagnostics for J2EE shows performance metrics for the typical J2EE tiers such as
JSP/Servlets, EJBs, JMS, JDBC, and JNDI across a clustered environment in a single dashboard,
allowing you to isolate and identify slow components, detect memory leaks, and more.
Mercury Diagnostics for J2EE for Mercury Business Availability
Mercury Diagnostics for J2EE is fully integrated with Mercury Business Availability Center to maximize theperformance and availability of J2EE applications by providing a unified solution for your operations and
application support teams to quickly monitor, diagnose, and optimize production applications. It provides
teams with visibility across user, business process, application server, application platform suites such as
portal servers, and system tiers down to the component and method level. Ultimately, high-performing,
mission-critical applications protect service-level agreements (SLAs) and the bottom line.
WWW.MERCURY.COM 8
Figure 3: Tracing remote method invocation across JVMs.
7/30/2019 Diagnosing J2EE Performance Problems
9/12
J2EE
The solution provides the
industrys broadest monitoring
capabilities that allow 24x7
monitoring of the entire
application infrastructure,
including the J2EE environment,
from a single web-based console.
Agentless monitoring for
enterprise infrastructure
components can optimize the
total cost of systems
management. In addition, it can
facilitate capacity planning by
ensuring that purchases of hardware
and software are optimized.
Mercury Business Availability Center provides correlation of end-user performance with root cause in
the infrastructure and application layers. Its J2EE-specific diagnostics identify problems involving EJB
components, methods, and JDBC calls with SQL statements. Mercury Diagnostics for J2EE integrated
with Mercury Business Availability Center provides the same transaction breakdown capabilities as
Mercury LoadRunner and Mercury Performance Center, using aggregated data capture, while optimized
for low-overhead, real-time monitoring, triage, and problem isolation.
Unlike other implementations that might need inclusion of client-side scripting code in web application
pages to measure the response time from the browser, Mercurys diagnostics solution uses industry-
leading scripting technology to capture this information from an end-user perspective with synthetic
scripts. A similar noninvasive technique is also employed to get diagnostics information for real users
(actual people using the production systems as opposed to synthetic scripts) for production usage.
J2EE Deep Diagnostics
Complex problems, such as finding the cause of thread deadlocks or objects that arent de-allocated
and removed by Java Garbage Collection, are responsible for the majority of the time and cost spent in
diagnosing the root cause of performance problems.
J2EE Deep Diagnostics uses Mercurys Total Trace data capture, which captures every event in aselected area of the application with low overhead, by providing carefully tuned, predefined sets of
byte-code instrumentation and a GUI for custom selection and automatic application of narrowly
targeted instrumentation. In production, a Mercury Business Availability Center alert can trigger a deep
WWW.MERCURY.COM 9
Figure 4: Transaction breakdown for real user for Portal Server environment showing the specific portlet
performance.
7/30/2019 Diagnosing J2EE Performance Problems
10/12
J2EE
diagnostic data capture for offline analysis. This stops the common frustration of persistently recurring
production performance problems that cannot be reproduced in a test bed. The same J2EE Deep
Diagnostics works with Mercury LoadRunner when performance problems under load need better
characterization for effective bug definition and rapid fixes.
J2EE Deep Diagnostics provides multi-layer correlation of HTTP, servlets, JSPs, EJBs/objects, methods,
and SQL calls. It correlates these internal component measurements with JMX and OS metrics to show
problem details. It delivers graphical views of transaction traces, call chains, memory leaks and
thrashing problems, synchronization details of locks held and threads blocked, latency charts, and
source code views of problem methods. J2EE Deep Diagnostics captures details of specific instances
of method invocations, including parameters passed and individual latencies, to pinpoint the source of
intermittent problems caused by specific user data values or usage patterns.
For further examples and details on how to solve synchronization problems, intermittent slow methods,
and other J2EE performance problems using Mercury Diagnostics for J2EE, contact your Mercury
representative.
Mercury Diagnostics Profiler for J2EE
Mercury Diagnostics Profiler for J2EE extends the application lifecycle solution into the development
environment. Developers can use the same lifecycle probe technology used for load testing and
production to find and fix common application problems earlier in the lifecycle. This also enables
applications to be readied for load testing. Often, when QA groups start to load test the application,
they find that the application is not ready for load testing. Specifically, the application breaks or doesnt
scale beyond two to five users. Lack of application readiness for load testing and production wastes
time and effort.
Developers are often very interested in application performance data that is actionable. Mercury
Diagnostics Profiler not only provides developers the ability to solve the most common problems
discussed earlier, but also to find specific problems such as application exceptions being thrown,
slowest SQL queries, memory leaks in primitive object types, JAVA objects, and JAVA collections. The
following screen shots show how a developer can understand if and specifically what exceptions are
being thrown in the application at run time. These could migrate to production as painful timebombs if
left undetected earlier.
For diagnosing problems in JAVA collections, its not only important to understand which collections are
growing fastest and which are the largest, but also to understand the contained class and where the
collection is being allocated. For example, is the contained class in a collection a critical resource such
WWW.MERCURY.COM 10
7/30/2019 Diagnosing J2EE Performance Problems
11/12
as DB connection or socket or a less
important Integer object? Figure 5
shows how Mercury Diagnostics
Profiler can provide this visibility to
help find and solve more problems
earlier in the lifecycle.
Summary
With Mercury Diagnostics for J2EE,
enterprises can improve the readiness
of their J2EE applications before they
are placed into production, and
dramatically accelerate problemidentification, isolation, and resolution.
Mercury Diagnostics for J2EE allows
companies to reduce their costs of
operations by providing a business
process view into the application and
by isolating and resolving problems
before business users and the bottom
line are impacted. The solution can also
help companies optimize their existing
infrastructure and avoid unnecessary
hardware or software expenditures.
Choosing the appropriate application
profilers, application delivery tools, and
application management tools is critical
to guaranteeing application quality and
ensuring that SLAs can be met.
Mercury Diagnostics for J2EE is the
industrys first, most complete
application delivery and application
management suite for the J2EE
ecosystem that improves the quality,
performance, and scalability of J2EE
applications across the entire
application lifecycle.
J2EE
WWW.MERCURY.COM 11
Figure 5: Exception stack traces.
Figure 6: JAVA collection memory diagnostics.
7/30/2019 Diagnosing J2EE Performance Problems
12/12
2006 Mercury Interactive Corporation. Patents pending. All rights reserved. Mercury Interactive, Mercury the Mercury logo, Mercury Diagnostics for J2EE, Mercury Diagnostics Profiler, Mercury LoadRunner, Mercury Performance Center,
and Mercury Business Availability Centerare trademarks or registered trademarks of Mercury Interactive Corporation in the United States and/or other foreign countries. All other company, brand, and product names are marks of their
respective holders. WP-1355-0406
Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business outcome of IT.
WWW.MERCURY.COM
Top Related