Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)

Performance Tuning and Troubleshooting for Oracle OC4J

Jurijs Velikanovs7+ years Oracle DBA, OCP 7/8/8i/9i, OCA 9iAS R2

Topics:

IntroductionUsage considerationsOC4J InternalsOC4J TroubleshootingOC4J Performance tuningBC4J

Introduction

Who am I? And what are my responsibilities?

Who am I?I am a DBA team leaderI’m neither a Java fan nor a Java Developer nor a Java GuruMy team has spent a lot of time researching and struggling to stabilize the production environments based on OC4J

ResponsibilitiesTo support production and development environmentsTo ensure production availabilityTo troubleshoot production systemTo provide all the necessary information to the developers or support organizations to solve a problem

How were we involved?

Local Oracle Marketing team has pushed customers to demand the use of OC4J in their tenders’ conditionsIn 2002 the company management decided to switch the whole Development from Oracle Forms to OC4J/BC4J/JDeveloper technologyWe had

100 forms developers without any Java experienceA few Java fansOracle 9iAS R2 (9.0.2), later 9.0.3.1In 2003 the first phase of the first application was deployed

Example of the Applications we are working with

Healthcare financial systemEnterprise application2+ years in the production and still under developmentOracle 9iAS 9.0.3.1 OC4J, Oracle 9.2.0.5 DBJSP = 1448 files, CLASS = 3283 files965 tables, 593 views, 251 types, 220 packages, DB Size 50 GB, REDO 2-8 GB Daily800 heavy users per day

10x5 hour availability

Why is it so important?

OC4J is the present strategic direction for Oracle JDeveloper, Forms, Reports, Discoverer, OEM, etc.

Difficult to troubleshootComplex technology Many componentsJava on Java

Why is it so important?

OC4J is a strategic direction for Oracle JDeveloper, Forms, Reports, Discoverer, OEM

Difficult to troubleshootComplex technology Many componentsJava on Java

There is not much information about this productPoor vendor supportMany Oracle customers use it and have troubles

I will not tell you how to develop in Java …

I’m not going to tell you aboutWeb and EJB application technology VMC etc.

The technical layer is of great interest

I’m going to speak about OC4J internals …

ApplicationServerThread



Java native



JMSServerPORT: 3201

RMI ServerRMI ServerPORT: 3101

TaskManagerTaskManager

JMSServerJMSServerJMSServerJMSServerThreadThread

ONS

OC4JMonitor Thread

Thread

Thread

AJPConnection

ListenerPO

RT:

300

1

VM Thread VM Periodic Task Thread

Signal Dispatcher

Suspend Checker Thread

Reference Handler

Finalizer

Java native


Signal Dispatcher


Reference Handler

Finalizer

Memory Structures

Http Session List JDBC Connection pool

Application objects

...


...


AJP 1.3 ONet

new req

3101

3201

Usage considerations or

how to avoid troubles

Oracle AS Architecture Considerations

You may make the configuration complicated, with many components highly dependent on each other

Oracle AS Architecture Considerations

BUT:Keep it as simple as possibleAt first you should be sure that the system works properly in a simple configurationMake components as independent as possibleDon’t use the infrastructure if there is no strong need for it (i.e. Single-Signon, Discoverer, Clustering)

Oracle AS Typical Architecture

PO

RT:

80,

778

0

OHS (c)

Child ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server



OC4J

DB

WAN

ApacheChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild ServerChild Server

JVM

AJPConnection

Listener




TaskManagerOC4JMonitor Thread

JMSServerJMSServerThreadThread

Java native


Signal Dispatcher


Reference Handler

Finalizer

TaskManager

JVM(java)

AJPConnection

Listener




JMSServerPORT: 3201

POR

T: 3

001

TaskManagerOC4JMonitor Thread (dms)

JMSServerJMSServerThreadThread

Java native


Signal Dispatcher


Reference Handler

Finalizer

TaskManager


HTTP AJP 1.3

PM, ONS (c)strat, stop, ping,

notification

OPMNDCM (java)

configuration management

DCM-daemonDMS.JAR (java)

statistics reflecting

DMS-metrics

opmn.xml

httpd.conf server.xml

DCM repository

Installation Considerations

Use the latest version of Oracle AS depending on your circumstancesUse the latest versions of Oracle Forms, Reports, Discoverer, Portal. Install them in the OH separated from your custom development OC4J OHInstall the latest patches for OS, JDK, OC4J, JDBC componentsInstall the version of BC4J which was used by the Development. Don’t use the default version of BC4J/ADF/JHeadstart (See. JDeveloper install.html)

Operational Considerations

Keep running only the components you needDon’t startup in the production environment:

Oracle Enterprise Manager 10g Application Server ControlOC4J_Demos component“home” component

Don’t deploy your applications to the “home”componentCreate your own component and use it to deploy and run your applicationsUse the opmnctl, dcmctl, dmstool utilities to maintain your AS environment

Deployment Considerations

Get a commitment from the Development to use industry standards for the Application development

Use EAR files to deploy any application changesUse Ant tool (http://ant.apache.org) to make EAR/WAR filesIn emergency cases use WAR files to deploy small changes to application

Use undeployApplication/deployApplication combination rather than redeployApplication dcmctl commandUse dcmctl saveInstance before deploying to backup current configurationDeploy BC4J.EAR to your OC4J component to keep “home” down

Versions Considerations

Application

OC4Jc.c.c.x9.0.3.1

JDK c.c.x_x, 1.3.1_15

OS 2.4.9-e.38enterprise

JDBCc.c.c.x9.2.0.5

BC4J 9.0.3.10.7

APP Frame Work

RecommendationsHave a dedicated test environment and verify all the changes there before implementing in productionIf you have the automatic stresstestingscenarios, run them after each change in the test environmentMonitor the impact of changes and if the situation becomes worse rollback them


Application

OC4Jc.c.c.x9.0.3.1

JDK c.c.x_x, 1.3.1_15


JDBCc.c.c.x9.2.0.5

BC4J 9.0.3.10.7

APP Frame Work

OSIt is important to install the latest patches from an OS vendor (Kernel, Network, etc.)Usually the patches can be rollbacked easilyThe change of OS main release requires more testing and efforts than patching

JDKInstall the latest versions of JDK within c.c. boundariesUse the JDK from Sun ww.JavaSoft.comTo install it replace the OH/jdkdirectory. These changes can be rollbacked easilyThe change of c.c. version requires the Development involvement and more testing


Application

OC4Jc.c.c.x9.0.3.1

JDK c.c.x_x, 1.3.1_15


JDBCc.c.c.x9.2.0.5

BC4J 9.0.3.10.7

APP Frame Work

JDBCInstall the latest version of JDBC within c.c.c. versionTo install it replace OH/jdbcdirectory. These changes can be rollbacked easily It requires the minimal functionality testing

OC4JCan be easily upgraded within a c.c.c. version boundariesIt requires the minimal functionality testingFollow the patch installation instructions (9iAS 9.0.3.1)There are 50+ one-off patches (*.class)The change of c.c.c. version requires the Development involvement and more testing


Application

OC4Jc.c.c.x9.0.3.1

JDK c.c.x_x, 1.3.1_15


JDBCc.c.c.x9.2.0.5

BC4J 9.0.3.10.7

APP Frame Work

BC4JA BC4J version sets the requirements for OC4J versionIt can’t be changed without Development involvement even within x.x.x.x. version boundariesOften there are needs for application changesIt requires a lot of testingNew version can have negative impact to your applicationThe change of c.c.c. version requires more application code changes, testing, and other efforts


How to Determine the VersionsOS: uname -r

$ uname -r2.4.9-e.38enterprise

JRE: java –version$ $ORACLE_HOME/jdk/bin/java -versionjava version "1.3.1_13"Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)

OC4J: java -jar oc4j.jar –version$ $ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/j2ee/home/oc4j.jar -versionOracle9iAS (9.0.3.1.0) Containers for J2EE (build 031214.BP1.2075)

JDBC: a program with a call to meta.getDriverVersion(location $OH/jdbc/lib/classes12dms.jar, Notes:73629.1/244074.1)

$ $ORACLE_HOME/jdk/bin/java JDBCVersionOracle JDBC driver 9.2.0.5.0

BC4J: a program with a call to meta.getDriverVersion(location $BC4J/lib/bc4jct.jar)

$ $ORACLE_HOME/jdk/bin/java BC4JVersion 9.0.3.10.7

Other Considerations

Have at least one senior level developer, who understands used technology well, is capable to make technical decisions and to lead the framework development, maintenance and troubleshootingSynchronize OC4J, BC4J versions with DevelopmentUse an Automatic testing tool for the application stresstestingA commitment from Oracle is desirable. If you can manage a connection to the Oracle JDeveloper development team to support your project, it will be an invaluable help during troubleshooting and decision makingHave a skilled DBA for environments maintenance, monitoring and troubleshooting. The DBA provides all possible information to the development for application troubleshootingHave a monitoring framework

Before going to life

!!! Stresstest your application !!!(or ask for results)

OC4J Internals

OC4J Internals




Java native



JMSServerPORT: 3201




ONS

OC4JMonitor Thread

Thread

Thread

AJPConnection

ListenerPO

RT:

300

1


Signal Dispatcher


Reference Handler

Finalizer

Java native


Signal Dispatcher


Reference Handler

Finalizer

Memory Structures

Http Session List JDBC Connection pool

Application objects

...


...


AJP 1.3 ONet

new req

3101

3201

Processes (threads) within OC4J

Java native threadsVM Thread, VM Periodic Task Thread, Signal Dispatcher, Suspend Checker Thread, Finalizer, Reference HandlerExist in any Java based application

OC4J Listener - threadsAJP/HttpConnectionListener, RMIServer (2), JMSServerAccepting clients requests, assigning them to the worker threadsThe configuration of listened ports is in opmn.xml file

Set of processes for communicating with ONSOC4JMonitorThread, Thread(2)These processes make connection to the OPMN immediately after the startup of OC4JThey send event messages to the OPMN to be broadcasted to subscribers

Processes (threads) within OC4J

TaskManagerResponsible for a periodic tasks executing (housekeeping)

server.http.SessionTimeoutTask (drops expired session)server.ApplicationServerTask (clear expired threads)sql.DriverManagerXADataSource…

You can set an executing frequency in server.xml filetaskmanager-granularity=1000 (default 1 sec)

Use the dmstool to monitor the statistics of the processdmstool -table oc4j_task

ApplicationServerThreads (Global Thread Pool)Execute users requests, run an application code, communicate with the Apache processes and the databaseInitial count of the processes can be set in server.xml fileglobal-thread-pool min=“200" max="200" queue="20“

OC4J Troubleshooting

Classification of Problem AreasAp

ache

OC4J

DB




TaskManager

JMSServerJMSServer

JVM(java) AJP

ConnectionListener







POR

T: 3

001

New Request

assign

1.2.3.

JDK

OS

4.

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)2. Interface to the Apache processes3. Interface to the database4. 500 Internal Error

Possible Approaches & Information Sources

Java Full thread dumpOther

Log filesOS Monitoring commandsOS Tracing commandsApache side informationJava code decompilationDocumentation, External searches and Metalink

Troubleshooting information gathering (example)

Java “Full thread dump”

How to get a Full thread dump?The Full thread dump interpretation

Different types of the threads within OC4J containerDifferent states of the threadsSome threads examples

Useful java runtime parameterJava DEADLOCK problem

How to get a “Full thread dump”?

Get pid of the OC4J processUse : ps, pstree or opmnadmin

$ opmnadmin debug comp=pmHTTP/1.1 200 OK Content-Length: 677Content-Type: text/htmlResponse:

======== PM ========

PM requests processed: 5

PROCESS TABLE

UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS---------- ----- -------- ------- ------- ------ ----- ----- ----- -----

456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201194092 13338 00000040 Apache Alive 1 7780 0 0 0128556 13281 00000040 Generic Alive 1 0 0 0 0


Find out the output file name (OH/opmn/logs/*)$ ps -fw --width 4000 -p 16114UID PID PPID C STIME TTY TIME CMDoracle 15874 15902 0 23:40 ? 00:00:00

/u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java -Xms512M -Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 -Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-8000 -Doracle.ons.indexid=PANDA.default_island.1 -DOPMN=true -jar oc4j.jar -config /u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml -properties -instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.-8000#J2EE01.ap1.voava.lv#521772#PANDA#PANDA#default_island -ajp 3001 -rmi 3102 -jms 3202

$ ps -wo args --width 4000 -p 15874 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}'

PANDA.default_island.1

Send SIGQUIT (Quit from keyboard) signal$ kill -3 15874


Check the output fileOH/opmn/logs/PANDA.default_island.1Full thread dump:

"Thread-13" daemon prio=1 tid=0x82c8690 nid=0x3f15 waiting on monitor [0xa51ff000..0xa51ff89c]

at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:427)at java.util.TimerThread.run(Timer.java:380)

"Thread-12" prio=1 tid=0x67b0cce0 nid=0x3e02 waiting on monitor [0..0xbffeb62c]

"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c]

at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)

"Thread-10" daemon prio=1 tid=0x676930f0 nid=0x3ef1 waiting on monitor [0xa55ff000..0xa55ff89c]

at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:415)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:253)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:213)at oracle.ons.SenderThread.run(SenderThread.java:81)

The “Full thread dump” Interpretation

====================================================="OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor

[0xa53ff000..0xa53ff89c]at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)

=====================================================

[OC4JMonitorThread] – name (type) of the thread OC4JMonitorThread.javaThread.currentThread().setName("OC4JMonitorThread");

[nid=0x3ef2] – Thread ID (in decimal = 16114)

[waiting on monitor] – State of the thread at the snap time

[oracle.ons.NotificationQueue.internalDequeue] – java stack

TYPES of the Threads

OC4J SpecificAJPConnectionListener [0.0.0.0/0.0.0.0:3001] 1RMIServer [0.0.0.0/0.0.0.0:3102] count:1 1RMIServer [0.0.0.0/0.0.0.0:3102] count:2 1JMSServer 1OC4JMonitorThread 1TaskManager 1Thread 5ApplicationServerThread 195JAVA GENERICVM Thread 1VM Periodic Task Thread 1Reference Handler 1Signal Dispatcher 1Suspend Checker Thread 1Finalizer 1

1

STATES of the Threads

Waiting on monitorIn most of cases it is an idle state of the thread. The thread is waiting for a task to execute or voluntary sleepsAll the available ApplicationServerThreads are in this state

RunnableThe thread is executing application code or reading from the socketAll listeners must be in this state

Waiting for monitor entryWaiting to lock an object (other thread is holding the lock). This happens if two or more threads are trying to execute synchronized java codeNone of threads must be in this state for a long timeIf the system thread stays in this state for a long time you are in troubleIf many threads are in this state you have the situation of serialization

Threads Examples

1. OC4JMonitorThread is waiting for a work. Usual state for most of the threads

"OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c]at java.lang.Object.wait(Native Method)at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.java:251)at oracle.ons.NotificationQueue.dequeue(NotificationQueue.java:224)at oracle.ons.Subscriber.receive(Subscriber.java:136)at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.java:419)at java.lang.Thread.run(Thread.java:479)

2. "ApplicationServerThread-5" – idle worker thread, is waiting for a task"ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c]

at java.lang.Object.wait(Native Method)at EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.poll(BoundedBuffer.java:170)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:768)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:802)at java.lang.Thread.run(Thread.java:479)

3. "Thread-9" is reading from the socket (connected to the ONS process, port 7100). This thread most of time is waiting for a data to be read from the socket

"Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c]at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.ons.InputBuffer.readMoreData(InputBuffer.java:267)at oracle.ons.InputBuffer.getNextString(InputBuffer.java:222)at oracle.ons.ReceiverThread.run(ReceiverThread.java:225)

Threads Examples

4. "AJPConnectionListener” is waiting for new connections from the Apache processes

"AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable[0xb19ff000..0xb19ff89c]at java.net.PlainSocketImpl.socketAccept(Native Method)at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)at java.net.ServerSocket.implAccept(ServerSocket.java:238)at java.net.ServerSocket.accept(ServerSocket.java:217)at com.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:58)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)

5. "ApplicationServerThread-5" is reading the request from the Apache process

"ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c]at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at com.evermind.io.SingleReadBufferInputStream.readChunk(SingleReadBufferInputStream.java:116)at com.evermind.io.SingleReadBufferInputStream.read(SingleReadBufferInputStream.java:35)at com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:345)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:131)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)

Apache

Threads Examples6. "ApplicationServerThread-71" is executing database query, waiting for the results to be retrieved"ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c]

at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.net.ns.Packet.receive(Unknown Source)at oracle.net.ns.DataPacket.receive(Unknown Source)at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.net.ns.NetInputStream.read(Unknown Source)at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986)at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144)at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2484)at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2850)at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609)at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:537)at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:1522)at oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:1530)at oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:1482)at oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:4596)at _common._comp._CMN__DataScrollerComponent._jspService(_CMN__DataScrollerComponent.java:115)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)at com.evermind.server.http.ServletRequestDispatcher.include(ServletRequestDispatcher.java:119)at com.evermind.server.http.GetParametersRequestDispatcher.include(GetParametersRequestDispatcher.java:95)at com.evermind.server.http.EvermindPageContext.include(EvermindPageContext.java:280)at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.java:70)at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.java:249)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306)at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166)at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)at java.lang.Thread.run(Thread.java:479)

DB

Useful Java Runtime Parameter

-XX:+JavaMonitorsInStackTrace (opmn.xml)<java-option value="-server -Xms512M -Xmx512M -verbose:gc -XX:+JavaMonitorsInStackTrace"/>

The parameter wasn’t set"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c]

at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.java:150)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

...

The parameter was set"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c]

at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.java:376)- waiting to lock <460e0338> (a oracle.jbo.JboSyncLock)at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.java:1659)at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.java:4626)at lv.vovaa.panda.common.datatags.DataSourceCmn.doStartTag(DataSourceCmn.java:189)- locked <45f722d0> (a lv.vovaa.panda.common.datatags.DataSourceCmn)at _nsk._nsk001._AP3220F__Tickets__browse._jspService(_AP3220F__Tickets__browse.java:105)at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

...

OC4J

Useful Java Runtime Parameter

It is possible to determine the blocking thread easily"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c]

at java.net.SocketInputStream.socketRead(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:85)at oracle.net.ns.Packet.receive(Unknown Source)

...at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:549)at oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:2600)- locked <460e0338> (a oracle.jbo.JboSyncLock)at lv.vovaa.panda.common.pbo.PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.java:324)at lv.vovaa.panda.common.pbo.PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.java:309)at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:521)

...

Then we can locate the source code for troubleshooting purposesViewObjectImpl.java

protected void executeQueryForCollection(Object obj, Object aobj[], int i){

synchronized(getSyncLock()){

int j = 0;if(InstrumentedEvent.isActive)

j = InstrumentedEvent.startEvent(EventGroup.EXECUTE_QUERY, "ViewObjectexecuteQueryForCollection " + getName());

((QueryCollection)obj).executeQuery(aobj, i);if(InstrumentedEvent.isActive)

InstrumentedEvent.endEvent(j);}

}

Java DEADLOCK

Information from the OC4J process “Full thread dump”Full thread dump:"ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor

entry [0xa19ff000..0xa19ff89c]......FOUND A JAVA LEVEL DEADLOCK:----------------------------"ApplicationServerThread-58":waiting to lock monitor 0x80b9804 (object 0x4595b408, a java.lang.Object),which is locked by "ApplicationServerThread-51"

"ApplicationServerThread-51":waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession),

which is locked by "ApplicationServerThread-58"JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:------------------------------------------------Java Stack for "ApplicationServerThread-58":==========

at oracle.jbo.http.HttpContainer.setValue(HttpContainer.java:352)- waiting to lock <4595b408> (a java.lang.Object)

JDK/JRE (before 1.5.X) is unable to solve DEADLOCK situationsOPMN doesn’t detect such kind of situationsYou need to take action from outside

Gather troubleshooting information and restart OC4J process

Classification of Problem AreasAp

ache

OC4J

DB




TaskManager

JMSServerJMSServer

JVM(java) AJP

ConnectionListener







POR

T: 3

001

New Request

assign

1.2.3.

JDK

OS

4.

1. OC4J container inside (infinitive cycles, waits on other threads, deadlocks)2. Interface to the Apache processes3. Interface to the database4. 500 Internal Error

Java “Full thread dump” (conclusion)

It is an extremely useful source of information for understanding OC4J internals and troubleshooting

Take a few snaps to see dynamics

You need to analyze whole OC4J process “Full thread dump” to make conclusions

Java on Java. The ideal person for such analysis is the person who has a good knowledge of java, used technologies, application framework as well as application running on the container

DBA/Administrator should ensure all necessary information to be available to application troubleshooting person

Log Files

OC4J output fileLocated at OH/opmn/log/<comp_name>.default_island.<proc_num> (example PANDA.default_island.1)Store standard output of OC4J process (JAVA DUMP, task manager info, debug info -Dajp.io.debug=true)It can be useful to add time stamp information

* * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1

OC4J default-web-access.logLocated at OH/ j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log

It is an analogue of access_log for Apache

OS Monitoring Commands

System wide: vmstat, iostat, uptime, pstree

Top resources consumers:top, ps -eo pcpu,pid,args | sort -k1n | tail -10

Particular process information$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ;

done25.325.325.3$ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ;

done1095372 2582441095372 2582441095372 258244$ ps -wo args --width 4000 -p 6018

OS Tracing Commands

strace - trace system calls and signals for particular process$ strace –s1000 -f -p 7495read(19,$ strace –s1000 –f -p 27021--- SIGSTOP (Stopped (signal)) ---read(104, "\0224\0\2\0\0", 4096) = 6send(104, "A", 1, 0) = 1send(104, "B", 1, 0) = 1send(104, "\0", 1, 0) = 1send(104, "\3", 1, 0) = 1send(104, "\6", 1, 0) = 1send(104, "\37", 1, 0) = 1send(104, "\372", 1, 0) = 1read(104, "\0224\0\2\0\0", 4096) = 6send(104, "A", 1, 0) = 1send(104, "B", 1, 0) = 1send(104, "\0", 1, 0) = 1send(104, "\3", 1, 0) = 1send(104, "\6", 1, 0) = 1send(104, "\37", 1, 0) = 1send(104, "\372", 1, 0) = 1read(104, "\0224\0\2\0\0", 4096) = 6...

pstack - print a stack trace of running processes

Apache Side Information

Apache Server Status pagehttp://ap1.voava.lv:7780/server-status?refresh=15

log filesApache log files $OH/Apache/Apache/logsSometimes information can mislead

access_log192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET

/PANDA/NSK/NSK003/NS0291F_Browse.jsp HTTP/1.1" 500 0erorr_log[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling

Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data.

[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13.

[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndlingajp13 response message, got an invalid value: -2.

[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDA_15 and it is not recoverable.

[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDA's service() to service the request.

http://ap1.voava.lv:7780/server-status?refresh=15

Java Code Decompilation

Extremely useful for troubleshooting purposesUnzip archive (jar/zip)Decompile *.class files (http://www.kpdus.com/jad.html)

OC4JOH/j2ee/home/oc4j.jar

JDBCOH/jdbc/lib/classes12.zipOH/jdbc/lib/classes12dms.jar

DCMOH/dcm/lib/dcm.jarOH/dcm/lib/oc4j_deploy_tools.jarOH/dcm/lib/oc4j_remote_deploy.jar

DMSOH/lib/dms.jar

BC4JOH/BC4J/redist/bc4j.earOH/BC4J/jlib/*OH/BC4J/lib/*

Java Code Decompilation Usage Example

Decompile all oc4j.jar classesGrep through *.java files for “System.getProperty”

9.0.4 oc4j.jar…opmnPingIntervalopmn.query.debugoracle.arraylist.deepCopyoracle.aurora.jem.scheduling.threadsoracle.dms.console.DMSConsoleoracle.dms.gateoracle.dms.sensorsoracle.dms.transtrace.ecidenabledoracle.homeoracle.ias.jcache

Debugging Switches

Debug switches are documented inContainers for J2EE User's Guide 10g Release 2 (10.1.2)Part No. B14011-01B Additional InformationB.8 OC4J Command-Line Options and System Properties

Debug information example (-Dajp.io.debug=true)PANDA.default_island.1

1107387907360:ThrP[50] ThrQ[0]1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0] AJP: Input request length is: 1AJP: AJPOutputStream::write() of 1AJP: AJPOutputStream::writeChunk(): 11107387908370:ThrP[50] ThrQ[0]1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]

Some of available categories are : ajp, http, jms, rmi, jdbc

Documentation and Internet

The information in 9.0.X documentation is very limitedOracle AS Documentation Library 10g Release 2 (10.1.2)Highly Recommended books

Containers for J2EE User's Guide (144)Containers for J2EE Standalone User's Guide (134)Server Administrator's Guide (542)OPMN Administrator's Guide (158)DCM Administrator's Guide (116)Server Performance Guide (234)

10g Best Practices 10g (9.0.4) Part No. B12223-01BC4J JDeveloper Online Help system.www.google.comhttp://otn.oracle.comhttp://metalink.oracle.com

http://metalink.oracle.com/

Metalink Patches

Look through available patches.

………………………………………………………………..

Metalink Patches

Strongly recommended in case of 9.0.3 OC4J

Metalink PatchesMonitor for new patches, which can be relative to your environment

Troubleshooting information gathering (example)

Performance tuning

Our Findings (Java Workers)

Preset java workers threads for OC4J processOH/j2ee/<comp_name>/config/server.xml<global-thread-pool

min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1"

debug="true"/>

Set min to high value. It allows OC4J not to create the additional threads on the fly

You don’t have to take care on max, keepAlive and queue parameters thenOC4J doesn’t need to manage queue

debug="true" – Allows to see statistics in the OC4J process standard output file

It is useful as troubleshooting information as well. Record appears in the output file at the end of each Task manager cycle.

<PANDA.default_island.1>1107390315893:ThrP[100] ThrQ[0]1107390315893:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1107390316903:ThrP[100] ThrQ[0]1107390316903:CxP[100] CxQ[0] - WebC[0] RmiC[0] ...

Our Findings (OC4J Processes)

Use many OC4J processes without enabled state replicationOH/opmn/conf/opmn.xml<oc4j maxRetry="3" instanceName="PANDA" gid="PANDA" numProcs="1">

<config-file path="/u01/app/oracle/product/J2EE01/j2ee/PANDA/config/server.xml"/><java-option value="-server -Xms512M -Xmx512M"/><oc4j-option value="-properties "/><port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/><island id="default_island" numProcs="3"/>

</oc4j>

Initially it was recommended by Oracle support while TAR resolution OC4J container is more stable on a lower users loadmod_oc4j distributes requests through 3 OC4J identical containers1/3 of users load is handled by each OC4J processDo not use state replication

It can add significant load on CPUIt complicates configurationIt can lead to opposite effect

Be carefulYou have to test this configuration for your application. It can lead to unpredictable DB connection behaviorMemory/threads configuration is for each process of the OC4J instance

Our Findings (Memory)

OH/opmn/conf/opmn.xml<oc4j …

<java-option value="-server -Xms512M -Xmx512M"/></oc4j>

Set max=minJVM doesn’t need to manage memory allocation dynamicallyYou have reserved memory on the server

Java Garbage collectorThere is a lot of information around

http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/http://www.cons.org/cmucl/doc/gc-tuning.html

-verbose:gcJava Profilers

Add significant overhead to be used in production environmentsExtremely useful in test/dev environments for troubleshooting

http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/

http://www.cons.org/cmucl/doc/gc-tuning.html

Our Findings (Memory)

Monitor memory usage in productiongrep log files for “out of memory” erroruse OS utilitiesuse dmstool

$ dmstool -table JVM | grep 3001 | grep Memory/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytesap1.voava.lv*J2EE01-/u01/app/oracle/product$

Our Findings (Stresstesting)

The best way to tune performance is a stresstestingAccording to our experience

After Development had performed the performance testing, a lot of problems were solvedAn automatic testing tool has been used (Rational Robot)Different modules were testedDifferent load profiles were testedTesting was repeated several times until acceptable results were achieved

Response Time Monitoring by DMS

OHS [c]

OC4J

DB

ApacheJVM [java]


DMS.JAR (java)statistics reflecting

DMS-client

mod_oc4j handle

mod_plsql handle

mod_perlmod_php

...

Request B

Request E

Handle B

Handle EHandle BHandle E

ReadResolve

Process B

Process E

JSP ProcessingServlet Processing

DMS Sensors DMS Sensors

Recive

Send

Response Time Breakdown Example

$ dmstool -table ohs_server | grep time.../Apache/connection.time 48123608767 usecs.../Apache/request.time 15916131942 usecs.../Apache/handle.time 15913988395 usecsrhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf$$ dmstool -table ohs_module | grep time | sort -k2n | tail -2.../Apache/Modules/mod_dms.c/handle.time 8081655 usecs.../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs$ $ dmstool -table oc4j_jspExec | grep time .../oc4j/PANDA/WEBs/PANDA/JSPs/processRequest.time 17 563 793 msecs$$ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0285F_searchForm.jsp/service.time 1 072 570 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/NSK/NSK003/NS0291F_Browse.jsp/service.time 2 942 655 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2261F_Edit.jsp/service.time 3 151 184 msecs.../oc4j/PANDA/WEBs/PANDA/JSPs/SKD/d001/D2260F_Browse.jsp/service.time 4 847 408 msecsrhoracle.voava.lv*J2EE01-/u01/app/oracle/product/J2EE01/Apache/Apache/conf$

DMS Sensors Classification

Informative sensors – reflect a configuration information and some system wide statistics

ohs_child, ohs_responses, JVM, oc4j_ear, oc4j_opmn, oc4j_task, JDBC_Driver , JDBC_Connection

Average performance sensors – reflect response time totals by modules, applications, etc.

ohs_server , ohs_module, ohs_virtualHost, oc4j_context, oc4j_web_module, oc4j_jspExec

Units performance sensors – reflect response time statistics for each unit

oc4j_servletoc4j_jsp(threadsafe=true)

Collecting and Reflecting DMS Statistics

DMS Sensors statistics is incrementalMake periodic snaps of statistics for further analysisDevelop interface to access statistics easily

Graphical interface for system wide statistics

Textual interface for units (JSP/Servlet) response time analysis

Collecting and Reflecting DMS StatisticsSYS:PROD> @sp_jspCompleted Snapshots

Snap Id Snap Started

------- -------------------530 2005-02-03 12:33:26531 2005-02-03 13:03:24532 2005-02-03 13:33:27549 2005-02-03 14:03:32

Specify the Begin and End Snapshot Ids~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Enter value for begin_snap: 532Enter value for end_snap: 549Saved in c:\temp\532_549.log

Snap Id Snap Time ------- ------------------

Begin Snap: 532 2005-02-03 13:33:27End Snap: 549 2005-02-03 14:03:32Elapsed: 30.08 (mins)

JSP ordered by Response time

JSP Name Response Execution Average(sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ----------../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08

SYS:PROD>

http://pmon.alise.lv/cacti/graph.php?local_graph_id=300&rra_id=all

Monitor All the Tiers of the System

Do not forget to monitorOS and HW resources utilizationNetworkDatabase



Instrument Your Code

AskTom.oracle.comhttp://asktom.oracle.com/pls/ask/f?p=4950:8:16675873436862412624::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12836314571537

…Another example is asktom.oracle.com. Just go there and click on any article from the home page. You'll see a URL similar to:

http://asktom.oracle.com/pls/ask/f?p=...::NO::

If you simply type over that URL and replace the word NO with YES - you'll see the same page but with lots and lots of state/timing information dumped into it.…

OC4J Internals & BC4J




Java native



JMSServerPORT: 3201




ONS

OC4JMonitor Thread

Thread

Thread

AJPConnection

ListenerPO

RT:

300

1


Signal Dispatcher


Reference Handler

Finalizer

Java native


Signal Dispatcher


Reference Handler

Finalizer

Memory Structures

Http Session List ...

Application objects

...


...


AJP 1.3 ONet

new req

3101

3201

JBO Connection poolJBO Application Module PoolJBO Application Module PoolJBO Application Module PoolJBO Application Module Pool

JBO View Objects

Usage Considerations

Most information is available under JDevelopercoverUse development version of BC4JDeploy BC4J.EAR to your OC4J component (do not use home component)There is the BC4J monitoring page

Useful Information Sources

0. Metalink Note:273350.1 BC4J configuration tuning tips1. Metalink Note:283476.1 BC4J (ADF BC) Configuration Tuning Tips2. Metalink Note:238634.1 JBO / BC4J Parameters And Defaults3. Metalink Note:231141.1 BC4J Row Spillover and tuning4. How to Performance Tune an ADF Business Components (ADF BC) Application

September 23, 2004Steve Anderson and Michael Gantmanhttp://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html5. Understanding Application Module Pooling Concepts and Configuration Parameters

Author: Steve Muench, BC4J Development TeamDate: August 5, 2004http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html6. Getting Application Module Pool Statistics to Aid with Size TuningBy Steve Muenchhttp://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html7. Overview of Temporary Tables Created By BC4JAn Oracle technical white paperApril 2002http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html8. How To Support Dynamic JDBC Credentials

An Oracle BC4J technoteVersion 3: April 23rd 2002

http://www.oracle.com/technology/products/jdev/howtos/bc4j/howto_dynamic_jdbc.html9. BC4J/JClient Performance StudyWritten by Steve Muench, Oracle CorporationJanuary, 2004http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html10. Dive into BC4J and ADFTips and tricks from Steve Muench on using Oracle's J2EE frameworks, JDeveloper Java IDE, and XML technologies...http://radio.weblogs.com/0118231/


http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html

http://www.oracle.com/technology/products/jdev/tips/muench/ampooling/index.html

http://radio.weblogs.com/0118231/stories/2004/05/10/gettingApplicationModulePoolStatisticsToAidWithSizeTuning.html

http://www.oracle.com/technology/products/jdev/htdocs/bc4j/bc4j_temp_tables.html

http://www.oracle.com/technology/products/jdev/tips/muench/jclientperf/index.html

http://radio.weblogs.com/0118231/

Performance Tuning Parametersjbo.pers.max.active.nodes=1000jbo.pers.max.rows.per.node=1000jbo.doconnectionpooling=truejbo.poolminavailablesize=0jbo.poolmaxavailablesize=600jbo.poolmonitorsleepinterval=600000jbo.poolmaxinactiveage=600000jbo.ampool.monitorsleepinterval=300000jbo.ampool.maxinactiveage=600000

JBO Parameter lookup order:- Set programmatically or configuration in bc4j.xcfg file)- Applet tags- -D flags (or oc4j.properties file) - BC4J.properties file in current working directory of Java VM- /oracle/jbo/BC4J.properties resource in the runtime class path- /oracle/jbo/server/jboserver.properties resource (by default found inside bc4jmt.jar)- /oracle/jbo/common/Diagnostic.properties resource (by default found inside bc4jmt.jar)- System defined default as a fallback

Summary

Summary

IntroductionUsage considerationsOC4J InternalsOC4J TroubleshootingOC4J Performance tuningBC4J

Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)

Documents

Transcript of Velikanovs - Performance Tuning and Troubleshooting for Oracle OC4J (Slides - Long Version)