Taming the OACore JVM Session ID - OATUG
Transcript of Taming the OACore JVM Session ID - OATUG
Session ID:
Prepared by:
Remember to complete your evaluation for this session within the app!
10886Taming the OACore JVM
April 10, 2019
Ahmed Alomari
Cybernoor Corporation
cybernoor.com
Agenda
• Architecture
• JVM Tuning
• Java Object Cache (JOC)
• Applications Connection Pool
• OA Framework
• Thread Dumps
• Heap Dumps
• Q & A
Architecture
R12.2 Architecture
R12.2 Architecture
• R12.2 uses Oracle Fusion Middleware (11gR1)
• WebLogic Server (WLS) -10.1.3.6
• WebLogic JSP Compiler and Engine 11.1.1.9
• Dual File System (FS1 & FS2) – Run and Patch Filesystems
• ADOP (online Patching) utility
• 64 bit Java Support
– Can deploy Managed servers with larger heaps such as 8GB
– Useful for components with large memory footprints
• Oracle Configurator
• Reporting
• Large number of online users
• New Middle Tier Code level checker (MT –ETCC)
Terminology
• JVM – Java Virtual Machine
• JRE – Java Runtime Environment
• JDK – Java Development Kit
• Java SE – Java Standard Edition
• Java EE – Java Enterprise Edition
• Java ME – Java Micro Edition
• Hotspot – Oracle (Sun’s) VM Implementation
➢Server VM and Client VM
• JNI - Java Native Interface (Native Applications)
Terminology
• JSP - Java Server Pages
➢Web pages which can embed Java, JavaScript, and/or HTML. Usually compiled into a servlet.
➢Located in $OA_HTML
• JDBC – Java Database Connectivity
➢ Interface for SQL execution for Java Applications.
• Servlets – Server side Java code used to interact with Web based applications.
➢Configurator
➢ iPayment
➢AppsLogin
Terminology
• Applet – Client side Java code such as Java code which runs inside a browser [JRE Plug-in or Java Web Start (JWS)]
• Oracle WebForms
• javac – Compiler used to compile java source code files into class files (i.e. byte code).
• JAR – Java Archive
➢Typically used to package an Application including class files, configuration files, properties, manifest, etc..
• Manifest – Descriptor file
• JAR Signing
➢Attach a digital signature to the JAR file in order to improve security.
Terminology
• Class file
➢Well-defined format of compiled Java source (i.e. byte code).
• Package
➢Allows class files to be organized into a hierarchy
• java.lang
• java.util
• oracle.apps.fnd.common
• Class hierarchy
➢oracle.apps.fnd.common.WebAppsContext →
• $JAVA_TOP/oracle/apps/fnd/common/WebAppsContext.class
Terminology
• JDK Vendors
➢Oracle – Linux, Solaris, Windows, macOS
➢ IBM – AIX, Linux, z/OS, Windows
➢HP – HP UX
➢JRockit - Linux, Solaris, Windows
➢Extended Support ended December 2018
➢Oracle JVM – DB Server platforms.
Terminology
• Web Container
➢A Java Application Server which provides
deployment and runtime services
➢JSP engine
➢Pooling Services
➢Thread Pool
➢JDBC Connection Pool
➢Security
➢Logging
➢Transaction Management
➢Cluster Services
Terminology
• Web Container
➢Oracle Containers for J2EE (OC4J)
➢Oracle WebLogic
➢ IBM WebSphere
➢Apache Tomcat
➢RedHat JBoss
➢GlassFish
Architecture
Architecture• HotSpot VM / Mixed-Mode VM
➢ Compiles “hot” classes using thresholds and other criteria into native machine code.
• Oracle HotSpot VM provides Client and Server compilers.
• -server• -client
➢ Uses background thread (CompilerThread0)• To perform compilation.
➢ Switch -XX:+PrintCompilation can be used to report compilation activity including method names.
• java.lang.String::charAt (33 bytes)• java.lang.String::hashCode (60 bytes)
Architecture
Architecture
.
• Java Object Life Cycle
Architecture
• Finalizers
➢ Object has been declared unreachable.
➢ The finalize() method on the object is invoked.
Architecture
• Leaks
try {
webAppsCtx = new WebAppsContext(l_dbc);
requestType = httpservletrequest.getParameter("requestType");
String jvm_url =
webAppsCtx.getProfileStore().getProfile("APPS_FRAMEWORK_AGENT");
webAppsCtx.createAnonymousSession();
connection = webAppsCtx.getJDBCConnection();
. . . . . . .
} catch (Exception e)
{
MessageErrorHandler(e);
}
Architecture
• Garbage Collection
➢ Process of re-claiming memory for objects which are
either no longer reachable (i.e. strong reference cleared)
or object reference type permits collection.
➢ Memory leaks in Java are due to references which are not
cleared such as adding objects to a collection without
removing the entry or setting the reference to null.
• Connection leaks
• Cursor leaks
• Data Structure leaks (hash tables, hash maps, other
collection types, etc..)
➢ For example, cursor leaks are fixed by calling the close()
method on the statement object (i.e. reference is cleared
therefore making the object collectable).
Architecture
• Garbage Collection
➢ The JVM provides various collectors based on the heap
being collected as well as the JVM configuration options.
➢ Garbage Collection can be monitored via the following
switches:
• -verbosegc
• Enables GC messages
• -XX:+PrintGCDetails
• Records heap level details of GC event
• -Xloggc
• Redirects GC messages to a dedicated log file
• Can use tools to open the GC log file and plot the data
points.
Architecture
• Collectors➢ Serial Collector (-XX:+UseSerialGC)
➢ Parallel New Space Collector
• -XX:+UseParallelGC / -XX:ParallelGCThreads
➢ Concurrent Collector (CMS)
• -XX:+UseConcMarkSweepGC
• Tries to minimize GC Pause times for stop the world collections.
• CMS optimal for old generation collections as well as Applications which require a large permanent footprint (i.e. large data caches).
• Much more efficient than the conventional Full GC.
➢ Garbage First (G1) Garbage Collector – 1.7+
• Successor to CMS
• -XX:+UseG1GC
• Utilizes compaction
• Divides heap into regions rather than by generation.
Architecture
• Java Heaps (-Xmx/-Xms)➢ Xmx (maximum heap size)
➢ -Xms (minimum heap size at startup)
• Heap segment is memory mapped or a shared memory segment is
created if large pages is used.
➢ Young Generation
• Eden (Nursery)
• NewRatio or NewSize/MaxNewSize
• Survivor Space
• Can be tuned via SurvivorRatio
➢ Tenured Generation (Old Generation)
➢ Meta Space heap (Permanent Generation)
• MaxMetaspaceSize (formerly PermSize and MaxPermSize)
• -XX:MetaspaceSize / -XX:MaxMetaspaceSize
• Code Cache (classes)
• Constants
• Intern strings
Architecture
• Java Heaps (-Xmx/-Xms)
➢ Total OS memory size will be the sum of the value of:
• Xmx + MaxMetaspaceSize + Thread Stacks + Control
Structures
➢ Gather and review verbose GC data to optimally size the
JVM heaps.
• Need to understand the permanent memory footprint (i.e.
tenured generation footprint after caches are loaded and
steady state achieved).
• Need to understand object creation and allocation rates.
• Need to understand lifespan of transient objects.
• Need to understand user level concurrency traffic as well as
end-user response time SLAs.
• Choose optimal collector based on above data points.
Infrastructure Tuning
Infrastructure Tuning
• Ensure kernel/shell limits are set sufficiently as per
the load.
• Low settings can result in JVM errors, hangs, or
OutOfMemoryError(s).
• /etc/security/limits.confaebizprd hard core unlimited
aebizprd hard nofile 131072
aebizprd hard nproc 131072
aebizprd soft core unlimited
aebizprd soft nofile 131072
aebizprd soft nproc 131072
Infrastructure Tuning
• Ensure optimal performance of NAS device and the
shared application filesystem (NFS)
• OHS and JVM logs and output files (runtime)
• JVM performance is impacted due to NAS latency.
• Utilize NFS V4 for application file system mounts
• Ensure NAS device supports V4
• nas-app-prod:/app01 on /app01 type nfs4
(rw,relatime,vers=4.0,rsize=64k,wsize=64k,namlen=2
55,soft,nordirplus, local_lock=none)
Infrastructure Tuning
• Monitor CPU and memory utilization on the
application tier hosts.
• Top processes (by CPU and memory)
• Leverage EM grid or install OSWatcher.
• Monitor Entropy pool to ensure sufficient availabilityIf Entropy values are low or exhausted, the JVM can hang
with threads blocked waiting on random key generation.
applmgr> cat /proc/sys/kernel/random/entropy_avail
3520
applmgr> cat /proc/sys/kernel/random/entropy_avail
118
Might need to switch to /dev/urandom or use options to increase
Entropy availability.
JVM Tuning
Application Tier - JVM
• Ensure you are running a current version of the JDK as
well as current patch set.
JDK
Version
Current
Patchset
MetaLink Note Applications
Version
6.0 Update 211
(28414647)
2244851.1
1459546.1 (12.2)
Extended Support ended 12/2018
7.0 Update 211
(28916517)
1467892.1 (12.1)
1530033.1 (12.2)
12.1.3 /12.2.X
8.0 Update 202
(28916765)
393931.1 (R12) JRE
Application Tier - JVM
• JDK 7.0 New Features
➢ Invokedynamic instruction for dynamic languages.
➢ Garbage-First (G1) Garbage Collector.
➢ Improved Class loading
➢ Parallel ClassLoader
➢ try with block
➢ java.lang.AutoCloseable
➢ Tiered Compilation
➢ -XX:+TieredCompilation
➢ Compressed Ordinary Object Pointers
➢ -XX:+UseCompressedOops
➢ NUMA Aware Collectors
➢ -XX:+UseNUMA
➢ Switch statement can reference String Objects as expressions.
➢ ThreadLocalRandom
➢ Improve performance of random number generators.
Application Tier - JVM
• JDK 1.5 JVM monitoring tools
➢ jps
➢ jstat
➢ jconsole
➢ jmap (utility to obtain heap information)
• JDK 1.6
➢ jhat (Memory Analysis tool)
➢ jmap enhanced to specify heap dump file name/path.
➢ JDK 1.7
➢ jcmd (diagnostic commands utility)
➢ jvisualvm (JVM monitoring tool)
➢ JDK 1.8
➢ jmc (Java Mission Control)
Application Tier - JVM
➢ jvisualvm (JVM monitoring tool)
Application Tier - JVM
➢ jvisualvm (JVM monitoring tool)
Application Tier - JVM
➢ jmc (Java Mission Control)
Application Tier - JVM
➢ jmc (Java Mission Control) – Flight Recorder
Application Tier - JVM
➢ jmc (Java Mission Control) – Flight Recorder
Application Tier - JVM
• Set initial and max. heap sizes appropriately. Sizes
below are starting points and should be adjusted as per
the load and footprint.
Usage Model Recommended Starting Heap
Size
Small Xms/Xmx=1G
Medium Xms/Xmx=3-4G
Large Xms/Xmx=6-8G
Application Tier - JVM
• Add sufficient managed servers across the app-tier nodes to handle the load.
• Minimum two OACore managed servers per app tier host.
• Ensure sufficient CPU and memory exist to support two managed servers plus other processes.
• Use a hardware load balancer (VIP) to load balance the requests across the OACore managed servers.
• Refer to MOS 1905593.1.
perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl \
ebs-create-managedserver -contextfile=<CONTEXT_FILE> \
-managedsrvname=oacore_server4 -servicetype=oacore \
-managedsrvport=21205 -logfile=<APPLRGF>/TXK/addMS_oacoreserver4.log
Application Tier – Logging
• Set logging level to Notice (Severity Levels)• WebLogic Admin Console for oacore_server<n>
• Logging tab
Application Tier – Logging
• Set logging level to Notice (Severity Levels)
➢ Review JVM log files for exceptions or errors
• 12.2: $EBS_DOMAIN_HOME/servers/oa*/logs/
• oacore_server2.out
• oacore_server2.log
• OPMN logs:• $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn
• 12.1: $INST_TOP/logs/ora/10.1.3/opmn/*oacore*• oacore_default_group_<n>
• oacorestd.out
• oacorestd.err• log4j:
• log4j.rootLogger=error, stdout, R
Application Tier – JVM
• Enable verbose GC
➢ $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml (12.1)
• -verbose:gc
• -XX:+PrintGCTimeStamps
• -XX:+PrintGCDetails
➢ Direct verbosegc output to a specific file
• 12.1: $INST_TOP/admin/scripts/java.sh
• -Xloggc:$JVMLOGDIR/oacore.gc.$$
• 12.2: WebLogic Admin Console
• Configuration->Server Start->Arguments
Application Tier - JVM
263888.839: [GC [PSYoungGen: 378131K->9061K(393280K)] 1175648K->807594K(1212480K), 0.0249310 secs] [Times:
user=0.19 sys=0.01, real=0.02 secs]
264062.955: [GC [PSYoungGen: 385509K->2107K(392768K)] 1184042K->807774K(1211968K), 0.0263420 secs] [Times:
user=0.24 sys=0.00, real=0.02 secs]
264062.982: [Full GC[Unloading class sun.reflect.GeneratedMethodAccessor443]
[PSYoungGen: 2107K->0K(392768K)] [PSOldGen: 805666K->247375K(819200K)] 807774K->247375K(1211968K) [PSPermGen:
54677K->53972K(71680K)], 2.6600860 secs] [Times: user=2.63 sys=0.03, real=2.66 secs]
264164.928: [GC [PSYoungGen: 376448K->5127K(393600K)] 623823K->252503K(1212800K), 0.0142250 secs] [Times:
user=0.10 sys=0.00, real=0.01 secs]
264273.025: [GC [PSYoungGen: 382215K->2381K(393088K)] 629591K->252854K(1212288K), 0.0193010 secs] [Times:
user=0.13 sys=0.00, real=0.02 secs]
264393.122: [GC [PSYoungGen: 379469K->11043K(392320K)] 629942K->262082K(1211520K), 0.0258080 secs] [Times:
New Space
GC
Elapsed Time
Size Before
GC Size After
GC
Perm. Gen
Type of GC
• Review verbose GC data
Application Tier – JVM
• Jstat• S0C Current survivor space 0 capacity (KB).
• S1C Current survivor space 1 capacity (KB).
• S0U Survivor space 0 utilization (KB).
• S1U Survivor space 1 utilization (KB).
• EC Current eden space capacity (KB).
• EU Eden space utilization (KB).
• OC Current old space capacity (KB).
• OU Old space utilization (KB).
• PC Current permanent space capacity (KB).
• PU Permanent space utilization (KB).
• YGC Number of young generation GC Events.
• YGCT Young generation garbage collection time.
• FGC Number of full GC events.
• FGCT Full garbage collection time.
• GCT Total garbage collection time.
• jstat -gc 298291; jstat -gcutil 298291
• S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC
FGCT GCT
• 294400.0 303616.0 0.0 141953.0 6370816.0 1796791.0 13981696.0 2371051.0 1048576.0 194275.0 49
8.497 0 0.000 8.497
• S0 S1 E O P YGC YGCT FGC FGCT GCT
• 0.00 46.75 28.20 16.96 18.53 49 8.497 0 0.000 8.497
Application Tier – Session Timeout
• Set the served session timeout to 30 minutes
➢12.2: Set the profile “ICX:Session Timeout”
➢12.1: orion-web.xml
• <session-timeout>30</session-timeout>
• AutoConfig: s_oc4j_sesstimeout
• WebLogic: timeout-secs (1800)
➢Larger timeout values increase overall JVM
memory footprint.
Application Tier – Web Applications
• Ensure the users are trained to use the Logout or Home global buttons
when completing their transactions.
• Ensure users do not use the browser close (“x”) link.
• Logging out gracefully releases the memory and corresponding
resources (i.e. connections, etc..)
• Avoids memory leaks and timeout based invalidation.
Java Object Cache (JOC)
Application Tier – JOC
• Java Object Cache (JOC) is an Middleware Component
which provides a caching framework for Java based
applications.
• E-Business Suite uses JOC for many of the core caches:
• Page Metadata Cache (MDS)
• FND Caches
• Menus
• Responsibilities
• Profiles
• Function Security
Application Tier - JOC
• JOC Provides local caching (i.e. per JVM) as well as
distributed caching (i.e. all JVMs)
• Java System Property LONG_RUNNING_JVM
specifies the caching behavior
• TRUE – Distributed Caching
• FALSE – Local Caching
• LONG_RUNNING_JVM is set in oacore properties file
via the AutConfig context variable
s_long_running_jvm
Application Tier - JOC
• JOC Distributed Caching
– Endpoint Receiver thread is created on all OACore
JVMs, GSM JVMs, and XML Services JVMs
– Receives cache update messages
– Listens on Cache Port
• s_java_object_cache_port (AutoConfig)
• Profile: JTF_DIST_CACHE_PORT
• Creates several threads in each JVM
– EndPointConnection
– Ack Processor
– Packet Sender
– CL Packet Processor
Application Tier - JOC
• JOC Distributed Caching
• Endpoint Receiver thread receives cache update messages.
"EndPointConnection CL-65 to DI-3 at [10.16.240.16:22128]" daemon prio=10
tid=0x00007fd691dcf000 nid=0x3fe5 runnable [0x00007fd677dfc000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:153)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000007f55b4600> (a java.io.BufferedInputStream)
at oracle.ias.cache.groupv2.HttpHandler.read(HttpHandler.java:167)
- locked <0x00000007f55b4648> (a oracle.ias.cache.groupv2.HttpHandler)
at
oracle.ias.cache.groupv2.EndPointConnection.receive(EndPointConnection.java:383)
at oracle.ias.cache.groupv2.EndPointConnection.run(EndPointConnection.java:214)
Application Tier - JOC
• Monitor JOC log file for exceptions/errors
➢ $APPLRGF/javacache.log*
➢ Memory leaks and JVM instability can occur if JOC loses
contact with other JVMs.
➢ NoClassDef errors involving the Profiles class can occur
during user login if JOC errors occur.
• Check firewall configuration/rules to account for the Cache port
as well as external nodes (DMZ).
Application Tier - JOC
• Java Object Cache Full Exception:
oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full.
at oracle.ias.cache.CacheHandle.findObject(CacheHandle.java:1682)
at oracle.ias.cache.CacheHandle.locateObject(CacheHandle.java:1120)
at oracle.ias.cache.CacheAccess.get(CacheAccess.java:877)
at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:771)
at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802)
at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624)
at oracle.apps.fnd.cache.AppsCache.get(Unknown Source)
at
oracle.apps.fnd.profiles.Profiles.getProfileOptionValue(Profiles.java:1714)
• Increase JOC Cache sizes via javacache.xml underneath
$EBS_DOMAIN_HOME
• Change From: max-objects="5000" max-size="10“
• Change To: max-size=”1000″ max-objects=”5000000″
Apps Connection Pool
Application Tier – Connection Pool
• Apps Connection Pool (dbc configuration)
➢ Enable sanity checks in production environments
• FND_JDBC_USABLE_CHECK=true
➢ Start with a reasonable default such as 300.
• FND_JDBC_MAX_CONNECTIONS=300
➢ Tune FND_JDBC_MAX_CONNECTIONS as per the amount of user concurrency (per JVM).
➢ Apps automatically decays idle connections and resizes the pool
Application Tier – Connection Pool
• Apps Connection Pool – Enable Test Connection• WebLogic->Configuration->EBSDataSource->Connection Pool-
>Advanced
Application Tier – Connection Pool
• Apps Connection Pool (dbc configuration)
➢ Apps Supports the use of Services for RAC load balancing or
failover.
• dbc parameter APPS_JDBC_URL
APPS_JDBC_URL=jdbc\:oracle\:thin\:@(DESCRIPTION\=(LOAD_BALANCE\=YES)(FA
ILOVER\=YES)(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=tcp)(HOST\=db1host)(POR
T\=24601))(ADDRESS\=(PROTOCOL\=tcp)(HOST\=db2host)(PORT\=24601)))(CONNE
CT_DATA\=(SERVICE_NAME\= ebizprodOA)))
Application Tier – Connection Pool
• Apps Connection Pool Monitoring (12.1)
➢ Use the AOL/J Database Connection Pool
Status Page to monitor the connection pool
status.
• System Administration Responsibility
• AOL/J Database Connection Pool Status
(Function)
Application Tier – Connection Pool
• Apps Connection Pool Monitoring (12.1)
➢ Use the AOL/J Database Connection Pool
Status Page to monitor the connection pool
status.
• Lists DBC configuration parameter settings.
• Connection Statistics
• Locked Connections
• Available Connections
• Closed Connections
• Allows drill-downs to determine source of the
connection.
Application Tier – Connection Pool
• AOL/J Database Connection Pool Status Page (12.1)
Application Tier – Connection Pool
• AOL/J Database Connection Pool Status Page (12.1)
Application Tier – Connection Pool
• WebLogic Connection Pool Monitoring (12.2)
• WebLogic Console->Data Source->Monitoring-
>Statistics
Application Tier – Connection Pool
• Process Identification (GV$SESSION)
• Allows you to map the JDBC session from v$session to a particular JVM
process.
SID MACHINE PROCESS MODULE LOGON
---- ------------------------- ------------ ------------------ -----------------
41 aptier1.us.oracle.com 23806 JDBC Thin Client 04/08/19 17:26:43
42 aptier2.us.oracle.com 2723 JDBC Thin Client 04/08/19 17:27:01
43 aptier3.us.oracle.com 4201 JDBC Thin Client 04/08/19 17:28:15
44 aptier1.us.oracle.com 23807 JDBC Thin Client 04/08/19 17:29:17
. . . . . . .
. . . . . . .
aptier1{apps_a}-> ps -ef | grep 23806
apps_a 23806 23804 53 17:26:40 pts/20 8:55 /R12/fs1/EBSapps/comn/util/jdk64/bin/java -
Dweblogic.Name=oacore_server2 -DCLIENT_PROCESSID=24314 -server -verbose:gc . . . . . . .
Application Tier – Connection Pool
• Monitor the number of JDBC connections for both
APPS and APPLSYSPUB to check for leaks or
excessive connection usage.
• General rule of thumb is that the total number of
connections should not exceed 2*<peak number of
users>.
➢ Keep in mind that each JVM will create ~10 connections for
background threads and bootstrapping at JVM startup time.
• Query GV$SESSION and group by module to
determine which modules are consuming the
connections.
Application Tier – Connection Pool
• Ensure Inactive Connection Timeout is set to zero (12.2).
• Only enable if debugging connection leaks, and reset to
zero after collecting debug logs.
• A non-zero values can actually create additional connection
leaks due to the WebLogic Shrink activity.
Application Tier – Connection Pool
• Ensure Deadman detection is enabled in sqlnet.ora for
the DB listener
• $TNS_ADMIN/sqlnet.ora
• SQLNET.EXPIRE_TIME= 10
• If not enabled, connections can hang due to TCP timeouts,
firewall rules, abnormal connection terminations, etc..
• Ensure sqlnet.ora entry is present for all relevant listeners
used by the application:
• DB listener (AutoConfig instance listener directory)
• GI/ASM listeners
• Scan listener
Application Tier – Connection Pool
• 12.2 Connection Leaks
• Check MOS for latest patches
• Shipping: Patch 27508480
• WebADI: MOS 2141923.1
• Payroll: Patch 22684327
• FND: Patch 26599059 and 28831943
• Ping Database checks: Patches 25969099 and 20141119
OA Framework
68
Database
OA Framework
JDBC
Statement Cache
Apps
Connection Pool
MDS
Metadata Cache
BC4J
Metadata Cache
Data Cache
UIX
HTML Generation
JSP Engine
OA.jsp / OAP.jsp
Module Code
Controller Object
http
OA Framework Architecture
OA Framework Architecture
• Java Server Page Source (JSPs)
➢$OA_HTML
• Java Server Page Cache
➢$COMMON_TOP/_pages
• Java Classes
➢$JAVA_TOP
Java Server Pages (JSP)
• Java Server Page (JSP) Execution
➢Upon JSP invocation, the JSP page cache is
examined.
• If a cache entry exists, the JSP class is executed.
• If a cache entry does not exist, the JSP is
processed as follows:
• Translation (.jsp -> .java)
• Compilation (.java -> .class)
Java Server Pages (JSP)
• OA.jsp
➢Centralized JSP which executes OA FWK based
pages such as the Home Page.
➢Uses redirection to render the relevant page.
• RF.jsp
➢Centralized JSP which is used to execute standard
web functions such as:
• Launching Oracle Forms
• Launching Certain Web pages
OA Framework Architecture
• User navigates to the OA FWK Page such as iExpenses
or iProcurement
➢Page Metadata is read and loaded into the MDS
Cache
➢Controller Page Object is invoked
➢Controller Page Object invokes server side
components such as the Entity Object or View Object.
• About this Page link can be used to display page
components.
Application Tier – About this Page
• About this Page link
Application Tier – About this Page
• About this Page link (page definition details)
Application Tier – About this Page
• About this Page link (personalization details)
Application Tier – About this Page
• About this Page link – Technology Components
Pool Monitor
Application Tier – Pool Monitor
• Pool Monitor can be used to monitor the Framework Applications as well as the JVM utilization
• Application Module Pool
• Memory Utilization
• Servlet Sessions
• Navigation Path: Home>Settings->Diagnostics->Show Pool Monitor.
Application Tier – Pool Monitor
• Pool Monitor Home Page
Application Tier – Pool Monitor
• Pool Monitor – Application Module Pool
Application Tier – AM Pooling
• Application Module (AM) Pool
➢AM Pooling can be enabled/disabled via the profile
“FND: Application Module Pool Enabled”
➢ If AM Pooling is disabled, the values for Creations
and Removals on the AM Pool Page of Pool Monitor
should be close.
➢ If AM Pooling is enabled, monitor Check-ins and
Check-Outs as well as the Available and Unavailable
counters to ensure AM instances are not being
leaked.
Application Tier – WebLogic Console
• Managed Server Performance page
➢Monitor JVM metrics
➢Heap Utilization
➢Garbage Collections
➢Thread information
➢Stuck Threads
Application Server – WebLogic Console
Application Server – WebLogic Console
Application Server – WebLogic Console
WebLogic Tuning
• Monitor Thread and Connection Pools
• Thread Count
• Maximum number of connections
• JDBC Ping Check Tuning
• Consider disabling if connections are stable and not hard-closed due to
firewall or inactivity termination.• Test Connections On Reserve
• Test Frequency
• Enable JDBC Statement Caching• Test thoroughly and watch for cursor leaks
• Utilize Hardware Load Balancers in place of Web Server clustering
• Monitor Heap utilization and GC traffic
WebLogic Tuning
• Monitor Long Running and Stuck Threads• Stuck Thread Max Time
• Stuck Thread Timer Interval
Java Thread Dumps
Application Tier – Thread Dumps
➢ Thread dumps show the stack trace for each thread
that is alive.
➢ To generate a thread dump, identify the process id of
the JVM process.
• Then, execute the following:
-kill –QUIT <pid>
➢ Thread dumps show deadlocks which can cause the
JVM to stop responding to user requests. If that’s the
case, the JVM should be restarted.
➢ Helps identify deadlocks, infinite loops, long running
queries submitted by the user, etc..
Application Tier – Thread Dumps
➢ Thread dumps show the stack trace for each thread
that is alive.
➢ To generate a thread dump, identify the oacore
Managed Server name and click on the Dump Thread
Stacks button.
Application Tier – Thread Dumps2019-04-09 14:58:31
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.201-b11 mixed mode):
"Worker-5" daemon prio=10 tid=0x00007f17d402b000 nid=0x5e751 in Object.wait() [0x00007f1712bfd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)
"Worker-4" daemon prio=10 tid=0x00007f16f8023000 nid=0x5e750 in Object.wait() [0x00007f17397fc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)
"Worker-2" daemon prio=10 tid=0x00007f175d85a000 nid=0x5e6cb in Object.wait() [0x00007f1713ffe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)
Application Tier – Thread Dumps
Found one Java-level deadlock:
=============================
"Thread-65103":
waiting to lock monitor 0x0813e334 (object 0xb4006318, a
oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock),
which is held by "Thread-65088"
"Thread-65088":
waiting to lock monitor 0x0813e374 (object 0xb3fa1418, a org.apache.jserv.JServSession),
which is held by "Thread-65103"
Java stack information for the threads listed above:
===================================================
"Thread-65103":
at oracle.apps.fnd.framework.webui.OAHttpSessionCookieImpl.timeout(OAHttpSessionCookieImpl.java:607)
- waiting to lock <0xb4006318> (a oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock)
at oracle.jbo.http.HttpSessionCookieImpl.timeout(HttpSessionCookieImpl.java:146)
at oracle.jbo.http.HttpContainer.fireTimeout(HttpContainer.java:512)
at oracle.jbo.http.HttpContainer.timeout(HttpContainer.java:232)
at oracle.jbo.http.HttpContainer.valueUnbound(HttpContainer.java:300)
at org.apache.jserv.JServSession.invalidate(JServSession.java:240)
- locked <0xb3fa1418> (a org.apache.jserv.JServSession)
. . . . . . . .
"Thread-65088":
at org.apache.jserv.JServSession.getValue(JServSession.java:293)
- waiting to lock <0xb3fa1418> (a org.apache.jserv.JServSession)
at oracle.jsp.provider.JspUniversalHttpRequest.getSession(JspUniversalHttpRequest.java:408)
at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)
at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)
at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)
at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)
at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)
. . . . . . . .
Application Tier – Thread Dumps
• Java Thread dump output (standard out)
➢12.2:
$EBS_DOMAIN_HOME/servers/oacore_server<n>/lo
gs/oacore_server<n>.out
➢12.1:
• $INST_TOP/logs/ora/10.1.3/opmn/default_group~o
acore~default_group*
• What should you look for?
➢Deadlocked threads.
➢Threads stuck in an infinite loop.
• Check to see if the stack is changing.
Java Heap Dumps
Application Tier – Heap Dumps
• Java Heap Dumps
➢Useful when trying to debug memory leaks,
excessive footprint/garbage collection, and/or
OutofMemoryErrors.
➢There are two types of heap dumps:
• Summary (histogram of all class instances and
shallow size)
• Full heap dump (complete dump of the entire
Java heap) which includes old generation and
permanent generation.
-Can result in very large files depending on the
heap sizes.
Application Tier – Heap Dumps
• Java Heap Dumps – Summary Heap Dump
➢Summary (histogram of all class instances and
shallow size)
• Can be generated using jmap
• jmap -histo:live <pid>
• Can be generated using a thread dump (kill -3) if
the JVM switch (-XX:+PrintClassHistogram) is
present. The summary histogram will be rewritten
to the oacore*.stdout log files (by default).
Application Tier – Heap Dumps
• Java Heap Dumps - Summary Heap Dumpnum #instances #bytes class name
----------------------------------------------
1: 21437694 686006208 com.sun.java.util.collections.HashMap$Entry
2: 5553282 646005184 [C
3: 19853550 476485200 oracle.apps.cz.dio.FieldCache
4: 615172 218915592 [Lcom.sun.java.util.collections.HashMap$Entry;
5: 315197 143264864 [B
6: 5539474 132947376 java.lang.String
7: 1453342 81990664 [Ljava.lang.Object;
8: 1545022 61800880 java.math.BigDecimal
9: 341887 54284456 <constMethodKlass>
10: 2119195 50860680 java.lang.Long
11: 341887 43778960 <methodKlass>
12: 283528 40828032 oracle.apps.cz.dio.model.DbBomOptionClass
13: 29342 37480752 <constantPoolKlass>
14: 615172 34449632 com.sun.java.util.collections.HashMap
15: 1004869 32155808 java.sql.Timestamp
16: 1124467 26987208 com.sun.java.util.collections.ArrayList
17: 1120206 26884944 java.lang.Double
18: 839012 26848384 EDU.oswego.cs.dl.util.concurrent.ConcurrentHashMap$Entry
19: 600378 24015120 oracle.apps.cz.utilities.LongHashMap$Entry
20: 29342 23976200 <instanceKlassKlass>
Application Tier – Heap Dumps
• Java Heap Dumps – Summary Histogram
– Quick to generate (several seconds)
• -XX:+PrintClassHistogram (and kill -3) is much faster
than jmap because it uses the native signal handler to
dump the heap.
– Good place to start when debugging leaks as it can help
you narrow down which objects are growing and which
objects consume the most memory.
• Once you have narrowed down the suspect objects, a
full heap dump can be generated to identify the root
cause and examine the reference graph.
Application Tier – Heap Dumps
• Java Heap Dumps – Full Heap Dump– Dumps all live objects to a heap dump file usually in hprof
format.
– Can be generated using jmap• jmap -heap:format=b <pid>
– Trigger a heap dump when an OutofMemoryError occurs
• -XX:+HeapDumpOnOutOfMemoryError
• -XX:HeapDumpPath=<path>
– Use Memory Profiling tools to analyze the heap dump:
• jhat (part of JDK)
– Can also compare two heap dumps which will help narrow down the objects which continue to grow
• Eclipse Memory Analyzer (MAT)
• NetBeans IDE (Memory Profiler)
References
• https://docs.oracle.com/javase/8/docs
• http://www.oracle.com/technetwork/tutorials/tutorials
-1876574.html
• http://www.oracle.com/technetwork/java/index.html
• http://bugs.java.com/bugdatabase
• http://www.oracle.com/technetwork/java/javase/docu
mentation/index.html
• http://www.oracle.com/technetwork/java/javaseprod
ucts/mission-control/java-mission-control-
1998576.html
Session ID:
Remember to complete your evaluation for this session within the app!
Taming the OACore JVM
10886